3904 Commits

Author SHA1 Message Date
Aseem Bansal
d6263a6c0c
fix(ingest/s3): catch no such bucket exception instead of failing (#8549) 2023-08-02 19:21:39 +05:30
TusharM
e2919cfd06
feat(ingest/kafka-connect): add support for Confluent S3 Sink Connector (#8298) 2023-08-02 15:09:50 +05:30
Harshal Sheth
08a39c4b0d
fix(ingest/presto): fix presto on hive test failures (#8548) 2023-08-02 12:46:39 +05:30
Aseem Bansal
bb33f015ca
fix(ingest/s3): wrong sorting in case of multi-partition key (#8536) 2023-08-02 09:54:33 +05:30
Felix Lüdin
ea95e36c2b
fix(ingest): fix data platform urn in dataset_urn_to_key and dataset_key_to_urn (#8209) 2023-08-02 09:54:21 +05:30
Jarod Smilkstein
f51bd01a70
feat(ingest): add ability to read other method types than GET for OAS ingest recipes (#8303) 2023-08-02 09:54:09 +05:30
Kirill Popov
eec89a884a
feat(ingest): Add metabase database id to platform instance mapping (#8359) 2023-08-02 09:53:48 +05:30
Mayuri Nehate
e67f811034
feat(classification): allow parallelisation to reduce time (#8368) 2023-08-02 09:53:39 +05:30
zheyu001
bf47d65412
feat(ingest/presto-on-hive): enable partition key for presto-on-hive (#8380) 2023-08-02 09:53:27 +05:30
Gabe Lyons
843f82b943
feat(presto-on-hive): allow v1 fieldpaths in the presto-on-hive source (#8474) 2023-08-01 14:05:50 -07:00
Pedro Silva
a4a8182001
feat(cli): Adds ability to upload recipes to DataHub's UI (#8317)
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
2023-08-01 17:35:42 -03:00
Benjamin Dornel
2e2a6748ac
fix(ingest/json-schema): convert non-string enums to strings (#8479) 2023-08-01 19:35:40 +05:30
mohdsiddique
547e1f498e
feat(ingestion/tableau): support column level lineage for custom sql (#8466)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-01 19:35:18 +05:30
Harshal Sheth
66074341f7
test(ingest): test case statements with sql parser (#8437) 2023-08-01 19:34:48 +05:30
VISHAL KUMAR
ef3b9489aa
feat(ingest/vertica): performance improvement and bug fixes (#8328)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-01 19:34:35 +05:30
Aseem Bansal
09ef6e080c
fix(ingest): remove duplication of tags (#8532) 2023-08-01 19:31:26 +05:30
Harshal Sheth
d8b2397b93
fix(ingest): pin boto3-stubs in CI (#8527)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-07-31 19:48:05 -07:00
Tamas Nemeth
1a47a51f1b
fix(ingest/build): Fix sagemaker mypy and flake8 issues (#8530) 2023-07-31 16:13:07 +02:00
Harshal Sheth
89f23d3c36
chore(ingest): bump sqllineage and sqlparse (#8481) 2023-07-28 13:10:19 -07:00
Harshal Sheth
9718505fc7
fix(ingest): respect max_threads for ingestion reporter (#8521) 2023-07-28 13:09:32 -07:00
Harshal Sheth
99f1624ce7
docs(ingest/lookml): clarify connection map config (#8508) 2023-07-27 17:06:04 +05:30
Mayuri Nehate
b9060db515
chore(ingest/tableau): miscellaneous cleanup refractor (#8417)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-07-27 17:02:48 +05:30
Mayuri Nehate
5a3f91de53
docs(ingest/bigquery): add permissions to profile google drive backed… (#8490) 2023-07-27 17:01:39 +05:30
Harshal Sheth
2495b50f8c
ci: fix commandLine usage in build.gradle (#8510) 2023-07-26 11:12:22 +05:30
Harshal Sheth
c585a1bcc7
fix(ingest): hint at --update-golden-files option when tests fail (#8507) 2023-07-26 09:50:27 +05:30
Harshal Sheth
b12de099aa
docs: add docs on sqlglot lineage (#8482) 2023-07-25 10:48:37 -07:00
Harshal Sheth
eac003ccf4
feat(ingest): enable pipeline reporting by default (#8472) 2023-07-25 14:16:27 +05:30
Aseem Bansal
cc46729137
chore(ingest): add example of training metric/hyper parameters (#8491) 2023-07-25 13:53:16 +05:30
Andrew Sikowitz
bf9f380350
fix(ingest): Generate browse paths v2 for more sources; properly pass platform_instance (#8501) 2023-07-25 11:35:34 +05:30
Aseem Bansal
c0dbea8363
feat(ingest/elastic): reduce number of calls made (#8477) 2023-07-23 17:12:31 +05:30
Aseem Bansal
8fb5912978
fix(mysql): getting table name correctly (#8476)
Co-authored-by: Ellie O'Neil <oneile729@gmail.com>
2023-07-23 17:10:54 +05:30
Harshal Sheth
056d3619f0
feat(sdk): easily generate container urns (#8198)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-07-21 21:14:06 +00:00
mohdsiddique
bec018257c
fix(ingestion/powerbi-report-srever): handle requests.exceptions.JSONDecodeError (#8442)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-07-21 14:01:57 -07:00
Aseem Bansal
f4c0ed3aab
ingest(mysql): add storage bytes information (#8294)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-07-20 17:31:06 -04:00
Mayuri Nehate
f4fde21168
feat(ingest/nifi): add support for basic auth in nifi (#8457) 2023-07-20 14:12:33 -04:00
Harshal Sheth
d733363bed
chore(ingest): drop bigquery-beta and snowflake-beta aliases (#8451) 2023-07-20 14:05:25 -04:00
Harshal Sheth
47616ff093
feat(ingest/lookml): fail when nothing was produced (#8464) 2023-07-20 14:04:30 -04:00
Aseem Bansal
9df70d7355
ingest(elasticsearch): add basic profiling (#8351) 2023-07-20 08:25:30 +05:30
Ellie O'Neil
d734b2849e
feat(ingest/mysql): Add estimate row count for mysql (#8420) 2023-07-20 08:06:26 +05:30
Harshal Sheth
695de42a8c
chore(ingest): hide ignore old/new state options (#8438) 2023-07-19 16:46:32 +05:30
JifeiMei
0def0e5a2e
fix(GreatExpections): AssertionRunEventClass does not match the examp… (#8243) 2023-07-19 16:08:00 +05:30
Harshal Sheth
addf76c849
fix(ingest/bigquery): make sql parsing more robust (#8450) 2023-07-19 16:06:42 +05:30
Arun Vasudevan
cc2dc342c6
fix(ingestion-redshift): Fix Redshift ingestion logs (#8454) 2023-07-18 22:24:54 -04:00
Andrew Sikowitz
48c1dc820e
build(ingest/boto3): Update boto3-stubs to fix CI (#8452) 2023-07-18 21:29:50 +00:00
Andrew Sikowitz
20b3adb7b1
fix(ingest/snowflake): Add sqlglot as snowflake dependency (#8427) 2023-07-14 21:31:24 -04:00
Andrew Sikowitz
61611b7202
feat(ingest/snowflake): View CLL from sql parsing of view definition (#8419) 2023-07-14 18:50:32 -04:00
Andrew Sikowitz
f41f642eaf
build(ingest/boto3): Update boto3-stubs to fix CI (#8425) 2023-07-14 15:48:04 -07:00
Harshal Sheth
4fb77e4a25
fix(ingest): tweak ingestion exit codes (#8418) 2023-07-14 15:47:16 -07:00
Tamas Nemeth
a91c78cf31
fix(ingest/s3): fix test flakiness (#8416) 2023-07-14 00:42:00 +02:00
mohdsiddique
cbbe083731
fix(ingestion/powerbi): increment msal version (#8385)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-07-13 17:33:19 +05:30