1604 Commits

Author SHA1 Message Date
Mayuri Nehate
e285da3e75
feat(ingest/snowflake): tables from snowflake shares as siblings (#8531) 2023-08-24 10:23:07 -04:00
RChygir
43d48ddde4
feat(ingest/mssql): load jobs and stored procedures (#5363) 2023-08-24 14:48:03 +05:30
Alexander
bcef25acd3
feat(ingest/looker): Record observed lineage timestamps for Looker and LookML sources (#7735) 2023-08-24 14:47:04 +05:30
Andrew Sikowitz
22c35f1a23
fix(ingest/bigquery): Add config option to create DataPlatformInstance, default off (#8659) 2023-08-24 14:46:06 +05:30
Andrew Sikowitz
8141e2d649
remove(ingest/snowflake): Remove legacy snowflake lineage (#8653)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-08-23 15:57:46 -04:00
siddiquebagwan-gslab
8ee58af0c2
feat(ingestion/powerbi): support multiple tables as upstream in native SQL parsing (#8592) 2023-08-23 14:38:58 +05:30
Andrew Sikowitz
439cf4d7dc
test(ingest/vertica): Skip integration test failing CI; support arm Macs (#8694) 2023-08-22 16:27:46 -04:00
Alexander
c0addf6eef
feat(ingest/bigquery): add tag to BigQuery clustering columns (#8495)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-17 12:44:15 -04:00
Jinlin Yang
6748aecdc0
fix(ingest/s3): emit data_platform_instance aspect if the config has platform_instance (#8585) 2023-08-17 10:40:54 +05:30
kr_Deepankar
23ac9062fe
feat(ingestion/ldap): flag to ingest ldap users with email instead of username (#8606) 2023-08-16 10:33:18 +05:30
kr_Deepankar
94ce753bb0
fix(ingest/kafka): use SchemaReference properties instead of dict access (#8615)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-16 10:33:09 +05:30
Andrew Sikowitz
526e626146
feat(ingest): Add DataHub source (#8561)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-15 17:49:20 -04:00
Mayuri Nehate
ddcd5109dc
feat(ingest): allow relative start time config (#8562)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-14 17:48:06 -07:00
alplatonov
11fdfcf956
Fix(ingestion/clickhouse) move to two tier sqlalchemy (#8300)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-08-11 16:11:40 -04:00
alplatonov
b58f9bb396
Feat(ingest/ldap)fix list index out of range error (#8525)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-08-09 13:13:27 -04:00
Mayuri Nehate
b4e104f190
fix(ingest/snowflake): maintain qualified name casing, do not lowercase (#8574) 2023-08-04 10:43:22 -07:00
Aseem Bansal
dac89fb1fb
feat(ingest): allow lower freq profiling based on date of month/day of week (#8489) 2023-08-04 10:13:48 +05:30
mohdsiddique
6a36118b4f
feat(ingestion/snowflake): use user email-id in urn generation for top users stat (#8513)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-03 08:30:50 +05:30
mohdsiddique
05ef7db45e
fix(ingetion/mssql): convert dataset urns to lowercase (#8551)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-02 13:44:05 -07:00
TusharM
e2919cfd06
feat(ingest/kafka-connect): add support for Confluent S3 Sink Connector (#8298) 2023-08-02 15:09:50 +05:30
Harshal Sheth
08a39c4b0d
fix(ingest/presto): fix presto on hive test failures (#8548) 2023-08-02 12:46:39 +05:30
Aseem Bansal
bb33f015ca
fix(ingest/s3): wrong sorting in case of multi-partition key (#8536) 2023-08-02 09:54:33 +05:30
Jarod Smilkstein
f51bd01a70
feat(ingest): add ability to read other method types than GET for OAS ingest recipes (#8303) 2023-08-02 09:54:09 +05:30
Kirill Popov
eec89a884a
feat(ingest): Add metabase database id to platform instance mapping (#8359) 2023-08-02 09:53:48 +05:30
Mayuri Nehate
e67f811034
feat(classification): allow parallelisation to reduce time (#8368) 2023-08-02 09:53:39 +05:30
zheyu001
bf47d65412
feat(ingest/presto-on-hive): enable partition key for presto-on-hive (#8380) 2023-08-02 09:53:27 +05:30
Gabe Lyons
843f82b943
feat(presto-on-hive): allow v1 fieldpaths in the presto-on-hive source (#8474) 2023-08-01 14:05:50 -07:00
Benjamin Dornel
2e2a6748ac
fix(ingest/json-schema): convert non-string enums to strings (#8479) 2023-08-01 19:35:40 +05:30
mohdsiddique
547e1f498e
feat(ingestion/tableau): support column level lineage for custom sql (#8466)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-01 19:35:18 +05:30
Harshal Sheth
66074341f7
test(ingest): test case statements with sql parser (#8437) 2023-08-01 19:34:48 +05:30
VISHAL KUMAR
ef3b9489aa
feat(ingest/vertica): performance improvement and bug fixes (#8328)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-01 19:34:35 +05:30
Tamas Nemeth
1a47a51f1b
fix(ingest/build): Fix sagemaker mypy and flake8 issues (#8530) 2023-07-31 16:13:07 +02:00
Harshal Sheth
89f23d3c36
chore(ingest): bump sqllineage and sqlparse (#8481) 2023-07-28 13:10:19 -07:00
Harshal Sheth
9718505fc7
fix(ingest): respect max_threads for ingestion reporter (#8521) 2023-07-28 13:09:32 -07:00
Mayuri Nehate
b9060db515
chore(ingest/tableau): miscellaneous cleanup refractor (#8417)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-07-27 17:02:48 +05:30
Andrew Sikowitz
bf9f380350
fix(ingest): Generate browse paths v2 for more sources; properly pass platform_instance (#8501) 2023-07-25 11:35:34 +05:30
Aseem Bansal
8fb5912978
fix(mysql): getting table name correctly (#8476)
Co-authored-by: Ellie O'Neil <oneile729@gmail.com>
2023-07-23 17:10:54 +05:30
mohdsiddique
bec018257c
fix(ingestion/powerbi-report-srever): handle requests.exceptions.JSONDecodeError (#8442)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-07-21 14:01:57 -07:00
Mayuri Nehate
f4fde21168
feat(ingest/nifi): add support for basic auth in nifi (#8457) 2023-07-20 14:12:33 -04:00
Aseem Bansal
9df70d7355
ingest(elasticsearch): add basic profiling (#8351) 2023-07-20 08:25:30 +05:30
Ellie O'Neil
d734b2849e
feat(ingest/mysql): Add estimate row count for mysql (#8420) 2023-07-20 08:06:26 +05:30
Harshal Sheth
4fb77e4a25
fix(ingest): tweak ingestion exit codes (#8418) 2023-07-14 15:47:16 -07:00
Tamas Nemeth
a91c78cf31
fix(ingest/s3): fix test flakiness (#8416) 2023-07-14 00:42:00 +02:00
Mayuri Nehate
a488557962
feat(ingest/kafka-connect): allow setting platform_instance for kafka… (#8299) 2023-07-13 16:00:42 +05:30
Mayuri Nehate
60b9623675
fix(ingest/tableau): graceful handling of get all datasources failure (#8406) 2023-07-13 12:58:19 +05:30
Andrew Sikowitz
851c5bab9e
test(ingest/minio): Configure delta lake minio tests for arm64 (#8364)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-07-13 02:41:07 -04:00
Tamas Nemeth
54c7aef1bc
feat(ingest/presto-on-hive): Extracting all the table properties from Hive Metastore (#8348)
Co-authored-by: Pedro Silva <pedro@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-12 15:56:13 -03:00
Mayuri Nehate
a2fc02294c
fix(ingest/snowflake): fix azure cloud region ids in external url (#8376)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-12 17:11:33 +05:30
Andrew Sikowitz
ea3e119210
test(ingest/kafka-connect): Attempt to fix flaky test (#8404) 2023-07-12 00:46:10 -04:00
Harshal Sheth
d4135d57b7
feat(ingest/bigquery): support column-level lineage (#8382) 2023-07-11 11:12:51 -07:00