945 Commits

Author SHA1 Message Date
siddiquebagwan-gslab
c415d63dda
feat(ingestion/powerbi): column level lineage extraction for M-Query (#8796) 2023-10-04 16:22:51 +05:30
Harshal Sheth
9deb7be3fc
fix(ingest): refactor test markers + fix disk space issues in CI (#8938) 2023-10-03 20:17:49 -07:00
hariishaa
622816dcb8
feat(metadata-ingestion): implement mlflow source (#7971)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-26 13:51:30 -04:00
Shubham Jagtap
501522d891
feat(ingest/kafka-connect): Lineage for Kafka Connect > Snowflake (#8811)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:12:48 -07:00
Mayuri Nehate
5c40390a92
feat(ingest/kafka): support metadata mapping from kafka avro schemas (#8825)
Co-authored-by: Daniel Messias <danielcmessias@gmail.com>
Co-authored-by: Deepankarkr <deepankar.kumar@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:11:42 -07:00
Tony Ouyang
f4da93988e
feat(ingestion/dynamodb): Add DynamoDB as new metadata ingestion source (#8768)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-09-15 13:26:17 -07:00
Mayuri Nehate
cdb9f5ba62
feat(bigquery): add better timers around every API call (#8626) 2023-09-15 11:55:39 -07:00
cjm98332
a021053a72
fix(ingest/mssql): Add UNIQUEIDENTIFIER data type as String (#8642)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-12 19:23:39 +05:30
siddiquebagwan-gslab
95b2d437ca
feat(ingestion/looker): Add view file-path as option in view_naming_pattern config (#8713) 2023-09-11 16:55:17 +05:30
cccs-eric
6fe60a274e
feat(iceberg): Upgrade Iceberg ingestion source to pyiceberg 0.4.0 (#8357)
Co-authored-by: cccs-Dustin <96579982+cccs-Dustin@users.noreply.github.com>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-31 13:01:05 -04:00
Tamas Nemeth
d86b336e70
chore(ingest/s3) Bump Deequ and Pyspark version (#8638)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-29 18:11:37 +02:00
Kirill Popov
3acd25ba1d
feat(ingest/metabase): detect source table for cards sourced from other cards (#8577) 2023-08-28 13:02:41 -04:00
Mayuri Nehate
cc94ffbf6c
fix(ingest): stateful redundant run skip handler (#8467) 2023-08-28 15:03:31 +05:30
RChygir
43d48ddde4
feat(ingest/mssql): load jobs and stored procedures (#5363) 2023-08-24 14:48:03 +05:30
Alexander
bcef25acd3
feat(ingest/looker): Record observed lineage timestamps for Looker and LookML sources (#7735) 2023-08-24 14:47:04 +05:30
Andrew Sikowitz
22c35f1a23
fix(ingest/bigquery): Add config option to create DataPlatformInstance, default off (#8659) 2023-08-24 14:46:06 +05:30
Andrew Sikowitz
8141e2d649
remove(ingest/snowflake): Remove legacy snowflake lineage (#8653)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-08-23 15:57:46 -04:00
siddiquebagwan-gslab
8ee58af0c2
feat(ingestion/powerbi): support multiple tables as upstream in native SQL parsing (#8592) 2023-08-23 14:38:58 +05:30
Andrew Sikowitz
439cf4d7dc
test(ingest/vertica): Skip integration test failing CI; support arm Macs (#8694) 2023-08-22 16:27:46 -04:00
Jinlin Yang
6748aecdc0
fix(ingest/s3): emit data_platform_instance aspect if the config has platform_instance (#8585) 2023-08-17 10:40:54 +05:30
kr_Deepankar
23ac9062fe
feat(ingestion/ldap): flag to ingest ldap users with email instead of username (#8606) 2023-08-16 10:33:18 +05:30
Mayuri Nehate
ddcd5109dc
feat(ingest): allow relative start time config (#8562)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-14 17:48:06 -07:00
alplatonov
11fdfcf956
Fix(ingestion/clickhouse) move to two tier sqlalchemy (#8300)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-08-11 16:11:40 -04:00
alplatonov
b58f9bb396
Feat(ingest/ldap)fix list index out of range error (#8525)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-08-09 13:13:27 -04:00
Mayuri Nehate
b4e104f190
fix(ingest/snowflake): maintain qualified name casing, do not lowercase (#8574) 2023-08-04 10:43:22 -07:00
mohdsiddique
6a36118b4f
feat(ingestion/snowflake): use user email-id in urn generation for top users stat (#8513)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-03 08:30:50 +05:30
mohdsiddique
05ef7db45e
fix(ingetion/mssql): convert dataset urns to lowercase (#8551)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-02 13:44:05 -07:00
TusharM
e2919cfd06
feat(ingest/kafka-connect): add support for Confluent S3 Sink Connector (#8298) 2023-08-02 15:09:50 +05:30
Harshal Sheth
08a39c4b0d
fix(ingest/presto): fix presto on hive test failures (#8548) 2023-08-02 12:46:39 +05:30
Mayuri Nehate
e67f811034
feat(classification): allow parallelisation to reduce time (#8368) 2023-08-02 09:53:39 +05:30
zheyu001
bf47d65412
feat(ingest/presto-on-hive): enable partition key for presto-on-hive (#8380) 2023-08-02 09:53:27 +05:30
Gabe Lyons
843f82b943
feat(presto-on-hive): allow v1 fieldpaths in the presto-on-hive source (#8474) 2023-08-01 14:05:50 -07:00
mohdsiddique
547e1f498e
feat(ingestion/tableau): support column level lineage for custom sql (#8466)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-01 19:35:18 +05:30
VISHAL KUMAR
ef3b9489aa
feat(ingest/vertica): performance improvement and bug fixes (#8328)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-01 19:34:35 +05:30
Mayuri Nehate
b9060db515
chore(ingest/tableau): miscellaneous cleanup refractor (#8417)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-07-27 17:02:48 +05:30
Andrew Sikowitz
bf9f380350
fix(ingest): Generate browse paths v2 for more sources; properly pass platform_instance (#8501) 2023-07-25 11:35:34 +05:30
Aseem Bansal
8fb5912978
fix(mysql): getting table name correctly (#8476)
Co-authored-by: Ellie O'Neil <oneile729@gmail.com>
2023-07-23 17:10:54 +05:30
mohdsiddique
bec018257c
fix(ingestion/powerbi-report-srever): handle requests.exceptions.JSONDecodeError (#8442)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-07-21 14:01:57 -07:00
Mayuri Nehate
f4fde21168
feat(ingest/nifi): add support for basic auth in nifi (#8457) 2023-07-20 14:12:33 -04:00
Ellie O'Neil
d734b2849e
feat(ingest/mysql): Add estimate row count for mysql (#8420) 2023-07-20 08:06:26 +05:30
Tamas Nemeth
a91c78cf31
fix(ingest/s3): fix test flakiness (#8416) 2023-07-14 00:42:00 +02:00
Mayuri Nehate
a488557962
feat(ingest/kafka-connect): allow setting platform_instance for kafka… (#8299) 2023-07-13 16:00:42 +05:30
Mayuri Nehate
60b9623675
fix(ingest/tableau): graceful handling of get all datasources failure (#8406) 2023-07-13 12:58:19 +05:30
Andrew Sikowitz
851c5bab9e
test(ingest/minio): Configure delta lake minio tests for arm64 (#8364)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-07-13 02:41:07 -04:00
Tamas Nemeth
54c7aef1bc
feat(ingest/presto-on-hive): Extracting all the table properties from Hive Metastore (#8348)
Co-authored-by: Pedro Silva <pedro@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-12 15:56:13 -03:00
Andrew Sikowitz
ea3e119210
test(ingest/kafka-connect): Attempt to fix flaky test (#8404) 2023-07-12 00:46:10 -04:00
Tamas Nemeth
e4736d7b46
fix(ingest/s3): Deequ import rearragement (#8389) 2023-07-10 14:40:06 -04:00
Andrew Sikowitz
d92cbe99e0
test(ingest/kafka): Configure kafka connect tests for arm64 (#8362) 2023-07-10 12:27:29 -04:00
Andrew Sikowitz
1f84bf5b2b
fix(ingest/sql-common): Fix profile_table_level_only (#8331)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-07 19:05:50 -04:00
Andrew Sikowitz
74301ecb60
test(ingest/nifi): Configure nifi tests for arm64 (#8363)
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-07-07 12:43:01 -04:00