427 Commits

Author SHA1 Message Date
Shubham Jagtap
cb8002413e
feat(ingestion/transformer): Add dataset dataproduct transformer (#9491)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-01-05 17:03:22 -05:00
Mayuri Nehate
29f2142a2c
feat(databricks): add hive metastore analyze profiling (#9511) 2024-01-02 23:18:43 -05:00
Shubham Jagtap
3635c1c221
feat(ingestion/bigquery): Use sqlglot_lineage for usage and add more perf timers (#9247)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-12-28 04:54:26 -05:00
cburroughs
754d881447
build(ingest/feast): upgrade to latest feast version (#9439) 2023-12-27 19:33:41 -05:00
Harshal Sheth
52687f3eea
feat(ingest): support CLL for redshift materialized views with auto refresh (#9508) 2023-12-22 08:18:22 +01:00
Harshal Sheth
ca518d6c78
feat(ingest): remove librdkafka hacks (#9507) 2023-12-21 20:28:45 -05:00
Tamas Nemeth
cfc641f0d0
fix(ingest/databricks): Pinning databricks sdk to not fail on mypy issues (#9500) 2023-12-21 20:32:51 +01:00
Harshal Sheth
ecda3e6187
feat(ingest): pydantic v2 compatibility (#9434) 2023-12-18 18:26:33 -05:00
Tamas Nemeth
03590a1948
fix(ingest/snowflake) - Fixing snowflake url with default region (#9443) 2023-12-18 13:54:31 -05:00
Harshal Sheth
caef6771b8
feat(ingest/redshift): drop repeated operations (#9440) 2023-12-15 15:07:56 -05:00
Mayuri Nehate
aac1c55a14
feat(ingest/unity): enable hive metastore ingestion (#9416) 2023-12-14 10:35:06 -05:00
Aseem Bansal
cff32e9c74
fix(ingest/transformer): correct registration (#9418) 2023-12-08 16:07:00 -08:00
Andrew Sikowitz
27f23ecdd5
feat(ingest/unity): GE Profiling (#8951) 2023-12-06 13:59:23 -05:00
Andrew Sikowitz
806f09ae23
test(ingest/unity): Unity catalog data generation (#8949) 2023-12-05 12:33:00 -05:00
Tamas Nemeth
3ee82590cd
fix(ingest/redshift): Identify materialized views properly + fix connection args support (#9368) 2023-12-05 17:42:29 +01:00
Harshal Sheth
4dd6738ae7
fix(ingest): bump pyhive to fix headers issue (#9328) 2023-11-29 10:25:33 +01:00
terratrue-daniel
ab10e6bc58
feat(ingest/mssql): enable TLS encryption for SQLServer using pytds (#9256) 2023-11-29 13:32:26 +05:30
Andrew Sikowitz
b778def10a
build(ingest/hive): Update thrift pin (#8964)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-16 17:19:25 -05:00
Harshal Sheth
9d41a8f9f0
fix(ingest): drop redshift-legacy and redshift-usage-legacy sources (#9244) 2023-11-16 13:33:35 -05:00
Harshal Sheth
19aa215068
feat(ingest/dbt): dbt column-level lineage (#8991) 2023-11-13 19:00:30 -05:00
david-leifker
ebd2e2312b
fix(datahub-ingestion): remove old jars, sync pyspark version (#9217) 2023-11-10 22:10:00 -06:00
Shubham Jagtap
a187127ac5
feat(ingestion): file-based state checkpoint provider (#9029) 2023-11-10 14:36:00 -08:00
Shubham Jagtap
e73e926999
feat(integration/fivetran): Fivetran connector integration (#9018)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-07 23:02:41 -08:00
Harshal Sheth
55f14530a3
feat(ingest): bump sqlglot parser (#9155) 2023-11-01 12:42:52 +05:30
Harshal Sheth
ea1273281e
feat(ingest): sql parser perf + asyncio fixes (#9119) 2023-10-30 20:57:59 -07:00
Harshal Sheth
58bcedcd6a
fix(ingest/clickhouse): pin version to solve column reflection regression (#9143) 2023-10-30 14:18:48 -07:00
Mayuri Nehate
f402090c1e
feat(ingest): support view lineage for all sqlalchemy sources (#9039) 2023-10-26 09:14:32 -07:00
Andrew Sikowitz
8a80e858a7
build(ingest): Bump avro pin: security vulnerability (#9042) 2023-10-25 13:06:12 -04:00
siddiquebagwan-gslab
b612545220
feat(ingestion/redshift): support auto_incremental_lineage (#9010) 2023-10-25 15:26:06 +05:30
Harshal Sheth
c849246e63
fix(ingest): add releases link (#9014) 2023-10-24 09:09:41 +02:00
Tim
1eaf9c8c5f
feature(ingest/athena): introduce support for complex and nested schemas in Athena (#8137)
Co-authored-by: dnks23 <dominik.s23@live.de>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Tim <tim@MBP-von-Tim.fritz.box>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-18 09:39:59 -07:00
VISHAL KUMAR
5937937819
feat(ingestion/Vertica): Fixed vertica integration test Updated vertica dialect (#9011) 2023-10-18 14:52:07 +05:30
Andrew Sikowitz
9ccd1d4f5d
build(ingest/redshift): Add sqlglot dependency (#9021) 2023-10-16 14:34:15 -04:00
Mayuri Nehate
c81a339bfc
build(ingest): remove ratelimiter dependency (#9008) 2023-10-16 09:27:57 -07:00
Tamas Nemeth
c2e8041d77
Adding missing sqlparser libs to setup.py (#9015) 2023-10-13 20:59:18 +00:00
Tamas Nemeth
a8f0080c08
feat(ingest/teradata): Teradata source (#8977) 2023-10-12 15:14:45 -07:00
Pedro Silva
f6e1312063
feat(ingestion): Adds support for memory profiling (#8856)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-12 18:43:14 +01:00
Andrew Sikowitz
d3346a04e4
feat(ingest/unity): Ingest notebooks and their lineage (#8940) 2023-10-04 10:22:45 -04:00
Harshal Sheth
a300b39f15
feat(ingest/airflow): airflow plugin v2 (#8853) 2023-10-04 16:23:15 +05:30
Aseem Bansal
ad313ad282
feat(transfomer): add transformer to get ownership from tags (#8748) 2023-10-04 14:06:03 +05:30
Harshal Sheth
9deb7be3fc
fix(ingest): refactor test markers + fix disk space issues in CI (#8938) 2023-10-03 20:17:49 -07:00
Andrew Sikowitz
8396829dbc
build(ingest/databricks): Relax databricks-sdk pin (#8855) 2023-09-27 14:03:34 -04:00
hariishaa
622816dcb8
feat(metadata-ingestion): implement mlflow source (#7971)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-26 13:51:30 -04:00
Harshal Sheth
dc9141a6f4
fix(ingest): bump typing-extensions (#8897) 2023-09-26 10:34:36 -07:00
Harshal Sheth
53eaac5963
fix(ingest): bound types-requests (#8895) 2023-09-25 13:24:19 -07:00
Shubham Jagtap
501522d891
feat(ingest/kafka-connect): Lineage for Kafka Connect > Snowflake (#8811)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:12:48 -07:00
Harshal Sheth
791e2e7bf5
feat(python): support custom models without forking (#8774) 2023-09-22 16:43:58 -07:00
Harshal Sheth
2a0200b047
feat(ingest): bump acryl-sqlglot (#8882) 2023-09-21 14:28:51 -07:00
Harshal Sheth
6c6216aaa2
fix(airflow): fix provider loading exception (#8861) 2023-09-20 12:00:23 -07:00
Tony Ouyang
f4da93988e
feat(ingestion/dynamodb): Add DynamoDB as new metadata ingestion source (#8768)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-09-15 13:26:17 -07:00