455 Commits

Author SHA1 Message Date
Shubham Jagtap
e73e926999
feat(integration/fivetran): Fivetran connector integration (#9018)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-07 23:02:41 -08:00
Harshal Sheth
55f14530a3
feat(ingest): bump sqlglot parser (#9155) 2023-11-01 12:42:52 +05:30
Harshal Sheth
ea1273281e
feat(ingest): sql parser perf + asyncio fixes (#9119) 2023-10-30 20:57:59 -07:00
Harshal Sheth
58bcedcd6a
fix(ingest/clickhouse): pin version to solve column reflection regression (#9143) 2023-10-30 14:18:48 -07:00
Mayuri Nehate
f402090c1e
feat(ingest): support view lineage for all sqlalchemy sources (#9039) 2023-10-26 09:14:32 -07:00
Andrew Sikowitz
8a80e858a7
build(ingest): Bump avro pin: security vulnerability (#9042) 2023-10-25 13:06:12 -04:00
siddiquebagwan-gslab
b612545220
feat(ingestion/redshift): support auto_incremental_lineage (#9010) 2023-10-25 15:26:06 +05:30
Harshal Sheth
c849246e63
fix(ingest): add releases link (#9014) 2023-10-24 09:09:41 +02:00
Tim
1eaf9c8c5f
feature(ingest/athena): introduce support for complex and nested schemas in Athena (#8137)
Co-authored-by: dnks23 <dominik.s23@live.de>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Tim <tim@MBP-von-Tim.fritz.box>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-18 09:39:59 -07:00
VISHAL KUMAR
5937937819
feat(ingestion/Vertica): Fixed vertica integration test Updated vertica dialect (#9011) 2023-10-18 14:52:07 +05:30
Andrew Sikowitz
9ccd1d4f5d
build(ingest/redshift): Add sqlglot dependency (#9021) 2023-10-16 14:34:15 -04:00
Mayuri Nehate
c81a339bfc
build(ingest): remove ratelimiter dependency (#9008) 2023-10-16 09:27:57 -07:00
Tamas Nemeth
c2e8041d77
Adding missing sqlparser libs to setup.py (#9015) 2023-10-13 20:59:18 +00:00
Tamas Nemeth
a8f0080c08
feat(ingest/teradata): Teradata source (#8977) 2023-10-12 15:14:45 -07:00
Pedro Silva
f6e1312063
feat(ingestion): Adds support for memory profiling (#8856)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-12 18:43:14 +01:00
Andrew Sikowitz
d3346a04e4
feat(ingest/unity): Ingest notebooks and their lineage (#8940) 2023-10-04 10:22:45 -04:00
Harshal Sheth
a300b39f15
feat(ingest/airflow): airflow plugin v2 (#8853) 2023-10-04 16:23:15 +05:30
Aseem Bansal
ad313ad282
feat(transfomer): add transformer to get ownership from tags (#8748) 2023-10-04 14:06:03 +05:30
Harshal Sheth
9deb7be3fc
fix(ingest): refactor test markers + fix disk space issues in CI (#8938) 2023-10-03 20:17:49 -07:00
Andrew Sikowitz
8396829dbc
build(ingest/databricks): Relax databricks-sdk pin (#8855) 2023-09-27 14:03:34 -04:00
hariishaa
622816dcb8
feat(metadata-ingestion): implement mlflow source (#7971)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-26 13:51:30 -04:00
Harshal Sheth
dc9141a6f4
fix(ingest): bump typing-extensions (#8897) 2023-09-26 10:34:36 -07:00
Harshal Sheth
53eaac5963
fix(ingest): bound types-requests (#8895) 2023-09-25 13:24:19 -07:00
Shubham Jagtap
501522d891
feat(ingest/kafka-connect): Lineage for Kafka Connect > Snowflake (#8811)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:12:48 -07:00
Harshal Sheth
791e2e7bf5
feat(python): support custom models without forking (#8774) 2023-09-22 16:43:58 -07:00
Harshal Sheth
2a0200b047
feat(ingest): bump acryl-sqlglot (#8882) 2023-09-21 14:28:51 -07:00
Harshal Sheth
6c6216aaa2
fix(airflow): fix provider loading exception (#8861) 2023-09-20 12:00:23 -07:00
Tony Ouyang
f4da93988e
feat(ingestion/dynamodb): Add DynamoDB as new metadata ingestion source (#8768)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-09-15 13:26:17 -07:00
Andrew Sikowitz
e75900b9a9
build(ingest): Remove constraint on jsonschema for Python >= 3.8 (#8842) 2023-09-14 12:25:41 -07:00
Andrew Sikowitz
1474ac01b1
build(ingest): Bump jsonschema for Python >= 3.8 (#8836) 2023-09-13 12:32:45 -07:00
Mayuri Nehate
303a2d0863
build(ingest): upgrade to sqlalchemy 1.4, drop 1.3 support (#8810)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-12 11:30:24 -07:00
Harshal Sheth
0e8000cf18
feat(ingest): drop sql_metadata parser (#8765) 2023-09-07 11:32:28 -07:00
Harshal Sheth
4ffad4d9b9
chore(ingest): upgrade sqlglot fork (#8775)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-09-06 12:49:44 -07:00
cccs-eric
6fe60a274e
feat(iceberg): Upgrade Iceberg ingestion source to pyiceberg 0.4.0 (#8357)
Co-authored-by: cccs-Dustin <96579982+cccs-Dustin@users.noreply.github.com>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-31 13:01:05 -04:00
Mayuri Nehate
e867dbc3da
ci: separate airflow build and test (#8688)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-30 14:08:42 -07:00
Andrew Sikowitz
19ce0036c7
build(ingest): Pin mypy-boto3-sagemaker directly (#8746) 2023-08-29 12:37:27 -05:00
Tamas Nemeth
d86b336e70
chore(ingest/s3) Bump Deequ and Pyspark version (#8638)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-29 18:11:37 +02:00
Andrew Sikowitz
6659ff26ef
feat(ingest/sql-queries): Add sql queries source, SqlParsingBuilder, sqlglot_lineage performance optimizations (#8494)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-08-24 10:35:46 -04:00
Harshal Sheth
a97548ce46
fix(ingest/powerbi): add sqlglot python dep (#8704) 2023-08-23 22:05:53 -07:00
Andrew Sikowitz
68abf9c6a1
build(ingest): Bump pydantic pin (#8660) 2023-08-23 16:55:51 +05:30
skrydal
baae3d261d
fix(ingest/okta): fix event_loop RuntimeError with nested asyncio (#8637) 2023-08-16 10:32:57 +05:30
Andrew Sikowitz
526e626146
feat(ingest): Add DataHub source (#8561)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-15 17:49:20 -04:00
Harshal Sheth
ebf42e2702
fix(ingest): use hive pure_sasl variant (#8570) 2023-08-09 13:04:36 -04:00
Harshal Sheth
3428bcaaad
fix(ingest): add tableau sqlglot dep (#8552) 2023-08-02 15:18:06 -03:00
Pedro Silva
a4a8182001
feat(cli): Adds ability to upload recipes to DataHub's UI (#8317)
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
2023-08-01 17:35:42 -03:00
VISHAL KUMAR
ef3b9489aa
feat(ingest/vertica): performance improvement and bug fixes (#8328)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-01 19:34:35 +05:30
Harshal Sheth
d8b2397b93
fix(ingest): pin boto3-stubs in CI (#8527)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-07-31 19:48:05 -07:00
Harshal Sheth
89f23d3c36
chore(ingest): bump sqllineage and sqlparse (#8481) 2023-07-28 13:10:19 -07:00
Harshal Sheth
d733363bed
chore(ingest): drop bigquery-beta and snowflake-beta aliases (#8451) 2023-07-20 14:05:25 -04:00
Aseem Bansal
9df70d7355
ingest(elasticsearch): add basic profiling (#8351) 2023-07-20 08:25:30 +05:30