3089 Commits

Author SHA1 Message Date
Andrew Sikowitz
9fec6024fb
feat(ingest/unity-catalog): Support external S3 lineage (#9025) 2023-10-17 09:58:38 -04:00
Andrew Sikowitz
6366b63e48
feat(ingest/teradata): Add option to not use file backed dict for view definitions (#9024) 2023-10-16 19:13:23 -04:00
Mayuri Nehate
c81a339bfc
build(ingest): remove ratelimiter dependency (#9008) 2023-10-16 09:27:57 -07:00
Tamas Nemeth
1007204cda
feat(ingest/teradata): view parsing (#9005) 2023-10-13 15:07:19 -04:00
Andrew Sikowitz
6bc7425353
feat(cli/datacontract): Add data quality assertion support (#8968)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-10-13 12:36:18 -04:00
Tamas Nemeth
a8f0080c08
feat(ingest/teradata): Teradata source (#8977) 2023-10-12 15:14:45 -07:00
Pedro Silva
f6e1312063
feat(ingestion): Adds support for memory profiling (#8856)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-12 18:43:14 +01:00
Harshal Sheth
8813ae2fb1
feat(ingest/s3): support .gzip and fix decompression bug (#8990) 2023-10-12 13:58:10 +02:00
Tamas Nemeth
c381806110
feat(ingestion): Adding config option to auto lowercase dataset urns (#8928) 2023-10-12 13:56:30 +02:00
Tamas Nemeth
dd418de76d
fix(ingest/bigquery): Fix shard regexp to match without underscore as well (#8934) 2023-10-12 13:10:59 +02:00
Harshal Sheth
84bba4dc44
feat(ingest): add output schema inference for sql parser (#8989) 2023-10-11 22:31:17 -07:00
Sergio Gómez Villamor
245c5c0008
fix(ingest/looker): stop emitting tag owner (#8942) 2023-10-11 17:06:19 -07:00
Mayuri Nehate
1b06c6a30c
fix(ingest/snowflake): fix sample fraction for very large tables (#8988) 2023-10-11 12:01:42 -07:00
Harshal Sheth
932fbcddbf
refactor(ingest/dbt): move dbt tests logic to dedicated file (#8984) 2023-10-11 11:17:02 -07:00
Harsha Mandadi
4b6b941a2a
fix(ingest): Fix postgres lineage within views (#8906)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Maggie Hays <maggiem.hays@gmail.com>
2023-10-11 14:01:46 -04:00
siddiquebagwan-gslab
10a190470e
feat(ingestion/redshift): CLL support in redshift (#8921) 2023-10-10 20:24:08 -07:00
Andrew Sikowitz
1a72fa499c
feat(ingest/tableau): Allow parsing of database name from fullName (#8981) 2023-10-10 17:34:06 -04:00
Mayuri Nehate
57f855ecd1
feat(ingest): refactor + simplify incremental lineage helper (#8976) 2023-10-09 23:48:21 -07:00
Mayuri Nehate
8d175ef7ef
feat(ingest): incremental lineage source helper (#8941)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-09 13:34:25 -07:00
Mayuri Nehate
8e7f286e71
feat(ingest/snowflake): support profiling with sampling (#8902)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-10-06 10:42:39 -04:00
Andrew Sikowitz
c80da8f949
feat(ingest/unity): Use ThreadPoolExecutor for CLL (#8952) 2023-10-06 10:06:36 -04:00
Andrew Sikowitz
26bc039b96
fix(ingest/unity): Remove metastore from ingestion and urns; standardize platform instance; add notebook filter (#8943) 2023-10-05 23:23:15 -04:00
Harshal Sheth
3cede10ab3
feat(ingest/dbt): support use_compiled_code and test_warnings_are_errors (#8956) 2023-10-05 10:29:47 -07:00
Jinlin Yang
6310e51eb0
feat(ingestion/dynamodb): implement pagination for list_tables (#8910) 2023-10-05 09:33:31 +05:30
Harshal Sheth
817c371fbf
feat: data contracts models + CLI (#8923)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: John Joyce <john@acryl.io>
2023-10-04 20:11:06 -07:00
ethan-cartwright
e2afd44bfe
feat(dbt-ingestion): add documentation link from dbt source to institutionalMemory (#8686)
Co-authored-by: Ethan Cartwright <ethan.cartwright@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-04 20:38:58 +00:00
Andrew Sikowitz
301d3e6b1c
test(ingest/unity): Add Unity Catalog memory performance testing (#8932) 2023-10-04 10:23:13 -04:00
Andrew Sikowitz
d3346a04e4
feat(ingest/unity): Ingest notebooks and their lineage (#8940) 2023-10-04 10:22:45 -04:00
Upendra Rao Vedullapalli
13508a9d88
feat(bigquery): excluding projects without any datasets from ingestion (#8535)
Co-authored-by: Upendra Vedullapalli <upendra.rao.vedullapalli@entur.org>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-10-04 09:20:51 -04:00
Mayuri Nehate
e3780c2d75
feat(ingest/snowflake): initialize schema resolver from datahub for l… (#8903) 2023-10-04 16:23:31 +05:30
Harshal Sheth
a300b39f15
feat(ingest/airflow): airflow plugin v2 (#8853) 2023-10-04 16:23:15 +05:30
siddiquebagwan-gslab
c415d63dda
feat(ingestion/powerbi): column level lineage extraction for M-Query (#8796) 2023-10-04 16:22:51 +05:30
Aseem Bansal
ad313ad282
feat(transfomer): add transformer to get ownership from tags (#8748) 2023-10-04 14:06:03 +05:30
Harshal Sheth
419b8a7cc2
fix(cli): make quickstart docker compose up command more robust (#8929) 2023-10-03 20:20:32 -07:00
Ellie O'Neil
0a5e7d176e
fix(client): use value for RelationshipDirection (#8912) 2023-10-03 11:53:05 -07:00
Andrew Sikowitz
833daa8efd
refactor(ingest/graph): Factor out filter logic (#8888) 2023-09-28 13:52:50 -04:00
Andrew Sikowitz
e738e16157
refactor(ingest/bigquery): Clarify table / view queries (#8913) 2023-09-28 13:52:35 -04:00
Andrew Sikowitz
2e2cd87d99
test(ingest/delta-lake): Fix minio test for new version of delta-lake (#8914) 2023-09-27 17:00:02 -04:00
hariishaa
622816dcb8
feat(metadata-ingestion): implement mlflow source (#7971)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-26 13:51:30 -04:00
Tamas Nemeth
0a869dd6f8
fix(ingest/s3): Converting windows style path to posix one on local fs (#8757) 2023-09-26 10:28:03 +02:00
Mayuri Nehate
874109f76e
feat(ingest/snowflake): allow shares config without platform instance (#8803) 2023-09-25 14:04:05 +05:30
Shubham Jagtap
501522d891
feat(ingest/kafka-connect): Lineage for Kafka Connect > Snowflake (#8811)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:12:48 -07:00
Mayuri Nehate
5c40390a92
feat(ingest/kafka): support metadata mapping from kafka avro schemas (#8825)
Co-authored-by: Daniel Messias <danielcmessias@gmail.com>
Co-authored-by: Deepankarkr <deepankar.kumar@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:11:42 -07:00
Sergio Gómez Villamor
e254a50b50
fix(report): too long report causes MSG_SIZE_TOO_LARGE in kafka (#8857) 2023-09-22 16:54:34 -07:00
Harshal Sheth
791e2e7bf5
feat(python): support custom models without forking (#8774) 2023-09-22 16:43:58 -07:00
Harshal Sheth
c946c01199
fix(ingest/bigquery): show report in output (#8867)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-22 13:01:38 -07:00
Mayuri Nehate
5481e19e0a
feat(ingest): bulk fetch schema info for schema resolver (#8865)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-21 18:05:26 -04:00
Mayuri Nehate
6ce35c9654
fix(ingest): fix mode lint error (#8875) 2023-09-21 09:05:58 -07:00
Mayuri Nehate
99d7eb756c
feat(ingest/bigquery): support bigquery profiling with sampling (#8794) 2023-09-15 13:36:04 -07:00
Tony Ouyang
f4da93988e
feat(ingestion/dynamodb): Add DynamoDB as new metadata ingestion source (#8768)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-09-15 13:26:17 -07:00