3904 Commits

Author SHA1 Message Date
Shubham Jagtap
501522d891
feat(ingest/kafka-connect): Lineage for Kafka Connect > Snowflake (#8811)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:12:48 -07:00
Mayuri Nehate
5c40390a92
feat(ingest/kafka): support metadata mapping from kafka avro schemas (#8825)
Co-authored-by: Daniel Messias <danielcmessias@gmail.com>
Co-authored-by: Deepankarkr <deepankar.kumar@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:11:42 -07:00
Harshal Sheth
5bb9f30895
docs(ingest/lookml): add guide on debugging lkml parse errors (#8890) 2023-09-22 16:55:15 -07:00
Sergio Gómez Villamor
e254a50b50
fix(report): too long report causes MSG_SIZE_TOO_LARGE in kafka (#8857) 2023-09-22 16:54:34 -07:00
Harshal Sheth
791e2e7bf5
feat(python): support custom models without forking (#8774) 2023-09-22 16:43:58 -07:00
Harshal Sheth
c946c01199
fix(ingest/bigquery): show report in output (#8867)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-22 13:01:38 -07:00
Mayuri Nehate
5481e19e0a
feat(ingest): bulk fetch schema info for schema resolver (#8865)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-21 18:05:26 -04:00
Harshal Sheth
2a0200b047
feat(ingest): bump acryl-sqlglot (#8882) 2023-09-21 14:28:51 -07:00
Mayuri Nehate
6ce35c9654
fix(ingest): fix mode lint error (#8875) 2023-09-21 09:05:58 -07:00
Harshal Sheth
6c6216aaa2
fix(airflow): fix provider loading exception (#8861) 2023-09-20 12:00:23 -07:00
Gabe Lyons
67af68284f
dcs(ml-models): enhancing ml model documentation (#8848) 2023-09-19 09:02:24 -07:00
Mayuri Nehate
99d7eb756c
feat(ingest/bigquery): support bigquery profiling with sampling (#8794) 2023-09-15 13:36:04 -07:00
Tony Ouyang
f4da93988e
feat(ingestion/dynamodb): Add DynamoDB as new metadata ingestion source (#8768)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-09-15 13:26:17 -07:00
Mayuri Nehate
cdb9f5ba62
feat(bigquery): add better timers around every API call (#8626) 2023-09-15 11:55:39 -07:00
Andrew Sikowitz
e75900b9a9
build(ingest): Remove constraint on jsonschema for Python >= 3.8 (#8842) 2023-09-14 12:25:41 -07:00
Hyejin Yoon
31abf383d1
ci: add markdown-link-check (#8771) 2023-09-14 11:34:21 +09:00
Andrew Sikowitz
493d31531a
feat(ingest/rest-emitter): Do not raise error on retry failure to get better error messages (#8837) 2023-09-13 14:00:58 -07:00
Andrew Sikowitz
1474ac01b1
build(ingest): Bump jsonschema for Python >= 3.8 (#8836) 2023-09-13 12:32:45 -07:00
Adriano Vega Llobell
3cc0f76d17
docs(transformer): fix names in sample code of 'pattern_add_dataset_domain' (#8755) 2023-09-12 14:34:24 -07:00
Pedro Silva
138f6c0f74
feat(cli): fix upload ingest cli endpoint (#8826) 2023-09-12 14:26:30 -07:00
Harshal Sheth
449cc9ba91
ci: make wheel builds more robust (#8815) 2023-09-12 13:15:05 -07:00
Harshal Sheth
f7fee743bf
fix(ingest): use epoch 1 for dev build versions (#8824) 2023-09-12 13:11:01 -07:00
Mayuri Nehate
303a2d0863
build(ingest): upgrade to sqlalchemy 1.4, drop 1.3 support (#8810)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-12 11:30:24 -07:00
cjm98332
a021053a72
fix(ingest/mssql): Add UNIQUEIDENTIFIER data type as String (#8642)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-12 19:23:39 +05:30
siddiquebagwan-gslab
95b2d437ca
feat(ingestion/looker): Add view file-path as option in view_naming_pattern config (#8713) 2023-09-11 16:55:17 +05:30
Harshal Sheth
eb4107a6e3
fix(ingest): drop wrap_aspect_as_workunit method (#8766) 2023-09-07 11:32:41 -07:00
Harshal Sheth
0e8000cf18
feat(ingest): drop sql_metadata parser (#8765) 2023-09-07 11:32:28 -07:00
Harshal Sheth
4ffad4d9b9
chore(ingest): upgrade sqlglot fork (#8775)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-09-06 12:49:44 -07:00
Mayuri Nehate
e680a97046
fix(ingest/bigquery): fix partition and median queries for profiling (#8778) 2023-09-06 12:48:11 -07:00
Mayuri Nehate
8bf28bfa92
fix(ingest/tableau): fix tableau native CLL for snowflake, add type annotations (#8779)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-06 12:47:36 -07:00
dominik s
25148f4a65
refactor(ingest): Add support for group-owners in dataflow entities (#8154)
Co-authored-by: Dominik Schüssele <dominik.schuessele@inovex.de>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-06 15:12:14 -04:00
Andrew Sikowitz
ac025e508d
fix(ingest/datahub): Support postgres; build(postgres): Modernize postgres docker setup (#8762) 2023-09-06 12:18:29 -04:00
Aseem Bansal
c38bb91519
fix(elastic): improve error handling for profiling (#8785) 2023-09-05 09:20:27 -07:00
Hyejin Yoon
065a290bd5
fix:change global graph url to static-assets (#8742) 2023-09-04 15:49:00 +09:00
cccs-eric
6fe60a274e
feat(iceberg): Upgrade Iceberg ingestion source to pyiceberg 0.4.0 (#8357)
Co-authored-by: cccs-Dustin <96579982+cccs-Dustin@users.noreply.github.com>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-31 13:01:05 -04:00
Andrew Sikowitz
a4e726872b
fix(ingest/bigquery): Filter out fine grained lineage with no upstreams (#8758) 2023-08-31 12:44:24 -04:00
Harshal Sheth
21b2851be7
feat(sql-parser): schema-aware output column casing (#8760)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-08-31 09:43:39 -07:00
Harshal Sheth
4c69f9a1d6
fix(ingest/athena): fix container linting (#8761) 2023-08-30 19:36:05 -04:00
Mayuri Nehate
e867dbc3da
ci: separate airflow build and test (#8688)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-30 14:08:42 -07:00
Harshal Sheth
1282e5bf93
feat(systemMetadata): add pipeline names to system metadata (#8684) 2023-08-30 13:19:28 -07:00
Tamas Nemeth
c193b1dc70
fix(ingest/athena): Fixing db container id (#8689) 2023-08-30 22:12:02 +02:00
Andrew Sikowitz
026f7abe9c
feat(ingest/usage): Make cumulative query character limit configurable (#8751) 2023-08-30 15:53:08 -04:00
Andrew Sikowitz
fa0c43c031
fix(ingest/bigquery): Handle null view_definition; remove view definition hash ids (#8747) 2023-08-30 15:47:08 -04:00
Harshal Sheth
5032af9123
feat(cli): support recursive deletes (#8709) 2023-08-30 12:07:41 -07:00
skrydal
2776903315
fix(ingest/okta): Removed code closing okta's event_loop (#8675)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-29 13:25:35 -07:00
Andrew Sikowitz
40d17f00ea
feat(ingest/datahub): Improvements, bug fixes, and docs (#8735) 2023-08-29 14:33:40 -04:00
Andrew Sikowitz
19ce0036c7
build(ingest): Pin mypy-boto3-sagemaker directly (#8746) 2023-08-29 12:37:27 -05:00
Andrew Sikowitz
04bf8866c5
docs(ingest/openapi): Downgrade status from CERTIFIED to INCUBATING (#8736) 2023-08-29 12:32:27 -04:00
Tamas Nemeth
d86b336e70
chore(ingest/s3) Bump Deequ and Pyspark version (#8638)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-29 18:11:37 +02:00
Jinlin Yang
437b787747
(ingestion) bug fix: emit platform instance aspect for dataset in Databricks ingestion (#8671) 2023-08-28 19:17:07 -04:00