3975 Commits

Author SHA1 Message Date
Mayuri Nehate
8bf28bfa92
fix(ingest/tableau): fix tableau native CLL for snowflake, add type annotations (#8779)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-06 12:47:36 -07:00
dominik s
25148f4a65
refactor(ingest): Add support for group-owners in dataflow entities (#8154)
Co-authored-by: Dominik Schüssele <dominik.schuessele@inovex.de>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-06 15:12:14 -04:00
Andrew Sikowitz
ac025e508d
fix(ingest/datahub): Support postgres; build(postgres): Modernize postgres docker setup (#8762) 2023-09-06 12:18:29 -04:00
Aseem Bansal
c38bb91519
fix(elastic): improve error handling for profiling (#8785) 2023-09-05 09:20:27 -07:00
Hyejin Yoon
065a290bd5
fix:change global graph url to static-assets (#8742) 2023-09-04 15:49:00 +09:00
cccs-eric
6fe60a274e
feat(iceberg): Upgrade Iceberg ingestion source to pyiceberg 0.4.0 (#8357)
Co-authored-by: cccs-Dustin <96579982+cccs-Dustin@users.noreply.github.com>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-31 13:01:05 -04:00
Andrew Sikowitz
a4e726872b
fix(ingest/bigquery): Filter out fine grained lineage with no upstreams (#8758) 2023-08-31 12:44:24 -04:00
Harshal Sheth
21b2851be7
feat(sql-parser): schema-aware output column casing (#8760)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-08-31 09:43:39 -07:00
Harshal Sheth
4c69f9a1d6
fix(ingest/athena): fix container linting (#8761) 2023-08-30 19:36:05 -04:00
Mayuri Nehate
e867dbc3da
ci: separate airflow build and test (#8688)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-30 14:08:42 -07:00
Harshal Sheth
1282e5bf93
feat(systemMetadata): add pipeline names to system metadata (#8684) 2023-08-30 13:19:28 -07:00
Tamas Nemeth
c193b1dc70
fix(ingest/athena): Fixing db container id (#8689) 2023-08-30 22:12:02 +02:00
Andrew Sikowitz
026f7abe9c
feat(ingest/usage): Make cumulative query character limit configurable (#8751) 2023-08-30 15:53:08 -04:00
Andrew Sikowitz
fa0c43c031
fix(ingest/bigquery): Handle null view_definition; remove view definition hash ids (#8747) 2023-08-30 15:47:08 -04:00
Harshal Sheth
5032af9123
feat(cli): support recursive deletes (#8709) 2023-08-30 12:07:41 -07:00
skrydal
2776903315
fix(ingest/okta): Removed code closing okta's event_loop (#8675)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-29 13:25:35 -07:00
Andrew Sikowitz
40d17f00ea
feat(ingest/datahub): Improvements, bug fixes, and docs (#8735) 2023-08-29 14:33:40 -04:00
Andrew Sikowitz
19ce0036c7
build(ingest): Pin mypy-boto3-sagemaker directly (#8746) 2023-08-29 12:37:27 -05:00
Andrew Sikowitz
04bf8866c5
docs(ingest/openapi): Downgrade status from CERTIFIED to INCUBATING (#8736) 2023-08-29 12:32:27 -04:00
Tamas Nemeth
d86b336e70
chore(ingest/s3) Bump Deequ and Pyspark version (#8638)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-29 18:11:37 +02:00
Jinlin Yang
437b787747
(ingestion) bug fix: emit platform instance aspect for dataset in Databricks ingestion (#8671) 2023-08-28 19:17:07 -04:00
Kirill Popov
3acd25ba1d
feat(ingest/metabase): detect source table for cards sourced from other cards (#8577) 2023-08-28 13:02:41 -04:00
Harshal Sheth
7b66c32b70
feat(ingest): support writing configs to files (#8696) 2023-08-28 09:55:50 -07:00
Akhil
ada40e6edd
fix(superset): get alternate platform value if sqlalchemy_uri param is missing (#8667) 2023-08-28 15:03:43 +05:30
Mayuri Nehate
cc94ffbf6c
fix(ingest): stateful redundant run skip handler (#8467) 2023-08-28 15:03:31 +05:30
Mayuri Nehate
47571e6257
fix(examples): fix typo in business glossary bootstrap yml (#8703) 2023-08-28 15:02:47 +05:30
Andrew Sikowitz
d169ee8ae5
config(ingest/rest): Update rest sink defaults to retry more often (#8729) 2023-08-25 20:27:35 -04:00
Hyejin Yoon
04ecf4f75a
docs(docs): add native versioning (#8714) 2023-08-25 14:10:13 -07:00
Andrew Sikowitz
6659ff26ef
feat(ingest/sql-queries): Add sql queries source, SqlParsingBuilder, sqlglot_lineage performance optimizations (#8494)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-08-24 10:35:46 -04:00
Mayuri Nehate
e285da3e75
feat(ingest/snowflake): tables from snowflake shares as siblings (#8531) 2023-08-24 10:23:07 -04:00
Adriano Vega Llobell
d6a935e3ca
fix(ingestion/kafka-connect): update retrieval of database name in Debezium SQL Server (#8608) 2023-08-24 14:49:04 +05:30
RChygir
43d48ddde4
feat(ingest/mssql): load jobs and stored procedures (#5363) 2023-08-24 14:48:03 +05:30
Alexander
bcef25acd3
feat(ingest/looker): Record observed lineage timestamps for Looker and LookML sources (#7735) 2023-08-24 14:47:04 +05:30
Andrew Sikowitz
22c35f1a23
fix(ingest/bigquery): Add config option to create DataPlatformInstance, default off (#8659) 2023-08-24 14:46:06 +05:30
Harshal Sheth
090f8af8e4
ci(ingest): make ingestion caching rules correct (#8685) 2023-08-23 22:06:28 -07:00
Harshal Sheth
a97548ce46
fix(ingest/powerbi): add sqlglot python dep (#8704) 2023-08-23 22:05:53 -07:00
Andrew Sikowitz
01ae5d96da
fix(ingest/ldap): Handle case when 'objectClass' not in attrs (#8658) 2023-08-23 15:58:34 -04:00
Andrew Sikowitz
8141e2d649
remove(ingest/snowflake): Remove legacy snowflake lineage (#8653)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-08-23 15:57:46 -04:00
Andrew Sikowitz
68abf9c6a1
build(ingest): Bump pydantic pin (#8660) 2023-08-23 16:55:51 +05:30
siddiquebagwan-gslab
8ee58af0c2
feat(ingestion/powerbi): support multiple tables as upstream in native SQL parsing (#8592) 2023-08-23 14:38:58 +05:30
Andrew Sikowitz
4116716a15
logs(ingestion/unity): Hide stack trace on sql parse failure logs (#8657) 2023-08-23 14:38:10 +05:30
Andrew Sikowitz
439cf4d7dc
test(ingest/vertica): Skip integration test failing CI; support arm Macs (#8694) 2023-08-22 16:27:46 -04:00
Mayuri Nehate
3681e1a128
docs(ingest/kafka-connect): add details on platform instance mapping (#8654) 2023-08-18 18:51:14 +05:30
Harshal Sheth
c5a2e8923e
feat(ingest/dbt-cloud): use job-based graphql queries (#8647) 2023-08-17 13:29:10 -04:00
Harshal Sheth
130f908a49
fix(ingest/snowflake): fix usage enum bug (#8649) 2023-08-17 13:27:19 -04:00
Alexander
c0addf6eef
feat(ingest/bigquery): add tag to BigQuery clustering columns (#8495)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-17 12:44:15 -04:00
Jinlin Yang
6748aecdc0
fix(ingest/s3): emit data_platform_instance aspect if the config has platform_instance (#8585) 2023-08-17 10:40:54 +05:30
david-leifker
40caf35812
feat(datahub-ingestion): refactor datahub ingestion slim images (#8515) 2023-08-17 10:40:17 +05:30
Harshal Sheth
2f9b5a8273
fix(ingest): avoid mutable defaults in powerbi dataclass (#8609) 2023-08-17 10:36:52 +05:30
kr_Deepankar
23ac9062fe
feat(ingestion/ldap): flag to ingest ldap users with email instead of username (#8606) 2023-08-16 10:33:18 +05:30