3975 Commits

Author SHA1 Message Date
Harshal Sheth
b26da57928
feat(ingest): remove dep on termcolor (#11733) 2024-10-29 17:48:55 +01:00
Aseem Bansal
02f0a3dee7
feat(ingest/transform): extend ownership transformer to other entities (#11700) 2024-10-29 15:28:41 +05:30
Harshal Sheth
bea253a064
feat(ingest): unpin traitlets (#11731) 2024-10-29 09:21:20 +01:00
Mayuri Nehate
20d0999601
feat(ingest/databricks): report unique query count from usage (#11576) 2024-10-29 09:20:55 +01:00
sid-acryl
8c53c54081
feat(ingestion/powerbi): ingest powerbi app (#11629) 2024-10-29 09:19:37 +01:00
sagar-salvi-apptware
bb63cbd9db
fix(ingestion/bigquery): Add lineage extraction for BigQuery with GCS source (#11442) 2024-10-29 09:18:08 +01:00
Harshal Sheth
a11ac8d104
feat(ingest/fivetran): avoid duplicate table lineage entries (#11712) 2024-10-29 09:13:34 +01:00
Mayuri Nehate
e90d744cbd
feat(bigquery): support config for region qualifiers to fetch jobs (#11728) 2024-10-28 14:16:34 -07:00
Mayuri Nehate
87fa5b89e8
feat: multi-query lineage for temp upstreams (#11708) 2024-10-25 16:56:55 +05:30
haeniya
7c8dba4bd9
feat(ingest/tableau): support ingestion of access roles (#11157)
Co-authored-by: Yanik Häni <Yanik.Haeni1@swisscom.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-10-24 11:56:00 -07:00
Harshal Sheth
b89ca3f081
feat(ingest/fivetran): support overriding destination db (#11701) 2024-10-24 19:31:34 +02:00
Tamas Nemeth
4f6f69d076
fet(ingest/bigquery): Add support ingesting foreign keys and primary keys for BigQuery tables (#11686) 2024-10-24 19:24:44 +02:00
Tamas Nemeth
f6760436b7
feat(ingestion/dagster): Dagster assetless ingestion (#11262)
Co-authored-by: shubhamjagtap639 <shubham.jagtap@gslab.com>
2024-10-24 16:06:00 +02:00
Semion Sidorenko
6787db7124
fix(ingest/superset): Don't set schema/db for druid (#11682) 2024-10-23 14:42:27 -07:00
Jonny Dixon
8b062eb8bd
feat(ingest/oracle): retire deprecated cx_oracle library (#11607)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-10-23 14:39:57 -07:00
pankajmahato-visa
e569fbc8a6
fix(ingest/mongodb): Add Collection Name as Dataset Name in MongoDB (#11698) 2024-10-23 18:10:20 +02:00
Mayuri Nehate
eab2ac7a2e
feat(ingest/snowflake): support lineage via rename and swap using que… (#11600) 2024-10-23 14:02:08 +05:30
Harshal Sheth
e96323a2a2
feat(ingest/fivetran): show connector filter reason (#11695) 2024-10-23 13:49:49 +05:30
Julien Jehannet
326afc6308
fix(ingestion/glue): manage table names from resource_links from nearest catalog correctly (#11578) 2024-10-23 11:39:23 +05:30
Harshal Sheth
581a39ad40
fix(ingest): add logging for mcp diff (#11683) 2024-10-22 20:22:40 -07:00
Harshal Sheth
35f30b7d3c
feat(ingest): use mainline sqlglot (#11693) 2024-10-22 19:57:46 -07:00
skrydal
b7958c4a91
fix(ingest/bigquery): Fix tags urn/name ingestion for BigQuery (#11691) 2024-10-22 21:54:30 +02:00
Harshal Sheth
7ba77a959c
fix(ingest): cache sql is_profiling_enabled method (#11665)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2024-10-22 11:49:57 -07:00
Harshal Sheth
7f6f18cf77
feat(ingest): generate urn types for all entities (#11676) 2024-10-22 11:49:20 -07:00
Harshal Sheth
dd1a06fb55
feat(ingest/fivetran): add safeguards on table/column lineage (#11674) 2024-10-21 12:29:45 +05:30
Shirshanka Das
3b1b76244d
feat(sdk):platform-resource - complex queries (#11675) 2024-10-19 14:53:28 -07:00
Harshal Sheth
8f7f2c1724
feat(ingest/fivetran): protect against high sync volume (#11589) 2024-10-18 14:29:03 -07:00
Harshal Sheth
dfd7293f8d
feat(ingest/looker): include project name in model/explore properties (#11664)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2024-10-18 13:05:06 -07:00
Jay Feldman
7e4d4aba12
fix(ingest/looker): Remove bad imports from looker_common (#11663) 2024-10-18 12:58:52 -07:00
Andrew Sikowitz
72d1236669
feat(ingest/transformer/domain): Add support for on conflict do nothing to dataset domain transformers (#11649) 2024-10-18 12:01:39 -07:00
Andrew Sikowitz
dcf4793c3d
fix(ingest/dbt): Prevent lineage cycles when parsing sql of dbt models (#11666) 2024-10-18 03:13:18 -07:00
Shirshanka Das
011e5b97e7
fix(sdk): platform resource - support indexed queries when urns are i… (#11660) 2024-10-18 10:12:05 +02:00
Mayuri Nehate
179a6714a6
feat(ingest): ensure sqlite file delete on clean exit (#11612) 2024-10-18 10:33:28 +05:30
Harshal Sheth
6e3724b2da
perf(ingest): streamline CLL generation (#11645) 2024-10-17 17:50:59 -07:00
Harshal Sheth
523a456f5c
fix(ingest/redshift): fix syntax error in temp sql (#11661) 2024-10-17 12:33:04 -07:00
Harshal Sheth
6f193221c9
fix(ingest/dbt): fix bug in CLL pruning (#11614) 2024-10-17 20:50:42 +02:00
Tamas Nemeth
cf3634a6a9
fix(ingest/bigquery): Not setting platform instance for bigquery platform resources (#11659) 2024-10-17 19:57:02 +02:00
Harshal Sheth
68cd17b34e
fix(ingest/redshift): reduce sequence limit for LISTAGG (#11621)
Co-authored-by: treff7es <treff7es@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2024-10-17 10:08:37 -07:00
Harshal Sheth
6b09346ca5
feat(ingest): add offline flag to SQL parser CLI (#11635) 2024-10-17 08:14:06 -07:00
Harshal Sheth
8b42ac8cde
fix(ingest): run sqllineage in process by default (#11650) 2024-10-16 20:47:48 -07:00
Harshal Sheth
b8144699fd
chore(ingest): reorganize unit tests (#11636) 2024-10-16 19:18:32 -07:00
Harshal Sheth
d34717fd82
fix(ingest): remove default value from DatahubClientConfig.server (#11570) 2024-10-16 13:50:33 -07:00
sid-acryl
ce909c8b8b
feat(ingest/powerbi): link cross workspace dataset into assets (#11560) 2024-10-16 08:28:05 -07:00
Harshal Sheth
58f5b4a0cb
fix(ingest): add preset deps (#11637) 2024-10-16 17:23:01 +05:30
Tamas Nemeth
e76647dd7a
feat(ingest/bigquery): Generate platform resource entities for BigQuery labels (#11602)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2024-10-16 10:04:01 +02:00
Hyejin Yoon
2f2a7af584
feat: add quickstart post (#11623) 2024-10-15 23:49:59 -07:00
david-leifker
5d3e464c21
feat(validations): Ingest and metadata schema validators (#11619)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2024-10-15 12:35:41 -05:00
Aseem Bansal
1eec2c4238
fix(ingest): drop empty fields (#11613) 2024-10-15 14:53:45 +05:30
Andrew Sikowitz
3078c5a4c9
fix(ingest/preset): Add skip_on_failure to root_validator decorator (#11615) 2024-10-14 13:13:17 -07:00
Semion Sidorenko
62c39cdcd3
fix(ingest/superset): parse postgres platform correctly (#11540) 2024-10-14 11:57:38 -07:00