35 Commits

Author SHA1 Message Date
Sergio Gómez Villamor
8cae980286
tests(ingestion): moving some tests so they are available for sdk users (#13540) 2025-05-19 08:39:53 +02:00
Harshal Sheth
7c791db087
feat(ingest/sql): column logic + join extraction (#13426) 2025-05-12 17:19:44 -07:00
Harshal Sheth
cb3988a5f3
feat(ingest): associate queries with operations (#13404) 2025-05-05 11:27:33 -07:00
Hyejin Yoon
72aab9fe63
feat(sdk): add sdk lineage client (#13244)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2025-04-17 17:44:07 +09:00
Sergio Gómez Villamor
29d05c214a
feat(hex): warehouse integration via Query enrichment (#12949) 2025-03-26 09:35:50 +01:00
Aseem Bansal
0fc074b50a
dev: enable ruff rule (#12742) 2025-02-28 10:59:06 +05:30
sagar-salvi-apptware
a814cb389f
fix(ingest/bigquery): All View generation when queries_v2 is turned off (#12181)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2025-01-02 17:59:22 +05:30
Harshal Sheth
5519a330e2
chore(ingest): bump black (#11898) 2024-11-20 13:33:54 -08:00
Mayuri Nehate
87fa5b89e8
feat: multi-query lineage for temp upstreams (#11708) 2024-10-25 16:56:55 +05:30
skrydal
b7958c4a91
fix(ingest/bigquery): Fix tags urn/name ingestion for BigQuery (#11691) 2024-10-22 21:54:30 +02:00
Mayuri Nehate
179a6714a6
feat(ingest): ensure sqlite file delete on clean exit (#11612) 2024-10-18 10:33:28 +05:30
Tamas Nemeth
cf3634a6a9
fix(ingest/bigquery): Not setting platform instance for bigquery platform resources (#11659) 2024-10-17 19:57:02 +02:00
Tamas Nemeth
e76647dd7a
feat(ingest/bigquery): Generate platform resource entities for BigQuery labels (#11602)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2024-10-16 10:04:01 +02:00
Tamas Nemeth
3cb1678ecc
feat(ingest/bigquery): Add way to reference existing DataHub Tag from a bigquery label (#11544) 2024-10-09 17:01:34 +02:00
Mayuri Nehate
6a58493011
fix(ingest/bq): do not query PARTITIONS for biglake tables (#11463) 2024-09-27 16:46:37 +05:30
sid-acryl
9fb2df11f3
fix(ingest): sort by last modified not working in the UI (#11343) 2024-09-23 10:06:05 -07:00
Sergio Gómez Villamor
31edb46dbc
feat(ingestion): adds env property in ContainerProperties (#11214)
Co-authored-by: siladitya2 <siladitya2@gmail.com>
2024-09-18 14:56:52 +05:30
Mayuri Nehate
cf49f80e77
feat(ingest/sql): auto extract and use mode query user metadata (#11307)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-09 12:38:24 -07:00
Harshal Sheth
9589adebb3
feat(ingest): enable query usage stats by default (#11281) 2024-09-06 10:02:40 -07:00
Mayuri Nehate
08a5956e26
fix(bigquery): followups on bigquery queries v2 integration (#11291) 2024-09-04 07:42:35 -07:00
Mayuri Nehate
a7fc7f519a
feat(ingest/bq): integrate bigquery-queries into main source (#11247) 2024-08-30 18:16:45 -07:00
Mayuri Nehate
223650dd7a
feat(ingest): add bigquery-queries source (#10994)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-25 22:51:00 -07:00
sid-acryl
627c5abfd6
feat(ingestion/bigquery): Add ability to filter GCP project ingestion based on project labels (#11169)
Co-authored-by: Alice Naghshineh <alice.naghshineh@nytimes.com>
Co-authored-by: Alice Naghshineh <45885699+anaghshineh@users.noreply.github.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2024-08-20 14:42:00 -04:00
Mayuri Nehate
ff1c6b895e
feat(ingest/BigQuery): refactor+parallelize dataset metadata extraction (#10884) 2024-07-16 11:46:42 -07:00
sagar-salvi-apptware
d69966074a
fix(ingest/bigquery): Map BigQuery policy tags to datahub column-level tags (#10669) 2024-06-14 16:43:12 +05:30
Harshal Sheth
3d5735cbc5
chore(ingest): run pyupgrade for python 3.8 (#10513) 2024-05-15 22:31:05 -07:00
Harshal Sheth
7d31420b69
feat(ingest): materialize terms produced by ingestion (#10249) 2024-04-18 10:48:16 -07:00
Mayuri Nehate
77c72dad01
feat(ingest): add classification to bigquery, redshift (#10031) 2024-03-13 22:45:28 -07:00
AvaniSiddhapuraAPT
782d33db99
fix(ingest/bigquery): escape special characters for table descriptions (#9932) 2024-03-05 21:00:09 +05:30
Mayuri Nehate
cdb9f5ba62
feat(bigquery): add better timers around every API call (#8626) 2023-09-15 11:55:39 -07:00
Andrew Sikowitz
22c35f1a23
fix(ingest/bigquery): Add config option to create DataPlatformInstance, default off (#8659) 2023-08-24 14:46:06 +05:30
Andrew Sikowitz
3a21c27f06
feat(ingest): Turn on browse path v2 creation (#8342)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-06 16:43:42 -04:00
Andrew Sikowitz
c5cc53b99a
feat(ingest/bigquery_v2): enable platform instance using project id (#8216)
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-06-14 09:50:21 -07:00
Andrew Sikowitz
369a04ae30
revert(ingest/bigquery): Do not emit DataPlatformInstance; remove references to platform_instance (#8196) 2023-06-09 13:44:24 +05:30
Adrián Pertíñez
743439c11d
feat(ingest/bigquery_v2): enable platform instance using project id (#8142)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-06-05 15:17:40 -07:00