73 Commits

Author SHA1 Message Date
Andrew R Smith
647fb792de
feat(ingest): add snowflake ingestion config options (#12841)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2025-05-27 09:22:16 -07:00
Sergio Gómez Villamor
8cae980286
tests(ingestion): moving some tests so they are available for sdk users (#13540) 2025-05-19 08:39:53 +02:00
Harshal Sheth
d3944ded93
feat(ingest/snowflake): generate lineage through temp views (#13517) 2025-05-16 21:27:13 -07:00
Harshal Sheth
7c791db087
feat(ingest/sql): column logic + join extraction (#13426) 2025-05-12 17:19:44 -07:00
Harshal Sheth
287f373a9c
fix(ingest/snowflake): fix previously broken tests (#13428) 2025-05-06 10:19:25 -07:00
Harshal Sheth
f48c6b53ee
feat(ingest/snowflake): show returned query row counts (#13246) 2025-04-21 14:41:40 -07:00
Chakru
54156ea78a
fix(cli): use patch to update dataset properties (#13226) 2025-04-21 14:41:31 -07:00
Harshal Sheth
9f7f3cb886
chore(ingest/snowflake): remove unused query code (#13245) 2025-04-21 10:31:36 -07:00
Andrew Sikowitz
ca51df880f
fix(ingest/snowflake): Use CREATE change type when creating structured properties; support MCP headers (#13158) 2025-04-10 15:13:13 -05:00
Mayuri Nehate
ac9997d970
feat(ingest/snowflake): ingest stored procedures (#12929)
Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com>
2025-03-26 20:02:55 +05:30
Mayuri Nehate
fbed37b374
fix(ingest/snowflake): update logic to identify hybrid tables (#12672) 2025-02-20 11:23:57 +05:30
Mayuri Nehate
180087f992
fix(ingest): remove duplicate mcps,more typing (#12557) 2025-02-07 14:21:31 +05:30
Brock Griffey
ac13f255e6
feat(ingestion/snowflake):adds streams as a new dataset with lineage and properties. (#12318)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2025-02-05 10:48:37 +05:30
Tamas Nemeth
4dc9bfcde7
fix(ingest/snowflake): Create all structured propery templates before assignation (#12469) 2025-01-30 13:49:20 -08:00
Mayuri Nehate
0f538d8df2
fix(ingest): fix reporting for missing secure view lineage (#12430)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2025-01-24 14:51:52 +05:30
Andrew Sikowitz
ebbadf65ad
feat(ingest/snowflake): Support ingesting snowflake tags as structured properties (#12285) 2025-01-14 01:50:14 -08:00
Mayuri Nehate
3ca8d09100
feat(ingest/snowflake): support email_as_user_identifier for queries v2 (#12219) 2024-12-27 11:40:00 +05:30
Mayuri Nehate
4d990b06bd
fix(ingest/snowflake): always ingest view and external table ddl lineage (#12191) 2024-12-24 18:14:51 +05:30
Mayuri Nehate
d38427a773
feat(ingest/snowflake): secure view lineage without owner permissions (#12123) 2024-12-16 17:49:49 +05:30
Mayuri Nehate
355a7e69aa
feat(ingest/snowflake): allow option for incremental properties (#12080) 2024-12-11 16:08:39 +05:30
Harshal Sheth
5519a330e2
chore(ingest): bump black (#11898) 2024-11-20 13:33:54 -08:00
Mayuri Nehate
179a6714a6
feat(ingest): ensure sqlite file delete on clean exit (#11612) 2024-10-18 10:33:28 +05:30
Sergio Gómez Villamor
31edb46dbc
feat(ingestion): adds env property in ContainerProperties (#11214)
Co-authored-by: siladitya2 <siladitya2@gmail.com>
2024-09-18 14:56:52 +05:30
Alisa Aylward
675b225c5b
feat(ingest/snowflake): Add cluster formula to dataset properties (#11254) 2024-08-29 16:11:35 -07:00
Alisa Aylward
ca2e134d34
fix(ingest/snowflake): exclude snowflake excluded tags (#11250) 2024-08-27 08:48:59 -07:00
Alisa Aylward
88b3893b5c
feat(ingest/snowflake): allow iceberg tables in lineage and access metadata (#10961) 2024-08-22 01:20:51 +00:00
Harshal Sheth
a4bce6af1c
feat(ingest): add snowflake-queries source (#10835) 2024-07-12 15:08:51 -07:00
John Joyce
fa3e381f83
refactor(ingest): Refactor structured logging to support infos, warnings, and failures structured reporting to UI (#10828)
Co-authored-by: John Joyce <john@Johns-MBP.lan>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-07-03 19:20:58 -07:00
Harshal Sheth
0d677e4992
fix(ingest/snowflake): fix column batcher (#10781) 2024-06-25 22:21:54 -07:00
Harshal Sheth
0dc0bc5761
feat(ingest/snowflake): performance improvements (#10746) 2024-06-25 14:46:55 -07:00
Harshal Sheth
2d727a960b
feat(ingest/snowflake): support more than 10k views in a db (#10718) 2024-06-18 07:37:39 +02:00
Harshal Sheth
62c6704f69
feat(ingest/snowflake): refactor + parallel schema extraction (#10653) 2024-06-14 13:23:07 -07:00
Harshal Sheth
e842161849
feat(ingest): add fast query fingerprinting (#10619) 2024-06-05 13:47:44 -07:00
Harshal Sheth
3d5735cbc5
chore(ingest): run pyupgrade for python 3.8 (#10513) 2024-05-15 22:31:05 -07:00
Mayuri Nehate
f6627efe71
fix(ingest/snowflake): add more reporting for usage aggregation, handle lineage errors (#10279)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-05-07 08:42:39 -07:00
Harshal Sheth
7d31420b69
feat(ingest): materialize terms produced by ingestion (#10249) 2024-04-18 10:48:16 -07:00
Harshal Sheth
a70e775a12
feat(ingest): emit platform for query entities (#10103) 2024-03-26 11:22:53 -07:00
Mayuri Nehate
77c72dad01
feat(ingest): add classification to bigquery, redshift (#10031) 2024-03-13 22:45:28 -07:00
Harshal Sheth
7e2076e852
feat(ingest): add query formatting to sql aggregator (#10025) 2024-03-11 15:34:44 -07:00
Mayuri Nehate
8047d0b514
feat(ingest/snowflake): generate query nodes for snowflake (#9966)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-03-06 15:11:37 -08:00
Harshal Sheth
ecda3e6187
feat(ingest): pydantic v2 compatibility (#9434) 2023-12-18 18:26:33 -05:00
Tamas Nemeth
03590a1948
fix(ingest/snowflake) - Fixing snowflake url with default region (#9443) 2023-12-18 13:54:31 -05:00
Tamas Nemeth
0ea6145a9d
fix(ingest/profiling): Add option to enable external table profiling (#9463) 2023-12-15 00:12:45 +01:00
Mayuri Nehate
08fb730676
fix(ingest): drop deprecated database_alias from sql sources (#9299)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-28 15:49:49 -05:00
Tamas Nemeth
f794a90515
fix(ingest/snowflake): Apply email filter on all usage metrics (#9269) 2023-11-23 09:45:25 +01:00
Harshal Sheth
ec9725026d
chore(ingest): remove legacy memory_leak_detector (#9158) 2023-11-02 09:39:08 -07:00
Mayuri Nehate
e02b9096bd
fix(ingest/snowflake): missing view downstream cll if platform instance is set (#8966) 2023-10-27 12:03:43 -07:00
Mayuri Nehate
f402090c1e
feat(ingest): support view lineage for all sqlalchemy sources (#9039) 2023-10-26 09:14:32 -07:00
Mayuri Nehate
8d175ef7ef
feat(ingest): incremental lineage source helper (#8941)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-09 13:34:25 -07:00
Aseem Bansal
2bc685d3b9
ci: tweak ci to decrease wait time of devs (#8945) 2023-10-05 09:31:32 +05:30