3975 Commits

Author SHA1 Message Date
Mayuri Nehate
01a2c0c779
fix(ingest/kafka): update dependency, tests (#12159) 2024-12-18 19:02:44 +05:30
Aseem Bansal
2285436a62
fix(ingest/gc): more logging, error handling, explicit flag (#12124) 2024-12-18 17:50:38 +05:30
Harshal Sheth
76cfac3700
fix(cli/properties): allow structured properties without a graph instance (#12144)
Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com>
2024-12-18 09:04:51 +00:00
Alice-sky
5946558c01
fix(ingest/pulsar): handle Avro schema with missing namespace or name (#12058)
Co-authored-by: Alice <spongebob2023123@163.com>
Co-authored-by: Shirshanka Das <shirshanka+github@gmail.com>
Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-12-18 07:21:41 +00:00
Harshal Sheth
826437612e
fix(ingest/snowflake): improve lineage parse failure logging (#12153) 2024-12-17 20:21:05 -08:00
Harshal Sheth
d5ab001a97
feat(ingest/git): add subdir support to GitReference (#12131)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: harshal.sheth@acryl.io <harshal.sheth@acryl.io>
2024-12-17 10:54:37 -08:00
Andrew Sikowitz
ff385edbb1
build(ingest/sqlglot): Bump pin to support snowflake CREATE ... WITH TAG (#12003) 2024-12-17 03:49:47 -08:00
Harshal Sheth
d2359e259a
feat(ingest/datahub): report progress on db ingestion (#12117) 2024-12-17 09:58:47 +01:00
Austin SeungJun Park
8f9659fadf
fix(ingest/s3): incorrectly parsing path in s3_uri (#12135) 2024-12-17 09:57:03 +01:00
kevinkarchacryl
d0b4f7a7d3
feat(cli): added cli option for ingestion source (#11980) 2024-12-16 16:03:11 -05:00
Andrew Sikowitz
6b8d21a2ab
feat(ingest/sqlite): Support sqlite < 3.24.0 (#12137) 2024-12-16 12:50:25 -08:00
kousiknandy
ca6f435d03
docs(ingest): fix sink recipe to correct config parameter (#12132) 2024-12-16 12:08:21 -08:00
sid-acryl
74927969aa
feat(ingest/tableau): verify role assignment to user in test_connection (#12042)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-12-16 11:55:58 -08:00
Tamas Nemeth
0ea2e36226
fix(ingest/datahub): Use server side cursor instead of local one (#12129) 2024-12-16 20:38:24 +01:00
sid-acryl
c97fd1f8c0
fix(ingest/tableau): honor the key projectNameWithin in pagination (#12107) 2024-12-16 11:32:05 -08:00
Mayuri Nehate
d38427a773
feat(ingest/snowflake): secure view lineage without owner permissions (#12123) 2024-12-16 17:49:49 +05:30
Harshal Sheth
05ba40355d
refactor(ingest): cleanup structured properties validation (#12115) 2024-12-14 07:12:53 -08:00
Harshal Sheth
f9ca305600
feat(ingest): additional limits on ingestProposalBatch (#12130) 2024-12-14 07:09:04 -08:00
Aseem Bansal
0689c4de32
fix(ingest/snowflake): improve warn message (#12125)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2024-12-13 23:09:30 +05:30
Mayuri Nehate
d5e05131d5
fix(ingest/kafka-connect): update connection test url, handle api failures (#12082) 2024-12-13 16:39:29 +05:30
Jonny Dixon
06edf23a33
fix(ingestion/dremio): Ignore filtered containers in schema allowdeny pattern (#11959)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2024-12-13 14:55:31 +05:30
Harshal Sheth
bc4c7c633e
docs(ingest): add docs on the SQL parser (#12103) 2024-12-12 13:51:24 -08:00
Harshal Sheth
e730afdb68
feat(ingest): improve query fingerprinting (#12104) 2024-12-12 13:51:18 -08:00
Harshal Sheth
4683bc73a3
fix(ingest/snowflake): further improve dot handling (#12110) 2024-12-12 13:51:08 -08:00
kevinkarchacryl
507087a070
feat(glossary): support multiple ownership types (#12050)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2024-12-12 08:13:08 -05:00
sid-acryl
2ec9cb0536
fix(ingestion/lookml): resolve CLL issue caused by column name casing. (#11876) 2024-12-12 15:32:56 +05:30
Mayuri Nehate
c0b49a638f
docs(ingest/snowflake): update permissions for dynamic tables (#12074) 2024-12-12 15:32:15 +05:30
sid-acryl
47bd4464a3
fix(ingest/tableau): apply page_size regardless of object count (#12026) 2024-12-12 15:31:50 +05:30
Harshal Sheth
93c8ae2267
fix(ingest/snowflake): handle dots in snowflake table names (#12105) 2024-12-12 15:31:32 +05:30
Aseem Bansal
16a5db1f0c
feat(cli): add --workers arg in delete command (#12102) 2024-12-12 15:29:40 +05:30
Aseem Bansal
9a7f98cd05
fix(ingest/sdk): report recipe correctly (#12101) 2024-12-12 15:29:00 +05:30
Mayuri Nehate
3ae82f6b49
fix(ingest): always resolve platform for browse path v2 (#12045) 2024-12-12 15:24:35 +05:30
Mayuri Nehate
84de42d718
feat(ingest/snowflake): ingest secure, dynamic, hybrid table metadata (#12094) 2024-12-12 12:04:06 +05:30
sid-acryl
d062411c68
fix(ingest/tableau): project_path_pattern use in _is_denied_project (#12010) 2024-12-11 11:38:52 -08:00
Chris Collins
f1ef4f8e5f
feat(structuredProperties) Add new settings aspect plus graphql changes for structured props (#12052) 2024-12-11 13:59:14 -05:00
skrydal
b091e4615d
feat(ingest/kafka): Flag for optional schemas ingestion (#12077) 2024-12-11 16:02:31 +00:00
Jonny Dixon
e6cc676b23
fix(ingest/abs): detect jsonl schema (#11775) 2024-12-11 18:24:47 +05:30
Aseem Bansal
54d53cfc62
fix(ingest/gc): minor tweak gc source (#12093) 2024-12-11 18:24:18 +05:30
Mayuri Nehate
becf51575e
docs(ingest/athena): update recipe with aws key pair example (#12076) 2024-12-11 18:21:58 +05:30
Aseem Bansal
ff7ac48021
fix(cli): don't use /api in gms url (#12083) 2024-12-11 16:11:08 +05:30
Mayuri Nehate
355a7e69aa
feat(ingest/snowflake): allow option for incremental properties (#12080) 2024-12-11 16:08:39 +05:30
Harshal Sheth
d953718ab7
feat(ingest): allow max_workers=1 with ASYNC_BATCH rest sink (#12088) 2024-12-10 18:32:52 -05:00
sagar-salvi-apptware
57b12bd9cb
fix(ingest): replace sqllineage/sqlparse with our SQL parser (#12020) 2024-12-10 08:36:01 -08:00
Shirshanka Das
e4ea993df1
fix(py-sdk): DataJobPatchBuilder handling timestamps, output edges (#12067)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-12-09 19:40:31 -05:00
haeniya
4811de1c1d
feat(ingestion/tableau): hidden asset handling (#11559)
Co-authored-by: Yanik Häni <Yanik.Haeni1@swisscom.com>
2024-12-09 14:29:11 +05:30
Sergio Gómez Villamor
0e7ebaf0b2
feat(tableau): adds more reporting metrics to better understand lineage construction in tableau ingestion (#12008)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-12-09 07:29:42 +00:00
Tamas Nemeth
70bec48088
fix(ingest/gc): Additional dataprocess cleanup fixes (#12049) 2024-12-07 13:45:14 +01:00
Tamas Nemeth
1d5ddf0c04
fix(ingest/sagemaker): Adding option to control retry for any aws source (#8727) 2024-12-07 13:40:32 +01:00
hwmarkcheng
46aa962bad
feat(ingest/superset): initial support for superset datasets (#11972) 2024-12-06 13:48:00 -08:00
Harshal Sheth
bd7649ed89
refactor(ingest/sql): add _get_view_definition helper method (#12033) 2024-12-06 08:17:41 -08:00