724 Commits

Author SHA1 Message Date
Mayuri Nehate
8b79461bd5
feat(ingest/looker): browse path followups (#10217) 2024-04-12 10:21:06 -07:00
Dotan Mor
fa0c1b3fa9
feat(ingest/cockroachdb): add cockroachdb ingestion (#10226) 2024-04-09 18:36:51 -07:00
Mayuri Nehate
6997abd42e
feat(ingest/nifi): ingest process group as browse path v2, incremental lineage (#10202)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-04-09 17:44:20 +05:30
Tamas Nemeth
00a890f84f
fix(ingest/bigquery): fix lineage if multiple sql expression passed in and destination table set (#10212)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-04-09 11:54:02 +02:00
Harshal Sheth
bdf2c9a5c1
feat(ingest/sql): normalize bigquery partitioned tables when parsing (#10224) 2024-04-07 17:17:28 +02:00
Harshal Sheth
e74347812b
fix(ingest/dbt): better dbt timestamp parsing (#10223) 2024-04-05 16:13:18 -07:00
ACHRAF BOUAOUDA
b5615fac54
feat(ingest/great_expectations): support in-memory (Pandas) data assets (#9811)
Co-authored-by: Achraf BOUAOUDA <achraf_bouada@carrefour.com>
2024-04-04 12:46:59 -07:00
Shubham Jagtap
fa139a582e
feat(ingestion/transformer): Handle overlapping while mapping in extract ownership from tags transformer (#10201) 2024-04-04 12:19:11 -07:00
dushayntAW
3c7c3ec904
fix(ingestion/glue): fix to ingest the comment for partition key as description (#10189) 2024-04-03 17:34:02 +05:30
Tamas Nemeth
5c06f7a245
fix(ingest/bigquery): Supporting lineage extraction in case the select query result's target table is set on job (#10191)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-04-02 22:13:05 +02:00
Harshal Sheth
c9b9afc530
feat(ingest/dbt): enable model performance and compiled code by default (#10164) 2024-04-02 09:29:27 -05:00
RyanHolstien
ef637ccb37
fix(docker): fix versioning for compose file post release (#10176) 2024-04-01 15:01:09 -05:00
Shubham Jagtap
9f2c5d36f3
feat(ingestion/bigquery): BigQuery Owner Label to Datahub Ownership (#10047) 2024-03-28 15:50:25 -07:00
Harshal Sheth
25d9d6656c
feat(ingest): fix validators (#10115)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-03-27 15:20:55 -07:00
Harshal Sheth
f0bdc24fc9
feat(ingest/dbt): dbt model performance (#9992) 2024-03-26 17:18:54 -07:00
Harshal Sheth
a70e775a12
feat(ingest): emit platform for query entities (#10103) 2024-03-26 11:22:53 -07:00
dushayntAW
dd502ae662
fix(ingest): added new transformer to cleanup suffix/prefix in owner URN (#10067) 2024-03-22 15:23:03 +05:30
Harshal Sheth
af06f95c5e
fix(ingest/dbt): fix config validator for skip_sources_in_lineage (#10098) 2024-03-21 15:18:37 -07:00
alexs-101
e6e5c091ed
feat(tableau): ability to force extraction of table/column level linage from SQL queries (#9838) 2024-03-21 09:27:22 +01:00
Harshal Sheth
8c21b178df
feat(ingest): support incremental column-level lineage (#10090) 2024-03-21 09:18:12 +01:00
siladitya
43ac405c57
fix(metadata-ingestion)glue connector failure when Optional field Type of PartitionKey is absent for a Table (#10052) 2024-03-20 11:02:28 +01:00
Harshal Sheth
630f2a13f6
feat(ingest): check for private import path usages (#10059) 2024-03-18 14:59:18 -07:00
Harshal Sheth
d4d175da1a
fix(ingest/redshift): tweak lineage v2 queries (#10045) 2024-03-15 12:16:18 -07:00
Harshal Sheth
b0163c4885
feat(ingest): utilities for query logs (#10036) 2024-03-12 23:20:46 -07:00
Tamas Nemeth
4535f2adfd
fix(ingest/redshift): Temp table lineage fix (#10008)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-03-12 14:31:00 -07:00
Harshal Sheth
7e2076e852
feat(ingest): add query formatting to sql aggregator (#10025) 2024-03-11 15:34:44 -07:00
Harshal Sheth
92a3ac6f11
fix(ingest): use contextvar for cooperative timeout (#10021) 2024-03-11 14:14:39 -07:00
ethan-cartwright
7540e642a5
fix(ingest/redshift): patch instead of replace redshift custom properties (#9293)
Co-authored-by: Ethan Cartwright <ethan.cartwright@acryl.io>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2024-03-11 11:16:44 -07:00
Harshal Sheth
b6956f9a5c
feat(ingest): update sqlglot fork (#10022) 2024-03-11 15:22:30 +01:00
Mayuri Nehate
8047d0b514
feat(ingest/snowflake): generate query nodes for snowflake (#9966)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-03-06 15:11:37 -08:00
Shubham Jagtap
fda5eb89f7
feat(ingest): enable stateful_ingestion by default for DataHub rest sink (#9934) 2024-03-05 11:18:03 -08:00
AvaniSiddhapuraAPT
782d33db99
fix(ingest/bigquery): escape special characters for table descriptions (#9932) 2024-03-05 21:00:09 +05:30
AvaniSiddhapuraAPT
1b4f31bcfe
fix(ingest/json-schema): adding support descriptions for array (#9757) 2024-03-01 13:03:47 -08:00
david-leifker
ed10a8d8cc
feat(search): search access controls (#9892)
Co-authored-by: Chris Collins <chriscollins3456@gmail.com>
2024-02-28 16:57:26 -06:00
Harshal Sheth
1736edf8f5
feat(ingest): fix bugs in SqlParsingAggregator (#9926) 2024-02-28 15:06:33 +01:00
Aditya Malik
92b1cfa194
feat(ingest): Support for JSONL in s3 source with max_rows support (#9921)
Co-authored-by: Aditya <aditya.malik@quillbot.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-02-28 15:05:30 +01:00
RyanHolstien
ddf0b7d2cd
feat(patch): refactor cll patch (#9922)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-02-27 12:00:44 -06:00
Aseem Bansal
336d0543c1
feat(ingest/csv-enricher): add support for ownership type urn (#9935) 2024-02-27 17:47:04 +00:00
Tamas Nemeth
d2d9661a1e
fix(ingest/patch): Adding proper json patch path quoting (#9872) 2024-02-26 18:46:21 -08:00
Harshal Sheth
392158809d
feat(ingest/redshift): redshift lineage v2 (#9904) 2024-02-23 16:32:51 -08:00
Harshal Sheth
b01416c11b
feat(ingest): add timeout to sqlglot parser (#9868) 2024-02-23 11:11:02 -08:00
Harshal Sheth
ef3a8148ec
fix(ingest): bump sqlglot + add some debug info to tests (#9867) 2024-02-21 13:07:59 -08:00
Harshal Sheth
b15b352a00
fix(ingest): better logging line attribution (#9876) 2024-02-21 11:37:57 -08:00
Harshal Sheth
f13ae77966
feat(ingest): add escape hatch methods to SqlParsingAggregator (#9860) 2024-02-21 11:33:05 -08:00
Harshal Sheth
24da7a573b
test(ingest): use pytest-random-order for unit tests (#9753) 2024-02-09 18:20:05 -05:00
Harshal Sheth
0d780e5f8f
feat(ingest): sql parsing aggregator (#9786) 2024-02-09 16:27:45 -05:00
Harshal Sheth
f8f1af2735
feat(ingest/dbt): speed up test result only ingestion (#9791) 2024-02-08 23:17:32 +01:00
skrydal
bf1ce96ae6
feat(cli): Add support for custom ownership types to dataproduct CLI (#9762)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-02-05 08:29:14 -08:00
Andrea Scarpino
f743d3c044
feat(ingest/metabase): Allow to configure a different url for links (#9727) 2024-02-05 17:36:50 +05:30
sid-acryl
533130408a
feat(ingestion/redshift): collapse lineage to permanent table (#9704)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: treff7es <treff7es@gmail.com>
2024-02-01 12:47:09 -08:00