3089 Commits

Author SHA1 Message Date
Pierre Guivarch
8475fc92b4
docs(ingest): fix typo in snowflake ingestion docs (#9239) 2023-11-14 21:42:13 -05:00
Harshal Sheth
ec13847f54
feat(airflow): make RUN_IN_THREAD configurable (#9226) 2023-11-14 14:25:26 -05:00
Tony Ouyang
cfeecd799d
feat(ingest/mongodb): support AWS DocumentDB for MongoDB (#9201) 2023-11-14 12:12:39 -05:00
Harshal Sheth
906a5b91a8
chore(ingest): cleanup various methods (#9221) 2023-11-13 21:51:11 -05:00
Harshal Sheth
19aa215068
feat(ingest/dbt): dbt column-level lineage (#8991) 2023-11-13 19:00:30 -05:00
Shubham Jagtap
a187127ac5
feat(ingestion): file-based state checkpoint provider (#9029) 2023-11-10 14:36:00 -08:00
Harshal Sheth
89dff8f7bd
chore(ingest): start working on pydantic v2 support (#9220) 2023-11-10 09:34:08 -08:00
Teppo Naakka
107713846f
fix(ingest/powerbi): use dataset workspace id as key for parent container (#8994) 2023-11-09 16:48:06 -08:00
Andrew Sikowitz
f73ecfdcbb
style(ingest/tableau): Rename tableau_constant to c (#9207) 2023-11-08 18:17:49 -05:00
Shubham Jagtap
e73e926999
feat(integration/fivetran): Fivetran connector integration (#9018)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-07 23:02:41 -08:00
Andrew Sikowitz
2c58c63780
fix(ingest/datahub-source): Order by version in memory (#9185) 2023-11-06 15:58:57 -05:00
Harshal Sheth
86d2b08d2b
docs(ingest): inherit capabilities from superclasses (#9174) 2023-11-06 15:58:07 -05:00
Harshal Sheth
148ad1ad9f
feat(ingest/looker): support emitting unused explores (#9159) 2023-11-02 09:44:35 -07:00
Harshal Sheth
ec9725026d
chore(ingest): remove legacy memory_leak_detector (#9158) 2023-11-02 09:39:08 -07:00
Mayuri Nehate
12b41713b4
fix(ingest/bigquery): use correct row count in null count profiling c… (#9123)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-11-01 21:35:24 -07:00
Harshal Sheth
bab9d1c931
fix(ingest): fully support MCPs in urn_iter primitive (#9157) 2023-11-01 20:47:18 -07:00
deepgarg-visa
f7cd80283a
fix(ingest/db2): fix handling for table properties (#9128)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-01 20:46:58 -07:00
Harshal Sheth
50789224a1
fix(ingest/mongodb): fix schema inference for lists of values (#9145) 2023-11-01 16:58:37 -07:00
Mayuri Nehate
95d9ff2cc2
fix(ingest): handle exceptions in min, max, mean profiling (#9129) 2023-11-01 13:14:16 -07:00
Tony Ouyang
876de214c9
feat(ingest/mongodb): support stateful ingestion (#9118) 2023-11-01 12:43:17 +05:30
Harshal Sheth
ea1273281e
feat(ingest): sql parser perf + asyncio fixes (#9119) 2023-10-30 20:57:59 -07:00
Harshal Sheth
ce0f36b8bc
feat(sqlparser): support more update syntaxes + fix bug with subqueries (#9105) 2023-10-30 14:23:19 -07:00
Harshal Sheth
0bd2d9a36c
feat(ingest): add entity_supports_aspect helper (#9120) 2023-10-30 14:22:05 -07:00
Harshal Sheth
51d6d1f453
feat(ingest/looker): cleanup error handling (#9135) 2023-10-30 14:19:52 -07:00
Alex Klavens
2c019148ad
fix(metadata-ingestion): adds default value to _resolved_domain_urn i… (#9115)
Co-authored-by: Alex Klavens <alexklavens@A14603.lan>
2023-10-30 14:13:52 +05:30
Dmytro Kulyk
758ed47644
feat(ingest/jsonschema) enable schema-aware JsonSchemaTranslator (#8971)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-29 21:14:32 -07:00
Mayuri Nehate
e02b9096bd
fix(ingest/snowflake): missing view downstream cll if platform instance is set (#8966) 2023-10-27 12:03:43 -07:00
Mayuri Nehate
5166d90433
fix(ingest/sql-server): update queries to use escaped procedure name (#9127) 2023-10-27 10:19:51 -07:00
David Sanchez
cf617d77f3
Fix(ingest/bigquery): fix extracting comments from complex types (#8950)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-10-26 22:01:30 -07:00
nicholas.fwang
852267972c
refactor(ingestion/mongodb): Add platform_instance to mongodb (#8663)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-26 09:57:43 -07:00
Tamas Nemeth
a96a512166
fix(ingest/bigquery): Fixing lineage filter query (#9114) 2023-10-26 18:46:10 +02:00
Mayuri Nehate
f402090c1e
feat(ingest): support view lineage for all sqlalchemy sources (#9039) 2023-10-26 09:14:32 -07:00
Andrew Sikowitz
6c932e8afe
feat(ingest/bigquery): Attempt to support raw dataset pattern (#9109) 2023-10-25 13:17:09 -07:00
Harshal Sheth
32f5dcb154
fix(ingest): better handling around sink errors (#9003) 2023-10-25 13:16:49 -07:00
siddiquebagwan-gslab
b9508e6dd5
fix(ingestion/redshift): fix schema field data type mappings 2023-10-25 11:18:15 -07:00
Andrew Sikowitz
8a80e858a7
build(ingest): Bump avro pin: security vulnerability (#9042) 2023-10-25 13:06:12 -04:00
siddiquebagwan-gslab
b612545220
feat(ingestion/redshift): support auto_incremental_lineage (#9010) 2023-10-25 15:26:06 +05:30
Andrew Sikowitz
2d1584b12f
feat(ingest/datahub-source): Allow ingesting aspects from the entitiesV2 API (#9089) 2023-10-24 23:08:24 -04:00
Harshal Sheth
916235d31a
fix(ingest): update athena type mapping (#9061) 2023-10-24 19:59:42 -07:00
Harshal Sheth
d13553f53a
feat(sqlparser): extract CLL from updates (#9078) 2023-10-24 12:24:50 -07:00
Andrew Sikowitz
eb0b03d2f2
fix(ingest/bigquery): Correctly apply table pattern to read events; fix end time calculation; deprecate match_fully_qualified_names (#9077) 2023-10-24 10:45:09 -04:00
Andrew Sikowitz
adf8c8db38
refactor(ingest): Move sqlalchemy import out of sql_types.py (#9065) 2023-10-24 08:59:56 +02:00
Harshal Sheth
8fb95e88a1
feat(sqlparser): parse create DDL statements (#9002) 2023-10-23 12:40:42 -07:00
Tamas Nemeth
04216e30bb
feat(ingest/s3): S3 add partition to schema (#8900)
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-10-21 16:21:54 +01:00
Harshal Sheth
2fea466d48
docs: fix lineage capability annotations (#8954) 2023-10-20 13:47:52 -07:00
Tim
1eaf9c8c5f
feature(ingest/athena): introduce support for complex and nested schemas in Athena (#8137)
Co-authored-by: dnks23 <dominik.s23@live.de>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Tim <tim@MBP-von-Tim.fritz.box>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-18 09:39:59 -07:00
Andrew Sikowitz
d2eb42373f
fix(ingest/sqlalchemy): Fix URL parsing when sqlalchemy_uri provided (#9032) 2023-10-18 17:34:45 +02:00
Mayuri Nehate
9bb0bf25ae
fix(ingest): fix typo in parsing list of groups (#9037) 2023-10-18 14:09:27 +05:30
Andrew Sikowitz
da6cc54d63
fix(ingest/bigquery): Remove table name restrictions (allow $ and @) (#9030) 2023-10-17 16:18:39 -04:00
Tamas Nemeth
10eb205cb8
fix(ingest) - Fix file backed collection temp directory removal (#9027) 2023-10-17 10:16:25 -04:00