3975 Commits

Author SHA1 Message Date
Andrew Sikowitz
b778def10a
build(ingest/hive): Update thrift pin (#8964)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-16 17:19:25 -05:00
Andrew Sikowitz
5cb04981b2
test(ingest/unity): Update goldens (#9254) 2023-11-16 17:18:17 -05:00
Adriano Vega Llobell
d0fa5de9f0
fix(sql-parser): convert platform instance to lowercase when building table urns (#9181) 2023-11-16 13:58:49 -05:00
Harshal Sheth
417ffb12d8
feat(ingest): support advanced configs for aws (#9237) 2023-11-16 13:34:18 -05:00
Harshal Sheth
9d41a8f9f0
fix(ingest): drop redshift-legacy and redshift-usage-legacy sources (#9244) 2023-11-16 13:33:35 -05:00
Andrew Sikowitz
78abeb9beb
feat(ingest/unity): Support specifying catalogs directly; pass env correctly (#9110) 2023-11-16 12:41:12 -05:00
Pierre Guivarch
8475fc92b4
docs(ingest): fix typo in snowflake ingestion docs (#9239) 2023-11-14 21:42:13 -05:00
Harshal Sheth
ec13847f54
feat(airflow): make RUN_IN_THREAD configurable (#9226) 2023-11-14 14:25:26 -05:00
Tony Ouyang
cfeecd799d
feat(ingest/mongodb): support AWS DocumentDB for MongoDB (#9201) 2023-11-14 12:12:39 -05:00
Harshal Sheth
906a5b91a8
chore(ingest): cleanup various methods (#9221) 2023-11-13 21:51:11 -05:00
Harshal Sheth
19aa215068
feat(ingest/dbt): dbt column-level lineage (#8991) 2023-11-13 19:00:30 -05:00
david-leifker
ebd2e2312b
fix(datahub-ingestion): remove old jars, sync pyspark version (#9217) 2023-11-10 22:10:00 -06:00
Shubham Jagtap
a187127ac5
feat(ingestion): file-based state checkpoint provider (#9029) 2023-11-10 14:36:00 -08:00
Harshal Sheth
89dff8f7bd
chore(ingest): start working on pydantic v2 support (#9220) 2023-11-10 09:34:08 -08:00
Teppo Naakka
107713846f
fix(ingest/powerbi): use dataset workspace id as key for parent container (#8994) 2023-11-09 16:48:06 -08:00
Andrew Sikowitz
f73ecfdcbb
style(ingest/tableau): Rename tableau_constant to c (#9207) 2023-11-08 18:17:49 -05:00
Shubham Jagtap
e73e926999
feat(integration/fivetran): Fivetran connector integration (#9018)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-07 23:02:41 -08:00
Andrew Sikowitz
2c58c63780
fix(ingest/datahub-source): Order by version in memory (#9185) 2023-11-06 15:58:57 -05:00
Harshal Sheth
86d2b08d2b
docs(ingest): inherit capabilities from superclasses (#9174) 2023-11-06 15:58:07 -05:00
Harshal Sheth
02156662b5
docs(ingest): clarify adding source guide (#9161) 2023-11-06 12:47:07 -08:00
Harshal Sheth
148ad1ad9f
feat(ingest/looker): support emitting unused explores (#9159) 2023-11-02 09:44:35 -07:00
Harshal Sheth
ec9725026d
chore(ingest): remove legacy memory_leak_detector (#9158) 2023-11-02 09:39:08 -07:00
Mayuri Nehate
12b41713b4
fix(ingest/bigquery): use correct row count in null count profiling c… (#9123)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-11-01 21:35:24 -07:00
Harshal Sheth
bab9d1c931
fix(ingest): fully support MCPs in urn_iter primitive (#9157) 2023-11-01 20:47:18 -07:00
deepgarg-visa
f7cd80283a
fix(ingest/db2): fix handling for table properties (#9128)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-01 20:46:58 -07:00
Harshal Sheth
50789224a1
fix(ingest/mongodb): fix schema inference for lists of values (#9145) 2023-11-01 16:58:37 -07:00
Mayuri Nehate
95d9ff2cc2
fix(ingest): handle exceptions in min, max, mean profiling (#9129) 2023-11-01 13:14:16 -07:00
Tony Ouyang
876de214c9
feat(ingest/mongodb): support stateful ingestion (#9118) 2023-11-01 12:43:17 +05:30
Harshal Sheth
55f14530a3
feat(ingest): bump sqlglot parser (#9155) 2023-11-01 12:42:52 +05:30
Harshal Sheth
73514ad9c5
fix(ingest): cleanup large images in CI (#9153) 2023-10-31 21:28:38 -07:00
sachinsaju
dae320c9bc
docs(teradata): teradata doc external link 404 fix (#9152) 2023-10-31 15:58:39 -07:00
Harshal Sheth
ea1273281e
feat(ingest): sql parser perf + asyncio fixes (#9119) 2023-10-30 20:57:59 -07:00
Harshal Sheth
ce0f36b8bc
feat(sqlparser): support more update syntaxes + fix bug with subqueries (#9105) 2023-10-30 14:23:19 -07:00
Harshal Sheth
0bd2d9a36c
feat(ingest): add entity_supports_aspect helper (#9120) 2023-10-30 14:22:05 -07:00
Harshal Sheth
51d6d1f453
feat(ingest/looker): cleanup error handling (#9135) 2023-10-30 14:19:52 -07:00
Harshal Sheth
58bcedcd6a
fix(ingest/clickhouse): pin version to solve column reflection regression (#9143) 2023-10-30 14:18:48 -07:00
Alex Klavens
2c019148ad
fix(metadata-ingestion): adds default value to _resolved_domain_urn i… (#9115)
Co-authored-by: Alex Klavens <alexklavens@A14603.lan>
2023-10-30 14:13:52 +05:30
Dmytro Kulyk
758ed47644
feat(ingest/jsonschema) enable schema-aware JsonSchemaTranslator (#8971)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-29 21:14:32 -07:00
Ellie O'Neil
9ae0e93d82
docs(graphql): Correct mutation -> query for searchAcrossLineage examples (#9134) 2023-10-27 20:18:31 -07:00
Mayuri Nehate
e02b9096bd
fix(ingest/snowflake): missing view downstream cll if platform instance is set (#8966) 2023-10-27 12:03:43 -07:00
Mayuri Nehate
5166d90433
fix(ingest/sql-server): update queries to use escaped procedure name (#9127) 2023-10-27 10:19:51 -07:00
David Sanchez
cf617d77f3
Fix(ingest/bigquery): fix extracting comments from complex types (#8950)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-10-26 22:01:30 -07:00
nicholas.fwang
852267972c
refactor(ingestion/mongodb): Add platform_instance to mongodb (#8663)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-26 09:57:43 -07:00
Tamas Nemeth
a96a512166
fix(ingest/bigquery): Fixing lineage filter query (#9114) 2023-10-26 18:46:10 +02:00
Mayuri Nehate
f402090c1e
feat(ingest): support view lineage for all sqlalchemy sources (#9039) 2023-10-26 09:14:32 -07:00
Andrew Sikowitz
6c932e8afe
feat(ingest/bigquery): Attempt to support raw dataset pattern (#9109) 2023-10-25 13:17:09 -07:00
Harshal Sheth
32f5dcb154
fix(ingest): better handling around sink errors (#9003) 2023-10-25 13:16:49 -07:00
siddiquebagwan-gslab
b9508e6dd5
fix(ingestion/redshift): fix schema field data type mappings 2023-10-25 11:18:15 -07:00
Andrew Sikowitz
8a80e858a7
build(ingest): Bump avro pin: security vulnerability (#9042) 2023-10-25 13:06:12 -04:00
siddiquebagwan-gslab
b612545220
feat(ingestion/redshift): support auto_incremental_lineage (#9010) 2023-10-25 15:26:06 +05:30