3089 Commits

Author SHA1 Message Date
Bumsoo Kim
e41f42b9db
fix(ingest): fix typo in urn utilities (#6520)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-11-23 01:05:32 -05:00
Mayuri Nehate
22847a987a
feat(ingest): automated term classification for snowflake (#6376) 2022-11-23 00:43:30 -05:00
Harshal Sheth
2b9b123a85
fix(ingest): make kafka emit callback optional (#6525) 2022-11-22 16:57:14 -05:00
Mayuri Nehate
a1b3929b38
fix(ingest): fix typo in unique count profiling (#6517) 2022-11-22 14:23:01 -05:00
Mayuri Nehate
e085a9e7dc
feat(ingest): add config for ingesting delta table without files (#6403)
Closes undefined
2022-11-22 14:15:40 -05:00
Harshal Sheth
2cb1e6576d
fix(ingest): add alias for bigquery-beta (#6521) 2022-11-22 14:15:20 -05:00
Harshal Sheth
1229c1f60d
feat(airflow): support kafka sinks in the airflow plugin (#6508) 2022-11-22 10:49:00 -08:00
Harshal Sheth
3fdaf13f2c
feat(ingest/csv-enrich): handle BOM character (#6509)
This has come up a number of times e.g. https://datahubspace.slack.com/archives/C029A3M079U/p1669000226732159?thread_ts=1669000226.732159&amp;cid=C029A3M079U
2022-11-22 12:53:34 -05:00
Harshal Sheth
74cc88f2df
fix(ingest): correctly handle transformer patch semantics (#6505) 2022-11-22 09:29:57 -08:00
Harshal Sheth
10a31b1aa0
fix(ingest/snowflake): fix lineage allow/deny pattern typo (#6506) 2022-11-22 07:09:48 +01:00
Harshal Sheth
490097e532
fix(ingest): remove redundant types (#6486)
Possible since https://github.com/python/typeshed/pull/9220 was merged.

Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-21 23:08:05 +01:00
Harshal Sheth
fdb74b0524
fix(tableau): extra defensive error-handling (#6503) 2022-11-21 14:40:18 -05:00
Harshal Sheth
05a0f3e2a6
feat(ingest): dbt cloud integration (#6323) 2022-11-21 14:14:33 -05:00
Mayuri Nehate
9c1577da13
feat(ingest): add stateful ingestion support to looker and lookml source (#6443) 2022-11-21 13:44:34 -05:00
Lukasz Stefaniak
bc5e2d43af
fix(ingest/tableau): check tableName existence on datasource response (#6478)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-11-21 11:39:19 -05:00
Tamas Nemeth
250f7ce1a8
feat(ingest): presto - Adding presto source (#6459) 2022-11-18 12:02:48 +01:00
Harshal Sheth
817406eadb
refactor(ingest): simplify stateful ingestion config (#6454)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-18 00:09:24 -05:00
Harshal Sheth
b7c03731c4
fix(ingest): fix athena and GE lint errors (#6482) 2022-11-17 19:07:05 -05:00
Aditya Radhakrishnan
738b6a28d0
fix(docs): update the Okta and Azure AD docs to clarify the point of ingesting users (#6465) 2022-11-17 11:09:41 -08:00
Harshal Sheth
b73ebe69b5
revert: mssql - lowercase db name in mssql ingestion (#6481) 2022-11-17 19:42:38 +01:00
Harshal Sheth
d3174e75b7
fix(ingest): dbt - disable incremental lineage by default (#6467) 2022-11-17 09:41:43 +01:00
mohdsiddique
73bf7a81a9
feat(ingestion): powerbi - support for report (#6339)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2022-11-16 22:52:23 -08:00
Harshal Sheth
29c6c43cb3
feat(ingest): detect old or missing docker compose (#6466) 2022-11-16 22:38:00 -08:00
Harshal Sheth
8c322ede35
feat(ingest): allow specific profiler config fields to override profile_table_level_only (#6366) 2022-11-16 23:49:31 +01:00
Harshal Sheth
a3aec72c3d
fix(ingest): handle missing schema in transformer (#6445) 2022-11-16 23:37:43 +01:00
Harshal Sheth
cf6fb91f8d
fix(ingest): mssql - lowercase db name in mssql ingestion (#6448) 2022-11-16 23:34:08 +01:00
Harshal Sheth
e530faf865
fix(ingest): add cli info to ingestion reporter (#6451) 2022-11-16 12:16:23 -05:00
Harshal Sheth
1ffd24131e
chore(ingest): fix mssql lint (#6453) 2022-11-15 23:11:57 -05:00
david-leifker
33fd876421
feat(elasticsearch): Updates to elasticsearch configuration, dao, and tests (#6269) 2022-11-15 18:03:11 -08:00
Tamas Nemeth
94d2672ebe
fix(snowflake): Fixing top query trimming in snowflake (#6447) 2022-11-15 15:27:55 -08:00
Pavel Klammert
9862eff95a
fix(ingestion): add output converters for ODBC unsuported datatype in… (#6134)
Co-authored-by: Pavel Klammert | LOGEX <Pavel.Klammert@logex.com>
2022-11-15 17:11:00 -05:00
Mayuri Nehate
e78b03762a
docs(ingest): move prerequisite section before the ingestion recipe example (#6341) 2022-11-15 14:24:35 -05:00
Harshal Sheth
d87dcfae75
fix(ingest): correctly compute entity change percentage (#6438)
Reported https://datahubspace.slack.com/archives/C02FKQAGRG9/p1668487704332469.
2022-11-15 14:37:58 +05:30
Harshal Sheth
f75bb470c3
fix(ingest): init dataset props if missing in transformer (#6429) 2022-11-14 15:36:03 -05:00
Mayuri Nehate
198c86a620
feat(ingest): support incremental lineage to dbt node from external platform (#6392) 2022-11-14 13:16:38 -05:00
Mayuri Nehate
443d15f346
fix(ingest): mysql - fix native data type for mysql set type (#6407) 2022-11-14 14:33:26 +01:00
Mayuri Nehate
e79b4e8c2b
feat(ingest): s3 - add status aspect for detected s3 datasets (#6402) 2022-11-13 17:29:42 -08:00
Harshal Sheth
3e907ab0d1
feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ (#6204)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-11 15:04:36 -05:00
Mayuri Nehate
6c42064332
fix(ingest): snowflake - fix missing snowflake lineage when table_pattern is set (#6410) 2022-11-11 16:47:09 +01:00
Mayuri Nehate
1b739319f0
fix(ingest/snowflake): fix TypeError when columns are returned as None in snowflake query (#6404) 2022-11-10 12:02:49 -08:00
Harshal Sheth
ae2ea524a4
fix(ingest): use profiler config settings correctly (#6354)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-09 19:03:22 -08:00
Harshal Sheth
96f782730e
feat(ingest/tableau): re-authenticate if the token expires (#6380)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-09 18:55:24 -08:00
Tamas Nemeth
ef5c7128da
fix(ingest): unity - Removing unneeded dependency for sqlalchemy from unity-catalog to fix connector setup (#6379) 2022-11-09 15:57:17 +01:00
Marcin Szymański
44cfc29af0
fix(ingest): unity - add missing date type (#6385) 2022-11-09 15:56:22 +01:00
wangsaisai
92192d2410
feat(ingest): enable container stateful ingestion (#6343) 2022-11-08 12:34:43 +01:00
Harshal Sheth
1a81d8de6a
feat(ingest): supports MCEs in domain transformer (#6364) 2022-11-05 11:41:43 -07:00
Tamas Nemeth
07a0e86470
fix(ingest): bigquery - Sending lineage aspects as patch (#6313)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-11-04 09:52:34 -07:00
Harshal Sheth
3080b9e3ee
fix(ingest): only log vars if requested (#6362) 2022-11-03 20:16:07 -07:00
Chris Collins
f24f74598b
fix(ui): Handle encoded schemaField urns on the frontend (#6321) 2022-11-03 13:43:10 -07:00
cburroughs
39c84c2f5b
feat(ingest): upgrade feast (#6186)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-11-03 11:23:19 -07:00