3904 Commits

Author SHA1 Message Date
Phong Vu
0eb54c6387
fix(ingest): handle groupby custom label case (#6456)
fix(ingest): handle groupby custom label case

Co-authored-by: Phong Vu <phongvt@meeyland.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-11-23 14:12:18 -08:00
Harshal Sheth
ebd685d40d
fix(ingest/airflow): respect enabled flag in airflow plugin (#6528) 2022-11-23 11:52:05 -08:00
Harshal Sheth
4db51b4c4f
feat(ingest): disable stateful_ingestion failsafe by default (#6537) 2022-11-23 11:51:56 -08:00
Tamas Nemeth
8d525d67a9
fix(ingest): kafka - properly picking doc from union type (#6472) 2022-11-23 20:42:21 +01:00
Mayuri Nehate
37e95c9f91
fix(ingest): fix trino properties and tests (#6518) 2022-11-23 01:10:07 -05:00
Bumsoo Kim
e41f42b9db
fix(ingest): fix typo in urn utilities (#6520)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-11-23 01:05:32 -05:00
Mayuri Nehate
22847a987a
feat(ingest): automated term classification for snowflake (#6376) 2022-11-23 00:43:30 -05:00
Harshal Sheth
2b9b123a85
fix(ingest): make kafka emit callback optional (#6525) 2022-11-22 16:57:14 -05:00
Mayuri Nehate
a1b3929b38
fix(ingest): fix typo in unique count profiling (#6517) 2022-11-22 14:23:01 -05:00
Mayuri Nehate
e085a9e7dc
feat(ingest): add config for ingesting delta table without files (#6403)
Closes undefined
2022-11-22 14:15:40 -05:00
Harshal Sheth
2cb1e6576d
fix(ingest): add alias for bigquery-beta (#6521) 2022-11-22 14:15:20 -05:00
Harshal Sheth
1229c1f60d
feat(airflow): support kafka sinks in the airflow plugin (#6508) 2022-11-22 10:49:00 -08:00
Harshal Sheth
3fdaf13f2c
feat(ingest/csv-enrich): handle BOM character (#6509)
This has come up a number of times e.g. https://datahubspace.slack.com/archives/C029A3M079U/p1669000226732159?thread_ts=1669000226.732159&amp;cid=C029A3M079U
2022-11-22 12:53:34 -05:00
Harshal Sheth
74cc88f2df
fix(ingest): correctly handle transformer patch semantics (#6505) 2022-11-22 09:29:57 -08:00
Harshal Sheth
10a31b1aa0
fix(ingest/snowflake): fix lineage allow/deny pattern typo (#6506) 2022-11-22 07:09:48 +01:00
Harshal Sheth
490097e532
fix(ingest): remove redundant types (#6486)
Possible since https://github.com/python/typeshed/pull/9220 was merged.

Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-21 23:08:05 +01:00
Harshal Sheth
fdb74b0524
fix(tableau): extra defensive error-handling (#6503) 2022-11-21 14:40:18 -05:00
Harshal Sheth
05a0f3e2a6
feat(ingest): dbt cloud integration (#6323) 2022-11-21 14:14:33 -05:00
Mayuri Nehate
9c1577da13
feat(ingest): add stateful ingestion support to looker and lookml source (#6443) 2022-11-21 13:44:34 -05:00
Lukasz Stefaniak
bc5e2d43af
fix(ingest/tableau): check tableName existence on datasource response (#6478)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-11-21 11:39:19 -05:00
Tamas Nemeth
250f7ce1a8
feat(ingest): presto - Adding presto source (#6459) 2022-11-18 12:02:48 +01:00
Harshal Sheth
0d474a1fc6
docs(ingest/tableau): required permissions + doc formatting (#6484) 2022-11-18 09:31:16 +01:00
Harshal Sheth
817406eadb
refactor(ingest): simplify stateful ingestion config (#6454)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-18 00:09:24 -05:00
Harshal Sheth
b7c03731c4
fix(ingest): fix athena and GE lint errors (#6482) 2022-11-17 19:07:05 -05:00
jakobhanna
3daeaccb2b
fix(docs): Ownership Type is deprecated (#6477)
The ownership type is deprecated (described in OwnershipTypeClass), thats why it should be changed from DATAOWNER to TECHNICAL_OWNER.
2022-11-17 12:20:40 -08:00
Aditya Radhakrishnan
738b6a28d0
fix(docs): update the Okta and Azure AD docs to clarify the point of ingesting users (#6465) 2022-11-17 11:09:41 -08:00
Dmytro Kulyk
496f61b608
build: remove Jinja2 dependency from superset (#6476) 2022-11-17 13:46:42 -05:00
Harshal Sheth
b73ebe69b5
revert: mssql - lowercase db name in mssql ingestion (#6481) 2022-11-17 19:42:38 +01:00
Harshal Sheth
94e86e9713
docs(ingest): trino - add example of http connection (#6461) 2022-11-17 16:00:09 +01:00
Harshal Sheth
d3174e75b7
fix(ingest): dbt - disable incremental lineage by default (#6467) 2022-11-17 09:41:43 +01:00
mohdsiddique
73bf7a81a9
feat(ingestion): powerbi - support for report (#6339)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2022-11-16 22:52:23 -08:00
Harshal Sheth
29c6c43cb3
feat(ingest): detect old or missing docker compose (#6466) 2022-11-16 22:38:00 -08:00
Harshal Sheth
8c322ede35
feat(ingest): allow specific profiler config fields to override profile_table_level_only (#6366) 2022-11-16 23:49:31 +01:00
Harshal Sheth
a3aec72c3d
fix(ingest): handle missing schema in transformer (#6445) 2022-11-16 23:37:43 +01:00
Harshal Sheth
cf6fb91f8d
fix(ingest): mssql - lowercase db name in mssql ingestion (#6448) 2022-11-16 23:34:08 +01:00
Dmytro Kulyk
ba7fc3a685
deps(jinja): loose jinja2 dependency in Superset (#6388) (#6433) 2022-11-16 14:13:14 -08:00
david-leifker
8902404e11
fix(python): Fix python dependencies for doc generation (#6460) 2022-11-16 12:29:24 -06:00
Harshal Sheth
e530faf865
fix(ingest): add cli info to ingestion reporter (#6451) 2022-11-16 12:16:23 -05:00
Harshal Sheth
1ffd24131e
chore(ingest): fix mssql lint (#6453) 2022-11-15 23:11:57 -05:00
david-leifker
33fd876421
feat(elasticsearch): Updates to elasticsearch configuration, dao, and tests (#6269) 2022-11-15 18:03:11 -08:00
Tamas Nemeth
94d2672ebe
fix(snowflake): Fixing top query trimming in snowflake (#6447) 2022-11-15 15:27:55 -08:00
Pavel Klammert
9862eff95a
fix(ingestion): add output converters for ODBC unsuported datatype in… (#6134)
Co-authored-by: Pavel Klammert | LOGEX <Pavel.Klammert@logex.com>
2022-11-15 17:11:00 -05:00
Mayuri Nehate
e78b03762a
docs(ingest): move prerequisite section before the ingestion recipe example (#6341) 2022-11-15 14:24:35 -05:00
Harshal Sheth
d87dcfae75
fix(ingest): correctly compute entity change percentage (#6438)
Reported https://datahubspace.slack.com/archives/C02FKQAGRG9/p1668487704332469.
2022-11-15 14:37:58 +05:30
Harshal Sheth
f75bb470c3
fix(ingest): init dataset props if missing in transformer (#6429) 2022-11-14 15:36:03 -05:00
Mayuri Nehate
198c86a620
feat(ingest): support incremental lineage to dbt node from external platform (#6392) 2022-11-14 13:16:38 -05:00
Mayuri Nehate
443d15f346
fix(ingest): mysql - fix native data type for mysql set type (#6407) 2022-11-14 14:33:26 +01:00
Harshal Sheth
fd518cdf58
fix(ingest): snowflake - loosen snowflake connector version requirement (#6418)
Related to https://github.com/datahub-project/datahub/pull/6204.
2022-11-14 13:44:57 +01:00
Mayuri Nehate
e79b4e8c2b
feat(ingest): s3 - add status aspect for detected s3 datasets (#6402) 2022-11-13 17:29:42 -08:00
Harshal Sheth
3e907ab0d1
feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ (#6204)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-11 15:04:36 -05:00