3316 Commits

Author SHA1 Message Date
Shuixi Li
f147b51fc8
feat(ingest): add preset source (#10954)
Co-authored-by: MARK CHENG <hcheng@wealthsimple.com>
Co-authored-by: hwmarkcheng <94201005+hwmarkcheng@users.noreply.github.com>
2024-10-09 20:27:31 -07:00
Harshal Sheth
e5643d56c6
feat(ingest): support __from_env__ special server value (#11569) 2024-10-09 13:08:38 -07:00
Harshal Sheth
0bdd83d9d3
feat(ingest): add extra reporting for rest sink ASYNC_BATCH mode (#11562) 2024-10-09 09:18:18 -07:00
Mayuri Nehate
26bbe02e44
feat(ingest/stateful): omit irrelevant urns for deletion (#11558) 2024-10-09 08:46:58 -07:00
Harshal Sheth
576ae8a056
fix(cli): update hard delete confirmation message in delete cli (#11550) 2024-10-09 21:04:46 +05:30
Tamas Nemeth
3cb1678ecc
feat(ingest/bigquery): Add way to reference existing DataHub Tag from a bigquery label (#11544) 2024-10-09 17:01:34 +02:00
Ellie O'Neil
ef4805d852
docs(csv-enricher): Add warning for write_semantics (#11561) 2024-10-09 13:41:28 +05:30
Ellie O'Neil
732543f201
docs(datahub source): Add urn exclusions to docs (#11568) 2024-10-09 13:35:57 +05:30
Shirshanka Das
f3a348a231
sdk(platform-resource): add entity type for ease of use (#11541) 2024-10-07 20:30:14 -07:00
sid-acryl
6b961133e9
fix(ingestion/powerbi): fix for databricks lineage m-query pattern (#11462) 2024-10-07 14:42:44 -07:00
Mayuri Nehate
2c188924ce
feat(ingest/bigquery): casing for tmp table check for queries v2 (#11534) 2024-10-07 14:27:13 -07:00
Aseem Bansal
ad24cf1aac
fix(ingest): add typeUrn in glossary sync source (#11545) 2024-10-07 19:29:21 +05:30
Harshal Sheth
849e2c1db1
fix(ingest): fix UnboundLocalError in dataproduct transformer (#11528) 2024-10-07 09:40:03 +02:00
Mayuri Nehate
364496c3d7
fix(ingest/sigma): handle members api paginated response (#11535) 2024-10-04 11:32:57 -07:00
david-leifker
04349cb9cd
feat(bootstrap): bootstrap template mcps (#11518) 2024-10-04 11:57:42 -05:00
skrydal
134ad21afe
fix(ingestion/nifi): Fix for incremental lineage ingestion for nifi (#11517) 2024-10-04 17:29:06 +05:30
sagar-salvi-apptware
107cd6484a
fix(ingest/looker) : Handle DeserializeError to improve error reporting. (#11457)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-10-03 14:44:21 -07:00
Bumyu
44645b9650
fix(ingest/elasticsearch): detect sub-properties in 'nested' type mapping (#11338)
Co-authored-by: Lawrence De Spiegeleire <lawrence.despiegeleire@pauwelconsulting.com>
2024-10-01 20:59:37 -07:00
skrydal
e1514d5e8e
fix(ingestion/nifi): Improve nifi lineage extraction performance (#11490) 2024-10-01 21:51:00 +02:00
sagar-salvi-apptware
660fbf8e57
fix(ingestion/transformer): Add container support for ownership and domains (#11375) 2024-10-01 11:39:07 -07:00
sid-acryl
67d7116055
fix(ingest/lookml): missing lineage for looker template -- if prod (#11426) 2024-10-01 11:26:00 -07:00
Martin Thøgersen
0187fc6c0f
feat(ingest/elasticsearch): add api_key auth (#11475) 2024-09-29 15:26:40 -07:00
Mayuri Nehate
efcc918cfe
fix(ingest/bigquery): optimise queries v2 session processing (#11492)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-29 15:26:11 -07:00
Harshal Sheth
07034caf09
feat(ingest): support DATAHUB_INCLUDE_ENV_IN_CONTAINER_PROPERTIES (#11476) 2024-09-27 10:24:22 -07:00
Jonny Dixon
a8e6a06ea4
docs(ingest/dbt): update run result paths examples (#11138) 2024-09-27 10:11:06 -07:00
haeniya
99bfcefb72
feat(ingestion/tableau): introduce project_path_pattern (#10855)
Co-authored-by: Yanik Häni <Yanik.Haeni1@swisscom.com>
2024-09-27 09:46:39 -07:00
Mayuri Nehate
1a73c664f0
feat(ingest/databricks): add usage perf report (#11480) 2024-09-27 09:46:22 -07:00
david-leifker
4296373c5f
config(rest-api): enable authentication and api authorization by default (#11484) 2024-09-27 11:31:25 -05:00
Mayuri Nehate
6a58493011
fix(ingest/bq): do not query PARTITIONS for biglake tables (#11463) 2024-09-27 16:46:37 +05:30
sagar-salvi-apptware
68fb97f7f1
fix(ingest/openapi): update recipe for DataHub OpenAPI with url_complement and bearer token (#10980) 2024-09-25 16:54:13 -07:00
Harshal Sheth
f8535c00c5
fix(ingest): include platform instance in looker usage urns (#11469) 2024-09-25 15:39:11 -07:00
Harshal Sheth
d754650875
feat(ingest/dbt): produce multiple assertions for multi-table dbt tests (#11451) 2024-09-25 13:10:11 -07:00
Harshal Sheth
ba9bfa0613
feat(ingest/dbt): speed up dbt CLL with node_name_patterns (#11450) 2024-09-25 13:09:05 -07:00
Mayuri Nehate
506d7a63bb
fix(ingest): do not cache temporary tables schema resolvers (#11432) 2024-09-24 09:30:39 -07:00
sid-acryl
59b21ac24f
fix(ingestion/looker): skip personal folder independent looks (#11415) 2024-09-24 20:31:06 +05:30
Harshal Sheth
32c870244e
fix(ingest/dbt): allow custom owner types for dbt meta (#11460) 2024-09-23 19:57:30 -07:00
sagar-salvi-apptware
d696dbef10
fix(ingest/mode): add connection timeouts to avoid RemoteDisconnected errors (#11245) 2024-09-23 12:28:30 -07:00
Andrew Sikowitz
3c1dcf99b0
fix(ingest/sqlglot): Make detach_ctes more robust (#11449) 2024-09-23 11:42:55 -07:00
sid-acryl
9fb2df11f3
fix(ingest): sort by last modified not working in the UI (#11343) 2024-09-23 10:06:05 -07:00
Harshal Sheth
1bea570b8d
docs(ingest): add docs on pydantic compatibility (#11423) 2024-09-20 13:22:15 -07:00
Harshal Sheth
ce0e9e2fb9
feat(ingest/sink): report datahub-rest sink mode (#11422) 2024-09-20 13:21:55 -07:00
sid-acryl
9eefedfdc0
fix(ingest/dbt): fix dbt catalog version check (#11350)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-19 22:42:27 -07:00
Harshal Sheth
aec5e1b249
fix(ingest/dbt): handle null index values (#11433) 2024-09-19 16:05:44 -07:00
Harshal Sheth
7713ea56dd
feat(ingest): add get_entity_as_mcps method to client (#11425) 2024-09-19 08:22:50 -07:00
Andrew Sikowitz
3e3a05f09c
test(ingest/mcp_diff): Fallback to overwriting file on more complicated diffs (#11407)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-18 16:40:08 -07:00
Tamas Nemeth
dea2ef00c4
feat(ingest/gc): Add dry run mode to gc recipe (#11413) 2024-09-18 13:43:22 +02:00
Sergio Gómez Villamor
31edb46dbc
feat(ingestion): adds env property in ContainerProperties (#11214)
Co-authored-by: siladitya2 <siladitya2@gmail.com>
2024-09-18 14:56:52 +05:30
Tamas Nemeth
e7a3890943
feat(ingest/gc): Add dataflow and soft deleted entities cleanup (#11102) 2024-09-18 10:01:16 +02:00
Gabe Lyons
7ab212dce5
fix(docs): Fix typo in bigquery permissions error (#11401) 2024-09-17 15:07:08 -07:00
Harshal Sheth
1d83131601
fix(ingest/airflow): simplify env configuration (#11371) 2024-09-17 12:54:03 -07:00