yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-18 23:02:11 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,004 Commits 200 Branches 121 Tags
Commit Graph

945 Commits

Author SHA1 Message Date
Harshal Sheth
feb9d59b4d
fix(ingest): remove unused mysql golden file (#6106) 2022-10-04 20:26:51 -07:00
Harshal Sheth
3c0f63c50a
fix(ingest): hide deprecated path_spec option from config (#5944) 2022-10-04 12:14:00 -07:00
Tamas Nemeth
a6930562fd
fix(ingest): presto-on-hive - Setting display name properly (#6065) 2022-09-29 16:20:17 +02:00
Shirshanka Das
74d9fa25a6
feat(ingest): looker - add support for simple column level lineage (#6084) 2022-09-28 20:41:30 -07:00
Harshal Sheth
59a2228d13
fix(ingest): correct tableau browse paths (#6064) 2022-09-27 12:30:52 -07:00
mohdsiddique
c062bf2201
fix(ingestion): Tableau test case execution fix (#6005) 2022-09-26 18:14:42 -07:00
John Joyce
78454888b5
refactor(ingest): Browse Paths Upgrade Tableau (#6008) 2022-09-23 13:02:34 -07:00
Ravindra Lanka
b8941ab190
feat(ingestion): Add fail-safe stale entity removal via configurable 'fail_safe_threshold' param. (#6027) 2022-09-22 16:09:22 -07:00
mohdsiddique
44a67b52cf
feat(ingestion): Tableau dashboard & chart usage stats (#5999) 
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2022-09-20 17:41:56 -07:00
Mayuri Nehate
b195b6c123
fix(ingest): encode reserved characters when creating dataset urn (#5977) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2022-09-20 16:59:02 -07:00
Tamas Nemeth
46494bc0ad
feat(ingest): presto-on-hive - Description and hive view support (#5993) 
* Presto on Hive Improvements like:
 - Capturing table/column descriptions
 - Adding Hive view support
 - Various bug fixes
 2022-09-21 00:40:10 +02:00
Aezo
6c865d00c4
feat(ingest): stateful, catalog and pascalcase con (#5890) 2022-09-20 14:54:22 +02:00
Harshal Sheth
8ffa7054e5
feat(ingest): dbt column-level meta mappings + add_terms operation (#5970) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-09-19 15:26:12 -07:00
John Joyce
df3633e7f3
refactor(looker): Migrating to new browse path format (#5688) 2022-09-19 14:43:35 -07:00
Harshal Sheth
c8cd7d5040
refactor(ingest): use typed objects in dbt ingestion (#5929) 2022-09-16 15:07:59 -07:00
Mayuri Nehate
a14617b6a4
fix(ingest): continue validation of s3 path_specs even if platform is set (#5951) 2022-09-16 12:03:57 -07:00
Mayuri Nehate
2558129391
refactor(snowflake): move snowflake-beta to certified snowflake source (#5923) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-09-15 09:53:54 -07:00
Mugdha Hardikar
93681d7a71
feat(ingest): delta-lake - support delta tables from minio (#5758) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-09-15 09:51:14 -07:00
Harshal Sheth
9384a52b23
feat(ingest): skip ssh known_hosts verification for git clone (#5945) 2022-09-14 16:43:21 -07:00
Ravindra Lanka
ee68f09624
feat(ingestion): Refactor standard state-handling tasks into a common handler that are common across all stateful ingestion sources. (#5766) 2022-09-14 09:30:42 -07:00
Shirshanka Das
15a33fadc8
feat(ingest): lookml - support for git checkout (#5924) 2022-09-13 16:40:59 -07:00
Mayuri Nehate
aedf1522fb
feat(ingest): snowflake-beta - minor changes, tests (#5910) 2022-09-12 10:42:52 -07:00
Harshal Sheth
3cd8ea8316
test(ingest): use pytest parameterization for dbt integration tests (#5879) 2022-09-10 18:35:58 -07:00
Harshal Sheth
954397a078
fix(ingest): remove dbt disable_dbt_node_creation and load_schema options (#5877) 2022-09-09 14:07:55 -07:00
Marcin Szymański
57996049fb
fix(ingest): sql - add missing trino types (#5870) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-09-09 14:07:00 -07:00
Harshal Sheth
2115d5bf1d
fix(ingest): remove dbt delete_tests_as_datasets option (#5865) 
This option was added for backwards compat and can be dropped now.
 2022-09-08 11:22:25 -07:00
Mayuri Nehate
a8c1397e5e
feat(ingest): tableau - improve lineage, workbooks query, fix pagination (#5756) 2022-09-05 23:19:22 -07:00
mohdsiddique
dbcd3099d4
fix(ingestion): looker - fix dashboard and chart-usage calculations (#5791) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
 2022-08-31 16:22:11 -07:00
mohdsiddique
1f1056d7cf
feat(ingestion): looker - chart usage statistics (#5652) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-30 00:21:31 -07:00
Harshal Sheth
fa0fd7b830
fix(ingest): tweak mongodb schema inference to fix test (#5744) 2022-08-26 11:15:16 -07:00
Mugdha Hardikar
e448bb8832
feat(ingest): mysql - support multiple database in single recipe (#5684) 2022-08-26 19:47:49 +02:00
Harshal Sheth
d7bdfc2b86
fix(ingest): redshift-usage: check full table names with AllowDenyPattern (#5702) 
* fix(ingest): redshift-usage: check full table/schema names with AllowDenyPattern

* schema pattern does not take dot syntax

* update redshift filter test
 2022-08-24 14:33:18 -07:00
mohdsiddique
7d532b8886
fix(ingestion): ingest tables from dba_tables in oracle source (#5592) 
* inspector object wrapper for oracle inspector

* WIP

* before golden file creation

* fix(ingestion): ingest tables from dba_tables in oracle source

* code review comments

* address review comments

Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
 2022-08-24 09:48:08 -07:00
Gabe Lyons
d9b2a45c5f
feat(looker): index looker charts and dashboards by business term (#5649) 2022-08-24 08:01:56 -07:00
Ravindra Lanka
2a56d2c276
fix(ci): ingest - fix nifi integration tests (#5718) 2022-08-23 21:46:01 -07:00
Peter Szalai
150a36e236
feat(ingest): presto-on-hive - Add mode option to presto-on-hive source (#5659) 2022-08-23 23:18:27 +02:00
Aditya Radhakrishnan
4e2bfdd236
fix(ingest): emit status aspect for entities ingested from okta and azure_ad (#5700) 2022-08-22 13:28:35 -07:00
Ravindra Lanka
228f3b50ea
feat(ingestion): send reports of ingestion runs to datahub (#5639) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-19 09:08:17 -07:00
Aezo
8d6a9e08c9
feat(ingest): GE profiler Trino support (#5361) 
* feat(ingest): GE Profile/Action Trino support

* bumping GE version

* updated golden test files

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2022-08-16 09:46:52 -07:00
Harshal Sheth
680c4a8f81
chore(ingest): drop data-lake source in favor of s3 source (#5628) 2022-08-15 22:30:40 -07:00
Mugdha Hardikar
a449e8ba7d
feat(mssql): add multi database ingest support (#5516) 
* feat(mssql): add multi database ingest support

* Delete older golden file.

* Update s3.md

* fix test setup
 2022-08-15 21:37:47 -07:00
Mayuri Nehate
dc08bedd6e
feat(ingest): snowflake - add snowflake-beta connector (#5517) 2022-08-15 20:54:02 -07:00
Aditya Radhakrishnan
b042ab1ab0
fix(ingest): fix indenting issue in azure ad connector (#5627) 2022-08-11 14:10:28 -07:00
Harshal Sheth
355c129c7c
chore(ingest): drop python 3.6 support (#5521) 2022-08-10 15:00:31 -07:00
liyuhui666
d33eecbad5
feat(ingest): clickhouse - add metadata modification time and data size (#5330) 
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-08-08 14:00:02 -07:00
Jordan Wolinsky
3a86ff3485
Fix profiling when using {table}. (#5531) 
* profiling fix for when using {table}

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-08-08 13:16:59 -07:00
mohdsiddique
0a3721cab3
feat(ingest): power-bi - make ownership ingestion optional (#5335) 
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2022-08-08 11:26:00 -07:00
Mugdha Hardikar
b32a0723a1
feat(ingest): delta-lake - extract table history into operation aspect (#5277) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-07 11:35:09 -07:00
Shirshanka Das
da0258c6ad
fix(ingest): moving delta-lake connector to be 3.7+ only (#5584) 2022-08-07 09:32:18 -07:00
Piotr Sierkin
828a711684
feat(ingest): dbt - control over emitting test_results, test_definitions, etc. (#5328) 
Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-06 21:42:53 -07:00