yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-08-01 05:47:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
7,922 Commits 212 Branches 123 Tags
Commit Graph

780 Commits

Author SHA1 Message Date
Harshal Sheth
c05f3970fd
feat(ingest): cli - add rewrite option for metadata file check (#5763) 2022-09-01 14:30:00 -07:00
mohdsiddique
dbcd3099d4
fix(ingestion): looker - fix dashboard and chart-usage calculations (#5791) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
 2022-08-31 16:22:11 -07:00
mohdsiddique
1f1056d7cf
feat(ingestion): looker - chart usage statistics (#5652) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-30 00:21:31 -07:00
Tamas Nemeth
4572c96d60
feat(ingestion): bigquery - Bigquery beta connector - first cut (#5663) 2022-08-30 07:33:24 +02:00
Harshal Sheth
eb87db9813
fix(ingest): proper null skip logic in serialization (#5749) 2022-08-29 16:34:58 -07:00
Ravindra Lanka
b23195d3df
Fix sqllineage parser to handle special tokens with a hyphens in the table and column names. (#5748) 2022-08-26 18:11:00 -07:00
Harshal Sheth
fa0fd7b830
fix(ingest): tweak mongodb schema inference to fix test (#5744) 2022-08-26 11:15:16 -07:00
Mugdha Hardikar
e448bb8832
feat(ingest): mysql - support multiple database in single recipe (#5684) 2022-08-26 19:47:49 +02:00
liyuhui666
08f5a44df0
feat(elasticsearch): Add nested type display (#5524) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
 2022-08-26 09:07:03 -07:00
David Haglund
b830247727
fix(superset): do not crash when display_uri is not set (#5711) 2022-08-24 23:26:02 -07:00
Harshal Sheth
d7bdfc2b86
fix(ingest): redshift-usage: check full table names with AllowDenyPattern (#5702) 
* fix(ingest): redshift-usage: check full table/schema names with AllowDenyPattern

* schema pattern does not take dot syntax

* update redshift filter test
 2022-08-24 14:33:18 -07:00
mohdsiddique
7d532b8886
fix(ingestion): ingest tables from dba_tables in oracle source (#5592) 
* inspector object wrapper for oracle inspector

* WIP

* before golden file creation

* fix(ingestion): ingest tables from dba_tables in oracle source

* code review comments

* address review comments

Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
 2022-08-24 09:48:08 -07:00
Gabe Lyons
d9b2a45c5f
feat(looker): index looker charts and dashboards by business term (#5649) 2022-08-24 08:01:56 -07:00
Ravindra Lanka
2a56d2c276
fix(ci): ingest - fix nifi integration tests (#5718) 2022-08-23 21:46:01 -07:00
Peter Szalai
150a36e236
feat(ingest): presto-on-hive - Add mode option to presto-on-hive source (#5659) 2022-08-23 23:18:27 +02:00
Aditya Radhakrishnan
4e2bfdd236
fix(ingest): emit status aspect for entities ingested from okta and azure_ad (#5700) 2022-08-22 13:28:35 -07:00
Shirshanka Das
bb788ac317
feat(ingest): file - add support for folders, large files, improve co… (#5692) 2022-08-21 14:18:22 +05:30
Ravindra Lanka
228f3b50ea
feat(ingestion): send reports of ingestion runs to datahub (#5639) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-19 09:08:17 -07:00
Kwanyoung Son
9143663f1f
fix(ingest): redash - fix redash dashboard url bug (#5500) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-16 13:39:27 -07:00
Aezo
8d6a9e08c9
feat(ingest): GE profiler Trino support (#5361) 
* feat(ingest): GE Profile/Action Trino support

* bumping GE version

* updated golden test files

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2022-08-16 09:46:52 -07:00
Harshal Sheth
680c4a8f81
chore(ingest): drop data-lake source in favor of s3 source (#5628) 2022-08-15 22:30:40 -07:00
Mugdha Hardikar
a449e8ba7d
feat(mssql): add multi database ingest support (#5516) 
* feat(mssql): add multi database ingest support

* Delete older golden file.

* Update s3.md

* fix test setup
 2022-08-15 21:37:47 -07:00
Mayuri Nehate
dc08bedd6e
feat(ingest): snowflake - add snowflake-beta connector (#5517) 2022-08-15 20:54:02 -07:00
Amanda Hernando
337087cac0
feat(ingest): glue - add stateful ingestion (#5553) 2022-08-15 20:50:45 -07:00
Aditya Radhakrishnan
b042ab1ab0
fix(ingest): fix indenting issue in azure ad connector (#5627) 2022-08-11 14:10:28 -07:00
Harshal Sheth
355c129c7c
chore(ingest): drop python 3.6 support (#5521) 2022-08-10 15:00:31 -07:00
liyuhui666
0481075705
fix(ingest): Fix ingest Clickhouse without password (#5511) 
* fix(ingest): Fix ingest Clickhouse without password

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-08-09 10:30:56 -07:00
liyuhui666
d33eecbad5
feat(ingest): clickhouse - add metadata modification time and data size (#5330) 
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-08-08 14:00:02 -07:00
Jordan Wolinsky
3a86ff3485
Fix profiling when using {table}. (#5531) 
* profiling fix for when using {table}

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-08-08 13:16:59 -07:00
Jordan Wolinsky
33339e2c89
Expose catalog_name in athena.py (#5548) 
* expose catalog_name to the sql alchemy uri that is passed into pyathena

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-08 11:40:05 -07:00
mohdsiddique
0a3721cab3
feat(ingest): power-bi - make ownership ingestion optional (#5335) 
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2022-08-08 11:26:00 -07:00
Mugdha Hardikar
b32a0723a1
feat(ingest): delta-lake - extract table history into operation aspect (#5277) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-07 11:35:09 -07:00
Shirshanka Das
da0258c6ad
fix(ingest): moving delta-lake connector to be 3.7+ only (#5584) 2022-08-07 09:32:18 -07:00
Harshal Sheth
9790f3cefa
feat(ingest): infer aspectName from aspect type in MCP (#5566) 2022-08-07 07:52:58 -07:00
Piotr Sierkin
828a711684
feat(ingest): dbt - control over emitting test_results, test_definitions, etc. (#5328) 
Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-08-06 21:42:53 -07:00
Marcin Szymański
fa42b59d9f
fix(ingestion): correct trino datatype handling (#5541) 
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-08-03 20:55:14 -07:00
mohdsiddique
f1abdc91ee
fix(dbt): fix issue of assertion error when stateful ingestion is used with dbt tests (#5540) 
* fix(dbt): fix issue of dbt stateful ingestion with tests

Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com>
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-08-03 13:22:04 -07:00
Harshal Sheth
b1ff0300e6
test(ingest): mark trino/hana tests as xfail due to flakes (#5549) 2022-08-03 10:30:36 +05:30
Alexey Kravtsov
c182fe7fca
feat(ingestion): add groups to ldap users (#5470) 2022-08-01 15:04:30 -07:00
Harshal Sheth
55cb34e5d4
test(ingest): limit GMS retries in test (#5509) 2022-07-29 15:11:31 +05:30
Aditya Radhakrishnan
c8fc246319
feat(identity): update azure and okta connectors to emit Origin aspects (#5495) 2022-07-28 16:03:57 -07:00
Harshal Sheth
64e7da8a68
fix(ingest): use temp dir for file generated during test (#5505) 2022-07-27 14:29:11 -07:00
Harshal Sheth
690443ce14
fix(ingest): cleanup unused flake8 noqa statements (#5492) 
* fix(ingest): cleanup unused flake8 noqa statements

In the future, we can discover these using `flake8-noqa`.

* add back c901
 2022-07-27 22:02:32 +05:30
Aezo
85a55ffac7
fix(ingest): dbt - add support for trino datatypes (#5379) 2022-07-26 17:02:00 -07:00
Mayuri Nehate
04de6c27b7
feat(ingest): snowflake - test_connection add support for capability report (#5472) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-07-26 09:29:57 -07:00
Mayuri Nehate
455ec182e5
feat(model, ingest): add user email in dashboard user usage counts (#5471) 2022-07-25 23:06:53 -07:00
Gabe Lyons
efc5602493
feat(ingest): dbt - updating source lineage logic (#5414) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-07-24 23:55:01 -07:00
Shirshanka Das
7ed9cd2838
feat(ingest): snowflake - basic test connection capability (#5464) 2022-07-22 09:14:37 +02:00
Shirshanka Das
14d764a26f
fix(ingest): fix serialization of report to handle nesting (#5455) 2022-07-20 18:25:07 -07:00
Aseem Bansal
acb9879eb4
feat(cli,build): remove deprecated variables GMS_HOST/_PORT (#5451) 2022-07-20 20:54:43 +05:30