3975 Commits

Author SHA1 Message Date
Harshal Sheth
a991dc9491
fix(ingest): delta-lake - fix iteration bug (#5625) 2022-08-11 18:25:19 -07:00
Aditya Radhakrishnan
b042ab1ab0
fix(ingest): fix indenting issue in azure ad connector (#5627) 2022-08-11 14:10:28 -07:00
Harshal Sheth
355c129c7c
chore(ingest): drop python 3.6 support (#5521) 2022-08-10 15:00:31 -07:00
Harshal Sheth
44328b66e3
fix(ingest): use default telemetry ID when config is unwritable (#5614) 2022-08-10 14:58:03 -07:00
mohdsiddique
34c2f94ede
fix(transformers): domain transformer - change config, add docs (#5606)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2022-08-10 12:14:05 -07:00
Ravindra Lanka
3aa74e0431
fix(ingest): glue - fix minor logging bug. (#5605) 2022-08-09 17:45:55 -07:00
liyuhui666
0481075705
fix(ingest): Fix ingest Clickhouse without password (#5511)
* fix(ingest): Fix ingest Clickhouse without password

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-08-09 10:30:56 -07:00
Harshal Sheth
c7b3413b11
fix(ingest): simplify upgrade version stats (#5588) 2022-08-09 22:07:34 +05:30
Harshal Sheth
b46a6cd63f
feat(ingest): remove orderedset dependency (#5591) 2022-08-09 22:07:20 +05:30
liyuhui666
d33eecbad5
feat(ingest): clickhouse - add metadata modification time and data size (#5330)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-08-08 14:00:02 -07:00
Jordan Wolinsky
3a86ff3485
Fix profiling when using {table}. (#5531)
* profiling fix for when using {table}

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-08-08 13:16:59 -07:00
Jordan Wolinsky
33339e2c89
Expose catalog_name in athena.py (#5548)
* expose catalog_name to the sql alchemy uri that is passed into pyathena

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-08 11:40:05 -07:00
mohdsiddique
0a3721cab3
feat(ingest): power-bi - make ownership ingestion optional (#5335)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-08-08 11:26:00 -07:00
Mugdha Hardikar
b32a0723a1
feat(ingest): delta-lake - extract table history into operation aspect (#5277)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-07 11:35:09 -07:00
Shirshanka Das
da0258c6ad
fix(ingest): moving delta-lake connector to be 3.7+ only (#5584) 2022-08-07 09:32:18 -07:00
Mugdha Hardikar
d6621b4481
fix(ingest): sql-common - db2, snowflake bug fixes to extract table descriptions (#5526)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-07 09:23:23 -07:00
Harshal Sheth
9790f3cefa
feat(ingest): infer aspectName from aspect type in MCP (#5566) 2022-08-07 07:52:58 -07:00
Piotr Sierkin
828a711684
feat(ingest): dbt - control over emitting test_results, test_definitions, etc. (#5328)
Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-06 21:42:53 -07:00
Harshal Sheth
69124a0a99
fix(ingest): handle when current server version is unavailable (#5547) 2022-08-06 20:59:52 -07:00
Harshal Sheth
3c2a9c585b
fix(ingest): activate mypy support for ParamSpec typing annotation (#5551) 2022-08-04 17:05:52 +05:30
mohdsiddique
c619ba9500
feat(transformers): Add domain transformer for dataset (#5456)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2022-08-03 21:11:42 -07:00
Harshal Sheth
858b132444
fix(ingest): fix some typos and logging issues (#5564) 2022-08-03 21:01:45 -07:00
xiphl
16ff4aaa94
feat(ingest) Allow ingestion of Elasticsearch index template (#5444)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-08-03 20:58:07 -07:00
Marcin Szymański
fa42b59d9f
fix(ingestion): correct trino datatype handling (#5541)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-08-03 20:55:14 -07:00
mohdsiddique
f1abdc91ee
fix(dbt): fix issue of assertion error when stateful ingestion is used with dbt tests (#5540)
* fix(dbt): fix issue of dbt stateful ingestion with tests

Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com>
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-08-03 13:22:04 -07:00
Aseem Bansal
0cbcaf333d
fix(quickstart): failure on a path not being present (#5554) 2022-08-03 09:03:36 -07:00
Dan Andreescu
865074e338
feat(ingestion): superset - add display_uri to config (#5408) 2022-08-03 11:42:51 +02:00
Harshal Sheth
b1ff0300e6
test(ingest): mark trino/hana tests as xfail due to flakes (#5549) 2022-08-03 10:30:36 +05:30
Alexey Kravtsov
c182fe7fca
feat(ingestion): add groups to ldap users (#5470) 2022-08-01 15:04:30 -07:00
Harshal Sheth
4f93aaa3a7
feat(ingest): add deprecation warning for Python 3.6 (#5519) 2022-08-01 10:20:41 -07:00
Harshal Sheth
465e2c6e9a
fix(ingest): fix unbound variable bug in cli ingest list-runs (#5527) 2022-07-29 17:30:35 -07:00
Vishal Shah
f11172f94c
fix(ingest): airflow: update subdag check for compatibility with older Airflow versions (#5523)
* use getattr to default None if no subdag

* add None check

* add other None check

* Apply suggestions from code review- double quotes

Co-authored-by: Harshal Sheth <hsheth2@gmail.com>

* minor tweak to fix lint

Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-07-29 13:28:20 -07:00
Harshal Sheth
55cb34e5d4
test(ingest): limit GMS retries in test (#5509) 2022-07-29 15:11:31 +05:30
Aditya Radhakrishnan
c8fc246319
feat(identity): update azure and okta connectors to emit Origin aspects (#5495) 2022-07-28 16:03:57 -07:00
Harshal Sheth
9c7d4bd465
fix(ingest): respect rest emitter timeout setting (#5508) 2022-07-28 09:42:14 +05:30
Harshal Sheth
64e7da8a68
fix(ingest): use temp dir for file generated during test (#5505) 2022-07-27 14:29:11 -07:00
Aseem Bansal
9b5afcce51
feat(cli): delete - add --only-soft-deleted option, perf improvements (#5485) 2022-07-27 11:22:24 -07:00
Harshal Sheth
690443ce14
fix(ingest): cleanup unused flake8 noqa statements (#5492)
* fix(ingest): cleanup unused flake8 noqa statements

In the future, we can discover these using `flake8-noqa`.

* add back c901
2022-07-27 22:02:32 +05:30
Aezo
85a55ffac7
fix(ingest): dbt - add support for trino datatypes (#5379) 2022-07-26 17:02:00 -07:00
Harshal Sheth
6840240355
build(ingest): make gradle build less chatty (#5491) 2022-07-26 16:37:46 -07:00
Harshal Sheth
c294e89317
refactor(ingest): cleanup importlib.import_module calls (#5490) 2022-07-26 16:34:54 -07:00
Harshal Sheth
97d508fa2e
fix(ingest): loosen confluent-kafka dep requirement (#5489) 2022-07-26 14:43:33 -07:00
Mayuri Nehate
04de6c27b7
feat(ingest): snowflake - test_connection add support for capability report (#5472)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-07-26 09:29:57 -07:00
Mayuri Nehate
455ec182e5
feat(model, ingest): add user email in dashboard user usage counts (#5471) 2022-07-25 23:06:53 -07:00
Aseem Bansal
5d71f73062
feat(cli): delete - hard delete deletes soft deleted entities (#5478) 2022-07-25 18:17:02 -07:00
Gabe Lyons
efc5602493
feat(ingest): dbt - updating source lineage logic (#5414)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-07-24 23:55:01 -07:00
Shirshanka Das
941770f3bc
feat(cli): quickstart - experimental support for backup restore (#5418) 2022-07-24 23:52:25 -07:00
Shirshanka Das
b010efa6cf
fix(ingest): kafka - revert deps back to < 1.9.0 (#5476) 2022-07-24 15:04:24 -07:00
Tamas Nemeth
cb05159474
fix(ingest): bigquery: multiproject profiling fix (#5474) 2022-07-23 18:44:46 +02:00
Sergio Gómez Villamor
0455ae00d5
doc(bigquery): groups grants by requirements (#5468) 2022-07-22 18:49:37 +05:30