284 Commits

Author SHA1 Message Date
Harshal Sheth
ea2b092fe8
chore(ingest): remove unused groupby_unsorted utility (#4011) 2022-02-10 21:03:33 -08:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging (#4077) 2022-02-10 20:02:23 -08:00
Claudio Benfatto
f944a9ba05
fix(ingest): enforce correct behaviour for commit policy (#4092) 2022-02-08 23:21:23 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] (#4063) 2022-02-08 14:26:44 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion (#4051) 2022-02-07 09:51:49 -08:00
ksrinath
4c24f386a6
feat(model): data quality model (#3787)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-02-02 22:56:08 -08:00
Swaroop Jagadish
ded16809da
feat(ingest): add tests for platform instance (#4047) 2022-02-02 22:52:50 -08:00
Ravindra Lanka
f20382f956
feat(ingest): framework - client side changes for monitoring and reporting (#3807) 2022-02-02 13:19:15 -08:00
Ravindra Lanka
f4209504f1
feat(ingest): support Kafka confluent external schema resolution by name or subject (#4035) 2022-02-02 07:44:56 -08:00
mayurinehate
1afe8876b7
feat(ingest): nifi - handle provenance api variation for older versions (#4022) 2022-02-01 10:03:05 -08:00
Tamas Nemeth
771c8567da
fix(ingest): snowflake - Run authentication validation if default value used (#4024) 2022-02-01 10:01:29 -08:00
Tamas Nemeth
68711222d4
feat(ingest): usage-stats - add ability to ignore users from top users calculation (#3735) 2022-02-01 00:11:23 -08:00
mayurinehate
faac911df8
feat(ingest): use trino python client with sqlalchemy, supports python3.6 (#3888) 2022-01-31 09:22:55 -08:00
Tamas Nemeth
646dd6d5f0
fix(ingest): profiler - enable profiling for low cardinality number columns (#3990) 2022-01-30 17:36:18 -08:00
Michael A. Schlosser
c36662f837
feat(ingest): snowflake - support for additional auth mechanisms (#4009) 2022-01-30 11:47:53 -08:00
Swaroop Jagadish
9a9f448ebf
feat(ingest): platform-instance - changes to support platform instances in sources (#3996) 2022-01-27 15:31:25 -08:00
Kevin Hu
62f7d05557
feat(ingest): data-lake - support for Avro files (#3913) 2022-01-26 10:29:00 -08:00
iasoon
5dac30c93e
fix(ingestion): match default username for Azure OIDC and Azure ingestion source (#3926) 2022-01-25 18:57:18 -08:00
Kevin Hu
9e30b42638
feat(ingest): data lake - support for profiling data lake assets (#3656) 2022-01-17 15:01:23 -08:00
Aseem Bansal
400e0fe838
feat(ingest): kafka - support schema references (#3862) 2022-01-17 14:29:54 -08:00
Ravindra Lanka
1efe04f88a
feat(ingest): glue - support for nested structs (#3895) 2022-01-17 14:21:53 -08:00
Swaroop Jagadish
7d986ec880
fix(ingest): populate system metadata for all metadata events (mcp, mcpw) (#3900) 2022-01-16 12:03:38 -08:00
Ravindra Lanka
a44b48a6b8
feat(ingest): elasticsearch - add Elasticsearch Source (#3893)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 13:10:12 -08:00
Tamas Nemeth
e95446be1c
fix(ingest): sqlparser - Not lowercasing looker source's special table name (#3891)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 12:22:17 -08:00
Kevin Hu
5f701ebb3c
fix(cli): disable telemetry in CLI tests (#3877) 2022-01-12 00:25:42 -08:00
Rickard Cardell
2f7e49b399
feat(ingest): datahub-rest - Make datahub-rest client more robust by configurable retries. (#3826) (#3860) 2022-01-11 19:35:06 -08:00
Aditya Radhakrishnan
c3e98645a2
feat(ingest): operational stats - show last updated for sql usage sources (#3845) 2022-01-07 17:27:09 -08:00
Paolo Genissel
744dee5206
fix(ingest): bigquery-usage - fix remove_extras to remove all partitions (#3842) 2022-01-07 12:30:45 -08:00
Ravindra Lanka
9f80e5487d
fix(ingest): metabase - fix the datetime parsing issue (#3831)
fixes #3803
2022-01-05 16:34:05 -08:00
Swaroop Jagadish
480ad24282
feat(ingest): dbt - add support for dbt meta mapping (#3832) 2022-01-05 16:32:05 -08:00
Tamas Nemeth
e5d8313788
feat(ingest): sql parsing - Replacing sqlmetadata sql parser lib with sqlineage parser lib (#3806) 2022-01-05 12:22:59 -08:00
Tamas Nemeth
5df5150e51
feat(ingest): Add config option to set Bigquery credential in source config (#3786) 2021-12-27 05:48:45 -08:00
Sergio Gómez Villamor
3fa22e1138
feat(transform): adds simple add dataset properties transform (#3778) 2021-12-22 08:51:38 -08:00
Hyun Min Choi
e76d40b57a
fix(lkml): bump lkml version up to 1.1.2 to support sql_preamble expression (#3757) 2021-12-21 12:13:13 -08:00
Tamas Nemeth
2ee1a78f4e
fix(ingestion): Fix for same schema foreign key reference (#3769) 2021-12-20 07:11:55 -08:00
Tamas Nemeth
599edd22ae
fix(ingest): profiling - disable expensive profilers by default (#3759) 2021-12-17 17:17:25 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method (#3746) 2021-12-16 20:07:38 -08:00
Ravindra Lanka
bd69e736ba
feat(Stateful Ingestion-2/3): Client side changes for checkpointing a source job state. (#3763)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-12-16 20:06:33 -08:00
Sergio Gómez Villamor
c59c63e90d
feat: enables dbt metadata files to be loaded from URIs (#3739) 2021-12-15 09:11:39 -08:00
Harshal Sheth
adf9d2ead7
test(ingest): fix pytest warning for class starting with Test (#3745) 2021-12-14 22:44:42 -08:00
jawadqu
578590e795
feat(ingestion) : Add Metabase Source Connector (#3602)
Co-authored-by: Jawad Qureshi <jqureshi@petabloc.com>
2021-12-13 23:02:47 -08:00
Tamas Nemeth
b9f67c5b65
feat(ingest): trim long sql queries in usage connector (#3725) 2021-12-13 09:16:24 -08:00
Gabe Lyons
5d8c813684
fix(mode): support definitions in mode query (#3721)
Co-authored-by: Jawad Qureshi <jqureshi@petabloc.com>
2021-12-10 17:56:39 -08:00
Gabe Lyons
8394fc62b0
feat(mode): add mode analytics ingestion source (#3710) 2021-12-09 16:10:08 -08:00
mayurinehate
bd4ecbc7b9
fix(nifi): add env in nifi config, add unit tests, fix nifi doc (#3703) 2021-12-09 13:34:13 -08:00
mayurinehate
1d7ec8dba8
feat(ingest): add nifi source (#3681) 2021-12-08 14:56:31 -08:00
ecooklin
1a5121a5ae
feat(ingest): adds glossary terms transformer (#3657) 2021-12-07 21:54:15 -08:00
Kevin Hu
d3081f4807
feat(ingestion): anonymous usage stats (#3668) 2021-12-07 08:57:12 -08:00
Swaroop Jagadish
a16c432a1b
feat(metadata-model): adding metadata model doc generation and upload… (#3667) 2021-12-05 12:22:17 -08:00
Harshal Sheth
1afcb661bd
feat(ingest): add --strict-warnings option (#3665) 2021-12-03 14:07:07 -08:00