Gabe Lyons
a9c1b7e6a3
fix(ingest): dbt - dont produce key aspects if the entity has no other aspects ( #4217 )
2022-02-22 16:02:58 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support ( #4057 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
Ravindra Lanka
6c75185445
fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. ( #4140 )
2022-02-16 19:45:07 -08:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion ( #4028 )
...
* test: test stateful ingestion for kafka
test: some more advancement
test: some improvements
refactoring
* refactor: remove some linter modifications
* tests: add unit tests for kafka state
* refactor: minor changes
* tests: improve test coverage
* fix: fix naming
* style: fix format with black
* fix: fix broken test
* revert: revert smoke tests to master
* feat: add reporting to kafka source
* tests: add smoke tests for kafka reporting
* revert: revert changes to the smoke tests
* test: add kafka integration test for stateful ingestion
* docs: update documentation on kafka source
* fix: return empty string when no platform instance
* revert: remove unwanted file
* fix: solve problem with platform instance
* chore: use console sink instead of file
* fix: disable complexity check for _extract_record
* fix: remove if condition in get_platform_instance_id
* chore: remove unneeded integration test
* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging ( #4077 )
2022-02-10 20:02:23 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] ( #4063 )
2022-02-08 14:26:44 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion ( #4051 )
2022-02-07 09:51:49 -08:00
ksrinath
4c24f386a6
feat(model): data quality model ( #3787 )
...
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-02-02 22:56:08 -08:00
Swaroop Jagadish
ded16809da
feat(ingest): add tests for platform instance ( #4047 )
2022-02-02 22:52:50 -08:00
mayurinehate
1afe8876b7
feat(ingest): nifi - handle provenance api variation for older versions ( #4022 )
2022-02-01 10:03:05 -08:00
mayurinehate
faac911df8
feat(ingest): use trino python client with sqlalchemy, supports python3.6 ( #3888 )
2022-01-31 09:22:55 -08:00
Tamas Nemeth
646dd6d5f0
fix(ingest): profiler - enable profiling for low cardinality number columns ( #3990 )
2022-01-30 17:36:18 -08:00
Swaroop Jagadish
9a9f448ebf
feat(ingest): platform-instance - changes to support platform instances in sources ( #3996 )
2022-01-27 15:31:25 -08:00
Kevin Hu
62f7d05557
feat(ingest): data-lake - support for Avro files ( #3913 )
2022-01-26 10:29:00 -08:00
iasoon
5dac30c93e
fix(ingestion): match default username for Azure OIDC and Azure ingestion source ( #3926 )
2022-01-25 18:57:18 -08:00
Kevin Hu
9e30b42638
feat(ingest): data lake - support for profiling data lake assets ( #3656 )
2022-01-17 15:01:23 -08:00
Swaroop Jagadish
7d986ec880
fix(ingest): populate system metadata for all metadata events (mcp, mcpw) ( #3900 )
2022-01-16 12:03:38 -08:00
Aditya Radhakrishnan
c3e98645a2
feat(ingest): operational stats - show last updated for sql usage sources ( #3845 )
2022-01-07 17:27:09 -08:00
Paolo Genissel
744dee5206
fix(ingest): bigquery-usage - fix remove_extras to remove all partitions ( #3842 )
2022-01-07 12:30:45 -08:00
Ravindra Lanka
9f80e5487d
fix(ingest): metabase - fix the datetime parsing issue ( #3831 )
...
fixes #3803
2022-01-05 16:34:05 -08:00
Swaroop Jagadish
480ad24282
feat(ingest): dbt - add support for dbt meta mapping ( #3832 )
2022-01-05 16:32:05 -08:00
Hyun Min Choi
e76d40b57a
fix(lkml): bump lkml version up to 1.1.2 to support sql_preamble expression ( #3757 )
2021-12-21 12:13:13 -08:00
Tamas Nemeth
599edd22ae
fix(ingest): profiling - disable expensive profilers by default ( #3759 )
2021-12-17 17:17:25 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method ( #3746 )
2021-12-16 20:07:38 -08:00
Ravindra Lanka
bd69e736ba
feat(Stateful Ingestion-2/3): Client side changes for checkpointing a source job state. ( #3763 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-12-16 20:06:33 -08:00
Sergio Gómez Villamor
c59c63e90d
feat: enables dbt metadata files to be loaded from URIs ( #3739 )
2021-12-15 09:11:39 -08:00
jawadqu
578590e795
feat(ingestion) : Add Metabase Source Connector ( #3602 )
...
Co-authored-by: Jawad Qureshi <jqureshi@petabloc.com>
2021-12-13 23:02:47 -08:00
Gabe Lyons
5d8c813684
fix(mode): support definitions in mode query ( #3721 )
...
Co-authored-by: Jawad Qureshi <jqureshi@petabloc.com>
2021-12-10 17:56:39 -08:00
Gabe Lyons
8394fc62b0
feat(mode): add mode analytics ingestion source ( #3710 )
2021-12-09 16:10:08 -08:00
mayurinehate
bd4ecbc7b9
fix(nifi): add env in nifi config, add unit tests, fix nifi doc ( #3703 )
2021-12-09 13:34:13 -08:00
mayurinehate
1d7ec8dba8
feat(ingest): add nifi source ( #3681 )
2021-12-08 14:56:31 -08:00
Swaroop Jagadish
a16c432a1b
feat(metadata-model): adding metadata model doc generation and upload… ( #3667 )
2021-12-05 12:22:17 -08:00
Harshal Sheth
1afcb661bd
feat(ingest): add --strict-warnings option ( #3665 )
2021-12-03 14:07:07 -08:00
varunbharill
a5ec05e2b2
fix(ingest): mongodb enable version check for document size filter. ( #3644 )
2021-11-30 11:33:33 -08:00
mayurinehate
637d05488b
fix(ingest): rectify platform urn in kafka connect source ( #3624 )
2021-11-26 14:38:53 -08:00
Tamas Nemeth
a36fefaa33
feat(ingest): Starburst Trino usage ( #3558 )
2021-11-18 09:56:24 -08:00
mayurinehate
69036acfca
feat(ingest): add bigquery sink connector lineage in kafka connect source ( #3590 )
2021-11-17 17:18:37 -08:00
Remi
a9754ebe83
fix(ingest): fix dbt source platform when disable_dbt_node_creation is False ( #3592 )
2021-11-17 16:49:48 -08:00
Zach Bluhm
42419ec9fa
feat(ingest): allow logs to be filtered in Bigquery Usage plugin ( #3567 )
2021-11-16 08:11:17 -08:00
Ravindra Lanka
87478a1658
fix(ingestion): redshift-usage should not append email domain to the username if already an email. ( #3569 )
2021-11-13 19:47:39 -08:00
Vincenzo Lavorini
6cd31b5bdc
feat(ingest): adding openapi ingestion ( #2706 )
2021-11-09 21:22:15 -08:00
Harshal Sheth
b0b275ee34
test(profiler): make profiling tests more comprehensive ( #3513 )
2021-11-07 18:53:53 -08:00
varunbharill
417b555fef
feat(dbt): dbt enhancements - dbt nodes, lineage, subtype, etc. ( #3519 )
2021-11-05 09:27:41 -07:00
mayurinehate
192f0d33a2
feat(ingest): kafka connect source improvements ( #3481 )
2021-11-03 15:03:05 -07:00
Gabe Lyons
cad9be6b1d
fix(ingest): fix whitespace in foreign key urns ( #3486 )
2021-10-28 23:14:42 -07:00
AndreasTA-AW
561c04bcf8
feat(dbt-ingestion): add ability to skip specific models ( #3340 )
2021-10-25 22:31:19 -07:00
mayurinehate
f5d37c0621
feat(ingest): add complex types support in hive and trino source ( #3375 )
2021-10-18 22:53:51 -07:00
varunbharill
73bd7657aa
fix(test): Fixing lookml integration test. ( #3405 )
2021-10-14 18:09:32 -07:00
Swaroop Jagadish
0cf157e991
fix(ingest): lookml view file resolution and looker spurious aspect issues ( #3397 )
2021-10-14 01:16:35 -07:00
Swaroop Jagadish
146e79b9c3
fix(ingest): fixing support for kafka key schemas when only key schemas are present ( #3379 )
2021-10-12 18:50:39 -07:00
