John Joyce
9f1c5a8f75
feat(assertions): Adding Assertions Entity & Great Expectations BETA ( #4305 )
2022-03-04 11:51:31 -08:00
Kevin Hu
02fe05eb8f
feat(ingest): data-lake - remove spark requirement if not profiling ( #4131 )
2022-02-24 23:26:06 -08:00
Edward Vaisman
6ff551cbcd
feat(ingest): lineage-file - add ability to provide lineage manually through a file ( #4116 )
2022-02-24 17:02:38 -08:00
Harshal Sheth
49a8ece02a
fix(ingestion): enable compat with avro 1.11 ( #4205 )
2022-02-22 22:13:50 -08:00
Xu Wang
aa3363bcc2
feat(ingest): lib - add better support for working with urns ( #4172 )
...
Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
2022-02-22 19:39:24 -08:00
Ravindra Lanka
7f4cb87c57
Revert "fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. ( #4140 )" ( #4218 )
...
This reverts commit 6c75185445bbb23974932ff64cb142ee6bf5b51b.
2022-02-22 16:21:40 -08:00
Ravindra Lanka
84005d3848
feat(ingest): kafka - add support for non-default schema registry subject name strategies ( #4215 )
2022-02-22 16:05:46 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support ( #4057 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
Tamas Nemeth
3d02b5bec8
feat(ingest): bigquery - ignore temporary tables from lineage and connect edges directly ( #4160 )
2022-02-20 14:23:23 -08:00
Harshal Sheth
1b60fae014
test(airflow): fix airflow version parsing ( #4142 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 18:13:01 -08:00
abiwill
8bbc66b3e6
fix(ingest): elasticsearch - http/https host config support ( #4191 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 11:42:01 -08:00
Ravindra Lanka
6c75185445
fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. ( #4140 )
2022-02-16 19:45:07 -08:00
Tamas Nemeth
b2664916e3
feat(ingest): Glue - Support for domains and containers ( #4110 )
...
* Add container and domain support for Glue.
Adding option to set aws profile for Glue.
* Adding domain doc for Glue
* Making get_workunits less complex
* Updating golden file
* Addressing pr review comments
* Remove unneded empty line
2022-02-16 08:29:14 -08:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion ( #4028 )
...
* test: test stateful ingestion for kafka
test: some more advancement
test: some improvements
refactoring
* refactor: remove some linter modifications
* tests: add unit tests for kafka state
* refactor: minor changes
* tests: improve test coverage
* fix: fix naming
* style: fix format with black
* fix: fix broken test
* revert: revert smoke tests to master
* feat: add reporting to kafka source
* tests: add smoke tests for kafka reporting
* revert: revert changes to the smoke tests
* test: add kafka integration test for stateful ingestion
* docs: update documentation on kafka source
* fix: return empty string when no platform instance
* revert: remove unwanted file
* fix: solve problem with platform instance
* chore: use console sink instead of file
* fix: disable complexity check for _extract_record
* fix: remove if condition in get_platform_instance_id
* chore: remove unneeded integration test
* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Tamas Nemeth
bfaec300b6
feat(ingest) Athena: Getting table properties for Athena datasets ( #4123 )
...
* Getting table properties for Athena datasets
* Isorting
* Fixing mypy error
* Addressing pr review comments
Adding tests
* Adding missing import
* black
* Fixing test run
* fixing flake8
* Adding athen to tox tests as well
* Not running athena tests on python < 3.7
* Adressing more pr comments
2022-02-14 13:51:45 -08:00
Harshal Sheth
ea2b092fe8
chore(ingest): remove unused groupby_unsorted utility ( #4011 )
2022-02-10 21:03:33 -08:00
Claudio Benfatto
f944a9ba05
fix(ingest): enforce correct behaviour for commit policy ( #4092 )
2022-02-08 23:21:23 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] ( #4063 )
2022-02-08 14:26:44 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion ( #4051 )
2022-02-07 09:51:49 -08:00
Ravindra Lanka
f20382f956
feat(ingest): framework - client side changes for monitoring and reporting ( #3807 )
2022-02-02 13:19:15 -08:00
Ravindra Lanka
f4209504f1
feat(ingest): support Kafka confluent external schema resolution by name or subject ( #4035 )
2022-02-02 07:44:56 -08:00
mayurinehate
1afe8876b7
feat(ingest): nifi - handle provenance api variation for older versions ( #4022 )
2022-02-01 10:03:05 -08:00
Tamas Nemeth
771c8567da
fix(ingest): snowflake - Run authentication validation if default value used ( #4024 )
2022-02-01 10:01:29 -08:00
Tamas Nemeth
68711222d4
feat(ingest): usage-stats - add ability to ignore users from top users calculation ( #3735 )
2022-02-01 00:11:23 -08:00
Michael A. Schlosser
c36662f837
feat(ingest): snowflake - support for additional auth mechanisms ( #4009 )
2022-01-30 11:47:53 -08:00
Aseem Bansal
400e0fe838
feat(ingest): kafka - support schema references ( #3862 )
2022-01-17 14:29:54 -08:00
Ravindra Lanka
1efe04f88a
feat(ingest): glue - support for nested structs ( #3895 )
2022-01-17 14:21:53 -08:00
Swaroop Jagadish
7d986ec880
fix(ingest): populate system metadata for all metadata events (mcp, mcpw) ( #3900 )
2022-01-16 12:03:38 -08:00
Ravindra Lanka
a44b48a6b8
feat(ingest): elasticsearch - add Elasticsearch Source ( #3893 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 13:10:12 -08:00
Tamas Nemeth
e95446be1c
fix(ingest): sqlparser - Not lowercasing looker source's special table name ( #3891 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 12:22:17 -08:00
Rickard Cardell
2f7e49b399
feat(ingest): datahub-rest - Make datahub-rest client more robust by configurable retries. ( #3826 ) ( #3860 )
2022-01-11 19:35:06 -08:00
Swaroop Jagadish
480ad24282
feat(ingest): dbt - add support for dbt meta mapping ( #3832 )
2022-01-05 16:32:05 -08:00
Tamas Nemeth
e5d8313788
feat(ingest): sql parsing - Replacing sqlmetadata sql parser lib with sqlineage parser lib ( #3806 )
2022-01-05 12:22:59 -08:00
Tamas Nemeth
5df5150e51
feat(ingest): Add config option to set Bigquery credential in source config ( #3786 )
2021-12-27 05:48:45 -08:00
Sergio Gómez Villamor
3fa22e1138
feat(transform): adds simple add dataset properties transform ( #3778 )
2021-12-22 08:51:38 -08:00
Tamas Nemeth
2ee1a78f4e
fix(ingestion): Fix for same schema foreign key reference ( #3769 )
2021-12-20 07:11:55 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method ( #3746 )
2021-12-16 20:07:38 -08:00
Harshal Sheth
adf9d2ead7
test(ingest): fix pytest warning for class starting with Test ( #3745 )
2021-12-14 22:44:42 -08:00
Tamas Nemeth
b9f67c5b65
feat(ingest): trim long sql queries in usage connector ( #3725 )
2021-12-13 09:16:24 -08:00
mayurinehate
bd4ecbc7b9
fix(nifi): add env in nifi config, add unit tests, fix nifi doc ( #3703 )
2021-12-09 13:34:13 -08:00
ecooklin
1a5121a5ae
feat(ingest): adds glossary terms transformer ( #3657 )
2021-12-07 21:54:15 -08:00
Adrian Garcia Badaracco
bd53f377ec
fix(tests): fix typo in test name ( #3582 )
...
* fix(tests): fix typo in test name
* find & replace function name
2021-11-28 21:09:42 -08:00
Swaroop Jagadish
6a00cfb4a1
feat(ingest): adding snowflake app name to connection string ( #3574 )
2021-11-15 13:08:07 -08:00
Vincenzo Lavorini
6cd31b5bdc
feat(ingest): adding openapi ingestion ( #2706 )
2021-11-09 21:22:15 -08:00
Noé López
cde1ce043d
feat(ingest): add transformer to add properties ( #3480 )
2021-11-09 20:03:21 -08:00
Ravindra Lanka
dcb8789f17
feat(ingestion): Support for converting AVRO schemas with logical types to MCE fields. ( #3546 )
2021-11-09 19:59:01 -08:00
Ravindra Lanka
c3f1bf0534
fix(ingestion): Make AVRO schema parsing robust to exceptions. ( #3541 )
2021-11-09 18:01:18 -08:00
Swaroop Jagadish
ec406c7928
feat(ingest): read-modify-write capabilities during ingestion, adding ownership patch transformer ( #3506 )
2021-11-03 21:39:52 -07:00
Kevin Hu
de41134a33
fix(ingestion): fix incorrect glue job names ( #3503 )
2021-11-02 22:54:47 -07:00
Harshal Sheth
dccef30daf
perf(ingest): lazy load ingestion plugins ( #3430 )
2021-10-25 14:10:24 -07:00
