Harshal Sheth
6f37ddfee2
chore(ingestion): upgrade mypy ( #4141 )
2022-02-17 00:31:20 -08:00
Ravindra Lanka
6c75185445
fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. ( #4140 )
2022-02-16 19:45:07 -08:00
Ravindra Lanka
51d72c6a29
feat(ingestion): Add support for snowflake view lineage. ( #4163 )
...
* Add support for snowflake view lineage.
* Add a config flag. Sepearate upstream & downstream view lineage computation. Update documentation.
2022-02-16 18:27:40 -08:00
Tamas Nemeth
b2664916e3
feat(ingest): Glue - Support for domains and containers ( #4110 )
...
* Add container and domain support for Glue.
Adding option to set aws profile for Glue.
* Adding domain doc for Glue
* Making get_workunits less complex
* Updating golden file
* Addressing pr review comments
* Remove unneded empty line
2022-02-16 08:29:14 -08:00
Aseem Bansal
c6c71ac8bf
fix(cli): add timeout for telemetry calls ( #4135 )
2022-02-15 14:39:46 -08:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion ( #4028 )
...
* test: test stateful ingestion for kafka
test: some more advancement
test: some improvements
refactoring
* refactor: remove some linter modifications
* tests: add unit tests for kafka state
* refactor: minor changes
* tests: improve test coverage
* fix: fix naming
* style: fix format with black
* fix: fix broken test
* revert: revert smoke tests to master
* feat: add reporting to kafka source
* tests: add smoke tests for kafka reporting
* revert: revert changes to the smoke tests
* test: add kafka integration test for stateful ingestion
* docs: update documentation on kafka source
* fix: return empty string when no platform instance
* revert: remove unwanted file
* fix: solve problem with platform instance
* chore: use console sink instead of file
* fix: disable complexity check for _extract_record
* fix: remove if condition in get_platform_instance_id
* chore: remove unneeded integration test
* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Tamas Nemeth
bfaec300b6
feat(ingest) Athena: Getting table properties for Athena datasets ( #4123 )
...
* Getting table properties for Athena datasets
* Isorting
* Fixing mypy error
* Addressing pr review comments
Adding tests
* Adding missing import
* black
* Fixing test run
* fixing flake8
* Adding athen to tox tests as well
* Not running athena tests on python < 3.7
* Adressing more pr comments
2022-02-14 13:51:45 -08:00
Tamas Nemeth
d24c52828b
fix(ingest) Athena: db filter was not applied ( #4127 )
...
* Fix for db filter on Athena
* Black formatting
* Addressing pr comments
* Remove unneeded imports
2022-02-11 13:42:41 -08:00
Harshal Sheth
ea2b092fe8
chore(ingest): remove unused groupby_unsorted utility ( #4011 )
2022-02-10 21:03:33 -08:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging ( #4077 )
2022-02-10 20:02:23 -08:00
Kevin Hu
9bdc9af7b9
fix(ingest): postgres - ignore information_schema tables by default ( #4069 )
2022-02-09 23:20:25 -08:00
Harshal Sheth
076848ff55
fix(ingest): oracle - support large view definitions ( #4027 )
2022-02-09 23:18:19 -08:00
Ravindra Lanka
2d7452d64a
feat(ingest): bigquery - enhance logging while processing audit logs ( #4101 )
2022-02-09 13:34:52 -08:00
Claudio Benfatto
f944a9ba05
fix(ingest): enforce correct behaviour for commit policy ( #4092 )
2022-02-08 23:21:23 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] ( #4063 )
2022-02-08 14:26:44 -08:00
John Joyce
f5a51f0a74
fix(ingest): rest-emitter - fix serialization helper conditional ( #4090 )
...
hot-fix for rest emission issues in 0.8.25.0 and 0.8.25.1
2022-02-08 12:17:38 -08:00
Tamas Nemeth
2de29dc623
feat(ingest) - bigquery: More verbose and faster lineage generation and option to set partition datetime for profiling ( #4079 )
2022-02-08 10:09:30 -08:00
Dexter Lee
60c17a2ba7
fix(ingest): datahub-rest - retry on POST for emitter ( #4081 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-07 19:58:16 -08:00
Gabe Lyons
ec062b6787
feat(glue): make ownership configurable in glue source ( #4078 )
2022-02-07 14:29:51 -08:00
Tamas Nemeth
622d7bfccb
fix(profile):bigquery - Check for every table if it is partitioned to not hit table quota ( #4074 )
2022-02-07 10:15:06 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion ( #4051 )
2022-02-07 09:51:49 -08:00
Danilo Peixoto
52272d6561
feat(cli): add --force option to ingest rollback subcommand ( #4032 )
2022-02-06 14:49:10 -08:00
John Joyce
d19241740e
refactor(model): refactor new Assertion models ( #4064 )
2022-02-06 14:30:40 -08:00
Aditya Radhakrishnan
61db4ed152
refactor(ingest) - remove snowflake_common dependency on aws_common ( #4054 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-04 23:24:20 -08:00
Kevin Hu
6fe062f37c
feat(ingest): log CLI invocations and completions ( #4062 )
2022-02-04 21:11:04 -08:00
Tamas Nemeth
cc32c30b2a
fix(ingest): bigquery - fix for hitting limit if there are too many partitioned tables ( #4056 )
2022-02-04 14:02:36 -08:00
ksrinath
4c24f386a6
feat(model): data quality model ( #3787 )
...
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-02-02 22:56:08 -08:00
Swaroop Jagadish
ded16809da
feat(ingest): add tests for platform instance ( #4047 )
2022-02-02 22:52:50 -08:00
Ravindra Lanka
f20382f956
feat(ingest): framework - client side changes for monitoring and reporting ( #3807 )
2022-02-02 13:19:15 -08:00
Ravindra Lanka
f4209504f1
feat(ingest): support Kafka confluent external schema resolution by name or subject ( #4035 )
2022-02-02 07:44:56 -08:00
Tamas Nemeth
928ab74f33
feat(ingest): bigquery - profile only the latest partition/shard ( #3930 )
2022-02-01 10:05:35 -08:00
mayurinehate
1afe8876b7
feat(ingest): nifi - handle provenance api variation for older versions ( #4022 )
2022-02-01 10:03:05 -08:00
Tamas Nemeth
771c8567da
fix(ingest): snowflake - Run authentication validation if default value used ( #4024 )
2022-02-01 10:01:29 -08:00
Tamas Nemeth
68711222d4
feat(ingest): usage-stats - add ability to ignore users from top users calculation ( #3735 )
2022-02-01 00:11:23 -08:00
mayurinehate
faac911df8
feat(ingest): use trino python client with sqlalchemy, supports python3.6 ( #3888 )
2022-01-31 09:22:55 -08:00
Swaroop Jagadish
e29d2cbe7e
fix(ingest): adding platform instance urn to data platform instance aspects ( #4015 )
2022-01-31 08:49:03 -08:00
Swaroop Jagadish
c27f1f9487
fix(ingest): lint fix a few files ( #4016 )
2022-01-30 23:32:29 -08:00
Tamas Nemeth
646dd6d5f0
fix(ingest): profiler - enable profiling for low cardinality number columns ( #3990 )
2022-01-30 17:36:18 -08:00
Kevin Hu
b84294804a
feat(ingest): emit flags and stats for profiling telemetry ( #3969 )
2022-01-30 17:19:00 -08:00
Aditya Radhakrishnan
b5ff558f03
feat(model) - Update FabricType enum to represent more fabrics ( #3997 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-30 18:19:03 -06:00
Michael A. Schlosser
c36662f837
feat(ingest): snowflake - support for additional auth mechanisms ( #4009 )
2022-01-30 11:47:53 -08:00
John Joyce
3c2a4fe39d
Refining docs ( #4001 )
2022-01-28 15:00:35 -08:00
Aseem Bansal
308dbf2b1a
fix(ingest): superset - fix connection inference for redshift ( #3944 )
2022-01-27 22:58:34 -08:00
RyanHolstien
f6cbeba515
fix(cli): add missing classes to typed aspect map ( #4000 )
...
Co-authored-by: Ryan Holstien <ryan@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-27 22:55:40 -08:00
Swaroop Jagadish
9a9f448ebf
feat(ingest): platform-instance - changes to support platform instances in sources ( #3996 )
2022-01-27 15:31:25 -08:00
John Joyce
9366a47f88
fix(quickstart): Assign correct mysql-setup container for M1 and remove "head" default version. ( #3987 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-26 18:15:54 -08:00
John Joyce
cf38b61a7f
feat(ingest): looker - optional external_base_url config for chart and dashboard urls ( #3985 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-26 15:06:34 -08:00
Kevin Hu
62f7d05557
feat(ingest): data-lake - support for Avro files ( #3913 )
2022-01-26 10:29:00 -08:00
Harshal Sheth
e6060c03be
fix(ingest): don't dump variables for config errors ( #3974 )
2022-01-25 22:30:18 -08:00
Harshal Sheth
ea5a66bbc5
fix(ingest): serialization - tighten conditions for restli json transformation ( #3973 )
2022-01-25 21:56:04 -08:00
