Ravindra Lanka
84005d3848
feat(ingest): kafka - add support for non-default schema registry subject name strategies ( #4215 )
2022-02-22 16:05:46 -08:00
Gabe Lyons
a9c1b7e6a3
fix(ingest): dbt - dont produce key aspects if the entity has no other aspects ( #4217 )
2022-02-22 16:02:58 -08:00
Jie Qiu
c372b93804
Fix config typo in stateful ingestion README ( #4202 )
2022-02-22 15:20:53 -08:00
Swaroop Jagadish
c6cb549918
fix(ingest): superset - adding missing greenlet dep ( #4203 )
2022-02-21 09:07:44 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support ( #4057 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
Guido Turtu
3a6d1bde64
feat(ingest): airflow - capture airflow task description ( #4147 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 00:07:08 -08:00
Kevin Hu
6b5fba882e
fix(ingest): sql-sources: add mapping for postgres types ( #4179 )
2022-02-20 19:32:15 -08:00
Tamas Nemeth
a4dc4137b7
feat(ingest): sql-sources - prevent hard failure on table/view ingestion exceptions ( #4185 )
2022-02-20 14:32:59 -08:00
Tamas Nemeth
3d02b5bec8
feat(ingest): bigquery - ignore temporary tables from lineage and connect edges directly ( #4160 )
2022-02-20 14:23:23 -08:00
Kevin Hu
c3f531fbf9
feat(ingest): profiling - record bucketed profiling runtimes ( #4068 )
2022-02-20 13:39:25 -08:00
Tamas Nemeth
c7ea6f0db2
fix(ingest): bigquery - profiling works with limit and offset using temp tables ( #4161 )
2022-02-19 19:20:26 -08:00
Harshal Sheth
1b60fae014
test(airflow): fix airflow version parsing ( #4142 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 18:13:01 -08:00
Sunil Patil
be64092c89
fix(ingest): elasticsearch - pinning down elasticsearch dependency to less than 8.0.0 ( #4182 )
...
Co-authored-by: Sunil Patil <spatil@twilio.com>
2022-02-19 11:44:32 -08:00
abiwill
8bbc66b3e6
fix(ingest): elasticsearch - http/https host config support ( #4191 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 11:42:01 -08:00
Ravindra Lanka
ab633d0047
fix(ingest): snowflake - handle external S3 bucket lineage for "External Tables". ( #4192 )
2022-02-18 15:52:13 -08:00
Swaroop Jagadish
eaf7b02b2a
docs(model): auto-generated docs and hand-written docs for the metadata model ( #4189 )
2022-02-18 09:45:45 -08:00
Tamas Nemeth
8cb312c4be
fix(ci): pinning markup_safe ( #4188 )
2022-02-18 08:53:28 -08:00
Tamas Nemeth
585aad1aac
Fix for wrong containers on Athena ( #4167 )
2022-02-17 01:34:02 -08:00
Tamas Nemeth
2e7f3ae6f3
Removing sql_common dep from glue ( #4168 )
2022-02-17 01:33:29 -08:00
Harshal Sheth
75d2ec2a39
ci(ingestion): fix airflow 1 deps for tox ( #4083 )
2022-02-17 00:33:28 -08:00
Harshal Sheth
6f37ddfee2
chore(ingestion): upgrade mypy ( #4141 )
2022-02-17 00:31:20 -08:00
Ravindra Lanka
6c75185445
fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. ( #4140 )
2022-02-16 19:45:07 -08:00
cuong-pham
ede6d91534
Update the doc to including options to include Views ( #4164 )
2022-02-16 19:42:09 -08:00
Ravindra Lanka
51d72c6a29
feat(ingestion): Add support for snowflake view lineage. ( #4163 )
...
* Add support for snowflake view lineage.
* Add a config flag. Sepearate upstream & downstream view lineage computation. Update documentation.
2022-02-16 18:27:40 -08:00
Amanda Ng
d17f2bf014
feat(ui): Host platform images on datahub-web-react ( #4118 )
...
* feat(react-images): host platform images on datahub-web-react
* feat(react-images): update data_platforms logoUrl
* feat(react-images): add explanation for CopyWebpackPlugin
Co-authored-by: Amanda Ng <amanda.ng@grabtaxi.com>
2022-02-16 11:28:06 -08:00
Tamas Nemeth
b2664916e3
feat(ingest): Glue - Support for domains and containers ( #4110 )
...
* Add container and domain support for Glue.
Adding option to set aws profile for Glue.
* Adding domain doc for Glue
* Making get_workunits less complex
* Updating golden file
* Addressing pr review comments
* Remove unneded empty line
2022-02-16 08:29:14 -08:00
Aseem Bansal
d33a868f19
fix(docs): fix example of delta lake ( #4149 )
2022-02-15 14:44:37 -08:00
Aseem Bansal
c6c71ac8bf
fix(cli): add timeout for telemetry calls ( #4135 )
2022-02-15 14:39:46 -08:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion ( #4028 )
...
* test: test stateful ingestion for kafka
test: some more advancement
test: some improvements
refactoring
* refactor: remove some linter modifications
* tests: add unit tests for kafka state
* refactor: minor changes
* tests: improve test coverage
* fix: fix naming
* style: fix format with black
* fix: fix broken test
* revert: revert smoke tests to master
* feat: add reporting to kafka source
* tests: add smoke tests for kafka reporting
* revert: revert changes to the smoke tests
* test: add kafka integration test for stateful ingestion
* docs: update documentation on kafka source
* fix: return empty string when no platform instance
* revert: remove unwanted file
* fix: solve problem with platform instance
* chore: use console sink instead of file
* fix: disable complexity check for _extract_record
* fix: remove if condition in get_platform_instance_id
* chore: remove unneeded integration test
* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Tamas Nemeth
bfaec300b6
feat(ingest) Athena: Getting table properties for Athena datasets ( #4123 )
...
* Getting table properties for Athena datasets
* Isorting
* Fixing mypy error
* Addressing pr review comments
Adding tests
* Adding missing import
* black
* Fixing test run
* fixing flake8
* Adding athen to tox tests as well
* Not running athena tests on python < 3.7
* Adressing more pr comments
2022-02-14 13:51:45 -08:00
Tamas Nemeth
d24c52828b
fix(ingest) Athena: db filter was not applied ( #4127 )
...
* Fix for db filter on Athena
* Black formatting
* Addressing pr comments
* Remove unneeded imports
2022-02-11 13:42:41 -08:00
Aseem Bansal
7fcc71ed81
fix(docs): make intro to metadata ingestion easier for beginners ( #4039 )
...
* fix(docs): fix sidebar titles for clarity
* re-arrange docs to make Intro to Metadata ingestion easier for beginners
* minor changes for readability
* add heading
* docs: add note for common question
2022-02-11 09:03:01 -08:00
Swaroop Jagadish
d1a14abb53
fix(docs): fixing metadata model doc generation script and updating png ( #4120 )
2022-02-10 22:56:28 -08:00
Harshal Sheth
ea2b092fe8
chore(ingest): remove unused groupby_unsorted utility ( #4011 )
2022-02-10 21:03:33 -08:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging ( #4077 )
2022-02-10 20:02:23 -08:00
Kevin Hu
9bdc9af7b9
fix(ingest): postgres - ignore information_schema tables by default ( #4069 )
2022-02-09 23:20:25 -08:00
Harshal Sheth
076848ff55
fix(ingest): oracle - support large view definitions ( #4027 )
2022-02-09 23:18:19 -08:00
Dexter Lee
47db7b5e2c
fix(ingest): dependencies - fix for redshift-usage, mode, superset and others ( #4103 )
2022-02-09 23:12:41 -08:00
Ravindra Lanka
2d7452d64a
feat(ingest): bigquery - enhance logging while processing audit logs ( #4101 )
2022-02-09 13:34:52 -08:00
Aseem Bansal
dbcfe9e50e
docs(kafka): add example for using domains, change for clarity ( #4100 )
2022-02-09 08:56:27 -08:00
Ravindra Lanka
917720989a
fix(ingest): dependencies - Add psutil dependency for stateful ingestion reporting. ( #4099 )
2022-02-08 23:25:12 -08:00
Claudio Benfatto
f944a9ba05
fix(ingest): enforce correct behaviour for commit policy ( #4092 )
2022-02-08 23:21:23 -08:00
John Joyce
3668de85e0
Fixing domains double icon on profile ( #4093 )
2022-02-08 15:21:07 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] ( #4063 )
2022-02-08 14:26:44 -08:00
John Joyce
f5a51f0a74
fix(ingest): rest-emitter - fix serialization helper conditional ( #4090 )
...
hot-fix for rest emission issues in 0.8.25.0 and 0.8.25.1
2022-02-08 12:17:38 -08:00
Tamas Nemeth
2de29dc623
feat(ingest) - bigquery: More verbose and faster lineage generation and option to set partition datetime for profiling ( #4079 )
2022-02-08 10:09:30 -08:00
Dexter Lee
60c17a2ba7
fix(ingest): datahub-rest - retry on POST for emitter ( #4081 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-07 19:58:16 -08:00
Gabe Lyons
ec062b6787
feat(glue): make ownership configurable in glue source ( #4078 )
2022-02-07 14:29:51 -08:00
Tamas Nemeth
622d7bfccb
fix(profile):bigquery - Check for every table if it is partitioned to not hit table quota ( #4074 )
2022-02-07 10:15:06 -08:00
iasoon
782e66f5cf
docs(ingestion) glue: document required IAM permissions ( #3929 )
2022-02-07 10:10:35 -08:00
