Gabe Lyons
c585f291e8
adding final transport options ( #4462 )
2022-03-22 10:03:11 +01:00
Aseem Bansal
c5f1d2c9bd
feat(ingestion): snowflake, bigquery - enhancements to log and bugfix ( #4442 )
...
feat(ingestion): add logging for snowflake, bigquery
2022-03-21 09:50:36 -07:00
Gabe Lyons
bf35d4c83e
fix tableau connector when it cannot connect to URI ( #4451 )
2022-03-18 15:46:46 -07:00
Kevin Neville
d8e6f890a9
fix: Replace old repository link with new link ( #4446 )
2022-03-18 14:12:19 -07:00
cuong-pham
12bb2e1231
getting database directly from upstream tables incase there are multiple databases in upstreamDatabases ( #4447 )
2022-03-18 14:11:07 -07:00
Tamas Nemeth
430ca10c15
Passing entity properly on deletion ( #4436 )
...
Query filter use platform property properly if paltform instance specified
2022-03-18 12:30:41 +01:00
Ravindra Lanka
60925e3e8c
Fix bug in the SchemaField type computation for AVRO logical types. ( #4433 )
2022-03-18 12:06:54 +01:00
mayurinehate
2f078c981c
feat(ingestion): tableau - support for tableau version 2021.1 and older ( #4437 )
...
fixes #4119
2022-03-17 14:07:36 -07:00
Pedro Silva
0a522e5c6a
Makes filtered search deletes include BOTH removed and non-removed ( #4440 )
2022-03-17 13:22:28 -07:00
Pedro Silva
aa593c32d8
Flexible search on soft delete ( #4405 )
...
* Adds filter logic to correct DB
* Fix build
* Adds documentation & fixes flag typo
* apply review comments
* Adds test for filtered search
* Adds warning log for redundant parameter combo
2022-03-16 16:35:04 -07:00
Aseem Bansal
d4d1635f2b
fix: don't set None default ( #4422 )
2022-03-16 14:59:58 -07:00
Tamas Nemeth
f557b2c1b3
fix(ingestion) containers: Adding platform instance to container keys ( #4279 )
2022-03-16 14:57:50 -07:00
Gabe Lyons
1ab3ad3986
feat(gql): make gql layer resistant to unresolvable relationships ( #4424 )
...
* query for custom properties on containers
* dont break gql if fine grained lineage is present
2022-03-16 14:19:10 -07:00
mayurinehate
9025bfb8d0
fix(ingest): extract redshift platform correctly from sqlalchemy uri ( #4421 )
...
* fix(ingest): extract redshift platform from sqlalchemy uri
2022-03-16 19:36:23 +01:00
Aseem Bansal
2d10d9905b
fix: change log levels to debug ( #4411 )
2022-03-15 19:32:03 -07:00
Gabe Lyons
431ba4b2a9
fix(ingestion): looker - various fixes ( #4394 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-15 15:48:34 -07:00
Pedro Silva
e8f6c4cabd
feat(cli) Changes rollback behaviour to apply soft deletes by default ( #4358 )
...
* Changes rollback behaviour to apply soft deletes by default
Summary:
Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back.
Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete.
Another PR will follow to perform garbage collection on these soft deleted entities.
To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.
* Adds restli specs
* Fixes deleteAspect endpoint & adds support for nested transactions
* Enable regression test & fix docker-compose for local development
* Add generated quickstart
* Fix quickstart generation script
* Adds missing var env to docker-compose-without-neo4j
* Sets status removed=true when ingesting resources
* Adds soft deletes for ElasticSearch + soft delete flags across ingestion sub-commands
* Makes elastic search consistent
* Update tests with new behaviour
* apply review comments
* apply review comment
* Forces Elastic search to add documents with status removed false when ingesting
* Reset gradle properties to default
* Fix tests
2022-03-15 12:05:52 -07:00
Ravindra Lanka
30ed5f2a2f
feat(ingestion): cli - Add the ability to query the latest timeseries aspect value via the get command. ( #4395 )
2022-03-14 19:03:56 -07:00
Jorgen Evens
af5c4ee4d0
fix(ingest): handle endpoints without 200 response in openapi ( #4332 )
2022-03-14 17:52:08 -07:00
Ravindra Lanka
50ef658339
fix(ingestion): Invoke SqlLineageSQLParser's implementation in a separate process ( #4391 )
2022-03-14 17:49:03 -07:00
Abhiram98
d82e1d31a4
fix(ingestion): redshift - read all tables ( #4345 )
2022-03-14 17:46:35 -07:00
WaStCo
bd3090ae86
fix(query_combiner): add try block to handle queries of type str ( #4397 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-03-14 17:45:28 -07:00
Hassan Shahid
eb9a167e0d
build(ingestion): update markupsafe pinning for Airflow compatibility ( #4388 )
2022-03-14 16:19:59 -07:00
Ravindra Lanka
5aaf187371
fix(ingestion): Fix mypy error stateful committable & restore mypy version. ( #4408 )
2022-03-14 14:15:14 -07:00
Aseem Bansal
1198123d78
fix: telemetry failure should not cause CLI failure ( #4406 )
2022-03-14 12:25:22 -07:00
Aseem Bansal
f0230b05f5
fix: add missing logo ( #4386 )
2022-03-14 09:05:50 -07:00
cccs-eric
cb9b99f0ba
fix(ingest) Azure AD: support nested groups ( #4367 ) ( #4368 )
...
LGTM - Thanks!
2022-03-14 08:59:04 -07:00
Aseem Bansal
4bcc2b3d12
feat(ingestion): improve logging, docs for bigquery, snowflake, redshift ( #4344 )
2022-03-14 08:50:29 -07:00
mayurinehate
3ea72869f3
feat(GE): add option to disable sql parsing, use default parser ( #4377 )
2022-03-10 17:36:59 -08:00
John Joyce
870407b8f9
fix(ui): Supporting unknown data platform type ( #4361 )
2022-03-09 20:06:02 -08:00
Ravindra Lanka
dc62feb1e5
feat(ingest): Add memory leak detection capability to the datahub cli command. ( #4363 )
2022-03-09 17:08:44 -08:00
Tamas Nemeth
799b5305c8
fix(ingest): bigquery - Fixing missing attribute error if credential was not set ( #4356 )
2022-03-09 10:27:27 -08:00
Vishal Shah
733413f58e
feat(ingest): mysql - add database_alias functionality ( #4319 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-09 09:29:58 -08:00
Tamas Nemeth
48380ada4c
fix(ingest) bigquery-usage: Adding credential support for bigquery usage ( #4111 )
2022-03-08 12:29:10 -08:00
MugdhaHardikar-GSLab
f198a92def
fix(config-parsing): add support for variable expansion for in variables in between string ( #4350 )
2022-03-08 12:24:08 -08:00
Aseem Bansal
7eec30b2ec
fix(hive): clean protocol for hive source ( #4330 )
2022-03-08 11:57:26 -08:00
mayurinehate
e19b12030e
fix(GE): fix dependencies for GE DataHubValidationAction, logic for s… ( #4347 )
2022-03-08 11:20:09 -08:00
Aseem Bansal
05f2507e16
fix(doc): remove duplicate entry for permission ( #4341 )
2022-03-08 09:32:32 -08:00
Aseem Bansal
beb51ebf59
fix(ingestion): add logging, make job more resilient to errors ( #4331 )
2022-03-07 14:32:44 -08:00
BZ
e2d05cd8eb
docs: postgres - update support for platform instance ( #4292 )
2022-03-07 13:16:39 -08:00
Salih Can
915798a5ad
fix(ingest): elasticsearch - connector should work with defaults for auth ( #4329 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-07 13:16:05 -08:00
Swaroop Jagadish
35b187a8d4
feat(ingest): transformers - add support for processing MCP-s ( #4337 )
2022-03-07 13:14:29 -08:00
Ravindra Lanka
bd6603bec5
fix(ingest): logging - insulate 'datahub' and child loggers from external modules. ( #4324 )
2022-03-07 12:00:06 -08:00
mayurinehate
92b0e1c7c7
feat(tableau): emit workbook as container entity in tableau source, some minor fixes in tableau source ( #4261 )
2022-03-04 11:52:04 -08:00
John Joyce
9f1c5a8f75
feat(assertions): Adding Assertions Entity & Great Expectations BETA ( #4305 )
2022-03-04 11:51:31 -08:00
Aseem Bansal
9a9a5c3084
fix: revert changes to OpenApi casing ( #4291 )
2022-03-04 10:35:44 -08:00
Aseem Bansal
182d4d1cd2
doc: improve doc on adding source ( #4316 )
2022-03-04 10:22:53 -08:00
Tamas Nemeth
a4c1d5d382
fix(ingest): bigquery - remove unneeded warning ( #4317 )
2022-03-04 09:11:30 -08:00
Tamas Nemeth
3158e49171
fix(ingest): Removing python 3.9.9 restriction ( #4312 )
2022-03-04 08:21:33 -08:00
Arun Vasudevan
2bf648f98a
feat (ingest): kafka-connect: adding Auth to Kafka Connect API ( #4298 )
2022-03-03 20:43:20 -08:00
