Tamas Nemeth
4358d8fb01
feat(ingest): athena - set Athena location as upstream ( #4503 )
2022-03-29 07:06:48 -07:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source ( #4490 )
...
* feat(data-lake): add containers and folder level dataset support
* docs(data-lake): Update readme for data lake
* doc(data-lake): fix examples, update doc
* lint fix
* feat(s3): add s3 source, restore old data-lake source
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-03-29 11:52:57 +02:00
Aseem Bansal
6b04dff913
docs: add example of database and schema allow/deny patterns ( #4505 )
2022-03-28 13:06:40 +02:00
Aseem Bansal
a702265824
fix(snowflake): allow/deny patterns ( #4504 )
...
* fix(snowflake): allow/deny patterns
* fix lint failures
* fix linting
2022-03-28 12:48:00 +02:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently ( #4496 )
2022-03-27 18:49:26 -05:00
Aseem Bansal
ab36ac0f12
feat(snowflake): stop querying for usage data when no mix/max dates ( #4480 )
2022-03-25 11:02:38 +01:00
Aseem Bansal
7f1a1a3dcf
fix(snowflake-usage): do not ingest for stage as a dataaset ( #4483 )
2022-03-25 10:59:37 +01:00
Aseem Bansal
1a20f76225
fix(ingestion): pin Jinja2 to version < 3.1.0 ( #4489 )
2022-03-24 12:26:38 -07:00
Aseem Bansal
f770ed5fea
fix(ingestion): stop CLI build failures ( #4484 )
2022-03-24 17:03:56 +01:00
Aseem Bansal
9596e73706
doc: add caveats to snowflake doc ( #4467 )
2022-03-24 16:24:38 +01:00
Aseem Bansal
611cc2ddb5
fix(snowflake): don't recommend accountadmin role for snowflake ( #4481 )
2022-03-24 07:29:52 -07:00
Aseem Bansal
dcd4af51bb
fix: change log level to debug ( #4479 )
2022-03-24 07:27:25 -07:00
Xu Wang
d04092e634
feat(ingest): add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn ( #4469 )
...
* feat: add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn
Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-03-23 16:07:57 -07:00
Kevin Hu
a7b0275b86
feat(ingest): simplify event IDs for function invocations ( #4398 )
...
* Simplify function call events
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-03-23 12:52:29 -07:00
Sergio Gómez Villamor
9fbb521bfe
chore: acryl-data 0.6.12 ( #4474 )
2022-03-23 10:24:48 -07:00
Tamas Nemeth
5c8017789a
fix(redshift) Properly handling database alias in redshift usage and redshift lineage generation ( #4473 )
...
* Fix database-alias in redshift usage and redshift lineage generation
2022-03-23 16:01:14 +01:00
Tamas Nemeth
2013d5ddbd
feat(ingest) data-lake: Add s3 properties metadata when ingesting s3 files ( #4453 )
...
* Add s3 porperties for data lake ingestion
2022-03-22 10:31:18 -07:00
Aseem Bansal
15d5e418fb
fix(snowflake-usage): add more error handling ( #4466 )
2022-03-22 08:10:03 -07:00
Fernanda de Camargo
e290e6e07e
fix(ingest): add fix to tableau connector when table has None fields ( #4445 )
...
Co-authored-by: Ludmila Ferreira <ludmila.ferreira@elo7.com>
2022-03-22 14:59:18 +01:00
mayurinehate
885cf26828
docs(hive): update recipe with example to specify kerberos auth ( #4457 )
2022-03-22 13:38:21 +01:00
pedro-iatzky
6a6d744667
fix(ingest): bigquery - fix ingestion of external tables ( #4313 )
2022-03-22 13:35:41 +01:00
Gabe Lyons
c585f291e8
adding final transport options ( #4462 )
2022-03-22 10:03:11 +01:00
Aseem Bansal
c5f1d2c9bd
feat(ingestion): snowflake, bigquery - enhancements to log and bugfix ( #4442 )
...
feat(ingestion): add logging for snowflake, bigquery
2022-03-21 09:50:36 -07:00
Gabe Lyons
bf35d4c83e
fix tableau connector when it cannot connect to URI ( #4451 )
2022-03-18 15:46:46 -07:00
Kevin Neville
d8e6f890a9
fix: Replace old repository link with new link ( #4446 )
2022-03-18 14:12:19 -07:00
cuong-pham
12bb2e1231
getting database directly from upstream tables incase there are multiple databases in upstreamDatabases ( #4447 )
2022-03-18 14:11:07 -07:00
Tamas Nemeth
430ca10c15
Passing entity properly on deletion ( #4436 )
...
Query filter use platform property properly if paltform instance specified
2022-03-18 12:30:41 +01:00
Ravindra Lanka
60925e3e8c
Fix bug in the SchemaField type computation for AVRO logical types. ( #4433 )
2022-03-18 12:06:54 +01:00
mayurinehate
2f078c981c
feat(ingestion): tableau - support for tableau version 2021.1 and older ( #4437 )
...
fixes #4119
2022-03-17 14:07:36 -07:00
Pedro Silva
0a522e5c6a
Makes filtered search deletes include BOTH removed and non-removed ( #4440 )
2022-03-17 13:22:28 -07:00
Pedro Silva
aa593c32d8
Flexible search on soft delete ( #4405 )
...
* Adds filter logic to correct DB
* Fix build
* Adds documentation & fixes flag typo
* apply review comments
* Adds test for filtered search
* Adds warning log for redundant parameter combo
2022-03-16 16:35:04 -07:00
Aseem Bansal
d4d1635f2b
fix: don't set None default ( #4422 )
2022-03-16 14:59:58 -07:00
Tamas Nemeth
f557b2c1b3
fix(ingestion) containers: Adding platform instance to container keys ( #4279 )
2022-03-16 14:57:50 -07:00
Gabe Lyons
1ab3ad3986
feat(gql): make gql layer resistant to unresolvable relationships ( #4424 )
...
* query for custom properties on containers
* dont break gql if fine grained lineage is present
2022-03-16 14:19:10 -07:00
mayurinehate
9025bfb8d0
fix(ingest): extract redshift platform correctly from sqlalchemy uri ( #4421 )
...
* fix(ingest): extract redshift platform from sqlalchemy uri
2022-03-16 19:36:23 +01:00
Aseem Bansal
2d10d9905b
fix: change log levels to debug ( #4411 )
2022-03-15 19:32:03 -07:00
Gabe Lyons
431ba4b2a9
fix(ingestion): looker - various fixes ( #4394 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-15 15:48:34 -07:00
Pedro Silva
e8f6c4cabd
feat(cli) Changes rollback behaviour to apply soft deletes by default ( #4358 )
...
* Changes rollback behaviour to apply soft deletes by default
Summary:
Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back.
Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete.
Another PR will follow to perform garbage collection on these soft deleted entities.
To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.
* Adds restli specs
* Fixes deleteAspect endpoint & adds support for nested transactions
* Enable regression test & fix docker-compose for local development
* Add generated quickstart
* Fix quickstart generation script
* Adds missing var env to docker-compose-without-neo4j
* Sets status removed=true when ingesting resources
* Adds soft deletes for ElasticSearch + soft delete flags across ingestion sub-commands
* Makes elastic search consistent
* Update tests with new behaviour
* apply review comments
* apply review comment
* Forces Elastic search to add documents with status removed false when ingesting
* Reset gradle properties to default
* Fix tests
2022-03-15 12:05:52 -07:00
Ravindra Lanka
30ed5f2a2f
feat(ingestion): cli - Add the ability to query the latest timeseries aspect value via the get command. ( #4395 )
2022-03-14 19:03:56 -07:00
Jorgen Evens
af5c4ee4d0
fix(ingest): handle endpoints without 200 response in openapi ( #4332 )
2022-03-14 17:52:08 -07:00
Ravindra Lanka
50ef658339
fix(ingestion): Invoke SqlLineageSQLParser's implementation in a separate process ( #4391 )
2022-03-14 17:49:03 -07:00
Abhiram98
d82e1d31a4
fix(ingestion): redshift - read all tables ( #4345 )
2022-03-14 17:46:35 -07:00
WaStCo
bd3090ae86
fix(query_combiner): add try block to handle queries of type str ( #4397 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-03-14 17:45:28 -07:00
Hassan Shahid
eb9a167e0d
build(ingestion): update markupsafe pinning for Airflow compatibility ( #4388 )
2022-03-14 16:19:59 -07:00
Ravindra Lanka
5aaf187371
fix(ingestion): Fix mypy error stateful committable & restore mypy version. ( #4408 )
2022-03-14 14:15:14 -07:00
Aseem Bansal
1198123d78
fix: telemetry failure should not cause CLI failure ( #4406 )
2022-03-14 12:25:22 -07:00
Aseem Bansal
f0230b05f5
fix: add missing logo ( #4386 )
2022-03-14 09:05:50 -07:00
cccs-eric
cb9b99f0ba
fix(ingest) Azure AD: support nested groups ( #4367 ) ( #4368 )
...
LGTM - Thanks!
2022-03-14 08:59:04 -07:00
Aseem Bansal
4bcc2b3d12
feat(ingestion): improve logging, docs for bigquery, snowflake, redshift ( #4344 )
2022-03-14 08:50:29 -07:00
mayurinehate
3ea72869f3
feat(GE): add option to disable sql parsing, use default parser ( #4377 )
2022-03-10 17:36:59 -08:00
