9 Commits

Author SHA1 Message Date
Harsha Mandadi
bf36c935fa
feat(ingest/s3): support path_specs of different S3 buckets in the same recipe (#7514) 2023-03-14 21:55:57 -07:00
mohdsiddique
c31e380489
feat(apache-ranger): Apache Ranger Authorizer support in datahub-gms (#4999) 2022-08-01 09:47:08 -07:00
Shirshanka Das
bfed6b8029
fix(ci): split out expensive build steps, increase memory (#4825) 2022-05-04 22:21:56 -07:00
Pedro Silva
e8f6c4cabd
feat(cli) Changes rollback behaviour to apply soft deletes by default (#4358)
* Changes rollback behaviour to apply soft deletes by default

Summary:
Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back.

Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete.
Another PR will follow to perform garbage collection on these soft deleted entities.

To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.

* Adds restli specs

* Fixes deleteAspect endpoint & adds support for nested transactions

* Enable regression test & fix docker-compose for local development

* Add generated quickstart

* Fix quickstart generation script

* Adds missing var env to docker-compose-without-neo4j

* Sets status removed=true when ingesting resources

* Adds soft deletes for ElasticSearch + soft delete flags across ingestion sub-commands

* Makes elastic search consistent

* Update tests with new behaviour

* apply review comments

* apply review comment

* Forces Elastic search to add documents with status removed false when ingesting

* Reset gradle properties to default

* Fix tests
2022-03-15 12:05:52 -07:00
Swaroop Jagadish
d23a5bb725
feat(spark-lineage): simplified jars, config, auto publish to maven (#3924) 2022-01-20 00:48:09 -08:00
Harshal Sheth
91e5fd457a
fix(build): increase retries for dependency fetches (#2867) 2021-07-13 09:11:42 -07:00
Jyoti Wadhwani
ad6f1653e1
metadata-models 62.0.3 -> 72.0.8 (#1693) 2020-06-11 10:21:51 -07:00
Mars Lan
4a77576d31
build: pegasus plugin doesn't work well with gradle caching (#1622)
Snapshot & IDL will not be regenerated consistently when gradle caching is enabled.
2020-04-02 19:27:24 -07:00
Kerem Sahin
3765c1d234 Add gradle.properties & enable parallel Gradle build 2020-03-11 18:36:12 -07:00