* Changes rollback behaviour to apply soft deletes by default
Summary:
Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back.
Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete.
Another PR will follow to perform garbage collection on these soft deleted entities.
To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.
* Adds restli specs
* Fixes deleteAspect endpoint & adds support for nested transactions
* Enable regression test & fix docker-compose for local development
* Add generated quickstart
* Fix quickstart generation script
* Adds missing var env to docker-compose-without-neo4j
* Sets status removed=true when ingesting resources
* Adds soft deletes for ElasticSearch + soft delete flags across ingestion sub-commands
* Makes elastic search consistent
* Update tests with new behaviour
* apply review comments
* apply review comment
* Forces Elastic search to add documents with status removed false when ingesting
* Reset gradle properties to default
* Fix tests
* Make docker files easier to use during development.
During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support.
Changes made to docker files:
- Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides.
- Remove redundant README files that provided little information.
- Rename docker/<dir> to match the service name in the docker-compose file for clarity.
- Move environment variables to .env files. We only provide dev / the default environment for quickstart.
- Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead.
- Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image).
- Added docs/docker documentation for this.
* build(docker): refactor docker build scripts
- add "build" option to docker-compose files to simplify rebuilding of images
- create "start.sh" script so it's easier to override "command" in the quickstart's docker-compose file
- use dockerize to wait for requisite services to start up
- add a dedicated Dockerfile for kafka-setup
This fixes https://github.com/linkedin/datahub/issues/1549 & https://github.com/linkedin/datahub/issues/1550
As we're now utilize GitHub Actions to build & publish docker images to docker hub under linkedin org
Also allow overriding image tags via DATAHUB_VERSION environment variable
* Convert MAE to Spring boot
* Fix after testing
* Changes after testing
* Add file appender for gms and update doc type for ESv5.6
* Review comments
* Fix Review Comments
