9 Commits

Author SHA1 Message Date
Dexter Lee
669160a677
feat(run): Create a describe run endpoint for fetching aspects created by the ingestion run (#4964) 2022-05-24 14:46:36 -07:00
Eddie (Xiao Di) Li
01a5b13a7a
feat(Cassandra): Add Cassandra implementation of EntityService (#3286) 2022-04-25 17:29:24 -07:00
Pedro Silva
e8f6c4cabd
feat(cli) Changes rollback behaviour to apply soft deletes by default (#4358)
* Changes rollback behaviour to apply soft deletes by default

Summary:
Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back.

Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete.
Another PR will follow to perform garbage collection on these soft deleted entities.

To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.

* Adds restli specs

* Fixes deleteAspect endpoint & adds support for nested transactions

* Enable regression test & fix docker-compose for local development

* Add generated quickstart

* Fix quickstart generation script

* Adds missing var env to docker-compose-without-neo4j

* Sets status removed=true when ingesting resources

* Adds soft deletes for ElasticSearch + soft delete flags across ingestion sub-commands

* Makes elastic search consistent

* Update tests with new behaviour

* apply review comments

* apply review comment

* Forces Elastic search to add documents with status removed false when ingesting

* Reset gradle properties to default

* Fix tests
2022-03-15 12:05:52 -07:00
stephenp-gr
fb80e6b7bc
Common ElasticsearchContainer creation (#4152) 2022-02-16 16:10:10 -08:00
Dexter Lee
999e9e5b49
fix(search): Improve search and recs performance (#3660) 2021-12-03 18:13:11 -08:00
Pedro Silva
d5eddf52ab
fix(test): Adds docker engine configuration checks before running docker-based tests for metadata-io (#3654) 2021-12-02 12:42:43 -08:00
Enrico Minack
66c862fea9
test(metadata-io): Improve speed of ElasticSearch tests (#3160) 2021-09-13 14:37:29 -07:00
Gabe Lyons
2fc71da58d
fix(restore-indices): add system metadata restoration to restore-indices (#3029) 2021-08-04 11:44:22 -07:00
Gabe Lyons
aa253f5b3b
feat(deletes): add run commands (list, show, rollback) to datahub ingest (#2960) 2021-07-29 20:04:40 -07:00