* fix(security): commons-text in frontend
* refactor(restli): set threads based on cpu cores
feat(mce-consumers): hit local restli endpoint
* testing docker build
* Add retry configuration options for entity client
* Kafka debugging
* fix(kafka-setup): parallelize topic creation
* Adjust docker build
* Docker build updates
* WIP
* fix(lint): metadata-ingestion lint
* fix(gradle-docker): fix docker frontend dep
* fix(elastic): fix race condition between gms and mae for index creation
* Revert "fix(elastic): fix race condition between gms and mae for index creation"
This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3.
* fix(test): fix datahub frontend test for clean/test cycle
* fix(test): datahub-frontend missing assets in test
* fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer
* gitingore update
* fix(docker): remove platform on docker base image, set by buildx
* refactor(kafka-producer): update kafka producer tracking/logging
* updates per PR feedback
* Add documentation around mce standalone consumer
Kafka consumer concurrency to follow thread count for restli & sql connection pool
Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
* feat(ingest): add async option to ingest proposal endpoint
* small tweak to validate before write to K, also keep existing path for timeseries aspects
* avoid double convert
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
* fix(ownership-differ): standardise ownership change event payload
* fix(ownership-differ): update test case
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: John Joyce <john@acryl.io>
* fix(patch-entity-registry): Remove exception for entities with key aspects.
* test(patch-entity-registry): Fix failing unit test for entities with key aspect
* test(patch-entity-registry): Fix checkstyle for PatchEntityRegistryTest.java
* fix(patch-entity-registry): Remove duplicate semicolon.
* fix(patch-entity-registry): Use AspectSpec to retrieve aspect class in order to support custom entity ingestion via patch mechanism.
* fix(patch-entity-registry): Fixed incorrect merge conflict resolution in BrowsePathUtils.java
* fix(patch-entity-registry): Remove extraneous import from BrowsePathUtils.java
Co-authored-by: Parham Ghazanfari <pghazanfari53@yahoo.com>
* fix(standalone-consumers) Removes Solr from spring boot application config
* Adds standalone consumer option to datahub quickstart cli
* Rename files
* Make dockerize platform agnostic & change docker compose utility to work with M1
* Fix MAE/MCE dockerfiles for arm & make smoke tests use standalone consumers
* Fix identation
* Use master code
* Adds ARM image publishing for consumers
* Fix linter
* Fix lint
* fixing dbt platform issues
* have sibling hook use entity client over entity service
* switching search service as well
* lint
* more lint
* more specific exceptions
* Changes rollback behaviour to apply soft deletes by default
Summary:
Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back.
Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete.
Another PR will follow to perform garbage collection on these soft deleted entities.
To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.
* Adds restli specs
* Fixes deleteAspect endpoint & adds support for nested transactions
* Enable regression test & fix docker-compose for local development
* Add generated quickstart
* Fix quickstart generation script
* Adds missing var env to docker-compose-without-neo4j
* Sets status removed=true when ingesting resources
* Adds soft deletes for ElasticSearch + soft delete flags across ingestion sub-commands
* Makes elastic search consistent
* Update tests with new behaviour
* apply review comments
* apply review comment
* Forces Elastic search to add documents with status removed false when ingesting
* Reset gradle properties to default
* Fix tests
