* fix(security): commons-text in frontend
* refactor(restli): set threads based on cpu cores
feat(mce-consumers): hit local restli endpoint
* testing docker build
* Add retry configuration options for entity client
* Kafka debugging
* fix(kafka-setup): parallelize topic creation
* Adjust docker build
* Docker build updates
* WIP
* fix(lint): metadata-ingestion lint
* fix(gradle-docker): fix docker frontend dep
* fix(elastic): fix race condition between gms and mae for index creation
* Revert "fix(elastic): fix race condition between gms and mae for index creation"
This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3.
* fix(test): fix datahub frontend test for clean/test cycle
* fix(test): datahub-frontend missing assets in test
* fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer
* gitingore update
* fix(docker): remove platform on docker base image, set by buildx
* refactor(kafka-producer): update kafka producer tracking/logging
* updates per PR feedback
* Add documentation around mce standalone consumer
Kafka consumer concurrency to follow thread count for restli & sql connection pool
Co-authored-by: leifker <dleifker@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
* refactor(elasticsearch-setup-job): create-indices.sh readability
The script contains many copy-pasting and is not easy to follow.
Add comments, extract commonly used operations into functions, unify approaches.
* fix(elasticsearch-setup-job): AWS indices creation
Fix the issue where Amazon OpenSearch (AWS ES) indices are incorrectly initialised
and the Analytics screen shows errors only.
* feat(elasticsearch-setup-job): configuration hint
mention USE_AWS_ELASTICSEARCH env value if it seems it's set the wrong way
* fix(elasticsearch-setup-job): silent curl
* fix(elasticsearch-setup-job): better USE_AWS_ELASTICSEARCH hint
* docs(elasticsearch-setup-job): index dropping explained
- more comments
- more defensive approach
- index file renamed
* fix(elasticsearch-setup-job): script fixes
* merge(elasticsearch-setup-job): merging in PR #5937
* merge(elasticsearch-setup-job): merging in PR #5963
* merge(elasticsearch-setup-job): merging in PR #5975
Co-authored-by: Pedro Silva <pedro@acryl.io>
Merges in changes from our ES7 branch, and drops support for ES5.
This is a breaking change due to the upgrade, we have a ES5 branch at the commit before this.
These tests verify that, given an index settings and mappings, data can be written to the index, and read from it with a query_all query. These are very simple sanity tests.
We can, and should, write more complex tests that specific to each index in the future.
* Make docker files easier to use during development.
During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support.
Changes made to docker files:
- Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides.
- Remove redundant README files that provided little information.
- Rename docker/<dir> to match the service name in the docker-compose file for clarity.
- Move environment variables to .env files. We only provide dev / the default environment for quickstart.
- Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead.
- Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image).
- Added docs/docker documentation for this.
