ecc01b9a46
* fix(security): commons-text in frontend * refactor(restli): set threads based on cpu cores feat(mce-consumers): hit local restli endpoint * testing docker build * Add retry configuration options for entity client * Kafka debugging * fix(kafka-setup): parallelize topic creation * Adjust docker build * Docker build updates * WIP * fix(lint): metadata-ingestion lint * fix(gradle-docker): fix docker frontend dep * fix(elastic): fix race condition between gms and mae for index creation * Revert "fix(elastic): fix race condition between gms and mae for index creation" This reverts commit 9629d12c3bdb3c0dab87604d409ca4c642c9c6d3. * fix(test): fix datahub frontend test for clean/test cycle * fix(test): datahub-frontend missing assets in test * fix(security): set protobuf lib datahub-upgrade & mce/mae-consumer * gitingore update * fix(docker): remove platform on docker base image, set by buildx * refactor(kafka-producer): update kafka producer tracking/logging * updates per PR feedback * Add documentation around mce standalone consumer Kafka consumer concurrency to follow thread count for restli & sql connection pool Co-authored-by: leifker <dleifker@gmail.com> Co-authored-by: Pedro Silva <pedro@acryl.io>
| title | hide_title |
|---|---|
| Deploying with Docker | true |
Docker Images
Prerequisites
You need to install docker and docker-compose (if using Linux; on Windows and Mac compose is included with Docker Desktop).
Make sure to allocate enough hardware resources for Docker engine. Tested & confirmed config: 2 CPUs, 8GB RAM, 2GB Swap area.
Quickstart
The easiest way to bring up and test DataHub is using DataHub Docker images which are continuously deployed to Docker Hub with every commit to repository.
You can easily download and run all these images and their dependencies with our quick start guide.
DataHub Docker Images:
Do not use latest or debug tags for any of the image as those are not supported and present only due to leagcy reasons. Please use head or tags specific for versions like v0.8.40. For production we recommend using version specific tags not head.
- linkedin/datahub-ingestion - This contains the Python CLI. If you are looking for docker image for every minor CLI release you can find them under acryldata/datahub-ingestion.
- linkedin/datahub-gms.
- linkedin/datahub-frontend-react
- linkedin/datahub-mae-consumer
- linkedin/datahub-mce-consumer
- acryldata/datahub-upgrade
- linkedin/datahub-kafka-setup
- linkedin/datahub-elasticsearch-setup
- acryldata/datahub-mysql-setup
- acryldata/datahub-postgres-setup
- acryldata/datahub-actions. Do not use
acryldata/acryl-datahub-actionsas that is deprecated and no longer used.
Dependencies:
Ingesting demo data.
If you want to test ingesting some data once DataHub is up, use the ./docker/ingestion/ingestion.sh script or datahub docker ingest-sample-data. See the quickstart guide for more details.
Using Docker Images During Development
See Using Docker Images During Development.
Building And Deploying Docker Images
We use GitHub Actions to build and continuously deploy our images. There should be no need to do this manually; a successful release on Github will automatically publish the images.
Building images
This is not our recommended development flow and most developers should be following the Using Docker Images During Development guide.
To build the full images (that we are going to publish), you need to run the following:
COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub build
This is because we're relying on builtkit for multistage builds. It does not hurt also set DATAHUB_VERSION to
something unique.
Community Built Images
As the open source project grows, community members would like to contribute additions to the docker images. Not all contributions to the images can be accepted because those changes are not useful for all community members, it will increase build times, add dependencies and possible security vulns. In those cases this section can be used to point to Dockerfiles hosted by the community which build on top of the images published by the DataHub core team along with any container registry links where the result of those images are maintained.