yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-05 16:22:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/docker/README.md

90 lines
4.4 KiB
Markdown
Raw Permalink Normal View History

docs: hosted documentation website (#2174)
2021-03-05 00:12:12 -08:00
---
title: "Deploying with Docker"
hide_title: true
---
Add documentation
2019-09-08 20:25:58 -07:00
# Docker Images
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
## Prerequisites
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
You need to install [docker](https://docs.docker.com/install/) and
[docker-compose](https://docs.docker.com/compose/install/) (if using Linux; on Windows and Mac compose is included with
Docker Desktop).
Make sure to allocate enough hardware resources for Docker engine. Tested & confirmed config: 2 CPUs, 8GB RAM, 2GB Swap
area.
Support different container runtimes aliased as docker (#13207) Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2025-05-14 13:49:43 -05:00
_If you prefer not to use Docker Desktop (which requires a license for commercial use), you can opt for free and open-source alternatives such as [Podman Desktop](https://podman-desktop.io/) or [Rancher Desktop](https://rancherdesktop.io/). To configure them, you can add the following aliases to your `~/.bashrc` file:_
```bash
# podman
alias docker=podman
alias docker-compose="podman compose"
# Rancher (or nerdctl)
alias docker=nerdctl
alias docker-compose="nerdctl compose"
```
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
## Quickstart
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
The easiest way to bring up and test DataHub is using DataHub [Docker](https://www.docker.com) images
docs(docker): fix typo in docker hub url (#12402)
2025-01-22 06:53:30 +01:00
which are continuously deployed to [Docker Hub](https://hub.docker.com/u/acryldata) with every commit to repository.
Add documentation
2019-09-08 20:25:58 -07:00
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
You can easily download and run all these images and their dependencies with our
[quick start guide](../docs/quickstart.md).
DataHub Docker Images:
docs(docker): fix typo in README.md (#7729)
2023-04-05 22:32:18 +01:00
Do not use `latest` or `debug` tags for any of the image as those are not supported and present only due to legacy reasons. Please use `head` or tags specific for versions like `v0.8.40`. For production we recommend using version specific tags not `head`.
fix(docs,build): remove base image not needed, cleanup docs (#5469)
2022-07-22 19:43:48 +05:30
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
- [acryldata/datahub-ingestion](https://hub.docker.com/r/acryldata/datahub-ingestion/)
- [acryldata/datahub-gms](https://hub.docker.com/repository/docker/acryldata/datahub-gms/)
- [acryldata/datahub-frontend-react](https://hub.docker.com/repository/docker/acryldata/datahub-frontend-react/)
- [acryldata/datahub-mae-consumer](https://hub.docker.com/repository/docker/acryldata/datahub-mae-consumer/)
- [acryldata/datahub-mce-consumer](https://hub.docker.com/repository/docker/acryldata/datahub-mce-consumer/)
- [acryldata/datahub-upgrade](https://hub.docker.com/r/acryldata/datahub-upgrade/)
- [acryldata/datahub-kafka-setup](https://hub.docker.com/r/acryldata/datahub-kafka-setup/)
- [acryldata/datahub-elasticsearch-setup](https://hub.docker.com/r/acryldata/datahub-elasticsearch-setup/)
- [acryldata/datahub-mysql-setup](https://hub.docker.com/r/acryldata/datahub-mysql-setup/)
- [acryldata/datahub-postgres-setup](https://hub.docker.com/r/acryldata/datahub-postgres-setup/)
- [acryldata/datahub-actions](https://hub.docker.com/r/acryldata/datahub-actions). Do not use `acryldata/acryl-datahub-actions` as that is deprecated and no longer used.
Add documentation
2019-09-08 20:25:58 -07:00
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
Dependencies:
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
- [Kafka, Zookeeper, and Schema Registry](kafka-setup)
- [Elasticsearch](elasticsearch-setup)
- [MySQL](mysql)
- [(Optional) Neo4j](neo4j)
Add documentation
2019-09-08 20:25:58 -07:00
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
### Ingesting demo data.
Brought up ingestion docker image.
2019-12-16 12:05:42 -08:00
docs: remove a few outdated docs (#2981)
2021-07-29 15:32:23 -07:00
If you want to test ingesting some data once DataHub is up, use the `./docker/ingestion/ingestion.sh` script or `datahub docker ingest-sample-data`. See the [quickstart guide](../docs/quickstart.md) for more details.
Add documentation
2019-09-08 20:25:58 -07:00
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
## Using Docker Images During Development
See [Using Docker Images During Development](../docs/docker/development.md).
## Building And Deploying Docker Images
Update README.md
2020-08-08 04:58:55 -07:00
We use GitHub Actions to build and continuously deploy our images. There should be no need to do this manually; a
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
successful release on Github will automatically publish the images.
### Building images
Update README.md
2020-08-08 04:58:55 -07:00
> This is **not** our recommended development flow and most developers should be following the
Update README.md
2020-09-30 06:20:14 -07:00
> [Using Docker Images During Development](../docs/docker/development.md) guide.
Update README.md
2020-08-08 04:58:55 -07:00
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
To build the full images (that we are going to publish), you need to run the following:
```
bump(docker-compose): migrate to v2 docker compose cli (#9569)
2024-01-05 09:48:59 -06:00
COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -p datahub build
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
```
This is because we're relying on builtkit for multistage builds. It does not hurt also set `DATAHUB_VERSION` to
something unique.
Deploy React via Docker
2021-02-10 16:10:08 -08:00
docs(docker): add section for adding community images (#6770)
2022-12-16 04:56:31 +05:30
### Community Built Images
docs(docker): fix typo in README.md (#7729)
2023-04-05 22:32:18 +01:00
As the open source project grows, community members would like to contribute additions to the docker images. Not all contributions to the images can be accepted because those changes are not useful for all community members, it will increase build times, add dependencies and possible security vulns. In those cases this section can be used to point to `Dockerfiles` hosted by the community which build on top of the images published by the DataHub core team along with any container registry links where the result of those images are maintained.
Reference in New Issue Copy Permalink