mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-10-31 18:48:35 +00:00
6fcdf803e6
* Prepare Docs V1 structure * Point to the v1.0.0 images dir * Use the same ssh key * Use new key * Add connectors icons * Update images
300 lines
12 KiB
Markdown
300 lines
12 KiB
Markdown
---
|
||
title: Docker Deployment
|
||
slug: /deployment/docker
|
||
---
|
||
|
||
# Docker Deployment
|
||
|
||
Deploying OpenMetadata in Docker is a great start!
|
||
Before starting with the deployment make sure you follow all the below Prerequisites.
|
||
|
||
## Prerequisites
|
||
|
||
### Docker (version 20.10.0 or greater)
|
||
|
||
[Docker](https://docs.docker.com/get-started/overview/) is an open-source platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure, so you can deliver software quickly using OS-level virtualization. It helps deliver software in packages called Containers.
|
||
|
||
To check what version of Docker you have, please use the following command.
|
||
|
||
```commandline
|
||
docker --version
|
||
```
|
||
|
||
If you need to install Docker, please visit [Get Docker](https://docs.docker.com/get-docker/).
|
||
|
||
### Docker Compose (version v2.2.3 or greater)
|
||
|
||
The Docker compose package enables you to define and run multi-container Docker applications. The compose command integrates compose functions into the Docker platform, making them available from the Docker command-line interface ( CLI). The Python packages you will install in the procedure below use compose to deploy OpenMetadata.
|
||
|
||
- **MacOS X**: Docker on MacOS X ships with compose already available in the Docker CLI.
|
||
- **Linux**: To install compose on Linux systems, please visit the Docker CLI command documentation and follow the
|
||
instructions.
|
||
|
||
To verify that the docker compose command is installed and accessible on your system, run the following command.
|
||
|
||
```commandline
|
||
docker compose version
|
||
```
|
||
|
||
Upon running this command you should see output similar to the following.
|
||
|
||
```commandline
|
||
Docker Compose version v2.2.3
|
||
```
|
||
|
||
### Install Docker Compose Version 2 on Linux
|
||
|
||
Follow the instructions [here](https://docs.docker.com/compose/cli-command/#install-on-linux) to install docker compose version 2
|
||
|
||
1. Run the following command to download the current stable release of Docker Compose
|
||
|
||
```
|
||
DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
|
||
|
||
mkdir -p $DOCKER_CONFIG/cli-plugins
|
||
curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o
|
||
$DOCKER_CONFIG/cli-plugins/docker-compose
|
||
```
|
||
|
||
This command installs Compose V2 for the active user under $HOME directory. To install Docker Compose for all users
|
||
on your system, replace` ~/.docker/cli-plugins` with `/usr/local/lib/docker/cli-plugins`.
|
||
|
||
2. Apply executable permissions to the binary
|
||
```
|
||
chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
|
||
```
|
||
3. Test your installation
|
||
```
|
||
docker compose version
|
||
> Docker Compose version v2.2.3
|
||
```
|
||
|
||
## Steps for Deploying OpenMetadata using Docker
|
||
|
||
- First download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases). The latest version is at the top of the page
|
||
|
||
- Deploying with MySQL: Download `docker-compose.yml` file from the above link.
|
||
- Deploying with PostgreSQL: Download `docker-compose-postgres.yml` file from the above link.
|
||
|
||
- Create the directory for host volumes
|
||
|
||
```commandline
|
||
mkdir -p $PWD/docker-volume/db-data
|
||
```
|
||
|
||
- Run the below command to deploy the OpenMetadata
|
||
|
||
```commandline
|
||
docker compose up --build -d
|
||
```
|
||
|
||
This command will pull the docker images of Openmetadata for MySQL, OpenMetadat-Server, OpenMetadata-Ingestion and Elasticsearch.
|
||
|
||
Upon running this command you should see output similar to the following.
|
||
|
||
```commandline
|
||
+] Running 7/8
|
||
⠿ Network metadata_app_net Created 0.2s
|
||
⠿ Volume "metadata_ingestion-volume-dag-airflow" Created 0.0s
|
||
⠿ Volume "metadata_ingestion-volume-dags" Created 0.0s
|
||
⠿ Volume "metadata_ingestion-volume-tmp" Created 0.0s
|
||
⠿ Container openmetadata_elasticsearch Started 5.9s
|
||
⠿ Container openmetadata_mysql Started 38.3s
|
||
⠿ Container openmetadata_server Started 124.8s
|
||
⠿ Container openmetadata_ingestion Started 0.3s
|
||
```
|
||
|
||
You can validate that all containers are up by running with command `docker ps`.
|
||
|
||
```commandline
|
||
❯ docker ps
|
||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||
470cc8149826 openmetadata/server:0.13.2 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server
|
||
63578aacbff5 openmetadata/ingestion:0.13.2 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion
|
||
9f5ee8334f4b docker.elastic.co/elasticsearch/elasticsearch:7.10.2 "/tini -- /usr/local…" 45 seconds ago Up 44 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp openmetadata_elasticsearch
|
||
08947ab3424b openmetadata/db:0.13.2 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql
|
||
```
|
||
|
||
In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585)
|
||
|
||
## Port Mapping / Port Forwarding
|
||
|
||
### For OpenMetadata-Server
|
||
|
||
We are shipping the OpenMetadata server and UI at `8585`, and the ingestion container (Airflow) at `8080`. You can
|
||
change the port number's according to your requirement. As an example, You could
|
||
update the ports to serve OpenMetadata Server and UI at port `80`
|
||
|
||
To achieve this
|
||
|
||
- You just have to update the ports mapping of the openmetadata-server in the `docker-compose.yml` file under `openmetadata-server` docker service section.
|
||
|
||
```yaml
|
||
ports:
|
||
- "80:8585"
|
||
```
|
||
|
||
- Once the port is updated if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command
|
||
|
||
```commandline
|
||
docker compose up --build -d
|
||
```
|
||
|
||
### For Ingestion-Server
|
||
|
||
We are shipping the OpenMetadata server and UI at `8585`, and the ingestion container (Airflow) at `8080`. You can
|
||
change the port number's according to your requirement. As an example, You could
|
||
update the ports to serve Ingestion Server and UI at port `80`
|
||
|
||
To achieve this
|
||
|
||
- You just have to update the ports mapping of the openmetadata-server in the `docker-compose.yml` file under `ingestion-server` docker service section.
|
||
|
||
```yaml
|
||
ports:
|
||
- "80:8080"
|
||
```
|
||
|
||
- Also update the Airflow environment variable in openmetadata-server section
|
||
|
||
```commandline
|
||
AIRFLOW_HOST: '<AIRFLOW_HOST:-<AIRFLOW_HOST:80>'
|
||
```
|
||
|
||
- Once the port is updated if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command
|
||
|
||
```commandline
|
||
docker compose up --build -d
|
||
```
|
||
|
||
## PROD Deployment of OpenMetadata Using Docker
|
||
|
||
If you are planning on going to PROD, we recommend to validate below points:
|
||
|
||
- MySQL and OpenSearch (ElasticSearch) are available.
|
||
- OpenMetadata-Server require the minimum configuration of 2vCPU and 6Memory (GiB)
|
||
- OpenMetadata-Ingestion require the minimum configuration of 2vCPU and 8Memory (GiB)
|
||
- We also recommend to bind Docker Volumes for data persistence. Minimum disck space required would be 128 Gib. Learn how to do so [here](/deployment/docker/volumes).
|
||
|
||
### Steps for Deploying Ingestion
|
||
|
||
- Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases).
|
||
- Update the environment variables below for OpenMetadata-Ingestion Docker Compose backed systems to connect with Database.
|
||
|
||
```
|
||
# MySQL Environment Variables for ingestion service
|
||
DB_HOST: '<DB_HOST_NAME>'
|
||
DB_PORT: '<DB_PORT>'
|
||
AIRFLOW_DB: '<AIRFLOW_DATABASE>'
|
||
AIRFLOW_DB_SCHEME: '<AIRFLOW_DB_SCHEME>'
|
||
DB_USER: '<AIRFLOW_DB_USER>'
|
||
DB_PASSWORD: '<AIRFLOW_DB_PASSWORD>'
|
||
```
|
||
|
||
Once the environment variables values with the RDS are updated then provide this environment variable file as part of docker compose command.
|
||
|
||
```
|
||
docker compose --env-file ./config/.env.prod up -d openmetadata_ingestion
|
||
```
|
||
|
||
### Steps for Deploying OpenMetadata-Server
|
||
|
||
- Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases).
|
||
- Update the environment variables below for OpenMetadata-Ingestion Docker Compose backed systems to connect with Database and ElasticSearch and Ingestion.
|
||
|
||
```
|
||
# MySQL Environment Variables
|
||
DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver'
|
||
DB_SCHEME='mysql'
|
||
DB_USE_SSL='true'
|
||
DB_USER_PASSWORD='<OPENMETADATA_DB_USER_PASSWORD>'
|
||
DB_SCHEME='mysql'
|
||
DB_HOST='<DB_HOST>'
|
||
DB_USER='<OPENMETADATA__USER_NAME>'
|
||
OM_DATABASE='<OPENMETADATA_DATABASE_NAME>'
|
||
DB_PORT='<DB_PORT>'
|
||
# ElasticSearch Environment Variables
|
||
ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
|
||
ELASTICSEARCH_USER='<ELASTICSEARCH_USERNAME>'
|
||
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
|
||
ELASTICSEARCH_PORT='443'
|
||
ELASTICSEARCH_SCHEME='https'
|
||
ELASTICSEARCH_BATCH_SIZE='10'
|
||
ELASTICSEARCH_HOST='<ELASTICSEARCH_HOST_URL>'
|
||
ELASTICSEARCH_PASSWORD='<ELASTICSEARCH_PASSWORD>'
|
||
# Ingestion or Airflow Environment Variables
|
||
AIRFLOW_HOST: '<AIRFLOW_HOST_URL>'
|
||
SERVER_HOST_API_URL: '<OPENMETADATA_HOST_URL_WITH_SCHEME/api>'
|
||
```
|
||
|
||
Once the environment variables values with the RDS are updated then provide this environment variable file as part of docker compose command.
|
||
|
||
```
|
||
docker compose --env-file ./config/.env.prod up -d openmetadata_server
|
||
```
|
||
|
||
## Run OpenMetadata with AWS Services
|
||
|
||
If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway).
|
||
|
||
We support
|
||
|
||
- Amazon RDS (MySQL) engine version upto 8.0.29
|
||
- Amazon OpenSearch (ElasticSearch) engine version upto 7.10 or Amazon OpenSearch engine version upto 1.3
|
||
- Amazon RDS (PostgreSQL) engine version upto 14.2-R1
|
||
|
||
For Production Systems, we recommend Amazon RDS to be in Multiple Availibility Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availibility Zones with minimum 3 Master Nodes.
|
||
|
||
Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata Docker Compose backed systems to connect with Database and ElasticSearch.
|
||
|
||
```
|
||
# MySQL Environment Variables
|
||
DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver'
|
||
DB_SCHEME='mysql'
|
||
DB_USE_SSL='true'
|
||
DB_USER_PASSWORD='<DATABASE_USER_PASSWORD>'
|
||
DB_SCHEME='mysql'
|
||
DB_HOST='<DATABASE_HOST_NAME>'
|
||
DB_USER='<DATABASE_USER_NAME>'
|
||
OM_DATABASE='<DATABASE_NAME>'
|
||
DB_PORT='<DATABASE_PORT>'
|
||
# ElasticSearch Environment Variables
|
||
ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
|
||
ELASTICSEARCH_USER='<ELASTICSEARCH_USERNAME>'
|
||
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
|
||
ELASTICSEARCH_PORT='443'
|
||
ELASTICSEARCH_SCHEME='https'
|
||
ELASTICSEARCH_BATCH_SIZE='10'
|
||
ELASTICSEARCH_HOST='<ELASTICSEARCH_HOST_URL>'
|
||
ELASTICSEARCH_PASSWORD='<ELASTICSEARCH_PASSWORD>'
|
||
```
|
||
|
||
Replace the environment variables values with the RDS and OpenSearch Service ones and then provide this environment variable file as part of docker compose command.
|
||
|
||
```
|
||
docker compose --env-file ./config/.env.prod up -d openmetadata_server
|
||
```
|
||
|
||
# Production Deployment
|
||
|
||
If you are planning on going to PROD, we also recommend taking a look at the following
|
||
other deployment strategies:
|
||
|
||
{%inlineCalloutContainer%}
|
||
|
||
{%inlineCallout
|
||
icon="storage"
|
||
bold="Deploy on Bare Metal"
|
||
href="/deployment/bare-metal" %}
|
||
Deploy OpenMetadata directly using the binaries.
|
||
{%/inlineCallout%}
|
||
|
||
{%inlineCallout
|
||
icon="fit_screen"
|
||
bold="Deploy on Kubernetes"
|
||
href="/deployment/kubernetes" %}
|
||
Deploy and scale with Kubernetes
|
||
{%/inlineCallout%}
|
||
|
||
{%/inlineCalloutContainer%}
|