From 987e3a66fb58c7dab81b0706dc6fc01feb98171a Mon Sep 17 00:00:00 2001
From: Vj-L <102945356+Vj-L@users.noreply.github.com>
Date: Mon, 7 Nov 2022 11:00:00 +0530
Subject: [PATCH] Updated the document for docker deployment (#8527)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
* Updated the document for docker deployment
* Updated the document for docker deployment
* Updated the document for docker deployment
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
---
.../content/deployment/docker/index.md | 210 +++++++++++++++---
1 file changed, 183 insertions(+), 27 deletions(-)
diff --git a/openmetadata-docs/content/deployment/docker/index.md b/openmetadata-docs/content/deployment/docker/index.md
index 69bded1f769..d33695cbecf 100644
--- a/openmetadata-docs/content/deployment/docker/index.md
+++ b/openmetadata-docs/content/deployment/docker/index.md
@@ -4,39 +4,195 @@ slug: /deployment/docker
---
# Docker Deployment
+Deploying OpenMetadata in Docker is a great start!
+Before starting with the deployment make sure you follow all the below Prerequisites.
+## Prerequisites
+### Docker (version 20.10.0 or greater)
+[Docker](https://docs.docker.com/get-started/overview/) is an open-source platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure, so you can deliver software quickly using OS-level virtualization. It helps deliver software in packages called Containers.
-Deploying OpenMetadata in Docker is a great start! Take a look at our
-[Quickstart](/quick-start/local-deployment) guide to learn how to get OpenMetadata
-up and running locally in less than 7 minutes!
+To check what version of Docker you have, please use the following command.
+```commandline
+docker --version
+```
-If those steps are already done, you might want to bind Docker Volumes
-for data persistence. Learn how to do so [here](/deployment/docker/volumes).
+If you need to install Docker, please visit [Get Docker](https://docs.docker.com/get-docker/).
+### Docker Compose (version v2.2.3 or greater)
+The Docker compose package enables you to define and run multi-container Docker applications. The compose command integrates compose functions into the Docker platform, making them available from the Docker command-line interface ( CLI). The Python packages you will install in the procedure below use compose to deploy OpenMetadata.
-To test out your security integration, check out how to
-[Enable Security](/deployment/docker/security).
+- **MacOS X**: Docker on MacOS X ships with compose already available in the Docker CLI.
+- **Linux**: To install compose on Linux systems, please visit the Docker CLI command documentation and follow the
+ instructions.
-## Changing ports
+To verify that the docker compose command is installed and accessible on your system, run the following command.
-
+```commandline
+docker compose version
+```
-To make changes to the `docker-compose.yaml` file you will need to download it from the release page [here](https://github.com/open-metadata/OpenMetadata/releases). The latest version is at the top of the page
-
+Upon running this command you should see output similar to the following.
-This docker deployment is powered by `docker compose`, and uses the `docker-compose.yml` files shipped during
-each release [example](https://github.com/open-metadata/OpenMetadata/releases/tag/0.11.4-release).
+```commandline
+Docker Compose version v2.2.3
+```
-As with the [Docker Volumes](/deployment/docker/volumes), you might want to tune a bit the compose file to modify
-the default ports.
+### Install Docker Compose Version 2 on Linux
+Follow the instructions [here](https://docs.docker.com/compose/cli-command/#install-on-linux) to install docker compose version 2
+
+1. Run the following command to download the current stable release of Docker Compose
+ ```
+ DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
+
+ mkdir -p $DOCKER_CONFIG/cli-plugins
+ curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o
+ $DOCKER_CONFIG/cli-plugins/docker-compose
+ ```
+
+ This command installs Compose V2 for the active user under $HOME directory. To install Docker Compose for all users
+ on your system, replace` ~/.docker/cli-plugins` with `/usr/local/lib/docker/cli-plugins`.
+
+2. Apply executable permissions to the binary
+ ```
+ chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
+ ```
+3. Test your installation
+ ```
+ docker compose version
+ > Docker Compose version v2.2.3
+ ```
+
+## Steps for Deploying OpenMetadata using Docker
+
+- First download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases). The latest version is at the top of the page
+ - Deploying with MySQL: Download `docker-compose.yml` file from the above link.
+ - Deploying with PostgreSQL: Download `docker-compose-postgres.yml` file from the above link.
+
+- Run the below command to deploy the OpenMetadata
+
+```commandline
+docker compose up --build -d
+```
+This command will pull the docker images of Openmetadata for MySQL, OpenMetadat-Server, OpenMetadata-Ingestion and Elasticsearch.
+
+Upon running this command you should see output similar to the following.
+```commandline
++] Running 7/8
+ ⠿ Network metadata_app_net Created 0.2s
+ ⠿ Volume "metadata_ingestion-volume-dag-airflow" Created 0.0s
+ ⠿ Volume "metadata_ingestion-volume-dags" Created 0.0s
+ ⠿ Volume "metadata_ingestion-volume-tmp" Created 0.0s
+ ⠿ Container openmetadata_elasticsearch Started 5.9s
+ ⠿ Container openmetadata_mysql Started 38.3s
+ ⠿ Container openmetadata_server Started 124.8s
+ ⠿ Container openmetadata_ingestion Started 0.3s
+```
+
+You can validate that all containers are up by running with command `docker ps`.
+
+```commandline
+❯ docker ps
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
+470cc8149826 openmetadata/server:0.12.2 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server
+63578aacbff5 openmetadata/ingestion:0.12.2 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion
+9f5ee8334f4b docker.elastic.co/elasticsearch/elasticsearch:7.10.2 "/tini -- /usr/local…" 45 seconds ago Up 44 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp openmetadata_elasticsearch
+08947ab3424b openmetadata/db:0.12.2 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql
+```
+
+In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585)
+## Port Mapping / Port Forwarding
+
+### For OpenMetadata-Server
We are shipping the OpenMetadata server and UI at `8585`, and the ingestion container (Airflow) at `8080`. You can
-take a look at the official Docker [docs](https://docs.docker.com/compose/compose-file/#ports). As an example, You could
-update the ports to serve Airflow at `1234` with:
+change the port number's according to your requirement. As an example, You could
+update the ports to serve OpenMetadata Server and UI at port `80`
+
+To achieve this
+- You just have to update the ports mapping of the openmetadata-server in the `docker-compose.yml` file under `openmetadata-server` docker service section.
```yaml
ports:
- - "1234:8080"
+ - "80:8585"
```
+- Once the port is updated if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command
+```commandline
+docker compose up --build -d
+```
+### For Ingestion-Server
+We are shipping the OpenMetadata server and UI at `8585`, and the ingestion container (Airflow) at `8080`. You can
+change the port number's according to your requirement. As an example, You could
+update the ports to serve Ingestion Server and UI at port `80`
+To achieve this
+- You just have to update the ports mapping of the openmetadata-server in the `docker-compose.yml` file under `ingestion-server` docker service section.
+
+```yaml
+ports:
+ - "80:8080"
+```
+- Also update the Airflow environment variable in openmetadata-server section
+ ```commandline
+ AIRFLOW_HOST: ''
+ ```
+
+- Once the port is updated if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command
+```commandline
+docker compose up --build -d
+```
+## PROD Deployment of OpenMetadata Using Docker
+If you are planning on going to PROD, we recommend to validate below points:
+- MySQL and OpenSearch (ElasticSearch) are available.
+- OpenMetadata-Server require the minimum configuration of 2vCPU and 6Memory (GiB)
+- OpenMetadata-Ingestion require the minimum configuration of 2vCPU and 8Memory (GiB)
+- We also recommend to bind Docker Volumes for data persistence. Learn how to do so [here](/deployment/docker/volumes).
+### Steps for Deploying Ingestion
+- Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases).
+- Update the environment variables below for OpenMetadata-Ingestion Docker Compose backed systems to connect with Database.
+```
+# MySQL Environment Variables for ingestion service
+DB_HOST: ''
+DB_PORT: ''
+AIRFLOW_DB: ''
+AIRFLOW_DB_SCHEME: ''
+DB_USER: ''
+DB_PASSWORD: ''
+```
+Once the environment variables values with the RDS are updated then provide this environment variable file as part of docker compose command.
+
+```
+docker compose --env-file ./config/.env.prod up -d openmetadata_ingestion
+```
+### Steps for Deploying OpenMetadata-Server
+- Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases).
+- Update the environment variables below for OpenMetadata-Ingestion Docker Compose backed systems to connect with Database and ElasticSearch and Ingestion.
+```
+# MySQL Environment Variables
+DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver'
+DB_SCHEME='mysql'
+DB_USE_SSL='true'
+DB_USER_PASSWORD=''
+DB_SCHEME='mysql'
+DB_HOST=''
+DB_USER=''
+DB_DATABASE=''
+DB_PORT=''
+# ElasticSearch Environment Variables
+ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
+ELASTICSEARCH_USER=''
+ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
+ELASTICSEARCH_PORT='443'
+ELASTICSEARCH_SCHEME='https'
+ELASTICSEARCH_BATCH_SIZE='10'
+ELASTICSEARCH_HOST=''
+ELASTICSEARCH_PASSWORD=''
+# Ingestion or Airflow Environment Variables
+AIRFLOW_HOST: ''
+SERVER_HOST_API_URL: ''
+```
+Once the environment variables values with the RDS are updated then provide this environment variable file as part of docker compose command.
+
+```
+docker compose --env-file ./config/.env.prod up -d openmetadata_server
+```
## Run OpenMetadata with AWS Services
If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway).
@@ -56,21 +212,21 @@ Once you have the RDS and OpenSearch Services Setup, you can update the environm
DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver'
DB_SCHEME='mysql'
DB_USE_SSL='true'
-MYSQL_USER_PASSWORD=''
+DB_USER_PASSWORD=''
DB_SCHEME='mysql'
-MYSQL_HOST=''
-MYSQL_USER=''
-MYSQL_DATABASE=''
-MYSQL_PORT=''
+DB_HOST=''
+DB_USER=''
+DB_DATABASE=''
+DB_PORT=''
# ElasticSearch Environment Variables
ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
-ELASTICSEARCH_USER=''
+ELASTICSEARCH_USER=''
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
ELASTICSEARCH_PORT='443'
ELASTICSEARCH_SCHEME='https'
ELASTICSEARCH_BATCH_SIZE='10'
-ELASTICSEARCH_HOST='vpc-..es.amazonaws.com'
-ELASTICSEARCH_PASSWORD=''
+ELASTICSEARCH_HOST=''
+ELASTICSEARCH_PASSWORD=''
```
Replace the environment variables values with the RDS and OpenSearch Service ones and then provide this environment variable file as part of docker compose command.
@@ -82,7 +238,7 @@ docker compose --env-file ./config/.env.prod up -d openmetadata_server
# Production Deployment
If you are planning on going to PROD, we also recommend taking a look at the following
-deployment strategies:
+other deployment strategies: