yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-22 16:08:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

Updated the document for docker deployment (#8527)

Browse Source 
* Updated the document for docker deployment

* Updated the document for docker deployment

* Updated the document for docker deployment

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
This commit is contained in:
Vj-L 2022-11-07 11:00:00 +05:30 committed by GitHub
parent 292672a970
commit 987e3a66fb
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 183 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

210
openmetadata-docs/content/deployment/docker/index.md
View File

@ -4,39 +4,195 @@ slug: /deployment/docker
--- ---
# Docker Deployment # Docker Deployment
Deploying OpenMetadata in Docker is a great start!
Before starting with the deployment make sure you follow all the below Prerequisites.
## Prerequisites
### Docker (version 20.10.0 or greater)
[Docker](https://docs.docker.com/get-started/overview/) is an open-source platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure, so you can deliver software quickly using OS-level virtualization. It helps deliver software in packages called Containers.
Deploying OpenMetadata in Docker is a great start! Take a look at our To check what version of Docker you have, please use the following command.
[Quickstart](/quick-start/local-deployment) guide to learn how to get OpenMetadata ```commandline
up and running locally in less than 7 minutes! docker --version
```
If those steps are already done, you might want to bind Docker Volumes If you need to install Docker, please visit [Get Docker](https://docs.docker.com/get-docker/).
for data persistence. Learn how to do so [here](/deployment/docker/volumes). ### Docker Compose (version v2.2.3 or greater)
The Docker compose package enables you to define and run multi-container Docker applications. The compose command integrates compose functions into the Docker platform, making them available from the Docker command-line interface ( CLI). The Python packages you will install in the procedure below use compose to deploy OpenMetadata.
To test out your security integration, check out how to - **MacOS X**: Docker on MacOS X ships with compose already available in the Docker CLI.
[Enable Security](/deployment/docker/security). - **Linux**: To install compose on Linux systems, please visit the Docker CLI command documentation and follow the
instructions.
## Changing ports To verify that the docker compose command is installed and accessible on your system, run the following command.
<Note> ```commandline
docker compose version
```
To make changes to the `docker-compose.yaml` file you will need to download it from the release page [here](https://github.com/open-metadata/OpenMetadata/releases). The latest version is at the top of the page Upon running this command you should see output similar to the following.
</Note>
This docker deployment is powered by `docker compose`, and uses the `docker-compose.yml` files shipped during ```commandline
each release [example](https://github.com/open-metadata/OpenMetadata/releases/tag/0.11.4-release). Docker Compose version v2.2.3
```
As with the [Docker Volumes](/deployment/docker/volumes), you might want to tune a bit the compose file to modify ### Install Docker Compose Version 2 on Linux
the default ports.
Follow the instructions [here](https://docs.docker.com/compose/cli-command/#install-on-linux) to install docker compose version 2
1. Run the following command to download the current stable release of Docker Compose
```
DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o
$DOCKER_CONFIG/cli-plugins/docker-compose
```
This command installs Compose V2 for the active user under $HOME directory. To install Docker Compose for all users
on your system, replace` ~/.docker/cli-plugins` with `/usr/local/lib/docker/cli-plugins`.
2. Apply executable permissions to the binary
```
chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
```
3. Test your installation
```
docker compose version
> Docker Compose version v2.2.3
```
## Steps for Deploying OpenMetadata using Docker
- First download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases). The latest version is at the top of the page
- Deploying with MySQL: Download `docker-compose.yml` file from the above link.
- Deploying with PostgreSQL: Download `docker-compose-postgres.yml` file from the above link.
- Run the below command to deploy the OpenMetadata
```commandline
docker compose up --build -d
```
This command will pull the docker images of Openmetadata for MySQL, OpenMetadat-Server, OpenMetadata-Ingestion and Elasticsearch.
Upon running this command you should see output similar to the following.
```commandline
+] Running 7/8
⠿ Network metadata_app_net Created 0.2s
⠿ Volume "metadata_ingestion-volume-dag-airflow" Created 0.0s
⠿ Volume "metadata_ingestion-volume-dags" Created 0.0s
⠿ Volume "metadata_ingestion-volume-tmp" Created 0.0s
⠿ Container openmetadata_elasticsearch Started 5.9s
⠿ Container openmetadata_mysql Started 38.3s
⠿ Container openmetadata_server Started 124.8s
⠿ Container openmetadata_ingestion Started 0.3s
```
You can validate that all containers are up by running with command `docker ps`.
```commandline
❯ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
470cc8149826 openmetadata/server:0.12.2 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server
63578aacbff5 openmetadata/ingestion:0.12.2 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion
9f5ee8334f4b docker.elastic.co/elasticsearch/elasticsearch:7.10.2 "/tini -- /usr/local…" 45 seconds ago Up 44 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp openmetadata_elasticsearch
08947ab3424b openmetadata/db:0.12.2 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql
```
In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585)
## Port Mapping / Port Forwarding
### For OpenMetadata-Server
We are shipping the OpenMetadata server and UI at `8585`, and the ingestion container (Airflow) at `8080`. You can We are shipping the OpenMetadata server and UI at `8585`, and the ingestion container (Airflow) at `8080`. You can
take a look at the official Docker [docs](https://docs.docker.com/compose/compose-file/#ports). As an example, You could change the port number's according to your requirement. As an example, You could
update the ports to serve Airflow at `1234` with: update the ports to serve OpenMetadata Server and UI at port `80`
To achieve this
- You just have to update the ports mapping of the openmetadata-server in the `docker-compose.yml` file under `openmetadata-server` docker service section.
```yaml ```yaml
ports: ports:
- "1234:8080" - "80:8585"
``` ```
- Once the port is updated if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command
```commandline
docker compose up --build -d
```
### For Ingestion-Server
We are shipping the OpenMetadata server and UI at `8585`, and the ingestion container (Airflow) at `8080`. You can
change the port number's according to your requirement. As an example, You could
update the ports to serve Ingestion Server and UI at port `80`
To achieve this
- You just have to update the ports mapping of the openmetadata-server in the `docker-compose.yml` file under `ingestion-server` docker service section.
```yaml
ports:
- "80:8080"
```
- Also update the Airflow environment variable in openmetadata-server section
```commandline
AIRFLOW_HOST: '<AIRFLOW_HOST:-<AIRFLOW_HOST:80>'
```
- Once the port is updated if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command
```commandline
docker compose up --build -d
```
## PROD Deployment of OpenMetadata Using Docker
If you are planning on going to PROD, we recommend to validate below points:
- MySQL and OpenSearch (ElasticSearch) are available.
- OpenMetadata-Server require the minimum configuration of 2vCPU and 6Memory (GiB)
- OpenMetadata-Ingestion require the minimum configuration of 2vCPU and 8Memory (GiB)
- We also recommend to bind Docker Volumes for data persistence. Learn how to do so [here](/deployment/docker/volumes).
### Steps for Deploying Ingestion
- Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases).
- Update the environment variables below for OpenMetadata-Ingestion Docker Compose backed systems to connect with Database.
```
# MySQL Environment Variables for ingestion service
DB_HOST: '<DB_HOST_NAME>'
DB_PORT: '<DB_PORT>'
AIRFLOW_DB: '<AIRFLOW_DATABASE>'
AIRFLOW_DB_SCHEME: '<AIRFLOW_DB_SCHEME>'
DB_USER: '<AIRFLOW_DB_USER>'
DB_PASSWORD: '<AIRFLOW_DB_PASSWORD>'
```
Once the environment variables values with the RDS are updated then provide this environment variable file as part of docker compose command.
```
docker compose --env-file ./config/.env.prod up -d openmetadata_ingestion
```
### Steps for Deploying OpenMetadata-Server
- Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases).
- Update the environment variables below for OpenMetadata-Ingestion Docker Compose backed systems to connect with Database and ElasticSearch and Ingestion.
```
# MySQL Environment Variables
DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver'
DB_SCHEME='mysql'
DB_USE_SSL='true'
DB_USER_PASSWORD='<OPENMETADATA_DB_USER_PASSWORD>'
DB_SCHEME='mysql'
DB_HOST='<DB_HOST>'
DB_USER='<OPENMETADATA__USER_NAME>'
DB_DATABASE='<OPENMETADATA_DATABASE_NAME>'
DB_PORT='<DB_PORT>'
# ElasticSearch Environment Variables
ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
ELASTICSEARCH_USER='<ELASTICSEARCH_USERNAME>'
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
ELASTICSEARCH_PORT='443'
ELASTICSEARCH_SCHEME='https'
ELASTICSEARCH_BATCH_SIZE='10'
ELASTICSEARCH_HOST='<ELASTICSEARCH_HOST_URL>'
ELASTICSEARCH_PASSWORD='<ELASTICSEARCH_PASSWORD>'
# Ingestion or Airflow Environment Variables
AIRFLOW_HOST: '<AIRFLOW_HOST_URL>'
SERVER_HOST_API_URL: '<OPENMETADATA_HOST_URL_WITH_SCHEME/api>'
```
Once the environment variables values with the RDS are updated then provide this environment variable file as part of docker compose command.
```
docker compose --env-file ./config/.env.prod up -d openmetadata_server
```
## Run OpenMetadata with AWS Services ## Run OpenMetadata with AWS Services
If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway). If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway).
@ -56,21 +212,21 @@ Once you have the RDS and OpenSearch Services Setup, you can update the environm
DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver' DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver'
DB_SCHEME='mysql' DB_SCHEME='mysql'
DB_USE_SSL='true' DB_USE_SSL='true'
MYSQL_USER_PASSWORD='<YOUR_RDS_USER_PASSWORD>' DB_USER_PASSWORD='<DATABASE_USER_PASSWORD>'
DB_SCHEME='mysql' DB_SCHEME='mysql'
MYSQL_HOST='<YOUR_RDS_HOST_NAME>' DB_HOST='<DATABASE_HOST_NAME>'
MYSQL_USER='<YOUR_RDS_USER_NAME>' DB_USER='<DATABASE_USER_NAME>'
MYSQL_DATABASE='<YOUR_RDS_DATABASE_NAME>' DB_DATABASE='<DATABASE_NAME>'
MYSQL_PORT='<YOUR_RDS_PORT>' DB_PORT='<DATABASE_PORT>'
# ElasticSearch Environment Variables # ElasticSearch Environment Variables
ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60' ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
ELASTICSEARCH_USER='<ES_USERNAME>' ELASTICSEARCH_USER='<ELASTICSEARCH_USERNAME>'
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5' ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
ELASTICSEARCH_PORT='443' ELASTICSEARCH_PORT='443'
ELASTICSEARCH_SCHEME='https' ELASTICSEARCH_SCHEME='https'
ELASTICSEARCH_BATCH_SIZE='10' ELASTICSEARCH_BATCH_SIZE='10'
ELASTICSEARCH_HOST='vpc-<random_characters>.<aws_region>.es.amazonaws.com' ELASTICSEARCH_HOST='<ELASTICSEARCH_HOST_URL>'
ELASTICSEARCH_PASSWORD='<ES_PASSWORD>' ELASTICSEARCH_PASSWORD='<ELASTICSEARCH_PASSWORD>'
``` ```
Replace the environment variables values with the RDS and OpenSearch Service ones and then provide this environment variable file as part of docker compose command. Replace the environment variables values with the RDS and OpenSearch Service ones and then provide this environment variable file as part of docker compose command.
@ -82,7 +238,7 @@ docker compose --env-file ./config/.env.prod up -d openmetadata_server
# Production Deployment # Production Deployment
If you are planning on going to PROD, we also recommend taking a look at the following If you are planning on going to PROD, we also recommend taking a look at the following
deployment strategies: other deployment strategies:
<InlineCalloutContainer> <InlineCalloutContainer>
<InlineCallout <InlineCallout