mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-10-06 14:26:28 +00:00
Update Docker upgrade doc (#7891)
This commit is contained in:
parent
2a9732dbda
commit
11f7f6e3cf
@ -17,6 +17,11 @@ To test out your security integration, check out how to
|
||||
|
||||
## Changing ports
|
||||
|
||||
<Note>
|
||||
|
||||
To make changes to the `docker-compose.yaml` file you will need to download it from the release page [here](https://github.com/open-metadata/OpenMetadata/releases). The latest version is at the top of the page
|
||||
</Note>
|
||||
|
||||
This docker deployment is powered by `docker compose`, and uses the `docker-compose.yml` files shipped during
|
||||
each release [example](https://github.com/open-metadata/OpenMetadata/releases/tag/0.11.4-release).
|
||||
|
||||
|
@ -8,6 +8,11 @@ Advance section
|
||||
|
||||
Volumes provide the ability to connect specific filesystem paths of the container back to the host machine. If a directory or a file in the container is mounted, changes in that directory or file can also be seen on the host machine.we are going to use a mapping of a directory present on the host macine with the container path.
|
||||
|
||||
<Note>
|
||||
|
||||
To make changes to the `docker-compose.yaml` file you will need to download it from the release page [here](https://github.com/open-metadata/OpenMetadata/releases). The latest version is at the top of the page
|
||||
</Note>
|
||||
|
||||
# Docker Volumes
|
||||
|
||||
## Volumes for MYSQL container:
|
||||
|
@ -5,18 +5,86 @@ slug: /deployment/upgrade/docker
|
||||
|
||||
# Upgrade on Docker
|
||||
|
||||
For running OpenMetadata on Docker, we downloaded a `docker-compose.yml` file. Optionally, we added some
|
||||
To run OpenMetadata with Docker, you can simply downloaded the `docker-compose.yml` file. Optionally, we added some
|
||||
Named Volumes to handle data persistence.
|
||||
|
||||
<Note>
|
||||
|
||||
You can refresh these steps [here](/deployment/docker)
|
||||
You can find more details about Docker deployment [here](/deployment/docker)
|
||||
|
||||
</Note>
|
||||
|
||||
Now,
|
||||
Below we have highlited the steps needed to upgrade to the latest version with Docker. Make sure to also look [here](/deployment/upgrade/versions/011-to-012) for the general details related to upgrading to 0.12
|
||||
|
||||
1. Visit [github.com/open-metadata/OpenMetadata/releases](https://github.com/open-metadata/OpenMetadata/releases). The
|
||||
latest release will be at the top of this page.
|
||||
2. Download the new `docker-compose.yml` file.
|
||||
3. Run the `docker compose up -d` command on the new compose file.
|
||||
### 1. Download `docker-compose.yaml` file
|
||||
|
||||
Go to [github.com/open-metadata/OpenMetadata/releases](https://github.com/open-metadata/OpenMetadata/releases). The latest release will be at the top of this page.
|
||||
|
||||
Download the new `docker-compose.yml` file. You can run the below command to download the file (replacing `{version}` with the correct version) or simply visit the page and download the file from your browser.
|
||||
```
|
||||
wget https://github.com/open-metadata/OpenMetadata/releases/download/{version}-release/docker-compose.yml
|
||||
```
|
||||
or if you wish to use postgres as the database
|
||||
```
|
||||
wget https://github.com/open-metadata/OpenMetadata/releases/download/0.12.0-release/docker-compose-postgres.yml
|
||||
```
|
||||
|
||||
### 2. Backup your Data [IMPORTANT]
|
||||
|
||||
Make a backup of your database. You can find the steps to follow [here](=/deployment/upgrade/backup-metadata#backup-metadata). Please note this is an important step as it would allow you to revert to a stable state if any issues were to happen during your upgrade.
|
||||
|
||||
### 3. Add Volumes and Publish Ports
|
||||
|
||||
Update the docker compose file you just downloaded to add persistent volume for MySQL. You can follow the steps [here](/deployment/docker/volumes#docker-volumes). If you had previously configured volumes for MySQL and the ingestion container make sure the 0.12.1 compose file is referencing the correct volume so that 0.11.5 data will be available when upgrading to 0.12.1.
|
||||
|
||||
**Note:** in 0.12.1 we are not publishing MySQL ports to the host container. If you wish to do so you will need to update the MySQL service in the compose file to have the following configuration
|
||||
```yaml
|
||||
services:
|
||||
mysql:
|
||||
...
|
||||
ports:
|
||||
- "3306:3306"
|
||||
...
|
||||
```
|
||||
|
||||
### 4. Prepare Airflow upgrade to 2.3 (Only if upgrading from 0.11.x)
|
||||
|
||||
(If you are upgrading from 0.11.x to 0.12.x) In 0.12.x we are using airflow 2.3.3. There is a known issue when upgrading from airflow 2.1.x to 2.3.3. Make sure to look at the following notes [here](/deployment/upgrade/versions/011-to-012#airflow-version) for more details
|
||||
|
||||
### 5. Stop, Remove and Start your Containers
|
||||
Stop and remove (without deleting volumes) the already running containers. Then run the `docker compose up -d` command on the new compose file.
|
||||
|
||||
### 6. Update DAGs config
|
||||
#### 6.1 Update imports
|
||||
If you are using `openmetadata/ingestion` Docker image and you've upgraded to 0.12.x reusing volumes mounted to the `openmetadata/ingestion:0.11.5` container you will need to run the `metadata openmetadata-imports-migration` command inside the `openmetadata/ingestion:0.12.x` container. Indeed, `openmetadata-airflow-managed-apis` has been renamed to `openmetadata-managed-apis` and its import from `import openmetadata` to `import openmetadata_managed_apis`. If the import paths are not updated through the `metadata` command it will result in broken DAGs. By default the command will look for DAGs stored in `/opt/airflow/dags`. If you have changed where generated DAGs are stored you can specify the path to where your DAGs are stored `metadata openmetadata-imports-migration -d <path/to/folder>`
|
||||
|
||||
#### 6.2 Update config file path
|
||||
If you are using `openmetadata/ingestion` Docker image in 0.12.1 and migrated either from 0.12.0 or 0.11.x `dags` and `dag_generated_config` have changed path. Dags and generated config files are now stored respectively in `/opt/airflow/dags` and `/opt/airflow/dag_generated_config`. You will need to run the `metadata openmetadata-imports-migration` command with the `--change-config-file-path` inside the `openmetadata/ingestion:0.12.x` container to make sure DAGs workflow are pointing to the correct config file.
|
||||
|
||||
You can run 6.1 and 6.2 together using the following command `metadata openmetadata-imports-migration --change-config-file-path`.
|
||||
|
||||
### Troubleshooting
|
||||
#### Permision Denied when running `metadata openmetadata-imports-migration`
|
||||
If you have a `Permission Denied` error thrown when runnign `metadata openmetadata-imports-migration --change-config-file-path` you might need to change the permission on the `/opt/airflow/dags` folder. SSH into the ingestion container and check the permission on the folder running the below commands
|
||||
```
|
||||
ls -l /opt/airflow
|
||||
```
|
||||
```
|
||||
ls -l /opt/airflow/dags
|
||||
```
|
||||
both the `dags` folder and the files inside `dags/` should have `airflow root` permission. if this is not the case simply run the below command
|
||||
```
|
||||
chown -R airflow:root /opt/airflow/dags
|
||||
```
|
||||
|
||||
#### Broken DAGs can't load config file: Permission Denied
|
||||
You might need to change the permission on the `/opt/airflow/dag_generated_config` folder. SSH into the ingestion container and check the permission on the folder running the below commands
|
||||
```
|
||||
ls -l /opt/airflow
|
||||
```
|
||||
```
|
||||
ls -l /opt/airflow/dag_generated_config
|
||||
```
|
||||
both the `dags` folder and the files inside `dags/` should have `airflow root` permission. if this is not the case simply run the below command
|
||||
```
|
||||
chown -R airflow:root /opt/airflow/dag_generated_config
|
@ -5,7 +5,7 @@ slug: /deployment/upgrade/versions/011-to-012
|
||||
|
||||
# Upgrade from 0.11 to 0.12
|
||||
|
||||
Upgrading from 0.11 to 0.12 can be done directly on your instances. This page will list a couple of details that you should take into consideration when running the upgrade.
|
||||
Upgrading from 0.11 to 0.12 can be done directly on your instances. This page will list few general details you should take into consideration when running the upgrade.
|
||||
|
||||
## Highlights
|
||||
|
||||
@ -86,13 +86,7 @@ to Airflow 2.3.3.
|
||||
If you are using your own Airflow to prepare the ingestion from the UI, which is stuck in version 2.1.4, and you cannot
|
||||
upgrade that, but you want to use OM 0.12, reach out to us.
|
||||
|
||||
**Note #1**
|
||||
If you are using `openmetadata/ingestion` Docker image and you've upgraded to 0.12.x reusing volumes mounted to the `openmetadata/ingestion:0.11.5` container you will need to run the `metadata openmetadata-imports-migration` command inside the `openmetadata/ingestion:0.12.x` container. Indeed, `openmetadata-airflow-managed-apis` has been renamed to `openmetadata-managed-apis` and its import from `import openmetadata` to `import openmetadata_managed_apis`. If the import paths are not updated through the `metadata` command it will result in broken DAGs. By default the command will look for DAGs stored in `/ingestion/examples/airflow/dags`. If you have changed where generated DAGs are stored you can specify the path to where your DAGs are stored `metadata openmetadata-imports-migration -d <path/to/folder>`
|
||||
|
||||
**Note #2**
|
||||
If you are using `openmetadata/ingestion` Docker image in 0.12.1, dags and generated config files are stored respectively in `/opt/airflow/dags` and `/opt/airflow/dag_generated_config`. You will need to run the `metadata openmetadata-imports-migration` command with the `--change-config-file-path` inside the `openmetadata/ingestion:0.12.x` container to make sure DAGs workflow are pointing to the correct config file.
|
||||
|
||||
**Note #3**
|
||||
**Note**
|
||||
Upgrading airflow from 2.1.4 to 2.3 requires a few steps. If you are using your airflow instance only to run OpenMetadata workflow we recommend you to simply drop the airflow database. You can simply connect to your database engine where your airflow database exist, make a back of your database, drop it and recreate it.
|
||||
|
||||
```
|
||||
|
Loading…
x
Reference in New Issue
Block a user