yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-21 15:38:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

Added information for airflow migration and upgrade (#7818)

Browse Source
This commit is contained in:
Teddy 2022-10-01 17:32:26 +02:00 committed by GitHub
parent 5b3b9ec811
commit ace235922a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 20 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
openmetadata-docs/content/deployment/upgrade/versions/011-to-012.md
View File

@ -86,8 +86,26 @@ to Airflow 2.3.3.
If you are using your own Airflow to prepare the ingestion from the UI, which is stuck in version 2.1.4, and you cannot
upgrade that, but you want to use OM 0.12, reach out to us.
**Note**
If you are using `openmetadata/ingestion` Docker image and you've upgraded to 0.12.0 reusing volumes mounted to the `openmetadata/ingestion:0.11.5` container you will need to run the `metadata openmetadata-imports-migration` command inside the `openmetadata/ingestion:0.12.x` container. Indeed, `openmetadata-airflow-managed-apis` has been renamed to `openmetadata-managed-apis` and its import from `import openmetadata` to `import openmetadata_managed_apis`. If theimport paths are not updated through the `metadata` command it will result in broken DAGs. By default the command will look for DAGs stored in `/ingestion/examples/airflow/dags`. If you have changed where generated DAGs are stored you can specify the path to where your DAGs are stored `metadata openmetadata-imports-migration -d <path/to/folder>`
**Note #1**
If you are using `openmetadata/ingestion` Docker image and you've upgraded to 0.12.x reusing volumes mounted to the `openmetadata/ingestion:0.11.5` container you will need to run the `metadata openmetadata-imports-migration` command inside the `openmetadata/ingestion:0.12.x` container. Indeed, `openmetadata-airflow-managed-apis` has been renamed to `openmetadata-managed-apis` and its import from `import openmetadata` to `import openmetadata_managed_apis`. If the import paths are not updated through the `metadata` command it will result in broken DAGs. By default the command will look for DAGs stored in `/ingestion/examples/airflow/dags`. If you have changed where generated DAGs are stored you can specify the path to where your DAGs are stored `metadata openmetadata-imports-migration -d <path/to/folder>`
**Note #2**
If you are using `openmetadata/ingestion` Docker image in 0.12.1, dags and generated config files are stored respectively in `/opt/airflow/dags` and `/opt/airflow/dag_generated_config`. You will need to run the `metadata openmetadata-imports-migration` command with the `--change-config-file-path` inside the `openmetadata/ingestion:0.12.x` container to make sure DAGs workflow are pointing to the correct config file.
**Note #3**
Upgrading airflow from 2.1.4 to 2.3 requires a few steps. If you are using your airflow instance only to run OpenMetadata workflow we recommend you to simply drop the airflow database. You can simply connect to your database engine where your airflow database exist, make a back of your database, drop it and recreate it.
```
DROP DATABASE airflow_db;
CREATE DATABASE airflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON airflow_db.* TO 'airflow_user'@'%' WITH GRANT OPTION;
```
If you would like to keep the existing data in your airflow instance or simply cannot drop your airflow database you will need to follow the steps below:
1. Make a backup of your database (this will come handing if the migration fails and you need to perform any kind of restore)
1. Upgrade your airflow instance from 2.1.x to 2.2.x
2. Before upgrading to the 2.3.x version perform the steps describe on the airflow documentation page [[here]](https://airflow.apache.org/docs/apache-airflow/2.4.0/installation/upgrading.html#wrong-encoding-in-mysql-database) to make sure character set/collation uses the correct encoding -- this has been changing across MySQL versions.
3. Once the above has been performed you should be able to upgrade to airflow 2.3.x
### Connector Improvements