From 297900f17a92fbf74b82f462f3d24d7f8a108d62 Mon Sep 17 00:00:00 2001 From: Shannon Bradshaw Date: Mon, 10 Jan 2022 23:53:25 +0000 Subject: [PATCH] GitBook: [#77] Upgrade production OpenMetadata deployment without losing data --- docs/SUMMARY.md | 2 +- docs/install/run-openmetadata.md | 96 ++++++++++++++++ docs/install/upgrade-openmetadata.md | 165 +++++++++++++++------------ 3 files changed, 191 insertions(+), 72 deletions(-) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 7e523a4f473..23c063420ce 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -94,8 +94,8 @@ * [Run OpenMetadata](install/run-openmetadata.md) * [Run in Production](install/run-in-production.md) -* [Run in Kubernetes](install/run-in-kubernetes.md) * [Upgrade OpenMetadata](install/upgrade-openmetadata.md) +* [Run in Kubernetes](install/run-in-kubernetes.md) * [Server Configuration](install/configuration.md) * [Connector Configuration](install/connector-configuration.md) * [Enable Security](install/enable-security/README.md) diff --git a/docs/install/run-openmetadata.md b/docs/install/run-openmetadata.md index c785654f160..99a4c982bd2 100644 --- a/docs/install/run-openmetadata.md +++ b/docs/install/run-openmetadata.md @@ -219,3 +219,99 @@ sudo apt install python3-pip python3-venv ``` Follow the [OSX instructions](run-openmetadata.md#1.-create-a-directory-for-openmetadata) + + + +## Upgrade OpenMetadata + +If you would like to upgrade your OpenMetadata deployment installed following the procedure above, this procedure will guide you through the upgrade process. + +### 1. Ensure your Python virtual environment is activated + +The procedure for [installing OpenMetadata](run-openmetadata.md) asks you to create a new directory and Python virtual environment. The procedure then asks you to install the `openmetadata-ingestion[docker]` Python module in this virtual environment. + +In your command-line environment, please navigate to the directory where you installed `openmetadata-ingestion[docker]` and activate the virtual environment by running the following command. + +``` +source env/bin/activate +``` + +### 2. Check the current version you have installed + +To check the version of `openmetadata-ingestion[docker]` that you have installed, run the following command. + +```bash +metadata --version +``` + +Upon running this command you should see output similar to the following. + +```bash +metadata, version metadata 0.5.0 from /Users/om/openmetadata-docker/env/lib/python3.8 (python 3.8) +``` + +### 3. Check available versions + +To confirm that there is a later version of `openmetadata-ingestion[docker]` available and identify the version you want to install, please run the following command. + +``` +pip3 install 'openmetadata-ingestion[docker]'== +``` + +Upon running this command, you should see output similar to the following. + +``` +ERROR: Could not find a version that satisfies the requirement +openmetadata-ingestion[docker]== (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, +0.3.0, 0.3.2, 0.4.0.dev0, 0.4.0.dev6, 0.4.0, 0.4.1.dev6, 0.4.1, 0.4.2.dev1, 0.4.2, +0.4.2.1, 0.4.3.dev1, 0.4.3.dev2, 0.4.3.dev3, 0.4.3.dev4, 0.4.3, 0.4.4, 0.4.5, 0.4.7, +0.4.8.dev0, 0.4.8.dev2, 0.4.8, 0.4.9, 0.4.10, 0.4.11, 0.5.0rc0, 0.5.0rc1, 0.5.0, +0.5.1.dev0, 0.6.0.dev0, 0.7.0.dev1, 0.7.0.dev2, 0.7.0.dev3, 0.7.0.dev4) +ERROR: No matching distribution found for openmetadata-ingestion[docker]== +``` + +The error messages are expected. This is the accepted means of checking available versions for a Python module using `pip`. + +The output provides a complete list of available versions and enables you to determine whether there are release versions later than the version you currently have installed. Release versions have the form `x.x.x`. Examples of release versions in the above output include, `0.2.0`, `0.4.2`, and `0.5.0`. + +From this output you can also find patch releases (e.g., `0.4.2.1`), release candidates (`0.5.0rc1`), and development releases (e.g., `0.7.0.dev4`). + +### 4. Stop your currently running deployment + +Before upgrading, if you are currently running an OpenMetadata deployment, please stop the deployment by running the following command. + +```bash +metadata docker --stop +``` + +### 5. Install the version of your choice + +#### Option 1. Install the latest release version + +You may install the latest release version by running the following command. + +```bash +pip3 install --upgrade 'openmetadata-ingestion[docker]' +``` + +#### Option 2. Install a specific release, patch, or development version + +You may install a specific version of `openmetadata-ingestion[docker]`by running the following command, specifying the version you want to install in place of ``. + +```bash +pip3 install --upgrade 'openmetadata-ingestion[docker]'== +``` + +For example, if you want to install the `0.7.0.dev4` release, you would run the following command. + +```bash +pip3 install --upgrade 'openmetadata-ingestion[docker]'==0.7.0.dev4 +``` + +### 6. Restart your deployment + +Once you have successfully installed your preferred version of `openmetadata-ingestion[docker]`, restart your deployment using the new version, by running the following command. + +```bash +metadata docker --start +``` diff --git a/docs/install/upgrade-openmetadata.md b/docs/install/upgrade-openmetadata.md index d97fb7c0e66..93a4a49693f 100644 --- a/docs/install/upgrade-openmetadata.md +++ b/docs/install/upgrade-openmetadata.md @@ -1,99 +1,122 @@ --- description: >- - This guide will help you upgrade an existing Docker deployment of OpenMetadata - to a later version. + This guide will help you upgrade an OpenMetadata deployment using release + binaries. --- # Upgrade OpenMetadata +## Requirements + +An OpenMetadata deployment that you installed and configured following the [Run in Production](run-in-production.md) guide. + ## Procedure -### 1. Ensure your Python virtual environment is activated +### 1. Download the release binaries you want to install -The procedure for [installing OpenMetadata](run-openmetadata.md) asks you to create a new directory and Python virtual environment. The procedure then asks you to install the `openmetadata-ingestion[docker]` Python module in this virtual environment. +OpenMetadata release binaries are maintained as GitHub releases. To download a specific release binary: -In your command-line environment, please navigate to the directory where you installed `openmetadata-ingestion[docker]` and activate the virtual environment by running the following command. +1. Visit [github.com/open-metadata/OpenMetadata/releases](https://github.com/open-metadata/OpenMetadata/releases). The latest release will be at the top of this page. +2. Locate the Assets section for the release you want to upgrade to. +3. Download the release binaries. The release binaries will be in a compressed tar file named using the following convention, `openmetadata-x.y.z.tar.gz` Where `x`, `y`, `z` are the major, minor, and patch release numbers respectively. + +### 2. Extract the release binaries from the download file + +Using the command line tool or application of your choice, uncompress and untar the release binary download. For example, to extract using tar, run the following command. ``` -source env/bin/activate +tar xfz openmetadata-0.7.1.tar.gz ``` -### 2. Check the current version you have installed +This will create a directory with the same name as the file minus the `.tar` and `.gz` extensions. -To check the version of `openmetadata-ingestion[docker]` that you have installed, run the following command. +### 3. Navigate into the directory created by extracting the release binaries + +Change into the new directory by issuing a command similar to the following. + +``` +cd openmetadata-x.x.x +``` + +For example, to navigate into the directory created by issuing the tar command above, you would run the following command. + +``` +cd openmetadata-0.7.1 +``` + +### 4. Stop the OpenMetadata server + +OpenMetadata ships with a few control scripts. One is `openmetadata.sh`. This script enables you to start, stop, and perform other deployment operations on the OpenMetadata server. + +Before you migrate your data to the new release you are upgrading to, stop the OpenMetadata server by running the following command. + +``` +./bin/openmetadata.sh stop +``` + +### 5. Migrate database schemas and Elasticsearch indexes + +The `bootstrap/bootstrap_storage.sh` script enables you to perform a number of operations on the OpenMetadata database (in MySQL) and index (in Elasticsearch). + +Migrate your data using this script by running the following command. + +``` +./bootstrap/bootstrap_storage.sh migrate-all +``` + +This will migrate the OpenMetadata database schema to the new version you are upgrading to. It will also migrate the Elasticsearch index to this version. + +### 6. Restart the OpenMetadata server + +Once you've migrated your data to the new version, restart the OpenMetadata server using the new release binaries. You may restart the server by running the following command. + +``` +./bin/openmetadata.sh start +``` + +### 7. Upgrade all your connectors + +If you are ingesting data manually using OpenMetadata connectors, upgrade all your connectors by running the following command for each connector. You will need to replace `` in the command below by the name of the connector you are upgrading. ```bash -metadata --version +pip3 install --upgrade 'openmetadata-ingestion[]' ``` -Upon running this command you should see output similar to the following. +## Troubleshooting + +### "migrate" option failed + +In some circumstances, when attempting to migrate your database schemas and Elasticsearch indexes, you might encounter an error similar to the following. + +``` +14:42:40.630 [main] DEBUG org.flywaydb.core.FlywayExecutor - Memory usage: +58 of 1024M +"migrate" option failed : org.flywaydb.core.api.exception.FlywayValidateException: +Validate failed: Migrations have failed validation +Migration checksum mismatch for migration version 002 +-> Applied to database : -163374426 +-> Resolved locally : 326575949. Either revert the changes to the migration, or +run repair to update the schema history. +Need more flexibility with validation rules? +Learn more: https://rd.gt/3AbJUZE +``` + +If you encounter this error, run the following command to repair your schema. ```bash -metadata, version metadata 0.5.0 from /Users/om/openmetadata-docker/env/lib/python3.8 (python 3.8) +./bootstrap/bootstrap_storage.sh repair ``` -### 3. Check available versions - -To confirm that there is a later version of `openmetadata-ingestion[docker]` available and identify the version you want to install, please run the following command. +After running this command, you should see output similar to the following in response. ``` -pip3 install 'openmetadata-ingestion[docker]'== +14:42:48.110 [main] INFO org.flywaydb.core.internal.command.DbRepair - +Successfully repaired schema +history table `openmetadata_db`.`DATABASE_CHANGE_LOG` (execution time 00:00.058s). +14:42:48.123 [main] DEBUG org.flywaydb.core.FlywayExecutor - +Memory usage: 58 of 1024M +"repair" option successful ``` -Upon running this command, you should see output similar to the following. +Once repair is successful, continue with the procedure above by rerunning the command in Step 5 and continuing from there. -``` -ERROR: Could not find a version that satisfies the requirement -openmetadata-ingestion[docker]== (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, -0.3.0, 0.3.2, 0.4.0.dev0, 0.4.0.dev6, 0.4.0, 0.4.1.dev6, 0.4.1, 0.4.2.dev1, 0.4.2, -0.4.2.1, 0.4.3.dev1, 0.4.3.dev2, 0.4.3.dev3, 0.4.3.dev4, 0.4.3, 0.4.4, 0.4.5, 0.4.7, -0.4.8.dev0, 0.4.8.dev2, 0.4.8, 0.4.9, 0.4.10, 0.4.11, 0.5.0rc0, 0.5.0rc1, 0.5.0, -0.5.1.dev0, 0.6.0.dev0, 0.7.0.dev1, 0.7.0.dev2, 0.7.0.dev3, 0.7.0.dev4) -ERROR: No matching distribution found for openmetadata-ingestion[docker]== -``` - -The error messages are expected. This is the accepted means of checking available versions for a Python module using `pip`. - -The output provides a complete list of available versions and enables you to determine whether there are release versions later than the version you currently have installed. Release versions have the form `x.x.x`. Examples of release versions in the above output include, `0.2.0`, `0.4.2`, and `0.5.0`. - -From this output you can also find patch releases (e.g., `0.4.2.1`), release candidates (`0.5.0rc1`), and development releases (e.g., `0.7.0.dev4`). - -### 4. Stop your currently running deployment - -Before upgrading, if you are currently running an OpenMetadata deployment, please stop the deployment by running the following command. - -```bash -metadata docker --stop -``` - -### 5. Install the version of your choice - -#### Option 1. Install the latest release version - -You may install the latest release version by running the following command. - -```bash -pip3 install --upgrade 'openmetadata-ingestion[docker]' -``` - -#### Option 2. Install a specific release, patch, or development version - -You may install a specific version of `openmetadata-ingestion[docker]`by running the following command, specifying the version you want to install in place of ``. - -```bash -pip3 install --upgrade 'openmetadata-ingestion[docker]'== -``` - -For example, if you want to install the `0.7.0.dev4` release, you would run the following command. - -```bash -pip3 install --upgrade 'openmetadata-ingestion[docker]'==0.7.0.dev4 -``` - -### 6. Restart your deployment - -Once you have successfully installed your preferred version of `openmetadata-ingestion[docker]`, restart your deployment using the new version, by running the following command. - -```bash -metadata docker --start -```