mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-12-24 14:08:45 +00:00
GitBook: [#77] Upgrade production OpenMetadata deployment without losing data
This commit is contained in:
Shannon Bradshaw
Sriharsha Chintalapani
parent
3896a2df3a
commit
297900f17a
@ -94,8 +94,8 @@
|
||||
|
||||
* [Run OpenMetadata](install/run-openmetadata.md)
|
||||
* [Run in Production](install/run-in-production.md)
|
||||
* [Run in Kubernetes](install/run-in-kubernetes.md)
|
||||
* [Upgrade OpenMetadata](install/upgrade-openmetadata.md)
|
||||
* [Run in Kubernetes](install/run-in-kubernetes.md)
|
||||
* [Server Configuration](install/configuration.md)
|
||||
* [Connector Configuration](install/connector-configuration.md)
|
||||
* [Enable Security](install/enable-security/README.md)
|
||||
|
||||
@ -219,3 +219,99 @@ sudo apt install python3-pip python3-venv
|
||||
```
|
||||
|
||||
Follow the [OSX instructions](run-openmetadata.md#1.-create-a-directory-for-openmetadata)
|
||||
|
||||
|
||||
|
||||
## Upgrade OpenMetadata
|
||||
|
||||
If you would like to upgrade your OpenMetadata deployment installed following the procedure above, this procedure will guide you through the upgrade process.
|
||||
|
||||
### 1. Ensure your Python virtual environment is activated
|
||||
|
||||
The procedure for [installing OpenMetadata](run-openmetadata.md) asks you to create a new directory and Python virtual environment. The procedure then asks you to install the `openmetadata-ingestion[docker]` Python module in this virtual environment.
|
||||
|
||||
In your command-line environment, please navigate to the directory where you installed `openmetadata-ingestion[docker]` and activate the virtual environment by running the following command.
|
||||
|
||||
```
|
||||
source env/bin/activate
|
||||
```
|
||||
|
||||
### 2. Check the current version you have installed
|
||||
|
||||
To check the version of `openmetadata-ingestion[docker]` that you have installed, run the following command.
|
||||
|
||||
```bash
|
||||
metadata --version
|
||||
```
|
||||
|
||||
Upon running this command you should see output similar to the following.
|
||||
|
||||
```bash
|
||||
metadata, version metadata 0.5.0 from /Users/om/openmetadata-docker/env/lib/python3.8 (python 3.8)
|
||||
```
|
||||
|
||||
### 3. Check available versions
|
||||
|
||||
To confirm that there is a later version of `openmetadata-ingestion[docker]` available and identify the version you want to install, please run the following command.
|
||||
|
||||
```
|
||||
pip3 install 'openmetadata-ingestion[docker]'==
|
||||
```
|
||||
|
||||
Upon running this command, you should see output similar to the following.
|
||||
|
||||
```
|
||||
ERROR: Could not find a version that satisfies the requirement
|
||||
openmetadata-ingestion[docker]== (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4,
|
||||
0.3.0, 0.3.2, 0.4.0.dev0, 0.4.0.dev6, 0.4.0, 0.4.1.dev6, 0.4.1, 0.4.2.dev1, 0.4.2,
|
||||
0.4.2.1, 0.4.3.dev1, 0.4.3.dev2, 0.4.3.dev3, 0.4.3.dev4, 0.4.3, 0.4.4, 0.4.5, 0.4.7,
|
||||
0.4.8.dev0, 0.4.8.dev2, 0.4.8, 0.4.9, 0.4.10, 0.4.11, 0.5.0rc0, 0.5.0rc1, 0.5.0,
|
||||
0.5.1.dev0, 0.6.0.dev0, 0.7.0.dev1, 0.7.0.dev2, 0.7.0.dev3, 0.7.0.dev4)
|
||||
ERROR: No matching distribution found for openmetadata-ingestion[docker]==
|
||||
```
|
||||
|
||||
The error messages are expected. This is the accepted means of checking available versions for a Python module using `pip`.
|
||||
|
||||
The output provides a complete list of available versions and enables you to determine whether there are release versions later than the version you currently have installed. Release versions have the form `x.x.x`. Examples of release versions in the above output include, `0.2.0`, `0.4.2`, and `0.5.0`. 
|
||||
|
||||
From this output you can also find patch releases (e.g., `0.4.2.1`), release candidates (`0.5.0rc1`), and development releases (e.g., `0.7.0.dev4`). 
|
||||
|
||||
### 4. Stop your currently running deployment
|
||||
|
||||
Before upgrading, if you are currently running an OpenMetadata deployment, please stop the deployment by running the following command.
|
||||
|
||||
```bash
|
||||
metadata docker --stop
|
||||
```
|
||||
|
||||
### 5. Install the version of your choice
|
||||
|
||||
#### Option 1. Install the latest release version
|
||||
|
||||
You may install the latest release version by running the following command.
|
||||
|
||||
```bash
|
||||
pip3 install --upgrade 'openmetadata-ingestion[docker]'
|
||||
```
|
||||
|
||||
#### Option 2. Install a specific release, patch, or development version
|
||||
|
||||
You may install a specific version of `openmetadata-ingestion[docker]`by running the following command, specifying the version you want to install in place of `<version>`.
|
||||
|
||||
```bash
|
||||
pip3 install --upgrade 'openmetadata-ingestion[docker]'==<version>
|
||||
```
|
||||
|
||||
For example, if you want to install the `0.7.0.dev4` release, you would run the following command.
|
||||
|
||||
```bash
|
||||
pip3 install --upgrade 'openmetadata-ingestion[docker]'==0.7.0.dev4
|
||||
```
|
||||
|
||||
### 6. Restart your deployment
|
||||
|
||||
Once you have successfully installed your preferred version of `openmetadata-ingestion[docker]`, restart your deployment using the new version, by running the following command.
|
||||
|
||||
```bash
|
||||
metadata docker --start
|
||||
```
|
||||
|
||||
@ -1,99 +1,122 @@
|
||||
---
|
||||
description: >-
|
||||
This guide will help you upgrade an existing Docker deployment of OpenMetadata
|
||||
to a later version.
|
||||
This guide will help you upgrade an OpenMetadata deployment using release
|
||||
binaries.
|
||||
---
|
||||
|
||||
# Upgrade OpenMetadata
|
||||
|
||||
## Requirements
|
||||
|
||||
An OpenMetadata deployment that you installed and configured following the [Run in Production](run-in-production.md) guide.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Ensure your Python virtual environment is activated
|
||||
### 1. Download the release binaries you want to install
|
||||
|
||||
The procedure for [installing OpenMetadata](run-openmetadata.md) asks you to create a new directory and Python virtual environment. The procedure then asks you to install the `openmetadata-ingestion[docker]` Python module in this virtual environment.
|
||||
OpenMetadata release binaries are maintained as GitHub releases. To download a specific release binary:
|
||||
|
||||
In your command-line environment, please navigate to the directory where you installed `openmetadata-ingestion[docker]` and activate the virtual environment by running the following command.
|
||||
1. Visit [github.com/open-metadata/OpenMetadata/releases](https://github.com/open-metadata/OpenMetadata/releases). The latest release will be at the top of this page.
|
||||
2. Locate the Assets section for the release you want to upgrade to.
|
||||
3. Download the release binaries. The release binaries will be in a compressed tar file named using the following convention, `openmetadata-x.y.z.tar.gz` Where `x`, `y`, `z` are the major, minor, and patch release numbers respectively.
|
||||
|
||||
### 2. Extract the release binaries from the download file
|
||||
|
||||
Using the command line tool or application of your choice, uncompress and untar the release binary download. For example, to extract using tar, run the following command.
|
||||
|
||||
```
|
||||
source env/bin/activate
|
||||
tar xfz openmetadata-0.7.1.tar.gz
|
||||
```
|
||||
|
||||
### 2. Check the current version you have installed
|
||||
This will create a directory with the same name as the file minus the `.tar` and `.gz` extensions.
|
||||
|
||||
To check the version of `openmetadata-ingestion[docker]` that you have installed, run the following command.
|
||||
### 3. Navigate into the directory created by extracting the release binaries
|
||||
|
||||
Change into the new directory by issuing a command similar to the following.
|
||||
|
||||
```
|
||||
cd openmetadata-x.x.x
|
||||
```
|
||||
|
||||
For example, to navigate into the directory created by issuing the tar command above, you would run the following command.
|
||||
|
||||
```
|
||||
cd openmetadata-0.7.1
|
||||
```
|
||||
|
||||
### 4. Stop the OpenMetadata server
|
||||
|
||||
OpenMetadata ships with a few control scripts. One is `openmetadata.sh`. This script enables you to start, stop, and perform other deployment operations on the OpenMetadata server.
|
||||
|
||||
Before you migrate your data to the new release you are upgrading to, stop the OpenMetadata server by running the following command.
|
||||
|
||||
```
|
||||
./bin/openmetadata.sh stop
|
||||
```
|
||||
|
||||
### 5. Migrate database schemas and Elasticsearch indexes
|
||||
|
||||
The `bootstrap/bootstrap_storage.sh` script enables you to perform a number of operations on the OpenMetadata database (in MySQL) and index (in Elasticsearch). 
|
||||
|
||||
Migrate your data using this script by running the following command.
|
||||
|
||||
```
|
||||
./bootstrap/bootstrap_storage.sh migrate-all
|
||||
```
|
||||
|
||||
This will migrate the OpenMetadata database schema to the new version you are upgrading to. It will also migrate the Elasticsearch index to this version.
|
||||
|
||||
### 6. Restart the OpenMetadata server
|
||||
|
||||
Once you've migrated your data to the new version, restart the OpenMetadata server using the new release binaries. You may restart the server by running the following command.
|
||||
|
||||
```
|
||||
./bin/openmetadata.sh start
|
||||
```
|
||||
|
||||
### 7. Upgrade all your connectors
|
||||
|
||||
If you are ingesting data manually using OpenMetadata connectors, upgrade all your connectors by running the following command for each connector. You will need to replace `<connectorname>` in the command below by the name of the connector you are upgrading.
|
||||
|
||||
```bash
|
||||
metadata --version
|
||||
pip3 install --upgrade 'openmetadata-ingestion[<connectorname>]'
|
||||
```
|
||||
|
||||
Upon running this command you should see output similar to the following.
|
||||
## Troubleshooting
|
||||
|
||||
### "migrate" option failed
|
||||
|
||||
In some circumstances, when attempting to migrate your database schemas and Elasticsearch indexes, you might encounter an error similar to the following.
|
||||
|
||||
```
|
||||
14:42:40.630 [main] DEBUG org.flywaydb.core.FlywayExecutor - Memory usage:
|
||||
58 of 1024M
|
||||
"migrate" option failed : org.flywaydb.core.api.exception.FlywayValidateException:
|
||||
Validate failed: Migrations have failed validation
|
||||
Migration checksum mismatch for migration version 002
|
||||
-> Applied to database : -163374426
|
||||
-> Resolved locally : 326575949. Either revert the changes to the migration, or
|
||||
run repair to update the schema history.
|
||||
Need more flexibility with validation rules?
|
||||
Learn more: https://rd.gt/3AbJUZE
|
||||
```
|
||||
|
||||
If you encounter this error, run the following command to repair your schema.
|
||||
|
||||
```bash
|
||||
metadata, version metadata 0.5.0 from /Users/om/openmetadata-docker/env/lib/python3.8 (python 3.8)
|
||||
./bootstrap/bootstrap_storage.sh repair
|
||||
```
|
||||
|
||||
### 3. Check available versions
|
||||
|
||||
To confirm that there is a later version of `openmetadata-ingestion[docker]` available and identify the version you want to install, please run the following command.
|
||||
After running this command, you should see output similar to the following in response.
|
||||
|
||||
```
|
||||
pip3 install 'openmetadata-ingestion[docker]'==
|
||||
14:42:48.110 [main] INFO org.flywaydb.core.internal.command.DbRepair -
|
||||
Successfully repaired schema
|
||||
history table `openmetadata_db`.`DATABASE_CHANGE_LOG` (execution time 00:00.058s).
|
||||
14:42:48.123 [main] DEBUG org.flywaydb.core.FlywayExecutor -
|
||||
Memory usage: 58 of 1024M
|
||||
"repair" option successful
|
||||
```
|
||||
|
||||
Upon running this command, you should see output similar to the following.
|
||||
Once repair is successful, continue with the procedure above by rerunning the command in Step 5 and continuing from there.
|
||||
|
||||
```
|
||||
ERROR: Could not find a version that satisfies the requirement
|
||||
openmetadata-ingestion[docker]== (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4,
|
||||
0.3.0, 0.3.2, 0.4.0.dev0, 0.4.0.dev6, 0.4.0, 0.4.1.dev6, 0.4.1, 0.4.2.dev1, 0.4.2,
|
||||
0.4.2.1, 0.4.3.dev1, 0.4.3.dev2, 0.4.3.dev3, 0.4.3.dev4, 0.4.3, 0.4.4, 0.4.5, 0.4.7,
|
||||
0.4.8.dev0, 0.4.8.dev2, 0.4.8, 0.4.9, 0.4.10, 0.4.11, 0.5.0rc0, 0.5.0rc1, 0.5.0,
|
||||
0.5.1.dev0, 0.6.0.dev0, 0.7.0.dev1, 0.7.0.dev2, 0.7.0.dev3, 0.7.0.dev4)
|
||||
ERROR: No matching distribution found for openmetadata-ingestion[docker]==
|
||||
```
|
||||
|
||||
The error messages are expected. This is the accepted means of checking available versions for a Python module using `pip`.
|
||||
|
||||
The output provides a complete list of available versions and enables you to determine whether there are release versions later than the version you currently have installed. Release versions have the form `x.x.x`. Examples of release versions in the above output include, `0.2.0`, `0.4.2`, and `0.5.0`. 
|
||||
|
||||
From this output you can also find patch releases (e.g., `0.4.2.1`), release candidates (`0.5.0rc1`), and development releases (e.g., `0.7.0.dev4`). 
|
||||
|
||||
### 4. Stop your currently running deployment
|
||||
|
||||
Before upgrading, if you are currently running an OpenMetadata deployment, please stop the deployment by running the following command.
|
||||
|
||||
```bash
|
||||
metadata docker --stop
|
||||
```
|
||||
|
||||
### 5. Install the version of your choice
|
||||
|
||||
#### Option 1. Install the latest release version
|
||||
|
||||
You may install the latest release version by running the following command.
|
||||
|
||||
```bash
|
||||
pip3 install --upgrade 'openmetadata-ingestion[docker]'
|
||||
```
|
||||
|
||||
#### Option 2. Install a specific release, patch, or development version
|
||||
|
||||
You may install a specific version of `openmetadata-ingestion[docker]`by running the following command, specifying the version you want to install in place of `<version>`.
|
||||
|
||||
```bash
|
||||
pip3 install --upgrade 'openmetadata-ingestion[docker]'==<version>
|
||||
```
|
||||
|
||||
For example, if you want to install the `0.7.0.dev4` release, you would run the following command.
|
||||
|
||||
```bash
|
||||
pip3 install --upgrade 'openmetadata-ingestion[docker]'==0.7.0.dev4
|
||||
```
|
||||
|
||||
### 6. Restart your deployment
|
||||
|
||||
Once you have successfully installed your preferred version of `openmetadata-ingestion[docker]`, restart your deployment using the new version, by running the following command.
|
||||
|
||||
```bash
|
||||
metadata docker --start
|
||||
```
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user