mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-07-18 14:39:44 +00:00
156 lines
5.8 KiB
Markdown
156 lines
5.8 KiB
Markdown
![]() |
---
|
||
|
title: Upgrade 0.9 to 0.10
|
||
|
slug: /deployment/upgrade/versions/090-to-010
|
||
|
---
|
||
|
|
||
|
# Upgrade from 0.9 to 0.10
|
||
|
|
||
|
**The 0.10 Release consists of backward-incompatible changes. We do not support database migration from the 0.9.0 release.
|
||
|
Please follow the steps carefully and backup your database before proceeding. 0.10.0 installations require brand new
|
||
|
installation, and we have a migration tool to transfer all your entity descriptions, tags, owners, etc. to the 0.10.0
|
||
|
release**
|
||
|
|
||
|
Please reach out to us on [Slack](https://slack.open-metadata.org), we can schedule a zoom session to help you upgrade your
|
||
|
production instance.
|
||
|
|
||
|
## Requirements
|
||
|
|
||
|
Update the `openmetadata-ingestion` client to the latest available version.
|
||
|
|
||
|
Make sure that you [backed up](/deployment/upgrade/backup-metadata) your metadata.
|
||
|
|
||
|
<Note>
|
||
|
|
||
|
To migrate the metadata from the 0.9 to the 0.10 version you need to ingest all the connectors in the 0.10 version with
|
||
|
the same service name as done in the 0.9 version. For example, if you have ingested snowflake connector in 0.9 version
|
||
|
with service name `Snowflake_Prod` then in the 0.10 version, you need to ingest the snowflake connector again with the
|
||
|
name `Snowflake_Prod`.
|
||
|
|
||
|
</Note>
|
||
|
|
||
|
## Metadata Migration
|
||
|
|
||
|
All connectors are now defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/entity/services/connections/metadata/openMetadataConnection.json)
|
||
|
you can find the structure to create a connection to Snowflake. In order to create and run a Metadata Ingestion workflow,
|
||
|
we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed,
|
||
|
and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/metadataIngestion/workflow.json).
|
||
|
|
||
|
## 1. Setup your 0.10.1 installation
|
||
|
|
||
|
1. OpenMetadata 0.9, keep it running as it is with its own MySQL and ES
|
||
|
2. Install OpenMetadata 0.10.1, make it completely isolated from your 0.9 install new MySQL DB and other ES indexes
|
||
|
3. Install 0.10.1 connectors
|
||
|
4. Run ingestion of all your entities using the new connector. Keep the service names the same as your 0.9
|
||
|
installations. Once all the entities are ingested into OpenMetadata 0.10.1. For example, if you have ingested
|
||
|
snowflake connector in 0.9 version with service name Snowflake_Prod then in the 0.10 version, you need to ingest the
|
||
|
snowflake connector again with the name `Snowflake_Prod`.
|
||
|
5. Run migration using the newly installed connector
|
||
|
|
||
|
## 2. Define the YAML Config
|
||
|
|
||
|
This is a sample config:
|
||
|
|
||
|
```yaml
|
||
|
source:
|
||
|
type: migrate
|
||
|
serviceName: local_metadata
|
||
|
serviceConnection:
|
||
|
config:
|
||
|
type: OpenMetadata
|
||
|
hostPort: http://<hostport of 0.9.0 Openmetadata Server>/api
|
||
|
authProvider: no-auth
|
||
|
includeTables: true
|
||
|
includeUsers: true
|
||
|
includeTopics: true
|
||
|
includePipelines: true
|
||
|
includeTags: true
|
||
|
includeGlossaryTerms: true
|
||
|
includeMessagingServices: true
|
||
|
includeDatabaseServices: true
|
||
|
includePipelineServices: true
|
||
|
enableVersionValidation: false
|
||
|
limitRecords: 100000
|
||
|
sourceConfig:
|
||
|
config:
|
||
|
enableDataProfiler: false
|
||
|
stage:
|
||
|
type: migrate
|
||
|
config:
|
||
|
dirPath: <Directory Path to store data>
|
||
|
bulkSink:
|
||
|
type: migrate
|
||
|
config:
|
||
|
dirPath: <Directory Path to store data>
|
||
|
workflowConfig:
|
||
|
openMetadataServerConfig:
|
||
|
hostPort: http://<hostport of 0.10.x Openmetadata Server>/api
|
||
|
authProvider: no-auth
|
||
|
```
|
||
|
|
||
|
### Source Configuration - Service Connection
|
||
|
|
||
|
You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/entity/services/connections/database/clickhouseConnection.json).
|
||
|
- `hostPort`: Enter the OpenMetadata 0.9 Server Config. Must include API end point ex: `http://localhost:8585/api`
|
||
|
- `includeTables`: Include tables for migration
|
||
|
- `includeUsers`: Include users for migration
|
||
|
- `includeTopics`: Include topics for migration
|
||
|
- `includePipelines`: Include pipelines for migration
|
||
|
- `includeGlossaryTerms`: include glossary terms for migration
|
||
|
- `includeTags`: Include tags for migration.
|
||
|
- `includeMessagingServices`: Include messaging services for migration.
|
||
|
- `includeDatabaseServices`: Include database services for migration.
|
||
|
- `enableVersionValidation`: Enable the server OpenMetadata and client OpenMetadata version validation. For migrating the metadata this has to be set to `false`
|
||
|
- `limitRecords`: Limit the number records that is fetched by 0.9 OpenMetadata API (1 to 1000000, default = 10).
|
||
|
|
||
|
### Stage Configuration
|
||
|
|
||
|
```yaml
|
||
|
stage:
|
||
|
type: migrate
|
||
|
config:
|
||
|
dirPath: <Directory Path to store data>
|
||
|
```
|
||
|
|
||
|
The data while migrating from the 0.9 to 0.10 version needs to be stored in a file directory. in the dir Path field
|
||
|
enter a valid file directory path to store the metadata.
|
||
|
|
||
|
### BulkSink Configuration
|
||
|
|
||
|
```yaml
|
||
|
bulkSink:
|
||
|
type: migrate
|
||
|
config:
|
||
|
dirPath: <Directory Path to store data>
|
||
|
```
|
||
|
|
||
|
The `dirPath` needs to be set as the same passed in stage config, which contains the file directory path where the
|
||
|
metadata is stored.
|
||
|
|
||
|
### Workflow Configuration
|
||
|
|
||
|
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your
|
||
|
OpenMetadata installation.
|
||
|
|
||
|
For a simple, local installation using our docker containers, this looks like:
|
||
|
|
||
|
```yaml
|
||
|
workflowConfig:
|
||
|
openMetadataServerConfig:
|
||
|
hostPort: http://<hostport of 0.10.x Openmetadata Server>/api
|
||
|
authProvider: no-auth
|
||
|
```
|
||
|
|
||
|
<Note>
|
||
|
|
||
|
In `openMetadataServerConfig` you need to pass details about the 0.10 OpenMetadata Server.
|
||
|
|
||
|
</Note>
|
||
|
|
||
|
## 3. Run with the CLI
|
||
|
|
||
|
First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
|
||
|
|
||
|
```commandline
|
||
|
metadata ingest -c <path-to-yaml>
|
||
|
```
|