mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2026-01-05 20:17:07 +00:00
0.13 to 1.00 docs changes (#11236)
* 0.13 to 1.00 changes * add superset changes * dbt gcs yaml fix --------- Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
This commit is contained in:
Mayur Singal
GitHub
parent
49d6a0a0e6
commit
c920c9afa3
@ -29,19 +29,20 @@ source:
|
||||
# dbtObjectPrefix: "dbt/"
|
||||
# # For GCS
|
||||
# dbtSecurityConfig: # These are modeled after all GCS credentials
|
||||
# type: My Type
|
||||
# projectId: project ID
|
||||
# privateKeyId: us-east-2
|
||||
# privateKey: |
|
||||
# -----BEGIN PRIVATE KEY-----
|
||||
# Super secret key
|
||||
# -----END PRIVATE KEY-----
|
||||
# clientEmail: client@mail.com
|
||||
# clientId: 1234
|
||||
# authUri: https://accounts.google.com/o/oauth2/auth (default)
|
||||
# tokenUri: https://oauth2.googleapis.com/token (default)
|
||||
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
|
||||
# clientX509CertUrl: https://cert.url (URI)
|
||||
# gcsConfig:
|
||||
# type: My Type
|
||||
# projectId: project ID
|
||||
# privateKeyId: us-east-2
|
||||
# privateKey: |
|
||||
# -----BEGIN PRIVATE KEY-----
|
||||
# Super secret key
|
||||
# -----END PRIVATE KEY-----
|
||||
# clientEmail: client@mail.com
|
||||
# clientId: 1234
|
||||
# authUri: https://accounts.google.com/o/oauth2/auth (default)
|
||||
# tokenUri: https://oauth2.googleapis.com/token (default)
|
||||
# authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs (default)
|
||||
# clientX509CertUrl: https://cert.url (URI)
|
||||
# dbtPrefixConfig:
|
||||
# dbtBucketName: bucket
|
||||
# dbtObjectPrefix: "dbt/"
|
||||
|
||||
@ -16,7 +16,7 @@ This guide assumes that you have an OpenMetadata deployment that you installed a
|
||||
|
||||
{% note render="Warning" description="Warning" %}
|
||||
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade/versions/012-to-013) before starting the upgrade process.
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade/versions/013-to-100) before starting the upgrade process.
|
||||
|
||||
{% /note %}
|
||||
|
||||
@ -33,9 +33,9 @@ version to have all the improvements shipped in the CLI.
|
||||
2. Create a virtual environment to install an upgraded `metadata` version to run the backup command:
|
||||
1. `python -m venv venv`
|
||||
2. `source venv/bin/activate`
|
||||
3. `pip install openmetadata-ingestion~=0.13.2`
|
||||
3. `pip install openmetadata-ingestion~=1.0.0`
|
||||
3. Validate the installed `metadata` version with `python -m metadata --version`, which should tell us that we are
|
||||
indeed at 0.13.2. Notice the `python -m metadata` vs. `metadata`.
|
||||
indeed at 1.0.0. Notice the `python -m metadata` vs. `metadata`.
|
||||
4. Run the backup using the updated `metadata` CLI:
|
||||
```
|
||||
python -m metadata backup -u openmetadata_user -p openmetadata_password -H mysql -d openmetadata_db --port 3306
|
||||
@ -85,7 +85,7 @@ command.
|
||||
|
||||
```commandline
|
||||
|
||||
cd openmetadata-0.13.2
|
||||
cd openmetadata-1.0.0
|
||||
|
||||
```
|
||||
|
||||
|
||||
@ -22,21 +22,21 @@ It is advised to go through [openmetadata release notes](/deployment/upgrade#bre
|
||||
|
||||
{% /note %}
|
||||
|
||||
## Upgrade from 0.13.1 to 0.13.2
|
||||
## Upgrade from 0.13 to 1.0.0
|
||||
|
||||
Upgrading from 0.13.1 to 0.13.2 can be done easily as version 0.13.1 compose files already provided volumes for the databases.
|
||||
Your production deployment should go from stable version to stable version. This translated to moving from 0.13 to 1.0.0 to get the latest stable OpenMetadata release.
|
||||
|
||||
Let's go through the required steps:
|
||||
|
||||
### 1. Backup 0.13.1 data
|
||||
### 1. Backup 0.13 data
|
||||
|
||||
1. Make sure your instance is connected to the Database server
|
||||
2. Create a virtual environment to install an upgraded `metadata` version to run the backup command:
|
||||
1. `python -m venv venv`
|
||||
2. `source venv/bin/activate`
|
||||
3. `pip install openmetadata-ingestion~=0.13.2`
|
||||
3. `pip install openmetadata-ingestion~=1.0.0`
|
||||
3. Validate the installed `metadata` version with `python -m metadata --version`, which should tell us that we are
|
||||
indeed at 0.13.2. Notice the `python -m metadata` vs. `metadata`.
|
||||
indeed at 1.0.0. Notice the `python -m metadata` vs. `metadata`.
|
||||
4. Run the backup using the updated `metadata` CLI:
|
||||
```
|
||||
python -m metadata backup -u openmetadata_user -p openmetadata_password -H mysql -d openmetadata_db --port 3306
|
||||
@ -52,110 +52,22 @@ Let's go through the required steps:
|
||||
### 2. Update the docker compose file
|
||||
|
||||
1. Stop the running compose deployment with `docker compose down`.
|
||||
2. On the compose file we ran the 0.13.1 version, update the image tag in the `ingestion` and
|
||||
`openmetadata-server` to 0.13.2. E.g., `image: openmetadata/server:0.13.2`.
|
||||
2. On the compose file we ran the 0.13 version, update the image tag in the `ingestion` and
|
||||
`openmetadata-server` to 1.0.0. E.g., `image: openmetadata/server:0.13.2`.
|
||||
3. Start the updated compose file.
|
||||
4. Run the reindex from the UI.
|
||||
|
||||
## Upgrade from 0.12.3 to 0.13.2
|
||||
|
||||
Your production deployment should go from stable version to stable version. This translated to moving from 0.12.3 to 0.13.2 to
|
||||
get the latest stable OpenMetadata release.
|
||||
|
||||
Database volumes were just introduced in the 0.13.0 feature release. This means that when moving from 0.12.3 to 0.13.2, we'll
|
||||
need to do some extra steps to ensure that all your metadata is safe and sound. In this section, we'll guide you step by step on
|
||||
what you'll need to do to:
|
||||
|
||||
1. Backup your data,
|
||||
2. Add volumes and port mapping to the databases and restore your data,
|
||||
3. Start OpenMetadata with the 0.13.2 release.
|
||||
|
||||
These steps are based on MySQL, but are analogous for Postgres.
|
||||
|
||||
Please, validate the process in your development or staging environment first before going into PROD.
|
||||
|
||||
|
||||
{% note %}
|
||||
|
||||
This guide is specific to installations using the default docker compose file we shipped on the 0.12.3 release. The
|
||||
whole process is about making sure that all the data will be properly backed up considering that there is no
|
||||
port mapping or volumes available.
|
||||
|
||||
If your docker compose already had volumes for the database, then upgrading the version of OpenMetadata and the
|
||||
Ingestion container should be good to go (as shown in step 4 below).
|
||||
|
||||
{% /note %}
|
||||
|
||||
### 1. Backup 0.12.3 data
|
||||
|
||||
1. Get inside the `openmetadata_ingestion` container with `docker exec -it openmetadata_ingestion bash`.
|
||||
2. Create a virtual environment to install an upgraded `metadata` version to run the backup command:
|
||||
1. `python -m venv venv`
|
||||
2. `source venv/bin/activate`
|
||||
3. `PIP_USER=false pip install openmetadata-ingestion~=0.13.2`
|
||||
3. Validate the installed `metadata` version with `python -m metadata --version`, which should tell us that we are
|
||||
indeed at 0.13.2. Notice the `python -m metadata` vs. `metadata`. As we're inside the container, the raw metadata
|
||||
command will use the system-wide one, which is at 0.12.3.
|
||||
4. Run the backup using the updated `metadata` CLI:
|
||||
```
|
||||
python -m metadata backup -u openmetadata_user -p openmetadata_password -H mysql -d openmetadata_db --port 3306
|
||||
```
|
||||
if using Postgres:
|
||||
```
|
||||
python -m metadata backup -u openmetadata_user -p openmetadata_password -H postgresql -d openmetadata_db --port 5432 -s public
|
||||
```
|
||||
5. From outside the container, copy the backup file to safety. For example `docker cp openmetadata_ingestion:/opt/airflow/openmetadata_202212201528_backup.sql .`
|
||||
In our case, the backup file was named `openmetadata_202212201528_backup.sql`. You can copy the name from the backup
|
||||
command output.
|
||||
|
||||
### 2. Add volumes and port mapping
|
||||
|
||||
This step is only required if you have not done that yet.
|
||||
|
||||
1. Stop docker compose
|
||||
2. Update the `mysql` service in the compose file so that it looks like:
|
||||
```
|
||||
mysql:
|
||||
container_name: openmetadata_mysql
|
||||
image: openmetadata/db:0.13.2
|
||||
restart: always
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: password
|
||||
expose:
|
||||
- 3306
|
||||
ports:
|
||||
- "3306:3306"
|
||||
volumes:
|
||||
- ./docker-volume/db-data:/var/lib/mysql
|
||||
networks:
|
||||
- app_net
|
||||
healthcheck:
|
||||
test: mysql --user=root --password=$$MYSQL_ROOT_PASSWORD --silent --execute "use openmetadata_db"
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 10
|
||||
```
|
||||
Note how we added the `ports` and `volumes` sections. If you're using postgres, these would look like:
|
||||
```
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- ./docker-volume/db-data-postgres:/var/lib/postgresql/data
|
||||
```
|
||||
3. Start docker compose with `docker compose up` on the updated file.
|
||||
1. This will run a clean database instance with ports and volumes.
|
||||
2. The OpenMetadata server will run the 0.12.3 migrations to populate the starting tables.
|
||||
|
||||
### 3. Restore the backup
|
||||
|
||||
We will now use the backup generated in the previous step to repopulate the database.
|
||||
|
||||
1. On your laptop/VM, prepare a virtual environment and install `openmetadata-ingestion==0.13.2`. It is important
|
||||
1. On your laptop/VM, prepare a virtual environment and install `openmetadata-ingestion==1.0.0`. It is important
|
||||
that the instance has access to the database.
|
||||
1. `python -m venv venv`
|
||||
2. `source venv/bin/activate`
|
||||
3. `pip install openmetadata-ingestion~=0.13.2`
|
||||
2. Validate the metadata version with `metadata --version`. it should be 0.13.2.
|
||||
3. `pip install openmetadata-ingestion~=1.0.0`
|
||||
2. Validate the metadata version with `metadata --version`. it should be 1.0.0.
|
||||
3. Run the restore with your file name
|
||||
```
|
||||
metadata restore -H localhost -u openmetadata_user -p openmetadata_password -d openmetadata_db --port 3306 --input openmetadata_202212201528_backup.sql
|
||||
@ -169,11 +81,11 @@ We will now use the backup generated in the previous step to repopulate the data
|
||||
### 4. Upgrade OpenMetadata versions
|
||||
|
||||
1. On the compose file we previously updated the database settings, update the image tag in the `ingestion` and
|
||||
`openmetadata-server` to 0.13.2. E.g., `image: openmetadata/server:0.13.2`.
|
||||
`openmetadata-server` to 1.0.0. E.g., `image: openmetadata/server:1.0.0`.
|
||||
2. Start the updated compose file.
|
||||
3. Run the reindex from the UI.
|
||||
|
||||
Now you should still have all your data with OpenMetadata version 0.13.2.
|
||||
Now you should still have all your data with OpenMetadata version 1.0.0.
|
||||
|
||||
## Upgrade ingestion patch versions
|
||||
|
||||
|
||||
@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Upgrade 0.13 to 1.0
|
||||
slug: /deployment/upgrade/versions/013-to-100
|
||||
---
|
||||
|
||||
# Upgrade from 0.13 to 1.0
|
||||
|
||||
Upgrading from 0.13 to 1.0 can be done directly on your instances. This page will list few general details you should take into consideration when running the upgrade.
|
||||
|
||||
## 1.0.0 Highlights
|
||||
|
||||
### Service Connection Updates
|
||||
|
||||
- Postgres:
|
||||
- The database field for `Postgres` connector has been made a required field as initial database is required to connect to Postgres.
|
||||
- `Ingest All Database` field has been added since earlier when database field was empty we were ingesting all the databases but since now we have made the database field required you need to enable `Ingest All Database` flag in case you want to ingest all the available database in OpenMetadata.
|
||||
|
||||
- BigQuery:
|
||||
- Removed: `partitionQueryDuration`, `partitionQuery` & `partitionField`
|
||||
- since these fields are not used during the ingestion and has already being moved to profiler hence we have removed these fields.
|
||||
|
||||
- Salesforce:
|
||||
- Removed: `scheme`, `hostPort`, `supportsProfiler` & `supportsQueryComment`
|
||||
- These fields are related to a SQLAlchemy based connector and since `Salesforce` is based on APIs and not SQLAlchemy, these fields has been removed
|
||||
|
||||
- Superset:
|
||||
- Removed: `connectionOptions`
|
||||
- This field was not being used anywhere, hence removed.
|
||||
@ -22,4 +22,11 @@ You can find further information about specific version upgrades in the followin
|
||||
href="/deployment/upgrade/versions/012-to-013" %}
|
||||
Upgrade from 0.12 to 0.13 inplace.
|
||||
{% /inlineCallout %}
|
||||
{% /inlineCalloutContainer %}
|
||||
{% inlineCallout
|
||||
color="violet-70"
|
||||
icon="10k"
|
||||
bold="Upgrade 1.0"
|
||||
href="/deployment/upgrade/versions/013-to-100" %}
|
||||
Upgrade from 0.13 to 1.0 inplace.
|
||||
{% /inlineCallout %}
|
||||
{% /inlineCalloutContainer %}
|
||||
@ -176,6 +176,8 @@ site_menu:
|
||||
url: /deployment/upgrade/versions/011-to-012
|
||||
- category: Deployment / Upgrade OpenMetadata / Upgrade Version Instructions / 0.12 to 0.13
|
||||
url: /deployment/upgrade/versions/012-to-013
|
||||
- category: Deployment / Upgrade OpenMetadata / Upgrade Version Instructions / 0.13 to 1.0
|
||||
url: /deployment/upgrade/versions/013-to-100
|
||||
|
||||
- category: Deployment / Backup & Restore Metadata
|
||||
url: /deployment/backup-restore-metadata
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user