mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-07-24 17:59:52 +00:00
Fix docs with upgrade instructions (#8985)
* Fix docs with upgrade instructions * Update openmetadata-docs/content/deployment/upgrade/bare-metal.md Co-authored-by: Teddy <teddy.crepineau@gmail.com> * Update openmetadata-docs/content/deployment/upgrade/bare-metal.md Co-authored-by: Teddy <teddy.crepineau@gmail.com> * Updated kubernetes and docker upgrade (#8987) * fix quotes * fix quotes * fix formatting * Update kubernetes.md * added missing quote * Add latest release details Co-authored-by: Teddy <teddy.crepineau@gmail.com>
This commit is contained in:
Sriharsha Chintalapani
GitHub
parent
0aa29b66ef
commit
28e4250001
@ -16,7 +16,7 @@ This guide assumes that you have an OpenMetadata deployment that you installed a
|
||||
|
||||
<Warning>
|
||||
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0121-release) before starting the upgrade process. We have introduced major stability and security changes as part of 0.12.1 OpenMetadata Release.
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade/versions/012-to-013) before starting the upgrade process.
|
||||
|
||||
</Warning>
|
||||
|
||||
@ -25,6 +25,7 @@ It is adviced to go through [openmetadata release notes](/deployment/upgrade#bre
|
||||
OpenMetadata release binaries are maintained as GitHub releases.
|
||||
|
||||
To download a specific release binary:
|
||||
|
||||
1. Visit [github.com/open-metadata/OpenMetadata/releases](https://github.com/open-metadata/OpenMetadata/releases). The latest
|
||||
release will be at the top of this page.
|
||||
2. Locate the Assets' section for the release you want to upgrade to.
|
||||
@ -81,13 +82,6 @@ MySQL) and index (in Elasticsearch).
|
||||
./bootstrap/bootstrap_storage.sh migrate-all
|
||||
```
|
||||
|
||||
<Note>
|
||||
|
||||
This step will be different in the 0.9 to 0.10 upgrade as it is a backward incompatible change.
|
||||
|
||||
Find specific instructions [here](/deployment/upgrade/versions/090-to-010).
|
||||
|
||||
</Note>
|
||||
|
||||
### 6. Restart the OpenMetadata server
|
||||
|
||||
@ -108,3 +102,15 @@ You will need to replace `<connectorname>` in the command below with the name of
|
||||
```commandline
|
||||
pip3 install --upgrade "openmetadata-ingestion[<connectorname>]"
|
||||
```
|
||||
|
||||
### Re-index all your metadata
|
||||
|
||||
Go to Settings -> Elasticsearch
|
||||
|
||||
<Image src="/images/deployment/upgrade/elasticsearch-re-index.png" alt="create-project" caption="Reindex"/>
|
||||
|
||||
Click on reindex all
|
||||
|
||||
in the dialog box choose Recreate Indexes to All
|
||||
|
||||
<Image src="/images/deployment/upgrade/reindex-ES.png" alt="create-project" caption="Reindex"/>
|
||||
|
||||
@ -14,11 +14,11 @@ You can find more details about Docker deployment [here](/deployment/docker)
|
||||
|
||||
</Note>
|
||||
|
||||
Below we have highlighted the steps needed to upgrade to the latest version with Docker. Make sure to also look [here](/deployment/upgrade/versions/011-to-012) for the specific details related to upgrading to 0.12
|
||||
Below we have highlighted the steps needed to upgrade to the latest version with Docker. Make sure to also look [here](/deployment/upgrade/versions/012-to-013) for the specific details related to upgrading to 0.13
|
||||
|
||||
<Warning>
|
||||
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0121-release) before starting the upgrade process. We have introduced major stability and security changes as part of 0.12.1 OpenMetadata Release.
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0130-release) before starting the upgrade process.
|
||||
|
||||
</Warning>
|
||||
|
||||
@ -41,9 +41,9 @@ Make a backup of your database. You can find the steps to follow [here](/deploym
|
||||
|
||||
### 3. Add Volumes and Publish Ports
|
||||
|
||||
Update the docker compose file you just downloaded to add persistent volume for MySQL. You can follow the steps [here](/deployment/docker/volumes#docker-volumes). If you had previously configured volumes for MySQL and the ingestion container make sure the 0.12.1 compose file is referencing the correct volume so that 0.11.5 data will be available when upgrading to 0.12.1.
|
||||
Update the docker compose file you just downloaded to add persistent volume for MySQL. You can follow the steps [here](/deployment/docker/volumes#docker-volumes). If you had previously configured volumes for MySQL and the ingestion container make sure the 0.13.0 compose file is referencing the correct volume so that 0.12.x data will be available when upgrading to 0.13.0.
|
||||
|
||||
**Note:** in 0.12.1 we are not publishing MySQL ports to the host container. If you wish to do so you will need to update the MySQL service in the compose file to have the following configuration
|
||||
**Note:** in 0.13.0 we are not publishing MySQL ports to the host container. If you wish to do so you will need to update the MySQL service in the compose file to have the following configuration
|
||||
```yaml
|
||||
services:
|
||||
mysql:
|
||||
@ -53,21 +53,10 @@ services:
|
||||
...
|
||||
```
|
||||
|
||||
### 4. Prepare Airflow upgrade to 2.3 (Only if upgrading from 0.11.x)
|
||||
|
||||
(If you are upgrading from 0.11.x to 0.12.x) In 0.12.x we are using airflow 2.3.3. There is a known issue when upgrading from airflow 2.1.x to 2.3.3. Make sure to look at the following notes [here](/deployment/upgrade/versions/011-to-012#airflow-version) for more details
|
||||
|
||||
### 5. Stop, Remove and Start your Containers
|
||||
Stop and remove (without deleting volumes) the already running containers. Then run the `docker compose up -d` command on the new compose file.
|
||||
|
||||
### 6. Update DAGs config
|
||||
#### 6.1 Update imports
|
||||
If you are using `openmetadata/ingestion` Docker image and you've upgraded to 0.12.x reusing volumes mounted to the `openmetadata/ingestion:0.11.5` container you will need to run the `metadata openmetadata-imports-migration` command inside the `openmetadata/ingestion:0.12.x` container. Indeed, `openmetadata-airflow-managed-apis` has been renamed to `openmetadata-managed-apis` and its import from `import openmetadata` to `import openmetadata_managed_apis`. If the import paths are not updated through the `metadata` command it will result in broken DAGs. By default the command will look for DAGs stored in `/opt/airflow/dags`. If you have changed where generated DAGs are stored you can specify the path to where your DAGs are stored `metadata openmetadata-imports-migration -d <path/to/folder>`
|
||||
|
||||
#### 6.2 Update config file path
|
||||
If you are using `openmetadata/ingestion` Docker image in 0.12.1 and migrated either from 0.12.0 or 0.11.x `dags` and `dag_generated_config` have changed path. Dags and generated config files are now stored respectively in `/opt/airflow/dags` and `/opt/airflow/dag_generated_config`. You will need to run the `metadata openmetadata-imports-migration` command with the `--change-config-file-path` inside the `openmetadata/ingestion:0.12.x` container to make sure DAGs workflow are pointing to the correct config file.
|
||||
|
||||
You can run 6.1 and 6.2 together using the following command `metadata openmetadata-imports-migration --change-config-file-path`.
|
||||
|
||||
### Troubleshooting
|
||||
#### Permission Denied when running `metadata openmetadata-imports-migration`
|
||||
|
||||
@ -8,17 +8,32 @@ slug: /deployment/upgrade
|
||||
## Releases
|
||||
|
||||
OpenMetadata community will be doing feature releases and stable releases.
|
||||
|
||||
- Feature releases are to upgrade your sandbox or POCs to give feedback to the community and any potential bugs that the community needs to fix.
|
||||
- Stable releases are to upgrade your production environments and share it with your users.
|
||||
|
||||
## 0.12.1 - Stable release
|
||||
|
||||
OpenMetadata 0.12.1 is a stable release. For Release notes please check here
|
||||
Join our slack https://slack.open-metadata.org provide your feedback and help community to get to the stable 0.12.1 release
|
||||
|
||||
## Breaking Changes from 0.12.1 Release
|
||||
## 0.13.0 - Feature Release
|
||||
|
||||
OpenMetadata Release 0.12.1 introduces below breaking changes -
|
||||
OpenMetadata 0.13.0 is a **feature release**.
|
||||
|
||||
**Don't upgrade your production with 0.13.0 feature release**
|
||||
|
||||
Explore 0.13.0 by following up [Deployment guides](https://docs.open-metadata.org/deployment) and please give us any feedback on our [community slack](https://slack.open-metadata.org)
|
||||
|
||||
|
||||
## 0.12.3 - Stable release
|
||||
|
||||
OpenMetadata 0.12.3 is a stable release. Please check the [release notes](https://github.com/open-metadata/OpenMetadata/releases/tag/0.12.3-release)
|
||||
|
||||
If you are upgrading production this is the recommended version to upgrade.
|
||||
|
||||
|
||||
|
||||
|
||||
## Breaking Changes from 0.13.0 Feature Release
|
||||
## Breaking Changes from 0.12.x Stable Release
|
||||
|
||||
OpenMetadata Release 0.12.x introduces below breaking changes -
|
||||
|
||||
### Change of OpenMetadata Service Namespace
|
||||
|
||||
|
||||
@ -19,11 +19,11 @@ We also assume that your helm chart release names are `openmetadata` and `openme
|
||||
|
||||
<Warning>
|
||||
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0121-release) before starting the upgrade process. We have introduced major stability and security changes as part of 0.12.1 OpenMetadata Release.
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0130-release)
|
||||
|
||||
</Warning>
|
||||
|
||||
Below document is valid for upgrading Helm Charts from **0.11.5 to 0.12.X**.
|
||||
Below document is valid for upgrading Helm Charts from **0.12.x to 0.13.0**.
|
||||
|
||||
### Back up metadata
|
||||
|
||||
@ -84,16 +84,7 @@ open-metadata/openmetadata-dependencies 0.0.34 0.11.4 Helm Dependenc
|
||||
```
|
||||
|
||||
## Upgrade OpenMetadata Dependencies
|
||||
|
||||
|
||||
### Step 1: Upgrade Airfow
|
||||
|
||||
We have **upgraded the Airflow version from 2.1.4 to 2.3.3** with OpenMetadata `0.12.X` releases.
|
||||
|
||||
Before you start upgrading OpenMetadata Dependencies, you must follow airflow migration docs [here](/deployment/upgrade/bare-metal#upgrade-ingestion-container) to upgrade Airflow.
|
||||
|
||||
|
||||
### Step 2: Upgrade OpenMetadata Dependencies with the below command
|
||||
### Step 1: Upgrade OpenMetadata Dependencies with the below command
|
||||
|
||||
```commandline
|
||||
helm upgrade openmetadata-dependencies open-metadata/openmetadata-dependencies
|
||||
@ -104,14 +95,14 @@ The above command uses configurations defined [here](https://raw.githubuserconte
|
||||
You can modify any configuration and deploy by passing your own `values.yaml`.
|
||||
|
||||
|
||||
### Step 3: Troubleshooting
|
||||
### Step 2: Troubleshooting
|
||||
|
||||
If your helm upgrade fails with the below command result -
|
||||
```
|
||||
Error: UPGRADE FAILED: cannot patch "mysql" with kind StatefulSet: StatefulSet.apps "mysql" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
|
||||
```
|
||||
|
||||
This is probably because with `0.12.1`, we have **default size of mysql persistence set to 50Gi**.
|
||||
This is probably because with `0.13.0`, we have **default size of mysql persistence set to 50Gi**.
|
||||
|
||||
Kubernetes does not allow changes to Persistent volume with helm upgrades.
|
||||
|
||||
@ -141,20 +132,13 @@ helm upgrade openmetadata open-metadata/openmetadata
|
||||
|
||||
You might need to pass your own `values.yaml` with the `--values` flag
|
||||
|
||||
## Reindex ElasticSearch
|
||||
|
||||
We have added a conditional suggestion mapping for all of the elasticsearch indexes. This may require re-indexing. With 0.12.1 its never been easier to index your metadata
|
||||
|
||||
### Go to Settings -> Event Publishers -> ElasticSearch
|
||||
|
||||
### Re-index all your metadata
|
||||
Go to Settings -> Elasticsearch
|
||||
<Image src="/images/deployment/upgrade/elasticsearch-re-index.png" alt="create-project" caption="Create a New Project"/>
|
||||
|
||||
### Make sure you select "Recreate Indexes"
|
||||
|
||||
Click on the "Recreate Indexes" lable and click "Re Index All"
|
||||
|
||||
|
||||
|
||||
Click on reindex all
|
||||
in the dialog box choose Recreate Indexes to All
|
||||
<Image src="/images/deployment/upgrade/reindex-ES.png" alt="create-project" caption="Reindex"/>
|
||||
|
||||
## Troubleshooting for 0.12 Release
|
||||
|
||||
|
||||
@ -28,7 +28,7 @@ You can find further information about specific version upgrades in the followin
|
||||
color="violet-70"
|
||||
icon="10k"
|
||||
bold="Upgrade 0.12"
|
||||
href="/deployment/upgrade/versions/010-to-011"
|
||||
href="/deployment/upgrade/versions/011-to-012"
|
||||
>
|
||||
Upgrade from 0.11 to 0.12 inplace.
|
||||
</InlineCallout>
|
||||
@ -36,7 +36,7 @@ You can find further information about specific version upgrades in the followin
|
||||
color="violet-70"
|
||||
icon="10k"
|
||||
bold="Upgrade 0.13"
|
||||
href="/deployment/upgrade/versions/010-to-011"
|
||||
href="/deployment/upgrade/versions/012-to-013"
|
||||
>
|
||||
Upgrade from 0.12 to 0.13 inplace.
|
||||
</InlineCallout>
|
||||
|
||||
@ -3,6 +3,43 @@ title: Latest Release
|
||||
slug: /overview/latest-release
|
||||
---
|
||||
|
||||
|
||||
# 0.13.0 Release - Nov 23rd 2022 🎉
|
||||
This is our stable release and we recommend community to upgrade to the 0.12.1 release
|
||||
|
||||
## Data Insights
|
||||
|
||||
Data insights aims to provide a single pane view of all the key metrics to best reflect the state of your data. Admins can define the Key Performance Indicators(KPIs) and set goals within OpenMetadata to work towards better documentation, owernship, tiering
|
||||
|
||||
|
||||
## Lienage Traceability
|
||||
|
||||
Lineage UI has been transformed to enhance user experience. Users can get a holistic view of an entity from the Lineage tab. When a entity is selected, the UI displays end-to-end lineage traceability for the table and column levels, Just search for an entity and expand the graph to unfold lineage. It'll display the upstream/downstream of nodes
|
||||
|
||||
## Search Relevancy and Advanced Search
|
||||
|
||||
- Default operator for search queries is chaned to "OR"
|
||||
- We are ranking search results based on weekly usage metric. In searches similarly ranked tables with higher weekly usage will be shown first in the results
|
||||
- An advanced search feature has been introduced in the 0.13.0 release, which helps users discover assets quickly with a syntax editor on AND/OR conditions
|
||||
|
||||
## Data Lake Profiler
|
||||
|
||||
- Users can now create and deploy profiling workflows for Data Lake connector which supports AWS S3 and GCS
|
||||
|
||||
## Security
|
||||
- LDAP login is now supported
|
||||
- Bot accounts now have policies
|
||||
|
||||
## Connectors
|
||||
New connectors are an essential part of every release in OpenMetadata. We are introducing four new connectors in this release:
|
||||
|
||||
- Domo is a cloud-based dashboard service. The Domo Business Cloud is a low-code data app platform that takes the power of BI to the next level by combining all your data and putting it to work across any business process or workflow. OpenMetadata supports Domo as a Database, Dashboard, as well as a Pipeline service.
|
||||
- AWS SageMaker is a fully managed machine learning service, where data scientists and developers can quickly and easily build and train machine learning models, and then directly deploy them into a production-ready hosted environment.
|
||||
- AWS Kinesis is a cloud-based messaging service that allows real-time processing of streaming large amounts of data per second.
|
||||
- AWS QuickSight is a cloud-scale business intelligence (BI) service that allows everyone in the organization to understand the data by asking questions in natural language, exploring through interactive dashboards, or automatically looking for patterns and outliers powered by machine learning.
|
||||
|
||||
|
||||
|
||||
# 0.12.1 Release - Oct 3rd 2022 🎉
|
||||
This is our stable release and we recommend community to upgrade to the 0.12.1 release
|
||||
|
||||
@ -12,7 +49,7 @@ This is our stable release and we recommend community to upgrade to the 0.12.1 r
|
||||
- Email notifications for forgotten password and new user signed up
|
||||
- Admin can add new users and send an email
|
||||
|
||||
## ElasticSearch full re-index through Uigur
|
||||
## ElasticSearch full re-index through UI
|
||||
|
||||
- Now admins can full re-index elasticsearch through the UI itself
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 313 KiB |
BIN
openmetadata-docs/images/deployment/upgrade/reindex-ES.png
Normal file
BIN
openmetadata-docs/images/deployment/upgrade/reindex-ES.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 333 KiB |
Loading…
x
Reference in New Issue
Block a user