mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-12-24 05:58:31 +00:00
Docs - Upgrade and breaking changes for 0.13.2 (#9882)
* Prep breaking changes docs * Prep breaking changes docs * highlights * Superset Changes * Metabase and Domo * Add missing task_id param * Update upgrade docs for 0.13.2 * Version alignments * What's new * doc: Added breaking change around entityName and entityLink * Update openmetadata-docs/content/overview/roadmap/index.md --------- Co-authored-by: ulixius9 <mayursingal9@gmail.com> Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
This commit is contained in:
Pere Miquel Brull
GitHub
parent
9e6a91d349
commit
23ba96a271
@ -50,11 +50,11 @@ If you want to use the connector from the UI, the Python environment running the
|
||||
the new code you just created. For example, if running via Docker, the `openmetadata-ingestion` image should be
|
||||
aware of your new package.
|
||||
|
||||
We will be running the demo against the OpenMetadata version `0.12.2`, therefore, our Dockerfile looks like:
|
||||
We will be running the demo against the OpenMetadata version `0.13.2`, therefore, our Dockerfile looks like:
|
||||
|
||||
```Dockerfile
|
||||
# Base image from the right version
|
||||
FROM openmetadata/ingestion:0.13.1
|
||||
FROM openmetadata/ingestion:0.13.2
|
||||
|
||||
# Let's use the same workdir as the ingestion image
|
||||
WORKDIR ingestion
|
||||
|
||||
@ -121,7 +121,7 @@ with models.DAG(
|
||||
) as dag:
|
||||
DockerOperator(
|
||||
command="python main.py",
|
||||
image="openmetadata/ingestion-base:0.13.1",
|
||||
image="openmetadata/ingestion-base:0.13.2",
|
||||
environment={"config": config, "pipelineType": "metadata"},
|
||||
docker_url="unix://var/run/docker.sock", # To allow to start Docker. Needs chmod 666 permissions
|
||||
tty=True,
|
||||
@ -138,7 +138,7 @@ Make sure to tune out the DAG configurations (`schedule_interval`, `start_date`,
|
||||
|
||||
</Note>
|
||||
|
||||
Note that the example uses the image `openmetadata/ingestion-base:0.13.1`. Update that accordingly for higher version
|
||||
Note that the example uses the image `openmetadata/ingestion-base:0.13.2`. Update that accordingly for higher version
|
||||
once they are released. Also, the image version should be aligned with your OpenMetadata server version to avoid
|
||||
incompatibilities.
|
||||
|
||||
|
||||
@ -131,8 +131,8 @@ We will now describe the steps, following the official AWS documentation.
|
||||
### 1. Create an ECS Cluster
|
||||
|
||||
- The cluster just needs a task to run in `FARGATE` mode.
|
||||
- The required image is `registry.hub.docker.com/openmetadata/ingestion-base:x.y.z`
|
||||
- The same logic as above applies. The `x.y.z` version needs to match the server version. For example, `registry.hub.docker.com/openmetadata/ingestion-base:0.13.1`
|
||||
- The required image is `docker.getcollate.io/openmetadata/ingestion-base:x.y.z`
|
||||
- The same logic as above applies. The `x.y.z` version needs to match the server version. For example, `docker.getcollate.io/openmetadata/ingestion-base:0.13.2`
|
||||
|
||||
We have tested this process with a Task Memory of 512MB and Task CPU (unit) of 256. This can be tuned depending on the amount of metadata that needs to be ingested.
|
||||
|
||||
|
||||
@ -223,7 +223,7 @@ with models.DAG(
|
||||
task_id="ingest",
|
||||
name="ingest",
|
||||
cmds=["python", "main.py"],
|
||||
image="openmetadata/ingestion-base:0.13.1",
|
||||
image="openmetadata/ingestion-base:0.13.2",
|
||||
namespace='default',
|
||||
env_vars={"config": config, "pipelineType": "metadata"},
|
||||
dag=dag,
|
||||
@ -238,7 +238,7 @@ You can name the task as you want (`task_id` and `name`). The important points h
|
||||
be changed, and the `env_vars`. The `main.py` script that gets shipped within the image will load the env vars
|
||||
as they are shown, so only modify the content of the config YAML, but not this dictionary.
|
||||
|
||||
Note that the example uses the image `openmetadata/ingestion-base:0.13.1`. Update that accordingly for higher version
|
||||
Note that the example uses the image `openmetadata/ingestion-base:0.13.2`. Update that accordingly for higher version
|
||||
once they are released. Also, the image version should be aligned with your OpenMetadata server version to avoid
|
||||
incompatibilities.
|
||||
|
||||
@ -247,7 +247,7 @@ KubernetesPodOperator(
|
||||
task_id="ingest",
|
||||
name="ingest",
|
||||
cmds=["python", "main.py"],
|
||||
image="openmetadata/ingestion-base:0.13.1",
|
||||
image="openmetadata/ingestion-base:0.13.2",
|
||||
namespace='default',
|
||||
env_vars={"config": config, "pipelineType": "metadata"},
|
||||
dag=dag,
|
||||
|
||||
@ -217,6 +217,6 @@ a couple of points to validate:
|
||||
|
||||
The main root cause here is a version mismatch between the server and the client. Make sure that the `openmetadata-ingestion`
|
||||
python package you installed on the Airflow host has the same version as the OpenMetadata server. For example, to set up
|
||||
OpenMetadata server 0.13.1 you will need to install `openmetadata-ingestion==0.13.1.0`. Note that we are validating
|
||||
OpenMetadata server 0.13.2 you will need to install `openmetadata-ingestion~=0.13.2`. Note that we are validating
|
||||
the version as in `x.y.z`. Any differences after the PATCH versioning are not taken into account, as they are usually
|
||||
small bugfixes on existing functionalities.
|
||||
|
||||
@ -16,7 +16,10 @@ now allows all users to perform backups regardless of the underlying infrastruct
|
||||
|
||||
## Requirements
|
||||
|
||||
The backup CLI needs to be used with `openmetadata-ingestion` version 0.11.5 or higher.
|
||||
The backup CLI needs to be used with `openmetadata-ingestion` version 0.13.1 or higher.
|
||||
|
||||
However, we recommend to make sure that you are always in the latest `openmetadata-ingestion`
|
||||
version to have all the improvements shipped in the CLI.
|
||||
|
||||
## Installation
|
||||
|
||||
@ -58,15 +61,6 @@ replacing it with whatever comes from the SQL script.
|
||||
|
||||
</Note>
|
||||
|
||||
|
||||
<Note>
|
||||
|
||||
Running the backup or restore commands with version 0.12.3 or lower? The host parameter is `-h`.
|
||||
|
||||
For 0.13 or higher, `-h` is the flag used for the help command. Pass the host via `-H`.
|
||||
|
||||
</Note>
|
||||
|
||||
## Backup CLI
|
||||
|
||||
After the installation, we can take a look at the different options to run the CLI:
|
||||
|
||||
@ -97,10 +97,10 @@ You can validate that all containers are up by running with command `docker ps`.
|
||||
```commandline
|
||||
❯ docker ps
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
470cc8149826 openmetadata/server:0.13.1 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server
|
||||
63578aacbff5 openmetadata/ingestion:0.13.1 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion
|
||||
470cc8149826 openmetadata/server:0.13.2 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server
|
||||
63578aacbff5 openmetadata/ingestion:0.13.2 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion
|
||||
9f5ee8334f4b docker.elastic.co/elasticsearch/elasticsearch:7.10.2 "/tini -- /usr/local…" 45 seconds ago Up 44 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp openmetadata_elasticsearch
|
||||
08947ab3424b openmetadata/db:0.13.1 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql
|
||||
08947ab3424b openmetadata/db:0.13.2 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql
|
||||
```
|
||||
|
||||
In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585)
|
||||
|
||||
@ -262,7 +262,7 @@ COPY setup.py .
|
||||
RUN pip install --no-deps .
|
||||
```
|
||||
|
||||
where `openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `openmetadata/ingestion:0.13.1`.
|
||||
where `openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `openmetadata/ingestion:0.13.2`.
|
||||
This image needs to be built and published to the container registry of your choice.
|
||||
|
||||
### 2. Update your dependency values YAML
|
||||
@ -276,7 +276,7 @@ airflow:
|
||||
airflow:
|
||||
image:
|
||||
repository: <your repository> # by default, openmetadata/ingestion
|
||||
tag: <your tag> # by default, the version you are deploying, e.g., 0.13.1
|
||||
tag: <your tag> # by default, the version you are deploying, e.g., 0.13.2
|
||||
pullPolicy: "IfNotPresent"
|
||||
```
|
||||
|
||||
|
||||
@ -95,42 +95,42 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
|
||||
|
||||
<Table>
|
||||
|
||||
| Key | Type | Default |
|
||||
| :---------- | :---------- | :---------- |
|
||||
| affinity | object | `{}` |
|
||||
| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` |
|
||||
| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" |
|
||||
| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" |
|
||||
| fullnameOverride | string | `"openmetadata"` |
|
||||
| image.pullPolicy | string | `"Always"` |
|
||||
| image.repository | string | `"openmetadata/server"` |
|
||||
| image.tag | string | `0.13.1` |
|
||||
| imagePullSecrets | list | `[]` |
|
||||
| ingress.annotations | object | `{}` |
|
||||
| ingress.className | string | `""` |
|
||||
| ingress.enabled | bool | `false` |
|
||||
| ingress.hosts[0].host | string | `"open-metadata.local"` |
|
||||
| ingress.hosts[0].paths[0].path | string | `"/"` |
|
||||
| Key | Type | Default |
|
||||
| :---------- | :---------- |:---------------------------|
|
||||
| affinity | object | `{}` |
|
||||
| extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` |
|
||||
| extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" |
|
||||
| extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" |
|
||||
| fullnameOverride | string | `"openmetadata"` |
|
||||
| image.pullPolicy | string | `"Always"` |
|
||||
| image.repository | string | `"openmetadata/server"` |
|
||||
| image.tag | string | `0.13.2` |
|
||||
| imagePullSecrets | list | `[]` |
|
||||
| ingress.annotations | object | `{}` |
|
||||
| ingress.className | string | `""` |
|
||||
| ingress.enabled | bool | `false` |
|
||||
| ingress.hosts[0].host | string | `"open-metadata.local"` |
|
||||
| ingress.hosts[0].paths[0].path | string | `"/"` |
|
||||
| ingress.hosts[0].paths[0].pathType | string | `"ImplementationSpecific"` |
|
||||
| livenessProbe.initialDelaySeconds | int | `60` |
|
||||
| livenessProbe.periodSeconds | int | `30` |
|
||||
| livenessProbe.failureThreshold | int | `5` |
|
||||
| nameOverride | string | `""` |
|
||||
| nodeSelector | object | `{}` |
|
||||
| podAnnotations | object | `{}` |
|
||||
| podSecurityContext | object | `{}` |
|
||||
| readinessProbe.initialDelaySeconds | int | `60` |
|
||||
| readinessProbe.periodSeconds | int | `30` |
|
||||
| readinessProbe.failureThreshold | int | `5` |
|
||||
| replicaCount | int | `1` |
|
||||
| resources | object | `{}` |
|
||||
| securityContext | object | `{}` |
|
||||
| service.port | int | `8585` |
|
||||
| service.type | string | `"ClusterIP"` |
|
||||
| serviceAccount.annotations | object | `{}` |
|
||||
| serviceAccount.create | bool | `true` |
|
||||
| serviceAccount.name | string | `nil` |
|
||||
| sidecars | list | `[]` |
|
||||
| tolerations | list | `[]` |
|
||||
| livenessProbe.initialDelaySeconds | int | `60` |
|
||||
| livenessProbe.periodSeconds | int | `30` |
|
||||
| livenessProbe.failureThreshold | int | `5` |
|
||||
| nameOverride | string | `""` |
|
||||
| nodeSelector | object | `{}` |
|
||||
| podAnnotations | object | `{}` |
|
||||
| podSecurityContext | object | `{}` |
|
||||
| readinessProbe.initialDelaySeconds | int | `60` |
|
||||
| readinessProbe.periodSeconds | int | `30` |
|
||||
| readinessProbe.failureThreshold | int | `5` |
|
||||
| replicaCount | int | `1` |
|
||||
| resources | object | `{}` |
|
||||
| securityContext | object | `{}` |
|
||||
| service.port | int | `8585` |
|
||||
| service.type | string | `"ClusterIP"` |
|
||||
| serviceAccount.annotations | object | `{}` |
|
||||
| serviceAccount.create | bool | `true` |
|
||||
| serviceAccount.name | string | `nil` |
|
||||
| sidecars | list | `[]` |
|
||||
| tolerations | list | `[]` |
|
||||
|
||||
</Table>
|
||||
|
||||
@ -20,15 +20,22 @@ It is adviced to go through [openmetadata release notes](/deployment/upgrade/ver
|
||||
|
||||
</Warning>
|
||||
|
||||
### Backup 0.12.3 data
|
||||
### Backup your data
|
||||
|
||||
<Note>
|
||||
|
||||
To run the backup and restore commands, please make sure that you are always in the latest `openmetadata-ingestion`
|
||||
version to have all the improvements shipped in the CLI.
|
||||
|
||||
</Note>
|
||||
|
||||
1. Make sure your instance is connected to the Database server
|
||||
2. Create a virtual environment to install an upgraded `metadata` version to run the backup command:
|
||||
1. `python -m venv venv`
|
||||
2. `source venv/bin/activate`
|
||||
3. `pip install openmetadata-ingestion~=0.13.1.0`
|
||||
3. `pip install openmetadata-ingestion~=0.13.2`
|
||||
3. Validate the installed `metadata` version with `python -m metadata --version`, which should tell us that we are
|
||||
indeed at 0.13.1. Notice the `python -m metadata` vs. `metadata`.
|
||||
indeed at 0.13.2. Notice the `python -m metadata` vs. `metadata`.
|
||||
4. Run the backup using the updated `metadata` CLI:
|
||||
```
|
||||
python -m metadata backup -u openmetadata_user -p openmetadata_password -H mysql -d openmetadata_db --port 3306
|
||||
@ -40,6 +47,7 @@ It is adviced to go through [openmetadata release notes](/deployment/upgrade/ver
|
||||
5. This will generate the .sql file which can be used for the backup
|
||||
In our case, the backup file was named `openmetadata_202212201528_backup.sql`. You can copy the name from the backup
|
||||
command output.
|
||||
|
||||
### 1. Download the binaries for the release you want to install
|
||||
|
||||
OpenMetadata release binaries are maintained as GitHub releases.
|
||||
@ -77,7 +85,7 @@ command.
|
||||
|
||||
```commandline
|
||||
|
||||
cd openmetadata-0.13.1
|
||||
cd openmetadata-0.13.2
|
||||
|
||||
```
|
||||
|
||||
@ -135,4 +143,4 @@ Click on reindex all
|
||||
|
||||
in the dialog box choose Recreate Indexes to All
|
||||
|
||||
<Image src="/images/deployment/upgrade/reindex-ES.png" alt="create-project" caption="Reindex"/>
|
||||
<Image src="/images/deployment/upgrade/reindex-ES.png" alt="create-project" caption="Reindex"/>
|
||||
|
||||
@ -18,10 +18,45 @@ Below we have highlighted the steps needed to upgrade to the latest version with
|
||||
|
||||
<Warning>
|
||||
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0130-release) before starting the upgrade process.
|
||||
It is advised to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0130-release) before starting the upgrade process.
|
||||
|
||||
</Warning>
|
||||
|
||||
## Upgrade from 0.13.1 to 0.13.2
|
||||
|
||||
Upgrading from 0.13.1 to 0.13.2 can be done easily as version 0.13.1 compose files already provided volumes for the databases.
|
||||
|
||||
Let's go through the required steps:
|
||||
|
||||
### 1. Backup 0.12.3 data
|
||||
|
||||
1. Make sure your instance is connected to the Database server
|
||||
2. Create a virtual environment to install an upgraded `metadata` version to run the backup command:
|
||||
1. `python -m venv venv`
|
||||
2. `source venv/bin/activate`
|
||||
3. `pip install openmetadata-ingestion~=0.13.2`
|
||||
3. Validate the installed `metadata` version with `python -m metadata --version`, which should tell us that we are
|
||||
indeed at 0.13.2. Notice the `python -m metadata` vs. `metadata`.
|
||||
4. Run the backup using the updated `metadata` CLI:
|
||||
```
|
||||
python -m metadata backup -u openmetadata_user -p openmetadata_password -H mysql -d openmetadata_db --port 3306
|
||||
```
|
||||
if using Postgres:
|
||||
```
|
||||
python -m metadata backup -u openmetadata_user -p openmetadata_password -H postgresql -d openmetadata_db --port 5432 -s public
|
||||
```
|
||||
5. This will generate the .sql file which can be used for the backup
|
||||
In our case, the backup file was named `openmetadata_202212201528_backup.sql`. You can copy the name from the backup
|
||||
command output.
|
||||
|
||||
### 2. Update the docker compose file
|
||||
|
||||
1. Stop the running compose deployment with `docker compose down`.
|
||||
2. On the compose file we ran the 0.13.1 version, update the image tag in the `ingestion` and
|
||||
`openmetadata-server` to 0.13.2. E.g., `image: openmetadata/server:0.13.2`.
|
||||
3. Start the updated compose file.
|
||||
4. Run the reindex from the UI.
|
||||
|
||||
## Upgrade from 0.12.3 to 0.13.1
|
||||
|
||||
Your production deployment should go from stable version to stable version. This translated to moving from 0.12.3 to 0.13.1 to
|
||||
|
||||
@ -12,12 +12,72 @@ OpenMetadata community will be doing feature releases and stable releases.
|
||||
- Feature releases are to upgrade your sandbox or POCs to give feedback to the community and any potential bugs that the community needs to fix.
|
||||
- Stable releases are to upgrade your production environments and share it with your users.
|
||||
|
||||
## 0.13.2 - Stable Release
|
||||
|
||||
OpenMetadata 0.13.2 is a stable release. Please check the [release notes](https://github.com/open-metadata/OpenMetadata/releases/tag/0.13.1-release)
|
||||
|
||||
If you are upgrading production this is the recommended version to upgrade.
|
||||
|
||||
## Breaking Changes for 0.13.2 Stable Release
|
||||
|
||||
### EntityName
|
||||
To better manage and harmonize `entityName` value and allow users to form better expectations around these values the team introduced an enforcement of the `entityName` format using regex pattern.
|
||||
|
||||
All the OpenMetadata entities `entityName` fields will enforce by the default the following regex pattern:
|
||||
- `^[\w'\- .&]+$`: match any word characters (equivalent to `[a-zA-Z0-9_]`) or the characters `'- .&`. For example when creating a pipeline service the following name will not be allowed `MyPipelineIngestion?!` while the following will be `My-Pipeline `.
|
||||
|
||||
Some entities are enforcing specific patterns:
|
||||
- Users: `([\w\-.]|[^@])+$` match any word characters (equivalent to `[a-zA-Z0-9_]`) or the characters `-.` and don't match the character `@`
|
||||
- Webhook: `^[\w'\-.]+$` match any word characters (equivalent to `[a-zA-Z0-9_]`) or the characters `-.`
|
||||
- Table: `^[\w'\- ./]+$` match any word characters (equivalent to `[a-zA-Z0-9_]`) or the characters `'-. /`
|
||||
- Location: `^[\w'\-./]+$` match any word characters (equivalent to `[a-zA-Z0-9_]`) or the characters `-.'/`
|
||||
- Type: `^[a-z][\w]+$` match any word characters (equivalent to `[a-zA-Z0-9_]`) starting with a lowercase letter (e.g. `tHisChar` will match, `ThisChar` will not)
|
||||
|
||||
If an entity name does not follow the pattern an error will be returned by the OpenMetadata platform.
|
||||
|
||||
**The change should be transparent for the end user and no action should be required.**
|
||||
|
||||
## EntityLink
|
||||
Similar to the implementation done for `entityName`, `entityLink` will now enforce a specific pattern. The structure of `entityLink` is in the form `<#E::{entities}::{entityType}::{field}::{fieldName}::{fieldValue}>`
|
||||
|
||||
All the OpenMetadata entities `entityLink` fields will enforce by the default the following regex pattern:
|
||||
- `^<#E::\w+::[\w'\- .&/\:+\"\\]+>$`: this means that `{entities}` value needs to match any word characters (equivalent to `[a-zA-Z0-9_]`) and the part after `{entities}` can match any word characters (equivalent to `[a-zA-Z0-9_]`) or the characters `'- .&/\:+\"\`
|
||||
|
||||
If an entity name does not follow the pattern an error will be returned by the OpenMetadata platform.
|
||||
|
||||
**The change should be transparent for the end user and no action should be required.**
|
||||
|
||||
### Tags API
|
||||
|
||||
Tags APIs were coded long before other entities' APIs were added. Due to this, tag API does not follow the API convention
|
||||
that all the other entities are following. This issue makes backward incompatible changes to follow the same convention as glossaries.
|
||||
|
||||
You can find the full list of API paths changes in the following [issue](https://github.com/open-metadata/OpenMetadata/issues/9259).
|
||||
|
||||
### Metabase and Domo Dashboards `name`
|
||||
|
||||
With the new restrictions on the `EntityName` and to ensure unicity of the assets, the **Metabase** and **Domo Dashboard** sources
|
||||
now ingest the `name` of the charts and dashboards with their internal ID value. Their `name` value will be used
|
||||
as the display name, but not as the OpenMetadata `name` anymore.
|
||||
|
||||
The recommended approach here is to create a new service and ingest the metadata again. If you ingest from the same
|
||||
service, the assets in there will end up being duplicated, as the `name` in OpenMetadata is used to identify each asset.
|
||||
|
||||
Let us know if you have any questions around this topic.
|
||||
|
||||
### Ingestion Framework sources directory structure
|
||||
|
||||
We have converted source modules (e.g., `redshift.py`) into packages containing all the necessary information (e.g., `redshift/metadata.py`).
|
||||
This has helped us reorganise functionalities and easily focus on each connector independently.
|
||||
|
||||
If you're extending any of the sources, you'll need to update your imports. You can take a look at the new
|
||||
structure [here](https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/src/metadata/ingestion/source).
|
||||
|
||||
|
||||
## 0.13.1 - Stable Release
|
||||
|
||||
OpenMetadata 0.13.1 is a stable release. Please check the [release notes](https://github.com/open-metadata/OpenMetadata/releases/tag/0.13.1-release)
|
||||
|
||||
If you are upgrading production this is the recommended version to upgrade.
|
||||
|
||||
## Breaking Changes for 0.13.1 Stable Release
|
||||
|
||||
OpenMetadata Release 0.13.1 introduces below breaking changes -
|
||||
@ -68,96 +128,6 @@ OpenMetadata 0.13.0 is a **feature release**.
|
||||
|
||||
Explore 0.13.0 by following up [Deployment guides](https://docs.open-metadata.org/deployment) and please give us any feedback on our [community slack](https://slack.open-metadata.org)
|
||||
|
||||
|
||||
## 0.12.3 - Stable release
|
||||
|
||||
OpenMetadata 0.12.3 is a stable release. Please check the [release notes](https://github.com/open-metadata/OpenMetadata/releases/tag/0.12.3-release)
|
||||
|
||||
If you are upgrading production this is the recommended version to upgrade.
|
||||
## Breaking Changes from 0.12.x Stable Release
|
||||
|
||||
OpenMetadata Release 0.12.x introduces below breaking changes -
|
||||
|
||||
### Change of OpenMetadata Service Namespace
|
||||
|
||||
Under the [openmetadata.yaml](https://github.com/open-metadata/OpenMetadata/blob/main/conf/openmetadata.yaml), all the class names are updated from `org.openmetadata.catalog.*` to `org.openmetadata.service.*`.
|
||||
|
||||
- If you are using a previous version of openmetadata.yaml config file with [bare metal](/deployment/bare-metal) installation, make sure to migrate all these values as per new openmetadata.yaml configurations. Check the below example code snippet from openmetadata.yaml configuration
|
||||
|
||||
```yaml
|
||||
...
|
||||
authorizerConfiguration:
|
||||
className: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
|
||||
containerRequestFilter: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
|
||||
...
|
||||
```
|
||||
|
||||
- If you are using [docker](/deployment/docker) installation with your custom env file, update all the environement variables from `org.openmetadata.catalog.*` to `org.openmetadata.service.*`.
|
||||
|
||||
```
|
||||
AUTHORIZER_CLASS_NAME=org.openmetadata.service.security.DefaultAuthorizer
|
||||
AUTHORIZER_REQUEST_FILTER=org.openmetadata.service.security.JwtFilter
|
||||
```
|
||||
|
||||
- If you are running openmetadata on [kubernetes with helm charts](/deployment/kubernetes), make sure to update `global.authorizer.className` and `global.authorizer.containerRequestFilter` with below values for your custom openmetadata helm chart values file.
|
||||
|
||||
```yaml
|
||||
global:
|
||||
...
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
...
|
||||
```
|
||||
|
||||
### Centralising of openmetadata/ingestion and openmetadata/airflow docker images
|
||||
|
||||
Starting 0.12.1 Release, we have centralized openmetadata/airflow and openmetadata/ingestion docker images
|
||||
with openmetadata/ingestion docker image which will be used with docker compose installation and kubernetes helm chart installation. This docker image is based on apache-airflow 2.3.3 image with python 3.9.9. This will be a rootless docker image for enhanced security.
|
||||
|
||||
- There is no change or effect with docker installation
|
||||
|
||||
- This is a breaking change if you are using a custom openmetadata-dependencies kubernetes helm chart values file.
|
||||
You will need to manually update the airflow image and tag with openmetadata/ingestion:0.13.1
|
||||
|
||||
```yaml
|
||||
...
|
||||
airflow:
|
||||
airflow:
|
||||
image:
|
||||
repository: openmetadata/ingestion
|
||||
tag: 0.13.1
|
||||
pullPolicy: "IfNotPresent"
|
||||
...
|
||||
```
|
||||
|
||||
<p>
|
||||
If you are extending openmetadata/airflow docker image with 0.13.1 release, you can safely replace that with openmetadata/ingestion:0.13.1 Docker Image.
|
||||
</p>
|
||||
|
||||
```Dockerfile
|
||||
FROM openmetadata/ingestion:0.13.1
|
||||
USER airflow
|
||||
...
|
||||
```
|
||||
|
||||
### Basic Authentication enabled by default
|
||||
|
||||
We have deprecated and removed no-auth as the authentication mechanism starting 0.12.1 Release with OpenMetadata.
|
||||
|
||||
The default Authentication mechanism will be basic authentication. You can login to OpenMetadata UI with below default credentials -
|
||||
|
||||
```
|
||||
Username - admin
|
||||
Password - admin
|
||||
```
|
||||
|
||||
### Enabled JWT Token Configuration by default
|
||||
|
||||
Starting 0.12.1 Release, OpenMetadata Installation will provide a default configuration that will enable JWT Token Configuration for the OpenMetadata Instance.
|
||||
|
||||
If you want to setup a production Open Metadata instance, it is recommended to follow [enable jwt tokens](/deployment/security/enable-jwt-tokens) to setup and configure your own JWT Token configurations.
|
||||
|
||||
## Backup Metadata
|
||||
|
||||
Before upgrading your OpenMetadata version we recommend backing up the metadata.
|
||||
|
||||
@ -19,23 +19,31 @@ We also assume that your helm chart release names are `openmetadata` and `openme
|
||||
|
||||
<Warning>
|
||||
|
||||
It is adviced to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0130-release)
|
||||
It is advised to go through [openmetadata release notes](/deployment/upgrade#breaking-changes-from-0130-release)
|
||||
|
||||
</Warning>
|
||||
|
||||
Below document is valid for upgrading Helm Charts from **0.12.x to 0.13.0**.
|
||||
### Backup your data
|
||||
|
||||
### Back up metadata
|
||||
<Note>
|
||||
|
||||
Before proceeding, please back up your MySQL/Postgres DB behind the OpenMetadata server. This step is crucial for restoring to your current state if any issues arise during the upgrade. It is recommended before upgrading your production instances.
|
||||
To run the backup and restore commands, please make sure that you are always in the latest `openmetadata-ingestion`
|
||||
version to have all the improvements shipped in the CLI.
|
||||
|
||||
Make sure you have connectivity between your database (MySQL / PostgreSQL) and the host machine where you will be running the below commands. If you are using the default database available with OpenMetadata Dependencies, make sure to port-forward the MySQL service using `kubectl port-forward service/mysql 3306:3306`.
|
||||
</Note>
|
||||
|
||||
Before proceeding, please back up your MySQL/Postgres DB behind the OpenMetadata server. This step is crucial for
|
||||
restoring to your current state if any issues arise during the upgrade. It is recommended before upgrading your production instances.
|
||||
|
||||
Make sure you have connectivity between your database (MySQL / PostgreSQL) and the host machine where you will be running
|
||||
the below commands. If you are using the default database available with OpenMetadata Dependencies, make sure to
|
||||
port-forward the MySQL service using `kubectl port-forward service/mysql 3306:3306`.
|
||||
|
||||
Then, follow the next steps to create a virtual environment and install the latest OpenMetadata Python package with the backup CLI:
|
||||
|
||||
1. `python -m venv venv`
|
||||
2. `source venv/bin/activate`
|
||||
3. `pip install openmetadata-ingestion~=0.13.1.0`
|
||||
3. `pip install openmetadata-ingestion~=0.13.2`
|
||||
4. Validate the installed `metadata` version with `python -m metadata --version`
|
||||
5. Run the backup using the updated `metadata` CLI:
|
||||
```
|
||||
@ -75,46 +83,27 @@ Verify with the below command to see the latest release available locally.
|
||||
```commandline
|
||||
helm search repo open-metadata --versions
|
||||
> NAME CHART VERSION APP VERSION DESCRIPTION
|
||||
open-metadata/openmetadata 0.0.48 0.13.0 A Helm chart for OpenMetadata on Kubernetes
|
||||
open-metadata/openmetadata 0.0.47 0.13.0 A Helm chart for OpenMetadata on Kubernetes
|
||||
open-metadata/openmetadata 0.0.46 0.12.3 A Helm chart for OpenMetadata on Kubernetes
|
||||
open-metadata/openmetadata 0.0.53 0.13.1 A Helm chart for OpenMetadata on Kubernetes
|
||||
open-metadata/openmetadata 0.0.52 0.13.1 A Helm chart for OpenMetadata on Kubernetes
|
||||
open-metadata/openmetadata 0.0.51 0.13.1 A Helm chart for OpenMetadata on Kubernetes
|
||||
...
|
||||
open-metadata/openmetadata-dependencies 0.0.48 0.13.0 Helm Dependencies for OpenMetadata
|
||||
open-metadata/openmetadata-dependencies 0.0.47 0.13.0 Helm Dependencies for OpenMetadata
|
||||
open-metadata/openmetadata-dependencies 0.0.46 0.12.3 Helm Dependencies for OpenMetadata
|
||||
open-metadata/openmetadata-dependencies 0.0.53 0.13.1 Helm Dependencies for OpenMetadata
|
||||
open-metadata/openmetadata-dependencies 0.0.52 0.13.1 Helm Dependencies for OpenMetadata
|
||||
open-metadata/openmetadata-dependencies 0.0.51 0.13.1 Helm Dependencies for OpenMetadata
|
||||
...
|
||||
```
|
||||
|
||||
## Upgrade OpenMetadata Dependencies
|
||||
|
||||
### Step 1: Upgrade OpenMetadata Dependencies with the below command
|
||||
|
||||
```commandline
|
||||
helm upgrade openmetadata-dependencies open-metadata/openmetadata-dependencies
|
||||
```
|
||||
|
||||
|
||||
The above command uses configurations defined [here](https://raw.githubusercontent.com/open-metadata/openmetadata-helm-charts/main/charts/deps/values.yaml).
|
||||
You can modify any configuration and deploy by passing your own `values.yaml`.
|
||||
|
||||
|
||||
### Step 2: Troubleshooting
|
||||
|
||||
If your helm upgrade fails with the below command result -
|
||||
```
|
||||
Error: UPGRADE FAILED: cannot patch "mysql" with kind StatefulSet: StatefulSet.apps "mysql" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
|
||||
```
|
||||
|
||||
This is probably because with `0.13.0`, we have **default size of mysql persistence set to 50Gi**.
|
||||
|
||||
Kubernetes does not allow changes to Persistent volume with helm upgrades.
|
||||
|
||||
In order to work around this issue, you can either default the persistence size to 8Gi or run the below command which will patch Persistent Volumes and Persistent Volume Claims for mysql helm and then run the above `helm upgrade` command.
|
||||
|
||||
```
|
||||
kubectl patch pvc data-mysql-0 -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
|
||||
kubectl patch pv <mysql-pv> -p '{"spec":{"storage":"50Gi"}}'
|
||||
```
|
||||
|
||||
<Tip>
|
||||
|
||||
Make sure that, when using your own `values.yaml`, you are not overwriting elements such as the `image` of the containers.
|
||||
@ -135,46 +124,29 @@ helm upgrade openmetadata open-metadata/openmetadata
|
||||
You might need to pass your own `values.yaml` with the `--values` flag
|
||||
|
||||
### Re-index all your metadata
|
||||
Go to Settings -> Elasticsearch
|
||||
|
||||
Go to Settings -> Elasticsearch
|
||||
<Image src="/images/deployment/upgrade/elasticsearch-re-index.png" alt="create-project" caption="Create a New Project"/>
|
||||
|
||||
Click on reindex all
|
||||
in the dialog box choose Recreate Indexes to All
|
||||
<Image src="/images/deployment/upgrade/reindex-ES.png" alt="create-project" caption="Reindex"/>
|
||||
Click on reindex all
|
||||
in the dialog box choose Recreate Indexes to All
|
||||
<Image src="/images/deployment/upgrade/reindex-ES.png" alt="create-project" caption="Reindex"/>
|
||||
|
||||
## Troubleshooting for 0.12 Release
|
||||
## Troubleshooting for 0.13.0 Release
|
||||
|
||||
### Using custom helm values
|
||||
|
||||
If you are facing an issue similar to below when openmetadata pod keeps on restarting.
|
||||
If your helm dependencies upgrade fails with the below command result -
|
||||
|
||||
```
|
||||
java.lang.ClassNotFoundException: org.openmetadata.service.security.DefaultAuthorizer
|
||||
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
|
||||
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
|
||||
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
|
||||
at java.base/java.lang.Class.forName0(Native Method)
|
||||
at java.base/java.lang.Class.forName(Class.java:315)
|
||||
at org.openmetadata.service.OpenMetadataApplication.registerAuthorizer(OpenMetadataApplication.java:240)
|
||||
at org.openmetadata.service.OpenMetadataApplication.run(OpenMetadataApplication.java:123)
|
||||
at org.openmetadata.service.OpenMetadataApplication.run(OpenMetadataApplication.java:92)
|
||||
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:59)
|
||||
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)
|
||||
at io.dropwizard.cli.Cli.run(Cli.java:78)
|
||||
at io.dropwizard.Application.run(Application.java:94)
|
||||
at org.openmetadata.service.OpenMetadataApplication.main(OpenMetadataApplication.java:323)
|
||||
Error: UPGRADE FAILED: cannot patch "mysql" with kind StatefulSet: StatefulSet.apps "mysql" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
|
||||
```
|
||||
|
||||
The root cause of the issue is the default helm values which are upgraded in helm charts but are getting overridden by your custom helm values. Please verify the config for Authorizer Class Name and Container Request Filter.
|
||||
This is probably because with `0.13.0`, we have **default size of mysql persistence set to 50Gi**.
|
||||
|
||||
We have changed `org.openmetadata.service.security.*` to `org.openmetadata.service.security.*`.
|
||||
Make sure to verify your helm values and update the below content.
|
||||
Kubernetes does not allow changes to Persistent volume with helm upgrades.
|
||||
|
||||
In order to work around this issue, you can either default the persistence size to 8Gi or run the below command which will patch Persistent Volumes and Persistent Volume Claims for mysql helm and then run the above `helm upgrade` command.
|
||||
|
||||
```
|
||||
global:
|
||||
...
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
...
|
||||
```
|
||||
kubectl patch pvc data-mysql-0 -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
|
||||
kubectl patch pv <mysql-pv> -p '{"spec":{"storage":"50Gi"}}'
|
||||
```
|
||||
@ -151,3 +151,93 @@ the `Enable Security` chapter.
|
||||
Note that the `ingestion-bot` bot is created (or updated if it already exists) as a system bot that cannot be deleted, and
|
||||
the credentials used for this bot, if they did not exist before, will be the ones present in the OpenMetadata configuration.
|
||||
Otherwise, a JWT Token will be generated to be the default authentication mechanism of the `ingestion-bot`.
|
||||
|
||||
## 0.12.3 - Stable release
|
||||
|
||||
OpenMetadata 0.12.3 is a stable release. Please check the [release notes](https://github.com/open-metadata/OpenMetadata/releases/tag/0.12.3-release)
|
||||
|
||||
If you are upgrading production this is the recommended version to upgrade.
|
||||
|
||||
## Breaking Changes from 0.12.x Stable Release
|
||||
|
||||
OpenMetadata Release 0.12.x introduces below breaking changes -
|
||||
|
||||
### Change of OpenMetadata Service Namespace
|
||||
|
||||
Under the [openmetadata.yaml](https://github.com/open-metadata/OpenMetadata/blob/main/conf/openmetadata.yaml), all the class names are updated from `org.openmetadata.catalog.*` to `org.openmetadata.service.*`.
|
||||
|
||||
- If you are using a previous version of openmetadata.yaml config file with [bare metal](/deployment/bare-metal) installation, make sure to migrate all these values as per new openmetadata.yaml configurations. Check the below example code snippet from openmetadata.yaml configuration
|
||||
|
||||
```yaml
|
||||
...
|
||||
authorizerConfiguration:
|
||||
className: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
|
||||
containerRequestFilter: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
|
||||
...
|
||||
```
|
||||
|
||||
- If you are using [docker](/deployment/docker) installation with your custom env file, update all the environement variables from `org.openmetadata.catalog.*` to `org.openmetadata.service.*`.
|
||||
|
||||
```
|
||||
AUTHORIZER_CLASS_NAME=org.openmetadata.service.security.DefaultAuthorizer
|
||||
AUTHORIZER_REQUEST_FILTER=org.openmetadata.service.security.JwtFilter
|
||||
```
|
||||
|
||||
- If you are running openmetadata on [kubernetes with helm charts](/deployment/kubernetes), make sure to update `global.authorizer.className` and `global.authorizer.containerRequestFilter` with below values for your custom openmetadata helm chart values file.
|
||||
|
||||
```yaml
|
||||
global:
|
||||
...
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
...
|
||||
```
|
||||
|
||||
### Centralising of openmetadata/ingestion and openmetadata/airflow docker images
|
||||
|
||||
Starting 0.12.1 Release, we have centralized openmetadata/airflow and openmetadata/ingestion docker images
|
||||
with openmetadata/ingestion docker image which will be used with docker compose installation and kubernetes helm chart installation. This docker image is based on apache-airflow 2.3.3 image with python 3.9.9. This will be a rootless docker image for enhanced security.
|
||||
|
||||
- There is no change or effect with docker installation
|
||||
|
||||
- This is a breaking change if you are using a custom openmetadata-dependencies kubernetes helm chart values file.
|
||||
You will need to manually update the airflow image and tag with `openmetadata/ingestion:0.12.3`
|
||||
|
||||
```yaml
|
||||
...
|
||||
airflow:
|
||||
airflow:
|
||||
image:
|
||||
repository: openmetadata/ingestion
|
||||
tag: 0.12.3
|
||||
pullPolicy: "IfNotPresent"
|
||||
...
|
||||
```
|
||||
|
||||
<p>
|
||||
If you are extending openmetadata/airflow docker image with 0.12.1 release, you can safely replace that with openmetadata/ingestion:0.12.3 Docker Image.
|
||||
</p>
|
||||
|
||||
```Dockerfile
|
||||
FROM openmetadata/ingestion:0.12.3
|
||||
USER airflow
|
||||
...
|
||||
```
|
||||
|
||||
### Basic Authentication enabled by default
|
||||
|
||||
We have deprecated and removed no-auth as the authentication mechanism starting 0.12.1 Release with OpenMetadata.
|
||||
|
||||
The default Authentication mechanism will be basic authentication. You can login to OpenMetadata UI with below default credentials -
|
||||
|
||||
```
|
||||
Username - admin
|
||||
Password - admin
|
||||
```
|
||||
|
||||
### Enabled JWT Token Configuration by default
|
||||
|
||||
Starting 0.12.1 Release, OpenMetadata Installation will provide a default configuration that will enable JWT Token Configuration for the OpenMetadata Instance.
|
||||
|
||||
If you want to setup a production Open Metadata instance, it is recommended to follow [enable jwt tokens](/deployment/security/enable-jwt-tokens) to setup and configure your own JWT Token configurations.
|
||||
|
||||
@ -7,12 +7,29 @@ slug: /deployment/upgrade/versions/012-to-013
|
||||
|
||||
Upgrading from 0.12 to 0.13 can be done directly on your instances. This page will list few general details you should take into consideration when running the upgrade.
|
||||
|
||||
## 0.13.2 Highlights
|
||||
|
||||
## Highlights
|
||||
### Service Connection Updates
|
||||
|
||||
- Oracle:
|
||||
- Oracle connector now supports thick mode enabled by default.
|
||||
- Added: `instantClientDirectory` with the path to the Oracle libraries. They are provided by default for release 19
|
||||
in the ingestion container.
|
||||
|
||||
- Superset:
|
||||
- The Superset connector now supports the ability to extract metadata directly from the database underlying Superset, making it useful for users who have SSO enabled.
|
||||
- Added: `connection` field the connection can be of type `ApiConnection`, `postgresConnection` or `mysqlConnection`
|
||||
- Removed: `username`, `password` & `provider` fields as now these fields will be part of `ApiConnection`
|
||||
|
||||
## 0.13.1 Highlights
|
||||
|
||||
### Deprecating botPrincipals from OpenMetadata Configuration
|
||||
|
||||
Starting with `0.13.0`, we have deprecated the initial configurations for Authorizer Bot Pincipals. This means that all the configs will be taken care from UI once it's up and running under settings >> bots section. The default once available and enabled will be `ingestion-bot` which will have JWT Token configured. You can edit the same for configuring service principals instead of using the default JWT Auth mechanism. For more information on configuring Bots, you can visit [Enable Security](/deployment/security).
|
||||
Starting with `0.13.0`, we have deprecated the initial configurations for Authorizer Bot Principals. This means that all
|
||||
the configs will be taken care from UI once it's up and running under settings >> bots section. The default once available
|
||||
and enabled will be `ingestion-bot` which will have JWT Token configured. You can edit the same for configuring service
|
||||
principals instead of using the default JWT Auth mechanism. For more information on configuring Bots, you can
|
||||
visit [Enable Security](/deployment/security).
|
||||
|
||||
### Service Connection Updates
|
||||
|
||||
@ -31,8 +48,6 @@ Starting with `0.13.0`, we have deprecated the initial configurations for Author
|
||||
|
||||
### Lineage Backend
|
||||
|
||||
In 0.13.1:
|
||||
|
||||
- The import for the Airflow Lineage Backend has been updated from `airflow_provider_openmetadata.lineage.openmetadata.OpenMetadataLineageBackend`
|
||||
to `airflow_provider_openmetadata.lineage.backend.OpenMetadataLineageBackend`.
|
||||
- We removed support from Airflow v1.
|
||||
@ -40,12 +55,13 @@ to `airflow_provider_openmetadata.lineage.backend.OpenMetadataLineageBackend`.
|
||||
|
||||
## Webhooks
|
||||
|
||||
In 0.13.1 , OpenMetadata will be deprecating the existing webhooks for Slack, MSTeams.
|
||||
OpenMetadata will be deprecating the existing webhooks for Slack, MSTeams.
|
||||
|
||||
Before upgrading to 0.13.1 it is recommended to save the existing Webhook configs(like webhook url) to use them later.
|
||||
|
||||
We have added Alerts/Notifications , which can be configured to receive customised alerts on updates in OM using Triggers, Filtering Information to different destinations like Slack, MsTeams or even Emails.
|
||||
Please use the same webhook config that you had saved from previous version to configure the Alerts Destination after upgrading.
|
||||
|
||||
### dbt Workflow
|
||||
|
||||
- Starting with `0.13.1` dbt ingestion has been separated from the metadata ingestion. It can now be configured as a separate workflow after completing the metadata ingestion workflow
|
||||
|
||||
@ -3,27 +3,36 @@ title: Latest Release
|
||||
slug: /overview/latest-release
|
||||
---
|
||||
|
||||
# [0.13.2 Release](https://github.com/open-metadata/OpenMetadata/releases/tag/0.13.2-release) - Jan 30th 2022 🎉
|
||||
|
||||
# [0.13.1 Release](https://github.com/open-metadata/OpenMetadata/releases/tag/0.13.1-release) - Dec 20th 2022 🎉
|
||||
## Profiler and Data Quality
|
||||
- Freshness Metric has been introduced. Data freshness shows DML operations performed against a table and the number of rows affected. All this is displayed within the data profiler with filterable graphs. This is currently supported for BigQuery, Snowflake, and Redshift.
|
||||
- Support has been added for data quality tests on Data Lake.
|
||||
- UI has been improved to show table and column profile data on seperate page. Legend is now selectable to filter for specific metrics
|
||||
## Improved SQL Lineage
|
||||
- We have collaborated with the [sqllineage](https://github.com/reata/sqllineage) and [sqlfluff](https://www.sqlfluff.com/) communities
|
||||
to improve the parsing capabilities of `sqllineage`. We'll continue to collaborate to ship further improvements in new releases.
|
||||
|
||||
## Alerts and Notification
|
||||
The logic for Notification Support has been improved. Users can define Alerts based on a Trigger (all data assets or a specific entity), Filters (events to consider), and Action (Slack, MS Teams, Email, Webhook) on where to send the alert.
|
||||
## New Glossary UI
|
||||
- Moved from a tree view in the left panel to an easy to navigate list of the terms sorted alphabetically.
|
||||
- The term list shows the tags and descriptions in the cards.
|
||||
|
||||
## Ingestion
|
||||
- Now, dbt has its own workflow. Previously, dbt was a part of metadata ingestion workflow.
|
||||
- Airflow Lineage Operator and the OpenMetadata Hook are now part of the ingestion package. Send Airflow metadata from your DAGs and safely store the OpenMetadata server connection directly in Airflow.
|
||||
- Multiple Databases (catalog) is now supported for the Databricks connector
|
||||
- Azure blob is now supported to backup your metadata into
|
||||
## Glossary Import & Export
|
||||
- You can now export your Glossary data as a CSV file.
|
||||
- In the same way, you can now bulk upload terms to a Glossary by adding their details in a CSV file.
|
||||
- The import utility will validate the file and show you a preview of the elements that are going to be imported to OpenMetadata.
|
||||
|
||||
## New Connectors
|
||||
- OpenMetadata now supports Azure Datalake Storage Gen 2
|
||||
## Unified Tag Category API
|
||||
- Renamed Tag Categories to Classification, a more widely used term.
|
||||
- Updated the API to conform with the rest of the specification. More info [here](https://github.com/open-metadata/OpenMetadata/issues/9259).
|
||||
|
||||
## General Improvements
|
||||
- Users can update the description and tags for Topic Schema. Previously, the topic schemas were read-only. We now support Avro/Protobuf parsing and field level details for topic schemas.
|
||||
- The layout for the Data Insight Report has been improved. We now display a line graph instead of a bar graph. The Most Viewed Data Assets are clickable to view the asset details page.
|
||||
- Improvements have been made to Advanced Search. Now, when a filter is applied, the details of the filter selected are displayed for clarity.
|
||||
- On the Explore page UI, the Side Preview is now available for all data assets. Previously it was only displayed for tables.
|
||||
## Mutually Exclusive Tags
|
||||
- When creating a Classification or a Glossary term, you can now make the tags to be mutually exclusive.
|
||||
- If tags are set to be mutually exclusive, you won't be able to set multiple tags from the same category in the same asset.
|
||||
|
||||
## EntityName
|
||||
- Special characters
|
||||
|
||||
## Ingestion Framework
|
||||
- Performance Improvements: We are now getting descriptions in batch, making connectors such as Redshift or Snowflake way faster!
|
||||
- The Oracle connector now ships with the Thick mode enabled.
|
||||
- AWS QuickSight fixes
|
||||
- DB2 constraints and profiler improvements
|
||||
- Added support for Postgres Foreign Tables
|
||||
- Added support for Datalake profiler row-based sampling
|
||||
|
||||
@ -72,20 +72,12 @@ You can check the latest release [here](/overview/releases).
|
||||
<li>Users will be able to add descriptions/tags at the field level</li>
|
||||
<li>Users can search based on fields in a schema of a topic.</li>
|
||||
</Tile>
|
||||
<Tile
|
||||
title="Reverse Metadata **beta**"
|
||||
text=""
|
||||
background="yellow-70"
|
||||
bordercolor="blue-70"
|
||||
>
|
||||
<li>We are super excited about this feature coming in 0.13.1</li>
|
||||
</Tile>
|
||||
|
||||
</TileContainer>
|
||||
|
||||
|
||||
|
||||
## 1.0 Release - Feb 6th, 2023
|
||||
## 1.0 Release - March 14th, 2023
|
||||
|
||||
<TileContainer>
|
||||
<Tile
|
||||
@ -149,7 +141,7 @@ You can check the latest release [here](/overview/releases).
|
||||
</Tile>
|
||||
</TileContainer>
|
||||
|
||||
## 1.1 Release - March 15th 2023
|
||||
## 1.1 Release - April 18th 2023
|
||||
|
||||
<TileContainer>
|
||||
<Tile
|
||||
@ -199,4 +191,11 @@ You can check the latest release [here](/overview/releases).
|
||||
<li> Add support for Notebook Entity </li>
|
||||
<li> Add support for Report Entity </li>
|
||||
</Tile>
|
||||
<Tile
|
||||
title="Reverse Metadata **beta**"
|
||||
text=""
|
||||
background="yellow-70"
|
||||
bordercolor="blue-70"
|
||||
>
|
||||
</Tile>
|
||||
</TileContainer>
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user