From 68499f87d59280acb41c2520b16fcae16ff7e905 Mon Sep 17 00:00:00 2001 From: Karan Hotchandani <33024356+karanh37@users.noreply.github.com> Date: Thu, 31 Oct 2024 16:40:33 +0530 Subject: [PATCH] Prep 1.5.10 release (#18487) --- .../content/partials/v1.5/releases/latest.md | 20 +- .../content/partials/v1.6/releases/latest.md | 20 +- .../quick-start/local-docker-deployment.md | 54 ++-- .../content/v1.5.x/releases/releases/index.md | 286 +++++++++++++---- .../releases/releases/index.md | 288 ++++++++++++++---- 5 files changed, 497 insertions(+), 171 deletions(-) diff --git a/openmetadata-docs/content/partials/v1.5/releases/latest.md b/openmetadata-docs/content/partials/v1.5/releases/latest.md index 1ec48796715..f4613bb5347 100644 --- a/openmetadata-docs/content/partials/v1.5/releases/latest.md +++ b/openmetadata-docs/content/partials/v1.5/releases/latest.md @@ -1,7 +1,7 @@ -# 1.5.9 Release 🎉 +# 1.5.10 Release 🎉 {% note noteType="Tip" %} -**Oct 29th, 2024** +**Oct 31st, 2024** {% /note %} {% inlineCalloutContainer %} @@ -10,19 +10,17 @@ color="violet-70" icon="celebration" bold="Upgrade OpenMetadata" href="/deployment/upgrade" %} -Learn how to upgrade your OpenMetadata instance to 1.5.9! +Learn how to upgrade your OpenMetadata instance to 1.5.10! {% /inlineCallout %} {% /inlineCalloutContainer %} -You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.5.8-release). +You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.5.10-release). # What's Changed -- Prepare App Framework to handle application limits -- Add Query Builder widget -- Revamp MetaPilot as Collate AI and add limits (Collate only) -- Fix EntityLink for names with brackets -- Fix backend database Azure auth -- Mask Greenplum secrets on the UI +- Fix encoding issue for teams search query. +- Fix disable empty rule for query builder widget. +- Fix unable to add more enum values for enum cp. +- Fix navigate to listing for deleting a service. -**Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.8-release...1.5.9-release +**Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.9-release...1.5.10-release diff --git a/openmetadata-docs/content/partials/v1.6/releases/latest.md b/openmetadata-docs/content/partials/v1.6/releases/latest.md index 1ec48796715..f4613bb5347 100644 --- a/openmetadata-docs/content/partials/v1.6/releases/latest.md +++ b/openmetadata-docs/content/partials/v1.6/releases/latest.md @@ -1,7 +1,7 @@ -# 1.5.9 Release 🎉 +# 1.5.10 Release 🎉 {% note noteType="Tip" %} -**Oct 29th, 2024** +**Oct 31st, 2024** {% /note %} {% inlineCalloutContainer %} @@ -10,19 +10,17 @@ color="violet-70" icon="celebration" bold="Upgrade OpenMetadata" href="/deployment/upgrade" %} -Learn how to upgrade your OpenMetadata instance to 1.5.9! +Learn how to upgrade your OpenMetadata instance to 1.5.10! {% /inlineCallout %} {% /inlineCalloutContainer %} -You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.5.8-release). +You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.5.10-release). # What's Changed -- Prepare App Framework to handle application limits -- Add Query Builder widget -- Revamp MetaPilot as Collate AI and add limits (Collate only) -- Fix EntityLink for names with brackets -- Fix backend database Azure auth -- Mask Greenplum secrets on the UI +- Fix encoding issue for teams search query. +- Fix disable empty rule for query builder widget. +- Fix unable to add more enum values for enum cp. +- Fix navigate to listing for deleting a service. -**Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.8-release...1.5.9-release +**Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.9-release...1.5.10-release diff --git a/openmetadata-docs/content/v1.5.x/quick-start/local-docker-deployment.md b/openmetadata-docs/content/v1.5.x/quick-start/local-docker-deployment.md index af6363df15a..ce637c764b4 100644 --- a/openmetadata-docs/content/v1.5.x/quick-start/local-docker-deployment.md +++ b/openmetadata-docs/content/v1.5.x/quick-start/local-docker-deployment.md @@ -65,26 +65,27 @@ Docker Compose version v2.1.1 Follow the instructions [here](https://docs.docker.com/compose/cli-command/#install-on-linux) to install docker compose version 2.0.0 1. Run the following command to download the current stable release of Docker Compose - ``` - DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker} - mkdir -p $DOCKER_CONFIG/cli-plugins - curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o - $DOCKER_CONFIG/cli-plugins/docker-compose - ``` - + ``` + DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker} + + mkdir -p $DOCKER_CONFIG/cli-plugins + curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o + $DOCKER_CONFIG/cli-plugins/docker-compose + ``` + This command installs Compose V2 for the active user under $HOME directory. To install Docker Compose for all users on your system, replace` ~/.docker/cli-plugins` with `/usr/local/lib/docker/cli-plugins`. 2. Apply executable permissions to the binary - ``` - chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose - ``` + ``` + chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose + ``` 3. Test your installation - ``` - docker compose version - > Docker Compose version v2.2.3 - ``` + ``` + docker compose version + > Docker Compose version v2.2.3 + ``` ## Windows @@ -97,7 +98,6 @@ Follow the instructions [here](https://docs.docker.com/compose/cli-command/#inst - After completion of the pre-requisites, please install `python3-pip` and `python3-venv` on your Ubuntu system. - Command: `apt install python3-pip python3-venv` (Ensure that you have the privilege to install packages, if not, please use Super User.) - ## Procedure ### 1. Create a directory for OpenMetadata @@ -113,21 +113,22 @@ mkdir openmetadata-docker && cd openmetadata-docker Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases/latest). The latest version is at the top of the page - - Deploying with MySQL: Download `docker-compose.yml` file from the above link. - - Deploying with PostgreSQL: Download `docker-compose-postgres.yml` file from the above link. + +- Deploying with MySQL: Download `docker-compose.yml` file from the above link. +- Deploying with PostgreSQL: Download `docker-compose-postgres.yml` file from the above link. You can use the curl or wget command as well to fetch the docker compose files from your terminal - ```commandline -curl -sL -o docker-compose.yml https://github.com/open-metadata/OpenMetadata/releases/download/1.5.9-release/docker-compose.yml +curl -sL -o docker-compose.yml https://github.com/open-metadata/OpenMetadata/releases/download/1.5.10-release/docker-compose.yml -curl -sL -o docker-compose-postgres.yml https://github.com/open-metadata/OpenMetadata/releases/download/1.5.9-release/docker-compose-postgres.yml +curl -sL -o docker-compose-postgres.yml https://github.com/open-metadata/OpenMetadata/releases/download/1.5.10-release/docker-compose-postgres.yml ``` ```commandline -wget https://github.com/open-metadata/OpenMetadata/releases/download/1.5.9-release/docker-compose.yml +wget https://github.com/open-metadata/OpenMetadata/releases/download/1.5.10-release/docker-compose.yml -wget https://github.com/open-metadata/OpenMetadata/releases/download/1.5.9-release/docker-compose-postgres.yml +wget https://github.com/open-metadata/OpenMetadata/releases/download/1.5.10-release/docker-compose-postgres.yml ``` ### 3. Start the Docker Compose Services @@ -137,7 +138,7 @@ Run the below command to deploy the OpenMetadata For OpenMetadata with MySQL Database - ```commandline -docker compose -f docker-compose.yml up --detach +docker compose -f docker-compose.yml up --detach ``` For OpenMetadata with PostgreSQL Database - @@ -149,6 +150,7 @@ docker compose -f docker-compose-postgres.yml up --detach These commands will pull the docker images of Openmetadata for MySQL / PostgreSQL, OpenMetadata-Server, OpenMetadata-Ingestion and Elasticsearch. Upon running this command you should see output similar to the following. + ```commandline +] Running 7/8 ⠿ Network metadata_app_net Created 0.2s @@ -166,10 +168,10 @@ You can validate that all containers are up by running with command `docker ps`. ```commandline ❯ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -470cc8149826 openmetadata/server:1.5.9 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server -63578aacbff5 openmetadata/ingestion:1.5.9 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion +470cc8149826 openmetadata/server:1.5.10 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server +63578aacbff5 openmetadata/ingestion:1.5.10 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion 9f5ee8334f4b docker.elastic.co/elasticsearch/elasticsearch:7.16.3 "/tini -- /usr/local…" 45 seconds ago Up 44 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp openmetadata_elasticsearch -08947ab3424b openmetadata/db:1.5.9 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql +08947ab3424b openmetadata/db:1.5.10 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql ``` In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585) @@ -197,6 +199,7 @@ via the UI. In the Airflow, you will also see some sample DAGs that will ingest sample data and serve as an example. You can access Airflow at [http://localhost:8080](http://localhost:8080). Use the following credentials to log in to Airflow. + - Username: `admin` - Password: `admin` @@ -266,7 +269,6 @@ installation. OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. - ### Volume Permissions: Operation not permitted If you are running on Windows (WSL2) and see permissions errors when starting the databases (either MySQL or Postgres), e.g., diff --git a/openmetadata-docs/content/v1.5.x/releases/releases/index.md b/openmetadata-docs/content/v1.5.x/releases/releases/index.md index 1c6e470ba97..feb0596eaaf 100644 --- a/openmetadata-docs/content/v1.5.x/releases/releases/index.md +++ b/openmetadata-docs/content/v1.5.x/releases/releases/index.md @@ -14,6 +14,25 @@ version. To see what's coming in next releases, please check our [Roadmap](/rele {% partial file="/v1.5/releases/latest.md" /%} +# 1.5.9 Release + +{% note noteType="Tip" %} +**Oct 29th, 2024** +{% /note %} + +You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.5.9-release). + +# What's Changed + +- Prepare App Framework to handle application limits +- Add Query Builder widget +- Revamp MetaPilot as Collate AI and add limits (Collate only) +- Fix EntityLink for names with brackets +- Fix backend database Azure auth +- Mask Greenplum secrets on the UI + +**Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.8-release...1.5.9-release + # 1.5.8 Release {% note noteType="Tip" %} @@ -38,7 +57,6 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fix: Task deserialization in Airflow metadata ingestion. - Fix: Web analytic activity being reset. - **Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.7-release...1.5.8-release # 1.5.7 Release @@ -121,8 +139,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Improved AlationSink connector. - Fixed sktime version to fix AUT - Fixed Expected ColumnLineage but got dict -- Improved Collate API with Knowledge Center routes. ${CollateIconWithLinkMD} - +- Improved Collate API with Knowledge Center routes. ${CollateIconWithLinkMD} **Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.5-release...1.5.6-release @@ -174,6 +191,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # What's Changed ## OpenMetadata + - Hotfix to the Term Aggregation size on Data Insights - ES pagination with error handling - Updated Domain in Docker Compose & Docs @@ -185,6 +203,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Unable to access the import glossary page ## Collate + - Fix token limitations using config - Fix Automator pagination - Fix MetaPilot push for no constraint @@ -254,17 +273,20 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # Backward Incompatible Changes ## Multi Owners + OpenMetadata allows a single user or a team to be tagged as owners for any data assets. In Release 1.5.1, we allow users to tag multiple individual owners or a single team. This will allow organizations to add ownership to multiple individuals without necessarily needing to create a team around them like previously. This is a backward incompatible change, if you are using APIs, please make sure the owner field is now changed to “owners” ## Import/Export Format + To support the multi-owner format, we have now changed how we export and import the CSV file in glossary, services, database, schema, table, etc. The new format will be user:userName;team:TeamName If you are importing an older file, please make sure to make this change. ## Pydantic V2 + The core of OpenMetadata are the JSON Schemas that define the metadata standard. These schemas are automatically translated into Java, Typescript, and Python code with Pydantic classes. In this release, we have [migrated](https://docs.pydantic.dev/latest/migration/) the codebase from Pydantic V1 to Pydantic V2. @@ -314,14 +336,17 @@ In the latest release, several updates and enhancements have been made to the JS - **Tableau**: Enhanced support by adding capabilities for connecting with `TableauPublishedDatasource` and `TableauEmbeddedDatasource`, providing more granular control over data visualization and reporting. ## Include DDL + During the Database Metadata ingestion, we can optionally pick up the DDL for both tables and views. During the metadata ingestion, we use the view DDLs to generate the View Lineage. To reduce the processing time for out-of-the-box workflows, we are disabling the include DDL by default, whereas before, it was enabled, which potentially led to long-running workflows. ## Secrets Manager + Starting with the release 1.5.1, the JWT Token for the bots will be sent to the Secrets Manager if you configured one. It won't appear anymore in your dag_generated_configs in Airflow. ## Python SDK + The `metadata insight` command has been removed. Since Data Insights application was moved to be an internal system application instead of relying on external pipelines the SDK command to run the pipeline was removed. # What's New @@ -346,11 +371,13 @@ We also have improved the Table Data quality dashboard to showcase the tests cat {% youtube videoId="bXcQBtZuyoU" start="0:00" end="2:10" width="560px" height="315px" /%} ## Freshness Data Quality Tests (Collate) + Working with old data can lead to making wrong decisions. With the new Freshness test, you can validate that your data arrives at the right time. Freshness tests are a critical part of any data team's toolset. Bringing these tests together with lineage information and the Incident Manager, your team will be able to quickly detect issues related to missing data or stuck pipelines. {% youtube videoId="QRcR3m9cCGo" start="0:00" end="1:09" width="560px" height="315px" /%} ## Data Diff Data Quality Tests + Data quality checks are important not only within a single table but also between different tables. These data diff checks can ensure key data remains unchanged after transformation, or conversely, ensure that the transformations were actually performed. We are introducing the **table difference data quality test** to validate that multiple appearances of the same information remain consistent. Note that the test allows you to specify which column to use as a key and which columns you want to compare, and even add filters in the data to give you more control over multiple use cases. @@ -358,11 +385,13 @@ We are introducing the **table difference data quality test** to validate that m {% youtube videoId="oxZVS_UGrE4" start="0:00" end="2:22" width="560px" height="315px" /%} ## Domains RBAC & Subdomains + OpenMetadata introduced Domains & Data Products in 1.3.0. Since then, many large organizations have started using Domains & Data Products to achieve better ownership and collaboration around domains that can span multiple teams. In the 1.5.1 release, we added support for subdomains. This will help teams to organize into multiple subdomains within each domain. ### RBAC for Domains + With the 1.5.1 release, we are adding more stricter controls around Domain. Now, teams, data assets, glossaries, and classification can have domain concepts and can get a policy such that only users within a domain can access the data within a domain. Domain owners can use Data Products to publish data products and showcase publicly available data assets from a specific domain. This will help large companies to use a single OpenMetadata platform to unify all of their data and teams but also provide more stringent controls to segment the data between domains @@ -370,6 +399,7 @@ This will help large companies to use a single OpenMetadata platform to unify al {% youtube videoId="r-_HaewjgTQ" start="0:00" end="0:44" width="560px" height="315px" /%} ## Improved Explore Page & Data Asset Widget + OpenMetadata, with its simple UI/UX and data collaboration features, is becoming more attractive to non-technical users as well. Data Governance teams are using OpenMetadata to add glossary terms and policies around metadata. Teams using Collate SaaS product are taking advantage of our Automations feature to gain productivity in their governance tasks. Our new improved navigation on the Explore page will help users navigate hierarchically and find the data they are looking for. Users will see the data assets now grouped by `service name -> database -> schema -> tables/stored procedures`. @@ -379,11 +409,13 @@ We are also making the discovery of data more accessible for users introducing a {% youtube videoId="45ekUIRO1Ec" start="0:00" end="1:11" width="560px" height="315px" /%} ## Pipeline Status Widget + We are also adding another widget you can use to customize the Landing Page of the User Personas in your organization. With the Pipeline Status widget, Data Engineers can easily track the pipelines that are not behaving as expected. This widget, together with the obervability alerts that are already in place, will help your teams jump even faster to solving any issues in the platform. ## API as Data Asset + The Internet runs using APIs, both producing and consuming data. Organizations today run many microservices and REST APIs to capture data from their users and update a transaction database in the backend. On top of the many supported connectors across Databases, Dashboards, ML Models, etc. We believe that providing support for API Services as data assets will help to get the full picture of how the data is coming through from various services and landing into databases, going to warehouses and BI tools. @@ -393,20 +425,25 @@ In 1.5.1 we are introducing APIs as another first-class entity. Teams can now ca {% youtube videoId="b9wrVnM3u80" start="0:00" end="0:33" width="560px" height="315px" /%} ## Glossary Improvements + OpenMetadata supports multiple glossaries, an import/export and review process, and bulk asset tagging with glossary terms. Many teams are taking advantage of these features, and with an amazing open-source community, we are receiving great feedback on improving glossary functionality. Here are some of the improvements coming in 1.5.1: + 1. Glossary Reviewers can be teams 2. Updating a glossary will enforce a re-review 3. Renaming the Glossary Term while it's under review will keep the task associated with it open ## Data Insights (Collate) + The Data Insights application is meant to give you a quick glance of your data's state and allow you to take action based on the information you receive. To continue pursuing this objective, the application was completely refactored to allow customizability. This is achieved by the possibility of now creating custom dashboards. On this release you can create charts based on your data assets metadata based on your needs. ## Ingestion Connectors + 80+ connectors to help teams to centralize metadata. We continue to push the boundaries of this mission, in + - **Apache Flink** as a Pipeline Connector - **SAP ERP**, after a long and successful collaboration with our community and SAP experts - **Teradata** as a community contribution from [gbpy](https://github.com/gpby) to broaden the integration capabilities for enterprise-scale analytics and data management. @@ -420,7 +457,7 @@ To continue pursuing this objective, the application was completely refactored t **August 6th, 2024** {% /note %} -- Make `Include ddl` disabled by default +- Make `Include ddl` disabled by default - Made DDL configuration consistent with views - Fix user profile task listing. - Fix import/export UI flow ${CollateIconWithLinkMD}. @@ -570,11 +607,13 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta ## Backward Incompatible Changes ### Tooling + - Metadata Backup/Recovery is deprecated. No further support will be provided. - Users are advised to use database native tools to backup and store it in their object store for recovery. - `bootstrap/bootstrap_storage.sh` has been deprecated in favor of bootstrap/openmetadata-ops.sh ### UI + - Activity has been improved. New update specific cards display critical information such as data quality test case updates, description, tag update or removal. - For Lineage, the Expand All button has been removed. A new Layers button is introduced at the bottom left corner. With the Layers button, you can add Column Level Lineage or Data Observability details to your Lineage view. - View Definition is now renamed as Schema Definition. @@ -582,6 +621,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - For Classification, users can set classification to be mutually exclusive only at the time of creation. Once created, you cannot change it back to mutually non-exclusive or vice-versa. This is to prevent conflicts of adding multiple tags that belong to same classification and later turning the mutually exclusive flag back to true. ### API + - Table Schema's `ViewDefinition` is now renamed to `SchemaDefinition` to capture Tables' Create Schema. - Bulk Import API now creates entities if they are not present during the import. - Table's TestSuite is migrated to EntityReference. Previously it used to store entire payload of TestSuite. @@ -605,6 +645,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - APIs are available in OSS. ## Data Quality Improvements + {% youtube videoId="UNOHvBMVcYM" start="0:00" end="1:28" width="560px" height="315px" /%} - The Table schema page now shows the Data Quality tests for each column. @@ -615,17 +656,20 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Support has been added for an empty string as a missing count. ## Data Profiler + - Implemented a global profiler configuration page, allowing admin to exclude certain metric computations for specific data types. - Added profiler support for Redshift complex types and DynamoDB. - Fixed an issue with performing sum operations for large values in profiler ingestion. - Fixed the histogram unit's issues with scientific notation. ## Incident Manager + - We now display a sample of failed rows for the latest failed test cases. Once the issue is resolved, the failed sample will be deleted. (Collate Only) - Fixed the Date time filter for the Incident Manager. - Notifications are sent for the tasks created by the Incident Manager. ## Lineage Improvements + https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Column Lineage Search @@ -650,10 +694,12 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Support has been added for dynamic tables. ## Data Insights + - Previously, the data insights reports displayed only the percentage coverage of ownership and description. Now, users can drill down to view the data assets with no owner or description. - Improved the UX for data insight filters. ## Cost Analysis (Collate Only) + - Lifecycle data for Cost Analysis has been implemented for BigQuery, Snowflake, and Redshift. ## Custom Theme @@ -671,10 +717,12 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Added a Data Quality Widget to list the summary of data quality tests belonging to a user or their team. ## Ingestion Performance Improvements + - Bigquery, Redshift, and Snowflake now support incremental metadata ingestions by scanning DML operations on the query history. - Database Services now support parallelizing the metadata ingestion at each schema. ## Connectors + - Now supports a new connector for QlikCloud. - New Kafka Connect connector - We now parse complex protobuf schemas for Kafka @@ -683,7 +731,7 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Added an option to include or exclude paused pipelines in Airflow. - Revamped SSL support to allow users to upload the required certificates directly in the UI. - The character support has been enhanced for tag ingestion to include /. -- In the Oracle connector, we rolled back to use all_ tables instead of dba_. +- In the Oracle connector, we rolled back to use all* tables instead of dba*. - Added support for Azure auth in Trino. - For QlikSense, we have added an option to disable SSL validation. @@ -694,25 +742,31 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Custom Properties now allow linking other assets in the platform, such as Tables, Dashboards, etc. To enable this, create a Custom Property as an Entity Reference or Entity Reference List. ## Glossary + - The glossary term parent can now be changed from the Details page. - On the data assets page, glossary terms are displayed by hierarchy. ## Alerts & Notification Improvements + - The Activity Feed provides more contextual information, removing the need to move to entity pages. - Alerts give more accurate information about the entity, as well as conversations and tasks. ## Localization + - Fixed localization issues in the confirmation logic for the delete function. - Fixed the search index language configuration. ## Roles + - Now, roles can be inherited from the user configuration in SSO. ## Search + - You can now filter by assets without a description or an owner. - Improved the match results for search results. ## Others + - The description is auto-expanded when the data asset has no data and has the space to accommodate a lengthy description. - User email IDs have been masked and are only visible to Admins. - Users can filter Queries by owner, tag, and creation date in the UI. @@ -723,7 +777,6 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Redundant scroll bars have been removed from the UI. - Improved the bot role binding to provide more control over which roles are passed to the system bots. - # 1.3.4 Release 🎉 {% note noteType="Tip" %} @@ -762,21 +815,25 @@ Learn how to upgrade your OpenMetadata instance to 1.3.3! You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.3.2-release). ## MetaPilot (Collate) + - New MetaPilot application shipped in preview mode. Try it out in the [Sandbox](https://sandbox.open-metadata.org/)! - Get automatically generated descriptions with GenAI. Now it’s easier than ever to document your data assets. - Chat with the MetaPilot and get SQL queries to help you extract relevant information from your data assets. - Let the MetaPilot help you understand and improve the queries used on your tables. ## Authentication Flow + - Added generic support for OIDC Authentication. This is SSO provider-agnostic. - You can now integrate Confidential Clients to manage the server authentication. - Now, the session renewal happens automatically in the backend. ## Data Quality + - Pagination support was added for the Data Quality tab for data assets. - Fixed an issue with execution summary timeout issue for the data quality test. ## Connectors + - New Bigtable connector. - Now, users can configure the external sample data storage path. - Added lineage support for Snowflake materialized view and masking policies. @@ -787,6 +844,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Improve PATCH generation for array fields. ## Other Changes + - Avoid creating duplicated queries. - Speed up the server start time by moving the Secrets Manager Migration to the migration container. - Fixed the issue with the date filter for the Incident Manager. @@ -795,6 +853,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fixed an issue with search indexing. - Fixed the missing input field for conversation source for alerts and notifications. - Filter dashboards by a project on the Explore page. + --- **Full Changelog**: [link](https://github.com/open-metadata/OpenMetadata/compare/1.3.1-release...1.3.2-release) @@ -807,30 +866,36 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.3.1-release). {% /note %} - ## Knowledge Center (Collate) + - Supports drag and drop for the hierarchy of knowledge articles. - Enhanced the layout and loading experience of the knowledge page. ## Lineage + - When adding a new node in Lineage, the Display Name is supported in search. - Fixed the issues with displaying lineage from Metabase. ## Glossary + - Improved the automation of performance tests for Glossary. - Performance improvements to display a large Glossary. ## Data Insights + - Data Insights report has been improved. - The cost Analysis report has been optimized. ## Notifications + - The format for Slack notifications has been improved. ## Custom Properties + - Added enum type support for custom properties. ## Connectors + - Now BigQuery connector supports Primary, Foreign, and Unique Constraints. It fetches the column description for views. - Captures the SQL query that powers a Tableau DataModel. - Azure Key Vault is supported as a Secrets Manager. @@ -840,6 +905,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fixed an issue with the service display name after ingestion. ## Other Changes + - The functionality for mutually exclusive tags has been disabled. - PodGC set up for Argo workflows to delete the pods from the Kubernetes environment on a successful run of the pods. - Fixed the issue with the display of the personal access token. @@ -850,11 +916,9 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fixed the issues with testing the email settings. - Fixed an issue with adding tags. - - # 1.3.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2024, February 5th** [OpenMetadata 1.3 Release - Intuitive Lineage UI, Data Observability Alerts, Data Quality Incident Manager, Custom Metrics for Profiler, Knowledge Center Improvements, and lots more](https://blog.open-metadata.org/openmetadata-release-1-3-ac801834ee80) @@ -920,12 +984,14 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Custom metrics can be created at Table and Column levels. ## Profiler and Data Quality + - The Profiler has been improved to support sample data ingestion without computing other metrics. - Admins can configure the profiler to fetch up to 10,000 rows of sample data. - Sample data can be stored in S3 buckets. - Refined the default time range on the test case results page, adjusting it from the Last 3 days to the Last 30 days for a more encompassing view. ## Connectors + - New Google Cloud Storage for storage services. (Collate) - New Alation connector to migrate metadata into Collate. (Collate) - New Iceberg, SAS Viya, and Doris connectors. @@ -940,6 +1006,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - We now extract table descriptions when ingesting metadata from Salesforce. ## Glossary + - Supports soft delete for the default glossaries in OpenMetadata. - Supports the creation of tasks to request tags or a description. - Only the Owner can edit the Glossary term. @@ -959,10 +1026,12 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - The Settings page UI has been revamped. ## Data Insights + - Cost Analysis expanded to support BigQuery & Redshift. (Collate) - Improved the Data Insights Report sent via email. ## Other Changes + - Announcements can be notified over email, Slack, or Teams. - Alerts are sent to a user when they are mentioned in a task or activity feed. - We have improved the display of search results for column matches. When searching for columns, the matched results will be displayed and highlighted in the Preview pane. @@ -978,7 +1047,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # 1.2.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, October 26th** [OpenMetadata 1.2 Release - Domains, Data Products, Search Index, Stored Procedures, Glossary Approval Workflow, Customizable Landing Page, Applications, Knowledge Center, Cost Analysis, and lots more](https://blog.open-metadata.org/openmetadata-release-1-2-531f0e3c6d9a) @@ -997,10 +1066,12 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Assets can also be added as Data Products in a Domain. ## Search Index + - Elasticsearch or Opensearch connectors can now bring in the search index metadata into OpenMetadata. - The connector will populate the index’s mapping, settings, and sample data. ## Stored Procedures + - Added support for Stored Procedures. - Snowflake, Redshift, and BigQuery connectors are updated to bring stored procedure metadata into OpenMetadata. - The metadata workflow will bring the Stored Procedures and parse their executions to extract lineage information. @@ -1018,8 +1089,9 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Color coding helps to visually differentiate and identify the data assets, when glossary terms are added to them. ## OpenMetadata Browser Extension + - Updated the Chrome browser extension for OpenMetadata with the new UI. -- Added support for Databases, Database Schemas, Tables, Dashboards, Charts, Pipelines, and Topics. +- Added support for Databases, Database Schemas, Tables, Dashboards, Charts, Pipelines, and Topics. ## Build Automation Applications @@ -1031,11 +1103,13 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - We will continue to add new Applications in upcoming releases. ## Lineage + - Performance improvements made for lineage based on the new release of SQLfluff. - Added support for `UPDATE … FROM Snowflake` queries - Added column-level lineage support for `SELECT *` queries ## Connectors + - Greenplum connector is now supported. - Couchbase connector is now supported. - Azure Data Lake Storage is now supported. (Collate) @@ -1066,21 +1140,24 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # 1.1.2 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, August 24th** {% /note %} ## Data Quality + - Added support for Postgres version 11.19. - Fixed MariaDB time column issues. ## Connectors + - Added JWT authentication support for Trino. - Fixed Snowflake connection test. - Fixed SageMaker ingestion. - Added external table support for BigQuery. ## UI Improvements + - Added Russian language support. - Supports Delete functionality for sample data. - Improved Schema page UX. @@ -1088,9 +1165,11 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fixed the version history list. ## Ingestion + - Improved performance when ingesting table constraints. ## Backend + - Improved Glossary import validations. - Fixed Test Suite migrations and naming. - Fixed Classification migration. @@ -1099,7 +1178,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # 1.1.1 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, August 7th** {% /note %} @@ -1112,15 +1191,18 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Support custom cron for schedule ingestion ## Data Quality + - Fix BigQuery, MSSQL, and Clickhouse profiling errors ## Ingestion + - Fixed Airflow lineage extraction. - Added support for Databricks complex columns comments. - Fixed Athena lineage and usage parameter validation. - Airflow Managed APIs now support Airflow 2.6 ## Connectors + - New [Qliksense](qlik.com) Connector. - Hive supports extracting metadata directly from the metastore to speed up the execution. Users whose metastore is not exposed can still run the extraction pointing to Hive. - Added Usage & Lineage connector for Trino. @@ -1129,14 +1211,14 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Added support for JSON fields in SingleStore. ## Backend + - Bumped table and column names length - Aggregation Improvements for Search - Test Suite Improvements - # 1.1.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, June 30th** [OpenMetadata 1.1.0 Release - UI Overhaul, New Connectors, Improved Lineage Parsing, PII Masking, and lots more](https://blog.open-metadata.org/openmetadata-1-1-0-release-97c1fb603bcf) @@ -1185,84 +1267,97 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Improved monitoring of the Pipeline Service Client health. Any status errors between the OpenMetadata server and the Pipeline Service Client are now surfaced in a Prometheus metric `pipelineServiceClientStatus_counter_total` - Added AWS OpenSearch client-specific support. This allows us to update the Elasticsearch version support up to 7.16. - # 1.0.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, April 25th** [OpenMetadata 1.0 Release - Improved Schemas & APIs, Ingestion Improvements, Storage Services, Dashboard Data Models, Auto PII Classification, Localization, and much more](https://blog.open-metadata.org/openmetadata-1-0-release-beb34762d916) {% /note %} ## APIs & Schema + - **Stabilized** and improved the Schemas and APIs. - The APIs are **backward compatible**. ## Ingestion + - Connecting to your data sources has never been easier. Find all the necessary **permissions** and **connection details** directly in the UI. - When testing the connection, we now have a comprehensive list of **validations** to let you know which pieces of metadata can be extracted with the provided configuration. - **Performance** improvements when extracting metadata from sources such as Snowflake, Redshift, Postgres, and dbt. - New **Apache Impala** connector. ## Storage Services + - Based on your [feedback](https://github.com/open-metadata/OpenMetadata/discussions/8124), we created a new service to extract metadata from your **cloud storage**. - The Data Lake connector ingested one table per file, which covered only some of the use cases in a Data Platform. With **Storage Services**, you can now present accurate metadata from your tables, even when **partitioned**. - The first implementation has been done on **S3**, and we will keep adding support for other sources in the upcoming releases. ## Dashboard Data Models + - Dashboard Services now support the concept of **Data Models**: data that can be directly defined and managed in the Dashboard tooling itself, e.g., LookML models in Looker. - Data Models will help us close the gap between engineering and business by providing all the necessary metadata from sources typically used and managed by analysts or business users. - The first implementation has been done for **Tableau** and **Looker**. ## Queries + - Improved UI for **SQL Queries**, with faster loading times and allowing users to **vote** for popular queries! - Users can now create and share a **Query** directly from the UI, linking it to multiple tables if needed. ## Localization + - In 1.0, we have added **Localization** support for OpenMetadata. - Now you can use OpenMetadata in **English**, **French**, **Chinese**, **Japanese**, **Portuguese**, and **Spanish**. ## Glossary + - New and Improved **Glossary UI** - Easily search for Glossaries and any Glossary Term directly in the **global search**. - Instead of searching and tagging their assets individually, users can add Glossary Terms to multiple **assets** from the Glossary UI. ## Auto PII Classification + - Implemented an automated way to **tag PII data**. - The auto-classification is an optional step of the **Profiler** workflow. We will analyze the column names, and if sample data is being ingested, we will run NLP models on top of it. ## Search + - **Improved Relevancy**, with added support for partial matches. - **Improved Ranking**, with most used or higher Tier assets at the top of the search. - Support for **Classifications** and **Glossaries** in the global search. ## Security + - **SAML** support has been added. - Added option to mask passwords in the API response except for the `ingestion-bot` by setting the environment variable `MASK_PASSWORDS_API=true`. More info [here](/deployment/security/enable-password-masking). - **Deprecation Notice**: **SSO** Service accounts for Bots will be deprecated. **JWT** authentication will be the preferred method for creating Bots. ## Lineage + - Enhanced Lineage UI to display a large number of **nodes (1000+)**. - Improved UI for **better navigation**. - Improved **SQL parser** to extract lineage in the Lineage Workflows. ## Chrome Browser Extension + - All the metadata is at your fingertips while browsing Looker, Superset, etc., with the OpenMetadata Chrome Browser Extension. - **Chrome extension** supports Google SSO, Azure SSO, Okta, and AWS Cognito authentication. - You can Install the Chrome extension from **Chrome Web Store**. ## Other Changes + - The **Explore page** cards will now display a maximum of **ten tags**. - **Entity names** support apostrophes. - The **Summary panel** has been improved to be consistent across the UI. # 0.13.3 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, March 30th** {% /note %} ## Ingestion Framework + - Datalake Avro & Json, JsonZip support - BigQuery Profiler Ingestion for all regions - Support for Snowflake Geometry Type @@ -1281,57 +1376,69 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Support for DBT manifest V8 ## Roles & Policies + - A Non-Privileged user can add new 'Roles' to Teams - Fix Permissions API to consider the leaf nodes tags as well, example: table's column tags ## Search + - Improve Search Relevancy, by adding functional scoring and add ngram analyzer; - Enable search for entities using both name and displayName ## Security + - Enable LDAP configuration to be configured via environment variable - LDAP-s support connection without MTLS ## EntityName + - Relax data asset name restrictions to allow the special characters except "::" - Allow unicode character and digits in Entity ## Data Quality + - Fix column values between test # 0.13.2 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, January 30th** [OpenMetadata 0.13.2 Release - Improved SQL Lineage, Glossary Bulk Upload, Unified Tag Category API, Mutually Exclusive Tags, Chrome Extension, and lots more](https://blog.open-metadata.org/openmetadata-0-13-2-release-e32c0de93361) {% /note %} ## Improved SQL Lineage + - We have collaborated with the [sqllineage](https://github.com/reata/sqllineage) and [sqlfluff](https://www.sqlfluff.com/) communities - to improve the parsing capabilities of `sqllineage`. We'll continue to collaborate to ship further improvements in new releases. + to improve the parsing capabilities of `sqllineage`. We'll continue to collaborate to ship further improvements in new releases. ## New Glossary UI + - Moved from a tree view in the left panel to an easy to navigate list of the terms sorted alphabetically. - The term list shows the tags and descriptions in the cards. ## Glossary Import & Export + - You can now export your Glossary data as a CSV file. - In the same way, you can now bulk upload terms to a Glossary by adding their details in a CSV file. - The import utility will validate the file and show you a preview of the elements that are going to be imported to OpenMetadata. ## Unified Tag Category API + - Renamed Tag Categories to Classification, a more widely used term. - Updated the API to conform with the rest of the specification. More info [here](https://github.com/open-metadata/OpenMetadata/issues/9259). ## Mutually Exclusive Tags + - When creating a Classification or a Glossary term, you can now make the tags to be mutually exclusive. - If tags are set to be mutually exclusive, you won't be able to set multiple tags from the same category in the same asset. ## EntityName + - Special characters ## Ingestion Framework + - Performance Improvements: We are now getting descriptions in batch, making connectors such as Redshift or Snowflake way faster! - The Oracle connector now ships with the Thick mode enabled. - AWS QuickSight fixes @@ -1341,36 +1448,41 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # 0.13.1 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, December 20th** {% /note %} ## Profiler and Data Quality + - Freshness Metric has been introduced. Data freshness shows DML operations performed against a table and the number of rows affected. All this is displayed within the data profiler with filterable graphs. This is currently supported for BigQuery, Snowflake, and Redshift. - Support has been added for data quality tests on Data Lake. - UI has been improved to show table and column profile data on separate page. Legend is now selectable to filter for specific metrics ## Alerts and Notification + The logic for Notification Support has been improved. Users can define Alerts based on a Trigger (all data assets or a specific entity), Filters (events to consider), and Action (Slack, MS Teams, Email, Webhook) on where to send the alert. ## Ingestion -- Now, dbt has its own workflow. Previously, dbt was a part of metadata ingestion workflow. + +- Now, dbt has its own workflow. Previously, dbt was a part of metadata ingestion workflow. - Airflow Lineage Operator and the OpenMetadata Hook are now part of the ingestion package. Send Airflow metadata from your DAGs and safely store the OpenMetadata server connection directly in Airflow. - Multiple Databases (catalog) is now supported for the Databricks connector - Azure blob is now supported to backup your metadata into ## New Connectors + - OpenMetadata now supports Azure Datalake Storage Gen 2 ## General Improvements + - Users can update the description and tags for Topic Schema. Previously, the topic schemas were read-only. We now support Avro/Protobuf parsing and field level details for topic schemas. -- The layout for the Data Insight Report has been improved. We now display a line graph instead of a bar graph. The Most Viewed Data Assets are clickable to view the asset details page. +- The layout for the Data Insight Report has been improved. We now display a line graph instead of a bar graph. The Most Viewed Data Assets are clickable to view the asset details page. - Improvements have been made to Advanced Search. Now, when a filter is applied, the details of the filter selected are displayed for clarity. - On the Explore page UI, the Side Preview is now available for all data assets. Previously it was only displayed for tables. # 0.13.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, December 8th** [OpenMetadata 0.13.0 Release — Data Insights & KPIs, Lineage Traceability, Data Lake Profiler, Search Improvements, and lots more](https://blog.open-metadata.org/openmetadata-0-13-0-release-ac8ac5bd87c1) @@ -1379,58 +1491,70 @@ The logic for Notification Support has been improved. Users can define Alerts ba {% youtube videoId="oNbMnTW5AkE" start="0:00" end="7:51" width="560px" height="315px" /%} ## Data Insights and KPI + Data Insight allows admins to take an active approach in their metadata management. Data Insight provides a single-pane view of all the key metrics to best reflect the state of your data. Admins can define the Key Performance Indicators (KPIs) and set goals within OpenMetadata to work towards better documentation, ownership, and tiering. Alerts can be set against the KPIs to be received on a specified schedule. ## Lineage + The lineage UI has been transformed to enhance user experience. Users can get a holistic view of an entity from the Lineage tab. When an entity is selected, the UI displays end-to-end lineage traceability for the table and column levels. ## Profiler + With the OpenMetadata UI, users can now create and deploy profiling workflows for the Datalake connector, which supports AWS S3 and GCS ## SSO + Support for LDAP SSO has been added in this release ## Advance Search + Syntax Editor has been introduced for advanced search with And/Or conditions that help discover assets quickly ## New Connectors + - AWS SageMaker - AWS QuickSight - AWS Kinesis - Domo ## Messaging Service Schemas Improvements + Major enhancements have been made to how data is extracted from Kafka and Redpanda Messaging services. Previously, OpenMetadata extracted all the Topics in the messaging queue and also connected to the Schema Registry to get the Schemas. These schemas were taken as one payload and published to OpenMetadata. We now parse Avro and Protobuf Schemas to extract the fields. Now, users can document each of these fields within a schema by adding descriptions and tags. Users can search based on the fields in the Schema of a Topic. ## General Improvements + - Soft deleted entities can be restored. Currently, only the ML Models are not supported. - Soft deleted teams can be restored. When restoring a soft deleted parent team, the child teams will not be restored by default. # 0.12.3 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, November 18th** {% /note %} ## Bug Fixes + - User suggestion index mapping - Tag and Glossary terms caching # 0.12.2 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, October 20th** {% /note %} ## Ingestion + - Databricks lineage - Added support for Airflow version 2.2.2 as a workflow scheduler + ## Bug Fixes + - Support same table across different databases for the profiler # 0.12.1 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, October 3rd** {% /note %} @@ -1438,7 +1562,7 @@ Major enhancements have been made to how data is extracted from Kafka and Redpan - User/Password signup and login - Email notifications for forgotten password and new user signed up -- Admin can add new users and send an email +- Admin can add new users and send an email ## ElasticSearch full re-index through UI @@ -1452,17 +1576,17 @@ Major enhancements have been made to how data is extracted from Kafka and Redpan - We support ingesting DBT tags into OpenMetadata -## Bots Integration +## Bots Integration - Admins can create bots and their security mechanism from UI itself ## Bug Fixes -- Around 136 Features/Improvements/Tests made it into 0.12.1 release +- Around 136 Features/Improvements/Tests made it into 0.12.1 release # 0.12.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, September 7th** [OpenMetadata 0.12.0 Release](https://blog.open-metadata.org/openmetadata-0-12-0-release-1ac059700de4) @@ -1471,6 +1595,7 @@ Major enhancements have been made to how data is extracted from Kafka and Redpan {% youtube videoId="tv3pyCLcJfQ" start="0:00" end="17:04" width="560px" height="315px" /%} ## Team Hierarchy + Prior releases supported a flat hierarchy of just Teams and Users. In 0.12, support has been added for the entire organizational hierarchy with Business Unit, Division, Department, and Groups. An organization from small to very large can now be modeled in OpenMetadata with this feature. ## Roles and Policies @@ -1503,18 +1628,18 @@ In 0.12, we’ve also streamlined the Notifications menu with two separate tabs Users can get timely updates about the metadata change events for all entities through APIs using webhooks. The webhook integration with Slack has been further improved in this release. -OpenMetadata also supports webhook integration to Microsoft Teams, just as it supports Slack. Users can choose to receive notifications for only the required entities by using event filters based on when an entity is created, updated, or deleted. +OpenMetadata also supports webhook integration to Microsoft Teams, just as it supports Slack. Users can choose to receive notifications for only the required entities by using event filters based on when an entity is created, updated, or deleted. ## Tasks In the 0.11 release, a request to add or update descriptions for data assets could be converted to a Task. In the 0.12 release, Tasks can be created based on requests to create or update tags. Also, a glossary term approval workflow can be converted to a Task. - ## Secret Management Store Interface In 0.12, we have completely revamped how that secret is stored, accessed, and by whom; by introducing a Secrets Manager Interface to communicate with any Key Management Store. The KMS will mediate between any OpenMetadata internal requirement and sensitive information. That way, users can choose to use the underlying database as KMS, or any external system. The OpenMetadata community has already added support for AWS Key Management Service and AWS SSM. ## Connectors + New connectors are an essential part of every release in OpenMetadata. We are introducing four new connectors in this release: - Redpanda is a Kafka API-compatible streaming data platform for developers that unifies historical and real-time data. OpenMetadata now supports Redpanda as a Messaging service, which allows users to document its topics and schemas. Refer to the Redpanda documentation for more info. @@ -1523,6 +1648,7 @@ New connectors are an essential part of every release in OpenMetadata. We are in - Apache NiFi automates the flow of data between systems. OpenMetadata now supports a NiFi connector as the third new pipeline service on this release. ## Lineage + We’ve enhanced the performance of workflows by having a separate workflow for Lineage and Usage. By using two workflows for computing specific pieces of information, we can effectively filter down the queries to extract lineage. During table usage ingestion, the tables retrieved successfully will be cached, so that there is no need to repeat the same calls multiple times as many queries would be referencing the same tables. @@ -1530,10 +1656,11 @@ Usage queries have been optimized. A result limit has been added to Usage queries. ## Global Settings + The OpenMetadata Settings dropdown menu has been transformed into a single, centralized Settings page for added convenience in viewing all the available options. The Global Settings comprises setting options for Team Members, Access based on Roles and Policies, Services, Data Quality, Collaboration, Custom Attributes, and Integrations for webhooks and bots. Admins can view or update settings for various services like Slack, MS Teams, Webhooks, etc from the Global Settings page. - ## UI/UX Improvements + The major UI UX improvements have been done around Roles and Policies and a Global Settings page. Quite a lot of tweaks have been made to the UI to improve the UX. When creating a new user or when a user is registering for the first time, the dropdown menu for Teams now displays an option to ‘Show All’ teams. Previously, we supported the display of only the first 10 teams. An option has also been provided to search and filter. @@ -1542,13 +1669,14 @@ Manage Tab has been replaced with the manage button on the UI. # 0.11.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, July 1st** [OpenMetadata 0.11.0 Release](https://blog.open-metadata.org/openmetadata-0-11-release-8b82c85636a) {% /note %} ## Data Collaboration - Tasks, Announcements, & Emojis + - Tasks have been introduced as an extension to the ability to create conversations and post replies. - Tasks can be created around descriptions for tables, pipelines, dashboards, and topics. - Users can Request a description, or even Suggest a new description and make edits to an existing description. @@ -1560,14 +1688,17 @@ Manage Tab has been replaced with the manage button on the UI. - Task owners can provide description or accept/reject suggestions and those tasks are automatically closed. ## Column Level Lineage + - Column level lineage API support has been added in the backend. - Supports table level and column level lineage from Snowflake, Redshift, and BigQuery. ## Custom Properties + - Now supports adding new types and extending entities when organizations need to capture custom metadata. - New types and custom fields can be added to entities either using API or in OpenMetadata UI. ## Advanced Search + - Users can search by column, schema, database, owner, tag, and service. - Users can search by multiple parameters to narrow down the search results. - Separate advanced search options are available for Tables, Topics, Dashboards, Pipelines, and ML Models. @@ -1575,11 +1706,13 @@ Manage Tab has been replaced with the manage button on the UI. - Entity specific search options are also available - table specific options include Column, Schema, and Database, pipeline specific options include Task, and dashboards specific option includes Chart. ## Glossary UI Updates + - The Glossary UI has been upgraded. - The arrangement to display the Summary, Related Terms, Synonyms, and References has been changed. - Reviewers are shown on the right panel with an option to add or remove existing reviewers. ## Profiler and Data Quality Improvements + - Seven additional data quality tests have been added as follows. - - tableColumnCountToBeBetween: Ensure the number of columns in your table stays within the expected range - - tableColumnNameToExist: Check that a specific column is in your table @@ -1593,10 +1726,12 @@ Manage Tab has been replaced with the manage button on the UI. - Developed a direct integration between Great Expectations and OpenMetadata. Now, you can add custom actions to your Great Expectations checkpoints file that will automatically ingest your data quality tests results into OpenMetadata at the end of your checkpoint file run. ## ML Models + - ML Model entities have been added to the UI. - Supports ingestion through the UI from MLflow. ## Connectors + - Five new connectors have been added - Airbyte, Mode, AWS Data Lake, Google Cloud Data Lake, and Apache Pinot. - DBT Cloud support was added and we now extract manifest and catalog files from API. - The ingestion scheduler now supports a minute level selection. @@ -1604,6 +1739,7 @@ Manage Tab has been replaced with the manage button on the UI. - The Looker connector now fetches the ‘Usage’ and ‘Access’ metadata for Dashboards and Charts. ## UI Improvements + - The OpenMetadata UI has a new layout. - In the Activity Feeds, the options to reply to a conversation, as well as to delete can now be found on hovering over the conversation. - Users can react with Emojis on the activity feeds, conversations and replies. @@ -1612,6 +1748,7 @@ Manage Tab has been replaced with the manage button on the UI. - A tooltip has been added to display the FQN on hover in the Activity Feed header. ## Other Changes + - Admin users define Roles and associate these roles to Teams. When a user picks a Team, the Role gets automatically assigned. - An option has been added to recreate a fresh index from the data available in Elasticsearch. - A simple webhook server has been added to the metadata command to register and listen to the metadata change events. @@ -1622,7 +1759,7 @@ Manage Tab has been replaced with the manage button on the UI. # 0.10.1 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, May 17th** {% /note %} @@ -1634,7 +1771,7 @@ Manage Tab has been replaced with the manage button on the UI. # 0.10.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, April 27th** [OpenMetadata 0.10.0 Release](https://blog.open-metadata.org/openmetadata-0-10-0-release-82c4f5533c3f) @@ -1642,7 +1779,7 @@ Manage Tab has been replaced with the manage button on the UI. ## Support for Database Schema -OpenMetadata supports databases, service name databases, and tables. We’ve added Database Schema as part of the FQN. +OpenMetadata supports databases, service name databases, and tables. We’ve added Database Schema as part of the FQN. For each external data source, we ingest the database, as well as the tables that are contained underneath the schemas. ## Support for Hard Delete @@ -1652,39 +1789,39 @@ and ingestion. Hard deleting an entity removes the entity and all of its relatio ## Deploy Ingestion from UI -OpenMetadata has refactored the service connections to simplify the ingestion jobs from both the ingestion framework +OpenMetadata has refactored the service connections to simplify the ingestion jobs from both the ingestion framework and the UI. We now use the pydantic models automatically generated from the JSON schemas for the connection definition. The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata. ## Download dbt Manifest Files from Amazon S3 or Google Cloud Storage -Previously, when ingesting the models and lineage from dbt, we passed the path of the dbt manifest and catalog files -directly into the workflow. We’ve worked on improving the quality of life of dbt. Now, we can dynamically download -these files from Amazon S3 or Google Cloud Storage. This way we can have any other process to connect to the dbt, -extract the catalog, and put it into any cloud service. We just need the path name and workflow job details from the +Previously, when ingesting the models and lineage from dbt, we passed the path of the dbt manifest and catalog files +directly into the workflow. We’ve worked on improving the quality of life of dbt. Now, we can dynamically download +these files from Amazon S3 or Google Cloud Storage. This way we can have any other process to connect to the dbt, +extract the catalog, and put it into any cloud service. We just need the path name and workflow job details from the metadata extraction to be able to ingest metadata. ## JSON Schema based Connection Definition -Each service (database, dashboard, messaging, or pipeline service) has its own configuration specifications, with some -unique requirements for some services. Instead of the ad hoc definitions of the source module in Python for each +Each service (database, dashboard, messaging, or pipeline service) has its own configuration specifications, with some +unique requirements for some services. Instead of the ad hoc definitions of the source module in Python for each connector, we’ve worked on the full refactoring of the ingestion framework. We now use the pydantic models automatically generated from the JSON schemas for the connection definition. ## Airflow Rest APIs -The Airflow REST APIs have been refactored. With our API centric model, we are creating a custom airflow rest API +The Airflow REST APIs have been refactored. With our API centric model, we are creating a custom airflow rest API directly on top of Airflow using plugins. This passes the connection information to automatically generate all the dags and prepares handy methods to help us test the connection to the source before creating the service. ## UI Changes - The UI improvements are directed toward providing a consistent user experience. -- Hard Deletion of Entities: With the support for the hard deletion of entities, we can permanently delete tables, - topics, or services. When the entity is hard deleted, the entity and all its relationships are removed. +- Hard Deletion of Entities: With the support for the hard deletion of entities, we can permanently delete tables, + topics, or services. When the entity is hard deleted, the entity and all its relationships are removed. This generates an ‘EntityDeleted’ change event. -- Dynamic “Add Service” Forms: The ‘Add Service’ form is automatically generated in the UI based on the JSON +- Dynamic “Add Service” Forms: The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata. - UI Support for Database Schema as part of FQN: The database schema has been introduced in the 0.10 release. All the entity pages now support Database Schema in the UI. @@ -1695,17 +1832,18 @@ and prepares handy methods to help us test the connection to the source before c - Add User: A user can be added from the Users page. ## Security Changes -- **Support Refresh Tokens for Auth0 and Okta SSO**: The JWT tokens generated by the SSO providers expire by default - in about an hour, making the user re-login often. In this release, we’ve added support for refresh tokens for Auth0 + +- **Support Refresh Tokens for Auth0 and Okta SSO**: The JWT tokens generated by the SSO providers expire by default + in about an hour, making the user re-login often. In this release, we’ve added support for refresh tokens for Auth0 and Okta SSO. The tokens are refreshed silently behind the scenes to provide an uninterrupted user experience. In future releases, we’ll continue to stabilize authentication and add refresh tokens for the other SSO providers. -- **Custom OIDC SSO**: OpenMetadata now supports integration with your custom-built OIDC SSO for authentication. +- **Custom OIDC SSO**: OpenMetadata now supports integration with your custom-built OIDC SSO for authentication. This is supported both on the front end for user authentication and on the ingestion side. - **Azure SSO**: Support has been added for Azure SSO on Airflow. # 0.9.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, March 10th** [OpenMetadata 0.9.0 Release](https://blog.open-metadata.org/openmetadata-0-9-0-release-8e7b93ab1882) @@ -1720,6 +1858,7 @@ and prepares handy methods to help us test the connection to the source before c - Table details - Click through on usage to see who or what services are using it, what queries are pulling from it. ## Data Quality + - Ability to create and monitor the test cases. - Data Quality Tests support with Json Schemas and APIs. - UI Integration to enable user to write tests and run them on Airflow. @@ -1729,10 +1868,11 @@ and prepares handy methods to help us test the connection to the source before c - Glossaries are a Controlled Vocabulary in an organization used to define the concepts and terminologies specific to a particular domain. - API & Schemas to support Glossary. -- UI support to add Glossary and Glossary Terms. +- UI support to add Glossary and Glossary Terms. - Support for using Glossary terms to annotate Entities and Search using Glossary Terms. ## Connectors + - Apache Iceberg - Azure SQL - Clickhouse @@ -1749,10 +1889,12 @@ and prepares handy methods to help us test the connection to the source before c - Amundsen, Import Metadata from Amundsen into OpenMetadata ## Lineage + - DataSource SQL Parsing support to extract Lineage - View Lineage support ## Pipeline + - Capture pipeline status as it happens ## Security @@ -1763,23 +1905,25 @@ and prepares handy methods to help us test the connection to the source before c # 0.8.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, January 22nd** [OpenMetadata 0.8.0 Release](https://blog.open-metadata.org/openmetadata-0-8-0-release-ca09bd2fbf54) {% /note %} ## Access Control Policies + - Design of Access Control Policies. - Provide Role based access control with community feedback. ## Eventing Webhook - Register webhooks to get metadata event notifications. -- Metadata Change Event integration into Slack and framework for integration into other services such as +- Metadata Change Event integration into Slack and framework for integration into other services such as Kafka or other Notification frameworks ## Connectors + - Delta Lake - Iceberg - PowerBI @@ -1787,25 +1931,29 @@ and prepares handy methods to help us test the connection to the source before c # 0.7.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2021, November 17th** [OpenMetadata 0.7.0 Release](https://blog.open-metadata.org/openmetadata-0-7-0-release-9f741b8d5089) {% /note %} ## UI - Activity Feed, Improved UX for Search + - Users will have access to Activity Feed of all the changes to the Metadata. - New and Improved UX for Search and Landing page. ## Support for Table Location + - Extract Location information from Glue, Redshift. - Show Location details on the Table Page. ## ElasticSearch Improvements + - Support SSL (including self-signed certs) enabled ElasticSearch. - New entities will be indexed into ElasticSearch directly ## Connectors + - Metabase - Apache Druid - Glue Improvements @@ -1814,88 +1962,104 @@ and prepares handy methods to help us test the connection to the source before c - Amundsen Import connector ## Other features + - Metadata Change Event integration into Slack and framework for integration into other services such as Kafka or other Notification frameworks - Delta Lake support, Databricks, Iceberg # 0.6.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2021, November 17th** [OpenMetadata 0.6.0 Release — Metadata Versioning, Events API, One-Click Ingestion, and more](https://blog.open-metadata.org/openmetadata-0-6-0-release-metadata-versioning-events-api-one-click-ingestion-and-more-4394c4f08e0b) {% /note %} ## Metadata Versioning and Eventing Framework + - Capture changes to Entity Metadata from source and user interactions as versions. - Versioned changes will be published as events for clients to consume to take actions on. ## Data Reliability + - Improvements to Data Reliability library. - Capture custom measurements through user provided SQL. ## Airflow APIs + - Airflow APIs to deploy DAGS and manage them. - UI integration to deploy ingestion workflows. ## Connectors + - AWS Glue - dbt - MariaDB # 0.5.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2021, October 19th** [OpenMetadata 0.5.0 Release is here — Lineage, Pipelines, Complex Types, Data Profiler and so much more](https://blog.open-metadata.org/openmetadata-0-5-0-1144a4000644) {% /note %} ## Support for Lineage + - Lineage related schemas and APIs. - Lineage metadata integration from AirFlow for tables. - UI changes to show lineage information to the users. ## Data Reliability + - Improvements to Data Profiler. - UI integration with Data Profiler to show how the table profile looks over the period of time. ## Complex Types + - Support complex types such as Struct, Array with nested fields. - UI support to add expand complex types and tag, add description for nested fields. ## Connectors + - Trino - Redash ## Other features + - Pipeline Entities are supported. - Integration with Airflow to extract Pipeline details. # 0.4.0 Release -{% note noteType="Tip" %} + +{% note noteType="Tip" %} **2021, September 20th** [OpenMetadata 0.4.0 Release — Dashboards, Topics, Data Reliability](https://blog.open-metadata.org/openmetadata-0-4-0-release-dashboards-topics-data-reliability-14e8672ae0f5) {% /note %} ## Support for Kafka (and Pulsar WIP) + - Support for Message Service and Topic entities in schemas, APIs, and UI. - Kafka connector and ingestion support for Confluent Schema Registry. ## Support for Dashboards + - Support for Dashboard services, Dashboards, and Charts entities in schemas, APIs, and UI. - Looker, Superset, Tableau connector, and ingestion support. ## User Interface + - Sort search results based on Usage, Relevance, and Last updated time. - Search string highlighted in search results. - Support for Kafka and Dashboards from Looker, Superset, and Tableau. ## Other features + - Pluggable SSO integration - Auth0 support. - Support for Presto. ## Work in progress + - Salesforce CRM connector. - Data profiler to profile tables in ingestion framework and show it table details page. diff --git a/openmetadata-docs/content/v1.6.x-SNAPSHOT/releases/releases/index.md b/openmetadata-docs/content/v1.6.x-SNAPSHOT/releases/releases/index.md index d4ac93c7666..fe825c9835d 100644 --- a/openmetadata-docs/content/v1.6.x-SNAPSHOT/releases/releases/index.md +++ b/openmetadata-docs/content/v1.6.x-SNAPSHOT/releases/releases/index.md @@ -19,7 +19,26 @@ version. To see what's coming in next releases, please check our [Roadmap](/rele ## Breaking Changes - The ingestion Framework now uses the OpenMetadata Ingestion Service Specification (OMISS) to specify -entrypoints to ingestion operations. [Click here](./todo-need-link) for more info. + entrypoints to ingestion operations. [Click here](./todo-need-link) for more info. + +# 1.5.9 Release + +{% note noteType="Tip" %} +**Oct 29th, 2024** +{% /note %} + +You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.5.9-release). + +# What's Changed + +- Prepare App Framework to handle application limits +- Add Query Builder widget +- Revamp MetaPilot as Collate AI and add limits (Collate only) +- Fix EntityLink for names with brackets +- Fix backend database Azure auth +- Mask Greenplum secrets on the UI + +**Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.8-release...1.5.9-release # 1.5.8 Release @@ -45,7 +64,6 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fix: Task deserialization in Airflow metadata ingestion. - Fix: Web analytic activity being reset. - **Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.7-release...1.5.8-release # 1.5.7 Release @@ -128,8 +146,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Improved AlationSink connector. - Fixed sktime version to fix AUT - Fixed Expected ColumnLineage but got dict -- Improved Collate API with Knowledge Center routes. ${CollateIconWithLinkMD} - +- Improved Collate API with Knowledge Center routes. ${CollateIconWithLinkMD} **Full Changelog**: https://github.com/open-metadata/OpenMetadata/compare/1.5.5-release...1.5.6-release @@ -181,6 +198,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # What's Changed ## OpenMetadata + - Hotfix to the Term Aggregation size on Data Insights - ES pagination with error handling - Updated Domain in Docker Compose & Docs @@ -192,6 +210,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Unable to access the import glossary page ## Collate + - Fix token limitations using config - Fix Automator pagination - Fix MetaPilot push for no constraint @@ -261,17 +280,20 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # Backward Incompatible Changes ## Multi Owners + OpenMetadata allows a single user or a team to be tagged as owners for any data assets. In Release 1.5.1, we allow users to tag multiple individual owners or a single team. This will allow organizations to add ownership to multiple individuals without necessarily needing to create a team around them like previously. This is a backward incompatible change, if you are using APIs, please make sure the owner field is now changed to “owners” ## Import/Export Format + To support the multi-owner format, we have now changed how we export and import the CSV file in glossary, services, database, schema, table, etc. The new format will be user:userName;team:TeamName If you are importing an older file, please make sure to make this change. ## Pydantic V2 + The core of OpenMetadata are the JSON Schemas that define the metadata standard. These schemas are automatically translated into Java, Typescript, and Python code with Pydantic classes. In this release, we have [migrated](https://docs.pydantic.dev/latest/migration/) the codebase from Pydantic V1 to Pydantic V2. @@ -321,14 +343,17 @@ In the latest release, several updates and enhancements have been made to the JS - **Tableau**: Enhanced support by adding capabilities for connecting with `TableauPublishedDatasource` and `TableauEmbeddedDatasource`, providing more granular control over data visualization and reporting. ## Include DDL + During the Database Metadata ingestion, we can optionally pick up the DDL for both tables and views. During the metadata ingestion, we use the view DDLs to generate the View Lineage. To reduce the processing time for out-of-the-box workflows, we are disabling the include DDL by default, whereas before, it was enabled, which potentially led to long-running workflows. ## Secrets Manager + Starting with the release 1.5.1, the JWT Token for the bots will be sent to the Secrets Manager if you configured one. It won't appear anymore in your dag_generated_configs in Airflow. ## Python SDK + The `metadata insight` command has been removed. Since Data Insights application was moved to be an internal system application instead of relying on external pipelines the SDK command to run the pipeline was removed. # What's New @@ -353,11 +378,13 @@ We also have improved the Table Data quality dashboard to showcase the tests cat {% youtube videoId="bXcQBtZuyoU" start="0:00" end="2:10" width="560px" height="315px" /%} ## Freshness Data Quality Tests (Collate) + Working with old data can lead to making wrong decisions. With the new Freshness test, you can validate that your data arrives at the right time. Freshness tests are a critical part of any data team's toolset. Bringing these tests together with lineage information and the Incident Manager, your team will be able to quickly detect issues related to missing data or stuck pipelines. {% youtube videoId="QRcR3m9cCGo" start="0:00" end="1:09" width="560px" height="315px" /%} ## Data Diff Data Quality Tests + Data quality checks are important not only within a single table but also between different tables. These data diff checks can ensure key data remains unchanged after transformation, or conversely, ensure that the transformations were actually performed. We are introducing the **table difference data quality test** to validate that multiple appearances of the same information remain consistent. Note that the test allows you to specify which column to use as a key and which columns you want to compare, and even add filters in the data to give you more control over multiple use cases. @@ -365,11 +392,13 @@ We are introducing the **table difference data quality test** to validate that m {% youtube videoId="oxZVS_UGrE4" start="0:00" end="2:22" width="560px" height="315px" /%} ## Domains RBAC & Subdomains + OpenMetadata introduced Domains & Data Products in 1.3.0. Since then, many large organizations have started using Domains & Data Products to achieve better ownership and collaboration around domains that can span multiple teams. In the 1.5.1 release, we added support for subdomains. This will help teams to organize into multiple subdomains within each domain. ### RBAC for Domains + With the 1.5.1 release, we are adding more stricter controls around Domain. Now, teams, data assets, glossaries, and classification can have domain concepts and can get a policy such that only users within a domain can access the data within a domain. Domain owners can use Data Products to publish data products and showcase publicly available data assets from a specific domain. This will help large companies to use a single OpenMetadata platform to unify all of their data and teams but also provide more stringent controls to segment the data between domains @@ -377,6 +406,7 @@ This will help large companies to use a single OpenMetadata platform to unify al {% youtube videoId="r-_HaewjgTQ" start="0:00" end="0:44" width="560px" height="315px" /%} ## Improved Explore Page & Data Asset Widget + OpenMetadata, with its simple UI/UX and data collaboration features, is becoming more attractive to non-technical users as well. Data Governance teams are using OpenMetadata to add glossary terms and policies around metadata. Teams using Collate SaaS product are taking advantage of our Automations feature to gain productivity in their governance tasks. Our new improved navigation on the Explore page will help users navigate hierarchically and find the data they are looking for. Users will see the data assets now grouped by `service name -> database -> schema -> tables/stored procedures`. @@ -386,11 +416,13 @@ We are also making the discovery of data more accessible for users introducing a {% youtube videoId="45ekUIRO1Ec" start="0:00" end="1:11" width="560px" height="315px" /%} ## Pipeline Status Widget + We are also adding another widget you can use to customize the Landing Page of the User Personas in your organization. With the Pipeline Status widget, Data Engineers can easily track the pipelines that are not behaving as expected. This widget, together with the obervability alerts that are already in place, will help your teams jump even faster to solving any issues in the platform. ## API as Data Asset + The Internet runs using APIs, both producing and consuming data. Organizations today run many microservices and REST APIs to capture data from their users and update a transaction database in the backend. On top of the many supported connectors across Databases, Dashboards, ML Models, etc. We believe that providing support for API Services as data assets will help to get the full picture of how the data is coming through from various services and landing into databases, going to warehouses and BI tools. @@ -400,20 +432,25 @@ In 1.5.1 we are introducing APIs as another first-class entity. Teams can now ca {% youtube videoId="b9wrVnM3u80" start="0:00" end="0:33" width="560px" height="315px" /%} ## Glossary Improvements + OpenMetadata supports multiple glossaries, an import/export and review process, and bulk asset tagging with glossary terms. Many teams are taking advantage of these features, and with an amazing open-source community, we are receiving great feedback on improving glossary functionality. Here are some of the improvements coming in 1.5.1: + 1. Glossary Reviewers can be teams 2. Updating a glossary will enforce a re-review 3. Renaming the Glossary Term while it's under review will keep the task associated with it open ## Data Insights (Collate) + The Data Insights application is meant to give you a quick glance of your data's state and allow you to take action based on the information you receive. To continue pursuing this objective, the application was completely refactored to allow customizability. This is achieved by the possibility of now creating custom dashboards. On this release you can create charts based on your data assets metadata based on your needs. ## Ingestion Connectors + 80+ connectors to help teams to centralize metadata. We continue to push the boundaries of this mission, in + - **Apache Flink** as a Pipeline Connector - **SAP ERP**, after a long and successful collaboration with our community and SAP experts - **Teradata** as a community contribution from [gbpy](https://github.com/gpby) to broaden the integration capabilities for enterprise-scale analytics and data management. @@ -427,7 +464,7 @@ To continue pursuing this objective, the application was completely refactored t **August 6th, 2024** {% /note %} -- Make `Include ddl` disabled by default +- Make `Include ddl` disabled by default - Made DDL configuration consistent with views - Fix user profile task listing. - Fix import/export UI flow ${CollateIconWithLinkMD}. @@ -577,11 +614,13 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta ## Backward Incompatible Changes ### Tooling + - Metadata Backup/Recovery is deprecated. No further support will be provided. - Users are advised to use database native tools to backup and store it in their object store for recovery. - `bootstrap/bootstrap_storage.sh` has been deprecated in favor of bootstrap/openmetadata-ops.sh ### UI + - Activity has been improved. New update specific cards display critical information such as data quality test case updates, description, tag update or removal. - For Lineage, the Expand All button has been removed. A new Layers button is introduced at the bottom left corner. With the Layers button, you can add Column Level Lineage or Data Observability details to your Lineage view. - View Definition is now renamed as Schema Definition. @@ -589,6 +628,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - For Classification, users can set classification to be mutually exclusive only at the time of creation. Once created, you cannot change it back to mutually non-exclusive or vice-versa. This is to prevent conflicts of adding multiple tags that belong to same classification and later turning the mutually exclusive flag back to true. ### API + - Table Schema's `ViewDefinition` is now renamed to `SchemaDefinition` to capture Tables' Create Schema. - Bulk Import API now creates entities if they are not present during the import. - Table's TestSuite is migrated to EntityReference. Previously it used to store entire payload of TestSuite. @@ -612,6 +652,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - APIs are available in OSS. ## Data Quality Improvements + {% youtube videoId="UNOHvBMVcYM" start="0:00" end="1:28" width="560px" height="315px" /%} - The Table schema page now shows the Data Quality tests for each column. @@ -622,17 +663,20 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Support has been added for an empty string as a missing count. ## Data Profiler + - Implemented a global profiler configuration page, allowing admin to exclude certain metric computations for specific data types. - Added profiler support for Redshift complex types and DynamoDB. - Fixed an issue with performing sum operations for large values in profiler ingestion. - Fixed the histogram unit's issues with scientific notation. ## Incident Manager + - We now display a sample of failed rows for the latest failed test cases. Once the issue is resolved, the failed sample will be deleted. (Collate Only) - Fixed the Date time filter for the Incident Manager. - Notifications are sent for the tasks created by the Incident Manager. ## Lineage Improvements + https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Column Lineage Search @@ -657,10 +701,12 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Support has been added for dynamic tables. ## Data Insights + - Previously, the data insights reports displayed only the percentage coverage of ownership and description. Now, users can drill down to view the data assets with no owner or description. - Improved the UX for data insight filters. ## Cost Analysis (Collate Only) + - Lifecycle data for Cost Analysis has been implemented for BigQuery, Snowflake, and Redshift. ## Custom Theme @@ -678,10 +724,12 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Added a Data Quality Widget to list the summary of data quality tests belonging to a user or their team. ## Ingestion Performance Improvements + - Bigquery, Redshift, and Snowflake now support incremental metadata ingestions by scanning DML operations on the query history. - Database Services now support parallelizing the metadata ingestion at each schema. ## Connectors + - Now supports a new connector for QlikCloud. - New Kafka Connect connector - We now parse complex protobuf schemas for Kafka @@ -690,7 +738,7 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Added an option to include or exclude paused pipelines in Airflow. - Revamped SSL support to allow users to upload the required certificates directly in the UI. - The character support has been enhanced for tag ingestion to include /. -- In the Oracle connector, we rolled back to use all_ tables instead of dba_. +- In the Oracle connector, we rolled back to use all* tables instead of dba*. - Added support for Azure auth in Trino. - For QlikSense, we have added an option to disable SSL validation. @@ -701,25 +749,31 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Custom Properties now allow linking other assets in the platform, such as Tables, Dashboards, etc. To enable this, create a Custom Property as an Entity Reference or Entity Reference List. ## Glossary + - The glossary term parent can now be changed from the Details page. - On the data assets page, glossary terms are displayed by hierarchy. ## Alerts & Notification Improvements + - The Activity Feed provides more contextual information, removing the need to move to entity pages. - Alerts give more accurate information about the entity, as well as conversations and tasks. ## Localization + - Fixed localization issues in the confirmation logic for the delete function. - Fixed the search index language configuration. ## Roles + - Now, roles can be inherited from the user configuration in SSO. ## Search + - You can now filter by assets without a description or an owner. - Improved the match results for search results. ## Others + - The description is auto-expanded when the data asset has no data and has the space to accommodate a lengthy description. - User email IDs have been masked and are only visible to Admins. - Users can filter Queries by owner, tag, and creation date in the UI. @@ -730,7 +784,6 @@ https://www.youtube.com/watch?v=KZdVb8DiHJs - Video on Column Lineage Search - Redundant scroll bars have been removed from the UI. - Improved the bot role binding to provide more control over which roles are passed to the system bots. - # 1.3.4 Release 🎉 {% note noteType="Tip" %} @@ -769,21 +822,25 @@ Learn how to upgrade your OpenMetadata instance to 1.3.3! You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.3.2-release). ## MetaPilot (Collate) + - New MetaPilot application shipped in preview mode. Try it out in the [Sandbox](https://sandbox.open-metadata.org/)! - Get automatically generated descriptions with GenAI. Now it’s easier than ever to document your data assets. - Chat with the MetaPilot and get SQL queries to help you extract relevant information from your data assets. - Let the MetaPilot help you understand and improve the queries used on your tables. ## Authentication Flow + - Added generic support for OIDC Authentication. This is SSO provider-agnostic. - You can now integrate Confidential Clients to manage the server authentication. - Now, the session renewal happens automatically in the backend. ## Data Quality + - Pagination support was added for the Data Quality tab for data assets. - Fixed an issue with execution summary timeout issue for the data quality test. ## Connectors + - New Bigtable connector. - Now, users can configure the external sample data storage path. - Added lineage support for Snowflake materialized view and masking policies. @@ -794,6 +851,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Improve PATCH generation for array fields. ## Other Changes + - Avoid creating duplicated queries. - Speed up the server start time by moving the Secrets Manager Migration to the migration container. - Fixed the issue with the date filter for the Incident Manager. @@ -802,6 +860,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fixed an issue with search indexing. - Fixed the missing input field for conversation source for alerts and notifications. - Filter dashboards by a project on the Explore page. + --- **Full Changelog**: [link](https://github.com/open-metadata/OpenMetadata/compare/1.3.1-release...1.3.2-release) @@ -814,30 +873,36 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta You can find the GitHub release [here](https://github.com/open-metadata/OpenMetadata/releases/tag/1.3.1-release). {% /note %} - ## Knowledge Center (Collate) + - Supports drag and drop for the hierarchy of knowledge articles. - Enhanced the layout and loading experience of the knowledge page. ## Lineage + - When adding a new node in Lineage, the Display Name is supported in search. - Fixed the issues with displaying lineage from Metabase. ## Glossary + - Improved the automation of performance tests for Glossary. - Performance improvements to display a large Glossary. ## Data Insights + - Data Insights report has been improved. - The cost Analysis report has been optimized. ## Notifications + - The format for Slack notifications has been improved. ## Custom Properties + - Added enum type support for custom properties. ## Connectors + - Now BigQuery connector supports Primary, Foreign, and Unique Constraints. It fetches the column description for views. - Captures the SQL query that powers a Tableau DataModel. - Azure Key Vault is supported as a Secrets Manager. @@ -847,6 +912,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fixed an issue with the service display name after ingestion. ## Other Changes + - The functionality for mutually exclusive tags has been disabled. - PodGC set up for Argo workflows to delete the pods from the Kubernetes environment on a successful run of the pods. - Fixed the issue with the display of the personal access token. @@ -857,11 +923,9 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fixed the issues with testing the email settings. - Fixed an issue with adding tags. - - # 1.3.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2024, February 5th** [OpenMetadata 1.3 Release - Intuitive Lineage UI, Data Observability Alerts, Data Quality Incident Manager, Custom Metrics for Profiler, Knowledge Center Improvements, and lots more](https://blog.open-metadata.org/openmetadata-release-1-3-ac801834ee80) @@ -927,12 +991,14 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Custom metrics can be created at Table and Column levels. ## Profiler and Data Quality + - The Profiler has been improved to support sample data ingestion without computing other metrics. - Admins can configure the profiler to fetch up to 10,000 rows of sample data. - Sample data can be stored in S3 buckets. - Refined the default time range on the test case results page, adjusting it from the Last 3 days to the Last 30 days for a more encompassing view. ## Connectors + - New Google Cloud Storage for storage services. (Collate) - New Alation connector to migrate metadata into Collate. (Collate) - New Iceberg, SAS Viya, and Doris connectors. @@ -947,6 +1013,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - We now extract table descriptions when ingesting metadata from Salesforce. ## Glossary + - Supports soft delete for the default glossaries in OpenMetadata. - Supports the creation of tasks to request tags or a description. - Only the Owner can edit the Glossary term. @@ -966,10 +1033,12 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - The Settings page UI has been revamped. ## Data Insights + - Cost Analysis expanded to support BigQuery & Redshift. (Collate) - Improved the Data Insights Report sent via email. ## Other Changes + - Announcements can be notified over email, Slack, or Teams. - Alerts are sent to a user when they are mentioned in a task or activity feed. - We have improved the display of search results for column matches. When searching for columns, the matched results will be displayed and highlighted in the Preview pane. @@ -985,7 +1054,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # 1.2.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, October 26th** [OpenMetadata 1.2 Release - Domains, Data Products, Search Index, Stored Procedures, Glossary Approval Workflow, Customizable Landing Page, Applications, Knowledge Center, Cost Analysis, and lots more](https://blog.open-metadata.org/openmetadata-release-1-2-531f0e3c6d9a) @@ -1004,10 +1073,12 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Assets can also be added as Data Products in a Domain. ## Search Index + - Elasticsearch or Opensearch connectors can now bring in the search index metadata into OpenMetadata. - The connector will populate the index’s mapping, settings, and sample data. ## Stored Procedures + - Added support for Stored Procedures. - Snowflake, Redshift, and BigQuery connectors are updated to bring stored procedure metadata into OpenMetadata. - The metadata workflow will bring the Stored Procedures and parse their executions to extract lineage information. @@ -1025,8 +1096,9 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Color coding helps to visually differentiate and identify the data assets, when glossary terms are added to them. ## OpenMetadata Browser Extension + - Updated the Chrome browser extension for OpenMetadata with the new UI. -- Added support for Databases, Database Schemas, Tables, Dashboards, Charts, Pipelines, and Topics. +- Added support for Databases, Database Schemas, Tables, Dashboards, Charts, Pipelines, and Topics. ## Build Automation Applications @@ -1038,11 +1110,13 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - We will continue to add new Applications in upcoming releases. ## Lineage + - Performance improvements made for lineage based on the new release of SQLfluff. - Added support for `UPDATE … FROM Snowflake` queries - Added column-level lineage support for `SELECT *` queries ## Connectors + - Greenplum connector is now supported. - Couchbase connector is now supported. - Azure Data Lake Storage is now supported. (Collate) @@ -1073,21 +1147,24 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # 1.1.2 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, August 24th** {% /note %} ## Data Quality + - Added support for Postgres version 11.19. - Fixed MariaDB time column issues. ## Connectors + - Added JWT authentication support for Trino. - Fixed Snowflake connection test. - Fixed SageMaker ingestion. - Added external table support for BigQuery. ## UI Improvements + - Added Russian language support. - Supports Delete functionality for sample data. - Improved Schema page UX. @@ -1095,9 +1172,11 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Fixed the version history list. ## Ingestion + - Improved performance when ingesting table constraints. ## Backend + - Improved Glossary import validations. - Fixed Test Suite migrations and naming. - Fixed Classification migration. @@ -1106,7 +1185,7 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # 1.1.1 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, August 7th** {% /note %} @@ -1119,15 +1198,18 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Support custom cron for schedule ingestion ## Data Quality + - Fix BigQuery, MSSQL, and Clickhouse profiling errors ## Ingestion + - Fixed Airflow lineage extraction. - Added support for Databricks complex columns comments. - Fixed Athena lineage and usage parameter validation. - Airflow Managed APIs now support Airflow 2.6 ## Connectors + - New [Qliksense](qlik.com) Connector. - Hive supports extracting metadata directly from the metastore to speed up the execution. Users whose metastore is not exposed can still run the extraction pointing to Hive. - Added Usage & Lineage connector for Trino. @@ -1136,14 +1218,14 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Added support for JSON fields in SingleStore. ## Backend + - Bumped table and column names length - Aggregation Improvements for Search - Test Suite Improvements - # 1.1.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, June 30th** [OpenMetadata 1.1.0 Release - UI Overhaul, New Connectors, Improved Lineage Parsing, PII Masking, and lots more](https://blog.open-metadata.org/openmetadata-1-1-0-release-97c1fb603bcf) @@ -1192,84 +1274,97 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Improved monitoring of the Pipeline Service Client health. Any status errors between the OpenMetadata server and the Pipeline Service Client are now surfaced in a Prometheus metric `pipelineServiceClientStatus_counter_total` - Added AWS OpenSearch client-specific support. This allows us to update the Elasticsearch version support up to 7.16. - # 1.0.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, April 25th** [OpenMetadata 1.0 Release - Improved Schemas & APIs, Ingestion Improvements, Storage Services, Dashboard Data Models, Auto PII Classification, Localization, and much more](https://blog.open-metadata.org/openmetadata-1-0-release-beb34762d916) {% /note %} ## APIs & Schema + - **Stabilized** and improved the Schemas and APIs. - The APIs are **backward compatible**. ## Ingestion + - Connecting to your data sources has never been easier. Find all the necessary **permissions** and **connection details** directly in the UI. - When testing the connection, we now have a comprehensive list of **validations** to let you know which pieces of metadata can be extracted with the provided configuration. - **Performance** improvements when extracting metadata from sources such as Snowflake, Redshift, Postgres, and dbt. - New **Apache Impala** connector. ## Storage Services + - Based on your [feedback](https://github.com/open-metadata/OpenMetadata/discussions/8124), we created a new service to extract metadata from your **cloud storage**. - The Data Lake connector ingested one table per file, which covered only some of the use cases in a Data Platform. With **Storage Services**, you can now present accurate metadata from your tables, even when **partitioned**. - The first implementation has been done on **S3**, and we will keep adding support for other sources in the upcoming releases. ## Dashboard Data Models + - Dashboard Services now support the concept of **Data Models**: data that can be directly defined and managed in the Dashboard tooling itself, e.g., LookML models in Looker. - Data Models will help us close the gap between engineering and business by providing all the necessary metadata from sources typically used and managed by analysts or business users. - The first implementation has been done for **Tableau** and **Looker**. ## Queries + - Improved UI for **SQL Queries**, with faster loading times and allowing users to **vote** for popular queries! - Users can now create and share a **Query** directly from the UI, linking it to multiple tables if needed. ## Localization + - In 1.0, we have added **Localization** support for OpenMetadata. - Now you can use OpenMetadata in **English**, **French**, **Chinese**, **Japanese**, **Portuguese**, and **Spanish**. ## Glossary + - New and Improved **Glossary UI** - Easily search for Glossaries and any Glossary Term directly in the **global search**. - Instead of searching and tagging their assets individually, users can add Glossary Terms to multiple **assets** from the Glossary UI. ## Auto PII Classification + - Implemented an automated way to **tag PII data**. - The auto-classification is an optional step of the **Profiler** workflow. We will analyze the column names, and if sample data is being ingested, we will run NLP models on top of it. ## Search + - **Improved Relevancy**, with added support for partial matches. - **Improved Ranking**, with most used or higher Tier assets at the top of the search. - Support for **Classifications** and **Glossaries** in the global search. ## Security + - **SAML** support has been added. - Added option to mask passwords in the API response except for the `ingestion-bot` by setting the environment variable `MASK_PASSWORDS_API=true`. More info [here](/deployment/security/enable-password-masking). - **Deprecation Notice**: **SSO** Service accounts for Bots will be deprecated. **JWT** authentication will be the preferred method for creating Bots. ## Lineage + - Enhanced Lineage UI to display a large number of **nodes (1000+)**. - Improved UI for **better navigation**. - Improved **SQL parser** to extract lineage in the Lineage Workflows. ## Chrome Browser Extension + - All the metadata is at your fingertips while browsing Looker, Superset, etc., with the OpenMetadata Chrome Browser Extension. - **Chrome extension** supports Google SSO, Azure SSO, Okta, and AWS Cognito authentication. - You can Install the Chrome extension from **Chrome Web Store**. ## Other Changes + - The **Explore page** cards will now display a maximum of **ten tags**. - **Entity names** support apostrophes. - The **Summary panel** has been improved to be consistent across the UI. # 0.13.3 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, March 30th** {% /note %} ## Ingestion Framework + - Datalake Avro & Json, JsonZip support - BigQuery Profiler Ingestion for all regions - Support for Snowflake Geometry Type @@ -1288,57 +1383,69 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta - Support for DBT manifest V8 ## Roles & Policies + - A Non-Privileged user can add new 'Roles' to Teams - Fix Permissions API to consider the leaf nodes tags as well, example: table's column tags ## Search + - Improve Search Relevancy, by adding functional scoring and add ngram analyzer; - Enable search for entities using both name and displayName ## Security + - Enable LDAP configuration to be configured via environment variable - LDAP-s support connection without MTLS ## EntityName + - Relax data asset name restrictions to allow the special characters except "::" - Allow unicode character and digits in Entity ## Data Quality + - Fix column values between test # 0.13.2 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2023, January 30th** [OpenMetadata 0.13.2 Release - Improved SQL Lineage, Glossary Bulk Upload, Unified Tag Category API, Mutually Exclusive Tags, Chrome Extension, and lots more](https://blog.open-metadata.org/openmetadata-0-13-2-release-e32c0de93361) {% /note %} ## Improved SQL Lineage + - We have collaborated with the [sqllineage](https://github.com/reata/sqllineage) and [sqlfluff](https://www.sqlfluff.com/) communities - to improve the parsing capabilities of `sqllineage`. We'll continue to collaborate to ship further improvements in new releases. + to improve the parsing capabilities of `sqllineage`. We'll continue to collaborate to ship further improvements in new releases. ## New Glossary UI + - Moved from a tree view in the left panel to an easy to navigate list of the terms sorted alphabetically. - The term list shows the tags and descriptions in the cards. ## Glossary Import & Export + - You can now export your Glossary data as a CSV file. - In the same way, you can now bulk upload terms to a Glossary by adding their details in a CSV file. - The import utility will validate the file and show you a preview of the elements that are going to be imported to OpenMetadata. ## Unified Tag Category API + - Renamed Tag Categories to Classification, a more widely used term. - Updated the API to conform with the rest of the specification. More info [here](https://github.com/open-metadata/OpenMetadata/issues/9259). ## Mutually Exclusive Tags + - When creating a Classification or a Glossary term, you can now make the tags to be mutually exclusive. - If tags are set to be mutually exclusive, you won't be able to set multiple tags from the same category in the same asset. ## EntityName + - Special characters ## Ingestion Framework + - Performance Improvements: We are now getting descriptions in batch, making connectors such as Redshift or Snowflake way faster! - The Oracle connector now ships with the Thick mode enabled. - AWS QuickSight fixes @@ -1348,36 +1455,41 @@ You can find the GitHub release [here](https://github.com/open-metadata/OpenMeta # 0.13.1 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, December 20th** {% /note %} ## Profiler and Data Quality + - Freshness Metric has been introduced. Data freshness shows DML operations performed against a table and the number of rows affected. All this is displayed within the data profiler with filterable graphs. This is currently supported for BigQuery, Snowflake, and Redshift. - Support has been added for data quality tests on Data Lake. - UI has been improved to show table and column profile data on separate page. Legend is now selectable to filter for specific metrics ## Alerts and Notification + The logic for Notification Support has been improved. Users can define Alerts based on a Trigger (all data assets or a specific entity), Filters (events to consider), and Action (Slack, MS Teams, Email, Webhook) on where to send the alert. ## Ingestion -- Now, dbt has its own workflow. Previously, dbt was a part of metadata ingestion workflow. + +- Now, dbt has its own workflow. Previously, dbt was a part of metadata ingestion workflow. - Airflow Lineage Operator and the OpenMetadata Hook are now part of the ingestion package. Send Airflow metadata from your DAGs and safely store the OpenMetadata server connection directly in Airflow. - Multiple Databases (catalog) is now supported for the Databricks connector - Azure blob is now supported to backup your metadata into ## New Connectors + - OpenMetadata now supports Azure Datalake Storage Gen 2 ## General Improvements + - Users can update the description and tags for Topic Schema. Previously, the topic schemas were read-only. We now support Avro/Protobuf parsing and field level details for topic schemas. -- The layout for the Data Insight Report has been improved. We now display a line graph instead of a bar graph. The Most Viewed Data Assets are clickable to view the asset details page. +- The layout for the Data Insight Report has been improved. We now display a line graph instead of a bar graph. The Most Viewed Data Assets are clickable to view the asset details page. - Improvements have been made to Advanced Search. Now, when a filter is applied, the details of the filter selected are displayed for clarity. - On the Explore page UI, the Side Preview is now available for all data assets. Previously it was only displayed for tables. # 0.13.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, December 8th** [OpenMetadata 0.13.0 Release — Data Insights & KPIs, Lineage Traceability, Data Lake Profiler, Search Improvements, and lots more](https://blog.open-metadata.org/openmetadata-0-13-0-release-ac8ac5bd87c1) @@ -1386,58 +1498,70 @@ The logic for Notification Support has been improved. Users can define Alerts ba {% youtube videoId="oNbMnTW5AkE" start="0:00" end="7:51" width="560px" height="315px" /%} ## Data Insights and KPI + Data Insight allows admins to take an active approach in their metadata management. Data Insight provides a single-pane view of all the key metrics to best reflect the state of your data. Admins can define the Key Performance Indicators (KPIs) and set goals within OpenMetadata to work towards better documentation, ownership, and tiering. Alerts can be set against the KPIs to be received on a specified schedule. ## Lineage + The lineage UI has been transformed to enhance user experience. Users can get a holistic view of an entity from the Lineage tab. When an entity is selected, the UI displays end-to-end lineage traceability for the table and column levels. ## Profiler + With the OpenMetadata UI, users can now create and deploy profiling workflows for the Datalake connector, which supports AWS S3 and GCS ## SSO + Support for LDAP SSO has been added in this release ## Advance Search + Syntax Editor has been introduced for advanced search with And/Or conditions that help discover assets quickly ## New Connectors + - AWS SageMaker - AWS QuickSight - AWS Kinesis - Domo ## Messaging Service Schemas Improvements + Major enhancements have been made to how data is extracted from Kafka and Redpanda Messaging services. Previously, OpenMetadata extracted all the Topics in the messaging queue and also connected to the Schema Registry to get the Schemas. These schemas were taken as one payload and published to OpenMetadata. We now parse Avro and Protobuf Schemas to extract the fields. Now, users can document each of these fields within a schema by adding descriptions and tags. Users can search based on the fields in the Schema of a Topic. ## General Improvements + - Soft deleted entities can be restored. Currently, only the ML Models are not supported. - Soft deleted teams can be restored. When restoring a soft deleted parent team, the child teams will not be restored by default. # 0.12.3 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, November 18th** {% /note %} ## Bug Fixes + - User suggestion index mapping - Tag and Glossary terms caching # 0.12.2 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, October 20th** {% /note %} ## Ingestion + - Databricks lineage - Added support for Airflow version 2.2.2 as a workflow scheduler + ## Bug Fixes + - Support same table across different databases for the profiler # 0.12.1 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, October 3rd** {% /note %} @@ -1445,7 +1569,7 @@ Major enhancements have been made to how data is extracted from Kafka and Redpan - User/Password signup and login - Email notifications for forgotten password and new user signed up -- Admin can add new users and send an email +- Admin can add new users and send an email ## ElasticSearch full re-index through UI @@ -1459,17 +1583,17 @@ Major enhancements have been made to how data is extracted from Kafka and Redpan - We support ingesting DBT tags into OpenMetadata -## Bots Integration +## Bots Integration - Admins can create bots and their security mechanism from UI itself ## Bug Fixes -- Around 136 Features/Improvements/Tests made it into 0.12.1 release +- Around 136 Features/Improvements/Tests made it into 0.12.1 release # 0.12.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, September 7th** [OpenMetadata 0.12.0 Release](https://blog.open-metadata.org/openmetadata-0-12-0-release-1ac059700de4) @@ -1478,6 +1602,7 @@ Major enhancements have been made to how data is extracted from Kafka and Redpan {% youtube videoId="tv3pyCLcJfQ" start="0:00" end="17:04" width="560px" height="315px" /%} ## Team Hierarchy + Prior releases supported a flat hierarchy of just Teams and Users. In 0.12, support has been added for the entire organizational hierarchy with Business Unit, Division, Department, and Groups. An organization from small to very large can now be modeled in OpenMetadata with this feature. ## Roles and Policies @@ -1510,18 +1635,18 @@ In 0.12, we’ve also streamlined the Notifications menu with two separate tabs Users can get timely updates about the metadata change events for all entities through APIs using webhooks. The webhook integration with Slack has been further improved in this release. -OpenMetadata also supports webhook integration to Microsoft Teams, just as it supports Slack. Users can choose to receive notifications for only the required entities by using event filters based on when an entity is created, updated, or deleted. +OpenMetadata also supports webhook integration to Microsoft Teams, just as it supports Slack. Users can choose to receive notifications for only the required entities by using event filters based on when an entity is created, updated, or deleted. ## Tasks In the 0.11 release, a request to add or update descriptions for data assets could be converted to a Task. In the 0.12 release, Tasks can be created based on requests to create or update tags. Also, a glossary term approval workflow can be converted to a Task. - ## Secret Management Store Interface In 0.12, we have completely revamped how that secret is stored, accessed, and by whom; by introducing a Secrets Manager Interface to communicate with any Key Management Store. The KMS will mediate between any OpenMetadata internal requirement and sensitive information. That way, users can choose to use the underlying database as KMS, or any external system. The OpenMetadata community has already added support for AWS Key Management Service and AWS SSM. ## Connectors + New connectors are an essential part of every release in OpenMetadata. We are introducing four new connectors in this release: - Redpanda is a Kafka API-compatible streaming data platform for developers that unifies historical and real-time data. OpenMetadata now supports Redpanda as a Messaging service, which allows users to document its topics and schemas. Refer to the Redpanda documentation for more info. @@ -1530,6 +1655,7 @@ New connectors are an essential part of every release in OpenMetadata. We are in - Apache NiFi automates the flow of data between systems. OpenMetadata now supports a NiFi connector as the third new pipeline service on this release. ## Lineage + We’ve enhanced the performance of workflows by having a separate workflow for Lineage and Usage. By using two workflows for computing specific pieces of information, we can effectively filter down the queries to extract lineage. During table usage ingestion, the tables retrieved successfully will be cached, so that there is no need to repeat the same calls multiple times as many queries would be referencing the same tables. @@ -1537,10 +1663,11 @@ Usage queries have been optimized. A result limit has been added to Usage queries. ## Global Settings + The OpenMetadata Settings dropdown menu has been transformed into a single, centralized Settings page for added convenience in viewing all the available options. The Global Settings comprises setting options for Team Members, Access based on Roles and Policies, Services, Data Quality, Collaboration, Custom Attributes, and Integrations for webhooks and bots. Admins can view or update settings for various services like Slack, MS Teams, Webhooks, etc from the Global Settings page. - ## UI/UX Improvements + The major UI UX improvements have been done around Roles and Policies and a Global Settings page. Quite a lot of tweaks have been made to the UI to improve the UX. When creating a new user or when a user is registering for the first time, the dropdown menu for Teams now displays an option to ‘Show All’ teams. Previously, we supported the display of only the first 10 teams. An option has also been provided to search and filter. @@ -1549,13 +1676,14 @@ Manage Tab has been replaced with the manage button on the UI. # 0.11.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, July 1st** [OpenMetadata 0.11.0 Release](https://blog.open-metadata.org/openmetadata-0-11-release-8b82c85636a) {% /note %} ## Data Collaboration - Tasks, Announcements, & Emojis + - Tasks have been introduced as an extension to the ability to create conversations and post replies. - Tasks can be created around descriptions for tables, pipelines, dashboards, and topics. - Users can Request a description, or even Suggest a new description and make edits to an existing description. @@ -1567,14 +1695,17 @@ Manage Tab has been replaced with the manage button on the UI. - Task owners can provide description or accept/reject suggestions and those tasks are automatically closed. ## Column Level Lineage + - Column level lineage API support has been added in the backend. - Supports table level and column level lineage from Snowflake, Redshift, and BigQuery. ## Custom Properties + - Now supports adding new types and extending entities when organizations need to capture custom metadata. - New types and custom fields can be added to entities either using API or in OpenMetadata UI. ## Advanced Search + - Users can search by column, schema, database, owner, tag, and service. - Users can search by multiple parameters to narrow down the search results. - Separate advanced search options are available for Tables, Topics, Dashboards, Pipelines, and ML Models. @@ -1582,11 +1713,13 @@ Manage Tab has been replaced with the manage button on the UI. - Entity specific search options are also available - table specific options include Column, Schema, and Database, pipeline specific options include Task, and dashboards specific option includes Chart. ## Glossary UI Updates + - The Glossary UI has been upgraded. - The arrangement to display the Summary, Related Terms, Synonyms, and References has been changed. - Reviewers are shown on the right panel with an option to add or remove existing reviewers. ## Profiler and Data Quality Improvements + - Seven additional data quality tests have been added as follows. - - tableColumnCountToBeBetween: Ensure the number of columns in your table stays within the expected range - - tableColumnNameToExist: Check that a specific column is in your table @@ -1600,10 +1733,12 @@ Manage Tab has been replaced with the manage button on the UI. - Developed a direct integration between Great Expectations and OpenMetadata. Now, you can add custom actions to your Great Expectations checkpoints file that will automatically ingest your data quality tests results into OpenMetadata at the end of your checkpoint file run. ## ML Models + - ML Model entities have been added to the UI. - Supports ingestion through the UI from MLflow. ## Connectors + - Five new connectors have been added - Airbyte, Mode, AWS Data Lake, Google Cloud Data Lake, and Apache Pinot. - DBT Cloud support was added and we now extract manifest and catalog files from API. - The ingestion scheduler now supports a minute level selection. @@ -1611,6 +1746,7 @@ Manage Tab has been replaced with the manage button on the UI. - The Looker connector now fetches the ‘Usage’ and ‘Access’ metadata for Dashboards and Charts. ## UI Improvements + - The OpenMetadata UI has a new layout. - In the Activity Feeds, the options to reply to a conversation, as well as to delete can now be found on hovering over the conversation. - Users can react with Emojis on the activity feeds, conversations and replies. @@ -1619,6 +1755,7 @@ Manage Tab has been replaced with the manage button on the UI. - A tooltip has been added to display the FQN on hover in the Activity Feed header. ## Other Changes + - Admin users define Roles and associate these roles to Teams. When a user picks a Team, the Role gets automatically assigned. - An option has been added to recreate a fresh index from the data available in Elasticsearch. - A simple webhook server has been added to the metadata command to register and listen to the metadata change events. @@ -1629,7 +1766,7 @@ Manage Tab has been replaced with the manage button on the UI. # 0.10.1 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, May 17th** {% /note %} @@ -1641,7 +1778,7 @@ Manage Tab has been replaced with the manage button on the UI. # 0.10.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, April 27th** [OpenMetadata 0.10.0 Release](https://blog.open-metadata.org/openmetadata-0-10-0-release-82c4f5533c3f) @@ -1649,7 +1786,7 @@ Manage Tab has been replaced with the manage button on the UI. ## Support for Database Schema -OpenMetadata supports databases, service name databases, and tables. We’ve added Database Schema as part of the FQN. +OpenMetadata supports databases, service name databases, and tables. We’ve added Database Schema as part of the FQN. For each external data source, we ingest the database, as well as the tables that are contained underneath the schemas. ## Support for Hard Delete @@ -1659,39 +1796,39 @@ and ingestion. Hard deleting an entity removes the entity and all of its relatio ## Deploy Ingestion from UI -OpenMetadata has refactored the service connections to simplify the ingestion jobs from both the ingestion framework +OpenMetadata has refactored the service connections to simplify the ingestion jobs from both the ingestion framework and the UI. We now use the pydantic models automatically generated from the JSON schemas for the connection definition. The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata. ## Download dbt Manifest Files from Amazon S3 or Google Cloud Storage -Previously, when ingesting the models and lineage from dbt, we passed the path of the dbt manifest and catalog files -directly into the workflow. We’ve worked on improving the quality of life of dbt. Now, we can dynamically download -these files from Amazon S3 or Google Cloud Storage. This way we can have any other process to connect to the dbt, -extract the catalog, and put it into any cloud service. We just need the path name and workflow job details from the +Previously, when ingesting the models and lineage from dbt, we passed the path of the dbt manifest and catalog files +directly into the workflow. We’ve worked on improving the quality of life of dbt. Now, we can dynamically download +these files from Amazon S3 or Google Cloud Storage. This way we can have any other process to connect to the dbt, +extract the catalog, and put it into any cloud service. We just need the path name and workflow job details from the metadata extraction to be able to ingest metadata. ## JSON Schema based Connection Definition -Each service (database, dashboard, messaging, or pipeline service) has its own configuration specifications, with some -unique requirements for some services. Instead of the ad hoc definitions of the source module in Python for each +Each service (database, dashboard, messaging, or pipeline service) has its own configuration specifications, with some +unique requirements for some services. Instead of the ad hoc definitions of the source module in Python for each connector, we’ve worked on the full refactoring of the ingestion framework. We now use the pydantic models automatically generated from the JSON schemas for the connection definition. ## Airflow Rest APIs -The Airflow REST APIs have been refactored. With our API centric model, we are creating a custom airflow rest API +The Airflow REST APIs have been refactored. With our API centric model, we are creating a custom airflow rest API directly on top of Airflow using plugins. This passes the connection information to automatically generate all the dags and prepares handy methods to help us test the connection to the source before creating the service. ## UI Changes - The UI improvements are directed toward providing a consistent user experience. -- Hard Deletion of Entities: With the support for the hard deletion of entities, we can permanently delete tables, - topics, or services. When the entity is hard deleted, the entity and all its relationships are removed. +- Hard Deletion of Entities: With the support for the hard deletion of entities, we can permanently delete tables, + topics, or services. When the entity is hard deleted, the entity and all its relationships are removed. This generates an ‘EntityDeleted’ change event. -- Dynamic “Add Service” Forms: The ‘Add Service’ form is automatically generated in the UI based on the JSON +- Dynamic “Add Service” Forms: The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata. - UI Support for Database Schema as part of FQN: The database schema has been introduced in the 0.10 release. All the entity pages now support Database Schema in the UI. @@ -1702,17 +1839,18 @@ and prepares handy methods to help us test the connection to the source before c - Add User: A user can be added from the Users page. ## Security Changes -- **Support Refresh Tokens for Auth0 and Okta SSO**: The JWT tokens generated by the SSO providers expire by default - in about an hour, making the user re-login often. In this release, we’ve added support for refresh tokens for Auth0 + +- **Support Refresh Tokens for Auth0 and Okta SSO**: The JWT tokens generated by the SSO providers expire by default + in about an hour, making the user re-login often. In this release, we’ve added support for refresh tokens for Auth0 and Okta SSO. The tokens are refreshed silently behind the scenes to provide an uninterrupted user experience. In future releases, we’ll continue to stabilize authentication and add refresh tokens for the other SSO providers. -- **Custom OIDC SSO**: OpenMetadata now supports integration with your custom-built OIDC SSO for authentication. +- **Custom OIDC SSO**: OpenMetadata now supports integration with your custom-built OIDC SSO for authentication. This is supported both on the front end for user authentication and on the ingestion side. - **Azure SSO**: Support has been added for Azure SSO on Airflow. # 0.9.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, March 10th** [OpenMetadata 0.9.0 Release](https://blog.open-metadata.org/openmetadata-0-9-0-release-8e7b93ab1882) @@ -1727,6 +1865,7 @@ and prepares handy methods to help us test the connection to the source before c - Table details - Click through on usage to see who or what services are using it, what queries are pulling from it. ## Data Quality + - Ability to create and monitor the test cases. - Data Quality Tests support with Json Schemas and APIs. - UI Integration to enable user to write tests and run them on Airflow. @@ -1736,10 +1875,11 @@ and prepares handy methods to help us test the connection to the source before c - Glossaries are a Controlled Vocabulary in an organization used to define the concepts and terminologies specific to a particular domain. - API & Schemas to support Glossary. -- UI support to add Glossary and Glossary Terms. +- UI support to add Glossary and Glossary Terms. - Support for using Glossary terms to annotate Entities and Search using Glossary Terms. ## Connectors + - Apache Iceberg - Azure SQL - Clickhouse @@ -1756,10 +1896,12 @@ and prepares handy methods to help us test the connection to the source before c - Amundsen, Import Metadata from Amundsen into OpenMetadata ## Lineage + - DataSource SQL Parsing support to extract Lineage - View Lineage support ## Pipeline + - Capture pipeline status as it happens ## Security @@ -1770,23 +1912,25 @@ and prepares handy methods to help us test the connection to the source before c # 0.8.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2022, January 22nd** [OpenMetadata 0.8.0 Release](https://blog.open-metadata.org/openmetadata-0-8-0-release-ca09bd2fbf54) {% /note %} ## Access Control Policies + - Design of Access Control Policies. - Provide Role based access control with community feedback. ## Eventing Webhook - Register webhooks to get metadata event notifications. -- Metadata Change Event integration into Slack and framework for integration into other services such as +- Metadata Change Event integration into Slack and framework for integration into other services such as Kafka or other Notification frameworks ## Connectors + - Delta Lake - Iceberg - PowerBI @@ -1794,25 +1938,29 @@ and prepares handy methods to help us test the connection to the source before c # 0.7.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2021, November 17th** [OpenMetadata 0.7.0 Release](https://blog.open-metadata.org/openmetadata-0-7-0-release-9f741b8d5089) {% /note %} ## UI - Activity Feed, Improved UX for Search + - Users will have access to Activity Feed of all the changes to the Metadata. - New and Improved UX for Search and Landing page. ## Support for Table Location + - Extract Location information from Glue, Redshift. - Show Location details on the Table Page. ## ElasticSearch Improvements + - Support SSL (including self-signed certs) enabled ElasticSearch. - New entities will be indexed into ElasticSearch directly ## Connectors + - Metabase - Apache Druid - Glue Improvements @@ -1821,88 +1969,104 @@ and prepares handy methods to help us test the connection to the source before c - Amundsen Import connector ## Other features + - Metadata Change Event integration into Slack and framework for integration into other services such as Kafka or other Notification frameworks - Delta Lake support, Databricks, Iceberg # 0.6.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2021, November 17th** [OpenMetadata 0.6.0 Release — Metadata Versioning, Events API, One-Click Ingestion, and more](https://blog.open-metadata.org/openmetadata-0-6-0-release-metadata-versioning-events-api-one-click-ingestion-and-more-4394c4f08e0b) {% /note %} ## Metadata Versioning and Eventing Framework + - Capture changes to Entity Metadata from source and user interactions as versions. - Versioned changes will be published as events for clients to consume to take actions on. ## Data Reliability + - Improvements to Data Reliability library. - Capture custom measurements through user provided SQL. ## Airflow APIs + - Airflow APIs to deploy DAGS and manage them. - UI integration to deploy ingestion workflows. ## Connectors + - AWS Glue - dbt - MariaDB # 0.5.0 Release -{% note noteType="Tip" %} +{% note noteType="Tip" %} **2021, October 19th** [OpenMetadata 0.5.0 Release is here — Lineage, Pipelines, Complex Types, Data Profiler and so much more](https://blog.open-metadata.org/openmetadata-0-5-0-1144a4000644) {% /note %} ## Support for Lineage + - Lineage related schemas and APIs. - Lineage metadata integration from AirFlow for tables. - UI changes to show lineage information to the users. ## Data Reliability + - Improvements to Data Profiler. - UI integration with Data Profiler to show how the table profile looks over the period of time. ## Complex Types + - Support complex types such as Struct, Array with nested fields. - UI support to add expand complex types and tag, add description for nested fields. ## Connectors + - Trino - Redash ## Other features + - Pipeline Entities are supported. - Integration with Airflow to extract Pipeline details. # 0.4.0 Release -{% note noteType="Tip" %} + +{% note noteType="Tip" %} **2021, September 20th** [OpenMetadata 0.4.0 Release — Dashboards, Topics, Data Reliability](https://blog.open-metadata.org/openmetadata-0-4-0-release-dashboards-topics-data-reliability-14e8672ae0f5) {% /note %} ## Support for Kafka (and Pulsar WIP) + - Support for Message Service and Topic entities in schemas, APIs, and UI. - Kafka connector and ingestion support for Confluent Schema Registry. ## Support for Dashboards + - Support for Dashboard services, Dashboards, and Charts entities in schemas, APIs, and UI. - Looker, Superset, Tableau connector, and ingestion support. ## User Interface + - Sort search results based on Usage, Relevance, and Last updated time. - Search string highlighted in search results. - Support for Kafka and Dashboards from Looker, Superset, and Tableau. ## Other features + - Pluggable SSO integration - Auth0 support. - Support for Presto. ## Work in progress + - Salesforce CRM connector. - Data profiler to profile tables in ingestion framework and show it table details page.