mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-10-30 18:17:53 +00:00
4adffef439
* Update the partial references for 1.7.x-SNAPSHOT * Delete image folder for 1.4 and add images for 1.7 * Update the new connector stages
243 lines
9.3 KiB
Markdown
243 lines
9.3 KiB
Markdown
---
|
||
title: OpenMetadata
|
||
slug: /features
|
||
---
|
||
|
||
# Features
|
||
|
||
OpenMetadata includes a rapidly growing set of features to address common needs in data discovery, quality,
|
||
observability, and collaboration.
|
||
|
||
## All Your Data in One Place
|
||
OpenMetadata provides all the data context you need for different use cases in a single place.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/all-your-data-in-one-place.gif"
|
||
/%}
|
||
|
||
|
||
## Data Discovery
|
||
OpenMetadata enables you to discover your data using a variety of strategies, including: keyword search, data associations
|
||
(e.g., frequently joined tables, lineage), and complex queries. Using OpenMetadata you can search across tables,
|
||
topics, dashboards, pipelines, and services. OpenMetadata supports detailed metadata for assets and their components
|
||
(e.g., columns, charts), including support for complex data types such as arrays and structs.
|
||
|
||
### Data Discovery
|
||
Find assets based on name, description, component metadata (e.g., for columns, charts), and the containing service.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/data-discovery.gif"
|
||
/%}
|
||
|
||
|
||
|
||
### Discover Data through Association
|
||
Discover assets through frequently joined tables and columns as measured by the data profiler. You can also discover
|
||
assets through relationships based on data lineage.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/data-discovery-association.gif"
|
||
/%}
|
||
|
||
|
||
### Advanced Search
|
||
Find assets matching strict criteria on metadata properties and Boolean operators.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/data-discovery-advanced-search.gif"
|
||
/%}
|
||
|
||
|
||
## Activity Feeds
|
||
The OpenMetadata home screen features a change activity feed that enables you view a summary of data change events.
|
||
This feed shows all changes to data sorted with the most recent changes at the top. The entities in the activity feed
|
||
are clickable including tables, dashboards, team names, etc. There are activity feeds for:
|
||
- All data
|
||
- Data for which you are an owner
|
||
- Data you are following
|
||
|
||
{% image
|
||
src="/images/v1.7/features/activity-feeds.gif"
|
||
/%}
|
||
|
||
|
||
## Event Notification via Webhooks and Slack Integration
|
||
|
||
The webhook interface allows you to build applications that receive all the data changes happening in your organization
|
||
through APIs. Register URLs to receive metadata event notifications. Slack integration through incoming webhooks is one
|
||
of many applications of this feature.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/event-notification-slack.gif"
|
||
/%}
|
||
|
||
## Add Descriptive Metadata
|
||
Add descriptions and tags to tables, columns, and other assets. OpenMetadata indexes assets based on descriptions, tags,
|
||
names, and other metadata to enable keyword, advanced search, and filtering to enable you and others in your
|
||
organization to discover your data.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/add-descriptive-metadata.gif"
|
||
/%}
|
||
|
||
|
||
## Complex Data Types
|
||
Add descriptions and tags to nested fields in complex data types like arrays and structs. Locate these assets using
|
||
keyword search or advanced search.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/complex-data-types.gif"
|
||
/%}
|
||
|
||
## Deleted Entity Metadata
|
||
Entities have a lot of user-generated metadata, such as descriptions, tags, ownership, tiering. There’s also rich
|
||
metadata generated by OpenMetadata through the data profiler, usage data, lineage, test results, and other graph
|
||
relationships with other entities. When an entity is deleted, all of this rich information is lost, and it’s not
|
||
easy to recreate it. OpenMetadata supports soft deletion in the UI and soft and permanent deletion in the API,
|
||
enabling you to choose whether to maintain metadata for deleted entities.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/deleted-entity.gif"
|
||
/%}
|
||
|
||
|
||
## Importance & Owners
|
||
Tier tags enable you to annotate assets with their importance relative to other assets. The Explore UI enables you to
|
||
filter assets based on importance.
|
||
|
||
Use ownership metadata to determine the primary points of contact for any assets of interest in order to get help with
|
||
any questions you might have.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/importance-owners.gif"
|
||
/%}
|
||
|
||
|
||
### Filter Assets by Importance
|
||
User Tier tags and usage data to identify the relative importance of data assets.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/filter-asset-by-importance.gif"
|
||
/%}
|
||
|
||
|
||
### Identify Asset Owners
|
||
Identify owners who can help with questions about an asset.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/identify-asset-owner.gif"
|
||
/%}
|
||
|
||
|
||
## Role Based Access Control
|
||
OpenMetadata supports Role Based Access Control (RBAC) policies for metadata operations. Each user may be assigned one
|
||
or more roles. Each role has a defined policy. Policies are composed of a set of rules. Rules allow/deny access to
|
||
metadata operations such as updating descriptions, tags, owners, and lineage.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/rbac.gif"
|
||
/%}
|
||
|
||
|
||
## Data Lineage
|
||
Trace the path of data across tables, pipelines, and dashboards.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/data-lineage.gif"
|
||
/%}
|
||
|
||
|
||
## Edit Data Lineage Manually
|
||
Edit lineage to provide a richer understanding of the provenance of data. The OpenMetadata no-code editor provides a
|
||
drag and drop interface. Drop tables, pipelines, and dashboards onto the lineage graph. You may add new edges or
|
||
delete existing edges to better represent data lineage.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/manual-lineage.gif"
|
||
/%}
|
||
|
||
|
||
## dbt Integration
|
||
A dbt model provides transformation logic that creates a table from raw data. While lineage tells us broadly what data
|
||
a table was generated from. A dbt model provides specifics. OpenMetadata includes an integration for dbt that enables
|
||
you to see what models are being used to generate tables.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/dbt-integration.gif"
|
||
/%}
|
||
|
||
|
||
## Data Reliability
|
||
Build trust in your data by creating tests to monitor that the data is complete, fresh, and accurate.
|
||
|
||
### Data Profiler
|
||
Enable the data profiler to capture table usage statistics over a period of time. This happens as part of metadata
|
||
ingestion. Data profiles enable you to check for null values in non-null columns, for duplicates in a unique column,
|
||
etc. You can gain a better understanding of column data distributions through descriptive statistics provided.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/data-profiler.gif"
|
||
/%}
|
||
|
||
|
||
## Service Connectors & One-Click Ingestion Pipelines
|
||
Integrate your database, dashboard, messaging, and pipeline services with OpenMetadata. OpenMetadata provides a UI
|
||
integration with Apache Airflow as a workflow engine to run ingestion, data profiling, data quality and other automation
|
||
jobs. Admins can configure a service to run the OpenMetadata pipelines and add an ingestion schedule to automatically
|
||
kick off the ingestion jobs directly from the OpenMetadata UI.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/ingestion-pipelines.gif"
|
||
/%}
|
||
|
||
|
||
See the [Connectors](/connectors) documentation for information on available connectors and how to
|
||
integrate your services with OpenMetadata.
|
||
|
||
## Metadata Versioning & Events API
|
||
Starting in 0.6, OpenMetadata captures changes in both technical metadata (e.g., table schemas) and business metadata
|
||
(e.g., tags, ownership, descriptions) as new versions of an entity. Metadata changes generate events that indicate
|
||
which entity changed, who changed it, and how it changed. You may use these events to integrate metadata into other
|
||
tools or trigger actions.
|
||
|
||
### Metadata Versioning
|
||
OpenMetadata maintains the version history for all entities using a number with the format `major.minor`,
|
||
starting with 0.1 as the initial version of an entity.
|
||
|
||
Changes in metadata result in version changes as follows:
|
||
- **Backward compatible** changes result in a Minor version change. A change in the description, tags, or ownership
|
||
will increase the version of the entity metadata by 0.1 (e.g., from 0.1 to 0.2).
|
||
- **Backward incompatible** changes result in a Major version change. For example, when a column in a table is deleted,
|
||
the version increases by 1.0 (e.g., from 0.2 to 1.2).
|
||
|
||
Metadata versioning helps **simplify debugging processes**. View the version history to see if a recent change led to
|
||
a data issue. Data owners and admins can review changes and revert if necessary.
|
||
|
||
|
||
{% image
|
||
src="/images/v1.7/features/data-versioning.gif"
|
||
/%}
|
||
|
||
|
||
Versioning also helps in broader collaboration among consumers and producers of data. Admins can provide access to more
|
||
users in the organization to change certain fields. Crow sourcing makes metadata the collective responsibility of the
|
||
entire organization.
|
||
|
||
### Events API
|
||
Starting in the 0.6 release, OpenMetadata captures changes in both technical metadata (e.g., table schemas) and
|
||
business metadata (e.g., tags, ownership, descriptions) as change events. This lays the groundwork for notifications.
|
||
In an upcoming release we will add support for push based events, enabling applications to register webhooks to be
|
||
notified of change events. We will also enable users to subscribe to notifications and alerts for assets of interest.
|
||
|
||
## Elasticsearch Integration
|
||
OpenMetadata supports SSL-enabled Elasticsearch (including self-signed certs). In prior versions of OpenMetadata it was
|
||
necessary to run an indexing workflow following any ingestion workflow to make ingested entities available in the
|
||
OpenMetadata UI. As of the 0.7 release, OpenMetadata automatically runs an indexing workflow as new entities are added
|
||
or updated through ingestion workflows.
|
||
|
||
{% image
|
||
src="/images/v1.7/features/elasticsearch.gif"
|
||
/%}
|
||
|
||
|