mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-10-29 17:49:14 +00:00
243 lines
9.3 KiB
Markdown
243 lines
9.3 KiB
Markdown
|
---
|
|||
|
|
title: OpenMetadata
|
|||
|
|
slug: /features
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Features
|
|||
|
|
|
|||
|
|
OpenMetadata includes a rapidly growing set of features to address common needs in data discovery, quality,
|
|||
|
|
observability, and collaboration.
|
|||
|
|
|
|||
|
|
## All Your Data in One Place
|
|||
|
|
OpenMetadata provides all the data context you need for different use cases in a single place.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/all-your-data-in-one-place.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Data Discovery
|
|||
|
|
OpenMetadata enables you to discover your data using a variety of strategies, including: keyword search, data associations
|
|||
|
|
(e.g., frequently joined tables, lineage), and complex queries. Using OpenMetadata you can search across tables,
|
|||
|
|
topics, dashboards, pipelines, and services. OpenMetadata supports detailed metadata for assets and their components
|
|||
|
|
(e.g., columns, charts), including support for complex data types such as arrays and structs.
|
|||
|
|
|
|||
|
|
### Data Discovery
|
|||
|
|
Find assets based on name, description, component metadata (e.g., for columns, charts), and the containing service.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/data-discovery.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
### Discover Data through Association
|
|||
|
|
Discover assets through frequently joined tables and columns as measured by the data profiler. You can also discover
|
|||
|
|
assets through relationships based on data lineage.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/data-discovery-association.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
### Advanced Search
|
|||
|
|
Find assets matching strict criteria on metadata properties and Boolean operators.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/data-discovery-advanced-search.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Activity Feeds
|
|||
|
|
The OpenMetadata home screen features a change activity feed that enables you view a summary of data change events.
|
|||
|
|
This feed shows all changes to data sorted with the most recent changes at the top. The entities in the activity feed
|
|||
|
|
are clickable including tables, dashboards, team names, etc. There are activity feeds for:
|
|||
|
|
- All data
|
|||
|
|
- Data for which you are an owner
|
|||
|
|
- Data you are following
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/activity-feeds.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Event Notification via Webhooks and Slack Integration
|
|||
|
|
|
|||
|
|
The webhook interface allows you to build applications that receive all the data changes happening in your organization
|
|||
|
|
through APIs. Register URLs to receive metadata event notifications. Slack integration through incoming webhooks is one
|
|||
|
|
of many applications of this feature.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/event-notification-slack.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
## Add Descriptive Metadata
|
|||
|
|
Add descriptions and tags to tables, columns, and other assets. OpenMetadata indexes assets based on descriptions, tags,
|
|||
|
|
names, and other metadata to enable keyword, advanced search, and filtering to enable you and others in your
|
|||
|
|
organization to discover your data.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/add-descriptive-metadata.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Complex Data Types
|
|||
|
|
Add descriptions and tags to nested fields in complex data types like arrays and structs. Locate these assets using
|
|||
|
|
keyword search or advanced search.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/complex-data-types.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
## Deleted Entity Metadata
|
|||
|
|
Entities have a lot of user-generated metadata, such as descriptions, tags, ownership, tiering. There’s also rich
|
|||
|
|
metadata generated by OpenMetadata through the data profiler, usage data, lineage, test results, and other graph
|
|||
|
|
relationships with other entities. When an entity is deleted, all of this rich information is lost, and it’s not
|
|||
|
|
easy to recreate it. OpenMetadata supports soft deletion in the UI and soft and permanent deletion in the API,
|
|||
|
|
enabling you to choose whether to maintain metadata for deleted entities.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/deleted-entity.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Importance & Owners
|
|||
|
|
Tier tags enable you to annotate assets with their importance relative to other assets. The Explore UI enables you to
|
|||
|
|
filter assets based on importance.
|
|||
|
|
|
|||
|
|
Use ownership metadata to determine the primary points of contact for any assets of interest in order to get help with
|
|||
|
|
any questions you might have.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/importance-owners.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
### Filter Assets by Importance
|
|||
|
|
User Tier tags and usage data to identify the relative importance of data assets.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/filter-asset-by-importance.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
### Identify Asset Owners
|
|||
|
|
Identify owners who can help with questions about an asset.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/identify-asset-owner.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Role Based Access Control
|
|||
|
|
OpenMetadata supports Role Based Access Control (RBAC) policies for metadata operations. Each user may be assigned one
|
|||
|
|
or more roles. Each role has a defined policy. Policies are composed of a set of rules. Rules allow/deny access to
|
|||
|
|
metadata operations such as updating descriptions, tags, owners, and lineage.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/rbac.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Data Lineage
|
|||
|
|
Trace the path of data across tables, pipelines, and dashboards.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/data-lineage.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Edit Data Lineage Manually
|
|||
|
|
Edit lineage to provide a richer understanding of the provenance of data. The OpenMetadata no-code editor provides a
|
|||
|
|
drag and drop interface. Drop tables, pipelines, and dashboards onto the lineage graph. You may add new edges or
|
|||
|
|
delete existing edges to better represent data lineage.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/manual-lineage.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## dbt Integration
|
|||
|
|
A dbt model provides transformation logic that creates a table from raw data. While lineage tells us broadly what data
|
|||
|
|
a table was generated from. A dbt model provides specifics. OpenMetadata includes an integration for dbt that enables
|
|||
|
|
you to see what models are being used to generate tables.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/dbt-integration.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Data Reliability
|
|||
|
|
Build trust in your data by creating tests to monitor that the data is complete, fresh, and accurate.
|
|||
|
|
|
|||
|
|
### Data Profiler
|
|||
|
|
Enable the data profiler to capture table usage statistics over a period of time. This happens as part of metadata
|
|||
|
|
ingestion. Data profiles enable you to check for null values in non-null columns, for duplicates in a unique column,
|
|||
|
|
etc. You can gain a better understanding of column data distributions through descriptive statistics provided.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/data-profiler.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Service Connectors & One-Click Ingestion Pipelines
|
|||
|
|
Integrate your database, dashboard, messaging, and pipeline services with OpenMetadata. OpenMetadata provides a UI
|
|||
|
|
integration with Apache Airflow as a workflow engine to run ingestion, data profiling, data quality and other automation
|
|||
|
|
jobs. Admins can configure a service to run the OpenMetadata pipelines and add an ingestion schedule to automatically
|
|||
|
|
kick off the ingestion jobs directly from the OpenMetadata UI.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/ingestion-pipelines.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
See the [Connectors](/connectors) documentation for information on available connectors and how to
|
|||
|
|
integrate your services with OpenMetadata.
|
|||
|
|
|
|||
|
|
## Metadata Versioning & Events API
|
|||
|
|
Starting in 0.6, OpenMetadata captures changes in both technical metadata (e.g., table schemas) and business metadata
|
|||
|
|
(e.g., tags, ownership, descriptions) as new versions of an entity. Metadata changes generate events that indicate
|
|||
|
|
which entity changed, who changed it, and how it changed. You may use these events to integrate metadata into other
|
|||
|
|
tools or trigger actions.
|
|||
|
|
|
|||
|
|
### Metadata Versioning
|
|||
|
|
OpenMetadata maintains the version history for all entities using a number with the format `major.minor`,
|
|||
|
|
starting with 0.1 as the initial version of an entity.
|
|||
|
|
|
|||
|
|
Changes in metadata result in version changes as follows:
|
|||
|
|
- **Backward compatible** changes result in a Minor version change. A change in the description, tags, or ownership
|
|||
|
|
will increase the version of the entity metadata by 0.1 (e.g., from 0.1 to 0.2).
|
|||
|
|
- **Backward incompatible** changes result in a Major version change. For example, when a column in a table is deleted,
|
|||
|
|
the version increases by 1.0 (e.g., from 0.2 to 1.2).
|
|||
|
|
|
|||
|
|
Metadata versioning helps **simplify debugging processes**. View the version history to see if a recent change led to
|
|||
|
|
a data issue. Data owners and admins can review changes and revert if necessary.
|
|||
|
|
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/data-versioning.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|
|||
|
|
Versioning also helps in broader collaboration among consumers and producers of data. Admins can provide access to more
|
|||
|
|
users in the organization to change certain fields. Crow sourcing makes metadata the collective responsibility of the
|
|||
|
|
entire organization.
|
|||
|
|
|
|||
|
|
### Events API
|
|||
|
|
Starting in the 0.6 release, OpenMetadata captures changes in both technical metadata (e.g., table schemas) and
|
|||
|
|
business metadata (e.g., tags, ownership, descriptions) as change events. This lays the groundwork for notifications.
|
|||
|
|
In an upcoming release we will add support for push based events, enabling applications to register webhooks to be
|
|||
|
|
notified of change events. We will also enable users to subscribe to notifications and alerts for assets of interest.
|
|||
|
|
|
|||
|
|
## Elasticsearch Integration
|
|||
|
|
OpenMetadata supports SSL-enabled Elasticsearch (including self-signed certs). In prior versions of OpenMetadata it was
|
|||
|
|
necessary to run an indexing workflow following any ingestion workflow to make ingested entities available in the
|
|||
|
|
OpenMetadata UI. As of the 0.7 release, OpenMetadata automatically runs an indexing workflow as new entities are added
|
|||
|
|
or updated through ingestion workflows.
|
|||
|
|
|
|||
|
|
{% image
|
|||
|
|
src="/images/v1.5/features/elasticsearch.gif"
|
|||
|
|
/%}
|
|||
|
|
|
|||
|
|
|