2022-06-27 15:14:04 +02:00
|
|
|
---
|
|
|
|
title: Metadata Ingestion
|
2022-10-05 21:54:02 -07:00
|
|
|
slug: /connectors/ingestion
|
2022-06-27 15:14:04 +02:00
|
|
|
---
|
|
|
|
|
|
|
|
# Metadata Ingestion
|
2022-07-11 15:19:11 +02:00
|
|
|
|
2022-08-23 11:53:40 +02:00
|
|
|
The goal of OpenMetadata is to serve as a centralised platform where users can gather and collaborate
|
|
|
|
around data. This is possible thanks for different workflows that users can deploy and schedule, which will
|
|
|
|
connect to the data sources to extract metadata.
|
2022-07-11 15:19:11 +02:00
|
|
|
|
2022-08-23 11:53:40 +02:00
|
|
|
Different metadata being ingested to OpenMetadata can be:
|
|
|
|
- Entities metadata, such as Tables, Dashboards, Topics...
|
|
|
|
- Query usage to rank the most used tables,
|
|
|
|
- Lineage between Entities,
|
|
|
|
- Data Profiles and Quality Tests.
|
|
|
|
|
|
|
|
In this section we will explore the different workflows, how they work and how to use them.
|
|
|
|
|
|
|
|
<InlineCalloutContainer>
|
|
|
|
<InlineCallout
|
|
|
|
color="violet-70"
|
|
|
|
bold="Metadata Ingestion"
|
|
|
|
icon="cable"
|
2022-10-05 21:54:02 -07:00
|
|
|
href="/connectors/ingestion/workflows/metadata"
|
2022-08-23 11:53:40 +02:00
|
|
|
>
|
|
|
|
Learn more about how to ingest metadata from dozens of connectors.
|
|
|
|
</InlineCallout>
|
|
|
|
<InlineCallout
|
|
|
|
color="violet-70"
|
|
|
|
bold="Metadata Profiler & Quality Tests"
|
|
|
|
icon="cable"
|
2022-10-05 21:54:02 -07:00
|
|
|
href="/connectors/ingestion/workflows/profiler"
|
2022-08-23 11:53:40 +02:00
|
|
|
>
|
|
|
|
Get metrics from your Tables and run automated Quality Tests!
|
|
|
|
</InlineCallout>
|
|
|
|
<InlineCallout
|
|
|
|
color="violet-70"
|
|
|
|
bold="Metadata Usage"
|
|
|
|
icon="cable"
|
2022-10-05 21:54:02 -07:00
|
|
|
href="/connectors/ingestion/workflows/usage"
|
2022-08-23 11:53:40 +02:00
|
|
|
>
|
|
|
|
To analyze popular entities.
|
|
|
|
</InlineCallout>
|
|
|
|
<InlineCallout
|
|
|
|
color="violet-70"
|
|
|
|
bold="Metadata Lineage"
|
|
|
|
icon="cable"
|
2022-10-05 21:54:02 -07:00
|
|
|
href="/connectors/ingestion/workflows/lineage"
|
2022-08-23 11:53:40 +02:00
|
|
|
>
|
|
|
|
To analyze relationships in your data platform.
|
|
|
|
</InlineCallout>
|
|
|
|
|
|
|
|
</InlineCalloutContainer>
|
|
|
|
|
|
|
|
## Metadata Versioning
|
|
|
|
|
|
|
|
One fundamental aspect of Metadata Ingestion is being able to analyze the evolution of your metadata. OpenMetadata
|
|
|
|
support Metadata Versioning, maintaining the history of changes of all your assets.
|
|
|
|
|
|
|
|
<InlineCalloutContainer>
|
|
|
|
<InlineCallout
|
|
|
|
color="violet-70"
|
|
|
|
bold="Metadata Versioning"
|
|
|
|
icon="360"
|
2022-10-05 21:54:02 -07:00
|
|
|
href="/connectors/ingestion/versioning"
|
2022-08-23 11:53:40 +02:00
|
|
|
>
|
|
|
|
Learn how OpenMetadata keeps track of your metadata evolution.
|
|
|
|
</InlineCallout>
|
|
|
|
</InlineCalloutContainer>
|
2022-10-01 20:01:39 +02:00
|
|
|
|
2022-11-07 09:27:53 +01:00
|
|
|
## Best Practices
|
|
|
|
|
|
|
|
You want to know some of the best practices around metadata ingestion? This is the right place!
|
|
|
|
|
|
|
|
<InlineCalloutContainer>
|
|
|
|
<InlineCallout
|
|
|
|
color="violet-70"
|
|
|
|
bold="Best Practices"
|
|
|
|
icon="360"
|
|
|
|
href="/connectors/ingestion/best-practices"
|
|
|
|
>
|
|
|
|
Learn the best practices to ingest metadata, both from the UI and using any custom orchestrator.
|
|
|
|
</InlineCallout>
|
|
|
|
</InlineCalloutContainer>
|
|
|
|
|
2022-10-01 20:01:39 +02:00
|
|
|
## Deep Dive
|
|
|
|
|
|
|
|
Understand how OpenMetadata deploys the workflows that are created from the UI.
|
|
|
|
|
|
|
|
<InlineCalloutContainer>
|
|
|
|
<InlineCallout
|
|
|
|
color="violet-70"
|
|
|
|
bold="Ingestion Pipeline UI Deployment"
|
|
|
|
icon="360"
|
2022-10-07 08:49:46 -07:00
|
|
|
href="/connectors/ingestion/deployment"
|
2022-10-01 20:01:39 +02:00
|
|
|
>
|
|
|
|
Learn about the Pipeline Service interface and how OpenMetadata handles workflow deployments.
|
|
|
|
</InlineCallout>
|
|
|
|
</InlineCalloutContainer>
|