| 
									
										
										
										
											2022-06-27 15:14:04 +02:00
										 |  |  | --- | 
					
						
							|  |  |  | title: Metadata Ingestion | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  | slug: /connectors/ingestion | 
					
						
							| 
									
										
										
										
											2022-06-27 15:14:04 +02:00
										 |  |  | --- | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | # Metadata Ingestion
 | 
					
						
							| 
									
										
										
										
											2022-07-11 15:19:11 +02:00
										 |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  | The goal of OpenMetadata is to serve as a centralised platform where users can gather and collaborate | 
					
						
							|  |  |  | around data. This is possible thanks for different workflows that users can deploy and schedule, which will | 
					
						
							|  |  |  | connect to the data sources to extract metadata. | 
					
						
							| 
									
										
										
										
											2022-07-11 15:19:11 +02:00
										 |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  | Different metadata being ingested to OpenMetadata can be: | 
					
						
							|  |  |  | - Entities metadata, such as Tables, Dashboards, Topics... | 
					
						
							|  |  |  | - Query usage to rank the most used tables, | 
					
						
							|  |  |  | - Lineage between Entities, | 
					
						
							|  |  |  | - Data Profiles and Quality Tests. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | In this section we will explore the different workflows, how they work and how to use them. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <InlineCalloutContainer> | 
					
						
							|  |  |  |   <InlineCallout | 
					
						
							|  |  |  |     color="violet-70" | 
					
						
							|  |  |  |     bold="Metadata Ingestion" | 
					
						
							|  |  |  |     icon="cable" | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  |     href="/connectors/ingestion/workflows/metadata" | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  |   > | 
					
						
							|  |  |  |     Learn more about how to ingest metadata from dozens of connectors. | 
					
						
							|  |  |  |   </InlineCallout> | 
					
						
							|  |  |  |   <InlineCallout | 
					
						
							|  |  |  |     color="violet-70" | 
					
						
							| 
									
										
										
										
											2022-12-21 19:11:37 +05:30
										 |  |  |     bold="Metadata Profiler" | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  |     icon="cable" | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  |     href="/connectors/ingestion/workflows/profiler" | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  |   > | 
					
						
							| 
									
										
										
										
											2022-12-21 19:11:37 +05:30
										 |  |  |     To get metrics from your Tables | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  |   </InlineCallout> | 
					
						
							| 
									
										
										
										
											2022-12-21 19:11:37 +05:30
										 |  |  |   <InlineCallout | 
					
						
							|  |  |  |     color="violet-70" | 
					
						
							|  |  |  |     bold="Metadata Data Quality Tests" | 
					
						
							|  |  |  |     icon="cable" | 
					
						
							|  |  |  |     href="/connectors/ingestion/workflows/data-quality" | 
					
						
							|  |  |  |   > | 
					
						
							|  |  |  |     To run automated Quality Tests on your Tables. | 
					
						
							|  |  |  |   </InlineCallout> | 
					
						
							|  |  |  |     | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  |   <InlineCallout | 
					
						
							|  |  |  |     color="violet-70" | 
					
						
							|  |  |  |     bold="Metadata Usage" | 
					
						
							|  |  |  |     icon="cable" | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  |     href="/connectors/ingestion/workflows/usage" | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  |   > | 
					
						
							|  |  |  |     To analyze popular entities. | 
					
						
							|  |  |  |   </InlineCallout> | 
					
						
							|  |  |  |   <InlineCallout | 
					
						
							|  |  |  |     color="violet-70" | 
					
						
							|  |  |  |     bold="Metadata Lineage" | 
					
						
							|  |  |  |     icon="cable" | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  |     href="/connectors/ingestion/workflows/lineage" | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  |   > | 
					
						
							|  |  |  |     To analyze relationships in your data platform. | 
					
						
							|  |  |  |   </InlineCallout> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | </InlineCalloutContainer> | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | ## Metadata Versioning
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | One fundamental aspect of Metadata Ingestion is being able to analyze the evolution of your metadata. OpenMetadata | 
					
						
							|  |  |  | support Metadata Versioning, maintaining the history of changes of all your assets. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <InlineCalloutContainer> | 
					
						
							|  |  |  |   <InlineCallout | 
					
						
							|  |  |  |     color="violet-70" | 
					
						
							|  |  |  |     bold="Metadata Versioning" | 
					
						
							|  |  |  |     icon="360" | 
					
						
							| 
									
										
										
										
											2022-10-05 21:54:02 -07:00
										 |  |  |     href="/connectors/ingestion/versioning" | 
					
						
							| 
									
										
										
										
											2022-08-23 11:53:40 +02:00
										 |  |  |   > | 
					
						
							|  |  |  |     Learn how OpenMetadata keeps track of your metadata evolution. | 
					
						
							|  |  |  |   </InlineCallout> | 
					
						
							|  |  |  | </InlineCalloutContainer> | 
					
						
							| 
									
										
										
										
											2022-10-01 20:01:39 +02:00
										 |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-11-07 09:27:53 +01:00
										 |  |  | ## Best Practices
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | You want to know some of the best practices around metadata ingestion? This is the right place! | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <InlineCalloutContainer> | 
					
						
							|  |  |  |   <InlineCallout | 
					
						
							|  |  |  |     color="violet-70" | 
					
						
							|  |  |  |     bold="Best Practices" | 
					
						
							|  |  |  |     icon="360" | 
					
						
							|  |  |  |     href="/connectors/ingestion/best-practices" | 
					
						
							|  |  |  |   > | 
					
						
							|  |  |  |     Learn the best practices to ingest metadata, both from the UI and using any custom orchestrator. | 
					
						
							|  |  |  |   </InlineCallout> | 
					
						
							|  |  |  | </InlineCalloutContainer> | 
					
						
							|  |  |  | 
 | 
					
						
							| 
									
										
										
										
											2022-10-01 20:01:39 +02:00
										 |  |  | ## Deep Dive
 | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | Understand how OpenMetadata deploys the workflows that are created from the UI. | 
					
						
							|  |  |  | 
 | 
					
						
							|  |  |  | <InlineCalloutContainer> | 
					
						
							|  |  |  |   <InlineCallout | 
					
						
							|  |  |  |     color="violet-70" | 
					
						
							|  |  |  |     bold="Ingestion Pipeline UI Deployment" | 
					
						
							|  |  |  |     icon="360" | 
					
						
							| 
									
										
										
										
											2022-10-07 08:49:46 -07:00
										 |  |  |     href="/connectors/ingestion/deployment" | 
					
						
							| 
									
										
										
										
											2022-10-01 20:01:39 +02:00
										 |  |  |   > | 
					
						
							|  |  |  |     Learn about the Pipeline Service interface and how OpenMetadata handles workflow deployments. | 
					
						
							|  |  |  |   </InlineCallout> | 
					
						
							|  |  |  | </InlineCalloutContainer> |