2023-12-13 14:03:08 +01:00
										 
									 
								 
							 | 
							
								
							 | 
							
								
							 | 
							
							
								---
							 | 
						
					
						
							
								
									
										
										
										
											2025-07-18 11:02:07 +05:30
										 
									 
								 
							 | 
							
								
									
										
									
								
							 | 
							
								
							 | 
							
							
								title: Ingestion Framework Deployment | Official Documentation
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								description: Configure ingestion from external, cloud-native, or hybrid environments using deployment guides.
							 | 
						
					
						
							
								
									
										
										
										
											2023-12-13 14:03:08 +01:00
										 
									 
								 
							 | 
							
								
							 | 
							
								
							 | 
							
							
								slug: /deployment/ingestion
							 | 
						
					
						
							
								
									
										
										
										
											2024-09-05 10:30:31 +02:00
										 
									 
								 
							 | 
							
								
									
										
									
								
							 | 
							
								
							 | 
							
							
								collate: false
							 | 
						
					
						
							
								
									
										
										
										
											2023-12-13 14:03:08 +01:00
										 
									 
								 
							 | 
							
								
							 | 
							
								
							 | 
							
							
								---
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								# Ingestion Framework Deployment
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								The Ingestion Framework is the module that takes care of bringing metadata in to OpenMetadata. It is used
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								for any type of workflow that is supported in the platform: Metadata, Lineage, Usage, Profiler, Data Quality,...
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								## Manage & Schedule the Ingestion Framework
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								In this guide, we will present the different alternatives to run and manage your ingestion workflows. There are mainly
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								2 ways of running the ingestion:
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								1. Internally, by managing the workflows from OpenMetadata.
							 | 
						
					
						
							
								
									
										
										
										
											2024-01-18 11:07:34 +01:00
										 
									 
								 
							 | 
							
								
									
										
									
								
							 | 
							
								
							 | 
							
							
								2. Externally, by using any other tool capable of running Python code.
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								Note that the end result is going to be the same. The only difference is that running the workflows internally,
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								OpenMetadata will dynamically generate the processes that will perform the metadata extraction. If configuring
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								the ingestion externally, you will be managing this processes directly on your platform of choice.
							 | 
						
					
						
							
								
									
										
										
										
											2023-12-13 14:03:08 +01:00
										 
									 
								 
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							
								
									
										
										
										
											2024-02-23 08:53:10 +01:00
										 
									 
								 
							 | 
							
								
									
										
									
								
							 | 
							
								
							 | 
							
							
								## Option 1 - From OpenMetadata
							 | 
						
					
						
							
								
									
										
										
										
											2023-12-13 14:03:08 +01:00
										 
									 
								 
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								If you want to learn how to configure your setup to run them from OpenMetadata, follow this guide:
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								{% inlineCalloutContainer %}
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								  {% inlineCallout
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    color="violet-70"
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    icon="10k"
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    bold="OpenMetadata UI"
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    href="/deployment/ingestion/openmetadata" %}
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    Deploy, configure and manage the ingestion workflows directly from the OpenMetadata UI.
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								  {% /inlineCallout %}
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								{% /inlineCalloutContainer %}
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							
								
									
										
										
										
											2024-02-23 08:53:10 +01:00
										 
									 
								 
							 | 
							
								
									
										
									
								
							 | 
							
								
							 | 
							
							
								## Option 2 - Externally
							 | 
						
					
						
							
								
									
										
										
										
											2023-12-13 14:03:08 +01:00
										 
									 
								 
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							
								
									
										
										
										
											2024-01-18 11:07:34 +01:00
										 
									 
								 
							 | 
							
								
									
										
									
								
							 | 
							
								
							 | 
							
							
								Any tool capable of running Python code can be used to configure the metadata extraction from your sources.
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								In this section, we are going to give you some background on how the Ingestion Framework works, how to configure
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								the metadata extraction, and some examples on how to host the ingestion in different platforms.
							 | 
						
					
						
							
								
									
										
										
										
											2023-12-13 14:03:08 +01:00
										 
									 
								 
							 | 
							
								
							 | 
							
								
							 | 
							
							
								
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								{% inlineCalloutContainer %}
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								  {% inlineCallout
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    color="violet-70"
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    icon="10k"
							 | 
						
					
						
							
								
									
										
										
										
											2024-02-23 08:53:10 +01:00
										 
									 
								 
							 | 
							
								
									
										
									
								
							 | 
							
								
							 | 
							
							
								    bold="External Ingestion"
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    href="/deployment/ingestion/external" %}
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								    Manage the Ingestion Framework from anywhere!
							 | 
						
					
						
							
								
									
										
										
										
											2023-12-13 14:03:08 +01:00
										 
									 
								 
							 | 
							
								
							 | 
							
								
							 | 
							
							
								  {% /inlineCallout %}
							 | 
						
					
						
							| 
								
							 | 
							
								
							 | 
							
								
							 | 
							
							
								{% /inlineCalloutContainer %}
							 |