mirror of
				https://github.com/open-metadata/OpenMetadata.git
				synced 2025-10-31 10:39:30 +00:00 
			
		
		
		
	[Docs] - Airflow Git sync (#11299)
* [Docs] - Airflow Git sync * Update deployment docs * upgrade and backup details * upgrade and backup details
This commit is contained in:
		
							parent
							
								
									7d2ab3a13f
								
							
						
					
					
						commit
						2860ebfb3e
					
				| @ -194,6 +194,23 @@ curl -XPOST <AIRFLOW_HOST>/api/v1/openmetadata/enable --data-raw '{"dag_id": "<D | ||||
| 
 | ||||
| Please update it accordingly. | ||||
| 
 | ||||
| ## Git Sync? | ||||
| 
 | ||||
| One recurrent question when setting up Airflow is the possibility of using [git-sync](https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-a-private-github-repo-using-git-sync-sidecar)  | ||||
| to manage the ingestion DAGs. | ||||
| 
 | ||||
| Let's remark the differences between `git-sync` and what we want to achieve by installing our custom API plugins: | ||||
| 1. `git-sync` will use Git as the source of truth for your DAGs. Meaning, any DAG you have on Git will eventually be used and scheduled in Airflow. | ||||
| 2. With the `openmetadata-managed-apis` we are using the OpenMetadata server as the source of truth. We are enabling dynamic DAG | ||||
|   creation from the OpenMetadata into your Airflow instance every time that you create a new Ingestion Workflow. | ||||
| 
 | ||||
| Then, should you use `git-sync`? | ||||
| 
 | ||||
| - If you have an existing Airflow instance, and you want to build and maintain your own ingestion DAGs ([example](https://docs.open-metadata.org/v1.0.0/connectors/database/snowflake/airflow#2.-prepare-the-ingestion-dag)), | ||||
| then you can go for it. | ||||
| - If instead, you want to use the full deployment process from OpenMetadata, `git-sync` would not be the right tool, since the DAGs won't be backed up by Git, but rather created from OpenMetadata. Note that if anything | ||||
|   would to happen where you might lose the Airflow volumes, etc. You can just redeploy the DAGs from OpenMetadata. | ||||
| 
 | ||||
| # Troubleshooting | ||||
| 
 | ||||
| ## Ingestion Pipeline deployment issues | ||||
|  | ||||
| @ -3,7 +3,29 @@ title: Deployment | ||||
| slug: /deployment | ||||
| --- | ||||
| 
 | ||||
| # Deployment | ||||
| # From 0 to OpenMetadata in 5 minutes | ||||
| 
 | ||||
| Are you exploring or doing a PoC? It won't get easier than following our Quickstart guide! | ||||
| 
 | ||||
| {% inlineCalloutContainer %} | ||||
| 
 | ||||
| {% inlineCallout | ||||
| icon="celebration" | ||||
| bold="Quickstart OpenMetadata" | ||||
| href="/quick-start/local-deployment" %} | ||||
| Get OpenMetadata up and running in under 5 minutes! | ||||
| {% /inlineCallout %} | ||||
| 
 | ||||
| {% inlineCallout | ||||
| icon="open_in_new" | ||||
| bold="Try the OpenMetadata Sandbox" | ||||
| href="/quick-start/sandbox" %} | ||||
| Interact with a sample installation with 0 setup to explore our Discovery, Governance and Collaboration features. | ||||
| {% /inlineCallout %} | ||||
| 
 | ||||
| {% /inlineCalloutContainer %} | ||||
| 
 | ||||
| # Architecture and Deployment | ||||
| 
 | ||||
| We support different kinds of deployment: | ||||
| 
 | ||||
|  | ||||
| @ -0,0 +1,69 @@ | ||||
| --- | ||||
| title: How do the Upgrade & Backup work? | ||||
| slug: /deployment/upgrade/how-does-it-work | ||||
| --- | ||||
| 
 | ||||
| # How do the Upgrade & Backup work? | ||||
| 
 | ||||
| If this is the first time you are trying to upgrade OpenMetadata, or you just have some doubts on how the | ||||
| backup and upgrade process work, this is the place to be. We will cover: | ||||
| 
 | ||||
| - What is being backed up? | ||||
| - When should we restore? | ||||
| - What happens during the migration? | ||||
| 
 | ||||
| ## Architecture Review | ||||
| 
 | ||||
| Let's start with a simplified design review of OpenMetadata. You can find further details [here](/main-concepts/high-level-design), | ||||
| but we'll now focus on the Server & the Database: | ||||
| 
 | ||||
| {% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-simple.drawio.png" alt="simple architecture" /%} | ||||
| 
 | ||||
| All the metadata is stored in a MySQL or Postgres instance. The shape of the data is managed using [Flyway](https://flywaydb.org/), and the migration | ||||
| scripts are handled [here](https://github.com/open-metadata/OpenMetadata/tree/main/bootstrap/sql). | ||||
| 
 | ||||
| In a nutshell, we have a data model in which we store all of this information. The definition of this model (table names, | ||||
| schemas,...) is managed using Flyway migrations. **In every release, the structure of this data model can change**. This means that | ||||
| the shape of the database is tightly coupled to your OpenMetadata Server version. | ||||
| 
 | ||||
| ## 1. What is being backed up? | ||||
| 
 | ||||
| {% note %} | ||||
| 
 | ||||
| You can find all the necessary information on how to run the backups [here](https://docs.open-metadata.org/v1.0.0/deployment/backup-restore-metadata). | ||||
| 
 | ||||
| {% /note %} | ||||
| 
 | ||||
| When we backup the data, we are creating an SQL file that follows the shape of the database of a specific version. Thus, if we have | ||||
| some issues on our instance, and we ever need to restore that data, it will only fit to a database with that same version shape. | ||||
| 
 | ||||
| {% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-backup.drawio.png" alt="backup" /%} | ||||
| 
 | ||||
| ## 2. When should we restore? | ||||
| 
 | ||||
| Now that we understand what is being backed up and how it looks like, when (and where) should we restore? | ||||
| 
 | ||||
| - **When**: We restore the data if we need to get back in time. Restoring is never needed during the upgrade process. | ||||
| - **Where**: We will restore the data to a clean database with the Flyway migrations at the same version as the backed up data. | ||||
|   The usual process will be: | ||||
|   - We start with a clean database (no tables in it). | ||||
|   - We run the migrations ([docs](https://docs.open-metadata.org/v1.0.0/deployment/bare-metal#4.-prepare-the-openmetadata-database-and-indexes)) | ||||
|      or we start the OpenMetadata server, which will automatically run the migrations. | ||||
|   - With the server stopped, we [restore](https://docs.open-metadata.org/v1.0.0/deployment/backup-restore-metadata#restore-metadata) the data. | ||||
| 
 | ||||
| {% note %} | ||||
| 
 | ||||
| Note that the restore process will not work if we try to restore some data taken from version X to a database shaped with version Y. | ||||
| 
 | ||||
| {% /note %} | ||||
| 
 | ||||
| ## 3. What happens during the migration? | ||||
| 
 | ||||
| We have been explaining how each OpenMetadata Server relies on a specific data model to store the metadata. What happens | ||||
| when we upgrade from version X to Y? | ||||
| 
 | ||||
| {% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-migrate.drawio.png" alt="backup" /%} | ||||
| 
 | ||||
| The migration process will take care of getting the data shaped as X and transform it to the Y shape. After the migration is done, | ||||
| the server in version Y will be able to run properly. | ||||
| 
 | ||||
| @ -168,6 +168,8 @@ site_menu: | ||||
| 
 | ||||
|   - category: Deployment / Upgrade OpenMetadata | ||||
|     url: /deployment/upgrade | ||||
|   - category: Deployment / Upgrade OpenMetadata / How do the Upgrade & Backup work? | ||||
|     url: /deployment/upgrade/how-does-it-work | ||||
|   - category: Deployment / Upgrade OpenMetadata / Upgrade on Bare Metal | ||||
|     url: /deployment/upgrade/bare-metal | ||||
|   - category: Deployment / Upgrade OpenMetadata / Upgrade on Docker | ||||
|  | ||||
| @ -18,14 +18,14 @@ We have two approaches to start exploring: | ||||
|     icon="celebration" | ||||
|     bold="Try OpenMetadata in Docker" | ||||
|     href="/quick-start/local-deployment" %} | ||||
| Get OpenMetadata up and running locally in under 5 minutes! | ||||
| Get OpenMetadata up and running in under 5 minutes! | ||||
| {% /inlineCallout %} | ||||
| 
 | ||||
| {% inlineCallout | ||||
|     icon="open_in_new" | ||||
|     bold="Try the OpenMetadata Sandbox" | ||||
|     href="/quick-start/sandbox" %} | ||||
| Interact with a sample installation with 0 setup. | ||||
| Interact with a sample installation with 0 setup to explore our Discovery, Governance and Collaboration features. | ||||
| {% /inlineCallout %} | ||||
| 
 | ||||
| {% /inlineCalloutContainer %} | ||||
|  | ||||
										
											Binary file not shown.
										
									
								
							| After Width: | Height: | Size: 60 KiB | 
										
											Binary file not shown.
										
									
								
							| After Width: | Height: | Size: 43 KiB | 
										
											Binary file not shown.
										
									
								
							| After Width: | Height: | Size: 47 KiB | 
		Loading…
	
	
			
			x
			
			
		
	
		Reference in New Issue
	
	Block a user
	 Pere Miquel Brull
						Pere Miquel Brull