mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-09-03 14:13:06 +00:00
[Docs] - Airflow Git sync (#11299)
* [Docs] - Airflow Git sync * Update deployment docs * upgrade and backup details * upgrade and backup details
This commit is contained in:
parent
7d2ab3a13f
commit
2860ebfb3e
@ -194,6 +194,23 @@ curl -XPOST <AIRFLOW_HOST>/api/v1/openmetadata/enable --data-raw '{"dag_id": "<D
|
||||
|
||||
Please update it accordingly.
|
||||
|
||||
## Git Sync?
|
||||
|
||||
One recurrent question when setting up Airflow is the possibility of using [git-sync](https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-a-private-github-repo-using-git-sync-sidecar)
|
||||
to manage the ingestion DAGs.
|
||||
|
||||
Let's remark the differences between `git-sync` and what we want to achieve by installing our custom API plugins:
|
||||
1. `git-sync` will use Git as the source of truth for your DAGs. Meaning, any DAG you have on Git will eventually be used and scheduled in Airflow.
|
||||
2. With the `openmetadata-managed-apis` we are using the OpenMetadata server as the source of truth. We are enabling dynamic DAG
|
||||
creation from the OpenMetadata into your Airflow instance every time that you create a new Ingestion Workflow.
|
||||
|
||||
Then, should you use `git-sync`?
|
||||
|
||||
- If you have an existing Airflow instance, and you want to build and maintain your own ingestion DAGs ([example](https://docs.open-metadata.org/v1.0.0/connectors/database/snowflake/airflow#2.-prepare-the-ingestion-dag)),
|
||||
then you can go for it.
|
||||
- If instead, you want to use the full deployment process from OpenMetadata, `git-sync` would not be the right tool, since the DAGs won't be backed up by Git, but rather created from OpenMetadata. Note that if anything
|
||||
would to happen where you might lose the Airflow volumes, etc. You can just redeploy the DAGs from OpenMetadata.
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
## Ingestion Pipeline deployment issues
|
||||
|
@ -3,7 +3,29 @@ title: Deployment
|
||||
slug: /deployment
|
||||
---
|
||||
|
||||
# Deployment
|
||||
# From 0 to OpenMetadata in 5 minutes
|
||||
|
||||
Are you exploring or doing a PoC? It won't get easier than following our Quickstart guide!
|
||||
|
||||
{% inlineCalloutContainer %}
|
||||
|
||||
{% inlineCallout
|
||||
icon="celebration"
|
||||
bold="Quickstart OpenMetadata"
|
||||
href="/quick-start/local-deployment" %}
|
||||
Get OpenMetadata up and running in under 5 minutes!
|
||||
{% /inlineCallout %}
|
||||
|
||||
{% inlineCallout
|
||||
icon="open_in_new"
|
||||
bold="Try the OpenMetadata Sandbox"
|
||||
href="/quick-start/sandbox" %}
|
||||
Interact with a sample installation with 0 setup to explore our Discovery, Governance and Collaboration features.
|
||||
{% /inlineCallout %}
|
||||
|
||||
{% /inlineCalloutContainer %}
|
||||
|
||||
# Architecture and Deployment
|
||||
|
||||
We support different kinds of deployment:
|
||||
|
||||
|
@ -0,0 +1,69 @@
|
||||
---
|
||||
title: How do the Upgrade & Backup work?
|
||||
slug: /deployment/upgrade/how-does-it-work
|
||||
---
|
||||
|
||||
# How do the Upgrade & Backup work?
|
||||
|
||||
If this is the first time you are trying to upgrade OpenMetadata, or you just have some doubts on how the
|
||||
backup and upgrade process work, this is the place to be. We will cover:
|
||||
|
||||
- What is being backed up?
|
||||
- When should we restore?
|
||||
- What happens during the migration?
|
||||
|
||||
## Architecture Review
|
||||
|
||||
Let's start with a simplified design review of OpenMetadata. You can find further details [here](/main-concepts/high-level-design),
|
||||
but we'll now focus on the Server & the Database:
|
||||
|
||||
{% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-simple.drawio.png" alt="simple architecture" /%}
|
||||
|
||||
All the metadata is stored in a MySQL or Postgres instance. The shape of the data is managed using [Flyway](https://flywaydb.org/), and the migration
|
||||
scripts are handled [here](https://github.com/open-metadata/OpenMetadata/tree/main/bootstrap/sql).
|
||||
|
||||
In a nutshell, we have a data model in which we store all of this information. The definition of this model (table names,
|
||||
schemas,...) is managed using Flyway migrations. **In every release, the structure of this data model can change**. This means that
|
||||
the shape of the database is tightly coupled to your OpenMetadata Server version.
|
||||
|
||||
## 1. What is being backed up?
|
||||
|
||||
{% note %}
|
||||
|
||||
You can find all the necessary information on how to run the backups [here](https://docs.open-metadata.org/v1.0.0/deployment/backup-restore-metadata).
|
||||
|
||||
{% /note %}
|
||||
|
||||
When we backup the data, we are creating an SQL file that follows the shape of the database of a specific version. Thus, if we have
|
||||
some issues on our instance, and we ever need to restore that data, it will only fit to a database with that same version shape.
|
||||
|
||||
{% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-backup.drawio.png" alt="backup" /%}
|
||||
|
||||
## 2. When should we restore?
|
||||
|
||||
Now that we understand what is being backed up and how it looks like, when (and where) should we restore?
|
||||
|
||||
- **When**: We restore the data if we need to get back in time. Restoring is never needed during the upgrade process.
|
||||
- **Where**: We will restore the data to a clean database with the Flyway migrations at the same version as the backed up data.
|
||||
The usual process will be:
|
||||
- We start with a clean database (no tables in it).
|
||||
- We run the migrations ([docs](https://docs.open-metadata.org/v1.0.0/deployment/bare-metal#4.-prepare-the-openmetadata-database-and-indexes))
|
||||
or we start the OpenMetadata server, which will automatically run the migrations.
|
||||
- With the server stopped, we [restore](https://docs.open-metadata.org/v1.0.0/deployment/backup-restore-metadata#restore-metadata) the data.
|
||||
|
||||
{% note %}
|
||||
|
||||
Note that the restore process will not work if we try to restore some data taken from version X to a database shaped with version Y.
|
||||
|
||||
{% /note %}
|
||||
|
||||
## 3. What happens during the migration?
|
||||
|
||||
We have been explaining how each OpenMetadata Server relies on a specific data model to store the metadata. What happens
|
||||
when we upgrade from version X to Y?
|
||||
|
||||
{% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-migrate.drawio.png" alt="backup" /%}
|
||||
|
||||
The migration process will take care of getting the data shaped as X and transform it to the Y shape. After the migration is done,
|
||||
the server in version Y will be able to run properly.
|
||||
|
@ -168,6 +168,8 @@ site_menu:
|
||||
|
||||
- category: Deployment / Upgrade OpenMetadata
|
||||
url: /deployment/upgrade
|
||||
- category: Deployment / Upgrade OpenMetadata / How do the Upgrade & Backup work?
|
||||
url: /deployment/upgrade/how-does-it-work
|
||||
- category: Deployment / Upgrade OpenMetadata / Upgrade on Bare Metal
|
||||
url: /deployment/upgrade/bare-metal
|
||||
- category: Deployment / Upgrade OpenMetadata / Upgrade on Docker
|
||||
|
@ -18,14 +18,14 @@ We have two approaches to start exploring:
|
||||
icon="celebration"
|
||||
bold="Try OpenMetadata in Docker"
|
||||
href="/quick-start/local-deployment" %}
|
||||
Get OpenMetadata up and running locally in under 5 minutes!
|
||||
Get OpenMetadata up and running in under 5 minutes!
|
||||
{% /inlineCallout %}
|
||||
|
||||
{% inlineCallout
|
||||
icon="open_in_new"
|
||||
bold="Try the OpenMetadata Sandbox"
|
||||
href="/quick-start/sandbox" %}
|
||||
Interact with a sample installation with 0 setup.
|
||||
Interact with a sample installation with 0 setup to explore our Discovery, Governance and Collaboration features.
|
||||
{% /inlineCallout %}
|
||||
|
||||
{% /inlineCalloutContainer %}
|
||||
|
Binary file not shown.
After Width: | Height: | Size: 60 KiB |
Binary file not shown.
After Width: | Height: | Size: 43 KiB |
Binary file not shown.
After Width: | Height: | Size: 47 KiB |
Loading…
x
Reference in New Issue
Block a user