[Docs] - Airflow Git sync (#11299)

* [Docs] - Airflow Git sync

* Update deployment docs

* upgrade and backup details

* upgrade and backup details
This commit is contained in:
Pere Miquel Brull 2023-04-26 21:41:24 +02:00 committed by GitHub
parent 7d2ab3a13f
commit 2860ebfb3e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 113 additions and 3 deletions

View File

@ -194,6 +194,23 @@ curl -XPOST <AIRFLOW_HOST>/api/v1/openmetadata/enable --data-raw '{"dag_id": "<D
Please update it accordingly.
## Git Sync?
One recurrent question when setting up Airflow is the possibility of using [git-sync](https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-a-private-github-repo-using-git-sync-sidecar)
to manage the ingestion DAGs.
Let's remark the differences between `git-sync` and what we want to achieve by installing our custom API plugins:
1. `git-sync` will use Git as the source of truth for your DAGs. Meaning, any DAG you have on Git will eventually be used and scheduled in Airflow.
2. With the `openmetadata-managed-apis` we are using the OpenMetadata server as the source of truth. We are enabling dynamic DAG
creation from the OpenMetadata into your Airflow instance every time that you create a new Ingestion Workflow.
Then, should you use `git-sync`?
- If you have an existing Airflow instance, and you want to build and maintain your own ingestion DAGs ([example](https://docs.open-metadata.org/v1.0.0/connectors/database/snowflake/airflow#2.-prepare-the-ingestion-dag)),
then you can go for it.
- If instead, you want to use the full deployment process from OpenMetadata, `git-sync` would not be the right tool, since the DAGs won't be backed up by Git, but rather created from OpenMetadata. Note that if anything
would to happen where you might lose the Airflow volumes, etc. You can just redeploy the DAGs from OpenMetadata.
# Troubleshooting
## Ingestion Pipeline deployment issues

View File

@ -3,7 +3,29 @@ title: Deployment
slug: /deployment
---
# Deployment
# From 0 to OpenMetadata in 5 minutes
Are you exploring or doing a PoC? It won't get easier than following our Quickstart guide!
{% inlineCalloutContainer %}
{% inlineCallout
icon="celebration"
bold="Quickstart OpenMetadata"
href="/quick-start/local-deployment" %}
Get OpenMetadata up and running in under 5 minutes!
{% /inlineCallout %}
{% inlineCallout
icon="open_in_new"
bold="Try the OpenMetadata Sandbox"
href="/quick-start/sandbox" %}
Interact with a sample installation with 0 setup to explore our Discovery, Governance and Collaboration features.
{% /inlineCallout %}
{% /inlineCalloutContainer %}
# Architecture and Deployment
We support different kinds of deployment:

View File

@ -0,0 +1,69 @@
---
title: How do the Upgrade & Backup work?
slug: /deployment/upgrade/how-does-it-work
---
# How do the Upgrade & Backup work?
If this is the first time you are trying to upgrade OpenMetadata, or you just have some doubts on how the
backup and upgrade process work, this is the place to be. We will cover:
- What is being backed up?
- When should we restore?
- What happens during the migration?
## Architecture Review
Let's start with a simplified design review of OpenMetadata. You can find further details [here](/main-concepts/high-level-design),
but we'll now focus on the Server & the Database:
{% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-simple.drawio.png" alt="simple architecture" /%}
All the metadata is stored in a MySQL or Postgres instance. The shape of the data is managed using [Flyway](https://flywaydb.org/), and the migration
scripts are handled [here](https://github.com/open-metadata/OpenMetadata/tree/main/bootstrap/sql).
In a nutshell, we have a data model in which we store all of this information. The definition of this model (table names,
schemas,...) is managed using Flyway migrations. **In every release, the structure of this data model can change**. This means that
the shape of the database is tightly coupled to your OpenMetadata Server version.
## 1. What is being backed up?
{% note %}
You can find all the necessary information on how to run the backups [here](https://docs.open-metadata.org/v1.0.0/deployment/backup-restore-metadata).
{% /note %}
When we backup the data, we are creating an SQL file that follows the shape of the database of a specific version. Thus, if we have
some issues on our instance, and we ever need to restore that data, it will only fit to a database with that same version shape.
{% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-backup.drawio.png" alt="backup" /%}
## 2. When should we restore?
Now that we understand what is being backed up and how it looks like, when (and where) should we restore?
- **When**: We restore the data if we need to get back in time. Restoring is never needed during the upgrade process.
- **Where**: We will restore the data to a clean database with the Flyway migrations at the same version as the backed up data.
The usual process will be:
- We start with a clean database (no tables in it).
- We run the migrations ([docs](https://docs.open-metadata.org/v1.0.0/deployment/bare-metal#4.-prepare-the-openmetadata-database-and-indexes))
or we start the OpenMetadata server, which will automatically run the migrations.
- With the server stopped, we [restore](https://docs.open-metadata.org/v1.0.0/deployment/backup-restore-metadata#restore-metadata) the data.
{% note %}
Note that the restore process will not work if we try to restore some data taken from version X to a database shaped with version Y.
{% /note %}
## 3. What happens during the migration?
We have been explaining how each OpenMetadata Server relies on a specific data model to store the metadata. What happens
when we upgrade from version X to Y?
{% image src="/images/v1.0.0/deployment/upgrade/how-does-it-work/upgrade-migrate.drawio.png" alt="backup" /%}
The migration process will take care of getting the data shaped as X and transform it to the Y shape. After the migration is done,
the server in version Y will be able to run properly.

View File

@ -168,6 +168,8 @@ site_menu:
- category: Deployment / Upgrade OpenMetadata
url: /deployment/upgrade
- category: Deployment / Upgrade OpenMetadata / How do the Upgrade & Backup work?
url: /deployment/upgrade/how-does-it-work
- category: Deployment / Upgrade OpenMetadata / Upgrade on Bare Metal
url: /deployment/upgrade/bare-metal
- category: Deployment / Upgrade OpenMetadata / Upgrade on Docker

View File

@ -18,14 +18,14 @@ We have two approaches to start exploring:
icon="celebration"
bold="Try OpenMetadata in Docker"
href="/quick-start/local-deployment" %}
Get OpenMetadata up and running locally in under 5 minutes!
Get OpenMetadata up and running in under 5 minutes!
{% /inlineCallout %}
{% inlineCallout
icon="open_in_new"
bold="Try the OpenMetadata Sandbox"
href="/quick-start/sandbox" %}
Interact with a sample installation with 0 setup.
Interact with a sample installation with 0 setup to explore our Discovery, Governance and Collaboration features.
{% /inlineCallout %}
{% /inlineCalloutContainer %}

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB