yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-19 15:31:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenMetadata/openmetadata-docs/content/v1.3.x/connectors/pipeline/airflow/index.md
Pere Miquel Brull ce15728327
MINOR - Docs 1.3 BETA & Slack Alert (#14961) 
* MINOR - Docs 1.3 BETA & Slack Alert

* MINOR - Docs 1.3 BETA & Slack Alert
2024-01-31 07:16:22 +01:00

2.5 KiB
Raw Blame History

title slug
Airflow /connectors/pipeline/airflow

Airflow

Stage PROD
Pipelines {% icon iconName="check" /%}
Pipeline Status {% icon iconName="check" /%}
Owners {% icon iconName="check" /%}
Tags {% icon iconName="cross" /%}
Lineage {% icon iconName="check" /%}

In this section, we provide guides and references to use the Airflow connector.

Configure and schedule Airflow metadata workflow from the OpenMetadata UI:

{% partial file="/v1.3/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/pipeline/airflow/yaml"} /%}

Requirements

{% note %} We only support officially supported Airflow versions. You can check the version list here. {% /note %}

Metadata Ingestion

{% partial file="/v1.3/connectors/metadata-ingestion-ui.md" variables={ connector: "Airflow", selectServicePath: "/images/v1.3/connectors/airflow/select-service.png", addNewServicePath: "/images/v1.3/connectors/airflow/add-new-service.png", serviceConnectionPath: "/images/v1.3/connectors/airflow/service-connection.png", } /%}

{% stepsContainer %} {% extraContent parentTagName="stepsContainer" %}

Connection Details

  • Host and Port: URL to the Airflow instance.
  • Number of Status: Number of status we want to look back to in every ingestion (e.g., Past executions from a DAG).
  • Connection: Airflow metadata database connection. See these docs for supported backends.

In terms of connection we support the following selections:

  • backend: Should not be used from the UI. This is only applicable when ingesting Airflow metadata locally by running the ingestion from a DAG. It will use the current Airflow SQLAlchemy connection to extract the data.
  • MySQL, Postgres, and SQLite: Pass the required credentials to reach out each of these services. We will create a connection to the pointed database and read Airflow data from there.

{% /extraContent %}

{% partial file="/v1.3/connectors/test-connection.md" /%}

{% partial file="/v1.3/connectors/pipeline/configure-ingestion.md" /%}

{% partial file="/v1.3/connectors/ingestion-schedule-and-deploy.md" /%}

{% /stepsContainer %}

{% partial file="/v1.3/connectors/troubleshooting.md" /%}