2022-07-11 16:54:40 +02:00
---
title: Run Airflow Connector using the CLI
2022-10-05 21:54:02 -07:00
slug: /connectors/pipeline/airflow/cli
2022-07-11 16:54:40 +02:00
---
2022-08-27 02:57:09 +02:00
# Run Airflow using the metadata CLI
2022-07-11 16:54:40 +02:00
2022-08-27 02:57:09 +02:00
In this section, we provide guides and references to use the Airbyte connector.
2022-07-11 16:54:40 +02:00
2022-08-27 02:57:09 +02:00
Configure and schedule Airbyte metadata and profiler workflows from the OpenMetadata UI:
- [Requirements ](#requirements )
- [Metadata Ingestion ](#metadata-ingestion )
## Requirements
< InlineCallout color = "violet-70" icon = "description" bold = "OpenMetadata 0.12 or later" href = "/deployment" >
To deploy OpenMetadata, check the < a href = "/deployment" > Deployment< / a > guides.
< / InlineCallout >
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
custom Airflow plugins to handle the workflow deployment.
### Python Requirements
2022-09-23 14:19:13 +02:00
To run the Airflow ingestion, you will need to install:
2022-08-27 02:57:09 +02:00
```bash
2022-09-23 14:19:13 +02:00
pip3 install "openmetadata-ingestion[airflow]"
2022-08-27 02:57:09 +02:00
```
2022-08-18 11:43:45 +02:00
Note that this installs the same Airflow version that we ship in the Ingestion Container, which is
Airflow `2.3.3` from Release `0.12` .
The ingestion using Airflow version 2.3.3 as a source package has been tested against Airflow 2.3.3 and Airflow 2.2.5.
2023-03-24 06:31:38 +01:00
< Note >
Note that we only support officially supported Airflow versions. You can check the version list [here ](https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html ).
< / Note >
2022-08-27 02:57:09 +02:00
## Metadata Ingestion
All connectors are defined as JSON Schemas.
2022-09-23 15:09:46 -07:00
[Here ](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/airbyteConnection.json )
2022-08-27 02:57:09 +02:00
you can find the structure to create a connection to Airbyte.
In order to create and run a Metadata Ingestion workflow, we will follow
the steps to create a YAML configuration able to connect to the source,
process the Entities if needed, and reach the OpenMetadata server.
The workflow is modeled around the following
2022-09-23 15:09:46 -07:00
[JSON Schema ](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json )
2022-08-27 02:57:09 +02:00
### 1. Define the YAML Config
This is a sample config for Airbyte:
```yaml
source:
type: airflow
serviceName: airflow_source
serviceConnection:
config:
type: Airflow
hostPort: http://localhost:8080
numberOfStatus: 10
# Connection needs to be one of Mysql, Postgres, Mssql or Sqlite
connection:
type: Mysql
username: airflow_user
password: airflow_pass
databaseSchema: airflow_db
hostPort: localhost:3306
# #
# type: Postgres
# username: airflow_user
# password: airflow_pass
# database: airflow_db
# hostPort: localhost:3306
# #
# type: Mssql
# username: airflow_user
# password: airflow_pass
# database: airflow_db
# hostPort: localhost:3306
# uriString: http://... (optional)
# #
# type: Sqlite
# username: airflow_user
# password: airflow_pass
# database: airflow_db
# hostPort: localhost:3306
# databaseMode: ":memory:" (optional)
sourceConfig:
config:
type: PipelineMetadata
2023-03-29 12:41:44 +05:30
# markDeletedPipelines: True
# includeTags: True
2022-08-27 02:57:09 +02:00
# includeLineage: true
# pipelineFilterPattern:
# includes:
# - pipeline1
# - pipeline2
# excludes:
# - pipeline3
# - pipeline4
sink:
type: metadata-rest
config: { }
workflowConfig:
2022-08-31 15:11:11 +02:00
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
2022-08-27 02:57:09 +02:00
openMetadataServerConfig:
2022-10-03 14:52:32 +05:30
hostPort: < OpenMetadata host and port >
authProvider: < OpenMetadata auth provider >
2022-08-27 02:57:09 +02:00
```
2022-07-11 16:54:40 +02:00
2022-08-27 02:57:09 +02:00
#### Source Configuration - Service Connection
2022-07-11 16:54:40 +02:00
- **hostPort**: URL to the Airflow instance.
- **numberOfStatus**: Number of status we want to look back to in every ingestion (e.g., Past executions from a DAG).
- **connection**: Airflow metadata database connection. See
these [docs ](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html )
for supported backends.
In terms of `connection` we support the following selections:
- `backend` : Should not be used from the UI. This is only applicable when ingesting Airflow metadata locally by running
the ingestion from a DAG. It will use the current Airflow SQLAlchemy connection to extract the data.
- `MySQL` , `Postgres` , `MSSQL` and `SQLite` : Pass the required credentials to reach out each of these services. We will
create a connection to the pointed database and read Airflow data from there.
2022-08-27 02:57:09 +02:00
#### Source Configuration - Source Config
2022-09-23 15:09:46 -07:00
The `sourceConfig` is defined [here ](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json ):
2022-08-27 02:57:09 +02:00
2023-01-31 20:47:40 +05:30
- `dbServiceNames` : Database Service Name for the creation of lineage, if the source supports it.
2022-08-27 02:57:09 +02:00
- `pipelineFilterPattern` and `chartFilterPattern` : Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g.,
2023-03-29 12:41:44 +05:30
- `includeTags` : Set the Include tags toggle to control whether or not to include tags as part of metadata ingestion.
- `markDeletedPipelines` : Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system.
2022-08-27 02:57:09 +02:00
```yaml
pipelineFilterPattern:
includes:
- users
- type_test
```
#### Sink Configuration
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest` .
#### Workflow Configuration
The main property here is the `openMetadataServerConfig` , where you can define the host and security provider of your OpenMetadata installation.
For a simple, local installation using our docker containers, this looks like:
```yaml
workflowConfig:
openMetadataServerConfig:
2022-10-03 14:52:32 +05:30
hostPort: 'http://localhost:8585/api'
authProvider: openmetadata
securityConfig:
jwtToken: '{bot_jwt_token}'
2022-08-27 02:57:09 +02:00
```
2022-09-23 15:09:46 -07:00
We support different security providers. You can find their definitions [here ](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client ).
2022-08-27 02:57:09 +02:00
You can find the different implementation of the ingestion below.
< Collapse title = "Configure SSO in the Ingestion Workflows" >
2022-10-03 14:52:32 +05:30
### Openmetadata JWT Auth
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: openmetadata
securityConfig:
jwtToken: '{bot_jwt_token}'
```
2022-08-27 02:57:09 +02:00
### Auth0 SSO
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: auth0
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
```
### Azure SSO
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: azure
securityConfig:
clientSecret: '{your_client_secret}'
authority: '{your_authority_url}'
clientId: '{your_client_id}'
scopes:
- your_scopes
```
### Custom OIDC SSO
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: custom-oidc
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
```
### Google SSO
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: google
securityConfig:
secretKey: '{path-to-json-creds}'
```
### Okta SSO
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: http://localhost:8585/api
authProvider: okta
securityConfig:
clientId: "{CLIENT_ID - SPA APP}"
orgURL: "{ISSUER_URL}/v1/token"
privateKey: "{public/private keypair}"
email: "{email}"
scopes:
- token
```
### Amazon Cognito SSO
The ingestion can be configured by [Enabling JWT Tokens ](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens )
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: auth0
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
```
### OneLogin SSO
Which uses Custom OIDC for the ingestion
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: custom-oidc
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
```
### KeyCloak SSO
Which uses Custom OIDC for the ingestion
```yaml
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: custom-oidc
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
```
< / Collapse >
### 2. Run with the CLI
First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
```bash
metadata ingest -c < path-to-yaml >
```
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration,
you will be able to extract metadata from different sources.