From cd3a2abff04b05505a033d9efc9d79c40472e2d5 Mon Sep 17 00:00:00 2001 From: Pere Miquel Brull Date: Wed, 26 Oct 2022 16:57:16 +0200 Subject: [PATCH] Airflow APIs docs (#8368) --- .../content/deployment/airflow/index.md | 80 ++++++++----------- 1 file changed, 35 insertions(+), 45 deletions(-) diff --git a/openmetadata-docs/content/deployment/airflow/index.md b/openmetadata-docs/content/deployment/airflow/index.md index 2370f76179b..63cb9a0bd9d 100644 --- a/openmetadata-docs/content/deployment/airflow/index.md +++ b/openmetadata-docs/content/deployment/airflow/index.md @@ -92,6 +92,23 @@ Then, check the Connector Modules guide above to learn how to install the `openm necessary plugins. They are necessary because even if we install the APIs, the Airflow instance needs to have the required libraries to connect to each source. +### AIRFLOW_HOME + +The APIs will look for the `AIRFLOW_HOME` environment variable to place the dynamically generated DAGs. Make +sure that the variable is set and reachable from Airflow. + +### Airflow APIs Basic Auth + +Note that the integration of OpenMetadata with Airflow requires Basic Auth in the APIs. Make sure that your +Airflow configuration supports that. You can read more about it [here](https://airflow.apache.org/docs/apache-airflow/stable/security/api.html). + +A possible approach here is to update your `airflow.cfg` entries with: + +``` +[api] +auth_backends = airflow.api.auth.backend.basic_auth +``` + ### Configure in the OpenMetadata Server After installing the Airflow APIs, you will need to update your OpenMetadata Server. @@ -134,50 +151,23 @@ It is important to do this validation passing the command as is (i.e., `curl -XG and allowing the environment to do the substitution for you. That's the only way we can be sure that the setup is correct. -#### Adding SSO +#### More validations in the installation -If you are running OpenMetadata with the security enabled, you can take a look at the server -configuration for each security mode: +If you have an existing DAG in Airflow, you can further test your setup by running the following: - - - Configure Auth0 SSO to access the UI and APIs - - - Configure Azure SSO to access the UI and APIs - - - Configure a Custom OIDC SSO to access the UI and APIs - - - Configure Google SSO to access the UI and APIs - - - Configure Okta SSO to access the UI and APIs - - +```bash +curl -XPOST http://localhost:8080/api/v1/openmetadata/enable --data-raw '{"dag_id": "example_bash_operator"}' -u "admin:admin" --header 'Content-Type: application/json' +``` + +Note that in this example we are assuming: +- There is an Airflow instance running at `localhost:8080`, +- There is a user `admin` with password `admin` +- There is a DAG named `example_bash_operator`. + +A generic call would look like: + +```bash +curl -XPOST /api/v1/openmetadata/enable --data-raw '{"dag_id": ""}' -u ":" --header 'Content-Type: application/json' +``` + +Please update it accordingly.