9b0b06fe25
* added flag for overriding owners * docs update * adding owners in patch call * adding owners in patch call * fix: python test case * changes as per comment * changes as per comment * ui changes
12 KiB
| title | slug |
|---|---|
| Run Tableau Connector using Airflow SDK | /connectors/dashboard/tableau/airflow |
Run Tableau using the Airflow SDK
In this section, we provide guides and references to use the Tableau connector.
Configure and schedule Tableau metadata and profiler workflows from the OpenMetadata UI:
Requirements
To ingest tableau metadata, minimum Site Role: Viewer is requried for the tableau user.
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment.
To create lineage between tableau dashboard and any database service via the queries provided from Tableau Metadata API, please enable the Tableau Metadata API for your tableau server. For more information on enabling the Tableau Metadata APIs follow the link here
Python Requirements
To run the Tableau ingestion, you will need to install:
pip3 install "openmetadata-ingestion[tableau]"
Metadata Ingestion
All connectors are defined as JSON Schemas. Here you can find the structure to create a connection to Tableau.
In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server.
The workflow is modeled around the following JSON Schema
1. Define the YAML Config
This is a sample config for Tableau:
source:
type: tableau
serviceName: local_tableau
serviceConnection:
config:
type: Tableau
username: username
password: password
env: tableau_prod
hostPort: http://localhost
siteName: site_name
siteUrl: site_url
apiVersion: api_version
# If not setting user and password
# personalAccessTokenName: personal_access_token_name
# personalAccessTokenSecret: personal_access_token_secret
sourceConfig:
config:
type: DashboardMetadata
# dbServiceNames:
# - service1
# - service2
# dashboardFilterPattern:
# includes:
# - dashboard1
# - dashboard2
# excludes:
# - dashboard3
# - dashboard4
# chartFilterPattern:
# includes:
# - chart1
# - chart2
# excludes:
# - chart3
# - chart4
sink:
type: metadata-rest
config: {}
workflowConfig:
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
openMetadataServerConfig:
hostPort: <OpenMetadata host and port>
authProvider: <OpenMetadata auth provider>
Example Source Configurations for default and non-default tableau sites
1. Sample config for default tableau site
For a default tableau site siteName and siteUrl fields should be kept as empty strings as shown in the below config.
source:
type: tableau
serviceName: local_tableau
serviceConnection:
config:
type: Tableau
username: username
password: password
env: tableau_prod
hostPort: http://localhost
siteName: ""
siteUrl: ""
apiVersion: api_version
# If not setting user and password
# personalAccessTokenName: personal_access_token_name
# personalAccessTokenSecret: personal_access_token_secret
sourceConfig:
config:
type: DashboardMetadata
# dbServiceNames:
# - service1
# - service2
# dashboardFilterPattern:
# includes:
# - dashboard1
# - dashboard2
# excludes:
# - dashboard3
# - dashboard4
# chartFilterPattern:
# includes:
# - chart1
# - chart2
# excludes:
# - chart3
# - chart4
sink:
type: metadata-rest
config: {}
workflowConfig:
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
openMetadataServerConfig:
hostPort: <OpenMetadata host and port>
authProvider: <OpenMetadata auth provider>
1. Sample config for non-default tableau site
For a non-default tableau site siteName and siteUrl fields are required.
If https://xxx.tableau.com/#/site/sitename/home represents the homepage url for your tableau site, the sitename from the url should be entered in the siteName and siteUrl fields in the config below.
source:
type: tableau
serviceName: local_tableau
serviceConnection:
config:
type: Tableau
username: username
password: password
env: tableau_prod
hostPort: http://localhost
siteName: openmetadata
siteUrl: openmetadata
apiVersion: api_version
# If not setting user and password
# personalAccessTokenName: personal_access_token_name
# personalAccessTokenSecret: personal_access_token_secret
sourceConfig:
config:
type: DashboardMetadata
overrideOwner: True
# dbServiceNames:
# - service1
# - service2
# dashboardFilterPattern:
# includes:
# - dashboard1
# - dashboard2
# excludes:
# - dashboard3
# - dashboard4
# chartFilterPattern:
# includes:
# - chart1
# - chart2
# excludes:
# - chart3
# - chart4
sink:
type: metadata-rest
config: {}
workflowConfig:
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
openMetadataServerConfig:
hostPort: <OpenMetadata host and port>
authProvider: <OpenMetadata auth provider>
Source Configuration - Service Connection
- hostPort: URL to the Tableau instance.
- username: Specify the User to connect to Tableau. It should have enough privileges to read all the metadata.
- password: Password for Tableau.
- apiVersion: Tableau API version.
- siteName: Tableau Site Name. To be kept empty if you are using the default Tableau site
- siteUrl: Tableau Site Url. To be kept empty if you are using the default Tableau site
- personalAccessTokenName: Access token. To be used if not logging in with user/password.
- personalAccessTokenSecret: Access token Secret. To be used if not logging in with user/password.
- env: Tableau Environment.
Source Configuration - Source Config
The sourceConfig is defined here:
dbServiceNames: Database Service Name for the creation of lineage, if the source supports it.dashboardFilterPatternandchartFilterPattern: Note that thedashboardFilterPatternandchartFilterPatternboth support regex as include or exclude. E.g.,overrideOwner: Flag to override current owner by new owner from source, if found during metadata ingestion
dashboardFilterPattern:
includes:
- users
- type_test
Sink Configuration
To send the metadata to OpenMetadata, it needs to be specified as type: metadata-rest.
Workflow Configuration
The main property here is the openMetadataServerConfig, where you can define the host and security provider of your OpenMetadata installation.
For a simple, local installation using our docker containers, this looks like:
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: openmetadata
securityConfig:
jwtToken: '{bot_jwt_token}'
We support different security providers. You can find their definitions here. You can find the different implementation of the ingestion below.
Openmetadata JWT Auth
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: openmetadata
securityConfig:
jwtToken: '{bot_jwt_token}'
Auth0 SSO
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: auth0
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
Azure SSO
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: azure
securityConfig:
clientSecret: '{your_client_secret}'
authority: '{your_authority_url}'
clientId: '{your_client_id}'
scopes:
- your_scopes
Custom OIDC SSO
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: custom-oidc
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
Google SSO
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: google
securityConfig:
secretKey: '{path-to-json-creds}'
Okta SSO
workflowConfig:
openMetadataServerConfig:
hostPort: http://localhost:8585/api
authProvider: okta
securityConfig:
clientId: "{CLIENT_ID - SPA APP}"
orgURL: "{ISSUER_URL}/v1/token"
privateKey: "{public/private keypair}"
email: "{email}"
scopes:
- token
Amazon Cognito SSO
The ingestion can be configured by Enabling JWT Tokens
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: auth0
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
OneLogin SSO
Which uses Custom OIDC for the ingestion
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: custom-oidc
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
KeyCloak SSO
Which uses Custom OIDC for the ingestion
workflowConfig:
openMetadataServerConfig:
hostPort: 'http://localhost:8585/api'
authProvider: custom-oidc
securityConfig:
clientId: '{your_client_id}'
secretKey: '{your_client_secret}'
domain: '{your_domain}'
2. Prepare the Ingestion DAG
Create a Python file in your Airflow DAGs directory with the following contents:
import pathlib
import yaml
from datetime import timedelta
from airflow import DAG
try:
from airflow.operators.python import PythonOperator
except ModuleNotFoundError:
from airflow.operators.python_operator import PythonOperator
from metadata.config.common import load_config_file
from metadata.ingestion.api.workflow import Workflow
from airflow.utils.dates import days_ago
default_args = {
"owner": "user_name",
"email": ["username@org.com"],
"email_on_failure": False,
"retries": 3,
"retry_delay": timedelta(minutes=5),
"execution_timeout": timedelta(minutes=60)
}
config = """
<your YAML configuration>
"""
def metadata_ingestion_workflow():
workflow_config = yaml.safe_load(config)
workflow = Workflow.create(workflow_config)
workflow.execute()
workflow.raise_from_status()
workflow.print_status()
workflow.stop()
with DAG(
"sample_data",
default_args=default_args,
description="An example DAG which runs a OpenMetadata ingestion workflow",
start_date=days_ago(1),
is_paused_upon_creation=False,
schedule_interval='*/5 * * * *',
catchup=False,
) as dag:
ingest_task = PythonOperator(
task_id="ingest_using_recipe",
python_callable=metadata_ingestion_workflow,
)
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources.