mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-08-23 16:38:17 +00:00
added docs for DOMO (#8543)
* added docs for DOMO * Changed As Per Comment * Change as per comment * Changes as per comment Co-authored-by: Meet <meet.s@deuexsolutions.com>
This commit is contained in:
Meet Parita Munesh Shah
GitHub
parent
c83690d07e
commit
9c360fb61c
@ -0,0 +1,320 @@
|
||||
---
|
||||
title: Run Domo Dashboard Connector using Airflow SDK
|
||||
slug: /connectors/dashboard/domo-dashboard/airflow
|
||||
---
|
||||
|
||||
# Run Domo Dashboard using the Airflow SDK
|
||||
|
||||
In this section, we provide guides and references to use the Domo Dashboard connector.
|
||||
|
||||
Configure and schedule Domo Dashboard metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
### Python Requirements
|
||||
|
||||
To run the Domo-Dashboard ingestion, you will need to install:
|
||||
|
||||
```bash
|
||||
pip3 install "openmetadata-ingestion[domo]"
|
||||
```
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
All connectors are defined as JSON Schemas.
|
||||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/lookerConnection.json)
|
||||
you can find the structure to create a connection to Domo Dashboard.
|
||||
|
||||
In order to create and run a Metadata Ingestion workflow, we will follow
|
||||
the steps to create a YAML configuration able to connect to the source,
|
||||
process the Entities if needed, and reach the OpenMetadata server.
|
||||
|
||||
The workflow is modeled around the following
|
||||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for Domo-Dashboard:
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: domodashboard
|
||||
serviceName: local_domodashboard
|
||||
serviceConnection:
|
||||
config:
|
||||
type: DomoDashboard
|
||||
clientId: clientid
|
||||
secretToken: secret-token
|
||||
accessToken: access-token
|
||||
apiHost: api.domo.com
|
||||
sandboxDomain: https://<api_domo>.domo.com
|
||||
sourceConfig:
|
||||
dashboardFilterPattern: {}
|
||||
chartFilterPattern: {}
|
||||
config:
|
||||
type: DashboardMetadata
|
||||
# dbServiceNames:
|
||||
# - service1
|
||||
# - service2
|
||||
# dashboardFilterPattern:
|
||||
# includes:
|
||||
# - dashboard1
|
||||
# - dashboard2
|
||||
# excludes:
|
||||
# - dashboard3
|
||||
# - dashboard4
|
||||
# chartFilterPattern:
|
||||
# includes:
|
||||
# - chart1
|
||||
# - chart2
|
||||
# excludes:
|
||||
# - chart3
|
||||
# - chart4
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
|
||||
#### Source Configuration - Service Connection
|
||||
|
||||
- **Client ID**: Client ID to Connect to DOMO Dashboard.
|
||||
- **Secret Token**: Secret Token to Connect DOMO Dashboard.
|
||||
- **Access Token**: Access to Connect to DOMO Dashboard.
|
||||
- **API Host**: API Host to Connect to DOMO Dashboard instance.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json):
|
||||
|
||||
- `dbServiceName`: Database Service Name for the creation of lineage, if the source supports it.
|
||||
- `dashboardFilterPattern` and `chartFilterPattern`: Note that the `dashboardFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g.,
|
||||
|
||||
```yaml
|
||||
dashboardFilterPattern:
|
||||
includes:
|
||||
- users
|
||||
- type_test
|
||||
```
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client).
|
||||
You can find the different implementation of the ingestion below.
|
||||
|
||||
<Collapse title="Configure SSO in the Ingestion Workflows">
|
||||
|
||||
### Openmetadata JWT Auth
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
### Auth0 SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Azure SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: azure
|
||||
securityConfig:
|
||||
clientSecret: '{your_client_secret}'
|
||||
authority: '{your_authority_url}'
|
||||
clientId: '{your_client_id}'
|
||||
scopes:
|
||||
- your_scopes
|
||||
```
|
||||
|
||||
### Custom OIDC SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Google SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: google
|
||||
securityConfig:
|
||||
secretKey: '{path-to-json-creds}'
|
||||
```
|
||||
|
||||
### Okta SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: okta
|
||||
securityConfig:
|
||||
clientId: "{CLIENT_ID - SPA APP}"
|
||||
orgURL: "{ISSUER_URL}/v1/token"
|
||||
privateKey: "{public/private keypair}"
|
||||
email: "{email}"
|
||||
scopes:
|
||||
- token
|
||||
```
|
||||
|
||||
### Amazon Cognito SSO
|
||||
|
||||
The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens)
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### OneLogin SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### KeyCloak SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
</Collapse>
|
||||
|
||||
## 2. Prepare the Ingestion DAG
|
||||
|
||||
Create a Python file in your Airflow DAGs directory with the following contents:
|
||||
|
||||
```python
|
||||
import pathlib
|
||||
import yaml
|
||||
from datetime import timedelta
|
||||
from airflow import DAG
|
||||
|
||||
try:
|
||||
from airflow.operators.python import PythonOperator
|
||||
except ModuleNotFoundError:
|
||||
from airflow.operators.python_operator import PythonOperator
|
||||
|
||||
from metadata.config.common import load_config_file
|
||||
from metadata.ingestion.api.workflow import Workflow
|
||||
from airflow.utils.dates import days_ago
|
||||
|
||||
default_args = {
|
||||
"owner": "user_name",
|
||||
"email": ["username@org.com"],
|
||||
"email_on_failure": False,
|
||||
"retries": 3,
|
||||
"retry_delay": timedelta(minutes=5),
|
||||
"execution_timeout": timedelta(minutes=60)
|
||||
}
|
||||
|
||||
config = """
|
||||
<your YAML configuration>
|
||||
"""
|
||||
|
||||
def metadata_ingestion_workflow():
|
||||
workflow_config = yaml.safe_load(config)
|
||||
workflow = Workflow.create(workflow_config)
|
||||
workflow.execute()
|
||||
workflow.raise_from_status()
|
||||
workflow.print_status()
|
||||
workflow.stop()
|
||||
|
||||
with DAG(
|
||||
"sample_data",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
schedule_interval='*/5 * * * *',
|
||||
catchup=False,
|
||||
) as dag:
|
||||
ingest_task = PythonOperator(
|
||||
task_id="ingest_using_recipe",
|
||||
python_callable=metadata_ingestion_workflow,
|
||||
)
|
||||
```
|
||||
|
||||
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will
|
||||
be able to extract metadata from different sources.
|
||||
@ -0,0 +1,275 @@
|
||||
---
|
||||
title: Run DomoDashboard Connector using the CLI
|
||||
slug: /connectors/dashboard/domo-dashboard/cli
|
||||
---
|
||||
|
||||
# Run Domo Dashboard using the metadata CLI
|
||||
|
||||
In this section, we provide guides and references to use the DomoDashboard connector.
|
||||
|
||||
Configure and schedule DomoDashboard metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
### Python Requirements
|
||||
|
||||
To run the DomoDashboard ingestion, you will need to install:
|
||||
|
||||
```bash
|
||||
pip3 install "openmetadata-ingestion[domo]"
|
||||
```
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
All connectors are defined as JSON Schemas.
|
||||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/lookerConnection.json)
|
||||
you can find the structure to create a connection to DomoDashboard.
|
||||
|
||||
In order to create and run a Metadata Ingestion workflow, we will follow
|
||||
the steps to create a YAML configuration able to connect to the source,
|
||||
process the Entities if needed, and reach the OpenMetadata server.
|
||||
|
||||
The workflow is modeled around the following
|
||||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for DomoDashboard:
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: domodashboard
|
||||
serviceName: local_domodashboard
|
||||
serviceConnection:
|
||||
config:
|
||||
type: DomoDashboard
|
||||
clientId: clientid
|
||||
secretToken: secret-token
|
||||
accessToken: access-token
|
||||
apiHost: api.domo.com
|
||||
sandboxDomain: https://<api_domo>.domo.com
|
||||
sourceConfig:
|
||||
config:
|
||||
dashboardFilterPattern: {}
|
||||
chartFilterPattern: {}
|
||||
type: DashboardMetadata
|
||||
# dbServiceNames:
|
||||
# - service1
|
||||
# - service2
|
||||
# dashboardFilterPattern:
|
||||
# includes:
|
||||
# - dashboard1
|
||||
# - dashboard2
|
||||
# excludes:
|
||||
# - dashboard3
|
||||
# - dashboard4
|
||||
# chartFilterPattern:
|
||||
# includes:
|
||||
# - chart1
|
||||
# - chart2
|
||||
# excludes:
|
||||
# - chart3
|
||||
# - chart4
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
|
||||
#### Source Configuration - Service Connection
|
||||
|
||||
- **Client ID**: Client ID to Connect to DOMO Dashboard.
|
||||
- **Secret Token**: Secret Token to Connect DOMO Dashboard.
|
||||
- **Access Token**: Access to Connect to DOMO Dashboard.
|
||||
- **API Host**: API Host to Connect to DOMO Dashboard instance.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json):
|
||||
|
||||
- `dbServiceName`: Database Service Name for the creation of lineage, if the source supports it.
|
||||
- `dashboardFilterPattern` and `chartFilterPattern`: Note that the `dashboardFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g.
|
||||
|
||||
```yaml
|
||||
dashboardFilterPattern:
|
||||
includes:
|
||||
- users
|
||||
- type_test
|
||||
```
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client).
|
||||
You can find the different implementation of the ingestion below.
|
||||
|
||||
<Collapse title="Configure SSO in the Ingestion Workflows">
|
||||
|
||||
### Openmetadata JWT Auth
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
### Auth0 SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Azure SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: azure
|
||||
securityConfig:
|
||||
clientSecret: '{your_client_secret}'
|
||||
authority: '{your_authority_url}'
|
||||
clientId: '{your_client_id}'
|
||||
scopes:
|
||||
- your_scopes
|
||||
```
|
||||
|
||||
### Custom OIDC SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Google SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: google
|
||||
securityConfig:
|
||||
secretKey: '{path-to-json-creds}'
|
||||
```
|
||||
|
||||
### Okta SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: okta
|
||||
securityConfig:
|
||||
clientId: "{CLIENT_ID - SPA APP}"
|
||||
orgURL: "{ISSUER_URL}/v1/token"
|
||||
privateKey: "{public/private keypair}"
|
||||
email: "{email}"
|
||||
scopes:
|
||||
- token
|
||||
```
|
||||
|
||||
### Amazon Cognito SSO
|
||||
|
||||
The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens)
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### OneLogin SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### KeyCloak SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
</Collapse>
|
||||
|
||||
|
||||
### 2. Run with the CLI
|
||||
|
||||
First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
|
||||
|
||||
```bash
|
||||
metadata ingest -c <path-to-yaml>
|
||||
```
|
||||
|
||||
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration,
|
||||
you will be able to extract metadata from different sources.
|
||||
@ -0,0 +1,207 @@
|
||||
---
|
||||
title: Domo Dashboard
|
||||
slug: /connectors/dashboard/domo-dashboard
|
||||
---
|
||||
|
||||
# Domo Dashboard
|
||||
|
||||
In this section, we provide guides and references to use the DomoDashboard connector.
|
||||
|
||||
Configure and schedule DomoDashboard metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
|
||||
If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check
|
||||
the following docs to connect using Airflow SDK or with the CLI.
|
||||
|
||||
<TileContainer>
|
||||
<Tile
|
||||
icon="air"
|
||||
title="Ingest with Airflow"
|
||||
text="Configure the ingestion using Airflow SDK"
|
||||
link="/connectors/dashboard/domo-dashboard/airflow"
|
||||
size="half"
|
||||
/>
|
||||
<Tile
|
||||
icon="account_tree"
|
||||
title="Ingest with the CLI"
|
||||
text="Run a one-time ingestion using the metadata CLI"
|
||||
link="/connectors/dashboard/domo-dashboard/cli"
|
||||
size="half"
|
||||
/>
|
||||
</TileContainer>
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
### 1. Visit the Services Page
|
||||
|
||||
The first step is ingesting the metadata from your sources. Under
|
||||
Settings, you will find a Services link an external source system to
|
||||
OpenMetadata. Once a service is created, it can be used to configure
|
||||
metadata, usage, and profiler workflows.
|
||||
|
||||
To visit the Services page, select Services from the Settings menu.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/visit-services.png"
|
||||
alt="Visit Services Page"
|
||||
caption="Find Services under the Settings menu"
|
||||
/>
|
||||
|
||||
### 2. Create a New Service
|
||||
|
||||
Click on the Add New Service button to start the Service creation.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/create-service.png"
|
||||
alt="Create a new service"
|
||||
caption="Add a new Service from the Services page"
|
||||
/>
|
||||
|
||||
### 3. Select the Service Type
|
||||
|
||||
Select Domo-Dashboard as the service type and click Next.
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domodashboard/domo-dashboard-service.png"
|
||||
alt="Select Service"
|
||||
caption="Select your service from the list"
|
||||
/>
|
||||
</div>
|
||||
|
||||
### 4. Name and Describe your Service
|
||||
|
||||
Provide a name and description for your service as illustrated below.
|
||||
|
||||
#### Service Name
|
||||
|
||||
OpenMetadata uniquely identifies services by their Service Name. Provide
|
||||
a name that distinguishes your deployment from other services, including
|
||||
the other {connector} services that you might be ingesting metadata
|
||||
from.
|
||||
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domodashboard/domo-dashboard-add-new-service.png"
|
||||
alt="Add New Service"
|
||||
caption="Provide a Name and description for your Service"
|
||||
/>
|
||||
</div>
|
||||
|
||||
|
||||
### 5. Configure the Service Connection
|
||||
|
||||
In this step, we will configure the connection settings required for
|
||||
this connector. Please follow the instructions below to ensure that
|
||||
you've configured the connector to read from your DomoDashboard service as
|
||||
desired.
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domodashboard/image1.png"
|
||||
alt="Configure service connection"
|
||||
caption="Configure the service connection by filling the form"
|
||||
/>
|
||||
</div>
|
||||
|
||||
|
||||
Once the credentials have been added, click on `Test Connection` and Save
|
||||
the changes.
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/test-connection.png"
|
||||
alt="Test Connection"
|
||||
caption="Test the connection and save the Service"
|
||||
/>
|
||||
</div>
|
||||
|
||||
#### Connection Options
|
||||
|
||||
- **Client ID**: Client ID to Connect to DOMO Dashboard.
|
||||
- **Secret Token**: Secret Token to Connect DOMO Dashboard.
|
||||
- **Access Token**: Access to Connect to DOMO Dashboard.
|
||||
- **API Host**: API Host to Connect to DOMO Dashboard instance.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
### 6. Configure Metadata Ingestion
|
||||
|
||||
In this step we will configure the metadata ingestion pipeline,
|
||||
Please follow the instructions below
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/configure-metadata-ingestion-dashboard.png"
|
||||
alt="Configure Metadata Ingestion"
|
||||
caption="Configure Metadata Ingestion Page"
|
||||
/>
|
||||
|
||||
#### Metadata Ingestion Options
|
||||
|
||||
- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
|
||||
- **Dashboard Filter Pattern (Optional)**: Use to dashboard filter patterns to control whether or not to include dashboard as part of metadata ingestion.
|
||||
- **Include**: Explicitly include dashboards by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be excluded.
|
||||
- **Exclude**: Explicitly exclude dashboards by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be included.
|
||||
- **Chart Pattern (Optional)**: Use to chart filter patterns to control whether or not to include charts as part of metadata ingestion.
|
||||
- **Include**: Explicitly include charts by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all charts with names matching one or more of the supplied regular expressions. All other charts will be excluded.
|
||||
- **Exclude**: Explicitly exclude charts by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all charts with names matching one or more of the supplied regular expressions. All other charts will be included.
|
||||
- **Database Service Name (Optional)**: Enter the name of Database Service which is already ingested in OpenMetadata to create lineage between dashboards and database tables.
|
||||
- **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug, these logs can be viewed later in Airflow.
|
||||
|
||||
### 7. Schedule the Ingestion and Deploy
|
||||
|
||||
Scheduling can be set up at an hourly, daily, or weekly cadence. The
|
||||
timezone is in UTC. Select a Start Date to schedule for ingestion. It is
|
||||
optional to add an End Date.
|
||||
|
||||
Review your configuration settings. If they match what you intended,
|
||||
click Deploy to create the service and schedule metadata ingestion.
|
||||
|
||||
If something doesn't look right, click the Back button to return to the
|
||||
appropriate step and change the settings as needed.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/schedule.png"
|
||||
alt="Schedule the Workflow"
|
||||
caption="Schedule the Ingestion Pipeline and Deploy"
|
||||
/>
|
||||
|
||||
After configuring the workflow, you can click on Deploy to create the
|
||||
pipeline.
|
||||
|
||||
### 8. View the Ingestion Pipeline
|
||||
|
||||
Once the workflow has been successfully deployed, you can view the
|
||||
Ingestion Pipeline running from the Service Page.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/view-ingestion-pipeline.png"
|
||||
alt="View Ingestion Pipeline"
|
||||
caption="View the Ingestion Pipeline from the Service Page"
|
||||
/>
|
||||
|
||||
### 9. Workflow Deployment Error
|
||||
|
||||
If there were any errors during the workflow deployment process, the
|
||||
Ingestion Pipeline Entity will still be created, but no workflow will be
|
||||
present in the Ingestion container.
|
||||
|
||||
You can then edit the Ingestion Pipeline and Deploy it again.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/workflow-deployment-error.png"
|
||||
alt="Workflow Deployment Error"
|
||||
caption="Edit and Deploy the Ingestion Pipeline"
|
||||
/>
|
||||
|
||||
From the Connection tab, you can also Edit the Service if needed.
|
||||
@ -12,3 +12,4 @@ slug: /connectors/dashboard
|
||||
- [Redash](/connectors/dashboard/redash)
|
||||
- [Superset](/connectors/dashboard/superset)
|
||||
- [Tableau](/connectors/dashboard/tableau)
|
||||
- [DomoDashboard](/connectors/dashboard/domo-dashboard)
|
||||
@ -0,0 +1,406 @@
|
||||
---
|
||||
title: Run DomoDatabase Connector using Airflow SDK
|
||||
slug: /connectors/database/domo-database/airflow
|
||||
---
|
||||
|
||||
# Run Domo Database using Airflow SDK
|
||||
|
||||
In this section, we provide guides and references to use the Domo Database connector
|
||||
|
||||
Configure and schedule DomoDatabase metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Data Profiler](#data-profiler)
|
||||
- [DBT Integration](#dbt-integration)
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
### Python Requirements
|
||||
|
||||
To run the DomoDatabase ingestion, you will need to install:
|
||||
|
||||
```bash
|
||||
pip3 install "openmetadata-ingestion[domo]"
|
||||
```
|
||||
## Metadata Ingestion
|
||||
|
||||
All connectors are defined as JSON Schemas.
|
||||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/verticaConnection.json)
|
||||
you can find the structure to create a connection to Vertica.
|
||||
|
||||
In order to create and run a Metadata Ingestion workflow, we will follow
|
||||
the steps to create a YAML configuration able to connect to the source,
|
||||
process the Entities if needed, and reach the OpenMetadata server.
|
||||
|
||||
The workflow is modeled around the following
|
||||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for DomoDatabase:
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: domodatabase
|
||||
serviceName: local_DomoDatabase
|
||||
serviceConnection:
|
||||
config:
|
||||
type: DomoDashboard
|
||||
clientId: client-id
|
||||
secretToken: secret-token
|
||||
accessToken: access-token
|
||||
apiHost: api.domo.com
|
||||
sandboxDomain: https://<api_domo>.domo.com
|
||||
|
||||
# database: database
|
||||
sourceConfig:
|
||||
config:
|
||||
markDeletedTables: true
|
||||
includeTables: true
|
||||
includeViews: true
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>2. Configure service settings
|
||||
```
|
||||
|
||||
#### Source Configuration - Service Connection
|
||||
|
||||
- **Client ID**: Client ID to Connect to DOMODatabase.
|
||||
- **Secret Token**: Secret Token to Connect DOMODatabase.
|
||||
- **Access Token**: Access to Connect to DOMODatabase.
|
||||
- **API Host**: API Host to Connect to DOMODatabase instance.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
- `markDeletedTables`: To flag tables as soft-deleted if they are not present anymore in the source system.
|
||||
- `includeTables`: true or false, to ingest table data. Default is true.
|
||||
- `includeViews`: true or false, to ingest views definitions.
|
||||
- `databaseFilterPattern`, `schemaFilterPattern`, `tableFilternPattern`: Note that the they support regex as include or exclude. E.g.,
|
||||
|
||||
```yaml
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- users
|
||||
- type_test
|
||||
```
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client).
|
||||
You can find the different implementation of the ingestion below.
|
||||
|
||||
<Collapse title="Configure SSO in the Ingestion Workflows">
|
||||
|
||||
### Openmetadata JWT Auth
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
### Auth0 SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Azure SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: azure
|
||||
securityConfig:
|
||||
clientSecret: '{your_client_secret}'
|
||||
authority: '{your_authority_url}'
|
||||
clientId: '{your_client_id}'
|
||||
scopes:
|
||||
- your_scopes
|
||||
```
|
||||
|
||||
### Custom OIDC SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
### Google SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: google
|
||||
securityConfig:
|
||||
secretKey: '{path-to-json-creds}'
|
||||
```
|
||||
|
||||
### Okta SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: okta
|
||||
securityConfig:
|
||||
clientId: "{CLIENT_ID - SPA APP}"
|
||||
orgURL: "{ISSUER_URL}/v1/token"
|
||||
privateKey: "{public/private keypair}"
|
||||
email: "{email}"
|
||||
scopes:
|
||||
- token
|
||||
```
|
||||
|
||||
### Amazon Cognito SSO
|
||||
|
||||
The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens)
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### OneLogin SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### KeyCloak SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
</Collapse>
|
||||
|
||||
### 2. Prepare the Ingestion DAG
|
||||
|
||||
Create a Python file in your Airflow DAGs directory with the following contents:
|
||||
|
||||
```python
|
||||
import pathlib
|
||||
import yaml
|
||||
from datetime import timedelta
|
||||
from airflow import DAG
|
||||
|
||||
try:
|
||||
from airflow.operators.python import PythonOperator
|
||||
except ModuleNotFoundError:
|
||||
from airflow.operators.python_operator import PythonOperator
|
||||
|
||||
from metadata.config.common import load_config_file
|
||||
from metadata.ingestion.api.workflow import Workflow
|
||||
from airflow.utils.dates import days_ago
|
||||
|
||||
default_args = {
|
||||
"owner": "user_name",
|
||||
"email": ["username@org.com"],
|
||||
"email_on_failure": False,
|
||||
"retries": 3,
|
||||
"retry_delay": timedelta(minutes=5),
|
||||
"execution_timeout": timedelta(minutes=60)
|
||||
}
|
||||
|
||||
config = """
|
||||
<your YAML configuration>
|
||||
"""
|
||||
|
||||
def metadata_ingestion_workflow():
|
||||
workflow_config = yaml.safe_load(config)
|
||||
workflow = Workflow.create(workflow_config)
|
||||
workflow.execute()
|
||||
workflow.raise_from_status()
|
||||
workflow.print_status()
|
||||
workflow.stop()
|
||||
|
||||
with DAG(
|
||||
"sample_data",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
schedule_interval='*/5 * * * *',
|
||||
catchup=False,
|
||||
) as dag:
|
||||
ingest_task = PythonOperator(
|
||||
task_id="ingest_using_recipe",
|
||||
python_callable=metadata_ingestion_workflow,
|
||||
)
|
||||
```
|
||||
Note that from connector to connector, this recipe will always be the same.
|
||||
By updating the YAML configuration, you will be able to extract metadata from different sources.
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for the profiler:
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: domodatabase
|
||||
serviceName: <service name>
|
||||
serviceConnection:
|
||||
config:
|
||||
type: DomoDatabase
|
||||
clientId: clientid
|
||||
secretToken: secret Token
|
||||
accessToken: access Token
|
||||
apiHost: api.domo.com
|
||||
sandboxDomain: https://<api_domo>.domo.com
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
|
||||
#### Source Configuration
|
||||
|
||||
- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/athenaConnection.json).
|
||||
- The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json).
|
||||
|
||||
Note that the filter patterns support regex as includes or excludes. E.g.,
|
||||
|
||||
```yaml
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- *users$
|
||||
```
|
||||
#### Workflow Configuration
|
||||
|
||||
The same as the metadata ingestion.
|
||||
|
||||
### 2. Prepare the Profiler DAG
|
||||
|
||||
Here, we follow a similar approach as with the metadata and usage pipelines, although we will use a different Workflow class:
|
||||
|
||||
```python
|
||||
import yaml
|
||||
from datetime import timedelta
|
||||
|
||||
from airflow import DAG
|
||||
|
||||
try:
|
||||
from airflow.operators.python import PythonOperator
|
||||
except ModuleNotFoundError:
|
||||
from airflow.operators.python_operator import PythonOperator
|
||||
|
||||
from airflow.utils.dates import days_ago
|
||||
|
||||
from metadata.orm_profiler.api.workflow import ProfilerWorkflow
|
||||
|
||||
|
||||
default_args = {
|
||||
"owner": "user_name",
|
||||
"email_on_failure": False,
|
||||
"retries": 3,
|
||||
"retry_delay": timedelta(seconds=10),
|
||||
"execution_timeout": timedelta(minutes=60),
|
||||
}
|
||||
|
||||
config = """
|
||||
<your YAML configuration>
|
||||
"""
|
||||
|
||||
def metadata_ingestion_workflow():
|
||||
workflow_config = yaml.safe_load(config)
|
||||
workflow = ProfilerWorkflow.create(workflow_config)
|
||||
workflow.execute()
|
||||
workflow.raise_from_status()
|
||||
workflow.print_status()
|
||||
workflow.stop()
|
||||
|
||||
with DAG(
|
||||
"profiler_example",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
catchup=False,
|
||||
) as dag:
|
||||
ingest_task = PythonOperator(
|
||||
task_id="profile_and_test_using_recipe",
|
||||
python_callable=metadata_ingestion_workflow,
|
||||
)
|
||||
```
|
||||
@ -0,0 +1,345 @@
|
||||
---
|
||||
title: Run DomoDatabase Connector using the CLI
|
||||
slug: /connectors/database/domo-database/cli
|
||||
---
|
||||
|
||||
# Run Domo Database using the metadata CLI
|
||||
|
||||
In this section, we provide guides and references to use the Domo Database connector.
|
||||
|
||||
Configure and schedule DomoDatabase metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Data Profiler](#data-profiler)
|
||||
- [DBT Integration](#dbt-integration)
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
### Python Requirements
|
||||
|
||||
To run the DomoDatabase ingestion, you will need to install:
|
||||
|
||||
```bash
|
||||
pip3 install "openmetadata-ingestion[domo]"
|
||||
```
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
All connectors are defined as JSON Schemas.
|
||||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/athenaConnection.json)
|
||||
you can find the structure to create a connection to DomoDatbase.
|
||||
|
||||
In order to create and run a Metadata Ingestion workflow, we will follow
|
||||
the steps to create a YAML configuration able to connect to the source,
|
||||
process the Entities if needed, and reach the OpenMetadata server.
|
||||
|
||||
The workflow is modeled around the following
|
||||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for DomoDatabase:
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: domodatabase
|
||||
serviceName: local_domodatabase
|
||||
serviceConnection:
|
||||
config:
|
||||
type: DomoDatabase
|
||||
clientId: clientid
|
||||
secretToken: secret-token
|
||||
accessToken: access-token
|
||||
apiHost: api.domo.com
|
||||
sandboxDomain: https://<api_domo>.domo.com
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
|
||||
```
|
||||
|
||||
#### Source Configuration - Service Connection
|
||||
|
||||
- **Client ID**: Client ID to Connect to DOMO Database.
|
||||
- **Secret Token**: Secret Token to Connect DOMO Database.
|
||||
- **Access Token**: Access to Connect to DOMO Database.
|
||||
- **API Host**: API Host to Connect to DOMO Database instance.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
- `markDeletedTables`: To flag tables as soft-deleted if they are not present anymore in the source system.
|
||||
- `includeTables`: true or false, to ingest table data. Default is true.
|
||||
- `includeViews`: true or false, to ingest views definitions.
|
||||
- `databaseFilterPattern`, `schemaFilterPattern`, `tableFilternPattern`: Note that the they support regex as include or exclude. E.g.,
|
||||
|
||||
```yaml
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- users
|
||||
- type_test
|
||||
```
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client).
|
||||
You can find the different implementation of the ingestion below.
|
||||
|
||||
<Collapse title="Configure SSO in the Ingestion Workflows">
|
||||
|
||||
### Openmetadata JWT Auth
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
### Auth0 SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Azure SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: azure
|
||||
securityConfig:
|
||||
clientSecret: '{your_client_secret}'
|
||||
authority: '{your_authority_url}'
|
||||
clientId: '{your_client_id}'
|
||||
scopes:
|
||||
- your_scopes
|
||||
```
|
||||
|
||||
### Custom OIDC SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Google SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: google
|
||||
securityConfig:
|
||||
secretKey: '{path-to-json-creds}'
|
||||
```
|
||||
|
||||
### Okta SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: okta
|
||||
securityConfig:
|
||||
clientId: "{CLIENT_ID - SPA APP}"
|
||||
orgURL: "{ISSUER_URL}/v1/token"
|
||||
privateKey: "{public/private keypair}"
|
||||
email: "{email}"
|
||||
scopes:
|
||||
- token
|
||||
```
|
||||
|
||||
### Amazon Cognito SSO
|
||||
|
||||
The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens)
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### OneLogin SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### KeyCloak SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
</Collapse>
|
||||
|
||||
### 2. Run with the CLI
|
||||
|
||||
First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
|
||||
|
||||
```bash
|
||||
metadata ingest -c <path-to-yaml>
|
||||
```
|
||||
|
||||
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration,
|
||||
you will be able to extract metadata from different sources.
|
||||
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for the profiler:
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: domodatabase
|
||||
serviceName: <service name>
|
||||
serviceConnection:
|
||||
config:
|
||||
type: DomoDatabase
|
||||
type: DomoDashboard
|
||||
clientId: client-id
|
||||
secretToken: secret-token
|
||||
accessToken: access-token
|
||||
apiHost: api.domo.com
|
||||
sandboxDomain: https://<api_domo>.domo.com
|
||||
# endPointURL: https://athena.us-east-2.amazonaws.com/
|
||||
# awsSessionToken: TOKEN
|
||||
s3StagingDir: s3 directory for datasource
|
||||
workgroup: workgroup name
|
||||
sourceConfig:
|
||||
config:
|
||||
type: Profiler
|
||||
# generateSampleData: true
|
||||
# profileSample: 85
|
||||
# threadCount: 5 (default)
|
||||
# databaseFilterPattern:
|
||||
# includes:
|
||||
# - database1
|
||||
# - database2
|
||||
# excludes:
|
||||
# - database3
|
||||
# - database4
|
||||
# schemaFilterPattern:
|
||||
# includes:
|
||||
# - schema1
|
||||
# - schema2
|
||||
# excludes:
|
||||
# - schema3
|
||||
# - schema4
|
||||
# tableFilterPattern:
|
||||
# includes:
|
||||
# - table1
|
||||
# - table2
|
||||
# excludes:
|
||||
# - table3
|
||||
# - table4
|
||||
- ...
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
|
||||
#### Source Configuration
|
||||
|
||||
- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/athenaConnection.json).
|
||||
- The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json).
|
||||
|
||||
Note that the filter patterns support regex as includes or excludes. E.g.,
|
||||
|
||||
```yaml
|
||||
tableFilterPattern:
|
||||
includes:
|
||||
- *users$
|
||||
```
|
||||
#### Workflow Configuration
|
||||
|
||||
The same as the metadata ingestion.
|
||||
|
||||
### 2. Run with the CLI
|
||||
|
||||
After saving the YAML config, we will run the command the same way we did for the metadata ingestion:
|
||||
|
||||
```bash
|
||||
metadata profile -c <path-to-yaml>
|
||||
```
|
||||
|
||||
Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow.
|
||||
@ -0,0 +1,217 @@
|
||||
---
|
||||
title: DomoDatabase
|
||||
slug: /connectors/database/domo-database
|
||||
---
|
||||
|
||||
# DomoDatabase
|
||||
|
||||
In this section, we provide guides and references to use the DomoDatabase connector.
|
||||
|
||||
Configure and schedule DomoDatabase metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Data Profiler](#data-profiler)
|
||||
- [DBT Integration](#dbt-integration)
|
||||
|
||||
If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check
|
||||
the following docs to connect using Airflow SDK or with the CLI.
|
||||
|
||||
<TileContainer>
|
||||
<Tile
|
||||
icon="air"
|
||||
title="Ingest with Airflow"
|
||||
text="Configure the ingestion using Airflow SDK"
|
||||
link="/connectors/database/domo-database/airflow"
|
||||
size="half"
|
||||
/>
|
||||
<Tile
|
||||
icon="account_tree"
|
||||
title="Ingest with the CLI"
|
||||
text="Run a one-time ingestion using the metadata CLI"
|
||||
link="/connectors/database/domo-database/cli"
|
||||
size="half"
|
||||
/>
|
||||
</TileContainer>
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
### 1. Visit the Services Page
|
||||
|
||||
The first step is ingesting the metadata from your sources. Under
|
||||
Settings, you will find a Services link an external source system to
|
||||
OpenMetadata. Once a service is created, it can be used to configure
|
||||
metadata, usage, and profiler workflows.
|
||||
|
||||
To visit the Services page, select Services from the Settings menu.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/visit-services.png"
|
||||
alt="Visit Services Page"
|
||||
caption="Find Services under the Settings menu"
|
||||
/>
|
||||
|
||||
### 2. Create a New Service
|
||||
|
||||
Click on the Add New Service button to start the Service creation.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/create-service.png"
|
||||
alt="Create a new service"
|
||||
caption="Add a new Service from the Services page"
|
||||
/>
|
||||
|
||||
### 3. Select the Service Type
|
||||
|
||||
Select Domo-Database as the service type and click Next.
|
||||
|
||||
<div className="w-110 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domodatabase/image.png"
|
||||
alt="Select Service"
|
||||
caption="Select your service from the list"
|
||||
/>
|
||||
</div>
|
||||
|
||||
### 4. Name and Describe your Service
|
||||
|
||||
Provide a name and description for your service as illustrated below.
|
||||
|
||||
#### Service Name
|
||||
|
||||
OpenMetadata uniquely identifies services by their Service Name. Provide
|
||||
a name that distinguishes your deployment from other services, including
|
||||
the other {connector} services that you might be ingesting metadata
|
||||
from.
|
||||
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domodatabase/domo-service-page.png"
|
||||
alt="Add New Service"
|
||||
caption="Provide a Name and description for your Service"
|
||||
/>
|
||||
</div>
|
||||
|
||||
|
||||
### 5. Configure the Service Connection
|
||||
|
||||
In this step, we will configure the connection settings required for
|
||||
this connector. Please follow the instructions below to ensure that
|
||||
you've configured the connector to read from your domodatabase service as
|
||||
desired.
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domodatabase/domo-ingestion-pipeline.png"
|
||||
alt="Configure service connection"
|
||||
caption="Configure the service connection by filling the form"
|
||||
/>
|
||||
</div>
|
||||
|
||||
|
||||
Once the credentials have been added, click on `Test Connection` and Save
|
||||
the changes.
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/test-connection.png"
|
||||
alt="Test Connection"
|
||||
caption="Test the connection and save the Service"
|
||||
/>
|
||||
</div>
|
||||
|
||||
#### Connection Options
|
||||
|
||||
- **Client ID**: Client ID for DOMO Database.
|
||||
- **Secret Token**: Secret Token to Connect DOMO Database.
|
||||
- **Access Token**: Access to Connect to DOMO Database.
|
||||
- **Api Host**: API Host to Connect to DOMO Database instance.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
### 6. Configure Metadata Ingestion
|
||||
|
||||
In this step we will configure the metadata ingestion pipeline,
|
||||
Please follow the instructions below
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/configure-metadata-ingestion-database.png"
|
||||
alt="Configure Metadata Ingestion"
|
||||
caption="Configure Metadata Ingestion Page"
|
||||
/>
|
||||
|
||||
#### Metadata Ingestion Options
|
||||
|
||||
- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
|
||||
- **Database Filter Pattern (Optional)**: Use to database filter patterns to control whether or not to include database as part of metadata ingestion.
|
||||
- **Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded.
|
||||
- **Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included.
|
||||
- **Schema Filter Pattern (Optional)**: Use to schema filter patterns to control whether or not to include schemas as part of metadata ingestion.
|
||||
- **Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
|
||||
- **Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included.
|
||||
- **Table Filter Pattern (Optional)**: Use to table filter patterns to control whether or not to include tables as part of metadata ingestion.
|
||||
- **Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded.
|
||||
- **Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included.
|
||||
- **Include views (toggle)**: Set the Include views toggle to control whether or not to include views as part of metadata ingestion.
|
||||
- **Include tags (toggle)**: Set the Include tags toggle to control whether or not to include tags as part of metadata ingestion.
|
||||
- **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug, these logs can be viewed later in Airflow.
|
||||
- **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system.
|
||||
- **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema
|
||||
|
||||
### 7. Schedule the Ingestion and Deploy
|
||||
|
||||
Scheduling can be set up at an hourly, daily, or weekly cadence. The
|
||||
timezone is in UTC. Select a Start Date to schedule for ingestion. It is
|
||||
optional to add an End Date.
|
||||
|
||||
Review your configuration settings. If they match what you intended,
|
||||
click Deploy to create the service and schedule metadata ingestion.
|
||||
|
||||
If something doesn't look right, click the Back button to return to the
|
||||
appropriate step and change the settings as needed.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/schedule.png"
|
||||
alt="Schedule the Workflow"
|
||||
caption="Schedule the Ingestion Pipeline and Deploy"
|
||||
/>
|
||||
|
||||
After configuring the workflow, you can click on Deploy to create the
|
||||
pipeline.
|
||||
|
||||
### 8. View the Ingestion Pipeline
|
||||
|
||||
Once the workflow has been successfully deployed, you can view the
|
||||
Ingestion Pipeline running from the Service Page.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/view-ingestion-pipeline.png"
|
||||
alt="View Ingestion Pipeline"
|
||||
caption="View the Ingestion Pipeline from the Service Page"
|
||||
/>
|
||||
|
||||
### 9. Workflow Deployment Error
|
||||
|
||||
If there were any errors during the workflow deployment process, the
|
||||
Ingestion Pipeline Entity will still be created, but no workflow will be
|
||||
present in the Ingestion container.
|
||||
|
||||
You can then edit the Ingestion Pipeline and Deploy it again.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/workflow-deployment-error.png"
|
||||
alt="Workflow Deployment Error"
|
||||
caption="Edit and Deploy the Ingestion Pipeline"
|
||||
/>
|
||||
|
||||
From the Connection tab, you can also Edit the Service if needed.
|
||||
|
||||
|
||||
@ -54,6 +54,7 @@ OpenMetadata can extract metadata from the following list of 55 connectors:
|
||||
- SQL Profiles (SQL based systems)
|
||||
- [Trino](/connectors/database/trino)
|
||||
- [Vertica](/connectors/database/vertica)
|
||||
- [Domo Database](/connectors/database/domo-database)
|
||||
|
||||
## Dashboard Services
|
||||
|
||||
@ -64,6 +65,7 @@ OpenMetadata can extract metadata from the following list of 55 connectors:
|
||||
- [Redash](/connectors/dashboard/redash)
|
||||
- [Superset](/connectors/dashboard/superset)
|
||||
- [Tableau](/connectors/dashboard/tableau)
|
||||
- [Domo Dashboard](/connectors/dashboard/domo-dashboard)
|
||||
|
||||
## Messaging Services
|
||||
|
||||
@ -79,6 +81,7 @@ OpenMetadata can extract metadata from the following list of 55 connectors:
|
||||
- [Dagster](/connectors/pipeline/dagster)
|
||||
- [Fivetran](/connectors/pipeline/fivetran)
|
||||
- [Glue](/connectors/pipeline/glue-pipeline)
|
||||
- [Domo Pipeline](/connectors/pipeline/domo-pipeline)
|
||||
- NiFi
|
||||
|
||||
## ML Model Services
|
||||
|
||||
@ -0,0 +1,305 @@
|
||||
---
|
||||
title: Run Domo Pipeline Connector using Airflow SDK
|
||||
slug: /connectors/pipeline/domo-pipeline/airflow
|
||||
---
|
||||
|
||||
# Run Domo Pipeline using the Airflow SDK
|
||||
|
||||
In this section, we provide guides and references to use the Domo-Pipeline connector.
|
||||
|
||||
Configure and schedule Domo-Pipeline metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
### Python Requirements
|
||||
|
||||
To run the domopipeline ingestion, you will need to install:
|
||||
|
||||
```bash
|
||||
pip3 install "openmetadata-ingestion[domo]"
|
||||
```
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
All connectors are defined as JSON Schemas.
|
||||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/airbyteConnection.json)
|
||||
you can find the structure to create a connection to Airbyte.
|
||||
|
||||
In order to create and run a Metadata Ingestion workflow, we will follow
|
||||
the steps to create a YAML configuration able to connect to the source,
|
||||
process the Entities if needed, and reach the OpenMetadata server.
|
||||
|
||||
The workflow is modeled around the following
|
||||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for Domo-Pipeline:
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: domopipeline
|
||||
serviceName: domo-pipeline_source
|
||||
serviceConnection:
|
||||
config:
|
||||
type: DomoPipeline
|
||||
clientID: clientid
|
||||
secretToken: secret-token
|
||||
accessToken: access-token
|
||||
apiHost: api.domo.com
|
||||
sandboxDomain: https://<api_domo>.domo.com
|
||||
sourceConfig:
|
||||
config:
|
||||
pipelineFilterPattern: {}
|
||||
type: PipelineMetadata
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
securityconfig:
|
||||
jwtToken:
|
||||
|
||||
```
|
||||
|
||||
#### Source Configuration - Service Connection
|
||||
|
||||
- **Client ID**: Client ID to Connect to DOMO Pipeline.
|
||||
- **Secret Token**: Secret Token to Connect DOMO Pipeline.
|
||||
- **Access Token**: Access to Connect to DOMO Pipeline.
|
||||
- **API Host**: API Host to Connect to DOMO Pipeline instance.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json):
|
||||
|
||||
- `dbServiceName`: Database Service Name for the creation of lineage, if the source supports it.
|
||||
- `pipelineFilterPattern` and `chartFilterPattern`: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g.,
|
||||
|
||||
```yaml
|
||||
pipelineFilterPattern:
|
||||
includes:
|
||||
- users
|
||||
- type_test
|
||||
```
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client).
|
||||
You can find the different implementation of the ingestion below.
|
||||
|
||||
<Collapse title="Configure SSO in the Ingestion Workflows">
|
||||
|
||||
### Openmetadata JWT Auth
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
### Auth0 SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Azure SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: azure
|
||||
securityConfig:
|
||||
clientSecret: '{your_client_secret}'
|
||||
authority: '{your_authority_url}'
|
||||
clientId: '{your_client_id}'
|
||||
scopes:
|
||||
- your_scopes
|
||||
```
|
||||
|
||||
### Custom OIDC SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Google SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: google
|
||||
securityConfig:
|
||||
secretKey: '{path-to-json-creds}'
|
||||
```
|
||||
|
||||
### Okta SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: okta
|
||||
securityConfig:
|
||||
clientId: "{CLIENT_ID - SPA APP}"
|
||||
orgURL: "{ISSUER_URL}/v1/token"
|
||||
privateKey: "{public/private keypair}"
|
||||
email: "{email}"
|
||||
scopes:
|
||||
- token
|
||||
```
|
||||
|
||||
### Amazon Cognito SSO
|
||||
|
||||
The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens)
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### OneLogin SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### KeyCloak SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
</Collapse>
|
||||
|
||||
## 2. Prepare the Ingestion DAG
|
||||
|
||||
Create a Python file in your Airflow DAGs directory with the following contents:
|
||||
|
||||
```python
|
||||
import pathlib
|
||||
import yaml
|
||||
from datetime import timedelta
|
||||
from airflow import DAG
|
||||
|
||||
try:
|
||||
from airflow.operators.python import PythonOperator
|
||||
except ModuleNotFoundError:
|
||||
from airflow.operators.python_operator import PythonOperator
|
||||
|
||||
from metadata.config.common import load_config_file
|
||||
from metadata.ingestion.api.workflow import Workflow
|
||||
from airflow.utils.dates import days_ago
|
||||
|
||||
default_args = {
|
||||
"owner": "user_name",
|
||||
"email": ["username@org.com"],
|
||||
"email_on_failure": False,
|
||||
"retries": 3,
|
||||
"retry_delay": timedelta(minutes=5),
|
||||
"execution_timeout": timedelta(minutes=60)
|
||||
}
|
||||
|
||||
config = """
|
||||
<your YAML configuration>
|
||||
"""
|
||||
|
||||
def metadata_ingestion_workflow():
|
||||
workflow_config = yaml.safe_load(config)
|
||||
workflow = Workflow.create(workflow_config)
|
||||
workflow.execute()
|
||||
workflow.raise_from_status()
|
||||
workflow.print_status()
|
||||
workflow.stop()
|
||||
|
||||
with DAG(
|
||||
"sample_data",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
schedule_interval='*/5 * * * *',
|
||||
catchup=False,
|
||||
) as dag:
|
||||
ingest_task = PythonOperator(
|
||||
task_id="ingest_using_recipe",
|
||||
python_callable=metadata_ingestion_workflow,
|
||||
)
|
||||
```
|
||||
|
||||
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will
|
||||
be able to extract metadata from different sources.
|
||||
@ -0,0 +1,265 @@
|
||||
---
|
||||
title: Run Domo Pipeline Connector using the CLI
|
||||
slug: /connectors/pipeline/domo-pipeline/cli
|
||||
---
|
||||
|
||||
# Run Domo Pipeline using the Metadata CLI
|
||||
|
||||
In this section, we provide guides and references to use the Domo Pipeline connector.
|
||||
|
||||
Configure and schedule Domo Pipeline metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
### Python Requirements
|
||||
|
||||
To run the Domo Pipeline ingestion, you will need to install:
|
||||
|
||||
```bash
|
||||
pip3 install "openmetadata-ingestion[domo]"
|
||||
```
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
All connectors are defined as JSON Schemas.
|
||||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/airbyteConnection.json)
|
||||
you can find the structure to create a connection to Airbyte.
|
||||
|
||||
In order to create and run a Metadata Ingestion workflow, we will follow
|
||||
the steps to create a YAML configuration able to connect to the source,
|
||||
process the Entities if needed, and reach the OpenMetadata server.
|
||||
|
||||
The workflow is modeled around the following
|
||||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for Domo-Pipeline:
|
||||
|
||||
```yaml
|
||||
source:
|
||||
type: domopipeline
|
||||
serviceName: domopipeline_source
|
||||
serviceConnection:
|
||||
config:
|
||||
type: DomoPipeline
|
||||
clientId: clientid
|
||||
secretToken: secret-token
|
||||
accessToken: access-token
|
||||
apiHost: api.domo.com
|
||||
sandboxDomain: https://<api_domo>.domo.com
|
||||
sourceConfig:
|
||||
config:
|
||||
pipelineFilterPattern: {}
|
||||
type: PipelineMetadata
|
||||
# pipelineFilterPattern:
|
||||
# includes:
|
||||
# - pipeline1
|
||||
# - pipeline2
|
||||
# excludes:
|
||||
# - pipeline3
|
||||
# - pipeline4
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
securityconfig:
|
||||
jwtToken:
|
||||
```
|
||||
|
||||
#### Source Configuration - Service Connection
|
||||
|
||||
- **Client ID**: Client ID to Connect to DOMO Pipeline.
|
||||
- **Secret Token**: Secret Token to Connect DOMO Pipeline.
|
||||
- **Access Token**: Access to Connect to DOMO Pipeline.
|
||||
- **API Host**: API Host to Connect to DOMO Pipeline instance.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json):
|
||||
|
||||
- `dbServiceName`: Database Service Name for the creation of lineage, if the source supports it.
|
||||
- `pipelineFilterPattern` and `chartFilterPattern`: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g.,
|
||||
|
||||
```yaml
|
||||
pipelineFilterPattern:
|
||||
includes:
|
||||
- users
|
||||
- type_test
|
||||
```
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
|
||||
We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client).
|
||||
You can find the different implementation of the ingestion below.
|
||||
|
||||
<Collapse title="Configure SSO in the Ingestion Workflows">
|
||||
|
||||
### Openmetadata JWT Auth
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: openmetadata
|
||||
securityConfig:
|
||||
jwtToken: '{bot_jwt_token}'
|
||||
```
|
||||
|
||||
### Auth0 SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Azure SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: azure
|
||||
securityConfig:
|
||||
clientSecret: '{your_client_secret}'
|
||||
authority: '{your_authority_url}'
|
||||
clientId: '{your_client_id}'
|
||||
scopes:
|
||||
- your_scopes
|
||||
```
|
||||
|
||||
### Custom OIDC SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### Google SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: google
|
||||
securityConfig:
|
||||
secretKey: '{path-to-json-creds}'
|
||||
```
|
||||
|
||||
### Okta SSO
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: okta
|
||||
securityConfig:
|
||||
clientId: "{CLIENT_ID - SPA APP}"
|
||||
orgURL: "{ISSUER_URL}/v1/token"
|
||||
privateKey: "{public/private keypair}"
|
||||
email: "{email}"
|
||||
scopes:
|
||||
- token
|
||||
```
|
||||
|
||||
### Amazon Cognito SSO
|
||||
|
||||
The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens)
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: auth0
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### OneLogin SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
### KeyCloak SSO
|
||||
|
||||
Which uses Custom OIDC for the ingestion
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: 'http://localhost:8585/api'
|
||||
authProvider: custom-oidc
|
||||
securityConfig:
|
||||
clientId: '{your_client_id}'
|
||||
secretKey: '{your_client_secret}'
|
||||
domain: '{your_domain}'
|
||||
```
|
||||
|
||||
</Collapse>
|
||||
|
||||
### 2. Run with the CLI
|
||||
|
||||
First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run:
|
||||
|
||||
```bash
|
||||
metadata ingest -c <path-to-yaml>
|
||||
```
|
||||
|
||||
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration,
|
||||
you will be able to extract metadata from different sources.
|
||||
@ -0,0 +1,204 @@
|
||||
---
|
||||
title: Domo-Pipeline
|
||||
slug: /connectors/pipeline/domo-pipeline
|
||||
---
|
||||
|
||||
# Domo Pipeline
|
||||
|
||||
In this section, we provide guides and references to use the Domo-Pipeline connector.
|
||||
|
||||
Configure and schedule Domo-Pipeline metadata and profiler workflows from the OpenMetadata UI:
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
|
||||
If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check
|
||||
the following docs to connect using Airflow SDK or with the CLI.
|
||||
|
||||
<TileContainer>
|
||||
<Tile
|
||||
icon="air"
|
||||
title="Ingest with Airflow"
|
||||
text="Configure the ingestion using Airflow SDK"
|
||||
link="/connectors/pipeline/domo-pipeline/airflow"
|
||||
size="half"
|
||||
/>
|
||||
<Tile
|
||||
icon="account_tree"
|
||||
title="Ingest with the CLI"
|
||||
text="Run a one-time ingestion using the metadata CLI"
|
||||
link="/connectors/pipeline/domo-pipeline/cli"
|
||||
size="half"
|
||||
/>
|
||||
</TileContainer>
|
||||
|
||||
## Requirements
|
||||
|
||||
<InlineCallout color="violet-70" icon="description" bold="OpenMetadata 0.12 or later" href="/deployment">
|
||||
To deploy OpenMetadata, check the <a href="/deployment">Deployment</a> guides.
|
||||
</InlineCallout>
|
||||
|
||||
To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with
|
||||
custom Airflow plugins to handle the workflow deployment.
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
### 1. Visit the Services Page
|
||||
|
||||
The first step is ingesting the metadata from your sources. Under
|
||||
Settings, you will find a Services link an external source system to
|
||||
OpenMetadata. Once a service is created, it can be used to configure
|
||||
metadata, usage, and profiler workflows.
|
||||
|
||||
To visit the Services page, select Services from the Settings menu.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/visit-services.png"
|
||||
alt="Visit Services Page"
|
||||
caption="Find Services under the Settings menu"
|
||||
/>
|
||||
|
||||
### 2. Create a New Service
|
||||
|
||||
Click on the Add New Service button to start the Service creation.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/create-service.png"
|
||||
alt="Create a new service"
|
||||
caption="Add a new Service from the Services page"
|
||||
/>
|
||||
|
||||
### 3. Select the Service Type
|
||||
|
||||
Select DomoPipeline as the service type and click Next.
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domopipeline/domo-pipeline-services.png"
|
||||
alt="Select Service"
|
||||
caption="Select your service from the list"
|
||||
/>
|
||||
</div>
|
||||
|
||||
### 4. Name and Describe your Service
|
||||
|
||||
Provide a name and description for your service as illustrated below.
|
||||
|
||||
#### Service Name
|
||||
|
||||
OpenMetadata uniquely identifies services by their Service Name. Provide
|
||||
a name that distinguishes your deployment from other services, including
|
||||
the other {connector} services that you might be ingesting metadata
|
||||
from.
|
||||
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domopipeline/domo-add-newservices.png"
|
||||
alt="Add New Service"
|
||||
caption="Provide a Name and description for your Service"
|
||||
/>
|
||||
</div>
|
||||
|
||||
|
||||
### 5. Configure the Service Connection
|
||||
|
||||
In this step, we will configure the connection settings required for
|
||||
this connector. Please follow the instructions below to ensure that
|
||||
you've configured the connector to read from your domopipeline service as
|
||||
desired.
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/domopipeline/domo-pipeline-connection.png"
|
||||
alt="Configure service connection"
|
||||
caption="Configure the service connection by filling the form"
|
||||
/>
|
||||
</div>
|
||||
|
||||
|
||||
Once the credentials have been added, click on `Test Connection` and Save
|
||||
the changes.
|
||||
|
||||
<div className="w-100 flex justify-center">
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/test-connection.png"
|
||||
alt="Test Connection"
|
||||
caption="Test the connection and save the Service"
|
||||
/>
|
||||
</div>
|
||||
|
||||
#### Connection Options
|
||||
|
||||
- **Client ID**: Client Id for DOMO Pipeline.
|
||||
- **Secret Token**: Secret Token to Connect to DOMO Pipeline.
|
||||
- **Access Token**: Access to Connect to DOMO Pipeline.
|
||||
- **API Host**: API Host to Connect to DOMO Pipeline.
|
||||
- **SandBox Domain**: Connect to SandBox Domain.
|
||||
|
||||
### 6. Configure Metadata Ingestion
|
||||
|
||||
In this step we will configure the metadata ingestion pipeline,
|
||||
Please follow the instructions below
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/configure-metadata-ingestion-pipeline.png"
|
||||
alt="Configure Metadata Ingestion"
|
||||
caption="Configure Metadata Ingestion Page"
|
||||
/>
|
||||
|
||||
#### Metadata Ingestion Options
|
||||
|
||||
- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
|
||||
- **Pipeline Filter Pattern (Optional)**: Use to pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion.
|
||||
- **Include**: Explicitly include pipeline by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
|
||||
- **Exclude**: Explicitly exclude pipeline by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be included.
|
||||
- **Include lineage (toggle)**: Set the Include lineage toggle to control whether or not to include lineage between pipelines and data sources as part of metadata ingestion.
|
||||
- **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug, these logs can be viewed later in Airflow.
|
||||
|
||||
### 7. Schedule the Ingestion and Deploy
|
||||
|
||||
Scheduling can be set up at an hourly, daily, or weekly cadence. The
|
||||
timezone is in UTC. Select a Start Date to schedule for ingestion. It is
|
||||
optional to add an End Date.
|
||||
|
||||
Review your configuration settings. If they match what you intended,
|
||||
click Deploy to create the service and schedule metadata ingestion.
|
||||
|
||||
If something doesn't look right, click the Back button to return to the
|
||||
appropriate step and change the settings as needed.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/schedule.png"
|
||||
alt="Schedule the Workflow"
|
||||
caption="Schedule the Ingestion Pipeline and Deploy"
|
||||
/>
|
||||
|
||||
After configuring the workflow, you can click on Deploy to create the
|
||||
pipeline.
|
||||
|
||||
### 8. View the Ingestion Pipeline
|
||||
|
||||
Once the workflow has been successfully deployed, you can view the
|
||||
Ingestion Pipeline running from the Service Page.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/view-ingestion-pipeline.png"
|
||||
alt="View Ingestion Pipeline"
|
||||
caption="View the Ingestion Pipeline from the Service Page"
|
||||
/>
|
||||
|
||||
### 9. Workflow Deployment Error
|
||||
|
||||
If there were any errors during the workflow deployment process, the
|
||||
Ingestion Pipeline Entity will still be created, but no workflow will be
|
||||
present in the Ingestion container.
|
||||
|
||||
You can then edit the Ingestion Pipeline and Deploy it again.
|
||||
|
||||
<Image
|
||||
src="/images/openmetadata/connectors/workflow-deployment-error.png"
|
||||
alt="Workflow Deployment Error"
|
||||
caption="Edit and Deploy the Ingestion Pipeline"
|
||||
/>
|
||||
|
||||
From the Connection tab, you can also Edit the Service if needed.
|
||||
@ -10,3 +10,4 @@ slug: /connectors/pipeline
|
||||
- [Glue Pipeline](/connectors/pipeline/glue-pipeline)
|
||||
- [Fivetran](/connectors/pipeline/fivetran)
|
||||
- [Dagster](/connectors/pipeline/dagster)
|
||||
- [Domo Pipeline](/connectors/pipeline/domo-pipeline)
|
||||
@ -318,6 +318,12 @@ site_menu:
|
||||
url: /connectors/database/mariadb/airflow
|
||||
- category: Connectors / Database / MariaDB / CLI
|
||||
url: /connectors/database/mariadb/cli
|
||||
- category: Connectors / Database / Domo Database
|
||||
url: /connectors/database/domo-database
|
||||
- category: Connectors / Database / Domo Database / Airflow
|
||||
url: /connectors/database/domo-database/airflow
|
||||
- category: Connectors / Database / Domo Database / CLI
|
||||
url: /connectors/database/domo-database/cli
|
||||
|
||||
- category: Connectors / Dashboard
|
||||
url: /connectors/dashboard
|
||||
@ -365,6 +371,13 @@ site_menu:
|
||||
url: /connectors/dashboard/mode/airflow
|
||||
- category: Connectors / Dashboard / Mode / CLI
|
||||
url: /connectors/dashboard/mode/cli
|
||||
- category: Connectors / Dashboard
|
||||
- category: Connectors / Dashboard / Domo Dashboard
|
||||
url: /connectors/dashboard/domo-dashboard
|
||||
- category: Connectors / Dashboard / Domo Dashboard / Airflow
|
||||
url: /connectors/dashboard/domo-dashboard/airflow
|
||||
- category: Connectors / Dashboard / Domo Dashboard / CLI
|
||||
url: /connectors/dashboard/domo-dashboard/cli
|
||||
|
||||
- category: Connectors / Messaging
|
||||
url: /connectors/messaging
|
||||
@ -415,6 +428,12 @@ site_menu:
|
||||
url: /connectors/pipeline/dagster/airflow
|
||||
- category: Connectors / Pipeline / Dagster / CLI
|
||||
url: /connectors/pipeline/dagster/cli
|
||||
- category: Connectors / Pipeline / Domo Pipeline
|
||||
url: /connectors/pipeline/domo-pipeline
|
||||
- category: Connectors / Pipeline / Domo Pipeline / Airflow
|
||||
url: /connectors/pipeline/domo-pipeline/airflow
|
||||
- category: Connectors / Pipeline / Domo Pipeline / CLI
|
||||
url: /connectors/pipeline/domo-pipeline/cli
|
||||
|
||||
- category: Connectors / ML Model
|
||||
url: /connectors/ml-model
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 395 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 372 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 395 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Loading…
x
Reference in New Issue
Block a user