diff --git a/openmetadata-docs/content/connectors/dashboard/domo-dashboard/airflow.md b/openmetadata-docs/content/connectors/dashboard/domo-dashboard/airflow.md new file mode 100644 index 00000000000..9cf8d23b60b --- /dev/null +++ b/openmetadata-docs/content/connectors/dashboard/domo-dashboard/airflow.md @@ -0,0 +1,320 @@ +--- +title: Run Domo Dashboard Connector using Airflow SDK +slug: /connectors/dashboard/domo-dashboard/airflow +--- + +# Run Domo Dashboard using the Airflow SDK + +In this section, we provide guides and references to use the Domo Dashboard connector. + +Configure and schedule Domo Dashboard metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +### Python Requirements + +To run the Domo-Dashboard ingestion, you will need to install: + +```bash +pip3 install "openmetadata-ingestion[domo]" +``` + +## Metadata Ingestion + +All connectors are defined as JSON Schemas. +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/lookerConnection.json) +you can find the structure to create a connection to Domo Dashboard. + +In order to create and run a Metadata Ingestion workflow, we will follow +the steps to create a YAML configuration able to connect to the source, +process the Entities if needed, and reach the OpenMetadata server. + +The workflow is modeled around the following +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) + +### 1. Define the YAML Config + +This is a sample config for Domo-Dashboard: + +```yaml +source: + type: domodashboard + serviceName: local_domodashboard + serviceConnection: + config: + type: DomoDashboard + clientId: clientid + secretToken: secret-token + accessToken: access-token + apiHost: api.domo.com + sandboxDomain: https://.domo.com + sourceConfig: + dashboardFilterPattern: {} + chartFilterPattern: {} + config: + type: DashboardMetadata + # dbServiceNames: + # - service1 + # - service2 + # dashboardFilterPattern: + # includes: + # - dashboard1 + # - dashboard2 + # excludes: + # - dashboard3 + # - dashboard4 + # chartFilterPattern: + # includes: + # - chart1 + # - chart2 + # excludes: + # - chart3 + # - chart4 +sink: + type: metadata-rest + config: {} +workflowConfig: + # loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR + openMetadataServerConfig: + hostPort: + authProvider: +``` + +#### Source Configuration - Service Connection + +- **Client ID**: Client ID to Connect to DOMO Dashboard. +- **Secret Token**: Secret Token to Connect DOMO Dashboard. +- **Access Token**: Access to Connect to DOMO Dashboard. +- **API Host**: API Host to Connect to DOMO Dashboard instance. +- **SandBox Domain**: Connect to SandBox Domain. + + +#### Source Configuration - Source Config +The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json): + +- `dbServiceName`: Database Service Name for the creation of lineage, if the source supports it. +- `dashboardFilterPattern` and `chartFilterPattern`: Note that the `dashboardFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g., + +```yaml +dashboardFilterPattern: + includes: + - users + - type_test +``` + +#### Sink Configuration + +To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. + +#### Workflow Configuration + +The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. + +For a simple, local installation using our docker containers, this looks like: + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client). +You can find the different implementation of the ingestion below. + + + +### Openmetadata JWT Auth + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +### Auth0 SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Azure SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: azure + securityConfig: + clientSecret: '{your_client_secret}' + authority: '{your_authority_url}' + clientId: '{your_client_id}' + scopes: + - your_scopes +``` + +### Custom OIDC SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Google SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: google + securityConfig: + secretKey: '{path-to-json-creds}' +``` + +### Okta SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: http://localhost:8585/api + authProvider: okta + securityConfig: + clientId: "{CLIENT_ID - SPA APP}" + orgURL: "{ISSUER_URL}/v1/token" + privateKey: "{public/private keypair}" + email: "{email}" + scopes: + - token +``` + +### Amazon Cognito SSO + +The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens) + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### OneLogin SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### KeyCloak SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + + + +## 2. Prepare the Ingestion DAG + +Create a Python file in your Airflow DAGs directory with the following contents: + +```python +import pathlib +import yaml +from datetime import timedelta +from airflow import DAG + +try: + from airflow.operators.python import PythonOperator +except ModuleNotFoundError: + from airflow.operators.python_operator import PythonOperator + +from metadata.config.common import load_config_file +from metadata.ingestion.api.workflow import Workflow +from airflow.utils.dates import days_ago + +default_args = { + "owner": "user_name", + "email": ["username@org.com"], + "email_on_failure": False, + "retries": 3, + "retry_delay": timedelta(minutes=5), + "execution_timeout": timedelta(minutes=60) +} + +config = """ + +""" + +def metadata_ingestion_workflow(): + workflow_config = yaml.safe_load(config) + workflow = Workflow.create(workflow_config) + workflow.execute() + workflow.raise_from_status() + workflow.print_status() + workflow.stop() + +with DAG( + "sample_data", + default_args=default_args, + description="An example DAG which runs a OpenMetadata ingestion workflow", + start_date=days_ago(1), + is_paused_upon_creation=False, + schedule_interval='*/5 * * * *', + catchup=False, +) as dag: + ingest_task = PythonOperator( + task_id="ingest_using_recipe", + python_callable=metadata_ingestion_workflow, + ) +``` + +Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will +be able to extract metadata from different sources. diff --git a/openmetadata-docs/content/connectors/dashboard/domo-dashboard/cli.md b/openmetadata-docs/content/connectors/dashboard/domo-dashboard/cli.md new file mode 100644 index 00000000000..af6807fbfb3 --- /dev/null +++ b/openmetadata-docs/content/connectors/dashboard/domo-dashboard/cli.md @@ -0,0 +1,275 @@ +--- +title: Run DomoDashboard Connector using the CLI +slug: /connectors/dashboard/domo-dashboard/cli +--- + +# Run Domo Dashboard using the metadata CLI + +In this section, we provide guides and references to use the DomoDashboard connector. + +Configure and schedule DomoDashboard metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +### Python Requirements + +To run the DomoDashboard ingestion, you will need to install: + +```bash +pip3 install "openmetadata-ingestion[domo]" +``` + +## Metadata Ingestion + +All connectors are defined as JSON Schemas. +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/lookerConnection.json) +you can find the structure to create a connection to DomoDashboard. + +In order to create and run a Metadata Ingestion workflow, we will follow +the steps to create a YAML configuration able to connect to the source, +process the Entities if needed, and reach the OpenMetadata server. + +The workflow is modeled around the following +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) + +### 1. Define the YAML Config + +This is a sample config for DomoDashboard: + +```yaml +source: + type: domodashboard + serviceName: local_domodashboard + serviceConnection: + config: + type: DomoDashboard + clientId: clientid + secretToken: secret-token + accessToken: access-token + apiHost: api.domo.com + sandboxDomain: https://.domo.com + sourceConfig: + config: + dashboardFilterPattern: {} + chartFilterPattern: {} + type: DashboardMetadata + # dbServiceNames: + # - service1 + # - service2 + # dashboardFilterPattern: + # includes: + # - dashboard1 + # - dashboard2 + # excludes: + # - dashboard3 + # - dashboard4 + # chartFilterPattern: + # includes: + # - chart1 + # - chart2 + # excludes: + # - chart3 + # - chart4 +sink: + type: metadata-rest + config: {} +workflowConfig: + # loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR + openMetadataServerConfig: + hostPort: + authProvider: +``` + +#### Source Configuration - Service Connection + +- **Client ID**: Client ID to Connect to DOMO Dashboard. +- **Secret Token**: Secret Token to Connect DOMO Dashboard. +- **Access Token**: Access to Connect to DOMO Dashboard. +- **API Host**: API Host to Connect to DOMO Dashboard instance. +- **SandBox Domain**: Connect to SandBox Domain. + + +#### Source Configuration - Source Config + +The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json): + +- `dbServiceName`: Database Service Name for the creation of lineage, if the source supports it. +- `dashboardFilterPattern` and `chartFilterPattern`: Note that the `dashboardFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g. + +```yaml +dashboardFilterPattern: + includes: + - users + - type_test +``` + +#### Sink Configuration + +To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. + +#### Workflow Configuration + +The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. + +For a simple, local installation using our docker containers, this looks like: + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client). +You can find the different implementation of the ingestion below. + + + +### Openmetadata JWT Auth + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +### Auth0 SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Azure SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: azure + securityConfig: + clientSecret: '{your_client_secret}' + authority: '{your_authority_url}' + clientId: '{your_client_id}' + scopes: + - your_scopes +``` + +### Custom OIDC SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Google SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: google + securityConfig: + secretKey: '{path-to-json-creds}' +``` + +### Okta SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: http://localhost:8585/api + authProvider: okta + securityConfig: + clientId: "{CLIENT_ID - SPA APP}" + orgURL: "{ISSUER_URL}/v1/token" + privateKey: "{public/private keypair}" + email: "{email}" + scopes: + - token +``` + +### Amazon Cognito SSO + +The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens) + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### OneLogin SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### KeyCloak SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + + + + +### 2. Run with the CLI + +First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: + +```bash +metadata ingest -c +``` + +Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, +you will be able to extract metadata from different sources. diff --git a/openmetadata-docs/content/connectors/dashboard/domo-dashboard/index.md b/openmetadata-docs/content/connectors/dashboard/domo-dashboard/index.md new file mode 100644 index 00000000000..d9ba89d9aea --- /dev/null +++ b/openmetadata-docs/content/connectors/dashboard/domo-dashboard/index.md @@ -0,0 +1,207 @@ +--- +title: Domo Dashboard +slug: /connectors/dashboard/domo-dashboard +--- + +# Domo Dashboard + +In this section, we provide guides and references to use the DomoDashboard connector. + +Configure and schedule DomoDashboard metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) + +If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check +the following docs to connect using Airflow SDK or with the CLI. + + + + + + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +## Metadata Ingestion + +### 1. Visit the Services Page + +The first step is ingesting the metadata from your sources. Under +Settings, you will find a Services link an external source system to +OpenMetadata. Once a service is created, it can be used to configure +metadata, usage, and profiler workflows. + +To visit the Services page, select Services from the Settings menu. + +Visit Services Page + +### 2. Create a New Service + +Click on the Add New Service button to start the Service creation. + +Create a new service + +### 3. Select the Service Type + +Select Domo-Dashboard as the service type and click Next. + +
+Select Service +
+ +### 4. Name and Describe your Service + +Provide a name and description for your service as illustrated below. + +#### Service Name + +OpenMetadata uniquely identifies services by their Service Name. Provide +a name that distinguishes your deployment from other services, including +the other {connector} services that you might be ingesting metadata +from. + + +
+Add New Service +
+ + +### 5. Configure the Service Connection + +In this step, we will configure the connection settings required for +this connector. Please follow the instructions below to ensure that +you've configured the connector to read from your DomoDashboard service as +desired. + +
+Configure service connection +
+ + +Once the credentials have been added, click on `Test Connection` and Save +the changes. + +
+Test Connection +
+ +#### Connection Options + +- **Client ID**: Client ID to Connect to DOMO Dashboard. +- **Secret Token**: Secret Token to Connect DOMO Dashboard. +- **Access Token**: Access to Connect to DOMO Dashboard. +- **API Host**: API Host to Connect to DOMO Dashboard instance. +- **SandBox Domain**: Connect to SandBox Domain. + +### 6. Configure Metadata Ingestion + +In this step we will configure the metadata ingestion pipeline, +Please follow the instructions below + +Configure Metadata Ingestion + +#### Metadata Ingestion Options + +- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name. +- **Dashboard Filter Pattern (Optional)**: Use to dashboard filter patterns to control whether or not to include dashboard as part of metadata ingestion. + - **Include**: Explicitly include dashboards by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be excluded. + - **Exclude**: Explicitly exclude dashboards by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be included. +- **Chart Pattern (Optional)**: Use to chart filter patterns to control whether or not to include charts as part of metadata ingestion. + - **Include**: Explicitly include charts by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all charts with names matching one or more of the supplied regular expressions. All other charts will be excluded. + - **Exclude**: Explicitly exclude charts by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all charts with names matching one or more of the supplied regular expressions. All other charts will be included. +- **Database Service Name (Optional)**: Enter the name of Database Service which is already ingested in OpenMetadata to create lineage between dashboards and database tables. +- **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug, these logs can be viewed later in Airflow. + +### 7. Schedule the Ingestion and Deploy + +Scheduling can be set up at an hourly, daily, or weekly cadence. The +timezone is in UTC. Select a Start Date to schedule for ingestion. It is +optional to add an End Date. + +Review your configuration settings. If they match what you intended, +click Deploy to create the service and schedule metadata ingestion. + +If something doesn't look right, click the Back button to return to the +appropriate step and change the settings as needed. + +Schedule the Workflow + +After configuring the workflow, you can click on Deploy to create the +pipeline. + +### 8. View the Ingestion Pipeline + +Once the workflow has been successfully deployed, you can view the +Ingestion Pipeline running from the Service Page. + +View Ingestion Pipeline + +### 9. Workflow Deployment Error + +If there were any errors during the workflow deployment process, the +Ingestion Pipeline Entity will still be created, but no workflow will be +present in the Ingestion container. + +You can then edit the Ingestion Pipeline and Deploy it again. + +Workflow Deployment Error + +From the Connection tab, you can also Edit the Service if needed. diff --git a/openmetadata-docs/content/connectors/dashboard/index.md b/openmetadata-docs/content/connectors/dashboard/index.md index 3b3a58cdf22..f9c369c8380 100644 --- a/openmetadata-docs/content/connectors/dashboard/index.md +++ b/openmetadata-docs/content/connectors/dashboard/index.md @@ -12,3 +12,4 @@ slug: /connectors/dashboard - [Redash](/connectors/dashboard/redash) - [Superset](/connectors/dashboard/superset) - [Tableau](/connectors/dashboard/tableau) +- [DomoDashboard](/connectors/dashboard/domo-dashboard) \ No newline at end of file diff --git a/openmetadata-docs/content/connectors/database/domo-database/airflow.md b/openmetadata-docs/content/connectors/database/domo-database/airflow.md new file mode 100644 index 00000000000..a96bd81d005 --- /dev/null +++ b/openmetadata-docs/content/connectors/database/domo-database/airflow.md @@ -0,0 +1,406 @@ +--- +title: Run DomoDatabase Connector using Airflow SDK +slug: /connectors/database/domo-database/airflow +--- + +# Run Domo Database using Airflow SDK + +In this section, we provide guides and references to use the Domo Database connector + +Configure and schedule DomoDatabase metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) +- [Data Profiler](#data-profiler) +- [DBT Integration](#dbt-integration) + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +### Python Requirements + +To run the DomoDatabase ingestion, you will need to install: + +```bash +pip3 install "openmetadata-ingestion[domo]" +``` +## Metadata Ingestion + +All connectors are defined as JSON Schemas. +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/verticaConnection.json) +you can find the structure to create a connection to Vertica. + +In order to create and run a Metadata Ingestion workflow, we will follow +the steps to create a YAML configuration able to connect to the source, +process the Entities if needed, and reach the OpenMetadata server. + +The workflow is modeled around the following +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) + +### 1. Define the YAML Config + +This is a sample config for DomoDatabase: + +```yaml +source: + type: domodatabase + serviceName: local_DomoDatabase + serviceConnection: + config: + type: DomoDashboard + clientId: client-id + secretToken: secret-token + accessToken: access-token + apiHost: api.domo.com + sandboxDomain: https://.domo.com + + # database: database + sourceConfig: + config: + markDeletedTables: true + includeTables: true + includeViews: true +sink: + type: metadata-rest + config: {} +workflowConfig: + # loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR + openMetadataServerConfig: + hostPort: + authProvider: 2. Configure service settings +``` + +#### Source Configuration - Service Connection + +- **Client ID**: Client ID to Connect to DOMODatabase. +- **Secret Token**: Secret Token to Connect DOMODatabase. +- **Access Token**: Access to Connect to DOMODatabase. +- **API Host**: API Host to Connect to DOMODatabase instance. +- **SandBox Domain**: Connect to SandBox Domain. + + +#### Source Configuration - Source Config + +The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json): + +- `markDeletedTables`: To flag tables as soft-deleted if they are not present anymore in the source system. +- `includeTables`: true or false, to ingest table data. Default is true. +- `includeViews`: true or false, to ingest views definitions. +- `databaseFilterPattern`, `schemaFilterPattern`, `tableFilternPattern`: Note that the they support regex as include or exclude. E.g., + +```yaml +tableFilterPattern: + includes: + - users + - type_test +``` + +#### Sink Configuration + +To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. + +#### Workflow Configuration + +The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. + +For a simple, local installation using our docker containers, this looks like: + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client). +You can find the different implementation of the ingestion below. + + + +### Openmetadata JWT Auth + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` +### Auth0 SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Azure SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: azure + securityConfig: + clientSecret: '{your_client_secret}' + authority: '{your_authority_url}' + clientId: '{your_client_id}' + scopes: + - your_scopes +``` + +### Custom OIDC SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` +### Google SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: google + securityConfig: + secretKey: '{path-to-json-creds}' +``` + +### Okta SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: http://localhost:8585/api + authProvider: okta + securityConfig: + clientId: "{CLIENT_ID - SPA APP}" + orgURL: "{ISSUER_URL}/v1/token" + privateKey: "{public/private keypair}" + email: "{email}" + scopes: + - token +``` + +### Amazon Cognito SSO + +The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens) + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### OneLogin SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### KeyCloak SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + + + +### 2. Prepare the Ingestion DAG + +Create a Python file in your Airflow DAGs directory with the following contents: + +```python +import pathlib +import yaml +from datetime import timedelta +from airflow import DAG + +try: + from airflow.operators.python import PythonOperator +except ModuleNotFoundError: + from airflow.operators.python_operator import PythonOperator + +from metadata.config.common import load_config_file +from metadata.ingestion.api.workflow import Workflow +from airflow.utils.dates import days_ago + +default_args = { + "owner": "user_name", + "email": ["username@org.com"], + "email_on_failure": False, + "retries": 3, + "retry_delay": timedelta(minutes=5), + "execution_timeout": timedelta(minutes=60) +} + +config = """ + +""" + +def metadata_ingestion_workflow(): + workflow_config = yaml.safe_load(config) + workflow = Workflow.create(workflow_config) + workflow.execute() + workflow.raise_from_status() + workflow.print_status() + workflow.stop() + +with DAG( + "sample_data", + default_args=default_args, + description="An example DAG which runs a OpenMetadata ingestion workflow", + start_date=days_ago(1), + is_paused_upon_creation=False, + schedule_interval='*/5 * * * *', + catchup=False, +) as dag: + ingest_task = PythonOperator( + task_id="ingest_using_recipe", + python_callable=metadata_ingestion_workflow, + ) +``` +Note that from connector to connector, this recipe will always be the same. +By updating the YAML configuration, you will be able to extract metadata from different sources. + +### 1. Define the YAML Config + +This is a sample config for the profiler: + +```yaml +source: + type: domodatabase + serviceName: + serviceConnection: + config: + type: DomoDatabase + clientId: clientid + secretToken: secret Token + accessToken: access Token + apiHost: api.domo.com + sandboxDomain: https://.domo.com + sourceConfig: + config: + type: DatabaseMetadata + +sink: + type: metadata-rest + config: {} +workflowConfig: + # loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR + openMetadataServerConfig: + hostPort: http://localhost:8585/api + authProvider: + ``` + +#### Source Configuration + +- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/athenaConnection.json). +- The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json). + +Note that the filter patterns support regex as includes or excludes. E.g., + +```yaml +tableFilterPattern: + includes: + - *users$ +``` +#### Workflow Configuration + +The same as the metadata ingestion. + +### 2. Prepare the Profiler DAG + +Here, we follow a similar approach as with the metadata and usage pipelines, although we will use a different Workflow class: + +```python +import yaml +from datetime import timedelta + +from airflow import DAG + +try: + from airflow.operators.python import PythonOperator +except ModuleNotFoundError: + from airflow.operators.python_operator import PythonOperator + +from airflow.utils.dates import days_ago + +from metadata.orm_profiler.api.workflow import ProfilerWorkflow + + +default_args = { + "owner": "user_name", + "email_on_failure": False, + "retries": 3, + "retry_delay": timedelta(seconds=10), + "execution_timeout": timedelta(minutes=60), +} + +config = """ + +""" + +def metadata_ingestion_workflow(): + workflow_config = yaml.safe_load(config) + workflow = ProfilerWorkflow.create(workflow_config) + workflow.execute() + workflow.raise_from_status() + workflow.print_status() + workflow.stop() + +with DAG( + "profiler_example", + default_args=default_args, + description="An example DAG which runs a OpenMetadata ingestion workflow", + start_date=days_ago(1), + is_paused_upon_creation=False, + catchup=False, +) as dag: + ingest_task = PythonOperator( + task_id="profile_and_test_using_recipe", + python_callable=metadata_ingestion_workflow, + ) +``` diff --git a/openmetadata-docs/content/connectors/database/domo-database/cli.md b/openmetadata-docs/content/connectors/database/domo-database/cli.md new file mode 100644 index 00000000000..d064ada7337 --- /dev/null +++ b/openmetadata-docs/content/connectors/database/domo-database/cli.md @@ -0,0 +1,345 @@ +--- +title: Run DomoDatabase Connector using the CLI +slug: /connectors/database/domo-database/cli +--- + +# Run Domo Database using the metadata CLI + +In this section, we provide guides and references to use the Domo Database connector. + +Configure and schedule DomoDatabase metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) +- [Data Profiler](#data-profiler) +- [DBT Integration](#dbt-integration) + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +### Python Requirements + +To run the DomoDatabase ingestion, you will need to install: + +```bash +pip3 install "openmetadata-ingestion[domo]" +``` + +## Metadata Ingestion + +All connectors are defined as JSON Schemas. +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/athenaConnection.json) +you can find the structure to create a connection to DomoDatbase. + +In order to create and run a Metadata Ingestion workflow, we will follow +the steps to create a YAML configuration able to connect to the source, +process the Entities if needed, and reach the OpenMetadata server. + +The workflow is modeled around the following +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) + +### 1. Define the YAML Config + +This is a sample config for DomoDatabase: + +```yaml +source: + type: domodatabase + serviceName: local_domodatabase + serviceConnection: + config: + type: DomoDatabase + clientId: clientid + secretToken: secret-token + accessToken: access-token + apiHost: api.domo.com + sandboxDomain: https://.domo.com + sourceConfig: + config: + type: DatabaseMetadata +sink: + type: metadata-rest + config: {} +workflowConfig: + # loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR + openMetadataServerConfig: + hostPort: + authProvider: + +``` + +#### Source Configuration - Service Connection + +- **Client ID**: Client ID to Connect to DOMO Database. +- **Secret Token**: Secret Token to Connect DOMO Database. +- **Access Token**: Access to Connect to DOMO Database. +- **API Host**: API Host to Connect to DOMO Database instance. +- **SandBox Domain**: Connect to SandBox Domain. + +#### Source Configuration - Source Config + +The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json): + +- `markDeletedTables`: To flag tables as soft-deleted if they are not present anymore in the source system. +- `includeTables`: true or false, to ingest table data. Default is true. +- `includeViews`: true or false, to ingest views definitions. +- `databaseFilterPattern`, `schemaFilterPattern`, `tableFilternPattern`: Note that the they support regex as include or exclude. E.g., + +```yaml +tableFilterPattern: + includes: + - users + - type_test +``` + +#### Sink Configuration + +To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. + +#### Workflow Configuration + +The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. + +For a simple, local installation using our docker containers, this looks like: + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client). +You can find the different implementation of the ingestion below. + + + +### Openmetadata JWT Auth + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +### Auth0 SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Azure SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: azure + securityConfig: + clientSecret: '{your_client_secret}' + authority: '{your_authority_url}' + clientId: '{your_client_id}' + scopes: + - your_scopes +``` + +### Custom OIDC SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' + ``` + + ### Google SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: google + securityConfig: + secretKey: '{path-to-json-creds}' +``` + +### Okta SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: http://localhost:8585/api + authProvider: okta + securityConfig: + clientId: "{CLIENT_ID - SPA APP}" + orgURL: "{ISSUER_URL}/v1/token" + privateKey: "{public/private keypair}" + email: "{email}" + scopes: + - token +``` + +### Amazon Cognito SSO + +The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens) + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### OneLogin SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### KeyCloak SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + + + +### 2. Run with the CLI + +First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: + +```bash +metadata ingest -c +``` + +Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, +you will be able to extract metadata from different sources. + + +### 1. Define the YAML Config + +This is a sample config for the profiler: + +```yaml +source: + type: domodatabase + serviceName: + serviceConnection: + config: + type: DomoDatabase + type: DomoDashboard + clientId: client-id + secretToken: secret-token + accessToken: access-token + apiHost: api.domo.com + sandboxDomain: https://.domo.com + # endPointURL: https://athena.us-east-2.amazonaws.com/ + # awsSessionToken: TOKEN + s3StagingDir: s3 directory for datasource + workgroup: workgroup name + sourceConfig: + config: + type: Profiler + # generateSampleData: true + # profileSample: 85 + # threadCount: 5 (default) + # databaseFilterPattern: + # includes: + # - database1 + # - database2 + # excludes: + # - database3 + # - database4 + # schemaFilterPattern: + # includes: + # - schema1 + # - schema2 + # excludes: + # - schema3 + # - schema4 + # tableFilterPattern: + # includes: + # - table1 + # - table2 + # excludes: + # - table3 + # - table4 + - ... +sink: + type: metadata-rest + config: {} +workflowConfig: + # loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR + openMetadataServerConfig: + hostPort: + authProvider: +``` + +#### Source Configuration + +- You can find all the definitions and types for the `serviceConnection` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/athenaConnection.json). +- The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json). + +Note that the filter patterns support regex as includes or excludes. E.g., + +```yaml +tableFilterPattern: + includes: + - *users$ +``` +#### Workflow Configuration + +The same as the metadata ingestion. + +### 2. Run with the CLI + +After saving the YAML config, we will run the command the same way we did for the metadata ingestion: + +```bash +metadata profile -c +``` + +Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. diff --git a/openmetadata-docs/content/connectors/database/domo-database/index.md b/openmetadata-docs/content/connectors/database/domo-database/index.md new file mode 100644 index 00000000000..9f3ff74e737 --- /dev/null +++ b/openmetadata-docs/content/connectors/database/domo-database/index.md @@ -0,0 +1,217 @@ +--- +title: DomoDatabase +slug: /connectors/database/domo-database +--- + +# DomoDatabase + +In this section, we provide guides and references to use the DomoDatabase connector. + +Configure and schedule DomoDatabase metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) +- [Data Profiler](#data-profiler) +- [DBT Integration](#dbt-integration) + +If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check +the following docs to connect using Airflow SDK or with the CLI. + + + + + + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +## Metadata Ingestion + +### 1. Visit the Services Page + +The first step is ingesting the metadata from your sources. Under +Settings, you will find a Services link an external source system to +OpenMetadata. Once a service is created, it can be used to configure +metadata, usage, and profiler workflows. + +To visit the Services page, select Services from the Settings menu. + + + +### 2. Create a New Service + +Click on the Add New Service button to start the Service creation. + + + +### 3. Select the Service Type + +Select Domo-Database as the service type and click Next. + +
+Select Service +
+ +### 4. Name and Describe your Service + +Provide a name and description for your service as illustrated below. + +#### Service Name + +OpenMetadata uniquely identifies services by their Service Name. Provide +a name that distinguishes your deployment from other services, including +the other {connector} services that you might be ingesting metadata +from. + + +
+Add New Service +
+ + +### 5. Configure the Service Connection + +In this step, we will configure the connection settings required for +this connector. Please follow the instructions below to ensure that +you've configured the connector to read from your domodatabase service as +desired. + +
+Configure service connection +
+ + +Once the credentials have been added, click on `Test Connection` and Save +the changes. + +
+Test Connection +
+ +#### Connection Options + +- **Client ID**: Client ID for DOMO Database. +- **Secret Token**: Secret Token to Connect DOMO Database. +- **Access Token**: Access to Connect to DOMO Database. +- **Api Host**: API Host to Connect to DOMO Database instance. +- **SandBox Domain**: Connect to SandBox Domain. + +### 6. Configure Metadata Ingestion + +In this step we will configure the metadata ingestion pipeline, +Please follow the instructions below + + + +#### Metadata Ingestion Options + +- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name. +- **Database Filter Pattern (Optional)**: Use to database filter patterns to control whether or not to include database as part of metadata ingestion. + - **Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded. + - **Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included. +- **Schema Filter Pattern (Optional)**: Use to schema filter patterns to control whether or not to include schemas as part of metadata ingestion. + - **Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded. + - **Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included. +- **Table Filter Pattern (Optional)**: Use to table filter patterns to control whether or not to include tables as part of metadata ingestion. + - **Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded. + - **Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included. +- **Include views (toggle)**: Set the Include views toggle to control whether or not to include views as part of metadata ingestion. +- **Include tags (toggle)**: Set the Include tags toggle to control whether or not to include tags as part of metadata ingestion. +- **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug, these logs can be viewed later in Airflow. +- **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system. +- **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema + +### 7. Schedule the Ingestion and Deploy + +Scheduling can be set up at an hourly, daily, or weekly cadence. The +timezone is in UTC. Select a Start Date to schedule for ingestion. It is +optional to add an End Date. + +Review your configuration settings. If they match what you intended, +click Deploy to create the service and schedule metadata ingestion. + +If something doesn't look right, click the Back button to return to the +appropriate step and change the settings as needed. + + + +After configuring the workflow, you can click on Deploy to create the +pipeline. + +### 8. View the Ingestion Pipeline + +Once the workflow has been successfully deployed, you can view the +Ingestion Pipeline running from the Service Page. + + + +### 9. Workflow Deployment Error + +If there were any errors during the workflow deployment process, the +Ingestion Pipeline Entity will still be created, but no workflow will be +present in the Ingestion container. + +You can then edit the Ingestion Pipeline and Deploy it again. + + + +From the Connection tab, you can also Edit the Service if needed. + + diff --git a/openmetadata-docs/content/connectors/index.md b/openmetadata-docs/content/connectors/index.md index a3fea54f955..2f85e58bcca 100644 --- a/openmetadata-docs/content/connectors/index.md +++ b/openmetadata-docs/content/connectors/index.md @@ -54,6 +54,7 @@ OpenMetadata can extract metadata from the following list of 55 connectors: - SQL Profiles (SQL based systems) - [Trino](/connectors/database/trino) - [Vertica](/connectors/database/vertica) +- [Domo Database](/connectors/database/domo-database) ## Dashboard Services @@ -64,6 +65,7 @@ OpenMetadata can extract metadata from the following list of 55 connectors: - [Redash](/connectors/dashboard/redash) - [Superset](/connectors/dashboard/superset) - [Tableau](/connectors/dashboard/tableau) +- [Domo Dashboard](/connectors/dashboard/domo-dashboard) ## Messaging Services @@ -79,6 +81,7 @@ OpenMetadata can extract metadata from the following list of 55 connectors: - [Dagster](/connectors/pipeline/dagster) - [Fivetran](/connectors/pipeline/fivetran) - [Glue](/connectors/pipeline/glue-pipeline) +- [Domo Pipeline](/connectors/pipeline/domo-pipeline) - NiFi ## ML Model Services diff --git a/openmetadata-docs/content/connectors/pipeline/domo-pipeline/airflow.md b/openmetadata-docs/content/connectors/pipeline/domo-pipeline/airflow.md new file mode 100644 index 00000000000..e9f60e077ba --- /dev/null +++ b/openmetadata-docs/content/connectors/pipeline/domo-pipeline/airflow.md @@ -0,0 +1,305 @@ +--- +title: Run Domo Pipeline Connector using Airflow SDK +slug: /connectors/pipeline/domo-pipeline/airflow +--- + +# Run Domo Pipeline using the Airflow SDK + +In this section, we provide guides and references to use the Domo-Pipeline connector. + +Configure and schedule Domo-Pipeline metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +### Python Requirements + +To run the domopipeline ingestion, you will need to install: + +```bash +pip3 install "openmetadata-ingestion[domo]" +``` + +## Metadata Ingestion + +All connectors are defined as JSON Schemas. +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/airbyteConnection.json) +you can find the structure to create a connection to Airbyte. + +In order to create and run a Metadata Ingestion workflow, we will follow +the steps to create a YAML configuration able to connect to the source, +process the Entities if needed, and reach the OpenMetadata server. + +The workflow is modeled around the following +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) + +### 1. Define the YAML Config + +This is a sample config for Domo-Pipeline: + +```yaml +source: + type: domopipeline + serviceName: domo-pipeline_source + serviceConnection: + config: + type: DomoPipeline + clientID: clientid + secretToken: secret-token + accessToken: access-token + apiHost: api.domo.com + sandboxDomain: https://.domo.com + sourceConfig: + config: + pipelineFilterPattern: {} + type: PipelineMetadata +sink: + type: metadata-rest + config: {} +workflowConfig: + # loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR + openMetadataServerConfig: + hostPort: http://localhost:8585/api + authProvider: + securityconfig: + jwtToken: + +``` + +#### Source Configuration - Service Connection + +- **Client ID**: Client ID to Connect to DOMO Pipeline. +- **Secret Token**: Secret Token to Connect DOMO Pipeline. +- **Access Token**: Access to Connect to DOMO Pipeline. +- **API Host**: API Host to Connect to DOMO Pipeline instance. +- **SandBox Domain**: Connect to SandBox Domain. + +#### Source Configuration - Source Config + +The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json): + +- `dbServiceName`: Database Service Name for the creation of lineage, if the source supports it. +- `pipelineFilterPattern` and `chartFilterPattern`: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g., + +```yaml +pipelineFilterPattern: + includes: + - users + - type_test +``` + +#### Sink Configuration + +To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. + +#### Workflow Configuration + +The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. + +For a simple, local installation using our docker containers, this looks like: + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client). +You can find the different implementation of the ingestion below. + + + +### Openmetadata JWT Auth + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +### Auth0 SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Azure SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: azure + securityConfig: + clientSecret: '{your_client_secret}' + authority: '{your_authority_url}' + clientId: '{your_client_id}' + scopes: + - your_scopes +``` + +### Custom OIDC SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Google SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: google + securityConfig: + secretKey: '{path-to-json-creds}' +``` + +### Okta SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: http://localhost:8585/api + authProvider: okta + securityConfig: + clientId: "{CLIENT_ID - SPA APP}" + orgURL: "{ISSUER_URL}/v1/token" + privateKey: "{public/private keypair}" + email: "{email}" + scopes: + - token +``` + +### Amazon Cognito SSO + +The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens) + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### OneLogin SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### KeyCloak SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + + + +## 2. Prepare the Ingestion DAG + +Create a Python file in your Airflow DAGs directory with the following contents: + +```python +import pathlib +import yaml +from datetime import timedelta +from airflow import DAG + +try: + from airflow.operators.python import PythonOperator +except ModuleNotFoundError: + from airflow.operators.python_operator import PythonOperator + +from metadata.config.common import load_config_file +from metadata.ingestion.api.workflow import Workflow +from airflow.utils.dates import days_ago + +default_args = { + "owner": "user_name", + "email": ["username@org.com"], + "email_on_failure": False, + "retries": 3, + "retry_delay": timedelta(minutes=5), + "execution_timeout": timedelta(minutes=60) +} + +config = """ + +""" + +def metadata_ingestion_workflow(): + workflow_config = yaml.safe_load(config) + workflow = Workflow.create(workflow_config) + workflow.execute() + workflow.raise_from_status() + workflow.print_status() + workflow.stop() + +with DAG( + "sample_data", + default_args=default_args, + description="An example DAG which runs a OpenMetadata ingestion workflow", + start_date=days_ago(1), + is_paused_upon_creation=False, + schedule_interval='*/5 * * * *', + catchup=False, +) as dag: + ingest_task = PythonOperator( + task_id="ingest_using_recipe", + python_callable=metadata_ingestion_workflow, + ) +``` + +Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will +be able to extract metadata from different sources. diff --git a/openmetadata-docs/content/connectors/pipeline/domo-pipeline/cli.md b/openmetadata-docs/content/connectors/pipeline/domo-pipeline/cli.md new file mode 100644 index 00000000000..51195488d3f --- /dev/null +++ b/openmetadata-docs/content/connectors/pipeline/domo-pipeline/cli.md @@ -0,0 +1,265 @@ +--- +title: Run Domo Pipeline Connector using the CLI +slug: /connectors/pipeline/domo-pipeline/cli +--- + +# Run Domo Pipeline using the Metadata CLI + +In this section, we provide guides and references to use the Domo Pipeline connector. + +Configure and schedule Domo Pipeline metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +### Python Requirements + +To run the Domo Pipeline ingestion, you will need to install: + +```bash +pip3 install "openmetadata-ingestion[domo]" +``` + +## Metadata Ingestion + +All connectors are defined as JSON Schemas. +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/airbyteConnection.json) +you can find the structure to create a connection to Airbyte. + +In order to create and run a Metadata Ingestion workflow, we will follow +the steps to create a YAML configuration able to connect to the source, +process the Entities if needed, and reach the OpenMetadata server. + +The workflow is modeled around the following +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) + +### 1. Define the YAML Config + +This is a sample config for Domo-Pipeline: + +```yaml +source: + type: domopipeline + serviceName: domopipeline_source + serviceConnection: + config: + type: DomoPipeline + clientId: clientid + secretToken: secret-token + accessToken: access-token + apiHost: api.domo.com + sandboxDomain: https://.domo.com + sourceConfig: + config: + pipelineFilterPattern: {} + type: PipelineMetadata + # pipelineFilterPattern: + # includes: + # - pipeline1 + # - pipeline2 + # excludes: + # - pipeline3 + # - pipeline4 +sink: + type: metadata-rest + config: {} +workflowConfig: + # loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR + openMetadataServerConfig: + hostPort: + authProvider: + securityconfig: + jwtToken: +``` + +#### Source Configuration - Service Connection + +- **Client ID**: Client ID to Connect to DOMO Pipeline. +- **Secret Token**: Secret Token to Connect DOMO Pipeline. +- **Access Token**: Access to Connect to DOMO Pipeline. +- **API Host**: API Host to Connect to DOMO Pipeline instance. +- **SandBox Domain**: Connect to SandBox Domain. + +#### Source Configuration - Source Config + +The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json): + +- `dbServiceName`: Database Service Name for the creation of lineage, if the source supports it. +- `pipelineFilterPattern` and `chartFilterPattern`: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. E.g., + +```yaml +pipelineFilterPattern: + includes: + - users + - type_test +``` + +#### Sink Configuration + +To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. + +#### Workflow Configuration + +The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. + +For a simple, local installation using our docker containers, this looks like: + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + + +We support different security providers. You can find their definitions [here](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/security/client). +You can find the different implementation of the ingestion below. + + + +### Openmetadata JWT Auth + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: openmetadata + securityConfig: + jwtToken: '{bot_jwt_token}' +``` + +### Auth0 SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Azure SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: azure + securityConfig: + clientSecret: '{your_client_secret}' + authority: '{your_authority_url}' + clientId: '{your_client_id}' + scopes: + - your_scopes +``` + +### Custom OIDC SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### Google SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: google + securityConfig: + secretKey: '{path-to-json-creds}' +``` + +### Okta SSO + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: http://localhost:8585/api + authProvider: okta + securityConfig: + clientId: "{CLIENT_ID - SPA APP}" + orgURL: "{ISSUER_URL}/v1/token" + privateKey: "{public/private keypair}" + email: "{email}" + scopes: + - token +``` + +### Amazon Cognito SSO + +The ingestion can be configured by [Enabling JWT Tokens](https://docs.open-metadata.org/deployment/security/enable-jwt-tokens) + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: auth0 + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### OneLogin SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + +### KeyCloak SSO + +Which uses Custom OIDC for the ingestion + +```yaml +workflowConfig: + openMetadataServerConfig: + hostPort: 'http://localhost:8585/api' + authProvider: custom-oidc + securityConfig: + clientId: '{your_client_id}' + secretKey: '{your_client_secret}' + domain: '{your_domain}' +``` + + + +### 2. Run with the CLI + +First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: + +```bash +metadata ingest -c +``` + +Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, +you will be able to extract metadata from different sources. diff --git a/openmetadata-docs/content/connectors/pipeline/domo-pipeline/index.md b/openmetadata-docs/content/connectors/pipeline/domo-pipeline/index.md new file mode 100644 index 00000000000..2721e1f4456 --- /dev/null +++ b/openmetadata-docs/content/connectors/pipeline/domo-pipeline/index.md @@ -0,0 +1,204 @@ +--- +title: Domo-Pipeline +slug: /connectors/pipeline/domo-pipeline +--- + +# Domo Pipeline + +In this section, we provide guides and references to use the Domo-Pipeline connector. + +Configure and schedule Domo-Pipeline metadata and profiler workflows from the OpenMetadata UI: +- [Requirements](#requirements) +- [Metadata Ingestion](#metadata-ingestion) + +If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check +the following docs to connect using Airflow SDK or with the CLI. + + + + + + +## Requirements + + +To deploy OpenMetadata, check the Deployment guides. + + +To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with +custom Airflow plugins to handle the workflow deployment. + +## Metadata Ingestion + +### 1. Visit the Services Page + +The first step is ingesting the metadata from your sources. Under +Settings, you will find a Services link an external source system to +OpenMetadata. Once a service is created, it can be used to configure +metadata, usage, and profiler workflows. + +To visit the Services page, select Services from the Settings menu. + + + +### 2. Create a New Service + +Click on the Add New Service button to start the Service creation. + + + +### 3. Select the Service Type + +Select DomoPipeline as the service type and click Next. + +
+Select Service +
+ +### 4. Name and Describe your Service + +Provide a name and description for your service as illustrated below. + +#### Service Name + +OpenMetadata uniquely identifies services by their Service Name. Provide +a name that distinguishes your deployment from other services, including +the other {connector} services that you might be ingesting metadata +from. + + +
+Add New Service +
+ + +### 5. Configure the Service Connection + +In this step, we will configure the connection settings required for +this connector. Please follow the instructions below to ensure that +you've configured the connector to read from your domopipeline service as +desired. + +
+Configure service connection +
+ + +Once the credentials have been added, click on `Test Connection` and Save +the changes. + +
+Test Connection +
+ +#### Connection Options + +- **Client ID**: Client Id for DOMO Pipeline. +- **Secret Token**: Secret Token to Connect to DOMO Pipeline. +- **Access Token**: Access to Connect to DOMO Pipeline. +- **API Host**: API Host to Connect to DOMO Pipeline. +- **SandBox Domain**: Connect to SandBox Domain. + +### 6. Configure Metadata Ingestion + +In this step we will configure the metadata ingestion pipeline, +Please follow the instructions below + + + +#### Metadata Ingestion Options + +- **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name. +- **Pipeline Filter Pattern (Optional)**: Use to pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion. + - **Include**: Explicitly include pipeline by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be excluded. + - **Exclude**: Explicitly exclude pipeline by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be included. +- **Include lineage (toggle)**: Set the Include lineage toggle to control whether or not to include lineage between pipelines and data sources as part of metadata ingestion. +- **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug, these logs can be viewed later in Airflow. + +### 7. Schedule the Ingestion and Deploy + +Scheduling can be set up at an hourly, daily, or weekly cadence. The +timezone is in UTC. Select a Start Date to schedule for ingestion. It is +optional to add an End Date. + +Review your configuration settings. If they match what you intended, +click Deploy to create the service and schedule metadata ingestion. + +If something doesn't look right, click the Back button to return to the +appropriate step and change the settings as needed. + + + +After configuring the workflow, you can click on Deploy to create the +pipeline. + +### 8. View the Ingestion Pipeline + +Once the workflow has been successfully deployed, you can view the +Ingestion Pipeline running from the Service Page. + + + +### 9. Workflow Deployment Error + +If there were any errors during the workflow deployment process, the +Ingestion Pipeline Entity will still be created, but no workflow will be +present in the Ingestion container. + +You can then edit the Ingestion Pipeline and Deploy it again. + + + +From the Connection tab, you can also Edit the Service if needed. diff --git a/openmetadata-docs/content/connectors/pipeline/index.md b/openmetadata-docs/content/connectors/pipeline/index.md index 85e9a2981c6..25120c9dd5f 100644 --- a/openmetadata-docs/content/connectors/pipeline/index.md +++ b/openmetadata-docs/content/connectors/pipeline/index.md @@ -10,3 +10,4 @@ slug: /connectors/pipeline - [Glue Pipeline](/connectors/pipeline/glue-pipeline) - [Fivetran](/connectors/pipeline/fivetran) - [Dagster](/connectors/pipeline/dagster) +- [Domo Pipeline](/connectors/pipeline/domo-pipeline) \ No newline at end of file diff --git a/openmetadata-docs/content/menu.md b/openmetadata-docs/content/menu.md index 85fa1cb6fdc..f91f1327d85 100644 --- a/openmetadata-docs/content/menu.md +++ b/openmetadata-docs/content/menu.md @@ -318,6 +318,12 @@ site_menu: url: /connectors/database/mariadb/airflow - category: Connectors / Database / MariaDB / CLI url: /connectors/database/mariadb/cli + - category: Connectors / Database / Domo Database + url: /connectors/database/domo-database + - category: Connectors / Database / Domo Database / Airflow + url: /connectors/database/domo-database/airflow + - category: Connectors / Database / Domo Database / CLI + url: /connectors/database/domo-database/cli - category: Connectors / Dashboard url: /connectors/dashboard @@ -365,6 +371,13 @@ site_menu: url: /connectors/dashboard/mode/airflow - category: Connectors / Dashboard / Mode / CLI url: /connectors/dashboard/mode/cli + - category: Connectors / Dashboard + - category: Connectors / Dashboard / Domo Dashboard + url: /connectors/dashboard/domo-dashboard + - category: Connectors / Dashboard / Domo Dashboard / Airflow + url: /connectors/dashboard/domo-dashboard/airflow + - category: Connectors / Dashboard / Domo Dashboard / CLI + url: /connectors/dashboard/domo-dashboard/cli - category: Connectors / Messaging url: /connectors/messaging @@ -415,6 +428,12 @@ site_menu: url: /connectors/pipeline/dagster/airflow - category: Connectors / Pipeline / Dagster / CLI url: /connectors/pipeline/dagster/cli + - category: Connectors / Pipeline / Domo Pipeline + url: /connectors/pipeline/domo-pipeline + - category: Connectors / Pipeline / Domo Pipeline / Airflow + url: /connectors/pipeline/domo-pipeline/airflow + - category: Connectors / Pipeline / Domo Pipeline / CLI + url: /connectors/pipeline/domo-pipeline/cli - category: Connectors / ML Model url: /connectors/ml-model diff --git a/openmetadata-docs/images/openmetadata/connectors/domodashboard/domo-dashboard-add-new-service.png b/openmetadata-docs/images/openmetadata/connectors/domodashboard/domo-dashboard-add-new-service.png new file mode 100644 index 00000000000..88c44a7c9d6 Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domodashboard/domo-dashboard-add-new-service.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/domodashboard/domo-dashboard-service.png b/openmetadata-docs/images/openmetadata/connectors/domodashboard/domo-dashboard-service.png new file mode 100644 index 00000000000..b9c586e9c3d Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domodashboard/domo-dashboard-service.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/domodashboard/image1.png b/openmetadata-docs/images/openmetadata/connectors/domodashboard/image1.png new file mode 100644 index 00000000000..299052f27d9 Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domodashboard/image1.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/domodatabase/domo-ingestion-pipeline.png b/openmetadata-docs/images/openmetadata/connectors/domodatabase/domo-ingestion-pipeline.png new file mode 100644 index 00000000000..e9463fcb2dc Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domodatabase/domo-ingestion-pipeline.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/domodatabase/domo-service-page.png b/openmetadata-docs/images/openmetadata/connectors/domodatabase/domo-service-page.png new file mode 100644 index 00000000000..aa16e742f2b Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domodatabase/domo-service-page.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/domodatabase/image.png b/openmetadata-docs/images/openmetadata/connectors/domodatabase/image.png new file mode 100644 index 00000000000..41683f19224 Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domodatabase/image.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-add-newservices.png b/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-add-newservices.png new file mode 100644 index 00000000000..e7de1933df2 Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-add-newservices.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-pipeline-connection.png b/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-pipeline-connection.png new file mode 100644 index 00000000000..d9b70ecf81c Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-pipeline-connection.png differ diff --git a/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-pipeline-services.png b/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-pipeline-services.png new file mode 100644 index 00000000000..f1c5f111046 Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/domopipeline/domo-pipeline-services.png differ