yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-13 20:18:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenMetadata/openmetadata-docs/content/v1.1.0/connectors/pipeline/databricks-pipeline/index.md
Pere Miquel Brull 6e135302a7
[Docs] - Connectors docs with new UI, partials and external orchestrator structure (#12318) 
* Test markdoc variables

* Restructure connectors

* Fix paths

* Update index

* Database services

* Prep dashboard images

* Prep dashboard images

* Migrate dashboards

* Update partials

* Refactor pipelines

* Add missing services

* Add missing services

* Add missing services
2023-07-06 15:11:13 +02:00

2.1 KiB
Raw Blame History

title slug
Databricks Pipeline /connectors/pipeline/databricks-pipeline

Databricks Pipeline

In this section, we provide guides and references to use the Databricks Pipeline connector.

Configure and schedule Databricks Pipeline metadata workflows from the OpenMetadata UI:

{% partial file="/v1.1.0/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/pipeline/databricks-pipeline/yaml"} /%}

Requirements

{%inlineCallout icon="description" bold="OpenMetadata 0.12 or later" href="/deployment"%} To deploy OpenMetadata, check the Deployment guides. {% /inlineCallout %}

Metadata Ingestion

{% partial file="/v1.1.0/connectors/metadata-ingestion-ui.md" variables={ connector: "Databricks Pipeline", selectServicePath: "/images/v1.1.0/connectors/databrickspipeline/select-service.png", addNewServicePath: "/images/v1.1.0/connectors/databrickspipeline/add-new-service.png", serviceConnectionPath: "/images/v1.1.0/connectors/databrickspipeline/service-connection.png", } /%}

{% stepsContainer %} {% extraContent parentTagName="stepsContainer" %}

Connection Details

  • Host and Port: Enter the fully qualified hostname and port number for your Databricks Pipeline deployment in the Host and Port field.
  • Token: Generated Token to connect to Databricks Pipeline.
  • HTTP Path: Databricks Pipeline compute resources URL.
  • Connection Arguments (Optional): Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Databricks during the connection. These details must be added as Key-Value pairs.
    • In case you are using Single-Sign-On (SSO) for authentication, add the authenticator details in the Connection Arguments as a Key-Value pair as follows: "authenticator" : "sso_login_url"

{% partial file="/v1.1.0/connectors/test-connection.md" /%}

{% partial file="/v1.1.0/connectors/pipeline/configure-ingestion.md" /%}

{% partial file="/v1.1.0/connectors/ingestion-schedule-and-deploy.md" /%}

{% /stepsContainer %}

{% partial file="/v1.1.0/connectors/troubleshooting.md" /%}