mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-11-03 03:59:12 +00:00
Add dagster docs (#11198)
This commit is contained in:
Ayush Shah
GitHub
parent
93c0605012
commit
cdef9e54ee
@ -167,7 +167,12 @@ desired.
|
||||
|
||||
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
|
||||
- **Token** : Need pass token if connecting to `dagster cloud` instance
|
||||
|
||||
- Log in to your Dagster account.
|
||||
- Click on the "Settings" link in the top navigation bar.
|
||||
- Click on the "API Keys" tab.
|
||||
- Click on the "Create a New API Key" button.
|
||||
- Give your API key a name and click on the "Create API Key" button.
|
||||
- Copy the generated API key to your clipboard and paste it in the field.
|
||||
|
||||
{% /extraContent %}
|
||||
|
||||
|
||||
@ -9,9 +9,15 @@ In this section, we provide guides and references to use the Dagster connector.
|
||||
|
||||
Configure and schedule Dagster metadata and profiler workflows from the OpenMetadata UI:
|
||||
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Dagster Versions](#dagster-versions)
|
||||
- [Dagster](#dagster)
|
||||
- [Requirements](#requirements)
|
||||
- [Dagster Versions](#dagster-versions)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Service Name](#service-name)
|
||||
- [Connection Options](#connection-options)
|
||||
- [Metadata Ingestion Options](#metadata-ingestion-options)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Workflow Deployment Error](#workflow-deployment-error)
|
||||
|
||||
If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check
|
||||
the following docs to connect using Airflow SDK or with the CLI.
|
||||
@ -167,7 +173,12 @@ desired.
|
||||
|
||||
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
|
||||
- **Token** : Need pass token if connecting to `dagster cloud` instance
|
||||
|
||||
- Log in to your Dagster account.
|
||||
- Click on the "Settings" link in the top navigation bar.
|
||||
- Click on the "API Keys" tab.
|
||||
- Click on the "Create a New API Key" button.
|
||||
- Give your API key a name and click on the "Create API Key" button.
|
||||
- Copy the generated API key to your clipboard and paste it in the field.
|
||||
|
||||
{% /extraContent %}
|
||||
|
||||
@ -280,7 +291,7 @@ caption="View the Ingestion Pipeline from the Service Page" /%}
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Workflow Deployment Error
|
||||
### Workflow Deployment Error
|
||||
|
||||
If there were any errors during the workflow deployment process, the
|
||||
Ingestion Pipeline Entity will still be created, but no workflow will be
|
||||
|
||||
@ -91,7 +91,12 @@ If dagster is deployed on `localhost` and entering `https://localhost:3000` into
|
||||
- **ServiceConnection**
|
||||
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
|
||||
- **Token** : Need pass token if connecting to `dagster cloud` instance
|
||||
|
||||
- Log in to your Dagster account.
|
||||
- Click on the "Settings" link in the top navigation bar.
|
||||
- Click on the "API Keys" tab.
|
||||
- Click on the "Create a New API Key" button.
|
||||
- Give your API key a name and click on the "Create API Key" button.
|
||||
- Copy the generated API key to your clipboard and paste it in the field.
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json):
|
||||
|
||||
@ -89,7 +89,12 @@ If dagster is deployed on `localhost` and entering `https://localhost:3000` into
|
||||
- **ServiceConnection**
|
||||
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
|
||||
- **Token** : Need pass token if connecting to `dagster cloud` instance
|
||||
|
||||
- Log in to your Dagster account.
|
||||
- Click on the "Settings" link in the top navigation bar.
|
||||
- Click on the "API Keys" tab.
|
||||
- Click on the "Create a New API Key" button.
|
||||
- Give your API key a name and click on the "Create API Key" button.
|
||||
- Copy the generated API key to your clipboard and paste it in the field.
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json):
|
||||
|
||||
@ -138,7 +138,12 @@ the changes.
|
||||
|
||||
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
|
||||
- **Token** : Need pass token if connecting to `dagster cloud` instance
|
||||
|
||||
- Log in to your Dagster account.
|
||||
- Click on the "Settings" link in the top navigation bar.
|
||||
- Click on the "API Keys" tab.
|
||||
- Click on the "Create a New API Key" button.
|
||||
- Give your API key a name and click on the "Create API Key" button.
|
||||
- Copy the generated API key to your clipboard and paste it in the field.
|
||||
### 6. Configure Metadata Ingestion
|
||||
|
||||
In this step we will configure the metadata ingestion pipeline,
|
||||
|
||||
@ -3,7 +3,11 @@
|
||||
In this section, we provide guides and references to use the Dagster connector.
|
||||
|
||||
## Requirements
|
||||
<!-- to be updated -->
|
||||
|
||||
OpenMetadata is integrated with dagster upto version [1.0.13](https://docs.dagster.io/getting-started) and will continue to work for future dagster versions.
|
||||
|
||||
The ingestion framework uses [dagster graphql python client](https://docs.dagster.io/_apidocs/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient) to connect to the dagster instance and perform the API calls.
|
||||
|
||||
You can find further information on the Kafka connector in the [docs](https://docs.open-metadata.org/connectors/pipeline/dagster).
|
||||
|
||||
## Connection Details
|
||||
@ -12,12 +16,17 @@ $$section
|
||||
### Host $(id="host")
|
||||
|
||||
URL to the Dagster instance
|
||||
<!-- host to be updated -->
|
||||
Host and port of the Dagster service. For example: `localhost:1433`
|
||||
$$
|
||||
|
||||
$$section
|
||||
### Token $(id="token")
|
||||
|
||||
To Connect to Dagster Cloud
|
||||
<!-- token to be updated -->
|
||||
To Connect to Dagster Cloud.
|
||||
- Log in to your Dagster account.
|
||||
- Click on the "Settings" link in the top navigation bar.
|
||||
- Click on the "API Keys" tab.
|
||||
- Click on the "Create a New API Key" button.
|
||||
- Give your API key a name and click on the "Create API Key" button.
|
||||
- Copy the generated API key to your clipboard and paste it in the field.
|
||||
$$
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user