yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-11-13 01:13:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add dagster docs (#11198)

Browse Source
This commit is contained in:
Ayush Shah 2023-04-22 10:04:20 +05:30 committed by GitHub
parent 93c0605012
commit cdef9e54ee
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 53 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
openmetadata-docs-v1/content/v0.13.2/connectors/pipeline/dagster/index.md
View File

@ -167,7 +167,12 @@ desired.
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod` - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
- **Token** : Need pass token if connecting to `dagster cloud` instance - **Token** : Need pass token if connecting to `dagster cloud` instance
- Log in to your Dagster account.
- Click on the "Settings" link in the top navigation bar.
- Click on the "API Keys" tab.
- Click on the "Create a New API Key" button.
- Give your API key a name and click on the "Create API Key" button.
- Copy the generated API key to your clipboard and paste it in the field.
{% /extraContent %} {% /extraContent %}

21
openmetadata-docs-v1/content/v1.0.0/connectors/pipeline/dagster/index.md
View File

@ -9,9 +9,15 @@ In this section, we provide guides and references to use the Dagster connector.
Configure and schedule Dagster metadata and profiler workflows from the OpenMetadata UI: Configure and schedule Dagster metadata and profiler workflows from the OpenMetadata UI:
- [Requirements](#requirements) - [Dagster](#dagster)
- [Metadata Ingestion](#metadata-ingestion) - [Requirements](#requirements)
- [Dagster Versions](#dagster-versions) - [Dagster Versions](#dagster-versions)
- [Metadata Ingestion](#metadata-ingestion)
- [Service Name](#service-name)
- [Connection Options](#connection-options)
- [Metadata Ingestion Options](#metadata-ingestion-options)
- [Troubleshooting](#troubleshooting)
- [Workflow Deployment Error](#workflow-deployment-error)
If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check
the following docs to connect using Airflow SDK or with the CLI. the following docs to connect using Airflow SDK or with the CLI.
@ -167,7 +173,12 @@ desired.
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod` - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
- **Token** : Need pass token if connecting to `dagster cloud` instance - **Token** : Need pass token if connecting to `dagster cloud` instance
- Log in to your Dagster account.
- Click on the "Settings" link in the top navigation bar.
- Click on the "API Keys" tab.
- Click on the "Create a New API Key" button.
- Give your API key a name and click on the "Create API Key" button.
- Copy the generated API key to your clipboard and paste it in the field.
{% /extraContent %} {% /extraContent %}
@ -280,7 +291,7 @@ caption="View the Ingestion Pipeline from the Service Page" /%}
## Troubleshooting ## Troubleshooting
### Workflow Deployment Error ### Workflow Deployment Error
If there were any errors during the workflow deployment process, the If there were any errors during the workflow deployment process, the
Ingestion Pipeline Entity will still be created, but no workflow will be Ingestion Pipeline Entity will still be created, but no workflow will be

7
openmetadata-docs/content/connectors/pipeline/dagster/airflow.md
View File

@ -91,7 +91,12 @@ If dagster is deployed on `localhost` and entering `https://localhost:3000` into
- **ServiceConnection** - **ServiceConnection**
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod` - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
- **Token** : Need pass token if connecting to `dagster cloud` instance - **Token** : Need pass token if connecting to `dagster cloud` instance
- Log in to your Dagster account.
- Click on the "Settings" link in the top navigation bar.
- Click on the "API Keys" tab.
- Click on the "Create a New API Key" button.
- Give your API key a name and click on the "Create API Key" button.
- Copy the generated API key to your clipboard and paste it in the field.
#### Source Configuration - Source Config #### Source Configuration - Source Config
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json): The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json):

7
openmetadata-docs/content/connectors/pipeline/dagster/cli.md
View File

@ -89,7 +89,12 @@ If dagster is deployed on `localhost` and entering `https://localhost:3000` into
- **ServiceConnection** - **ServiceConnection**
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod` - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
- **Token** : Need pass token if connecting to `dagster cloud` instance - **Token** : Need pass token if connecting to `dagster cloud` instance
- Log in to your Dagster account.
- Click on the "Settings" link in the top navigation bar.
- Click on the "API Keys" tab.
- Click on the "Create a New API Key" button.
- Give your API key a name and click on the "Create API Key" button.
- Copy the generated API key to your clipboard and paste it in the field.
#### Source Configuration - Source Config #### Source Configuration - Source Config
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json): The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json):

7
openmetadata-docs/content/connectors/pipeline/dagster/index.md
View File

@ -138,7 +138,12 @@ the changes.
- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod` - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://<yourorghere>.dagster.cloud/prod`
- **Token** : Need pass token if connecting to `dagster cloud` instance - **Token** : Need pass token if connecting to `dagster cloud` instance
- Log in to your Dagster account.
- Click on the "Settings" link in the top navigation bar.
- Click on the "API Keys" tab.
- Click on the "Create a New API Key" button.
- Give your API key a name and click on the "Create API Key" button.
- Copy the generated API key to your clipboard and paste it in the field.
### 6. Configure Metadata Ingestion ### 6. Configure Metadata Ingestion
In this step we will configure the metadata ingestion pipeline, In this step we will configure the metadata ingestion pipeline,

17
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Pipeline/Dagster.md
View File

@ -3,7 +3,11 @@
In this section, we provide guides and references to use the Dagster connector. In this section, we provide guides and references to use the Dagster connector.
## Requirements ## Requirements
<!-- to be updated -->
OpenMetadata is integrated with dagster upto version [1.0.13](https://docs.dagster.io/getting-started) and will continue to work for future dagster versions.
The ingestion framework uses [dagster graphql python client](https://docs.dagster.io/_apidocs/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient) to connect to the dagster instance and perform the API calls.
You can find further information on the Kafka connector in the [docs](https://docs.open-metadata.org/connectors/pipeline/dagster). You can find further information on the Kafka connector in the [docs](https://docs.open-metadata.org/connectors/pipeline/dagster).
## Connection Details ## Connection Details
@ -12,12 +16,17 @@ $$section
### Host $(id="host") ### Host $(id="host")
URL to the Dagster instance URL to the Dagster instance
<!-- host to be updated --> Host and port of the Dagster service. For example: `localhost:1433`
$$ $$
$$section $$section
### Token $(id="token") ### Token $(id="token")
To Connect to Dagster Cloud To Connect to Dagster Cloud.
<!-- token to be updated --> - Log in to your Dagster account.
- Click on the "Settings" link in the top navigation bar.
- Click on the "API Keys" tab.
- Click on the "Create a New API Key" button.
- Give your API key a name and click on the "Create API Key" button.
- Copy the generated API key to your clipboard and paste it in the field.
$$ $$