From cdef9e54eee466b2f955275343d3a9dfcd4c0d60 Mon Sep 17 00:00:00 2001 From: Ayush Shah Date: Sat, 22 Apr 2023 10:04:20 +0530 Subject: [PATCH] Add dagster docs (#11198) --- .../connectors/pipeline/dagster/index.md | 7 ++++++- .../connectors/pipeline/dagster/index.md | 21 ++++++++++++++----- .../connectors/pipeline/dagster/airflow.md | 7 ++++++- .../connectors/pipeline/dagster/cli.md | 7 ++++++- .../connectors/pipeline/dagster/index.md | 7 ++++++- .../public/locales/en-US/Pipeline/Dagster.md | 17 +++++++++++---- 6 files changed, 53 insertions(+), 13 deletions(-) diff --git a/openmetadata-docs-v1/content/v0.13.2/connectors/pipeline/dagster/index.md b/openmetadata-docs-v1/content/v0.13.2/connectors/pipeline/dagster/index.md index 3a7d253c654..84f3179a48b 100644 --- a/openmetadata-docs-v1/content/v0.13.2/connectors/pipeline/dagster/index.md +++ b/openmetadata-docs-v1/content/v0.13.2/connectors/pipeline/dagster/index.md @@ -167,7 +167,12 @@ desired. - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://.dagster.cloud/prod` - **Token** : Need pass token if connecting to `dagster cloud` instance - + - Log in to your Dagster account. + - Click on the "Settings" link in the top navigation bar. + - Click on the "API Keys" tab. + - Click on the "Create a New API Key" button. + - Give your API key a name and click on the "Create API Key" button. + - Copy the generated API key to your clipboard and paste it in the field. {% /extraContent %} diff --git a/openmetadata-docs-v1/content/v1.0.0/connectors/pipeline/dagster/index.md b/openmetadata-docs-v1/content/v1.0.0/connectors/pipeline/dagster/index.md index 1c2ead347cb..06e64313bc7 100644 --- a/openmetadata-docs-v1/content/v1.0.0/connectors/pipeline/dagster/index.md +++ b/openmetadata-docs-v1/content/v1.0.0/connectors/pipeline/dagster/index.md @@ -9,9 +9,15 @@ In this section, we provide guides and references to use the Dagster connector. Configure and schedule Dagster metadata and profiler workflows from the OpenMetadata UI: -- [Requirements](#requirements) -- [Metadata Ingestion](#metadata-ingestion) -- [Dagster Versions](#dagster-versions) +- [Dagster](#dagster) + - [Requirements](#requirements) + - [Dagster Versions](#dagster-versions) + - [Metadata Ingestion](#metadata-ingestion) + - [Service Name](#service-name) + - [Connection Options](#connection-options) + - [Metadata Ingestion Options](#metadata-ingestion-options) + - [Troubleshooting](#troubleshooting) + - [Workflow Deployment Error](#workflow-deployment-error) If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to connect using Airflow SDK or with the CLI. @@ -167,7 +173,12 @@ desired. - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://.dagster.cloud/prod` - **Token** : Need pass token if connecting to `dagster cloud` instance - + - Log in to your Dagster account. + - Click on the "Settings" link in the top navigation bar. + - Click on the "API Keys" tab. + - Click on the "Create a New API Key" button. + - Give your API key a name and click on the "Create API Key" button. + - Copy the generated API key to your clipboard and paste it in the field. {% /extraContent %} @@ -280,7 +291,7 @@ caption="View the Ingestion Pipeline from the Service Page" /%} ## Troubleshooting - ### Workflow Deployment Error +### Workflow Deployment Error If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be diff --git a/openmetadata-docs/content/connectors/pipeline/dagster/airflow.md b/openmetadata-docs/content/connectors/pipeline/dagster/airflow.md index 788cf4965a1..aff86894baf 100644 --- a/openmetadata-docs/content/connectors/pipeline/dagster/airflow.md +++ b/openmetadata-docs/content/connectors/pipeline/dagster/airflow.md @@ -91,7 +91,12 @@ If dagster is deployed on `localhost` and entering `https://localhost:3000` into - **ServiceConnection** - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://.dagster.cloud/prod` - **Token** : Need pass token if connecting to `dagster cloud` instance - + - Log in to your Dagster account. + - Click on the "Settings" link in the top navigation bar. + - Click on the "API Keys" tab. + - Click on the "Create a New API Key" button. + - Give your API key a name and click on the "Create API Key" button. + - Copy the generated API key to your clipboard and paste it in the field. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json): diff --git a/openmetadata-docs/content/connectors/pipeline/dagster/cli.md b/openmetadata-docs/content/connectors/pipeline/dagster/cli.md index 121d841fbeb..461b087c9c0 100644 --- a/openmetadata-docs/content/connectors/pipeline/dagster/cli.md +++ b/openmetadata-docs/content/connectors/pipeline/dagster/cli.md @@ -89,7 +89,12 @@ If dagster is deployed on `localhost` and entering `https://localhost:3000` into - **ServiceConnection** - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://.dagster.cloud/prod` - **Token** : Need pass token if connecting to `dagster cloud` instance - + - Log in to your Dagster account. + - Click on the "Settings" link in the top navigation bar. + - Click on the "API Keys" tab. + - Click on the "Create a New API Key" button. + - Give your API key a name and click on the "Create API Key" button. + - Copy the generated API key to your clipboard and paste it in the field. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json): diff --git a/openmetadata-docs/content/connectors/pipeline/dagster/index.md b/openmetadata-docs/content/connectors/pipeline/dagster/index.md index a78d0c6da33..284bdffc727 100644 --- a/openmetadata-docs/content/connectors/pipeline/dagster/index.md +++ b/openmetadata-docs/content/connectors/pipeline/dagster/index.md @@ -138,7 +138,12 @@ the changes. - **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://.dagster.cloud/prod` - **Token** : Need pass token if connecting to `dagster cloud` instance - + - Log in to your Dagster account. + - Click on the "Settings" link in the top navigation bar. + - Click on the "API Keys" tab. + - Click on the "Create a New API Key" button. + - Give your API key a name and click on the "Create API Key" button. + - Copy the generated API key to your clipboard and paste it in the field. ### 6. Configure Metadata Ingestion In this step we will configure the metadata ingestion pipeline, diff --git a/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Pipeline/Dagster.md b/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Pipeline/Dagster.md index d227fc85c27..e62fe6357a7 100644 --- a/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Pipeline/Dagster.md +++ b/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Pipeline/Dagster.md @@ -3,7 +3,11 @@ In this section, we provide guides and references to use the Dagster connector. ## Requirements - + +OpenMetadata is integrated with dagster upto version [1.0.13](https://docs.dagster.io/getting-started) and will continue to work for future dagster versions. + +The ingestion framework uses [dagster graphql python client](https://docs.dagster.io/_apidocs/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient) to connect to the dagster instance and perform the API calls. + You can find further information on the Kafka connector in the [docs](https://docs.open-metadata.org/connectors/pipeline/dagster). ## Connection Details @@ -12,12 +16,17 @@ $$section ### Host $(id="host") URL to the Dagster instance - +Host and port of the Dagster service. For example: `localhost:1433` $$ $$section ### Token $(id="token") -To Connect to Dagster Cloud - +To Connect to Dagster Cloud. +- Log in to your Dagster account. +- Click on the "Settings" link in the top navigation bar. +- Click on the "API Keys" tab. +- Click on the "Create a New API Key" button. +- Give your API key a name and click on the "Create API Key" button. +- Copy the generated API key to your clipboard and paste it in the field. $$