From af9f45b00ac74405d7f9975da4979f92187bd58f Mon Sep 17 00:00:00 2001 From: Onkar Ravgan Date: Wed, 7 Aug 2024 20:50:31 +0530 Subject: [PATCH] MINOR: [DOCS] dbt cloud docs update (#17321) * dbt cloud docs update * fix token part --- .../ingestion/workflows/dbt/ingest-dbt-ui.md | 13 ++++++++++++- .../ingestion/workflows/dbt/ingest-dbt-yaml.md | 17 +++++++++++++++++ .../ingestion/workflows/dbt/ingest-dbt-ui.md | 13 ++++++++++++- .../ingestion/workflows/dbt/ingest-dbt-yaml.md | 17 +++++++++++++++++ .../locales/en-US/Database/workflows/dbt.md | 4 ++++ 5 files changed, 62 insertions(+), 2 deletions(-) diff --git a/openmetadata-docs/content/v1.4.x/connectors/ingestion/workflows/dbt/ingest-dbt-ui.md b/openmetadata-docs/content/v1.4.x/connectors/ingestion/workflows/dbt/ingest-dbt-ui.md index ed1b8d19912..c5ec41720ad 100644 --- a/openmetadata-docs/content/v1.4.x/connectors/ingestion/workflows/dbt/ingest-dbt-ui.md +++ b/openmetadata-docs/content/v1.4.x/connectors/ingestion/workflows/dbt/ingest-dbt-ui.md @@ -127,11 +127,20 @@ File server path of the `manifest.json`, `catalog.json` and `run_results.json` f #### 6. dbt Cloud Click on the the link [here](https://docs.getdbt.com/guides/getting-started) for getting started with dbt cloud account setup if not done already. -OpenMetadata uses dbt cloud APIs to fetch the `run artifacts` (manifest.json, catalog.json and run_results.json) from the most recent dbt run. The APIs need to be authenticated using an Authentication Token. Follow the link [here](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens) to generate an authentication token for your dbt cloud account. The `Account Viewer` permission is the minimum requirement for the dbt cloud token. +{% note %} + +The dbt Cloud workflow leverages the [dbt Cloud v2](https://docs.getdbt.com/dbt-cloud/api-v2#/) APIs to retrieve dbt run artifacts (manifest.json, catalog.json, and run_results.json) and ingest the dbt metadata. + +It uses the [/runs](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs) API to obtain the most recent successful dbt run, filtering by account_id, project_id and job_id if specified. The artifacts from this run are then collected using the [/artifacts](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Run%20Artifacts) API. + +Refer to the code [here](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/dbt/dbt_config.py#L142) + +{% /note %} + {% image src="/images/v1.4/features/ingestion/workflows/dbt/dbt-cloud.webp" alt="dbt-cloud" @@ -142,6 +151,8 @@ The `Account Viewer` permission is the minimum requirement for the dbt cloud tok The fields for `Dbt Cloud Account Id`, `Dbt Cloud Project Id` and `Dbt Cloud Job Id` should be numeric values. +To know how to get the values for `Dbt Cloud Account Id`, `Dbt Cloud Project Id` and `Dbt Cloud Job Id` fields check [here](/connectors/ingestion/workflows/dbt/ingest-dbt-yaml). + {% /note %} diff --git a/openmetadata-docs/content/v1.4.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md b/openmetadata-docs/content/v1.4.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md index 4a77091c0d6..c80db6e7207 100644 --- a/openmetadata-docs/content/v1.4.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md +++ b/openmetadata-docs/content/v1.4.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md @@ -495,6 +495,16 @@ In this configuration we will be fetching the dbt `manifest.json`, `catalog.json The `Account Viewer` permission is the minimum requirement for the dbt cloud token. +{% note %} + +The dbt Cloud workflow leverages the [dbt Cloud v2](https://docs.getdbt.com/dbt-cloud/api-v2#/) APIs to retrieve dbt run artifacts (manifest.json, catalog.json, and run_results.json) and ingest the dbt metadata. + +It uses the [/runs](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs) API to obtain the most recent successful dbt run, filtering by account_id, project_id and job_id if specified. The artifacts from this run are then collected using the [/artifacts](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Run%20Artifacts) API. + +Refer to the code [here](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/dbt/dbt_config.py#L142) + +{% /note %} + {% codePreview %} {% codeInfoContainer %} @@ -507,6 +517,7 @@ The `Account Viewer` permission is the minimum requirement for the dbt cloud tok {% codeInfo srNumber=61 %} - **dbtCloudAuthToken**: Please follow the instructions in [dbt Cloud's API](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens) documentation to create a dbt Cloud authentication token. +The `Account Viewer` permission is the minimum requirement for the dbt cloud token. {% /codeInfo %} {% codeInfo srNumber=62 %} @@ -518,12 +529,18 @@ For example, if the URL is `https://cloud.getdbt.com/#/accounts/1234/projects/67 {% codeInfo srNumber=63 %} - **dbtCloudJobId**: In case of multiple jobs in a dbt cloud account, specify the job's ID from which you want to extract the dbt run artifacts. If left empty, the dbt artifacts will be fetched from the most recent run on dbt cloud. + +After creating a dbt job, take note of the url which will be similar to `https://cloud.getdbt.com/#/accounts/1234/projects/6789/jobs/553344/`. The job ID is `553344`. + The value entered should be a `numeric` value. {% /codeInfo %} {% codeInfo srNumber=64 %} - **dbtCloudProjectId**: In case of multiple projects in a dbt cloud account, specify the project's ID from which you want to extract the dbt run artifacts. If left empty, the dbt artifacts will be fetched from the most recent run on dbt cloud. + +To find your project ID, sign in to your dbt cloud account and choose a specific project. Take note of the url which will be similar to `https://cloud.getdbt.com/#/accounts/1234/settings/projects/6789/`, the project ID is `6789`. + The value entered should be a `numeric` value. {% /codeInfo %} diff --git a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-ui.md b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-ui.md index b6db19a011f..8d8123d7f7d 100644 --- a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-ui.md +++ b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-ui.md @@ -127,11 +127,20 @@ File server path of the `manifest.json`, `catalog.json` and `run_results.json` f #### 6. dbt Cloud Click on the the link [here](https://docs.getdbt.com/guides/getting-started) for getting started with dbt cloud account setup if not done already. -OpenMetadata uses dbt cloud APIs to fetch the `run artifacts` (manifest.json, catalog.json and run_results.json) from the most recent dbt run. The APIs need to be authenticated using an Authentication Token. Follow the link [here](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens) to generate an authentication token for your dbt cloud account. The `Account Viewer` permission is the minimum requirement for the dbt cloud token. +{% note %} + +The dbt Cloud workflow leverages the [dbt Cloud v2](https://docs.getdbt.com/dbt-cloud/api-v2#/) APIs to retrieve dbt run artifacts (manifest.json, catalog.json, and run_results.json) and ingest the dbt metadata. + +It uses the [/runs](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs) API to obtain the most recent successful dbt run, filtering by account_id, project_id and job_id if specified. The artifacts from this run are then collected using the [/artifacts](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Run%20Artifacts) API. + +Refer to the code [here](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/dbt/dbt_config.py#L142) + +{% /note %} + {% image src="/images/v1.5/features/ingestion/workflows/dbt/dbt-cloud.webp" alt="dbt-cloud" @@ -142,6 +151,8 @@ The `Account Viewer` permission is the minimum requirement for the dbt cloud tok The fields for `Dbt Cloud Account Id`, `Dbt Cloud Project Id` and `Dbt Cloud Job Id` should be numeric values. +To know how to get the values for `Dbt Cloud Account Id`, `Dbt Cloud Project Id` and `Dbt Cloud Job Id` fields check [here](/connectors/ingestion/workflows/dbt/ingest-dbt-yaml). + {% /note %} diff --git a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md index 46b9ec58b73..c7225145b2d 100644 --- a/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md +++ b/openmetadata-docs/content/v1.5.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md @@ -495,6 +495,16 @@ In this configuration we will be fetching the dbt `manifest.json`, `catalog.json The `Account Viewer` permission is the minimum requirement for the dbt cloud token. +{% note %} + +The dbt Cloud workflow leverages the [dbt Cloud v2](https://docs.getdbt.com/dbt-cloud/api-v2#/) APIs to retrieve dbt run artifacts (manifest.json, catalog.json, and run_results.json) and ingest the dbt metadata. + +It uses the [/runs](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs) API to obtain the most recent successful dbt run, filtering by account_id, project_id and job_id if specified. The artifacts from this run are then collected using the [/artifacts](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Run%20Artifacts) API. + +Refer to the code [here](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/dbt/dbt_config.py#L142) + +{% /note %} + {% codePreview %} {% codeInfoContainer %} @@ -507,6 +517,7 @@ The `Account Viewer` permission is the minimum requirement for the dbt cloud tok {% codeInfo srNumber=61 %} - **dbtCloudAuthToken**: Please follow the instructions in [dbt Cloud's API](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens) documentation to create a dbt Cloud authentication token. +The `Account Viewer` permission is the minimum requirement for the dbt cloud token. {% /codeInfo %} {% codeInfo srNumber=62 %} @@ -518,12 +529,18 @@ For example, if the URL is `https://cloud.getdbt.com/#/accounts/1234/projects/67 {% codeInfo srNumber=63 %} - **dbtCloudJobId**: In case of multiple jobs in a dbt cloud account, specify the job's ID from which you want to extract the dbt run artifacts. If left empty, the dbt artifacts will be fetched from the most recent run on dbt cloud. + +After creating a dbt job, take note of the url which will be similar to `https://cloud.getdbt.com/#/accounts/1234/projects/6789/jobs/553344/`. The job ID is `553344`. + The value entered should be a `numeric` value. {% /codeInfo %} {% codeInfo srNumber=64 %} - **dbtCloudProjectId**: In case of multiple projects in a dbt cloud account, specify the project's ID from which you want to extract the dbt run artifacts. If left empty, the dbt artifacts will be fetched from the most recent run on dbt cloud. + +To find your project ID, sign in to your dbt cloud account and choose a specific project. Take note of the url which will be similar to `https://cloud.getdbt.com/#/accounts/1234/settings/projects/6789/`, the project ID is `6789`. + The value entered should be a `numeric` value. {% /codeInfo %} diff --git a/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/dbt.md b/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/dbt.md index a03718a6360..80094c8a134 100644 --- a/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/dbt.md +++ b/openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/dbt.md @@ -104,6 +104,8 @@ In case of multiple projects in a dbt cloud account, specify the project's ID fr If left empty, the dbt artifacts will be fetched from the most recent run on dbt cloud. +To find your project ID, sign in to your dbt cloud account and choose a specific project. Take note of the url which will be similar to `https://cloud.getdbt.com/#/accounts/1234/settings/projects/6789/`, the project ID is `6789`. + The value entered should be a `numeric` value. $$ @@ -114,6 +116,8 @@ In case of multiple jobs in a dbt cloud account, specify the job's ID from which If left empty, the dbt artifacts will be fetched from the most recent run on dbt cloud. +After creating a dbt job, take note of the url which will be similar to `https://cloud.getdbt.com/#/accounts/1234/projects/6789/jobs/553344/`. The job ID is `553344`. + The value entered should be a `numeric` value. $$