diff --git a/openmetadata-docs/content/connectors/ingestion/workflows/dbt/dbt-troubleshooting.md b/openmetadata-docs/content/connectors/ingestion/workflows/dbt/dbt-troubleshooting.md new file mode 100644 index 00000000000..2f784c74062 --- /dev/null +++ b/openmetadata-docs/content/connectors/ingestion/workflows/dbt/dbt-troubleshooting.md @@ -0,0 +1,72 @@ +--- +title: dbt Troubleshooting +slug: /connectors/ingestion/workflows/dbt/dbt-troubleshooting +--- + +# Troubleshooting + +### 1. dbt tab not displaying in the UI + +After the dbt workflow is finished, check the logs to see if the dbt files were successfuly validated or not. Any missing keys in the manifest.json or catalog.json files will displayed in the logs and those keys are needed to be added. + +The dbt workflow requires the below keys to be present in the node of a manifest.json file: +- resource_type (compulsory) +- alias/name (any one of them compulsory) +- schema (compulsory) +- database (compulsory) +- description (required if description needs to be updated) +- compiled_code/compiled_sql (required if the dbt model query is to be shown in dbt tab and for query lineage) +- depends_on (required if lineage information needs to exctracted) +- columns (required if column description is to be processed) + + + +The `name/alias, schema and database` values from dbt manifest.json should match values of the `name, schema and database` of the table/view ingested in OpenMetadata. + +dbt will only be processed if these values match + + + +Below is a sample manifest.json node for reference: +```json +"model.jaffle_shop.customers": { + "resource_type": "model", + "depends_on": { + "nodes": [ + "model.jaffle_shop.stg_customers", + "model.jaffle_shop.stg_orders", + "model.jaffle_shop.stg_payments" + ] + }, + "database": "dev", + "schema": "dbt_jaffle", + "name": "customers", + "alias": "customers", + "description": "sample description", + "columns": { + "customer_id": { + "name": "customer_id", + "description": "This is a unique identifier for a customer", + "meta": {}, + "data_type": null, + "quote": null, + "tags": [] + }, + "first_name": { + "name": "first_name", + "description": "Customer's first name. PII.", + "meta": {}, + "data_type": null, + "quote": null, + "tags": [] + } + }, + "compiled_code": "sample query", +} +``` + +### 2. Lineage not getting displayed from dbt +Follow to docs [here](/connectors/ingestion/workflows/dbt/ingest-dbt-lineage) to see if necessary details are present in the manifest.json file. + +Search for the following string `Processing DBT lineage for` in the dbt workflow logs and see if any errors are causing the lineage creation to fail. + diff --git a/openmetadata-docs/content/connectors/ingestion/workflows/dbt/index.md b/openmetadata-docs/content/connectors/ingestion/workflows/dbt/index.md index 38aba20e45f..e3d142402b7 100644 --- a/openmetadata-docs/content/connectors/ingestion/workflows/dbt/index.md +++ b/openmetadata-docs/content/connectors/ingestion/workflows/dbt/index.md @@ -34,7 +34,9 @@ Queries used to create the dbt models can be viewed in the dbt tab dbt-query ### 2. dbt Lineage -Lineage from dbt models can be viewed in the Lineage tab +Lineage from dbt models can be viewed in the Lineage tab. + +For more information on how lineage is extracted from dbt take a look [here](/connectors/ingestion/workflows/dbt/ingest-dbt-lineage) dbt-lineage @@ -62,4 +64,8 @@ By default descriptions from `manifest.json` will be imported. Descriptions from ### 6. dbt Tests and Test Results Tests from dbt will only be imported if the `run_results.json` file is passed. -dbt-tests \ No newline at end of file +dbt-tests + +## Troubleshooting + +For any issues please refer to the troubleshooting documentation [here](/connectors/ingestion/workflows/dbt/dbt-troubleshooting) \ No newline at end of file diff --git a/openmetadata-docs/content/connectors/ingestion/workflows/dbt/ingest-dbt-lineage.md b/openmetadata-docs/content/connectors/ingestion/workflows/dbt/ingest-dbt-lineage.md new file mode 100644 index 00000000000..8df9f752dde --- /dev/null +++ b/openmetadata-docs/content/connectors/ingestion/workflows/dbt/ingest-dbt-lineage.md @@ -0,0 +1,48 @@ +--- +title: Ingest Lineage from dbt +slug: /connectors/ingestion/workflows/dbt/ingest-dbt-lineage +--- + +# Ingest Lineage from dbt + +Ingest the lineage information from dbt `manifest.json` file into OpenMetadata. + +OpenMetadata exctracts the lineage information from the `depends_on` and `compiled_query/compiled_code` keys from the manifest file. + +### 1. Lineage information from dbt "depends_on" key +Openmetadata fetches the lineage information from the `manifest.json` file. Below is a sample `manifest.json` file node containing lineage information under `node_name->depends_on->nodes`. + +```json +"model.jaffle_shop.customers": { + "compiled": true, + "resource_type": "model", + "depends_on": { + "macros": [], + "nodes": [ + "model.jaffle_shop.stg_customers", + "model.jaffle_shop.stg_orders", + "model.jaffle_shop.stg_payments" + ] + } +} +``` + +For the above case the lineage will be created as shown in below: +dbt-lineage-customers + +### 2. Lineage information from dbt queries +Openmetadata fetches the dbt query information from the `manifest.json` file. + +Below is a sample `manifest.json` file node containing dbt query information under `node_name->compiled_code` or `node_name->compiled_sql`. + +```json +"model.jaffle_shop.customers": { + "compiled": true, + "resource_type": "model", + "compiled_code": "Query for the model" +} +``` + +The query from dbt will be parsed by the Lineage parser to extract source and target tables to create the lineage. + +The lineage may not be created if the lineage parser is not able to parse the query. Please check the logs for any errors in this case. diff --git a/openmetadata-docs/content/menu.md b/openmetadata-docs/content/menu.md index 16de12e3a97..6c239a74610 100644 --- a/openmetadata-docs/content/menu.md +++ b/openmetadata-docs/content/menu.md @@ -516,6 +516,10 @@ site_menu: url: /connectors/ingestion/workflows/dbt/ingest-dbt-owner - category: Connectors / Ingestion / Workflows / dbt / Ingest dbt Tags url: /connectors/ingestion/workflows/dbt/ingest-dbt-tags + - category: Connectors / Ingestion / Workflows / dbt / Ingest dbt Lineage + url: /connectors/ingestion/workflows/dbt/ingest-dbt-lineage + - category: Connectors / Ingestion / Workflows / dbt / dbt troubleshooting + url: /connectors/ingestion/workflows/dbt/dbt-troubleshooting - category: Connectors / Ingestion / Workflows / Profiler url: /connectors/ingestion/workflows/profiler - category: Connectors / Ingestion / Workflows / Profiler / Metrics diff --git a/openmetadata-docs/images/openmetadata/ingestion/workflows/dbt/dbt-lineage-customers.png b/openmetadata-docs/images/openmetadata/ingestion/workflows/dbt/dbt-lineage-customers.png new file mode 100644 index 00000000000..930eb5ab4b8 Binary files /dev/null and b/openmetadata-docs/images/openmetadata/ingestion/workflows/dbt/dbt-lineage-customers.png differ