Doc: Databricks and Unity Catalog Requirement Updation (#21576)

openmetadata-docs/content/v1.6.x/connectors/database/databricks/index.md

@@ -20,6 +20,7 @@ In this section, we provide guides and references to use the Databricks connecto
 
 Configure and schedule Databricks metadata and profiler workflows from the OpenMetadata UI:
 
+- [Requirements](#requirements)
 - [Unity Catalog](#unity-catalog)
 - [Metadata Ingestion](#metadata-ingestion)
 - [Query Usage](/connectors/ingestion/workflows/usage)
@@ -33,6 +34,51 @@ Configure and schedule Databricks metadata and profiler workflows from the OpenM
 
 {% partial file="/v1.6/connectors/external-ingestion-deployment.md" /%}
 
+## Requirements
+
+### Python Requirements
+
+{% partial file="/v1.8/connectors/python-requirements.md" /%}
+
+To run the Databricks ingestion, you will need to install:
+
+```bash
+pip3 install "openmetadata-ingestion[databricks]"
+```
+
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Unity Catalog
 
 If you are using unity catalog in Databricks, then checkout the [Unity Catalog](/connectors/database/unity-catalog) connector.

openmetadata-docs/content/v1.6.x/connectors/database/databricks/yaml.md

@@ -41,6 +41,39 @@ To run the Databricks ingestion, you will need to install:
 pip3 install "openmetadata-ingestion[databricks]"
 ```
 
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Metadata Ingestion
 
 All connectors are defined as JSON Schemas.

openmetadata-docs/content/v1.6.x/connectors/database/unity-catalog/index.md

@@ -16,6 +16,7 @@ In this section, we provide guides and references to use the Unity Catalog conne
 
 Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
 
+- [Requirements](#requirements)
 - [Metadata Ingestion](#metadata-ingestion)
 - [Query Usage](/connectors/ingestion/workflows/usage)
 - [Data Quality](/how-to-guides/data-quality-observability/quality)
@@ -27,6 +28,50 @@ Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
 
 {% partial file="/v1.6/connectors/external-ingestion-deployment.md" /%}
 
+## Requirements
+
+### Python Requirements
+
+{% partial file="/v1.8/connectors/python-requirements.md" /%}
+
+To run the Unity Catalog ingestion, you will need to install:
+
+```bash
+pip3 install "openmetadata-ingestion[databricks]"
+```
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Metadata Ingestion
 
 {% partial 

openmetadata-docs/content/v1.6.x/connectors/database/unity-catalog/yaml.md

@@ -35,6 +35,39 @@ To run the Unity Catalog ingestion, you will need to install:
 pip3 install "openmetadata-ingestion[databricks]"
 ```
 
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Metadata Ingestion
 
 All connectors are defined as JSON Schemas.

openmetadata-docs/content/v1.7.x/connectors/database/databricks/index.md

@@ -20,6 +20,7 @@ In this section, we provide guides and references to use the Databricks connecto
 
 Configure and schedule Databricks metadata and profiler workflows from the OpenMetadata UI:
 
+- [Requirements](#requirements)
 - [Unity Catalog](#unity-catalog)
 - [Metadata Ingestion](#metadata-ingestion)
 - [Query Usage](/connectors/ingestion/workflows/usage)
@@ -36,6 +37,51 @@ Configure and schedule Databricks metadata and profiler workflows from the OpenM
 
 {% partial file="/v1.7/connectors/external-ingestion-deployment.md" /%}
 
+## Requirements
+
+### Python Requirements
+
+{% partial file="/v1.8/connectors/python-requirements.md" /%}
+
+To run the Databricks ingestion, you will need to install:
+
+```bash
+pip3 install "openmetadata-ingestion[databricks]"
+```
+
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Unity Catalog
 
 If you are using unity catalog in Databricks, then checkout the [Unity Catalog](/connectors/database/unity-catalog) connector.

openmetadata-docs/content/v1.7.x/connectors/database/databricks/yaml.md

@@ -43,6 +43,39 @@ To run the Databricks ingestion, you will need to install:
 pip3 install "openmetadata-ingestion[databricks]"
 ```
 
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Metadata Ingestion
 
 All connectors are defined as JSON Schemas.

openmetadata-docs/content/v1.7.x/connectors/database/unity-catalog/index.md

@@ -16,6 +16,7 @@ In this section, we provide guides and references to use the Unity Catalog conne
 
 Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
 
+- [Requirements](#requirements)
 - [Metadata Ingestion](#metadata-ingestion)
 - [Data Quality](/how-to-guides/data-quality-observability/quality)
 - [Lineage](/connectors/ingestion/lineage)
@@ -29,6 +30,50 @@ Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
 
 {% partial file="/v1.7/connectors/external-ingestion-deployment.md" /%}
 
+## Requirements
+
+### Python Requirements
+
+{% partial file="/v1.8/connectors/python-requirements.md" /%}
+
+To run the Unity Catalog ingestion, you will need to install:
+
+```bash
+pip3 install "openmetadata-ingestion[databricks]"
+```
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Metadata Ingestion
 
 {% partial 

openmetadata-docs/content/v1.7.x/connectors/database/unity-catalog/yaml.md

@@ -38,6 +38,39 @@ To run the Unity Catalog ingestion, you will need to install:
 pip3 install "openmetadata-ingestion[databricks]"
 ```
 
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Metadata Ingestion
 
 All connectors are defined as JSON Schemas.

openmetadata-docs/content/v1.8.x-SNAPSHOT/connectors/database/databricks/index.md

@@ -20,6 +20,7 @@ In this section, we provide guides and references to use the Databricks connecto
 
 Configure and schedule Databricks metadata and profiler workflows from the OpenMetadata UI:
 
+- [Requirements](#requirements)
 - [Unity Catalog](#unity-catalog)
 - [Metadata Ingestion](#metadata-ingestion)
 - [Query Usage](/connectors/ingestion/workflows/usage)
@@ -36,6 +37,51 @@ Configure and schedule Databricks metadata and profiler workflows from the OpenM
 
 {% partial file="/v1.8/connectors/external-ingestion-deployment.md" /%}
 
+## Requirements
+
+### Python Requirements
+
+{% partial file="/v1.8/connectors/python-requirements.md" /%}
+
+To run the Databricks ingestion, you will need to install:
+
+```bash
+pip3 install "openmetadata-ingestion[databricks]"
+```
+
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Unity Catalog
 
 If you are using unity catalog in Databricks, then checkout the [Unity Catalog](/connectors/database/unity-catalog) connector.

openmetadata-docs/content/v1.8.x-SNAPSHOT/connectors/database/databricks/yaml.md

@@ -43,6 +43,39 @@ To run the Databricks ingestion, you will need to install:
 pip3 install "openmetadata-ingestion[databricks]"
 ```
 
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Metadata Ingestion
 
 All connectors are defined as JSON Schemas.

openmetadata-docs/content/v1.8.x-SNAPSHOT/connectors/database/unity-catalog/index.md

@@ -16,6 +16,7 @@ In this section, we provide guides and references to use the Unity Catalog conne
 
 Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
 
+- [Requirements](#requirements)
 - [Metadata Ingestion](#metadata-ingestion)
 - [Data Quality](/how-to-guides/data-quality-observability/quality)
 - [Lineage](/connectors/ingestion/lineage)
@@ -29,6 +30,50 @@ Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
 
 {% partial file="/v1.8/connectors/external-ingestion-deployment.md" /%}
 
+## Requirements
+
+### Python Requirements
+
+{% partial file="/v1.8/connectors/python-requirements.md" /%}
+
+To run the Unity Catalog ingestion, you will need to install:
+
+```bash
+pip3 install "openmetadata-ingestion[databricks]"
+```
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
+
 ## Metadata Ingestion
 
 {% partial 

openmetadata-docs/content/v1.8.x-SNAPSHOT/connectors/database/unity-catalog/yaml.md

@@ -37,6 +37,38 @@ To run the Unity Catalog ingestion, you will need to install:
 ```bash
 pip3 install "openmetadata-ingestion[databricks]"
 ```
+### Permission Requirement
+
+To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
+### Metadata and Profiling Permissions
+
+These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
+
+```sql
+GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA <schema_name> TO `<user>`;
+GRANT SELECT ON TABLE <table_name> TO `<user>`;
+```
+
+Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
+### Usage and Lineage
+
+These permissions enable OpenMetadata to extract query history and construct lineage information.
+
+```sql
+GRANT SELECT ON SYSTEM.QUERY.HISTORY TO `<user>`;
+GRANT USE SCHEMA ON SCHEMA system.query TO `<user>`;
+```
+
+These permissions allow access to Databricks system tables that track query activity, enabling lineage and usage statistics generation.
+
+{% note %}
+
+Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
+{% /note %}
 
 ## Metadata Ingestion