mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-06-27 04:22:05 +00:00
MINOR: documentation improvements for Reverse Metadata (#20728)
* docs: reverse metadata * added example query for managing tags in MSSQL * added caveat for snowflake's tag based masking policies * changed name to "reverse metadata" * added cross emojis where missing * added more docs
This commit is contained in:
Imri Paran
GitHub
parent
79629a59dd
commit
fdffe67fe3
@ -24,8 +24,7 @@ Configure and schedule Athena metadata and profiler workflows from the OpenMetad
|
||||
- [Troubleshooting](/connectors/database/athena/troubleshooting)
|
||||
- [Workflow Deployment Error](#workflow-deployment-error)
|
||||
- [Related](#related)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/athena/yaml"} /%}
|
||||
|
||||
## Requirements
|
||||
@ -240,13 +239,29 @@ Find more information about [Source Identity](https://docs.aws.amazon.com/STS/la
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
Athena supports the following reverse metadata ingestion features:
|
||||
- Table Description updates
|
||||
### Description Management
|
||||
|
||||
Athena supports description updates at the following level:
|
||||
- Table level
|
||||
|
||||
### Owner Management
|
||||
|
||||
❌ Owner management is not supported for Athena.
|
||||
|
||||
### Tag Management
|
||||
|
||||
❌ Tag management is not supported for Athena.
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
Athena supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -26,6 +26,8 @@ Configure and schedule BigQuery metadata and profiler workflows from the OpenMet
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Troubleshooting](/connectors/database/bigquery/troubleshooting)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/bigquery/yaml"} /%}
|
||||
|
||||
@ -164,15 +166,32 @@ We support cross-project lineage, but the data must be ingested within a single
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
BigQuery supports the following reverse metadata ingestion features:
|
||||
- Support for Description updates (Schema, Table)
|
||||
- Tag management (Schema, Table)
|
||||
### Description Management
|
||||
|
||||
BigQuery supports description updates at the following levels:
|
||||
- Schema level
|
||||
- Table level
|
||||
|
||||
### Owner Management
|
||||
|
||||
❌ Owner management is not supported for BigQuery.
|
||||
|
||||
### Tag Management
|
||||
|
||||
BigQuery supports tag management at the following levels:
|
||||
- Schema level
|
||||
- Table level
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
BigQuery supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -26,7 +26,7 @@ Configure and schedule Clickhouse metadata and profiler workflows from the OpenM
|
||||
- [Troubleshooting](/connectors/database/clickhouse/troubleshooting)
|
||||
- [Workflow Deployment Error](#workflow-deployment-error)
|
||||
- [Related](#related)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/clickhouse/yaml"} /%}
|
||||
@ -96,13 +96,30 @@ For the usage and lineage workflow, the user will need `SELECT` privilege. You c
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
Clickhouse supports the following reverse metadata ingestion features:
|
||||
- Support for Description updates (Table, Column)
|
||||
### Description Management
|
||||
|
||||
Clickhouse supports description updates at the following levels:
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Owner Management
|
||||
|
||||
❌ Owner management is not supported for Clickhouse.
|
||||
|
||||
### Tag Management
|
||||
|
||||
❌ Tag management is not supported for Clickhouse.
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
Clickhouse supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -27,7 +27,7 @@ Configure and schedule Databricks metadata and profiler workflows from the OpenM
|
||||
- [Data Quality](/how-to-guides/data-quality-observability/quality)
|
||||
- [Lineage](/how-to-guides/data-lineage/workflow)
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
- [Troubleshooting](/connectors/database/databricks/troubleshooting)
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/databricks/yaml"} /%}
|
||||
@ -76,15 +76,39 @@ If you are using unity catalog in Databricks, then checkout the [Unity Catalog](
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
Databricks supports the following reverse metadata ingestion features:
|
||||
- Full support for Description updates (Database, Schema, Table, Column)
|
||||
- Full support for Owner management (Database, Schema, Table)
|
||||
- Full support for Tag management (Database, Schema, Table, Column)
|
||||
### Description Management
|
||||
|
||||
Databricks supports description updates at all levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Owner Management
|
||||
|
||||
Databricks supports owner management at the following levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
|
||||
### Tag Management
|
||||
|
||||
Databricks supports tag management at all levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
Databricks supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -23,7 +23,7 @@ Configure and schedule MSSQL metadata and profiler workflows from the OpenMetada
|
||||
- [Data Quality](/how-to-guides/data-quality-observability/quality)
|
||||
- [Lineage](/connectors/ingestion/lineage)
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
- [Troubleshooting](/connectors/database/mssql/troubleshooting)
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/mssql/yaml"} /%}
|
||||
@ -124,14 +124,70 @@ For details step please refer to this [link](https://docs.microsoft.com/en-us/sq
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
MSSQL supports the following reverse metadata ingestion features:
|
||||
- Description updates (Schema, Table, Column)
|
||||
- Owner management (Database, Schema)
|
||||
### Description Management
|
||||
|
||||
MSSQL supports description updates at the following levels:
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Owner Management
|
||||
|
||||
MSSQL supports owner management at the following levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
|
||||
### Tag Management
|
||||
|
||||
❌ Tag management is not supported for MSSQL.
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
MSSQL supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Example custom SQL to handle tag updates using the 'labels' option:
|
||||
|
||||
```sql
|
||||
IF NOT EXISTS (SELECT 1
|
||||
FROM {database}.sys.extended_properties
|
||||
WHERE name = N'MS_Description' And
|
||||
major_id = OBJECT_ID('{database}.{schema}.{table}') AND
|
||||
minor_id = (SELECT column_id
|
||||
FROM {database}.sys.columns
|
||||
WHERE object_id = OBJECT_ID('{database}.{schema}.{table}') AND name = '{column}')
|
||||
)
|
||||
BEGIN
|
||||
EXEC {database}.{schema}.sp_addextendedproperty
|
||||
@name = N'MS_Description',
|
||||
@value = N{description},
|
||||
@level0type = N'SCHEMA',
|
||||
@level0name = N'{schema}',
|
||||
@level1type = N'TABLE',
|
||||
@level1name = N'{table}',
|
||||
@level2type = N'COLUMN',
|
||||
@level2name = N'{column}';
|
||||
END
|
||||
ELSE
|
||||
BEGIN
|
||||
EXEC {database}.{schema}.sp_updateextendedproperty
|
||||
@name = N'MS_Description',
|
||||
@value = N{description},
|
||||
@level0type = N'SCHEMA',
|
||||
@level0name = N'{schema}',
|
||||
@level1type = N'TABLE',
|
||||
@level1name = N'{table}',
|
||||
@level2type = N'COLUMN',
|
||||
@level2name = N'{column}';
|
||||
END;
|
||||
|
||||
|
||||
ALTER TABLE ` {project_id}.{dataset_id}.{table} ` SET OPTIONS (labels = {tags});
|
||||
```
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -22,7 +22,7 @@ Configure and schedule MySQL metadata and profiler workflows from the OpenMetada
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Enable Security](#securing-mysql-connection-with-ssl-in-openmetadata)
|
||||
- [Data Lineage](/how-to-guides/data-lineage/workflow)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
- [Troubleshooting](/connectors/database/mysql/troubleshooting)
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/mysql/yaml"} /%}
|
||||
@ -237,7 +237,7 @@ Executing the profiler workflow or data quality tests, will require the user to
|
||||
|
||||
## Securing MySQL Connection with SSL in OpenMetadata
|
||||
|
||||
To establish secure connections between OpenMetadata and MySQL, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `ssl_ca`. In this case, `ssl_cert` is used for the client’s SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `ssl_ca` for the CA certificate to validate the server’s certificate.
|
||||
To establish secure connections between OpenMetadata and MySQL, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `ssl_ca`. In this case, `ssl_cert` is used for the client's SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `ssl_ca` for the CA certificate to validate the server's certificate.
|
||||
|
||||
{% image
|
||||
src="/images/v1.7/connectors/ssl_connection.png"
|
||||
@ -247,23 +247,32 @@ To establish secure connections between OpenMetadata and MySQL, navigate to the
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
MySQL supports the following reverse metadata ingestion features:
|
||||
- Support for table description updates
|
||||
### Description Management
|
||||
|
||||
MySQL supports description updates at the following level:
|
||||
- Table level
|
||||
|
||||
### Owner Management
|
||||
|
||||
Owner management is not supported for MySQL.
|
||||
|
||||
### Tag Management
|
||||
|
||||
❌ Tag management is not supported for MySQL.
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
MySQL supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
### Requirements for Reverse Metadata
|
||||
|
||||
In addition to the basic ingestion requirements, for reverse metadata ingestion the user needs:
|
||||
- `ALTER` privileges on tables to update descriptions
|
||||
|
||||
```sql
|
||||
-- Grant required privileges for reverse metadata
|
||||
GRANT ALTER ON DATABASE_NAME.TABLE_NAME TO 'USER_NAME'@'%';
|
||||
```
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
- `
|
||||
@ -21,7 +21,7 @@ Configure and schedule Oracle metadata and profiler workflows from the OpenMetad
|
||||
- [Data Quality](/how-to-guides/data-quality-observability/quality)
|
||||
- [Lineage](/connectors/ingestion/lineage)
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
- [Troubleshooting](/connectors/database/oracle/troubleshooting)
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/oracle/yaml"} /%}
|
||||
@ -134,14 +134,30 @@ there is no routine out of the box in Oracle to grant SELECT to a full schema.
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
Oracle supports the following reverse metadata ingestion features:
|
||||
- Support for Description updates (Table, Column)
|
||||
### Description Management
|
||||
|
||||
Oracle supports description updates at the following levels:
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Owner Management
|
||||
|
||||
❌ Owner management is not supported for Oracle.
|
||||
|
||||
### Tag Management
|
||||
|
||||
❌ Tag management is not supported for Oracle.
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
Oracle supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -23,7 +23,7 @@ Configure and schedule PostgreSQL metadata and profiler workflows from the OpenM
|
||||
- [Lineage](/connectors/ingestion/lineage)
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Enable Security](#securing-postgres-connection-with-ssl-in-openmetadata)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
- [Troubleshooting](/connectors/database/postgres/troubleshooting)
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/postgres/yaml"} /%}
|
||||
@ -218,15 +218,36 @@ For IAM authentication, it is recommended to choose the `allow` mode or another
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
PostgreSQL supports the following reverse metadata ingestion features:
|
||||
- Support for Description updates (Database, Schema, Table, Column)
|
||||
- Owner management (Database, Schema, Table)
|
||||
### Description Management
|
||||
|
||||
PostgreSQL supports description updates at all levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Owner Management
|
||||
|
||||
PostgreSQL supports owner management at the following levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
|
||||
### Tag Management
|
||||
|
||||
❌ Tag management is not supported for PostgreSQL.
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
PostgreSQL supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
### Requirements for Reverse Metadata
|
||||
|
||||
|
||||
@ -24,8 +24,8 @@ Configure and schedule Redshift metadata and profiler workflows from the OpenMet
|
||||
- [Lineage](/connectors/ingestion/lineage)
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Enable Security](#securing-redshift-connection-with-ssl-in-openmetadata)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Troubleshooting](/connectors/database/redshift/troubleshooting)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
troubleshooting)
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/redshift/yaml"} /%}
|
||||
|
||||
@ -131,14 +131,35 @@ Under `Advanced Config`, specify the SSL mode appropriate for your connection, s
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
Redshift supports the following reverse metadata ingestion features:
|
||||
- Full support for Description updates (Database, Schema, Table, Column)
|
||||
- Owner management (Database, Schema, Table)
|
||||
### Description Management
|
||||
|
||||
Redshift supports description updates at all levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Owner Management
|
||||
|
||||
Redshift supports owner management at the following levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
|
||||
### Tag Management
|
||||
|
||||
❌ Tag management is not supported for Redshift.
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
Redshift supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -24,7 +24,7 @@ Configure and schedule Snowflake metadata and profiler workflows from the OpenMe
|
||||
- [Data Quality](/how-to-guides/data-quality-observability/quality)
|
||||
- [Lineage](/connectors/ingestion/lineage)
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
- [Troubleshooting](/connectors/database/snowflake/troubleshooting)
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/snowflake/yaml"} /%}
|
||||
@ -152,19 +152,52 @@ Depending on your view ddl you can grant the relevant privileged as per above qu
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
Snowflake supports the following reverse metadata ingestion features:
|
||||
- Full support for Description updates (Database, Schema, Table, Column)
|
||||
- Tag management (Schema, Table, Column)
|
||||
- Automated masking policy application based on tags
|
||||
- Integration with auto-classification workflows
|
||||
### Description Management
|
||||
|
||||
### Requirements for Reverse Metadata Ingestion
|
||||
Snowflake supports full description updates at all levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Owner Management
|
||||
|
||||
❌ Owner management is not supported for Snowflake.
|
||||
|
||||
### Tag Management
|
||||
|
||||
Snowflake supports tag management at the following levels:
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
Additionally, Snowflake supports automated masking policy application based on tags. For example, when you apply a `Sensitive` tag to a column in OpenMetadata, the corresponding masking policy will be automatically applied to that column in Snowflake.
|
||||
|
||||
{% note %}
|
||||
While OpenMetadata differentiates between "classification" values and "tag" values, Snowflake's masking policies only support actions based on classification values (tag keys). Therefore, it is advised not to use PII.None along with tag masking policies.
|
||||
{% /note %}
|
||||
|
||||
```sql
|
||||
-- Create masking policy
|
||||
CREATE MASKING POLICY SENSITIVE_DATA AS (VAL STRING) RETURNS STRING -> CASE WHEN VAL IS NOT NULL THEN '**********' ELSE NULL END;
|
||||
|
||||
-- Apply masking policy to PII.Sensitive tag
|
||||
ALTER TAG PII SET MASKING POLICY SENSITIVE_DATA;
|
||||
```
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
Snowflake supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
### Requirements for Reverse Metadata
|
||||
|
||||
In addition to the basic ingestion requirements, for reverse metadata ingestion the user needs:
|
||||
- `ACCOUNTADMIN` role or a role with similar privileges to modify descriptions and tags
|
||||
@ -184,25 +217,4 @@ GRANT OWNERSHIP ON TABLE DATABASE_NAME.SCHEMA_NAME.TABLE_NAME TO ROLE NEW_ROLE R
|
||||
GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE NEW_ROLE;
|
||||
```
|
||||
|
||||
### Automated Masking Policies with Tags
|
||||
|
||||
You can configure masking policies in Snowflake to be automatically applied when specific tags are set through OpenMetadata. For example, when you apply a `Sensitive` tag to a column in OpenMetadata, the corresponding masking policy will be automatically applied to that column in Snowflake.
|
||||
|
||||
```sql
|
||||
-- Create masking policy
|
||||
CREATE MASKING POLICY SENSITIVE_DATA AS (VAL STRING) RETURNS STRING -> CASE WHEN VAL IS NOT NULL THEN '**********' ELSE NULL END;
|
||||
|
||||
-- Apply masking policy to PII.Sensitive tag
|
||||
ALTER TAG PII SET MASKING POLICY SENSITIVE_DATA;
|
||||
```
|
||||
|
||||
### Auto-Classification Integration
|
||||
|
||||
The reverse ingestion workflow can be combined with OpenMetadata's auto-classification feature. When you run the auto-classification workflow:
|
||||
1. Tags will be automatically identified and added in OpenMetadata
|
||||
2. These tags will then be synchronized back to Snowflake through the reverse ingestion process
|
||||
|
||||
This creates a seamless workflow for identifying and protecting sensitive data across both platforms.
|
||||
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Ingestion Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -17,11 +17,10 @@ In this section, we provide guides and references to use the Unity Catalog conne
|
||||
Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
|
||||
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Query Usage](/connectors/ingestion/workflows/usage)
|
||||
- [Data Quality](/how-to-guides/data-quality-observability/quality)
|
||||
- [Lineage](/connectors/ingestion/lineage)
|
||||
- [dbt Integration](/connectors/ingestion/workflows/dbt)
|
||||
- [Reverse Metadata Ingestion](#reverse-metadata-ingestion)
|
||||
- [Reverse Metadata](#reverse-metadata)
|
||||
- [Troubleshooting](/connectors/database/unity-catalog/troubleshooting)
|
||||
|
||||
{% partial file="/v1.7/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/unity-catalog/yaml"} /%}
|
||||
@ -66,15 +65,39 @@ Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
|
||||
|
||||
{% partial file="/v1.7/connectors/database/related.md" /%}
|
||||
|
||||
## Reverse Metadata Ingestion
|
||||
## Reverse Metadata
|
||||
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
|
||||
Unity Catalog supports the following reverse metadata ingestion features:
|
||||
- Full support for Description updates (Database, Schema, Table, Column)
|
||||
- Full support for Owner management (Database, Schema, Table)
|
||||
- Full support for Tag management (Database, Schema, Table, Column)
|
||||
### Description Management
|
||||
|
||||
Unity Catalog supports description updates at all levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Owner Management
|
||||
|
||||
Unity Catalog supports owner management at the following levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
|
||||
### Tag Management
|
||||
|
||||
Unity Catalog supports tag management at all levels:
|
||||
- Database level
|
||||
- Schema level
|
||||
- Table level
|
||||
- Column level
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
Unity Catalog supports custom SQL templates for metadata changes. The template is interpreted using python f-strings.
|
||||
|
||||
Examples will be provided in a future update.
|
||||
|
||||
For more details about reverse metadata ingestion, visit our [Reverse Metadata Documentation](/connectors/ingestion/workflows/reverse-metadata).
|
||||
|
||||
@ -3,7 +3,7 @@ title: Reverse Metadata Ingestion Collate | Feature & Configuration
|
||||
slug: /connectors/ingestion/workflows/reverse-metadata
|
||||
---
|
||||
|
||||
# Reverse Metadata Ingestion
|
||||
# Reverse Metadata
|
||||
{% note %}
|
||||
This feature is specific to Collate and requires the Collate Enterprise License.
|
||||
{% /note %}
|
||||
@ -14,6 +14,7 @@ Reverse Metadata is an advanced feature in OpenMetadata that facilitates bi-dire
|
||||
Reverse Metadata uses the existing service connection configuration provided during the initial metadata ingestion. You do not need to reconfigure the service connection. In order to use Reverse Metadata, this connections must use a role with write permissions.
|
||||
{% /note %}
|
||||
|
||||
|
||||
## Supported Databases and Features
|
||||
|
||||
| Database | Update Description | Update Tags | Update Owners | Custom SQL Support | Documentation |
|
||||
@ -35,18 +36,34 @@ Reverse Metadata uses the existing service connection configuration provided dur
|
||||
### Description Management
|
||||
- **Multi-level Support**: Update descriptions at database, schema, table, and column levels.
|
||||
- **Consistency**: Maintain uniform documentation across all systems.
|
||||
- **Custom SQL Support**: Flexibility to provide custom SQL templates for updates.
|
||||
|
||||
### Ownership Management
|
||||
- **Owner Assignment**: Add new owners to data assets.
|
||||
- **Owner Removal**: Remove existing owners when needed.
|
||||
- **Custom SQL Support**: Flexibility to provide custom SQL templates for updates.
|
||||
|
||||
### Tag and Label Management
|
||||
- **Bidirectional Sync**: Synchronize tags between OpenMetadata and source systems.
|
||||
- **Tag Operations**: Add new tags or remove existing ones.
|
||||
- **Conflict Prevention**: Built-in protection against ambiguous tag situations.
|
||||
- **Custom SQL Support**: Flexibility to provide custom SQL templates for updates.
|
||||
|
||||
### Custom SQL Template
|
||||
|
||||
You can define a custom SQL template to handle the metadata changes. The template is interpreted using python f-strings.
|
||||
These are the available variables:
|
||||
|
||||
| Variable | Description | Context | Type |
|
||||
|----------|-------------| --- | --- |
|
||||
| `database` | The identifier of the database or catalog | Always | String |
|
||||
| `schema` | The identifier of the schema | Always | String |
|
||||
| `table` | The identifier of the table | Always | String |
|
||||
| `column` | The identifier of the column | Always | String |
|
||||
| `description` | Refers to the description | Description update | String |
|
||||
| `owner` | The identifier of the owner being added or removed. This value can be a username or an email and depends on the source system. | Owner update | String |
|
||||
| `tag_key` | The key of the tag being added or removed (Classification) | Tag update | String |
|
||||
| `tag_value` | The value of the tag being added or removed | Tag update | String |
|
||||
| `tags` | Refers to a list of tags (e.g., BigQuery). Example: [('my_tag', 'value'), ('env', 'prod')] | Always | List[Tuple[String, String]] |
|
||||
|
||||
You can see examples of custom SQL templates in the specific connector documentation.
|
||||
|
||||
{% note %}
|
||||
When updating tags during reverse ingestion, it is important to understand the difference in tag structures. Most databases support key-value pairs for tags, whereas OpenMetadata organizes tags using classifications and tag names. During reverse ingestion, OpenMetadata maps the classification name as the tag key and the tag name as the tag value. If a new tag from the same classification is applied to a data asset that already has a tag from that classification, the reverse ingestion process may raise an ambiguous tag error. This behavior is intended to prevent accidental overwriting of existing tags at the source system.
|
||||
@ -54,6 +71,13 @@ When updating tags during reverse ingestion, it is important to understand the d
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Channels
|
||||
|
||||
Reverse Metadata is handled in "channels". Each channel can be customized to handle different types of metadata changes.
|
||||
Channels can have different filtering configurations depending on the source system and different operations to be performed.
|
||||
If a metadata update is processed by multiple channels, the metadata will be updated multiple times. This should not be an issue
|
||||
for trivial cases but can be an issue when the logic is different for each channel.
|
||||
|
||||
### Configuration Setup
|
||||
|
||||
1. Navigate to the **Applications** section.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user