diff --git a/openmetadata-docs/content/v1.5.x/how-to-guides/data-governance/automation/index.md b/openmetadata-docs/content/v1.5.x/how-to-guides/data-governance/automation/index.md new file mode 100644 index 00000000000..9c64fc26241 --- /dev/null +++ b/openmetadata-docs/content/v1.5.x/how-to-guides/data-governance/automation/index.md @@ -0,0 +1,78 @@ +--- +title: Collate Automations Documentation +slug: /how-to-guides/data-governance/automation +collate: true +--- + +# Collate Automations + +{% youtube videoId="ug08aLUyTyE" start="0:00" end="14:52” width="560px" height="315px" /%} + +## Overview + +Collate's **Automation** feature is a powerful tool designed to simplify and streamline metadata management tasks. By automating repetitive actions such as assigning owners, domains, or tagging data, Collate helps maintain consistency in metadata across an organization's datasets. These automations reduce manual effort and ensure that metadata is always up-to-date, accurate, and governed according to predefined policies. + +## Why Automations are Useful + +Managing metadata manually can be challenging, particularly in dynamic environments where data constantly evolves. Collate's Automation feature addresses several key pain points: + +- **Maintaining Consistency**: Automation helps ensure that metadata such as ownership, tags, and descriptions are applied consistently across all data assets. +- **Saving Time**: Automations allow data teams to focus on higher-value tasks by eliminating the need for manual updates and maintenance. +- **Enforcing Governance Policies**: Automations help ensure that data follows organizational policies at all times by automatically applying governance rules (e.g., assigning data owners or domains). +- **Data Quality and Accountability**: Data quality suffers without clear ownership. Automating ownership assignments helps ensure that data quality issues are addressed efficiently. + +## Key Use Cases for Collate Automations + +### 1. Bulk Ownership and Domain Assignment + +{% image +src="/images/v1.5/how-to-guides/governance/bulk-ownership-and.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +- **Problem**: Many data assets lack proper ownership and domain assignment, leading to governance and accountability issues. Manually assigning owners can be error-prone and time-consuming. +- **Solution**: Automations can bulk-assign ownership and domains to datasets, ensuring all data assets are correctly categorized and owned. This process can be applied to tables, schemas, or other assets within Collate. +- **Benefit**: This use case ensures data assets have a designated owner and are organized under the appropriate domain, making data more discoverable and accountable. + +### 2. Bulk Tagging and Glossary Term Assignment + +{% image +src="/images/v1.5/how-to-guides/governance/bulk-tagging-glossary.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +- **Problem**: Manually applying the same tags or glossary terms to multiple datasets can be inefficient and inconsistent. +- **Solution**: Automations allow users to bulk-apply tags (e.g., PII) or glossary terms (e.g., Customer ID) to specific datasets, ensuring uniformity across the platform. +- **Benefit**: This automation reduces the risk of missing important tags like PII-sensitive and ensures that key metadata elements are applied consistently across datasets. + +### 3. Metadata Propagation via Lineage + +{% image +src="/images/v1.5/how-to-guides/governance/metadata-propogation.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +- **Problem**: When metadata such as tags, descriptions, or glossary terms are updated in one part of the data lineage, they may not be propagated across related datasets, leading to inconsistencies. +- **Solution**: Use automations to propagate metadata across related datasets, ensuring that all relevant data inherits the correct metadata properties from the source dataset. +- **Benefit**: Metadata consistency is ensured across the entire data lineage, reducing the need for manual updates and maintaining a single source of truth. + +### 4. Automatic PII Detection and Tagging + +{% image +src="/images/v1.5/how-to-guides/governance/automatic-detection.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +- **Problem**: Manually identifying and tagging Personally Identifiable Information (PII) across large datasets is labor-intensive and prone to errors. +- **Solution**: Automations can automatically detect PII data (e.g., emails, usernames) and apply relevant tags to ensure that sensitive data is flagged appropriately for compliance. +- **Benefit**: Ensures compliance with data protection regulations by consistently tagging sensitive data, reducing the risk of non-compliance. + +## Best Practices + +- **Validate Assets Before Applying Actions**: Always use the **Explore** page to verify the assets that will be affected by the automation. This ensures that only the intended datasets are updated. +- **Use Automation Logs**: Regularly check the **Recent Runs** logs to monitor automation activity and ensure that they are running as expected. +- **Propagate Metadata Thoughtfully**: When propagating metadata via lineage, make sure that the source metadata is correct before applying it across multiple datasets. diff --git a/openmetadata-docs/content/v1.5.x/how-to-guides/data-governance/automation/set-up-automation.md b/openmetadata-docs/content/v1.5.x/how-to-guides/data-governance/automation/set-up-automation.md new file mode 100644 index 00000000000..0b57f0a55b0 --- /dev/null +++ b/openmetadata-docs/content/v1.5.x/how-to-guides/data-governance/automation/set-up-automation.md @@ -0,0 +1,67 @@ +--- +title: How to Set Up Automations in OpenMetadata +slug: /how-to-guides/data-governance/automation/set-up-automation +collate: true +--- + +# How to Set Up Automations in Collate + +### Step 1: Access the Automations Section +In the OpenMetadata UI, navigate to **Govern>Automations**. +This will take you to the Automations page where you can view and manage your existing automations. + +{% image +src="/images/v1.5/how-to-guides/governance/automation-1.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +### Step 2: Add a New Automation +In the Automations page, click the **Add Automation** button located at the top right of the page. +A pop-up window will appear to begin the process of adding a new automation. + +{% image +src="/images/v1.5/how-to-guides/governance/automation-2.png" +alt="Add Automation" +caption="Add Automation" +/%} + +### Step 3: Fill in Automation Details +In the pop-up window, provide the necessary information to set up the automation: +- **Automation Name**: Give a meaningful name to the automation for easy identification. +- **Description**: Add a brief description explaining what this automation will do (e.g., "Daily metadata ingestion for database XYZ"). +- **Logic/Conditions**: Define any conditions or specific criteria needed for this automation to work (e.g., specific tables or columns to be included). + Ensure that the logic is set up as per your specific requirements to make the automation useful for your workflows. + +{% image +src="/images/v1.5/how-to-guides/governance/automation-4.png" +alt="Automation details" +caption="Automation details" +/%} + +{% image +src="/images/v1.5/how-to-guides/governance/automation-5.png" +alt="Automation logics" +caption="Automation logics" +/%} + +### Step 4: Configure Automation Interval +Once you've filled in the required details, click **Next**. +On the next page, you’ll be prompted to select the interval for the automation. This defines how frequently the automation should run (e.g., daily, weekly, or custom intervals). +Review your settings and click **Automate** once you are satisfied with the configuration. + +{% image +src="/images/v1.5/how-to-guides/governance/automation-6.png" +alt="Automation Interval" +caption="Automation Interval" +/%} + +### Step 5: Manage Your Automation +After completing the setup, your automation will appear in the Automations list. +To manage the automation, click on the three dots next to the automation entry. From here, you can **edit**, **re-deploy**, **delete**, etc. + +{% image +src="/images/v1.5/how-to-guides/governance/automation-7.png" +alt="Manage Your Automation" +caption="Manage Your Automation" +/%} diff --git a/openmetadata-docs/content/v1.5.x/menu.md b/openmetadata-docs/content/v1.5.x/menu.md index 04e9c027322..dacf205460f 100644 --- a/openmetadata-docs/content/v1.5.x/menu.md +++ b/openmetadata-docs/content/v1.5.x/menu.md @@ -880,6 +880,12 @@ site_menu: - category: How-to Guides / Data Governance url: /how-to-guides/data-governance + - category: How-to Guides / Data Governance / Automation + url: /how-to-guides/data-governance/automation + isCollateOnly: true + - category: How-to Guides / Data Governance / Automation / How to Set Up Automations in Collate + url: /how-to-guides/data-governance/automation/set-up-automation + isCollateOnly: true - category: How-to Guides / Data Governance / Glossary url: /how-to-guides/data-governance/glossary - category: How-to Guides / Data Governance / Glossary / What is a Glossary Term diff --git a/openmetadata-docs/content/v1.6.x-SNAPSHOT/how-to-guides/data-governance/automation/index.md b/openmetadata-docs/content/v1.6.x-SNAPSHOT/how-to-guides/data-governance/automation/index.md new file mode 100644 index 00000000000..1701954e108 --- /dev/null +++ b/openmetadata-docs/content/v1.6.x-SNAPSHOT/how-to-guides/data-governance/automation/index.md @@ -0,0 +1,78 @@ +--- +title: Collate Automations Documentation +slug: /how-to-guides/data-governance/automation +collate: true +--- + +# Collate Automations + +{% youtube videoId="ug08aLUyTyE" start="0:00" end="14:52” width="560px" height="315px" /%} + +## Overview + +Collate's **Automation** feature is a powerful tool designed to simplify and streamline metadata management tasks. By automating repetitive actions such as assigning owners, domains, or tagging data, Collate helps maintain consistency in metadata across an organization's datasets. These automations reduce manual effort and ensure that metadata is always up-to-date, accurate, and governed according to predefined policies. + +## Why Automations are Useful + +Managing metadata manually can be challenging, particularly in dynamic environments where data constantly evolves. Collate's Automation feature addresses several key pain points: + +- **Maintaining Consistency**: Automation helps ensure that metadata such as ownership, tags, and descriptions are applied consistently across all data assets. +- **Saving Time**: Automations allow data teams to focus on higher-value tasks by eliminating the need for manual updates and maintenance. +- **Enforcing Governance Policies**: Automations help ensure that data follows organizational policies at all times by automatically applying governance rules (e.g., assigning data owners or domains). +- **Data Quality and Accountability**: Data quality suffers without clear ownership. Automating ownership assignments helps ensure that data quality issues are addressed efficiently. + +## Key Use Cases for Collate Automations + +### 1. Bulk Ownership and Domain Assignment + +{% image +src="/images/v1.6/how-to-guides/governance/bulk-ownership-and.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +- **Problem**: Many data assets lack proper ownership and domain assignment, leading to governance and accountability issues. Manually assigning owners can be error-prone and time-consuming. +- **Solution**: Automations can bulk-assign ownership and domains to datasets, ensuring all data assets are correctly categorized and owned. This process can be applied to tables, schemas, or other assets within Collate. +- **Benefit**: This use case ensures data assets have a designated owner and are organized under the appropriate domain, making data more discoverable and accountable. + +### 2. Bulk Tagging and Glossary Term Assignment + +{% image +src="/images/v1.6/how-to-guides/governance/bulk-tagging-glossary.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +- **Problem**: Manually applying the same tags or glossary terms to multiple datasets can be inefficient and inconsistent. +- **Solution**: Automations allow users to bulk-apply tags (e.g., PII) or glossary terms (e.g., Customer ID) to specific datasets, ensuring uniformity across the platform. +- **Benefit**: This automation reduces the risk of missing important tags like PII-sensitive and ensures that key metadata elements are applied consistently across datasets. + +### 3. Metadata Propagation via Lineage + +{% image +src="/images/v1.6/how-to-guides/governance/metadata-propogation.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +- **Problem**: When metadata such as tags, descriptions, or glossary terms are updated in one part of the data lineage, they may not be propagated across related datasets, leading to inconsistencies. +- **Solution**: Use automations to propagate metadata across related datasets, ensuring that all relevant data inherits the correct metadata properties from the source dataset. +- **Benefit**: Metadata consistency is ensured across the entire data lineage, reducing the need for manual updates and maintaining a single source of truth. + +### 4. Automatic PII Detection and Tagging + +{% image +src="/images/v1.6/how-to-guides/governance/automatic-detection.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +- **Problem**: Manually identifying and tagging Personally Identifiable Information (PII) across large datasets is labor-intensive and prone to errors. +- **Solution**: Automations can automatically detect PII data (e.g., emails, usernames) and apply relevant tags to ensure that sensitive data is flagged appropriately for compliance. +- **Benefit**: Ensures compliance with data protection regulations by consistently tagging sensitive data, reducing the risk of non-compliance. + +## Best Practices + +- **Validate Assets Before Applying Actions**: Always use the **Explore** page to verify the assets that will be affected by the automation. This ensures that only the intended datasets are updated. +- **Use Automation Logs**: Regularly check the **Recent Runs** logs to monitor automation activity and ensure that they are running as expected. +- **Propagate Metadata Thoughtfully**: When propagating metadata via lineage, make sure that the source metadata is correct before applying it across multiple datasets. diff --git a/openmetadata-docs/content/v1.6.x-SNAPSHOT/how-to-guides/data-governance/automation/set-up-automation.md b/openmetadata-docs/content/v1.6.x-SNAPSHOT/how-to-guides/data-governance/automation/set-up-automation.md new file mode 100644 index 00000000000..a4afc9e61f4 --- /dev/null +++ b/openmetadata-docs/content/v1.6.x-SNAPSHOT/how-to-guides/data-governance/automation/set-up-automation.md @@ -0,0 +1,67 @@ +--- +title: How to Set Up Automations in OpenMetadata +slug: /how-to-guides/data-governance/automation/set-up-automation +collate: true +--- + +# How to Set Up Automations in Collate + +### Step 1: Access the Automations Section +In the OpenMetadata UI, navigate to **Govern>Automations**. +This will take you to the Automations page where you can view and manage your existing automations. + +{% image +src="/images/v1.6/how-to-guides/governance/automation-1.png" +alt="Getting started with Automation" +caption="Getting started with Automation" +/%} + +### Step 2: Add a New Automation +In the Automations page, click the **Add Automation** button located at the top right of the page. +A pop-up window will appear to begin the process of adding a new automation. + +{% image +src="/images/v1.6/how-to-guides/governance/automation-2.png" +alt="Add Automation" +caption="Add Automation" +/%} + +### Step 3: Fill in Automation Details +In the pop-up window, provide the necessary information to set up the automation: +- **Automation Name**: Give a meaningful name to the automation for easy identification. +- **Description**: Add a brief description explaining what this automation will do (e.g., "Daily metadata ingestion for database XYZ"). +- **Logic/Conditions**: Define any conditions or specific criteria needed for this automation to work (e.g., specific tables or columns to be included). + Ensure that the logic is set up as per your specific requirements to make the automation useful for your workflows. + +{% image +src="/images/v1.6/how-to-guides/governance/automation-4.png" +alt="Automation details" +caption="Automation details" +/%} + +{% image +src="/images/v1.6/how-to-guides/governance/automation-5.png" +alt="Automation logics" +caption="Automation logics" +/%} + +### Step 4: Configure Automation Interval +Once you've filled in the required details, click **Next**. +On the next page, you’ll be prompted to select the interval for the automation. This defines how frequently the automation should run (e.g., daily, weekly, or custom intervals). +Review your settings and click **Automate** once you are satisfied with the configuration. + +{% image +src="/images/v1.6/how-to-guides/governance/automation-6.png" +alt="Automation Interval" +caption="Automation Interval" +/%} + +### Step 5: Manage Your Automation +After completing the setup, your automation will appear in the Automations list. +To manage the automation, click on the three dots next to the automation entry. From here, you can **edit**, **re-deploy**, **delete**, etc. + +{% image +src="/images/v1.6/how-to-guides/governance/automation-7.png" +alt="Manage Your Automation" +caption="Manage Your Automation" +/%} diff --git a/openmetadata-docs/content/v1.6.x-SNAPSHOT/menu.md b/openmetadata-docs/content/v1.6.x-SNAPSHOT/menu.md index 5b8880f1e24..2516e26b41e 100644 --- a/openmetadata-docs/content/v1.6.x-SNAPSHOT/menu.md +++ b/openmetadata-docs/content/v1.6.x-SNAPSHOT/menu.md @@ -893,6 +893,12 @@ site_menu: - category: How-to Guides / Data Governance url: /how-to-guides/data-governance + - category: How-to Guides / Data Governance / Automation + url: /how-to-guides/data-governance/automation + isCollateOnly: true + - category: How-to Guides / Data Governance / Automation / How to Set Up Automations in Collate + url: /how-to-guides/data-governance/automation/set-up-automation + isCollateOnly: true - category: How-to Guides / Data Governance / Glossary url: /how-to-guides/data-governance/glossary - category: How-to Guides / Data Governance / Glossary / What is a Glossary Term diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/automatic-detection.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/automatic-detection.png new file mode 100644 index 00000000000..c5b8077414f Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/automatic-detection.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-1.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-1.png new file mode 100644 index 00000000000..f6cefd8c8d7 Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-1.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-2.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-2.png new file mode 100644 index 00000000000..cef6b5428fb Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-2.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-3.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-3.png new file mode 100644 index 00000000000..b9b195c7ca0 Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-3.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-4.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-4.png new file mode 100644 index 00000000000..4c49ede5a0b Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-4.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-5.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-5.png new file mode 100644 index 00000000000..7f18a8a2513 Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-5.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-6.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-6.png new file mode 100644 index 00000000000..155a4f05fb7 Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-6.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-7.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-7.png new file mode 100644 index 00000000000..23b11e4b2b7 Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/automation-7.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/bulk-ownership-and.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/bulk-ownership-and.png new file mode 100644 index 00000000000..b15cef41a7c Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/bulk-ownership-and.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/bulk-tagging-glossary.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/bulk-tagging-glossary.png new file mode 100644 index 00000000000..8497a169eb7 Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/bulk-tagging-glossary.png differ diff --git a/openmetadata-docs/images/v1.5/how-to-guides/governance/metadata-propogation.png b/openmetadata-docs/images/v1.5/how-to-guides/governance/metadata-propogation.png new file mode 100644 index 00000000000..92562cb3cfd Binary files /dev/null and b/openmetadata-docs/images/v1.5/how-to-guides/governance/metadata-propogation.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/automatic-detection.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/automatic-detection.png new file mode 100644 index 00000000000..c5b8077414f Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/automatic-detection.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-1.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-1.png new file mode 100644 index 00000000000..f6cefd8c8d7 Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-1.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-2.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-2.png new file mode 100644 index 00000000000..cef6b5428fb Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-2.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-3.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-3.png new file mode 100644 index 00000000000..b9b195c7ca0 Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-3.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-4.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-4.png new file mode 100644 index 00000000000..4c49ede5a0b Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-4.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-5.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-5.png new file mode 100644 index 00000000000..7f18a8a2513 Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-5.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-6.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-6.png new file mode 100644 index 00000000000..155a4f05fb7 Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-6.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-7.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-7.png new file mode 100644 index 00000000000..23b11e4b2b7 Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/automation-7.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/bulk-ownership-and.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/bulk-ownership-and.png new file mode 100644 index 00000000000..b15cef41a7c Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/bulk-ownership-and.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/bulk-tagging-glossary.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/bulk-tagging-glossary.png new file mode 100644 index 00000000000..8497a169eb7 Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/bulk-tagging-glossary.png differ diff --git a/openmetadata-docs/images/v1.6/how-to-guides/governance/metadata-propogation.png b/openmetadata-docs/images/v1.6/how-to-guides/governance/metadata-propogation.png new file mode 100644 index 00000000000..92562cb3cfd Binary files /dev/null and b/openmetadata-docs/images/v1.6/how-to-guides/governance/metadata-propogation.png differ