diff --git a/openmetadata-docs/content/v1.7.x/applications/autopilot.md b/openmetadata-docs/content/v1.7.x/applications/autopilot.md new file mode 100644 index 00000000000..a2c767637be --- /dev/null +++ b/openmetadata-docs/content/v1.7.x/applications/autopilot.md @@ -0,0 +1,91 @@ +--- +title: Autopilot Application +slug: /applications/autopilot +collate: true +--- + +# Autopilot Application + +The **Autopilot Application** simplifies and accelerates the onboarding of new data services by automatically deploying and triggering essential ingestion workflows. +With minimal input—just the service connection details—Autopilot sets up the entire metadata ingestion process using well-defined defaults. + +## Overview + +Autopilot is designed to help users get started quickly with metadata ingestion for newly added services by automating: + +- **Metadata ingestion** +- **Lineage extraction** +- **Usage statistics collection** + +> **Note:** +> Workflows like **Profiler** and **Auto Classification** are created but not run automatically, allowing teams to review and selectively execute them based on resource and cost considerations. + +## Key Benefits + +- **Fast Setup:** Start ingesting metadata by simply defining the connection. +- **Automated Deployment:** Automatically configures and deploys ingestion workflows. +- **Editable Configurations:** All generated workflows can be reviewed and updated later. +- **Integrated Insights:** Service insights are immediately available post-ingestion, highlighting areas that need further enrichment. + +## Installation + +1. Navigate to **Settings > Applications**. +2. Click **Add Apps** and select **Autopilot** from the marketplace. + +{% image +src="/images/v1.7/applications/autopilot.png" +alt="Select Autopilot Application" +caption="Select Autopilot Application" +/%} + +3. Click **Install**. + +{% image +src="/images/v1.7/applications/autopilot1.png" +alt="Install Autopilot" +caption="Install Autopilot" +/%} + +## Configuration + +Once installed, configure the application by filling in the following fields: + +| Parameter | Description | +|:----------------------|:------------| +| **Application Type** | Set to `AutoPilotAppConfig`. | +| **Active** | Enable to activate the workflow for the selected service. | +| **Service Entity Link** | Link the service for which the ingestion workflows should be triggered. | + +{% image +src="/images/v1.7/applications/autopilot2.png" +alt="Configuration" +caption="Configuration" +/%} + +## How It Works + +1. **Define Connection:** + Create a new service (e.g., Snowflake) and test the connection. + +2. **Apply Filters:** + Add filter sets that will be inherited by all workflows. + +3. **Autopilot Activation:** + Once configured, Autopilot deploys and runs the following workflows: + - Metadata Ingestion + - Usage + - Lineage + +4. **Selective Execution:** + Profiler and Auto Classification workflows are created but not executed automatically, giving you control over profiling scope and cost. + +## Post-Ingestion Insights + +After the initial run: + +- Navigate to **Service Insights** to view KPIs such as: + - Description coverage + - PII detection + - Data asset tiers + +- Use the insights to identify areas that need enrichment—e.g., tagging, ownership, documentation. diff --git a/openmetadata-docs/content/v1.7.x/applications/index.md b/openmetadata-docs/content/v1.7.x/applications/index.md new file mode 100644 index 00000000000..2b202ede0eb --- /dev/null +++ b/openmetadata-docs/content/v1.7.x/applications/index.md @@ -0,0 +1,25 @@ +--- +title: Collate Applications +slug: /applications/ +collate: true +--- + +Explore powerful Collate Applications to automate metadata management, streamline service onboarding, and enhance data intelligence. + +{% tilesContainer %} +{% tile + title="Reverse Metadata" + description="Seamlessly push metadata changes like descriptions, tags, and ownership from Collate back to your source systems, ensuring real-time metadata consistency." + link="/applications/reverse-metadata" + / %} +{% tile + title="Autopilot" + description="Automatically deploy and trigger metadata ingestion workflows when creating new services, accelerating service onboarding with minimal manual effort." + link="/applications/autopilot" + / %} +{% tile + title="Collate AI" + description="Unlock AI-powered metadata automation with Collate AI — enabling smart documentation, data quality validation, tiering, and natural language SQL generation." + link="/collateai" + / %} +{% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.7.x/applications/reverse-metadata.md b/openmetadata-docs/content/v1.7.x/applications/reverse-metadata.md index 9a2642c2879..373ec21fc3a 100644 --- a/openmetadata-docs/content/v1.7.x/applications/reverse-metadata.md +++ b/openmetadata-docs/content/v1.7.x/applications/reverse-metadata.md @@ -4,4 +4,82 @@ slug: /applications/reverse-metadata collate: true --- -[Reverse Metadata documentation is available here](/connectors/ingestion/workflows/reverse-metadata) \ No newline at end of file +# Reverse Metadata Application + +The **Reverse Metadata Application** enables seamless synchronization of metadata updates made in **Collate** back to the original data sources. +This ensures that Collate remains the single source of truth for metadata while maintaining consistency across platforms and reinforcing governance policies. + +## Overview + +With this application, you can automatically propagate metadata changes such as: + +- **Descriptions** +- **Owners** +- **Tags** (e.g., PII-sensitive classifications) + +These updates are pushed directly to supported data source systems, including **Snowflake**, enabling real-time enforcement of data governance controls such as masking policies. + +## Key Features + +- **Automated Metadata Propagation:** + Sync metadata updates (tags, owners, descriptions) from Collate to source systems without manual intervention. + +- **Configurable Channels:** + Define multiple sync channels to target different services, asset types, or metadata types. + +- **Custom SQL Templates:** + Use SQL templates to customize update behavior per connector. + +- **On-Demand or Scheduled Execution:** + Run synchronization workflows manually or on a predefined schedule. + +## Installation + +1. Navigate to **Settings > Applications**. + +{% image +src="/images/v1.7/applications/autopilot.png" +alt="Install Reverse Metadata Application" +caption="Install Reverse Metadata Application" +/%} + +2. Click **Add Apps** and install the **Reverse Metadata Application**. +3. After installation, configure the synchronization channels as described below. + +{% image +src="/images/v1.7/applications/reverse/reverse-metadata-application.png" +alt="Configuration" +caption="Configuration" +/%} + +## Channel Configuration + +Each sync process is managed through a **channel**. +You can define multiple channels for different services or metadata types. + +| Field | Description | +|:--------------------|:------------| +| **Channel Name** | A name for identifying the sync channel. | +| **Filter** | Use the UI Query Filter Builder to define the scope of the metadata updates. You can filter by properties such as service, schema, database, owner, domain, or custom attributes. | +| **Update Descriptions** | Enable to sync updated entity descriptions from Collate to the source. | +| **Update Owners** | Enable to sync owner assignments from Collate. | +| **Update Tags** | Enable to sync tag assignments (e.g., PII) to the source system. | +| **SQL Template** | Optional. Specify a custom SQL template for updates. | + +## Execution + +Once the configuration is complete, the application can be executed in two ways: + +- **Run Now:** + For manual execution. + +- **Scheduled Run:** + For recurring, automated sync based on a defined schedule. + +{% image +src="/images/v1.7/applications/reverse/reverse-metadata-application1.png" +alt="Scheduling" +caption="Scheduling" +/%} + +[Know more about Reverse Metadata documentation](/connectors/ingestion/workflows/reverse-metadata) diff --git a/openmetadata-docs/content/v1.7.x/collate-menu.md b/openmetadata-docs/content/v1.7.x/collate-menu.md index eb129a4f2ea..263d4152076 100644 --- a/openmetadata-docs/content/v1.7.x/collate-menu.md +++ b/openmetadata-docs/content/v1.7.x/collate-menu.md @@ -728,6 +728,16 @@ site_menu: - category: Collate AI / Collate AI Documentation Agent url: /collateai/documentation-agent + - category: Applications + url: /applications + + - category: Applications / Reverse Metadata Application + url: /applications/reverse-metadata + - category: Applications / Autopilot + url: /applications/autopilot + - category: Applications / Collate AI + url: /collateai + - category: How-to Guides url: /how-to-guides diff --git a/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/autopilot.md b/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/autopilot.md new file mode 100644 index 00000000000..98c28f36f1f --- /dev/null +++ b/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/autopilot.md @@ -0,0 +1,91 @@ +--- +title: Autopilot Application +slug: /applications/autopilot +collate: true +--- + +# Autopilot Application + +The **Autopilot Application** simplifies and accelerates the onboarding of new data services by automatically deploying and triggering essential ingestion workflows. +With minimal input—just the service connection details—Autopilot sets up the entire metadata ingestion process using well-defined defaults. + +## Overview + +Autopilot is designed to help users get started quickly with metadata ingestion for newly added services by automating: + +- **Metadata ingestion** +- **Lineage extraction** +- **Usage statistics collection** + +> **Note:** +> Workflows like **Profiler** and **Auto Classification** are created but not run automatically, allowing teams to review and selectively execute them based on resource and cost considerations. + +## Key Benefits + +- **Fast Setup:** Start ingesting metadata by simply defining the connection. +- **Automated Deployment:** Automatically configures and deploys ingestion workflows. +- **Editable Configurations:** All generated workflows can be reviewed and updated later. +- **Integrated Insights:** Service insights are immediately available post-ingestion, highlighting areas that need further enrichment. + +## Installation + +1. Navigate to **Settings > Applications**. +2. Click **Add Apps** and select **Autopilot** from the marketplace. + +{% image +src="/images/v1.8/applications/autopilot.png" +alt="Select Autopilot Application" +caption="Select Autopilot Application" +/%} + +3. Click **Install**. + +{% image +src="/images/v1.8/applications/autopilot1.png" +alt="Install Autopilot" +caption="Install Autopilot" +/%} + +## Configuration + +Once installed, configure the application by filling in the following fields: + +| Parameter | Description | +|:----------------------|:------------| +| **Application Type** | Set to `AutoPilotAppConfig`. | +| **Active** | Enable to activate the workflow for the selected service. | +| **Service Entity Link** | Link the service for which the ingestion workflows should be triggered. | + +{% image +src="/images/v1.8/applications/autopilot2.png" +alt="Configuration" +caption="Configuration" +/%} + +## How It Works + +1. **Define Connection:** + Create a new service (e.g., Snowflake) and test the connection. + +2. **Apply Filters:** + Add filter sets that will be inherited by all workflows. + +3. **Autopilot Activation:** + Once configured, Autopilot deploys and runs the following workflows: + - Metadata Ingestion + - Usage + - Lineage + +4. **Selective Execution:** + Profiler and Auto Classification workflows are created but not executed automatically, giving you control over profiling scope and cost. + +## Post-Ingestion Insights + +After the initial run: + +- Navigate to **Service Insights** to view KPIs such as: + - Description coverage + - PII detection + - Data asset tiers + +- Use the insights to identify areas that need enrichment—e.g., tagging, ownership, documentation. diff --git a/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/index.md b/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/index.md new file mode 100644 index 00000000000..2b202ede0eb --- /dev/null +++ b/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/index.md @@ -0,0 +1,25 @@ +--- +title: Collate Applications +slug: /applications/ +collate: true +--- + +Explore powerful Collate Applications to automate metadata management, streamline service onboarding, and enhance data intelligence. + +{% tilesContainer %} +{% tile + title="Reverse Metadata" + description="Seamlessly push metadata changes like descriptions, tags, and ownership from Collate back to your source systems, ensuring real-time metadata consistency." + link="/applications/reverse-metadata" + / %} +{% tile + title="Autopilot" + description="Automatically deploy and trigger metadata ingestion workflows when creating new services, accelerating service onboarding with minimal manual effort." + link="/applications/autopilot" + / %} +{% tile + title="Collate AI" + description="Unlock AI-powered metadata automation with Collate AI — enabling smart documentation, data quality validation, tiering, and natural language SQL generation." + link="/collateai" + / %} +{% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/reverse-metadata.md b/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/reverse-metadata.md new file mode 100644 index 00000000000..ff23a5ca9fe --- /dev/null +++ b/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/reverse-metadata.md @@ -0,0 +1,85 @@ +--- +title: Reverse Metadata +slug: /applications/reverse-metadata +collate: true +--- + +# Reverse Metadata Application + +The **Reverse Metadata Application** enables seamless synchronization of metadata updates made in **Collate** back to the original data sources. +This ensures that Collate remains the single source of truth for metadata while maintaining consistency across platforms and reinforcing governance policies. + +## Overview + +With this application, you can automatically propagate metadata changes such as: + +- **Descriptions** +- **Owners** +- **Tags** (e.g., PII-sensitive classifications) + +These updates are pushed directly to supported data source systems, including **Snowflake**, enabling real-time enforcement of data governance controls such as masking policies. + +## Key Features + +- **Automated Metadata Propagation:** + Sync metadata updates (tags, owners, descriptions) from Collate to source systems without manual intervention. + +- **Configurable Channels:** + Define multiple sync channels to target different services, asset types, or metadata types. + +- **Custom SQL Templates:** + Use SQL templates to customize update behavior per connector. + +- **On-Demand or Scheduled Execution:** + Run synchronization workflows manually or on a predefined schedule. + +## Installation + +1. Navigate to **Settings > Applications**. + +{% image +src="/images/v1.8/applications/autopilot.png" +alt="Install Reverse Metadata Application" +caption="Install Reverse Metadata Application" +/%} + +2. Click **Add Apps** and install the **Reverse Metadata Application**. +3. After installation, configure the synchronization channels as described below. + +{% image +src="/images/v1.8/applications/reverse/reverse-metadata-application.png" +alt="Configuration" +caption="Configuration" +/%} + +## Channel Configuration + +Each sync process is managed through a **channel**. +You can define multiple channels for different services or metadata types. + +| Field | Description | +|:--------------------|:------------| +| **Channel Name** | A name for identifying the sync channel. | +| **Filter** | Use the UI Query Filter Builder to define the scope of the metadata updates. You can filter by properties such as service, schema, database, owner, domain, or custom attributes. | +| **Update Descriptions** | Enable to sync updated entity descriptions from Collate to the source. | +| **Update Owners** | Enable to sync owner assignments from Collate. | +| **Update Tags** | Enable to sync tag assignments (e.g., PII) to the source system. | +| **SQL Template** | Optional. Specify a custom SQL template for updates. | + +## Execution + +Once the configuration is complete, the application can be executed in two ways: + +- **Run Now:** + For manual execution. + +- **Scheduled Run:** + For recurring, automated sync based on a defined schedule. + +{% image +src="/images/v1.8/applications/reverse/reverse-metadata-application1.png" +alt="Scheduling" +caption="Scheduling" +/%} + +[Know more about Reverse Metadata documentation](/connectors/ingestion/workflows/reverse-metadata) diff --git a/openmetadata-docs/content/v1.8.x-SNAPSHOT/collate-menu.md b/openmetadata-docs/content/v1.8.x-SNAPSHOT/collate-menu.md index eb129a4f2ea..263d4152076 100644 --- a/openmetadata-docs/content/v1.8.x-SNAPSHOT/collate-menu.md +++ b/openmetadata-docs/content/v1.8.x-SNAPSHOT/collate-menu.md @@ -728,6 +728,16 @@ site_menu: - category: Collate AI / Collate AI Documentation Agent url: /collateai/documentation-agent + - category: Applications + url: /applications + + - category: Applications / Reverse Metadata Application + url: /applications/reverse-metadata + - category: Applications / Autopilot + url: /applications/autopilot + - category: Applications / Collate AI + url: /collateai + - category: How-to Guides url: /how-to-guides diff --git a/openmetadata-docs/images/v1.7/applications/autopilot.png b/openmetadata-docs/images/v1.7/applications/autopilot.png new file mode 100644 index 00000000000..70c33ab4b7a Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/autopilot.png differ diff --git a/openmetadata-docs/images/v1.7/applications/autopilot1.png b/openmetadata-docs/images/v1.7/applications/autopilot1.png new file mode 100644 index 00000000000..79ac5c40398 Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/autopilot1.png differ diff --git a/openmetadata-docs/images/v1.7/applications/autopilot2.png b/openmetadata-docs/images/v1.7/applications/autopilot2.png new file mode 100644 index 00000000000..c9785e548fd Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/autopilot2.png differ diff --git a/openmetadata-docs/images/v1.7/applications/reverse/reverse-metadata-application.png b/openmetadata-docs/images/v1.7/applications/reverse/reverse-metadata-application.png new file mode 100644 index 00000000000..f8830bd3b7c Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/reverse/reverse-metadata-application.png differ diff --git a/openmetadata-docs/images/v1.7/applications/reverse/reverse-metadata-application1.png b/openmetadata-docs/images/v1.7/applications/reverse/reverse-metadata-application1.png new file mode 100644 index 00000000000..19776c0f0bd Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/reverse/reverse-metadata-application1.png differ diff --git a/openmetadata-docs/images/v1.8/applications/autopilot.png b/openmetadata-docs/images/v1.8/applications/autopilot.png new file mode 100644 index 00000000000..70c33ab4b7a Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/autopilot.png differ diff --git a/openmetadata-docs/images/v1.8/applications/autopilot1.png b/openmetadata-docs/images/v1.8/applications/autopilot1.png new file mode 100644 index 00000000000..79ac5c40398 Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/autopilot1.png differ diff --git a/openmetadata-docs/images/v1.8/applications/autopilot2.png b/openmetadata-docs/images/v1.8/applications/autopilot2.png new file mode 100644 index 00000000000..c9785e548fd Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/autopilot2.png differ diff --git a/openmetadata-docs/images/v1.8/applications/reverse/reverse-metadata-application.png b/openmetadata-docs/images/v1.8/applications/reverse/reverse-metadata-application.png new file mode 100644 index 00000000000..f8830bd3b7c Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/reverse/reverse-metadata-application.png differ diff --git a/openmetadata-docs/images/v1.8/applications/reverse/reverse-metadata-application1.png b/openmetadata-docs/images/v1.8/applications/reverse/reverse-metadata-application1.png new file mode 100644 index 00000000000..19776c0f0bd Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/reverse/reverse-metadata-application1.png differ