Doc: Applications Addition (#20999)

This commit is contained in:
Rounak Dhillon 2025-04-28 17:57:34 +05:30 committed by GitHub
parent 2e662f01ed
commit a25719b7c1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
18 changed files with 416 additions and 1 deletions

View File

@ -0,0 +1,91 @@
---
title: Autopilot Application
slug: /applications/autopilot
collate: true
---
# Autopilot Application
The **Autopilot Application** simplifies and accelerates the onboarding of new data services by automatically deploying and triggering essential ingestion workflows.
With minimal input—just the service connection details—Autopilot sets up the entire metadata ingestion process using well-defined defaults.
## Overview
Autopilot is designed to help users get started quickly with metadata ingestion for newly added services by automating:
- **Metadata ingestion**
- **Lineage extraction**
- **Usage statistics collection**
> **Note:**
> Workflows like **Profiler** and **Auto Classification** are created but not run automatically, allowing teams to review and selectively execute them based on resource and cost considerations.
## Key Benefits
- **Fast Setup:** Start ingesting metadata by simply defining the connection.
- **Automated Deployment:** Automatically configures and deploys ingestion workflows.
- **Editable Configurations:** All generated workflows can be reviewed and updated later.
- **Integrated Insights:** Service insights are immediately available post-ingestion, highlighting areas that need further enrichment.
## Installation
1. Navigate to **Settings > Applications**.
2. Click **Add Apps** and select **Autopilot** from the marketplace.
{% image
src="/images/v1.7/applications/autopilot.png"
alt="Select Autopilot Application"
caption="Select Autopilot Application"
/%}
3. Click **Install**.
{% image
src="/images/v1.7/applications/autopilot1.png"
alt="Install Autopilot"
caption="Install Autopilot"
/%}
## Configuration
Once installed, configure the application by filling in the following fields:
| Parameter | Description |
|:----------------------|:------------|
| **Application Type** | Set to `AutoPilotAppConfig`. |
| **Active** | Enable to activate the workflow for the selected service. |
| **Service Entity Link** | Link the service for which the ingestion workflows should be triggered. |
{% image
src="/images/v1.7/applications/autopilot2.png"
alt="Configuration"
caption="Configuration"
/%}
## How It Works
1. **Define Connection:**
Create a new service (e.g., Snowflake) and test the connection.
2. **Apply Filters:**
Add filter sets that will be inherited by all workflows.
3. **Autopilot Activation:**
Once configured, Autopilot deploys and runs the following workflows:
- Metadata Ingestion
- Usage
- Lineage
4. **Selective Execution:**
Profiler and Auto Classification workflows are created but not executed automatically, giving you control over profiling scope and cost.
## Post-Ingestion Insights
After the initial run:
- Navigate to **Service Insights** to view KPIs such as:
- Description coverage
- PII detection
- Data asset tiers
- Use the insights to identify areas that need enrichment—e.g., tagging, ownership, documentation.

View File

@ -0,0 +1,25 @@
---
title: Collate Applications
slug: /applications/
collate: true
---
Explore powerful Collate Applications to automate metadata management, streamline service onboarding, and enhance data intelligence.
{% tilesContainer %}
{% tile
title="Reverse Metadata"
description="Seamlessly push metadata changes like descriptions, tags, and ownership from Collate back to your source systems, ensuring real-time metadata consistency."
link="/applications/reverse-metadata"
/ %}
{% tile
title="Autopilot"
description="Automatically deploy and trigger metadata ingestion workflows when creating new services, accelerating service onboarding with minimal manual effort."
link="/applications/autopilot"
/ %}
{% tile
title="Collate AI"
description="Unlock AI-powered metadata automation with Collate AI — enabling smart documentation, data quality validation, tiering, and natural language SQL generation."
link="/collateai"
/ %}
{% /tilesContainer %}

View File

@ -4,4 +4,82 @@ slug: /applications/reverse-metadata
collate: true
---
[Reverse Metadata documentation is available here](/connectors/ingestion/workflows/reverse-metadata)
# Reverse Metadata Application
The **Reverse Metadata Application** enables seamless synchronization of metadata updates made in **Collate** back to the original data sources.
This ensures that Collate remains the single source of truth for metadata while maintaining consistency across platforms and reinforcing governance policies.
## Overview
With this application, you can automatically propagate metadata changes such as:
- **Descriptions**
- **Owners**
- **Tags** (e.g., PII-sensitive classifications)
These updates are pushed directly to supported data source systems, including **Snowflake**, enabling real-time enforcement of data governance controls such as masking policies.
## Key Features
- **Automated Metadata Propagation:**
Sync metadata updates (tags, owners, descriptions) from Collate to source systems without manual intervention.
- **Configurable Channels:**
Define multiple sync channels to target different services, asset types, or metadata types.
- **Custom SQL Templates:**
Use SQL templates to customize update behavior per connector.
- **On-Demand or Scheduled Execution:**
Run synchronization workflows manually or on a predefined schedule.
## Installation
1. Navigate to **Settings > Applications**.
{% image
src="/images/v1.7/applications/autopilot.png"
alt="Install Reverse Metadata Application"
caption="Install Reverse Metadata Application"
/%}
2. Click **Add Apps** and install the **Reverse Metadata Application**.
3. After installation, configure the synchronization channels as described below.
{% image
src="/images/v1.7/applications/reverse/reverse-metadata-application.png"
alt="Configuration"
caption="Configuration"
/%}
## Channel Configuration
Each sync process is managed through a **channel**.
You can define multiple channels for different services or metadata types.
| Field | Description |
|:--------------------|:------------|
| **Channel Name** | A name for identifying the sync channel. |
| **Filter** | Use the UI Query Filter Builder to define the scope of the metadata updates. You can filter by properties such as service, schema, database, owner, domain, or custom attributes. |
| **Update Descriptions** | Enable to sync updated entity descriptions from Collate to the source. |
| **Update Owners** | Enable to sync owner assignments from Collate. |
| **Update Tags** | Enable to sync tag assignments (e.g., PII) to the source system. |
| **SQL Template** | Optional. Specify a custom SQL template for updates. |
## Execution
Once the configuration is complete, the application can be executed in two ways:
- **Run Now:**
For manual execution.
- **Scheduled Run:**
For recurring, automated sync based on a defined schedule.
{% image
src="/images/v1.7/applications/reverse/reverse-metadata-application1.png"
alt="Scheduling"
caption="Scheduling"
/%}
[Know more about Reverse Metadata documentation](/connectors/ingestion/workflows/reverse-metadata)

View File

@ -728,6 +728,16 @@ site_menu:
- category: Collate AI / Collate AI Documentation Agent
url: /collateai/documentation-agent
- category: Applications
url: /applications
- category: Applications / Reverse Metadata Application
url: /applications/reverse-metadata
- category: Applications / Autopilot
url: /applications/autopilot
- category: Applications / Collate AI
url: /collateai
- category: How-to Guides
url: /how-to-guides

View File

@ -0,0 +1,91 @@
---
title: Autopilot Application
slug: /applications/autopilot
collate: true
---
# Autopilot Application
The **Autopilot Application** simplifies and accelerates the onboarding of new data services by automatically deploying and triggering essential ingestion workflows.
With minimal input—just the service connection details—Autopilot sets up the entire metadata ingestion process using well-defined defaults.
## Overview
Autopilot is designed to help users get started quickly with metadata ingestion for newly added services by automating:
- **Metadata ingestion**
- **Lineage extraction**
- **Usage statistics collection**
> **Note:**
> Workflows like **Profiler** and **Auto Classification** are created but not run automatically, allowing teams to review and selectively execute them based on resource and cost considerations.
## Key Benefits
- **Fast Setup:** Start ingesting metadata by simply defining the connection.
- **Automated Deployment:** Automatically configures and deploys ingestion workflows.
- **Editable Configurations:** All generated workflows can be reviewed and updated later.
- **Integrated Insights:** Service insights are immediately available post-ingestion, highlighting areas that need further enrichment.
## Installation
1. Navigate to **Settings > Applications**.
2. Click **Add Apps** and select **Autopilot** from the marketplace.
{% image
src="/images/v1.8/applications/autopilot.png"
alt="Select Autopilot Application"
caption="Select Autopilot Application"
/%}
3. Click **Install**.
{% image
src="/images/v1.8/applications/autopilot1.png"
alt="Install Autopilot"
caption="Install Autopilot"
/%}
## Configuration
Once installed, configure the application by filling in the following fields:
| Parameter | Description |
|:----------------------|:------------|
| **Application Type** | Set to `AutoPilotAppConfig`. |
| **Active** | Enable to activate the workflow for the selected service. |
| **Service Entity Link** | Link the service for which the ingestion workflows should be triggered. |
{% image
src="/images/v1.8/applications/autopilot2.png"
alt="Configuration"
caption="Configuration"
/%}
## How It Works
1. **Define Connection:**
Create a new service (e.g., Snowflake) and test the connection.
2. **Apply Filters:**
Add filter sets that will be inherited by all workflows.
3. **Autopilot Activation:**
Once configured, Autopilot deploys and runs the following workflows:
- Metadata Ingestion
- Usage
- Lineage
4. **Selective Execution:**
Profiler and Auto Classification workflows are created but not executed automatically, giving you control over profiling scope and cost.
## Post-Ingestion Insights
After the initial run:
- Navigate to **Service Insights** to view KPIs such as:
- Description coverage
- PII detection
- Data asset tiers
- Use the insights to identify areas that need enrichment—e.g., tagging, ownership, documentation.

View File

@ -0,0 +1,25 @@
---
title: Collate Applications
slug: /applications/
collate: true
---
Explore powerful Collate Applications to automate metadata management, streamline service onboarding, and enhance data intelligence.
{% tilesContainer %}
{% tile
title="Reverse Metadata"
description="Seamlessly push metadata changes like descriptions, tags, and ownership from Collate back to your source systems, ensuring real-time metadata consistency."
link="/applications/reverse-metadata"
/ %}
{% tile
title="Autopilot"
description="Automatically deploy and trigger metadata ingestion workflows when creating new services, accelerating service onboarding with minimal manual effort."
link="/applications/autopilot"
/ %}
{% tile
title="Collate AI"
description="Unlock AI-powered metadata automation with Collate AI — enabling smart documentation, data quality validation, tiering, and natural language SQL generation."
link="/collateai"
/ %}
{% /tilesContainer %}

View File

@ -0,0 +1,85 @@
---
title: Reverse Metadata
slug: /applications/reverse-metadata
collate: true
---
# Reverse Metadata Application
The **Reverse Metadata Application** enables seamless synchronization of metadata updates made in **Collate** back to the original data sources.
This ensures that Collate remains the single source of truth for metadata while maintaining consistency across platforms and reinforcing governance policies.
## Overview
With this application, you can automatically propagate metadata changes such as:
- **Descriptions**
- **Owners**
- **Tags** (e.g., PII-sensitive classifications)
These updates are pushed directly to supported data source systems, including **Snowflake**, enabling real-time enforcement of data governance controls such as masking policies.
## Key Features
- **Automated Metadata Propagation:**
Sync metadata updates (tags, owners, descriptions) from Collate to source systems without manual intervention.
- **Configurable Channels:**
Define multiple sync channels to target different services, asset types, or metadata types.
- **Custom SQL Templates:**
Use SQL templates to customize update behavior per connector.
- **On-Demand or Scheduled Execution:**
Run synchronization workflows manually or on a predefined schedule.
## Installation
1. Navigate to **Settings > Applications**.
{% image
src="/images/v1.8/applications/autopilot.png"
alt="Install Reverse Metadata Application"
caption="Install Reverse Metadata Application"
/%}
2. Click **Add Apps** and install the **Reverse Metadata Application**.
3. After installation, configure the synchronization channels as described below.
{% image
src="/images/v1.8/applications/reverse/reverse-metadata-application.png"
alt="Configuration"
caption="Configuration"
/%}
## Channel Configuration
Each sync process is managed through a **channel**.
You can define multiple channels for different services or metadata types.
| Field | Description |
|:--------------------|:------------|
| **Channel Name** | A name for identifying the sync channel. |
| **Filter** | Use the UI Query Filter Builder to define the scope of the metadata updates. You can filter by properties such as service, schema, database, owner, domain, or custom attributes. |
| **Update Descriptions** | Enable to sync updated entity descriptions from Collate to the source. |
| **Update Owners** | Enable to sync owner assignments from Collate. |
| **Update Tags** | Enable to sync tag assignments (e.g., PII) to the source system. |
| **SQL Template** | Optional. Specify a custom SQL template for updates. |
## Execution
Once the configuration is complete, the application can be executed in two ways:
- **Run Now:**
For manual execution.
- **Scheduled Run:**
For recurring, automated sync based on a defined schedule.
{% image
src="/images/v1.8/applications/reverse/reverse-metadata-application1.png"
alt="Scheduling"
caption="Scheduling"
/%}
[Know more about Reverse Metadata documentation](/connectors/ingestion/workflows/reverse-metadata)

View File

@ -728,6 +728,16 @@ site_menu:
- category: Collate AI / Collate AI Documentation Agent
url: /collateai/documentation-agent
- category: Applications
url: /applications
- category: Applications / Reverse Metadata Application
url: /applications/reverse-metadata
- category: Applications / Autopilot
url: /applications/autopilot
- category: Applications / Collate AI
url: /collateai
- category: How-to Guides
url: /how-to-guides

Binary file not shown.

After

Width:  |  Height:  |  Size: 284 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 266 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 260 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 442 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 201 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 284 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 266 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 260 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 442 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 201 KiB