Doc: Applications Addition (#20999)
91
openmetadata-docs/content/v1.7.x/applications/autopilot.md
Normal file
@ -0,0 +1,91 @@
|
|||||||
|
---
|
||||||
|
title: Autopilot Application
|
||||||
|
slug: /applications/autopilot
|
||||||
|
collate: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Autopilot Application
|
||||||
|
|
||||||
|
The **Autopilot Application** simplifies and accelerates the onboarding of new data services by automatically deploying and triggering essential ingestion workflows.
|
||||||
|
With minimal input—just the service connection details—Autopilot sets up the entire metadata ingestion process using well-defined defaults.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Autopilot is designed to help users get started quickly with metadata ingestion for newly added services by automating:
|
||||||
|
|
||||||
|
- **Metadata ingestion**
|
||||||
|
- **Lineage extraction**
|
||||||
|
- **Usage statistics collection**
|
||||||
|
|
||||||
|
> **Note:**
|
||||||
|
> Workflows like **Profiler** and **Auto Classification** are created but not run automatically, allowing teams to review and selectively execute them based on resource and cost considerations.
|
||||||
|
|
||||||
|
## Key Benefits
|
||||||
|
|
||||||
|
- **Fast Setup:** Start ingesting metadata by simply defining the connection.
|
||||||
|
- **Automated Deployment:** Automatically configures and deploys ingestion workflows.
|
||||||
|
- **Editable Configurations:** All generated workflows can be reviewed and updated later.
|
||||||
|
- **Integrated Insights:** Service insights are immediately available post-ingestion, highlighting areas that need further enrichment.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Navigate to **Settings > Applications**.
|
||||||
|
2. Click **Add Apps** and select **Autopilot** from the marketplace.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.7/applications/autopilot.png"
|
||||||
|
alt="Select Autopilot Application"
|
||||||
|
caption="Select Autopilot Application"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
3. Click **Install**.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.7/applications/autopilot1.png"
|
||||||
|
alt="Install Autopilot"
|
||||||
|
caption="Install Autopilot"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Once installed, configure the application by filling in the following fields:
|
||||||
|
|
||||||
|
| Parameter | Description |
|
||||||
|
|:----------------------|:------------|
|
||||||
|
| **Application Type** | Set to `AutoPilotAppConfig`. |
|
||||||
|
| **Active** | Enable to activate the workflow for the selected service. |
|
||||||
|
| **Service Entity Link** | Link the service for which the ingestion workflows should be triggered. |
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.7/applications/autopilot2.png"
|
||||||
|
alt="Configuration"
|
||||||
|
caption="Configuration"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
1. **Define Connection:**
|
||||||
|
Create a new service (e.g., Snowflake) and test the connection.
|
||||||
|
|
||||||
|
2. **Apply Filters:**
|
||||||
|
Add filter sets that will be inherited by all workflows.
|
||||||
|
|
||||||
|
3. **Autopilot Activation:**
|
||||||
|
Once configured, Autopilot deploys and runs the following workflows:
|
||||||
|
- Metadata Ingestion
|
||||||
|
- Usage
|
||||||
|
- Lineage
|
||||||
|
|
||||||
|
4. **Selective Execution:**
|
||||||
|
Profiler and Auto Classification workflows are created but not executed automatically, giving you control over profiling scope and cost.
|
||||||
|
|
||||||
|
## Post-Ingestion Insights
|
||||||
|
|
||||||
|
After the initial run:
|
||||||
|
|
||||||
|
- Navigate to **Service Insights** to view KPIs such as:
|
||||||
|
- Description coverage
|
||||||
|
- PII detection
|
||||||
|
- Data asset tiers
|
||||||
|
|
||||||
|
- Use the insights to identify areas that need enrichment—e.g., tagging, ownership, documentation.
|
25
openmetadata-docs/content/v1.7.x/applications/index.md
Normal file
@ -0,0 +1,25 @@
|
|||||||
|
---
|
||||||
|
title: Collate Applications
|
||||||
|
slug: /applications/
|
||||||
|
collate: true
|
||||||
|
---
|
||||||
|
|
||||||
|
Explore powerful Collate Applications to automate metadata management, streamline service onboarding, and enhance data intelligence.
|
||||||
|
|
||||||
|
{% tilesContainer %}
|
||||||
|
{% tile
|
||||||
|
title="Reverse Metadata"
|
||||||
|
description="Seamlessly push metadata changes like descriptions, tags, and ownership from Collate back to your source systems, ensuring real-time metadata consistency."
|
||||||
|
link="/applications/reverse-metadata"
|
||||||
|
/ %}
|
||||||
|
{% tile
|
||||||
|
title="Autopilot"
|
||||||
|
description="Automatically deploy and trigger metadata ingestion workflows when creating new services, accelerating service onboarding with minimal manual effort."
|
||||||
|
link="/applications/autopilot"
|
||||||
|
/ %}
|
||||||
|
{% tile
|
||||||
|
title="Collate AI"
|
||||||
|
description="Unlock AI-powered metadata automation with Collate AI — enabling smart documentation, data quality validation, tiering, and natural language SQL generation."
|
||||||
|
link="/collateai"
|
||||||
|
/ %}
|
||||||
|
{% /tilesContainer %}
|
@ -4,4 +4,82 @@ slug: /applications/reverse-metadata
|
|||||||
collate: true
|
collate: true
|
||||||
---
|
---
|
||||||
|
|
||||||
[Reverse Metadata documentation is available here](/connectors/ingestion/workflows/reverse-metadata)
|
# Reverse Metadata Application
|
||||||
|
|
||||||
|
The **Reverse Metadata Application** enables seamless synchronization of metadata updates made in **Collate** back to the original data sources.
|
||||||
|
This ensures that Collate remains the single source of truth for metadata while maintaining consistency across platforms and reinforcing governance policies.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
With this application, you can automatically propagate metadata changes such as:
|
||||||
|
|
||||||
|
- **Descriptions**
|
||||||
|
- **Owners**
|
||||||
|
- **Tags** (e.g., PII-sensitive classifications)
|
||||||
|
|
||||||
|
These updates are pushed directly to supported data source systems, including **Snowflake**, enabling real-time enforcement of data governance controls such as masking policies.
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
- **Automated Metadata Propagation:**
|
||||||
|
Sync metadata updates (tags, owners, descriptions) from Collate to source systems without manual intervention.
|
||||||
|
|
||||||
|
- **Configurable Channels:**
|
||||||
|
Define multiple sync channels to target different services, asset types, or metadata types.
|
||||||
|
|
||||||
|
- **Custom SQL Templates:**
|
||||||
|
Use SQL templates to customize update behavior per connector.
|
||||||
|
|
||||||
|
- **On-Demand or Scheduled Execution:**
|
||||||
|
Run synchronization workflows manually or on a predefined schedule.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Navigate to **Settings > Applications**.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.7/applications/autopilot.png"
|
||||||
|
alt="Install Reverse Metadata Application"
|
||||||
|
caption="Install Reverse Metadata Application"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
2. Click **Add Apps** and install the **Reverse Metadata Application**.
|
||||||
|
3. After installation, configure the synchronization channels as described below.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.7/applications/reverse/reverse-metadata-application.png"
|
||||||
|
alt="Configuration"
|
||||||
|
caption="Configuration"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
## Channel Configuration
|
||||||
|
|
||||||
|
Each sync process is managed through a **channel**.
|
||||||
|
You can define multiple channels for different services or metadata types.
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
|:--------------------|:------------|
|
||||||
|
| **Channel Name** | A name for identifying the sync channel. |
|
||||||
|
| **Filter** | Use the UI Query Filter Builder to define the scope of the metadata updates. You can filter by properties such as service, schema, database, owner, domain, or custom attributes. |
|
||||||
|
| **Update Descriptions** | Enable to sync updated entity descriptions from Collate to the source. |
|
||||||
|
| **Update Owners** | Enable to sync owner assignments from Collate. |
|
||||||
|
| **Update Tags** | Enable to sync tag assignments (e.g., PII) to the source system. |
|
||||||
|
| **SQL Template** | Optional. Specify a custom SQL template for updates. |
|
||||||
|
|
||||||
|
## Execution
|
||||||
|
|
||||||
|
Once the configuration is complete, the application can be executed in two ways:
|
||||||
|
|
||||||
|
- **Run Now:**
|
||||||
|
For manual execution.
|
||||||
|
|
||||||
|
- **Scheduled Run:**
|
||||||
|
For recurring, automated sync based on a defined schedule.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.7/applications/reverse/reverse-metadata-application1.png"
|
||||||
|
alt="Scheduling"
|
||||||
|
caption="Scheduling"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
[Know more about Reverse Metadata documentation](/connectors/ingestion/workflows/reverse-metadata)
|
||||||
|
@ -728,6 +728,16 @@ site_menu:
|
|||||||
- category: Collate AI / Collate AI Documentation Agent
|
- category: Collate AI / Collate AI Documentation Agent
|
||||||
url: /collateai/documentation-agent
|
url: /collateai/documentation-agent
|
||||||
|
|
||||||
|
- category: Applications
|
||||||
|
url: /applications
|
||||||
|
|
||||||
|
- category: Applications / Reverse Metadata Application
|
||||||
|
url: /applications/reverse-metadata
|
||||||
|
- category: Applications / Autopilot
|
||||||
|
url: /applications/autopilot
|
||||||
|
- category: Applications / Collate AI
|
||||||
|
url: /collateai
|
||||||
|
|
||||||
- category: How-to Guides
|
- category: How-to Guides
|
||||||
url: /how-to-guides
|
url: /how-to-guides
|
||||||
|
|
||||||
|
@ -0,0 +1,91 @@
|
|||||||
|
---
|
||||||
|
title: Autopilot Application
|
||||||
|
slug: /applications/autopilot
|
||||||
|
collate: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Autopilot Application
|
||||||
|
|
||||||
|
The **Autopilot Application** simplifies and accelerates the onboarding of new data services by automatically deploying and triggering essential ingestion workflows.
|
||||||
|
With minimal input—just the service connection details—Autopilot sets up the entire metadata ingestion process using well-defined defaults.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Autopilot is designed to help users get started quickly with metadata ingestion for newly added services by automating:
|
||||||
|
|
||||||
|
- **Metadata ingestion**
|
||||||
|
- **Lineage extraction**
|
||||||
|
- **Usage statistics collection**
|
||||||
|
|
||||||
|
> **Note:**
|
||||||
|
> Workflows like **Profiler** and **Auto Classification** are created but not run automatically, allowing teams to review and selectively execute them based on resource and cost considerations.
|
||||||
|
|
||||||
|
## Key Benefits
|
||||||
|
|
||||||
|
- **Fast Setup:** Start ingesting metadata by simply defining the connection.
|
||||||
|
- **Automated Deployment:** Automatically configures and deploys ingestion workflows.
|
||||||
|
- **Editable Configurations:** All generated workflows can be reviewed and updated later.
|
||||||
|
- **Integrated Insights:** Service insights are immediately available post-ingestion, highlighting areas that need further enrichment.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Navigate to **Settings > Applications**.
|
||||||
|
2. Click **Add Apps** and select **Autopilot** from the marketplace.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.8/applications/autopilot.png"
|
||||||
|
alt="Select Autopilot Application"
|
||||||
|
caption="Select Autopilot Application"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
3. Click **Install**.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.8/applications/autopilot1.png"
|
||||||
|
alt="Install Autopilot"
|
||||||
|
caption="Install Autopilot"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Once installed, configure the application by filling in the following fields:
|
||||||
|
|
||||||
|
| Parameter | Description |
|
||||||
|
|:----------------------|:------------|
|
||||||
|
| **Application Type** | Set to `AutoPilotAppConfig`. |
|
||||||
|
| **Active** | Enable to activate the workflow for the selected service. |
|
||||||
|
| **Service Entity Link** | Link the service for which the ingestion workflows should be triggered. |
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.8/applications/autopilot2.png"
|
||||||
|
alt="Configuration"
|
||||||
|
caption="Configuration"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
1. **Define Connection:**
|
||||||
|
Create a new service (e.g., Snowflake) and test the connection.
|
||||||
|
|
||||||
|
2. **Apply Filters:**
|
||||||
|
Add filter sets that will be inherited by all workflows.
|
||||||
|
|
||||||
|
3. **Autopilot Activation:**
|
||||||
|
Once configured, Autopilot deploys and runs the following workflows:
|
||||||
|
- Metadata Ingestion
|
||||||
|
- Usage
|
||||||
|
- Lineage
|
||||||
|
|
||||||
|
4. **Selective Execution:**
|
||||||
|
Profiler and Auto Classification workflows are created but not executed automatically, giving you control over profiling scope and cost.
|
||||||
|
|
||||||
|
## Post-Ingestion Insights
|
||||||
|
|
||||||
|
After the initial run:
|
||||||
|
|
||||||
|
- Navigate to **Service Insights** to view KPIs such as:
|
||||||
|
- Description coverage
|
||||||
|
- PII detection
|
||||||
|
- Data asset tiers
|
||||||
|
|
||||||
|
- Use the insights to identify areas that need enrichment—e.g., tagging, ownership, documentation.
|
@ -0,0 +1,25 @@
|
|||||||
|
---
|
||||||
|
title: Collate Applications
|
||||||
|
slug: /applications/
|
||||||
|
collate: true
|
||||||
|
---
|
||||||
|
|
||||||
|
Explore powerful Collate Applications to automate metadata management, streamline service onboarding, and enhance data intelligence.
|
||||||
|
|
||||||
|
{% tilesContainer %}
|
||||||
|
{% tile
|
||||||
|
title="Reverse Metadata"
|
||||||
|
description="Seamlessly push metadata changes like descriptions, tags, and ownership from Collate back to your source systems, ensuring real-time metadata consistency."
|
||||||
|
link="/applications/reverse-metadata"
|
||||||
|
/ %}
|
||||||
|
{% tile
|
||||||
|
title="Autopilot"
|
||||||
|
description="Automatically deploy and trigger metadata ingestion workflows when creating new services, accelerating service onboarding with minimal manual effort."
|
||||||
|
link="/applications/autopilot"
|
||||||
|
/ %}
|
||||||
|
{% tile
|
||||||
|
title="Collate AI"
|
||||||
|
description="Unlock AI-powered metadata automation with Collate AI — enabling smart documentation, data quality validation, tiering, and natural language SQL generation."
|
||||||
|
link="/collateai"
|
||||||
|
/ %}
|
||||||
|
{% /tilesContainer %}
|
@ -0,0 +1,85 @@
|
|||||||
|
---
|
||||||
|
title: Reverse Metadata
|
||||||
|
slug: /applications/reverse-metadata
|
||||||
|
collate: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Reverse Metadata Application
|
||||||
|
|
||||||
|
The **Reverse Metadata Application** enables seamless synchronization of metadata updates made in **Collate** back to the original data sources.
|
||||||
|
This ensures that Collate remains the single source of truth for metadata while maintaining consistency across platforms and reinforcing governance policies.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
With this application, you can automatically propagate metadata changes such as:
|
||||||
|
|
||||||
|
- **Descriptions**
|
||||||
|
- **Owners**
|
||||||
|
- **Tags** (e.g., PII-sensitive classifications)
|
||||||
|
|
||||||
|
These updates are pushed directly to supported data source systems, including **Snowflake**, enabling real-time enforcement of data governance controls such as masking policies.
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
- **Automated Metadata Propagation:**
|
||||||
|
Sync metadata updates (tags, owners, descriptions) from Collate to source systems without manual intervention.
|
||||||
|
|
||||||
|
- **Configurable Channels:**
|
||||||
|
Define multiple sync channels to target different services, asset types, or metadata types.
|
||||||
|
|
||||||
|
- **Custom SQL Templates:**
|
||||||
|
Use SQL templates to customize update behavior per connector.
|
||||||
|
|
||||||
|
- **On-Demand or Scheduled Execution:**
|
||||||
|
Run synchronization workflows manually or on a predefined schedule.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Navigate to **Settings > Applications**.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.8/applications/autopilot.png"
|
||||||
|
alt="Install Reverse Metadata Application"
|
||||||
|
caption="Install Reverse Metadata Application"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
2. Click **Add Apps** and install the **Reverse Metadata Application**.
|
||||||
|
3. After installation, configure the synchronization channels as described below.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.8/applications/reverse/reverse-metadata-application.png"
|
||||||
|
alt="Configuration"
|
||||||
|
caption="Configuration"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
## Channel Configuration
|
||||||
|
|
||||||
|
Each sync process is managed through a **channel**.
|
||||||
|
You can define multiple channels for different services or metadata types.
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
|:--------------------|:------------|
|
||||||
|
| **Channel Name** | A name for identifying the sync channel. |
|
||||||
|
| **Filter** | Use the UI Query Filter Builder to define the scope of the metadata updates. You can filter by properties such as service, schema, database, owner, domain, or custom attributes. |
|
||||||
|
| **Update Descriptions** | Enable to sync updated entity descriptions from Collate to the source. |
|
||||||
|
| **Update Owners** | Enable to sync owner assignments from Collate. |
|
||||||
|
| **Update Tags** | Enable to sync tag assignments (e.g., PII) to the source system. |
|
||||||
|
| **SQL Template** | Optional. Specify a custom SQL template for updates. |
|
||||||
|
|
||||||
|
## Execution
|
||||||
|
|
||||||
|
Once the configuration is complete, the application can be executed in two ways:
|
||||||
|
|
||||||
|
- **Run Now:**
|
||||||
|
For manual execution.
|
||||||
|
|
||||||
|
- **Scheduled Run:**
|
||||||
|
For recurring, automated sync based on a defined schedule.
|
||||||
|
|
||||||
|
{% image
|
||||||
|
src="/images/v1.8/applications/reverse/reverse-metadata-application1.png"
|
||||||
|
alt="Scheduling"
|
||||||
|
caption="Scheduling"
|
||||||
|
/%}
|
||||||
|
|
||||||
|
[Know more about Reverse Metadata documentation](/connectors/ingestion/workflows/reverse-metadata)
|
@ -728,6 +728,16 @@ site_menu:
|
|||||||
- category: Collate AI / Collate AI Documentation Agent
|
- category: Collate AI / Collate AI Documentation Agent
|
||||||
url: /collateai/documentation-agent
|
url: /collateai/documentation-agent
|
||||||
|
|
||||||
|
- category: Applications
|
||||||
|
url: /applications
|
||||||
|
|
||||||
|
- category: Applications / Reverse Metadata Application
|
||||||
|
url: /applications/reverse-metadata
|
||||||
|
- category: Applications / Autopilot
|
||||||
|
url: /applications/autopilot
|
||||||
|
- category: Applications / Collate AI
|
||||||
|
url: /collateai
|
||||||
|
|
||||||
- category: How-to Guides
|
- category: How-to Guides
|
||||||
url: /how-to-guides
|
url: /how-to-guides
|
||||||
|
|
||||||
|
BIN
openmetadata-docs/images/v1.7/applications/autopilot.png
Normal file
After Width: | Height: | Size: 284 KiB |
BIN
openmetadata-docs/images/v1.7/applications/autopilot1.png
Normal file
After Width: | Height: | Size: 266 KiB |
BIN
openmetadata-docs/images/v1.7/applications/autopilot2.png
Normal file
After Width: | Height: | Size: 260 KiB |
After Width: | Height: | Size: 442 KiB |
After Width: | Height: | Size: 201 KiB |
BIN
openmetadata-docs/images/v1.8/applications/autopilot.png
Normal file
After Width: | Height: | Size: 284 KiB |
BIN
openmetadata-docs/images/v1.8/applications/autopilot1.png
Normal file
After Width: | Height: | Size: 266 KiB |
BIN
openmetadata-docs/images/v1.8/applications/autopilot2.png
Normal file
After Width: | Height: | Size: 260 KiB |
After Width: | Height: | Size: 442 KiB |
After Width: | Height: | Size: 201 KiB |