Doc: Applications Addition (#20999)
91
openmetadata-docs/content/v1.7.x/applications/autopilot.md
Normal file
@ -0,0 +1,91 @@
|
||||
---
|
||||
title: Autopilot Application
|
||||
slug: /applications/autopilot
|
||||
collate: true
|
||||
---
|
||||
|
||||
# Autopilot Application
|
||||
|
||||
The **Autopilot Application** simplifies and accelerates the onboarding of new data services by automatically deploying and triggering essential ingestion workflows.
|
||||
With minimal input—just the service connection details—Autopilot sets up the entire metadata ingestion process using well-defined defaults.
|
||||
|
||||
## Overview
|
||||
|
||||
Autopilot is designed to help users get started quickly with metadata ingestion for newly added services by automating:
|
||||
|
||||
- **Metadata ingestion**
|
||||
- **Lineage extraction**
|
||||
- **Usage statistics collection**
|
||||
|
||||
> **Note:**
|
||||
> Workflows like **Profiler** and **Auto Classification** are created but not run automatically, allowing teams to review and selectively execute them based on resource and cost considerations.
|
||||
|
||||
## Key Benefits
|
||||
|
||||
- **Fast Setup:** Start ingesting metadata by simply defining the connection.
|
||||
- **Automated Deployment:** Automatically configures and deploys ingestion workflows.
|
||||
- **Editable Configurations:** All generated workflows can be reviewed and updated later.
|
||||
- **Integrated Insights:** Service insights are immediately available post-ingestion, highlighting areas that need further enrichment.
|
||||
|
||||
## Installation
|
||||
|
||||
1. Navigate to **Settings > Applications**.
|
||||
2. Click **Add Apps** and select **Autopilot** from the marketplace.
|
||||
|
||||
{% image
|
||||
src="/images/v1.7/applications/autopilot.png"
|
||||
alt="Select Autopilot Application"
|
||||
caption="Select Autopilot Application"
|
||||
/%}
|
||||
|
||||
3. Click **Install**.
|
||||
|
||||
{% image
|
||||
src="/images/v1.7/applications/autopilot1.png"
|
||||
alt="Install Autopilot"
|
||||
caption="Install Autopilot"
|
||||
/%}
|
||||
|
||||
## Configuration
|
||||
|
||||
Once installed, configure the application by filling in the following fields:
|
||||
|
||||
| Parameter | Description |
|
||||
|:----------------------|:------------|
|
||||
| **Application Type** | Set to `AutoPilotAppConfig`. |
|
||||
| **Active** | Enable to activate the workflow for the selected service. |
|
||||
| **Service Entity Link** | Link the service for which the ingestion workflows should be triggered. |
|
||||
|
||||
{% image
|
||||
src="/images/v1.7/applications/autopilot2.png"
|
||||
alt="Configuration"
|
||||
caption="Configuration"
|
||||
/%}
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Define Connection:**
|
||||
Create a new service (e.g., Snowflake) and test the connection.
|
||||
|
||||
2. **Apply Filters:**
|
||||
Add filter sets that will be inherited by all workflows.
|
||||
|
||||
3. **Autopilot Activation:**
|
||||
Once configured, Autopilot deploys and runs the following workflows:
|
||||
- Metadata Ingestion
|
||||
- Usage
|
||||
- Lineage
|
||||
|
||||
4. **Selective Execution:**
|
||||
Profiler and Auto Classification workflows are created but not executed automatically, giving you control over profiling scope and cost.
|
||||
|
||||
## Post-Ingestion Insights
|
||||
|
||||
After the initial run:
|
||||
|
||||
- Navigate to **Service Insights** to view KPIs such as:
|
||||
- Description coverage
|
||||
- PII detection
|
||||
- Data asset tiers
|
||||
|
||||
- Use the insights to identify areas that need enrichment—e.g., tagging, ownership, documentation.
|
25
openmetadata-docs/content/v1.7.x/applications/index.md
Normal file
@ -0,0 +1,25 @@
|
||||
---
|
||||
title: Collate Applications
|
||||
slug: /applications/
|
||||
collate: true
|
||||
---
|
||||
|
||||
Explore powerful Collate Applications to automate metadata management, streamline service onboarding, and enhance data intelligence.
|
||||
|
||||
{% tilesContainer %}
|
||||
{% tile
|
||||
title="Reverse Metadata"
|
||||
description="Seamlessly push metadata changes like descriptions, tags, and ownership from Collate back to your source systems, ensuring real-time metadata consistency."
|
||||
link="/applications/reverse-metadata"
|
||||
/ %}
|
||||
{% tile
|
||||
title="Autopilot"
|
||||
description="Automatically deploy and trigger metadata ingestion workflows when creating new services, accelerating service onboarding with minimal manual effort."
|
||||
link="/applications/autopilot"
|
||||
/ %}
|
||||
{% tile
|
||||
title="Collate AI"
|
||||
description="Unlock AI-powered metadata automation with Collate AI — enabling smart documentation, data quality validation, tiering, and natural language SQL generation."
|
||||
link="/collateai"
|
||||
/ %}
|
||||
{% /tilesContainer %}
|
@ -4,4 +4,82 @@ slug: /applications/reverse-metadata
|
||||
collate: true
|
||||
---
|
||||
|
||||
[Reverse Metadata documentation is available here](/connectors/ingestion/workflows/reverse-metadata)
|
||||
# Reverse Metadata Application
|
||||
|
||||
The **Reverse Metadata Application** enables seamless synchronization of metadata updates made in **Collate** back to the original data sources.
|
||||
This ensures that Collate remains the single source of truth for metadata while maintaining consistency across platforms and reinforcing governance policies.
|
||||
|
||||
## Overview
|
||||
|
||||
With this application, you can automatically propagate metadata changes such as:
|
||||
|
||||
- **Descriptions**
|
||||
- **Owners**
|
||||
- **Tags** (e.g., PII-sensitive classifications)
|
||||
|
||||
These updates are pushed directly to supported data source systems, including **Snowflake**, enabling real-time enforcement of data governance controls such as masking policies.
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Automated Metadata Propagation:**
|
||||
Sync metadata updates (tags, owners, descriptions) from Collate to source systems without manual intervention.
|
||||
|
||||
- **Configurable Channels:**
|
||||
Define multiple sync channels to target different services, asset types, or metadata types.
|
||||
|
||||
- **Custom SQL Templates:**
|
||||
Use SQL templates to customize update behavior per connector.
|
||||
|
||||
- **On-Demand or Scheduled Execution:**
|
||||
Run synchronization workflows manually or on a predefined schedule.
|
||||
|
||||
## Installation
|
||||
|
||||
1. Navigate to **Settings > Applications**.
|
||||
|
||||
{% image
|
||||
src="/images/v1.7/applications/autopilot.png"
|
||||
alt="Install Reverse Metadata Application"
|
||||
caption="Install Reverse Metadata Application"
|
||||
/%}
|
||||
|
||||
2. Click **Add Apps** and install the **Reverse Metadata Application**.
|
||||
3. After installation, configure the synchronization channels as described below.
|
||||
|
||||
{% image
|
||||
src="/images/v1.7/applications/reverse/reverse-metadata-application.png"
|
||||
alt="Configuration"
|
||||
caption="Configuration"
|
||||
/%}
|
||||
|
||||
## Channel Configuration
|
||||
|
||||
Each sync process is managed through a **channel**.
|
||||
You can define multiple channels for different services or metadata types.
|
||||
|
||||
| Field | Description |
|
||||
|:--------------------|:------------|
|
||||
| **Channel Name** | A name for identifying the sync channel. |
|
||||
| **Filter** | Use the UI Query Filter Builder to define the scope of the metadata updates. You can filter by properties such as service, schema, database, owner, domain, or custom attributes. |
|
||||
| **Update Descriptions** | Enable to sync updated entity descriptions from Collate to the source. |
|
||||
| **Update Owners** | Enable to sync owner assignments from Collate. |
|
||||
| **Update Tags** | Enable to sync tag assignments (e.g., PII) to the source system. |
|
||||
| **SQL Template** | Optional. Specify a custom SQL template for updates. |
|
||||
|
||||
## Execution
|
||||
|
||||
Once the configuration is complete, the application can be executed in two ways:
|
||||
|
||||
- **Run Now:**
|
||||
For manual execution.
|
||||
|
||||
- **Scheduled Run:**
|
||||
For recurring, automated sync based on a defined schedule.
|
||||
|
||||
{% image
|
||||
src="/images/v1.7/applications/reverse/reverse-metadata-application1.png"
|
||||
alt="Scheduling"
|
||||
caption="Scheduling"
|
||||
/%}
|
||||
|
||||
[Know more about Reverse Metadata documentation](/connectors/ingestion/workflows/reverse-metadata)
|
||||
|
@ -728,6 +728,16 @@ site_menu:
|
||||
- category: Collate AI / Collate AI Documentation Agent
|
||||
url: /collateai/documentation-agent
|
||||
|
||||
- category: Applications
|
||||
url: /applications
|
||||
|
||||
- category: Applications / Reverse Metadata Application
|
||||
url: /applications/reverse-metadata
|
||||
- category: Applications / Autopilot
|
||||
url: /applications/autopilot
|
||||
- category: Applications / Collate AI
|
||||
url: /collateai
|
||||
|
||||
- category: How-to Guides
|
||||
url: /how-to-guides
|
||||
|
||||
|
@ -0,0 +1,91 @@
|
||||
---
|
||||
title: Autopilot Application
|
||||
slug: /applications/autopilot
|
||||
collate: true
|
||||
---
|
||||
|
||||
# Autopilot Application
|
||||
|
||||
The **Autopilot Application** simplifies and accelerates the onboarding of new data services by automatically deploying and triggering essential ingestion workflows.
|
||||
With minimal input—just the service connection details—Autopilot sets up the entire metadata ingestion process using well-defined defaults.
|
||||
|
||||
## Overview
|
||||
|
||||
Autopilot is designed to help users get started quickly with metadata ingestion for newly added services by automating:
|
||||
|
||||
- **Metadata ingestion**
|
||||
- **Lineage extraction**
|
||||
- **Usage statistics collection**
|
||||
|
||||
> **Note:**
|
||||
> Workflows like **Profiler** and **Auto Classification** are created but not run automatically, allowing teams to review and selectively execute them based on resource and cost considerations.
|
||||
|
||||
## Key Benefits
|
||||
|
||||
- **Fast Setup:** Start ingesting metadata by simply defining the connection.
|
||||
- **Automated Deployment:** Automatically configures and deploys ingestion workflows.
|
||||
- **Editable Configurations:** All generated workflows can be reviewed and updated later.
|
||||
- **Integrated Insights:** Service insights are immediately available post-ingestion, highlighting areas that need further enrichment.
|
||||
|
||||
## Installation
|
||||
|
||||
1. Navigate to **Settings > Applications**.
|
||||
2. Click **Add Apps** and select **Autopilot** from the marketplace.
|
||||
|
||||
{% image
|
||||
src="/images/v1.8/applications/autopilot.png"
|
||||
alt="Select Autopilot Application"
|
||||
caption="Select Autopilot Application"
|
||||
/%}
|
||||
|
||||
3. Click **Install**.
|
||||
|
||||
{% image
|
||||
src="/images/v1.8/applications/autopilot1.png"
|
||||
alt="Install Autopilot"
|
||||
caption="Install Autopilot"
|
||||
/%}
|
||||
|
||||
## Configuration
|
||||
|
||||
Once installed, configure the application by filling in the following fields:
|
||||
|
||||
| Parameter | Description |
|
||||
|:----------------------|:------------|
|
||||
| **Application Type** | Set to `AutoPilotAppConfig`. |
|
||||
| **Active** | Enable to activate the workflow for the selected service. |
|
||||
| **Service Entity Link** | Link the service for which the ingestion workflows should be triggered. |
|
||||
|
||||
{% image
|
||||
src="/images/v1.8/applications/autopilot2.png"
|
||||
alt="Configuration"
|
||||
caption="Configuration"
|
||||
/%}
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Define Connection:**
|
||||
Create a new service (e.g., Snowflake) and test the connection.
|
||||
|
||||
2. **Apply Filters:**
|
||||
Add filter sets that will be inherited by all workflows.
|
||||
|
||||
3. **Autopilot Activation:**
|
||||
Once configured, Autopilot deploys and runs the following workflows:
|
||||
- Metadata Ingestion
|
||||
- Usage
|
||||
- Lineage
|
||||
|
||||
4. **Selective Execution:**
|
||||
Profiler and Auto Classification workflows are created but not executed automatically, giving you control over profiling scope and cost.
|
||||
|
||||
## Post-Ingestion Insights
|
||||
|
||||
After the initial run:
|
||||
|
||||
- Navigate to **Service Insights** to view KPIs such as:
|
||||
- Description coverage
|
||||
- PII detection
|
||||
- Data asset tiers
|
||||
|
||||
- Use the insights to identify areas that need enrichment—e.g., tagging, ownership, documentation.
|
@ -0,0 +1,25 @@
|
||||
---
|
||||
title: Collate Applications
|
||||
slug: /applications/
|
||||
collate: true
|
||||
---
|
||||
|
||||
Explore powerful Collate Applications to automate metadata management, streamline service onboarding, and enhance data intelligence.
|
||||
|
||||
{% tilesContainer %}
|
||||
{% tile
|
||||
title="Reverse Metadata"
|
||||
description="Seamlessly push metadata changes like descriptions, tags, and ownership from Collate back to your source systems, ensuring real-time metadata consistency."
|
||||
link="/applications/reverse-metadata"
|
||||
/ %}
|
||||
{% tile
|
||||
title="Autopilot"
|
||||
description="Automatically deploy and trigger metadata ingestion workflows when creating new services, accelerating service onboarding with minimal manual effort."
|
||||
link="/applications/autopilot"
|
||||
/ %}
|
||||
{% tile
|
||||
title="Collate AI"
|
||||
description="Unlock AI-powered metadata automation with Collate AI — enabling smart documentation, data quality validation, tiering, and natural language SQL generation."
|
||||
link="/collateai"
|
||||
/ %}
|
||||
{% /tilesContainer %}
|
@ -0,0 +1,85 @@
|
||||
---
|
||||
title: Reverse Metadata
|
||||
slug: /applications/reverse-metadata
|
||||
collate: true
|
||||
---
|
||||
|
||||
# Reverse Metadata Application
|
||||
|
||||
The **Reverse Metadata Application** enables seamless synchronization of metadata updates made in **Collate** back to the original data sources.
|
||||
This ensures that Collate remains the single source of truth for metadata while maintaining consistency across platforms and reinforcing governance policies.
|
||||
|
||||
## Overview
|
||||
|
||||
With this application, you can automatically propagate metadata changes such as:
|
||||
|
||||
- **Descriptions**
|
||||
- **Owners**
|
||||
- **Tags** (e.g., PII-sensitive classifications)
|
||||
|
||||
These updates are pushed directly to supported data source systems, including **Snowflake**, enabling real-time enforcement of data governance controls such as masking policies.
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Automated Metadata Propagation:**
|
||||
Sync metadata updates (tags, owners, descriptions) from Collate to source systems without manual intervention.
|
||||
|
||||
- **Configurable Channels:**
|
||||
Define multiple sync channels to target different services, asset types, or metadata types.
|
||||
|
||||
- **Custom SQL Templates:**
|
||||
Use SQL templates to customize update behavior per connector.
|
||||
|
||||
- **On-Demand or Scheduled Execution:**
|
||||
Run synchronization workflows manually or on a predefined schedule.
|
||||
|
||||
## Installation
|
||||
|
||||
1. Navigate to **Settings > Applications**.
|
||||
|
||||
{% image
|
||||
src="/images/v1.8/applications/autopilot.png"
|
||||
alt="Install Reverse Metadata Application"
|
||||
caption="Install Reverse Metadata Application"
|
||||
/%}
|
||||
|
||||
2. Click **Add Apps** and install the **Reverse Metadata Application**.
|
||||
3. After installation, configure the synchronization channels as described below.
|
||||
|
||||
{% image
|
||||
src="/images/v1.8/applications/reverse/reverse-metadata-application.png"
|
||||
alt="Configuration"
|
||||
caption="Configuration"
|
||||
/%}
|
||||
|
||||
## Channel Configuration
|
||||
|
||||
Each sync process is managed through a **channel**.
|
||||
You can define multiple channels for different services or metadata types.
|
||||
|
||||
| Field | Description |
|
||||
|:--------------------|:------------|
|
||||
| **Channel Name** | A name for identifying the sync channel. |
|
||||
| **Filter** | Use the UI Query Filter Builder to define the scope of the metadata updates. You can filter by properties such as service, schema, database, owner, domain, or custom attributes. |
|
||||
| **Update Descriptions** | Enable to sync updated entity descriptions from Collate to the source. |
|
||||
| **Update Owners** | Enable to sync owner assignments from Collate. |
|
||||
| **Update Tags** | Enable to sync tag assignments (e.g., PII) to the source system. |
|
||||
| **SQL Template** | Optional. Specify a custom SQL template for updates. |
|
||||
|
||||
## Execution
|
||||
|
||||
Once the configuration is complete, the application can be executed in two ways:
|
||||
|
||||
- **Run Now:**
|
||||
For manual execution.
|
||||
|
||||
- **Scheduled Run:**
|
||||
For recurring, automated sync based on a defined schedule.
|
||||
|
||||
{% image
|
||||
src="/images/v1.8/applications/reverse/reverse-metadata-application1.png"
|
||||
alt="Scheduling"
|
||||
caption="Scheduling"
|
||||
/%}
|
||||
|
||||
[Know more about Reverse Metadata documentation](/connectors/ingestion/workflows/reverse-metadata)
|
@ -728,6 +728,16 @@ site_menu:
|
||||
- category: Collate AI / Collate AI Documentation Agent
|
||||
url: /collateai/documentation-agent
|
||||
|
||||
- category: Applications
|
||||
url: /applications
|
||||
|
||||
- category: Applications / Reverse Metadata Application
|
||||
url: /applications/reverse-metadata
|
||||
- category: Applications / Autopilot
|
||||
url: /applications/autopilot
|
||||
- category: Applications / Collate AI
|
||||
url: /collateai
|
||||
|
||||
- category: How-to Guides
|
||||
url: /how-to-guides
|
||||
|
||||
|
BIN
openmetadata-docs/images/v1.7/applications/autopilot.png
Normal file
After Width: | Height: | Size: 284 KiB |
BIN
openmetadata-docs/images/v1.7/applications/autopilot1.png
Normal file
After Width: | Height: | Size: 266 KiB |
BIN
openmetadata-docs/images/v1.7/applications/autopilot2.png
Normal file
After Width: | Height: | Size: 260 KiB |
After Width: | Height: | Size: 442 KiB |
After Width: | Height: | Size: 201 KiB |
BIN
openmetadata-docs/images/v1.8/applications/autopilot.png
Normal file
After Width: | Height: | Size: 284 KiB |
BIN
openmetadata-docs/images/v1.8/applications/autopilot1.png
Normal file
After Width: | Height: | Size: 266 KiB |
BIN
openmetadata-docs/images/v1.8/applications/autopilot2.png
Normal file
After Width: | Height: | Size: 260 KiB |
After Width: | Height: | Size: 442 KiB |
After Width: | Height: | Size: 201 KiB |