diff --git a/openmetadata-docs/content/v1.7.x/applications/data-retention.md b/openmetadata-docs/content/v1.7.x/applications/data-retention.md new file mode 100644 index 00000000000..5d9655cb98a --- /dev/null +++ b/openmetadata-docs/content/v1.7.x/applications/data-retention.md @@ -0,0 +1,83 @@ +--- +title: Data Retention Application +slug: /applications/data-retention +collate: true +--- + +# Data Retention Application + +The **Data Retention** application in **OpenMetadata** automates the cleanup of the internal database to maintain long-term performance and prevent data bloat. It provides administrators with an efficient mechanism to define retention policies for event-related records, ensuring scalability and compliance with data management practices. + +## Overview + +As **OpenMetadata** processes events related to metadata changes and pipeline executions, it stores these events in internal tables. Over time, these tables can grow significantly and affect system performance. The **Data Retention App** enables automated and scheduled cleanup of outdated records. + +### Key Features + +- Automated cleanup of outdated internal records. +- Configurable retention periods for different data categories. +- Helps maintain system performance and database efficiency. +- Supports compliance with organizational data retention policies. + +## Configuration Options + +- **Configuration Key:** `changeEventRetentionPeriod` + **Description:** Number of days to retain change event records + **Default Value:** `7` + +{% note %} + +The retention period is applied to each cleanup target based on its timestamp. + +{% /note %} + +## Cleanup Targets + +The app deletes old records from the following internal tables: + +- **change_events:** Stores all metadata change event logs. +- **successful_sent_change_events:** Tracks successfully processed events. +- **consumers_dlq:** Dead Letter Queue for failed events. + +## Implementation Details + +- **Schedule:** Runs weekly by default. +- **Cron:** `0 0 * * 0` (every Sunday at midnight). +- **Batch Processing:** Cleans up records in batches of 10,000. +- **Triggering:** Can be executed manually or allowed to run on schedule. +- **Application Class:** `org.openmetadata.service.apps.bundles.dataRetention.DataRetention` +- **Permission Requirement:** Requires admin privileges to configure or trigger. + +## How to Access + +1. Go to **Settings > Applications** and then click on **Add apps**. + +{% image +src="/images/v1.7/applications/ret.png" +alt="Install Data Retention Application" +caption="Install Data Retention Application" +/%} + +2. Select or install the **Data Retention** application. + +{% image +src="/images/v1.7/applications/ret1.png" +alt="Install Data Retention Application" +caption="Install Data Retention Application" +/%} + +3. Configure the desired retention period. + +{% image +src="/images/v1.7/applications/ret2.png" +alt="configuration" +caption="Configuration" +/%} + +4. Run manually or let it operate on its default schedule. + +{% image +src="/images/v1.7/applications/ret3.png" +alt="scheduling" +caption="scheduling" +/%} diff --git a/openmetadata-docs/content/v1.7.x/collate-menu.md b/openmetadata-docs/content/v1.7.x/collate-menu.md index fdfae9bd8ea..75fa3438603 100644 --- a/openmetadata-docs/content/v1.7.x/collate-menu.md +++ b/openmetadata-docs/content/v1.7.x/collate-menu.md @@ -735,6 +735,8 @@ site_menu: url: /applications/reverse-metadata - category: Applications / Autopilot url: /applications/autopilot + - category: Applications / Data Retention Application + url: /applications/data-retention - category: How-to Guides url: /how-to-guides diff --git a/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/data-retention.md b/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/data-retention.md new file mode 100644 index 00000000000..cd00d48624a --- /dev/null +++ b/openmetadata-docs/content/v1.8.x-SNAPSHOT/applications/data-retention.md @@ -0,0 +1,83 @@ +--- +title: Data Retention Application +slug: /applications/data-retention +collate: true +--- + +# Data Retention Application + +The **Data Retention** application in **OpenMetadata** automates the cleanup of the internal database to maintain long-term performance and prevent data bloat. It provides administrators with an efficient mechanism to define retention policies for event-related records, ensuring scalability and compliance with data management practices. + +## Overview + +As **OpenMetadata** processes events related to metadata changes and pipeline executions, it stores these events in internal tables. Over time, these tables can grow significantly and affect system performance. The **Data Retention App** enables automated and scheduled cleanup of outdated records. + +### Key Features + +- Automated cleanup of outdated internal records. +- Configurable retention periods for different data categories. +- Helps maintain system performance and database efficiency. +- Supports compliance with organizational data retention policies. + +## Configuration Options + +- **Configuration Key:** `changeEventRetentionPeriod` + **Description:** Number of days to retain change event records + **Default Value:** `7` + +{% note %} + +The retention period is applied to each cleanup target based on its timestamp. + +{% /note %} + +## Cleanup Targets + +The app deletes old records from the following internal tables: + +- **change_events:** Stores all metadata change event logs. +- **successful_sent_change_events:** Tracks successfully processed events. +- **consumers_dlq:** Dead Letter Queue for failed events. + +## Implementation Details + +- **Schedule:** Runs weekly by default. +- **Cron:** `0 0 * * 0` (every Sunday at midnight). +- **Batch Processing:** Cleans up records in batches of 10,000. +- **Triggering:** Can be executed manually or allowed to run on schedule. +- **Application Class:** `org.openmetadata.service.apps.bundles.dataRetention.DataRetention` +- **Permission Requirement:** Requires admin privileges to configure or trigger. + +## How to Access + +1. Go to **Settings > Applications** and then click on **Add apps**. + +{% image +src="/images/v1.8/applications/ret.png" +alt="Install Data Retention Application" +caption="Install Data Retention Application" +/%} + +2. Select or install the **Data Retention** application. + +{% image +src="/images/v1.8/applications/ret1.png" +alt="Install Data Retention Application" +caption="Install Data Retention Application" +/%} + +3. Configure the desired retention period. + +{% image +src="/images/v1.8/applications/ret2.png" +alt="configuration" +caption="Configuration" +/%} + +4. Run manually or let it operate on its default schedule. + +{% image +src="/images/v1.8/applications/ret3.png" +alt="scheduling" +caption="scheduling" +/%} diff --git a/openmetadata-docs/content/v1.8.x-SNAPSHOT/collate-menu.md b/openmetadata-docs/content/v1.8.x-SNAPSHOT/collate-menu.md index fdfae9bd8ea..75fa3438603 100644 --- a/openmetadata-docs/content/v1.8.x-SNAPSHOT/collate-menu.md +++ b/openmetadata-docs/content/v1.8.x-SNAPSHOT/collate-menu.md @@ -735,6 +735,8 @@ site_menu: url: /applications/reverse-metadata - category: Applications / Autopilot url: /applications/autopilot + - category: Applications / Data Retention Application + url: /applications/data-retention - category: How-to Guides url: /how-to-guides diff --git a/openmetadata-docs/images/v1.7/applications/ret.png b/openmetadata-docs/images/v1.7/applications/ret.png new file mode 100644 index 00000000000..f557196273f Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/ret.png differ diff --git a/openmetadata-docs/images/v1.7/applications/ret1.png b/openmetadata-docs/images/v1.7/applications/ret1.png new file mode 100644 index 00000000000..3751eee4a41 Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/ret1.png differ diff --git a/openmetadata-docs/images/v1.7/applications/ret2.png b/openmetadata-docs/images/v1.7/applications/ret2.png new file mode 100644 index 00000000000..e83f168e1d9 Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/ret2.png differ diff --git a/openmetadata-docs/images/v1.7/applications/ret3.png b/openmetadata-docs/images/v1.7/applications/ret3.png new file mode 100644 index 00000000000..3472169038c Binary files /dev/null and b/openmetadata-docs/images/v1.7/applications/ret3.png differ diff --git a/openmetadata-docs/images/v1.8/applications/ret.png b/openmetadata-docs/images/v1.8/applications/ret.png new file mode 100644 index 00000000000..f557196273f Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/ret.png differ diff --git a/openmetadata-docs/images/v1.8/applications/ret1.png b/openmetadata-docs/images/v1.8/applications/ret1.png new file mode 100644 index 00000000000..3751eee4a41 Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/ret1.png differ diff --git a/openmetadata-docs/images/v1.8/applications/ret2.png b/openmetadata-docs/images/v1.8/applications/ret2.png new file mode 100644 index 00000000000..e83f168e1d9 Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/ret2.png differ diff --git a/openmetadata-docs/images/v1.8/applications/ret3.png b/openmetadata-docs/images/v1.8/applications/ret3.png new file mode 100644 index 00000000000..3472169038c Binary files /dev/null and b/openmetadata-docs/images/v1.8/applications/ret3.png differ