yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-12-28 02:17:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(): Updating datahub cloud actions source docs to include support for MCL events (#14786)

Browse Source 
Co-authored-by: John Joyce <john@Mac.lan>
This commit is contained in:
John Joyce 2025-09-17 09:54:06 -07:00 committed by GitHub
parent 6073e20f92
commit 6fe831387b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 10 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
docs/actions/sources/datahub-cloud-event-source.md
View File

@ -39,8 +39,7 @@ If you've configured your Action pipeline `failure_mode` to be `THROW`, then eve
The DataHub Cloud Event Source produces
- [Entity Change Event V1](../../managed-datahub/datahub-api/entity-events-api.md)
Note that the DataHub Cloud Event Source does _not_ yet support the full [Metadata Change Log V1](../events/metadata-change-log-event.md) event stream.
- [Metadata Change Log V1](../events/metadata-change-log-event.md) (By changing config > topics to include `MetadataChangeLog_Versioned_v1` and `MetadataChangeLog_Timeseries_v1`)
## Configure the Event Source
@ -75,6 +74,7 @@ datahub:
source:
type: "datahub-cloud"
config:
topics: ["PlatformEvent_v1"] # Add MetadataChangeLog_Versioned_v1 and / or MetadataChangeLog_Timeseries_v1 to generate raw MCL events.
lookback_days: 7 # Look back 7 days for events
reset_offsets: true # Ignore stored offsets and start fresh
kill_after_idle_timeout: true # Enable shutdown after idle period
@ -89,14 +89,14 @@ Note that the `datahub` configuration block is **required** to connect to your D
<details>
<summary>View All Configuration Options</summary>
| Field | Required | Default | Description |
| -------------------------------------------- | :------: | :----------------: | --------------------------------------------------------------------------------------------------------------- |
| `topic` | ❌ | `PlatformEvent_v1` | The name of the topic from which events will be consumed. Do not change this unless you know what you're doing! |
| `lookback_days` | ❌ | None | Optional number of days to look back when polling for events. |
| `reset_offsets` | ❌ | `False` | When set to `True`, the consumer will ignore any stored offsets and start fresh. |
| `kill_after_idle_timeout` | ❌ | `False` | If `True`, stops the consumer after being idle for the specified timeout duration. |
| `idle_timeout_duration_seconds` | ❌ | `30` | Duration in seconds after which, if no events are received, the consumer is considered idle. |
| `event_processing_time_max_duration_seconds` | ❌ | `30` | Maximum allowed time in seconds for processing events before timing out. |
| Field | Required | Default | Description |
| -------------------------------------------- | :------: | :----------------: | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `topics` | ❌ | `PlatformEvent_v1` | The name of the topic from which events will be consumed. By default only produces `EntityChangeEvent_v1` events. To include `MetadataChangeLog_v1` events, set this value to include ["MetadataChangeLog_Versioned_v1", "MetadataChangeLog_Timeseries_v1"] |
| `lookback_days` | ❌ | None | Optional number of days to look back when polling for events. |
| `reset_offsets` | ❌ | `False` | When set to `True`, the consumer will ignore any stored offsets and start fresh. |
| `kill_after_idle_timeout` | ❌ | `False` | If `True`, stops the consumer after being idle for the specified timeout duration. |
| `idle_timeout_duration_seconds` | ❌ | `30` | Duration in seconds after which, if no events are received, the consumer is considered idle. |
| `event_processing_time_max_duration_seconds` | ❌ | `30` | Maximum allowed time in seconds for processing events before timing out. |
</details>