yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-09 07:53:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix config typo in stateful ingestion README (#4202)

Browse Source
This commit is contained in:
Jie Qiu 2022-02-22 18:20:53 -05:00 committed by GitHub
parent c6cb549918
commit c372b93804
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 9 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
metadata-ingestion/source_docs/stateful_ingestion.md
View File

@ -25,10 +25,10 @@ Note that a `.` is used to denote nested fields in the YAML recipe.
| Field | Required | Default | Description |
|--------------------------------------------------------------| -------- |------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `source.config.stateful_ingestion.enabled` | | False | The type of the ingestion state provider registered with datahub. |
| `source.conifg.stateful_ingestion.ignore_old_state` | | False | If set to True, ignores the previous checkpoint state. |
| `source.conifg.stateful_ingestion.ignore_new_state` | | False | If set to True, ignores the current checkpoint state. |
| `source.conifg.stateful_ingestion.max_checkpoint_state_size` | | 2^24 (16MB) | The maximum size of the checkpoint state in bytes. |
| `source.conifg.stateful_ingestion.state_provider` | | The default [datahub ingestion state provider](#datahub-ingestion-state-provider) configuration. | The ingestion state provider configuration. |
| `source.config.stateful_ingestion.ignore_old_state` | | False | If set to True, ignores the previous checkpoint state. |
| `source.config.stateful_ingestion.ignore_new_state` | | False | If set to True, ignores the current checkpoint state. |
| `source.config.stateful_ingestion.max_checkpoint_state_size` | | 2^24 (16MB) | The maximum size of the checkpoint state in bytes. |
| `source.config.stateful_ingestion.state_provider` | | The default [datahub ingestion state provider](#datahub-ingestion-state-provider) configuration. | The ingestion state provider configuration. |
| `pipeline_name` | ✅ | | The name of the ingestion pipeline the checkpoint states of various source connector job runs are saved/retrieved against via the ingestion state provider. |
NOTE: If either `dry-run` or `preview` mode are set, stateful ingestion will be turned off regardless of the rest of the configuration.