yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-01 19:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(airflow): document background operation (#13907)

Browse Source
This commit is contained in:
Harshal Sheth 2025-06-30 17:35:19 -04:00 committed by GitHub
parent 31ee414008
commit 3583aa1eff
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/lineage/airflow.md
View File

@ -346,6 +346,10 @@ TypeError: on_task_instance_success() missing 3 required positional arguments: '
The solution is to upgrade `acryl-datahub-airflow-plugin>=0.12.0.4` or upgrade `pluggy>=1.2.0`. See this [PR](https://github.com/datahub-project/datahub/pull/9365) for details.
### Scheduler stalling
For extremely large Airflow deployments with thousands of tasks, you may see issues where the plugin interferes with the performance of the Airflow scheduler. In those cases, you can set the `DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT=0` environment variable. This makes the DataHub plugin run fully in background threads, but can cause us to miss some metadata if the scheduler shuts down soon after processing a task.
### Disabling the DataHub Plugin v2
There are two ways to disable the DataHub Plugin v2: