yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-12-27 09:58:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(airflow): add known limitations for automatic lineage (#11652)

Browse Source
This commit is contained in:
Harshal Sheth 2024-10-17 16:36:18 -07:00 committed by GitHub
parent 523a456f5c
commit d7ccf3b2c3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 5 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/lineage/airflow.md
View File

@ -132,7 +132,7 @@ conn_id = datahub_rest_default # or datahub_kafka_default
```
| Name | Default value | Description |
|----------------------------|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| -------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| enabled | true | If the plugin should be enabled. |
| conn_id | datahub_rest_default | The name of the datahub connection you set in step 1. |
| cluster | prod | name of the airflow cluster |
@ -191,6 +191,10 @@ These operators are supported by OpenLineage, but we haven't tested them yet:
There's also a few operators (e.g. BashOperator, PythonOperator) that have custom extractors, but those extractors don't generate lineage.
-->
Known limitations:
- We do not fully support operators that run multiple SQL statements at once. In these cases, we'll only capture lineage from the first SQL statement.
## Manual Lineage Annotation
### Using `inlets` and `outlets`