mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-08-05 23:58:18 +00:00
b0e1a136cf
* fix(airflow): correctly extract owners from serialized Airflow DAGs Airflow serialization format wraps tasks under `__var` and `__type`. Previously, the OpenMetadata Airflow connector failed to extract task owners properly in this format. This patch: - Flattens `__var` when parsing task owners - Fallbacks to `default_args["owner"]` if no task-level owner is explicitly present - Ensures correct DAG owner is picked as the most common task owner - Handles compatibility with older Airflow versions Fixes: #21106 * test(airflow): add tests for owner extraction from serialized Airflow DAGs Adds new test cases to validate owner extraction logic: - Owners from serialized task format (`__var`) - Fallback to `default_args['owner']` if task owners are missing - Resolution of most common owner - Compatibility with unstructured or missing owners * remove test version specific comment * simplify comments and warnings * fix return statement * fixing formatting * adding handling of default args * fixing and adding more tests
This guide will help you setup the Ingestion framework and connectors
OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites
- Python >= 3.9.x
Docs
Please refer to the documentation here https://docs.open-metadata.org/connectors
TopologyRunner
All the Ingestion Workflows run through the TopologyRunner.
The flow is depicted in the images below.
TopologyRunner Standard Flow
TopologyRunner Multithread Flow
