mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-10-17 19:57:56 +00:00
97fad806a2
* feat(app): add config to status add config to the reported status of the ingestion pipeline * added separate pipeline service client call for external apps * fix masking of pydantic model * - overload model_dump to mask secrets instead of a separate method - moved tests to test_custom_pydantic.py * fix: execution time * fix: mask secrets in dump json * fix: for python3.8 * fix: for python3.8 * fix: use mask_secrets=False when dumping a model for create * format * fix: update mask_secrets=False for workflow configurations * fix: use context directly when using model_dump_json * fix: default behavior when dumping json * format * fixed tests
This guide will help you setup the Ingestion framework and connectors
OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites
- Python >= 3.8.x
Docs
Please refer to the documentation here https://docs.open-metadata.org/connectors
TopologyRunner
All the Ingestion Workflows run through the TopologyRunner.
The flow is depicted in the images below.
TopologyRunner Standard Flow
TopologyRunner Multithread Flow
