yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-11-02 19:48:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenMetadata/ingestion
History
Keshav Mohta 7c0eeef049
Fixes #19692: Implemented Nifi Pipeline Lineage (#21802) 
* feat: implemented nifi pipeline lineage

* test: implemented tests for nifi pipeline lineage

* fix: yield_pipeline_bulk_lineage_details output type hinting

* fix: component check in connections

---------

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-06-18 07:31:04 +00:00
..
examples
Fix #21822: OpenSearch by default limits the number of characters it will analyze for highlighting to 1,000,000 characters. If your description field is very large (e.g. Markdown docs, embedded HTML, or verbose documentation), this limit gets exceeded. (#21821)
2025-06-17 14:22:11 -07:00
operators
update the snapshot to 1.8.0 (#20925)
2025-04-24 10:46:36 +05:30
pipelines
MINOR: Add failed rows sample to test case (#15682)
2024-04-10 17:00:00 +02:00
plugins
ISSUE #21101 - Implement BQ Partitioned Tests (#21348)
2025-05-22 17:22:05 +02:00
src
Fixes #19692: Implemented Nifi Pipeline Lineage (#21802)
2025-06-18 07:31:04 +00:00
tests
Fixes #19692: Implemented Nifi Pipeline Lineage (#21802)
2025-06-18 07:31:04 +00:00
__init__.py
ci/nox-setup-testing (#21377)
2025-05-27 10:56:52 +02:00
Dockerfile
update the snapshot to 1.8.0 (#20925)
2025-04-24 10:46:36 +05:30
Dockerfile.ci
MINOR - Remove airflow_lineage_operator from final ingestion image (#20551)
2025-04-02 11:53:55 +02:00
ingestion_dependency.sh
MINOR: Change ingestion licence header (#20549)
2025-04-03 10:39:47 +05:30
LICENSE
Docs - Ingestion License (#17893)
2024-09-17 08:58:53 -07:00
Makefile
ci/nox-setup-testing (#21377)
2025-05-27 10:56:52 +02:00
noxfile.py
fix/nox-ci-missing-steps (#21426)
2025-05-28 11:12:44 +02:00
pyproject.toml
MINOR: pytest opt out flaky test (#21800)
2025-06-17 14:23:28 +05:30
README.md
MINOR: Update README.md python version (#15893)
2024-04-18 21:33:16 +05:30
setup.py
MINOR: Implement dependency injection on ingestion (#21719)
2025-06-16 08:03:38 +02:00
sonar-project.properties
Add Sample data, modify regex pattern (#14467)
2024-01-11 14:23:33 +05:30

README.md

This guide will help you setup the Ingestion framework and connectors
This guide will help you setup the Ingestion framework and connectors

Python version 3.8+

OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites

  • Python >= 3.8.x

Docs

Please refer to the documentation here https://docs.open-metadata.org/connectors

TopologyRunner

All the Ingestion Workflows run through the TopologyRunner.

The flow is depicted in the images below.

TopologyRunner Standard Flow

image

TopologyRunner Multithread Flow

image