yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-31 10:39:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenMetadata/ingestion/README.md

31 lines
1.0 KiB
Markdown
Raw Normal View History

Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
---
This guide will help you setup the Ingestion framework and connectors
---
MINOR: Update README.md python version (#15893)
2024-04-18 18:03:16 +02:00
![Python version 3.8+](https://img.shields.io/badge/python-3.8%2B-blue)
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
[issue-1698] - Airflow DockerOperator setup (#1730) * Fix typo * Clean setup * Update ingestion local image to be barebone on connector dependencies * Prepare ingestion connectors base image * Add system dependencies * Prepare docker CLI * Add docker provider * Prepare entrypoint for the image * Remove DBT pipeline as per Issue 1658 * Add TODO for ingestion build * Bind docker socket * Update comment * Update README * Use DockerOperator in sample data * Build images with latest tag * Prepare symlink to pass the volume to the DockerOperator * Update README * Prepare Base image for CI * COPY multiple files into dir * COPY multiple files into dir * Remove DBT source as is now part of table ingestion * Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata.
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
**Prerequisites**
MINOR: Update README.md python version (#15893)
2024-04-18 18:03:16 +02:00
- Python >= 3.8.x
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
Update README.md
2021-11-24 07:18:33 -08:00
### Docs
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
Fixed Broken docs links (#9125)
2022-12-02 20:12:06 +05:30
Please refer to the documentation here https://docs.open-metadata.org/connectors
[WIP] - Add Scarf gateway (#9652) * Update ingestion Readme with scarf * Test scarf docker * Update ingestion Readme with scarf * Test scarf docker * Add docker domain
2023-01-11 09:33:09 +01:00
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=c1a30c7c-6dc7-4928-95bf-6ee08ca6aa6a" />
FIXES 15215: Implement initial Multithreading approach for the Metadata Ingestion on Databases (#15130) * Implement Initial MultiThread suggestion * Update all the ingestion sources to use the new ContextManager * Fix missing wraps on decorator * Fix Unittests * Fix linters * Fix linters * Fix BigQuery UnitTests * Add UnitTests to the newly created code * Fix unittest * change the threads from table to schemas * Update README.md * Small change suggested by Sonar * Slight change to test a different way to multithread over tables * Debug changes * More multithread tests * Remove uneeded wait time * Testing * refactor code based on removal of time.sleep * Fix wrong paste * Improve ExecutionTimeContextManager * Fix missing .get() and unit tests * Fix conflicting changes * Update Multithread logic with the incremental extraction * Fix linters * Fix unittest * Remove commented code * Fix Unittests * Fix checkstyle * Change default to threads = 1
2024-03-25 18:20:40 +01:00
### TopologyRunner
All the Ingestion Workflows run through the TopologyRunner.
The flow is depicted in the images below.
**TopologyRunner Standard Flow**
MINOR: Add Multithread Documentation (#15706) * Add Multithread Documentation * Add general considerations
2024-03-27 08:30:16 +01:00
![image](../openmetadata-docs/images/v1.4/features/ingestion/workflows/metadata/multithreading/single-thread-flow.png)
FIXES 15215: Implement initial Multithreading approach for the Metadata Ingestion on Databases (#15130) * Implement Initial MultiThread suggestion * Update all the ingestion sources to use the new ContextManager * Fix missing wraps on decorator * Fix Unittests * Fix linters * Fix linters * Fix BigQuery UnitTests * Add UnitTests to the newly created code * Fix unittest * change the threads from table to schemas * Update README.md * Small change suggested by Sonar * Slight change to test a different way to multithread over tables * Debug changes * More multithread tests * Remove uneeded wait time * Testing * refactor code based on removal of time.sleep * Fix wrong paste * Improve ExecutionTimeContextManager * Fix missing .get() and unit tests * Fix conflicting changes * Update Multithread logic with the incremental extraction * Fix linters * Fix unittest * Remove commented code * Fix Unittests * Fix checkstyle * Change default to threads = 1
2024-03-25 18:20:40 +01:00
**TopologyRunner Multithread Flow**
MINOR: Add Multithread Documentation (#15706) * Add Multithread Documentation * Add general considerations
2024-03-27 08:30:16 +01:00
![image](../openmetadata-docs/images/v1.4/features/ingestion/workflows/metadata/multithreading/multi-thread-flow.png)
Reference in New Issue Copy Permalink