yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-12-27 15:38:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add ingestion docs to implement a connector

Browse Source
This commit is contained in:
Suresh Srinivas 2021-08-16 15:56:58 -07:00
parent 9e23bd99e7
commit 4d74050884
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/open-source-community/developer/build-a-connector/README.md
View File

@ -22,7 +22,8 @@ Workflow execution happens in serial fashion.
1. It runs **source** component first. Source component retrieves a record from external sources and emits the record downstream.
2. if **processor** component is configured it sends the record to processor first
3. There can be multiple processors attached to the workflow it passes them in the order they are configurd
4. Once the **processors** finished , it sends the modified to record to Sink. All of these happens per record
4. Once the **processors** finished , it sends the modified to record to Sink.
5. The above steps repeats per record emitted from source component
In the cases where we need to aggregation over the records, we can use **stage** to write to a file or other store. Use the file written to in **stage** and pass it to **bulksink** to publish to external services such as **openmetadata** or **elasticsearch**