yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-04 23:57:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/metadata-ingestion/docs/sources/nifi/nifi_pre.md

1.5 KiB
Raw Permalink Blame History

Concept Mapping

Source Concept DataHub Concept Notes
"Nifi" Data Platform
Nifi flow Data Flow
Nifi Ingress / Egress Processor Data Job
Nifi Remote Port Data Job
Nifi Port with remote connections Dataset
Nifi Process Group Container Subtype Process Group

Caveats

  • This plugin extracts the lineage information between external datasets and ingress/egress processors by analyzing provenance events. Please check your Nifi configuration to confirm max rentention period of provenance events and make sure that ingestion runs frequent enough to read provenance events before they are disappear.

  • Limited ingress/egress processors are supported

    • S3: ListS3, FetchS3Object, PutS3Object
    • SFTP: ListSFTP, FetchSFTP, GetSFTP, PutSFTP