yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-12-30 03:18:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/metadata-jobs
History

README.md

MXE Consumer Jobs

Data Hub uses Kafka as the pub-sub message queue in the backend. There are 2 Kafka topics used by Data Hub which are MetadataChangeEvent and MetadataAuditEvent.

  • MetadataChangeEvent: This message is emitted by any data platform or crawler in which there is a change in the metadata.
  • MetadataAuditEvent: This message is emitted by Data Hub GMS to notify that metadata change is registered.

To be able to consume from these two topics, there are two Kafka Streams jobs Data Hub uses: