yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-28 20:09:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/metadata-jobs
History
david-leifker cd05f5b174
feat(schema-registry): replace confluent schema registry (#7930) 
Co-authored-by: Pedro Silva <pedro@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Ryan Holstien <ryan@acryl.io>
2023-05-01 13:18:41 -05:00
..
mae-consumer
feat(schema-registry): replace confluent schema registry (#7930)
2023-05-01 13:18:41 -05:00
mae-consumer-job
feat(schema-registry): replace confluent schema registry (#7930)
2023-05-01 13:18:41 -05:00
mce-consumer
feat(schema-registry): replace confluent schema registry (#7930)
2023-05-01 13:18:41 -05:00
mce-consumer-job
chore(snakeyaml): upgrade to snakeyaml 2 (#7786)
2023-04-13 12:01:51 -05:00
pe-consumer
Logging and test models fixes (#7884)
2023-04-21 15:45:20 -05:00
README.md
docs(): remove Kafka Streams from documentation (#6596)
2022-12-01 15:24:25 -08:00

README.md

MXE Processing Jobs

DataHub uses Kafka as the pub-sub message queue in the backend. There are 2 Kafka topics used by DataHub which are MetadataChangeEvent and MetadataAuditEvent.

  • MetadataChangeEvent: This message is emitted by any data platform or crawler in which there is a change in the metadata.
  • MetadataAuditEvent: This message is emitted by DataHub GMS to notify that metadata change is registered.

To be able to consume from these two topics, there are two Spring jobs DataHub uses: