yujunjun/datahub

mirror of https://github.com/datahub-project/datahub.git synced 2025-11-04 04:39:10 +00:00

History

udays-visa 73d8a4649d

Block http trace method on mce and mce consumers (#11513 )

2024-10-04 11:44:10 -05:00

..

feat(build): support jdk8 spark lineage (#9697 )

2024-01-23 17:49:23 -06:00

chore(cleanup): remove legacy bootstrap step (#11494 )

2024-09-30 14:28:30 -05:00

mae-consumer-job

Block http trace method on mce and mce consumers (#11513 )

2024-10-04 11:44:10 -05:00

chore(cleanup): remove legacy bootstrap step (#11494 )

2024-09-30 14:28:30 -05:00

mce-consumer-job

Block http trace method on mce and mce consumers (#11513 )

2024-10-04 11:44:10 -05:00

feat(graph): graph index soft-delete support (#11453 )

2024-09-23 10:10:03 -05:00

README.md

…

README.md

MXE Processing Jobs

DataHub uses Kafka as the pub-sub message queue in the backend. There are 2 Kafka topics used by DataHub which are MetadataChangeEvent and MetadataAuditEvent.

MetadataChangeEvent: This message is emitted by any data platform or crawler in which there is a change in the metadata.
MetadataAuditEvent: This message is emitted by DataHub GMS to notify that metadata change is registered.

To be able to consume from these two topics, there are two Spring jobs DataHub uses:

MCE Consumer Job: Writes to DataHub GMS
MAE Consumer Job: Writes to Elasticsearch & Neo4j