2020-02-28 22:08:28 -08:00
|
|
|
# MXE Processing Jobs
|
2025-04-16 16:55:51 -07:00
|
|
|
|
2022-12-02 00:24:25 +01:00
|
|
|
DataHub uses Kafka as the pub-sub message queue in the backend. There are 2 Kafka topics used by DataHub which are
|
2019-09-09 01:45:49 -07:00
|
|
|
`MetadataChangeEvent` and `MetadataAuditEvent`.
|
2025-04-16 16:55:51 -07:00
|
|
|
|
|
|
|
|
- `MetadataChangeEvent:` This message is emitted by any data platform or crawler in which there is a change in the metadata.
|
|
|
|
|
- `MetadataAuditEvent:` This message is emitted by [DataHub GMS](../metadata-service/README.md) to notify that metadata change is registered.
|
2019-09-09 01:45:49 -07:00
|
|
|
|
2022-12-02 00:24:25 +01:00
|
|
|
To be able to consume from these two topics, there are two Spring
|
2025-04-16 16:55:51 -07:00
|
|
|
jobs DataHub uses:
|
|
|
|
|
|
|
|
|
|
- [MCE Consumer Job](mce-consumer-job): Writes to [DataHub GMS](../metadata-service/README.md)
|
|
|
|
|
- [MAE Consumer Job](mae-consumer-job): Writes to [Elasticsearch](../docker/elasticsearch) & [Neo4j](../docker/neo4j)
|
