yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-01 19:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/metadata-jobs
History
Javier Sotelo ad4c80a753
feat(kafka-config): Add ability to configure other Kafka props (#1745) 
* Integarte spring-kafka & spring-boot for security props

- Upgrade spring-kafka to 2.1.14
- Use KafkaListener and KafkaTemplates to enable KafkaAutoConfiguration
- Integrates spring-boot's KafkaProperties into spring-kafka's config

* Cleanup imports

* Add DataHub kafka env vars

* Remove kafka-streams dependency

* Add KafkaProperties to gms; Add docs

* Add to Adoption

* Remove KAFKA_BOOTSTRAP_SERVER default

Co-authored-by: jsotelo <javier.sotelo@viasat.com>
Co-authored-by: Kerem Sahin <ksahin@linkedin.com>
2020-07-27 05:22:51 -07:00
..
mae-consumer-job
feat(kafka-config): Add ability to configure other Kafka props (#1745)
2020-07-27 05:22:51 -07:00
mce-consumer-job
feat(kafka-config): Add ability to configure other Kafka props (#1745)
2020-07-27 05:22:51 -07:00
README.md
Update README.md
2020-02-28 22:08:28 -08:00

README.md

MXE Processing Jobs

DataHub uses Kafka as the pub-sub message queue in the backend. There are 2 Kafka topics used by DataHub which are MetadataChangeEvent and MetadataAuditEvent.

  • MetadataChangeEvent: This message is emitted by any data platform or crawler in which there is a change in the metadata.
  • MetadataAuditEvent: This message is emitted by DataHub GMS to notify that metadata change is registered.

To be able to consume from these two topics, there are two Kafka Streams jobs DataHub uses: