yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-10-31 10:49:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/docker/kafka-setup
History
Esteban Gutierrez 8a17ba14d6
fix(): Fixes multiple minor security vulnerabilities (#13222) 
bug(snappy): Make sure right snappy version is installed
fix(docker): update Dockerize to version v0.9.3
fix(gms): fixes dgraph4j netty deps
fix(docker): remove SGID on /home/datahub and /home/datahub-integration
fix(datahub-actions): bump setuptools and wheel version
fix(docker): update c-ares version
fix(docker): datahub-actions addendum
2025-04-21 16:39:26 -05:00
..
env
feat(schema-registry): replace confluent schema registry (#7930)
2023-05-01 13:18:41 -05:00
build.gradle
CI speedup (#13057)
2025-04-07 10:13:07 +05:30
Dockerfile
fix(): Fixes multiple minor security vulnerabilities (#13222)
2025-04-21 16:39:26 -05:00
env_to_properties.py
change generate kafka connect properties from env (#10545)
2024-07-31 15:05:55 -05:00
kafka-config.sh
feat(kafka-setup): allow override KAFKA_HEAP_OPTS (#11400)
2024-09-17 13:00:24 -05:00
kafka-ready.sh
fix(docker): misc docker fixes (#8677)
2023-08-22 10:38:08 +05:30
kafka-setup.sh
change generate kafka connect properties from env (#10545)
2024-07-31 15:05:55 -05:00
kafka-topic-workers.sh
feat(kafka): increase kafka message size and enable compression (#9038)
2023-10-29 16:26:05 -05:00
README.md
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00

README.md

Kafka, Zookeeper and Schema Registry

DataHub uses Kafka as the pub-sub message queue in the backend. Official Confluent Kafka Docker images found in Docker Hub is used without any modification.

Debugging Kafka

You can install kafkacat to consume and produce messaged to Kafka topics. For example, to consume messages on MetadataAuditEvent topic, you can run below command.

kafkacat -b localhost:9092 -t MetadataAuditEvent

However, kafkacat currently doesn't support Avro deserialization at this point, but they have an ongoing work for that.