yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-04 15:50:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/docker/kafka-setup/README.md

18 lines
692 B
Markdown
Raw Permalink Normal View History

refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
# Kafka, Zookeeper and Schema Registry
DataHub uses Kafka as the pub-sub message queue in the backend.
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
[Official Confluent Kafka Docker images](https://hub.docker.com/u/confluentinc) found in Docker Hub is used without
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
any modification.
## Debugging Kafka
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
You can install [kafkacat](https://github.com/edenhill/kafkacat) to consume and produce messaged to Kafka topics.
For example, to consume messages on MetadataAuditEvent topic, you can run below command.
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
```
kafkacat -b localhost:9092 -t MetadataAuditEvent
```
ci(graphql,workflows): Format .md, .graphql, and workflow .yml files via prettier (#13220)
2025-04-16 16:55:51 -07:00
However, `kafkacat` currently doesn't support Avro deserialization at this point,
but they have an ongoing [work](https://github.com/edenhill/kafkacat/pull/151) for that.
Reference in New Issue Copy Permalink