yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-08-18 22:28:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/docs/how/kafka-config.md
Javier Sotelo ad4c80a753
feat(kafka-config): Add ability to configure other Kafka props (#1745) 
* Integarte spring-kafka & spring-boot for security props

- Upgrade spring-kafka to 2.1.14
- Use KafkaListener and KafkaTemplates to enable KafkaAutoConfiguration
- Integrates spring-boot's KafkaProperties into spring-kafka's config

* Cleanup imports

* Add DataHub kafka env vars

* Remove kafka-streams dependency

* Add KafkaProperties to gms; Add docs

* Add to Adoption

* Remove KAFKA_BOOTSTRAP_SERVER default

Co-authored-by: jsotelo <javier.sotelo@viasat.com>
Co-authored-by: Kerem Sahin <ksahin@linkedin.com>
2020-07-27 05:22:51 -07:00

1.5 KiB
Raw Blame History

How to configure Kafka?

With the exception of KAFKA_BOOTSTRAP_SERVER and KAFKA_SCHEMAREGISTRY_URL, Kafka is configured via spring-boot, specifically with KafkaProperties. See Integration Properties prefixed with spring.kafka.

Below is an example of how SASL/GSSAPI properties can be configured via environment variables:

export KAFKA_BOOTSTRAP_SERVER=broker:29092
export KAFKA_SCHEMAREGISTRY_URL=http://schema-registry:8081
export SPRING_KAFKA_PROPERTIES_SASL_KERBEROS_SERVICE_NAME=kafka
export SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL=SASL_PLAINTEXT
export SPRING_KAFKA_SASL_JAAS_CONFIG=com.sun.security.auth.module.Krb5LoginModule required principal='principal@REALM' useKeyTab=true storeKey=true keyTab='/keytab';

These properties can be specified via application.properties or application.yml files, or as command line switches, or as environment variables. See Spring's Externalized Configuration to see how this works.

See Kafka Connect Security for more ways to connect.

DataHub components that connect to Kafka are currently:

  • mce-consumer-job
  • mae-consumer-job
  • gms