mirror of
https://github.com/datahub-project/datahub.git
synced 2025-08-18 22:28:01 +00:00
ad4c80a753
* Integarte spring-kafka & spring-boot for security props - Upgrade spring-kafka to 2.1.14 - Use KafkaListener and KafkaTemplates to enable KafkaAutoConfiguration - Integrates spring-boot's KafkaProperties into spring-kafka's config * Cleanup imports * Add DataHub kafka env vars * Remove kafka-streams dependency * Add KafkaProperties to gms; Add docs * Add to Adoption * Remove KAFKA_BOOTSTRAP_SERVER default Co-authored-by: jsotelo <javier.sotelo@viasat.com> Co-authored-by: Kerem Sahin <ksahin@linkedin.com>
22 lines
1.5 KiB
Markdown
22 lines
1.5 KiB
Markdown
# How to configure Kafka?
|
|
|
|
With the exception of `KAFKA_BOOTSTRAP_SERVER` and `KAFKA_SCHEMAREGISTRY_URL`, Kafka is configured via [spring-boot](https://spring.io/projects/spring-boot), specifically with [KafkaProperties](https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/kafka/KafkaProperties.html). See [Integration Properties](https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#integration-properties) prefixed with `spring.kafka`.
|
|
|
|
Below is an example of how SASL/GSSAPI properties can be configured via environment variables:
|
|
```bash
|
|
export KAFKA_BOOTSTRAP_SERVER=broker:29092
|
|
export KAFKA_SCHEMAREGISTRY_URL=http://schema-registry:8081
|
|
export SPRING_KAFKA_PROPERTIES_SASL_KERBEROS_SERVICE_NAME=kafka
|
|
export SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL=SASL_PLAINTEXT
|
|
export SPRING_KAFKA_SASL_JAAS_CONFIG=com.sun.security.auth.module.Krb5LoginModule required principal='principal@REALM' useKeyTab=true storeKey=true keyTab='/keytab';
|
|
```
|
|
|
|
These properties can be specified via `application.properties` or `application.yml` files, or as command line switches, or as environment variables. See Spring's [Externalized Configuration](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config) to see how this works.
|
|
|
|
See [Kafka Connect Security](https://docs.confluent.io/current/connect/security.html) for more ways to connect.
|
|
|
|
DataHub components that connect to Kafka are currently:
|
|
- mce-consumer-job
|
|
- mae-consumer-job
|
|
- gms
|