yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-10-13 09:54:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(ingest): update links to Kafka docs (#2834)

Browse Source
This commit is contained in:
Harshal Sheth 2021-07-06 15:33:52 -07:00 committed by GitHub
parent 3d9f4ec1b4
commit 288d17f07e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 7 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
metadata-ingestion/README.md
View File

@ -157,7 +157,7 @@ source:
schema_registry_config: {} # passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.schema_registry.SchemaRegistryClient
```
For a full example with a number of security options, see this [example recipe](./examples/recipes/secured_kafka_to_console.yml).
For a full example with a number of security options, see this [example recipe](./examples/recipes/secured_kafka.yml).
### MySQL Metadata `mysql`
@ -895,11 +895,13 @@ sink:
config:
connection:
bootstrap: "localhost:9092"
producer_config: {} # passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/index.html#serializingproducer
producer_config: {} # passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#serializingproducer
schema_registry_url: "http://localhost:8081"
schema_registry_config: {} # passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.schema_registry.SchemaRegistryClient
```
For a full example with a number of security options, see this [example recipe](./examples/recipes/secured_kafka.yml).
### Console `console`
Simply prints each metadata event to stdout. Useful for experimentation and debugging purposes.

0
metadata-ingestion/examples/recipes/secured_kafka_to_console.yml → metadata-ingestion/examples/recipes/secured_kafka.yml
View File

6
metadata-ingestion/src/datahub/configuration/kafka.py
View File

@ -14,7 +14,7 @@ class _KafkaConnectionConfig(ConfigModel):
# Extra schema registry config.
# These options will be passed into Kafka's SchemaRegistryClient.
# See https://docs.confluent.io/platform/current/clients/confluent-kafka-python/index.html?highlight=schema%20registry#schemaregistryclient.
# See https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html?#schemaregistryclient
schema_registry_config: dict = Field(default_factory=dict)
@validator("bootstrap")
@ -43,7 +43,7 @@ class KafkaConsumerConnectionConfig(_KafkaConnectionConfig):
# Extra consumer config.
# These options will be passed into Kafka's DeserializingConsumer.
# See https://docs.confluent.io/platform/current/clients/confluent-kafka-python/index.html#deserializingconsumer
# See https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#deserializingconsumer
# and https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md.
consumer_config: dict = Field(default_factory=dict)
@ -53,6 +53,6 @@ class KafkaProducerConnectionConfig(_KafkaConnectionConfig):
# Extra producer config.
# These options will be passed into Kafka's SerializingProducer.
# See https://docs.confluent.io/platform/current/clients/confluent-kafka-python/index.html#serializingproducer
# See https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#serializingproducer
# and https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md.
producer_config: dict = Field(default_factory=dict)