mirror of
https://github.com/datahub-project/datahub.git
synced 2025-11-16 19:33:41 +00:00
6ece2d6469
Start adding java ETL examples, starting with kafka etl. We've had a few requests to start providing Java examples rather than Python due to type safety. I've also started to add these to metadata-ingestion-examples to make it clearer these are *examples*. They can be used directly or as a basis for other things. As we port to Java we'll move examples to contrib.
18 lines
599 B
Markdown
18 lines
599 B
Markdown
# Kafka ETL
|
|
|
|
## Ingest metadata from Kafka to DataHub
|
|
The kafka_etl provides you ETL channel to communicate with your kafka.
|
|
```
|
|
➜ Config your kafka environmental variable in the file.
|
|
ZOOKEEPER # Your zookeeper host.
|
|
|
|
➜ Config your Kafka broker environmental variable in the file.
|
|
AVROLOADPATH # Your model event in avro format.
|
|
KAFKATOPIC # Your event topic.
|
|
BOOTSTRAP # Kafka bootstrap server.
|
|
SCHEMAREGISTRY # Kafka schema registry host.
|
|
|
|
➜ python kafka_etl.py
|
|
```
|
|
This will bootstrap DataHub with your metadata in the kafka as a dataset entity.
|