yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-09 16:03:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/contrib/metadata-ingestion/python/kafka-etl
History
John Plaisted 821bce7d69
feat: Port mce-cli to Java. (#1871) 
Port mce-cli to Java.

Also moved off the avro format event file to json instead. Much nicer to use :)
2020-09-25 14:05:29 -07:00
..
kafka_etl.py
feat: Port mce-cli to Java. (#1871)
2020-09-25 14:05:29 -07:00
README.md
Start adding java ETL examples, starting with kafka etl. (#1805)
2020-09-11 13:04:21 -07:00
requirements.txt
Start adding java ETL examples, starting with kafka etl. (#1805)
2020-09-11 13:04:21 -07:00

README.md

Kafka ETL

Ingest metadata from Kafka to DataHub

The kafka_etl provides you ETL channel to communicate with your kafka.

➜  Config your kafka environmental variable in the file.
    ZOOKEEPER      # Your zookeeper host.
    
➜  Config your Kafka broker environmental variable in the file.
    AVROLOADPATH   # Your model event in avro format.
    KAFKATOPIC     # Your event topic.
    BOOTSTRAP      # Kafka bootstrap server.
    SCHEMAREGISTRY # Kafka schema registry host.

➜  python kafka_etl.py

This will bootstrap DataHub with your metadata in the kafka as a dataset entity.