yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-29 12:30:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/contrib/metadata-ingestion/python/kafka-etl
History
John Plaisted 6ece2d6469
Start adding java ETL examples, starting with kafka etl. (#1805) 
Start adding java ETL examples, starting with kafka etl.

We've had a few requests to start providing Java examples rather than Python due to type safety.

I've also started to add these to metadata-ingestion-examples to make it clearer these are *examples*. They can be used directly or as a basis for other things.

As we port to Java we'll move examples to contrib.
2020-09-11 13:04:21 -07:00
..
kafka_etl.py
Start adding java ETL examples, starting with kafka etl. (#1805)
2020-09-11 13:04:21 -07:00
README.md
Start adding java ETL examples, starting with kafka etl. (#1805)
2020-09-11 13:04:21 -07:00
requirements.txt
Start adding java ETL examples, starting with kafka etl. (#1805)
2020-09-11 13:04:21 -07:00

README.md

Kafka ETL

Ingest metadata from Kafka to DataHub

The kafka_etl provides you ETL channel to communicate with your kafka.

➜  Config your kafka environmental variable in the file.
    ZOOKEEPER      # Your zookeeper host.
    
➜  Config your Kafka broker environmental variable in the file.
    AVROLOADPATH   # Your model event in avro format.
    KAFKATOPIC     # Your event topic.
    BOOTSTRAP      # Kafka bootstrap server.
    SCHEMAREGISTRY # Kafka schema registry host.

➜  python kafka_etl.py

This will bootstrap DataHub with your metadata in the kafka as a dataset entity.