yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-03 04:10:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/contrib/metadata-ingestion/python
History
John Plaisted 821bce7d69
feat: Port mce-cli to Java. (#1871) 
Port mce-cli to Java.

Also moved off the avro format event file to json instead. Much nicer to use :)
2020-09-25 14:05:29 -07:00
..
kafka-etl
feat: Port mce-cli to Java. (#1871)
2020-09-25 14:05:29 -07:00
mysql-etl
feat: Port mce-cli to Java. (#1871)
2020-09-25 14:05:29 -07:00
openldap-etl
Start adding java ETL examples, starting with kafka etl. (#1805)
2020-09-11 13:04:21 -07:00
README.md
Start adding java ETL examples, starting with kafka etl. (#1805)
2020-09-11 13:04:21 -07:00

README.md

Python ETL examples

ETL scripts written in Python.

Prerequisites

  1. Before running any python metadata ingestion job, you should make sure that DataHub backend services are all running. The easiest way to do that is through Docker images.
  2. You also need to build the mxe-schemas module as below. 
    ./gradlew :metadata-events:mxe-schemas:build
    This is needed to generate MetadataChangeEvent.avsc which is the schema for MetadataChangeEvent Kafka topic.
  3. All the scripts are written using Python 3 and most likely won't work with Python 2.x interpreters. You can verify the version of your Python using the following command. 
    python --version
    We recommend using pyenv to install and manage your Python environment.
  4. Before launching each ETL ingestion pipeline, you can install/verify the library versions as below. 
    pip install --user -r requirements.txt