yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-09-26 01:23:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

Classified the ETL jobs in metadata-ingestion.

Browse Source
This commit is contained in:
Chris Lee 2019-09-16 12:52:37 -07:00
parent 28b876f323
commit 81a38f87f7
4 changed files with 8 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
metadata-ingestion/README.md
View File

@ -34,7 +34,7 @@ optional arguments:
``` ```
## Bootstrapping Data Hub ## Bootstrapping Data Hub
If you want to quickly ingest lots of sample data and test Data Hub in action, you can run below command: Leverage the mce-cli to quickly ingest lots of sample data and test Data Hub in action, you can run below command:
``` ```
➜ python mce_cli.py produce -d bootstrap_mce.dat ➜ python mce_cli.py produce -d bootstrap_mce.dat
Producing MetadataChangeEvent records to topic MetadataChangeEvent. ^c to exit. Producing MetadataChangeEvent records to topic MetadataChangeEvent. ^c to exit.
@ -45,7 +45,7 @@ Flushing records...
This will bootstrap Data Hub with sample datasets and sample users. This will bootstrap Data Hub with sample datasets and sample users.
## Ingest metadata from LDAP server to Data Hub ## Ingest metadata from LDAP server to Data Hub
The ldap_etl.py provides you ETL channel to communicate with your LDAP server. The ldap_etl provides you ETL channel to communicate with your LDAP server.
``` ```
➜ Config your LDAP server environmental variable in the file ➜ Config your LDAP server environmental variable in the file
LDAPSERVER # Your server host. LDAPSERVER # Your server host.

8
metadata-ingestion/ldap_etl.py → metadata-ingestion/ldap-etl/ldap_etl.py
View File

@ -14,10 +14,10 @@ PAGESIZE = PAGESIZE
ATTRLIST = ['cn', 'title', 'mail', 'sAMAccountName', 'department','manager'] ATTRLIST = ['cn', 'title', 'mail', 'sAMAccountName', 'department','manager']
SEARCHFILTER='SEARCHFILTER' SEARCHFILTER='SEARCHFILTER'
AVROLOADPATH = 'AVROLOADPATH' AVROLOADPATH = '../../metadata-events/mxe-schemas/src/renamed/avro/com/linkedin/mxe/MetadataChangeEvent.avsc'
KAFKATOPIC = 'KAFKATOPIC' KAFKATOPIC = 'MetadataChangeEvent'
BOOTSTRAP = 'BOOTSTRAP' BOOTSTRAP = 'localhost:9092'
SCHEMAREGISTRY = 'SCHEMAREGISTRY' SCHEMAREGISTRY = 'http://localhost:8081'
def create_controls(pagesize): def create_controls(pagesize):
""" """

0
metadata-ingestion/bootstrap_mce.dat → metadata-ingestion/mce-cli/bootstrap_mce.dat
View File

3
metadata-ingestion/mce_cli.py → metadata-ingestion/mce-cli/mce_cli.py
View File

@ -1,7 +1,8 @@
#! /usr/bin/python
import argparse import argparse
from confluent_kafka import avro from confluent_kafka import avro
record_schema = avro.load("../metadata-events/mxe-schemas/src/renamed/avro/com/linkedin/mxe/MetadataChangeEvent.avsc") record_schema = avro.load("../../metadata-events/mxe-schemas/src/renamed/avro/com/linkedin/mxe/MetadataChangeEvent.avsc")
topic = "MetadataChangeEvent" topic = "MetadataChangeEvent"
class MetadataChangeEvent(object): class MetadataChangeEvent(object):