yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-12-27 09:58:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Enabled the configuration lists in the LDAP ETL.

Browse Source
This commit is contained in:
Chris Lee 2019-09-11 11:30:19 -07:00
parent 9df9d336c9
commit 430f13108d
2 changed files with 26 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
metadata-ingestion/README.md
View File

@ -48,13 +48,19 @@ This will bootstrap Data Hub with sample datasets and sample users.
The ldap_etl.py provides you ETL channel to communicate with your LDAP server.
```
➜ Config your LDAP server environmental variable in the file
LDAPSERVER # Your server host.
BASEDN # Base dn as a container location.
LDAPUSER # Your credential.
LDAPPASSWORD # Your password.
PAGESIZE # Pagination size.
ATTRLIST # Return attributes relate to your model.
SEARCHFILTER # Filter to build the search query.
LDAPSERVER # Your server host.
BASEDN # Base dn as a container location.
LDAPUSER # Your credential.
LDAPPASSWORD # Your password.
PAGESIZE # Pagination size.
ATTRLIST # Return attributes relate to your model.
SEARCHFILTER # Filter to build the search query.
➜ Config your Kafka broker environmental variable in the file
AVROLOADPATH # Your model event in avro format.
KAFKATOPIC = # Your event topic.
BOOTSTRAP = # Kafka bootstrap server.
SCHEMAREGISTRY # Kafka schema registry host.
➜ python ldap_etl.py
```

21
metadata-ingestion/ldap_etl.py
View File

@ -6,13 +6,18 @@ from distutils.version import LooseVersion
LDAP24API = LooseVersion(ldap.__version__) >= LooseVersion('2.4')
LDAPSERVER='LDAPSERVER'
BASEDN='BASEDN'
LDAPSERVER ='LDAPSERVER'
BASEDN ='BASEDN'
LDAPUSER = 'LDAPUSER'
LDAPPASSWORD = 'LDAPPASSWORD'
PAGESIZE = 1000
PAGESIZE = PAGESIZE
ATTRLIST = ['cn', 'title', 'mail', 'sAMAccountName', 'department','manager']
SEARCHFILTER='objectClass=Person'
SEARCHFILTER='SEARCHFILTER'
AVROLOADPATH = 'AVROLOADPATH'
KAFKATOPIC = 'KAFKATOPIC'
BOOTSTRAP = 'BOOTSTRAP'
SCHEMAREGISTRY = 'SCHEMAREGISTRY'
def create_controls(pagesize):
"""
@ -79,13 +84,13 @@ def produce_corp_user_mce(mce):
from confluent_kafka import avro
from confluent_kafka.avro import AvroProducer
conf = {'bootstrap.servers': 'localhost:9092',
'schema.registry.url': 'http://localhost:8081'}
record_schema = avro.load('../metadata-events/mxe-schemas/src/renamed/avro/com/linkedin/mxe/MetadataChangeEvent.avsc')
conf = {'bootstrap.servers': BOOTSTRAP,
'schema.registry.url': SCHEMAREGISTRY}
record_schema = avro.load(AVROLOADPATH)
producer = AvroProducer(conf, default_value_schema=record_schema)
try:
producer.produce(topic='MetadataChangeEvent', value=mce)
producer.produce(topic=KAFKATOPIC, value=mce)
producer.poll(0)
sys.stdout.write('\n %s has been successfully produced!' % mce)
except ValueError as e: