Mars Lan
c75fa5e6dc
Use environmental variables to set ETL & Kafka job IDs. ( #418 )
...
This will allow us to set different job IDs in staging & production via cfg2.
2017-07-10 09:57:51 -07:00
Yi Wang
fcd6cf149e
Update MetastoreAuditProcessor to reduce storage, also refactor some code
2016-10-11 11:26:36 -07:00
Yi Wang
5049c847fa
Update Kafka consumer actors to reduce memory usage
2016-10-10 14:49:14 -07:00
Yi (Alan) Wang
c9dfb637af
Update MetadataChangeEvent APIs according to schema change ( #243 )
...
* Update MetadataChangeEvent APIs according to schema change
* Update MultiproductLoad to reflect new Owner types
* Add comments for Owner_type precedence (priority) and compliance
2016-10-06 13:33:45 -07:00
jbai
a11e4908dc
tracking the GobblinTrackingEvent_autit to get owner information
2016-09-29 15:01:32 -07:00
Yi Wang
b136fc6c37
Add MetadataInventoryEvent processor and API
2016-09-15 09:22:42 -07:00
Yi Wang
5515cbdde9
Add MatadataChangeEvent processor to call seperate APIs
2016-09-06 16:41:50 -07:00
Yi Wang
c0cfe1f5ca
Modify KafkaConsumerMaster to handle more than one kafka config, add error handling
2016-08-04 13:07:19 -07:00
Yi Wang
3d3b2a8075
Get kafka job id from applicatoin.conf and then get ref_id and configs from DB
2016-08-03 18:55:07 -07:00
Yi Wang
7edacc9a9f
get kafka config from wh_etl_job_property
2016-07-26 12:16:34 -07:00
Yi Wang
6d4706bc62
Ingest Gobblin tracking events into wherehows using Kafka consumer client
2016-07-25 15:03:29 -07:00
SunZhaonan
aff8f323e4
Scheduler check previous job is finished. Redirect remote outputstream into log. Fix avro parser bugs
2016-03-16 19:09:53 -07:00
SunZhaonan
d5c3d87d00
Initial commit
2015-11-19 14:39:21 -08:00
