28 Commits

Author	SHA1	Message	Date
Yi Wang	b4f5e438e2	Add JobExecutionLineageEvent and kafka processor	2016-11-08 19:11:37 -08:00
Yi Wang	664e4072bb	Upgrade to play 2.4.8	2016-10-19 17:42:28 -07:00
Yi Wang	3227412339	Login authentication support multiple LDAP servers, add login history	2016-10-13 14:30:43 -07:00
Yi Wang	fcd6cf149e	Update MetastoreAuditProcessor to reduce storage, also refactor some code	2016-10-11 11:26:36 -07:00
Yi Wang	5049c847fa	Update Kafka consumer actors to reduce memory usage	2016-10-10 14:49:14 -07:00
Yi Wang	ac34eb683f	Update Kafka processor casting Object to String, also add debug info if can't fetch schema from Registery	2016-09-26 15:06:33 -07:00
Yi Wang	1ad2b1528e	logback redirect ETL job logs into corresponding files	2016-09-23 16:54:52 -07:00
Yi Wang	5ce5a1425e	Add hostname and process_id to wh_etl_job_execution	2016-09-12 16:09:33 -07:00
Yi Wang	c0cfe1f5ca	Modify KafkaConsumerMaster to handle more than one kafka config, add error handling	2016-08-04 13:07:19 -07:00
Yi Wang	3d3b2a8075	Get kafka job id from applicatoin.conf and then get ref_id and configs from DB	2016-08-03 18:55:07 -07:00
Yi Wang	74ed769bab	add Oracle dataset metadata ETL job	2016-07-28 14:07:07 -07:00
Yi Wang	7edacc9a9f	get kafka config from wh_etl_job_property	2016-07-26 12:16:34 -07:00
Yi Wang	6d4706bc62	Ingest Gobblin tracking events into wherehows using Kafka consumer client	2016-07-25 15:03:29 -07:00
jbai	e7885f28db	comment out the flow tree builder since UI does not use it anymore	2016-06-13 10:15:32 -07:00
Rafal Kluszczynski	d06bfbfbd0	fix: use correctly directory path from properties when executing etl job	2016-05-30 08:59:05 +02:00
SunZhaonan	0b5c421311	Fix Hive column parser parent path bug	2016-05-19 16:36:30 -07:00
SunZhaonan	9d6a1b2649	Add optional config of ETL job white list	2016-05-12 16:28:23 -07:00
SunZhaonan	31de21ddcf	pass parameter through file.	2016-05-03 16:25:56 -07:00
SunZhaonan	c66b00e2f6	Fix dataset insert API bug. Fix load sql bug.	2016-03-28 16:27:43 -07:00
SunZhaonan	a0b7cb9d57	Fix process hanging bug. Add hive field ETL process.	2016-03-16 19:12:21 -07:00
SunZhaonan	aff8f323e4	Scheduler check previous job is finished. Redirect remote outputstream into log. Fix avro parser bugs	2016-03-16 19:09:53 -07:00
SunZhaonan	5e9ae37952	Change to multi processing instead of multi thread. Fix hive ETL bug	2016-02-29 16:37:03 -08:00
SunZhaonan	c3da00003e	Fix bugs. Reenforce logging. Format jython scripts. Add missing table DDL.	2016-02-03 19:22:18 -08:00
SunZhaonan	05d54b3070	Add Hive metadata ETL process	2015-12-21 12:07:08 -08:00
Zhen Chen	e21a6b8e75	remove unused variable, change load script and add etl type in backend service	2015-12-11 13:52:23 -08:00
Zhen Chen	ebbf9ec629	add ldap user and group metadata etl	2015-12-10 16:26:57 -08:00
Zhen Chen	5a08134b8d	add dataset owner metadata etl	2015-12-07 15:17:01 -08:00
SunZhaonan	d5c3d87d00	Initial commit	2015-11-19 14:39:21 -08:00