66 Commits

Author	SHA1	Message	Date
SunZhaonan	9d6a1b2649	Add optional config of ETL job white list	2016-05-12 16:28:23 -07:00
SunZhaonan	31de21ddcf	pass parameter through file.	2016-05-03 16:25:56 -07:00
SunZhaonan	c66b00e2f6	Fix dataset insert API bug. Fix load sql bug.	2016-03-28 16:27:43 -07:00
SunZhaonan	a0b7cb9d57	Fix process hanging bug. Add hive field ETL process.	2016-03-16 19:12:21 -07:00
SunZhaonan	aff8f323e4	Scheduler check previous job is finished. Redirect remote outputstream into log. Fix avro parser bugs	2016-03-16 19:09:53 -07:00
SunZhaonan	c4671d2579	Add field comments ETL ... Fix API bug of tech_matrix_id Add key in comment table	2016-03-14 14:23:33 -07:00
SunZhaonan	5e9ae37952	Change to multi processing instead of multi thread. Fix hive ETL bug	2016-02-29 16:37:03 -08:00
SunZhaonan	033a28faee	Backend job property API. 'update' change to 'insert on duplicate update'	2016-02-17 16:42:54 -08:00
SunZhaonan	c3da00003e	Fix bugs. Reenforce logging. Format jython scripts. Add missing table DDL.	2016-02-03 19:22:18 -08:00
SunZhaonan	05d54b3070	Add Hive metadata ETL process	2015-12-21 12:07:08 -08:00
SunZhaonan	cd44daba5d	Merge with master	2015-12-16 15:54:50 -08:00
SunZhaonan	07c46304b5	Fix bug of duplicate field loading. Fix bug of subflow process in azkaban lineage ETL.	2015-12-11 19:46:35 -08:00
Zhen Chen	e21a6b8e75	remove unused variable, change load script and add etl type in backend service	2015-12-11 13:52:23 -08:00
Zhen Chen	ebbf9ec629	add ldap user and group metadata etl	2015-12-10 16:26:57 -08:00
Zhen Chen	5a08134b8d	add dataset owner metadata etl	2015-12-07 15:17:01 -08:00
SunZhaonan	d5c3d87d00	Initial commit	2015-11-19 14:39:21 -08:00