8 Commits

Author SHA1 Message Date
Mars Lan
d57bce2c0b Redirect ETL job's stderr & stdout to files to make debugging easier. (#465) 2017-07-10 13:42:54 -07:00
Mars Lan
f5a7e0c9ec Make sure all intermediate directories are created for ETL job property files. (#450) 2017-07-10 13:42:51 -07:00
Mars Lan
fda572dd8a Allow the logback directory for ETL jobs to be overridden using system property (#448)
* Allow the logback directory for ETL jobs to be overridden using system property.

See https://logback.qos.ch/manual/configuration.html#variableSubstitution for more details.

* Add WHZ_ETL_TEMP_DIR env var and play config to control where the ETL job logs & temp files to be saved.

This enables us to move away from the default /var/tmp/wherehows directory.
2017-07-10 13:42:16 -07:00
Yi (Alan) Wang
8ede6f3314 Move logback.xml, modify etl job command generation (#364)
- Move logback.xml in metadata-etl to etl_logback.xml under backend/conf to avoid multiple logback config in classpath. ETL jobs are able to write to their own log file again.
- Replace generated single string command with String[] and invoke Runtime.getRuntime().exec(String[])
2017-07-10 09:54:20 -07:00
Yi (Alan) Wang
3360fe79cc Modify genearate java command to solve classpath issue (#362)
Remove the single quote around classpath.
2017-07-10 09:54:20 -07:00
Yi Wang
1ad2b1528e logback redirect ETL job logs into corresponding files 2016-09-23 16:54:52 -07:00
Rafal Kluszczynski
d06bfbfbd0 fix: use correctly directory path from properties when executing etl job 2016-05-30 08:59:05 +02:00
SunZhaonan
31de21ddcf pass parameter through file. 2016-05-03 16:25:56 -07:00