13 Commits

Author SHA1 Message Date
Mars Lan
d57bce2c0b Redirect ETL job's stderr & stdout to files to make debugging easier. (#465) 2017-07-10 13:42:54 -07:00
Mars Lan
f5a7e0c9ec Make sure all intermediate directories are created for ETL job property files. (#450) 2017-07-10 13:42:51 -07:00
Mars Lan
fda572dd8a Allow the logback directory for ETL jobs to be overridden using system property (#448)
* Allow the logback directory for ETL jobs to be overridden using system property.

See https://logback.qos.ch/manual/configuration.html#variableSubstitution for more details.

* Add WHZ_ETL_TEMP_DIR env var and play config to control where the ETL job logs & temp files to be saved.

This enables us to move away from the default /var/tmp/wherehows directory.
2017-07-10 13:42:16 -07:00
Yi (Alan) Wang
8ede6f3314 Move logback.xml, modify etl job command generation (#364)
- Move logback.xml in metadata-etl to etl_logback.xml under backend/conf to avoid multiple logback config in classpath. ETL jobs are able to write to their own log file again.
- Replace generated single string command with String[] and invoke Runtime.getRuntime().exec(String[])
2017-07-10 09:54:20 -07:00
Mars Lan
e36a40cd65 Generate code coverage reports (#334)
* Add playCoverage task to run code coverage using JaCoco for backend and web.

* Add jacocoTestReport task to run code coverage for testNG-based tests in wherehows-common & metadata-etl.
2017-07-10 09:53:28 -07:00
Mars Lan
bcc3cd9f76 Make unit tests buildable again for backend and web (#325)
* Make unit tests buildable again for backend and web.

* Add back fest dependency so the tests can stay more of less the same as before.
2017-07-10 09:53:28 -07:00
Yi Wang
664e4072bb Upgrade to play 2.4.8 2016-10-19 17:42:28 -07:00
jbai
9705a07ad8 provide the dataset dependency api 2016-06-14 16:17:24 -07:00
Rafal Kluszczynski
5f25e6a6b5 refactor: test assertions more readable 2016-05-30 12:39:15 +02:00
Rafal Kluszczynski
14e5ec497d test: check properties file and etl job command generation 2016-05-30 11:09:36 +02:00
SunZhaonan
31de21ddcf pass parameter through file. 2016-05-03 16:25:56 -07:00
SunZhaonan
5e9ae37952 Change to multi processing instead of multi thread. Fix hive ETL bug 2016-02-29 16:37:03 -08:00
SunZhaonan
d5c3d87d00 Initial commit 2015-11-19 14:39:21 -08:00