Mars Lan
d57bce2c0b
Redirect ETL job's stderr & stdout to files to make debugging easier. ( #465 )
2017-07-10 13:42:54 -07:00
Mars Lan
f5a7e0c9ec
Make sure all intermediate directories are created for ETL job property files. ( #450 )
2017-07-10 13:42:51 -07:00
Mars Lan
fda572dd8a
Allow the logback directory for ETL jobs to be overridden using system property ( #448 )
...
* Allow the logback directory for ETL jobs to be overridden using system property.
See https://logback.qos.ch/manual/configuration.html#variableSubstitution for more details.
* Add WHZ_ETL_TEMP_DIR env var and play config to control where the ETL job logs & temp files to be saved.
This enables us to move away from the default /var/tmp/wherehows directory.
2017-07-10 13:42:16 -07:00
Mars Lan
c75fa5e6dc
Use environmental variables to set ETL & Kafka job IDs. ( #418 )
...
This will allow us to set different job IDs in staging & production via cfg2.
2017-07-10 09:57:51 -07:00
Yi (Alan) Wang
8ede6f3314
Move logback.xml, modify etl job command generation ( #364 )
...
- Move logback.xml in metadata-etl to etl_logback.xml under backend/conf to avoid multiple logback config in classpath. ETL jobs are able to write to their own log file again.
- Replace generated single string command with String[] and invoke Runtime.getRuntime().exec(String[])
2017-07-10 09:54:20 -07:00
Yi (Alan) Wang
3360fe79cc
Modify genearate java command to solve classpath issue ( #362 )
...
Remove the single quote around classpath.
2017-07-10 09:54:20 -07:00
Yi Wang
14824c06bb
Change sleep to 10s after etl job init error
2017-01-30 09:27:42 -08:00
Yi (Alan) Wang
665a5dbded
Add retry for ETL jobs failed at initialization ( #308 )
2017-01-27 11:17:38 -08:00
Yi Wang
ea8f6e8551
Add retry for ETL jobs failed at initialization
2017-01-20 14:11:45 -08:00
Yi Wang
b4f5e438e2
Add JobExecutionLineageEvent and kafka processor
2016-11-08 19:11:37 -08:00
Yi Wang
664e4072bb
Upgrade to play 2.4.8
2016-10-19 17:42:28 -07:00
Yi Wang
3227412339
Login authentication support multiple LDAP servers, add login history
2016-10-13 14:30:43 -07:00
Yi Wang
fcd6cf149e
Update MetastoreAuditProcessor to reduce storage, also refactor some code
2016-10-11 11:26:36 -07:00
Yi Wang
5049c847fa
Update Kafka consumer actors to reduce memory usage
2016-10-10 14:49:14 -07:00
Yi Wang
ac34eb683f
Update Kafka processor casting Object to String, also add debug info if can't fetch schema from Registery
2016-09-26 15:06:33 -07:00
Yi Wang
1ad2b1528e
logback redirect ETL job logs into corresponding files
2016-09-23 16:54:52 -07:00
Yi Wang
5ce5a1425e
Add hostname and process_id to wh_etl_job_execution
2016-09-12 16:09:33 -07:00
Yi Wang
c0cfe1f5ca
Modify KafkaConsumerMaster to handle more than one kafka config, add error handling
2016-08-04 13:07:19 -07:00
Yi Wang
3d3b2a8075
Get kafka job id from applicatoin.conf and then get ref_id and configs from DB
2016-08-03 18:55:07 -07:00
Yi Wang
74ed769bab
add Oracle dataset metadata ETL job
2016-07-28 14:07:07 -07:00
Yi Wang
7edacc9a9f
get kafka config from wh_etl_job_property
2016-07-26 12:16:34 -07:00
Yi Wang
6d4706bc62
Ingest Gobblin tracking events into wherehows using Kafka consumer client
2016-07-25 15:03:29 -07:00
jbai
e7885f28db
comment out the flow tree builder since UI does not use it anymore
2016-06-13 10:15:32 -07:00
Rafal Kluszczynski
d06bfbfbd0
fix: use correctly directory path from properties when executing etl job
2016-05-30 08:59:05 +02:00
SunZhaonan
0b5c421311
Fix Hive column parser parent path bug
2016-05-19 16:36:30 -07:00
SunZhaonan
9d6a1b2649
Add optional config of ETL job white list
2016-05-12 16:28:23 -07:00
SunZhaonan
31de21ddcf
pass parameter through file.
2016-05-03 16:25:56 -07:00
SunZhaonan
c66b00e2f6
Fix dataset insert API bug. Fix load sql bug.
2016-03-28 16:27:43 -07:00
SunZhaonan
a0b7cb9d57
Fix process hanging bug. Add hive field ETL process.
2016-03-16 19:12:21 -07:00
SunZhaonan
aff8f323e4
Scheduler check previous job is finished. Redirect remote outputstream into log. Fix avro parser bugs
2016-03-16 19:09:53 -07:00
SunZhaonan
5e9ae37952
Change to multi processing instead of multi thread. Fix hive ETL bug
2016-02-29 16:37:03 -08:00
SunZhaonan
c3da00003e
Fix bugs. Reenforce logging. Format jython scripts. Add missing table DDL.
2016-02-03 19:22:18 -08:00
SunZhaonan
05d54b3070
Add Hive metadata ETL process
2015-12-21 12:07:08 -08:00
Zhen Chen
e21a6b8e75
remove unused variable, change load script and add etl type in backend service
2015-12-11 13:52:23 -08:00
Zhen Chen
ebbf9ec629
add ldap user and group metadata etl
2015-12-10 16:26:57 -08:00
Zhen Chen
5a08134b8d
add dataset owner metadata etl
2015-12-07 15:17:01 -08:00
SunZhaonan
d5c3d87d00
Initial commit
2015-11-19 14:39:21 -08:00
