Yi (Alan) Wang
e6978b97a7
Fix null comparison in SQL, refactor some queries ( #408 )
2017-04-04 13:02:46 -07:00
Yi (Alan) Wang
b6e644fbb1
Optimize dataset load scripts, improve speed ( #350 )
...
- When loading dataset fields in staging table, populate the dateset_id field first then use this in later JOIN.
- When JOIN two big tables such as dict_field_detail, use pre-select to reduce table JOIN size and DB resource.
- Refactor some SQL code.
- Modify logback setting to better capture log time.
- Remove unnecessary config in backend application.conf
2017-03-22 10:23:30 -07:00
Yi Wang
4bc63cd4bd
Fix bug for Teradata Load job, not loading samples on days that won't extract sample data.
2017-01-18 17:01:33 -08:00
jbai
ea1ac0da9f
load the teradata and hadoop data into table dict_dataset_instance
2016-07-29 10:59:33 -07:00
SunZhaonan
b202832741
Innodb engine DDL. Add config for timeout and load sample.
2016-04-05 12:43:02 -07:00
SunZhaonan
c4671d2579
Add field comments ETL
...
Fix API bug of tech_matrix_id
Add key in comment table
2016-03-14 14:23:33 -07:00
SunZhaonan
c3da00003e
Fix bugs. Reenforce logging. Format jython scripts. Add missing table DDL.
2016-02-03 19:22:18 -08:00
SunZhaonan
cd44daba5d
Merge with master
2015-12-16 15:54:50 -08:00
Zhen Chen
9ebe0b22ad
move close connection into finally block
2015-12-16 14:47:03 -08:00
SunZhaonan
57a09c64f6
Format sql
2015-12-16 12:10:06 -08:00
SunZhaonan
7bf3ecdb4a
Change flowPath, jobName format.
...
Fix sql nullable value comparison bug.
Formating.
2015-12-15 17:15:43 -08:00
SunZhaonan
07c46304b5
Fix bug of duplicate field loading. Fix bug of subflow process in azkaban lineage ETL.
2015-12-11 19:46:35 -08:00
SunZhaonan
d5c3d87d00
Initial commit
2015-11-19 14:39:21 -08:00
