Yi (Alan) Wang
b6e644fbb1
Optimize dataset load scripts, improve speed ( #350 )
...
- When loading dataset fields in staging table, populate the dateset_id field first then use this in later JOIN.
- When JOIN two big tables such as dict_field_detail, use pre-select to reduce table JOIN size and DB resource.
- Refactor some SQL code.
- Modify logback setting to better capture log time.
- Remove unnecessary config in backend application.conf
2017-03-22 10:23:30 -07:00
camelliazhang
fe1e698b8a
remove hive instance hardcode cluster name ( #236 )
2016-09-30 17:15:43 -07:00
Eric Sun
86bf71499f
Reformat the ETL job info message in log. ( #222 )
...
* Use ProcessBuilder and redirected log file for HDFS Extract
* relax urn validation rule
* continue process if hive sql parsor encounters error
* reformat etl job log message
2016-09-13 14:01:14 -07:00
Eric Sun
cd4853d0a5
Use ProcessBuilder and redirected log file for HDFS Extract ( #198 )
...
* Use ProcessBuilder and redirected log file for HDFS Extract
* relax urn validation rule
2016-08-08 14:02:34 -07:00
Eric Sun
1cd5872369
temp fix for hdfs_schema_crawler getRuntime().exec() hangs problem; exclude log4j
2016-08-03 15:50:00 -07:00
jbai
9fb5b09bd2
update dependency property name and fix the duplicated key issue when update cfg_object_name_map table
2016-07-20 19:07:16 -07:00
jbai
f3c299480f
update the column names from schema to schema_text and view_expanded_text to ddl_text
2016-07-20 18:01:25 -07:00
jbai
33b05cde4b
tracking the dalids schema and expanded text by versions
2016-07-20 15:59:11 -07:00
jbai
9166db7563
update the dict_dataset_instance data loading sql since table key changed
2016-06-29 18:00:10 -07:00
jbai
5062fa4ecc
load dali depends on and instance into final table
2016-06-09 18:51:05 -07:00
jbai
af976c3d5e
Dali Metadata integration - combine dali versions into one node
2016-06-02 18:29:44 -07:00
SunZhaonan
fb1198e4ae
Fix hive field ETL bug
2016-05-20 17:52:45 -07:00
SunZhaonan
0b5c421311
Fix Hive column parser parent path bug
2016-05-19 16:36:30 -07:00
SunZhaonan
b202832741
Innodb engine DDL. Add config for timeout and load sample.
2016-04-05 12:43:02 -07:00
SunZhaonan
c66b00e2f6
Fix dataset insert API bug. Fix load sql bug.
2016-03-28 16:27:43 -07:00
SunZhaonan
a0b7cb9d57
Fix process hanging bug. Add hive field ETL process.
2016-03-16 19:12:21 -07:00
SunZhaonan
c4671d2579
Add field comments ETL
...
Fix API bug of tech_matrix_id
Add key in comment table
2016-03-14 14:23:33 -07:00
SunZhaonan
c3da00003e
Fix bugs. Reenforce logging. Format jython scripts. Add missing table DDL.
2016-02-03 19:22:18 -08:00
SunZhaonan
0eddd8accf
close connection appropriatly, minior fix
2015-12-21 12:07:08 -08:00
SunZhaonan
05d54b3070
Add Hive metadata ETL process
2015-12-21 12:07:08 -08:00
