Mars Lan
b4fec37f61
Fix Kerberos authentication so that HIVE_DATASET_METADATA_ETL jobs can be run from non-grid cluster. ( #482 )
2017-07-10 13:42:56 -07:00
Na Zhang
4cf65fe57c
retrieve source modification time for hive and dalids
2017-07-10 13:42:55 -07:00
Yi (Alan) Wang
4b07be768c
Fix output file chmod mistake ( #468 )
2017-04-28 16:44:51 -07:00
Yi (Alan) Wang
ca4b079d5b
Fix Hive Extract job file writing issues, add more authentication log ( #419 )
...
Conflicts:
web/app/security/AuthenticationManager.java
2017-04-28 16:44:08 -07:00
Eric Sun
7b36d09b58
Add get_schema_literal_from_url() to fetch schema literal based on schema url ( #268 )
...
* use schema_url_helper to fetch avro schema from hdfs or http location
* trim space
* add dfs.namenode.kerberos.principal.pattern; include htrace for SchemaUrlHelper
2016-11-07 08:14:45 -08:00
jbai
33b05cde4b
tracking the dalids schema and expanded text by versions
2016-07-20 15:59:11 -07:00
jbai
0c68d9c4fb
fix the dataset field has duplicated records issue
2016-06-16 16:33:37 -07:00
jbai
38fdf1c132
fix the dalids dependency issue and add more log info in elasticsearch
2016-06-16 14:38:01 -07:00
jbai
5062fa4ecc
load dali depends on and instance into final table
2016-06-09 18:51:05 -07:00
jbai
f344f1cf2e
update code to follow the code review
2016-06-07 11:39:07 -07:00
jbai
e5880cf81a
fix the merge conflict
2016-06-07 11:31:00 -07:00
SunZhaonan
310e6e9f06
Catch hive table even though its schema is None or exception message
2016-06-03 11:30:08 -07:00
jbai
af976c3d5e
Dali Metadata integration - combine dali versions into one node
2016-06-02 18:29:44 -07:00
SunZhaonan
fb1198e4ae
Fix hive field ETL bug
2016-05-20 17:52:45 -07:00
SunZhaonan
a0b7cb9d57
Fix process hanging bug. Add hive field ETL process.
2016-03-16 19:12:21 -07:00
SunZhaonan
aff8f323e4
Scheduler check previous job is finished. Redirect remote outputstream into log. Fix avro parser bugs
2016-03-16 19:09:53 -07:00
SunZhaonan
6b024196cd
Fix Hive extract disorder bug. Add Hive database optional whitelist params
2016-03-16 19:09:52 -07:00
SunZhaonan
5e9ae37952
Change to multi processing instead of multi thread. Fix hive ETL bug
2016-02-29 16:37:03 -08:00
SunZhaonan
c3da00003e
Fix bugs. Reenforce logging. Format jython scripts. Add missing table DDL.
2016-02-03 19:22:18 -08:00
SunZhaonan
0eddd8accf
close connection appropriatly, minior fix
2015-12-21 12:07:08 -08:00
SunZhaonan
05d54b3070
Add Hive metadata ETL process
2015-12-21 12:07:08 -08:00
