yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-06 08:38:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,004 Commits 189 Branches 118 Tags
Commit Graph

12004 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yi (Alan) Wang
3bdaff934f Merge pull request #247 from alyiwang/master 
Update Kafka consumer actors to reduce memory usage
 2016-10-12 21:31:44 -07:00
Yi Wang
fcd6cf149e Update MetastoreAuditProcessor to reduce storage, also refactor some code 2016-10-11 11:26:36 -07:00
Yi Wang
5049c847fa Update Kafka consumer actors to reduce memory usage 2016-10-10 14:49:14 -07:00
Yi (Alan) Wang
286b93d7e8 Merge pull request #244 from alyiwang/master 
Update Azkaban_Execution job to fetch cronExpression in flow scheduling
 2016-10-06 17:22:10 -07:00
Yi Wang
c9f4f18d9c Update Azkaban_Execution job to fetch cronExpression in flow scheduling 2016-10-06 13:43:10 -07:00
Yi (Alan) Wang
c9dfb637af Update MetadataChangeEvent APIs according to schema change (#243) 
* Update MetadataChangeEvent APIs according to schema change

* Update MultiproductLoad to reflect new Owner types

* Add comments for Owner_type precedence (priority) and compliance
 2016-10-06 13:33:45 -07:00
Yi Wang
0356497124 Add comments for Owner_type precedence (priority) and compliance 2016-10-06 13:24:29 -07:00
Yi Wang
8ab5c824b0 Update MultiproductLoad to reflect new Owner types 2016-10-03 18:39:21 -07:00
Yi Wang
b74d58a33f Update MetadataChangeEvent APIs according to schema change 2016-10-03 10:56:23 -07:00
camelliazhang
fe1e698b8a remove hive instance hardcode cluster name (#236) 2016-09-30 17:15:43 -07:00
Na Zhang
10339690a9 Update HiveTransform and HiveLoad, remove hardcoded cluster name 2016-09-30 16:59:59 -07:00
Eric Sun
fd3b4baef8 avoid loop in LDAP org hierarchy (#242) 2016-09-30 16:45:38 -07:00
jerrybai2009
5f0426ea6b using the dynamic cursor to reduce the memory usage (#241) 2016-09-30 16:45:17 -07:00
jerrybai2009
2f1b31df35 Merge pull request #240 from jerrybai2009/master 
tracking the GobblinTrackingEvent_autit to get owner information
 2016-09-29 16:13:14 -07:00
jbai
a11e4908dc tracking the GobblinTrackingEvent_autit to get owner information 2016-09-29 15:01:32 -07:00
jerrybai2009
137d48a9b0 Merge pull request #237 from alyiwang/master 
Update Kafka processor casting Object to String
 2016-09-26 16:44:39 -07:00
Yi Wang
ac34eb683f Update Kafka processor casting Object to String, also add debug info if can't fetch schema from Registery 2016-09-26 15:06:33 -07:00
Na Zhang
5c76f47313 remove hive instance hardcode cluster name 2016-09-26 15:06:30 -07:00
Yi (Alan) Wang
bc06c1a882 Merge pull request #234 from alyiwang/master 
logback redirect ETL job logs into corresponding files
 2016-09-24 14:43:34 -07:00
Yi Wang
1ad2b1528e logback redirect ETL job logs into corresponding files 2016-09-23 16:54:52 -07:00
Yi (Alan) Wang
753de7de7c Merge pull request #233 from alyiwang/master 
Update backend APIs to cast SQL results back to Java record then to Json
 2016-09-21 08:59:04 -07:00
Eric Sun
89ff794ddf Add api to get dependents of a dataset (#232) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule

* continue process if hive sql parsor encounters error

* reformat etl job log message

* add API to find dataset dependents, such as which hive tables are based on an hdfs path
 2016-09-21 08:55:44 -07:00
Yi Wang
be65efb0cc Update backend APIs to cast SQL results back to Java record then serialize to Json reply 2016-09-20 18:56:49 -07:00
jerrybai2009
9db27f7048 Merge pull request #229 from jerrybai2009/master 
update the dataset ownership information
 2016-09-20 14:02:35 -07:00
Yi (Alan) Wang
c9a9d3a330 Merge pull request #231 from alyiwang/master 
Add REST proxy for Security API from backend to web
 2016-09-19 21:36:26 -07:00
Yi Wang(Data Infrastructure)
1171e00097 Add REST proxy for Security API from backend to web 2016-09-19 18:14:10 -07:00
jbai
44df5dd08a update the dataset ownership information 2016-09-19 16:10:23 -07:00
Daniel Grove
937637c3ea Add Database to create_tables readme (#225) 
* Add Database to create_tables readme

* Add -D to signify database
 2016-09-16 09:18:49 -07:00
Yi (Alan) Wang
b4d949b4ba Merge pull request #227 from alyiwang/master 
Add MetadataInventoryEvent processor and API
 2016-09-15 14:02:22 -07:00
Yi Wang
ee01d7c6c7 rename DatasetPropertiesRecord to DatasetInventoryPropertiesRecord 2016-09-15 11:46:26 -07:00
Yi Wang
b136fc6c37 Add MetadataInventoryEvent processor and API 2016-09-15 09:22:42 -07:00
jerrybai2009
f7878cdfe4 fix the elastic search index out of gc issue (#223) 2016-09-13 16:43:48 -07:00
Eric Sun
86bf71499f Reformat the ETL job info message in log. (#222) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule

* continue process if hive sql parsor encounters error

* reformat etl job log message
 2016-09-13 14:01:14 -07:00
Yi (Alan) Wang
9543d091dd Merge pull request #221 from alyiwang/master 
Modify HdfsLoad to improve speed, Add process_id and hostname to wh_etl_job_execution
 2016-09-12 16:28:33 -07:00
Yi Wang
5ce5a1425e Add hostname and process_id to wh_etl_job_execution 2016-09-12 16:09:33 -07:00
Yi Wang
33e592da14 Modify HdfsLoad to improve speed 2016-09-09 17:41:13 -07:00
Yi (Alan) Wang
34fe3b26a2 Merge pull request #216 from alyiwang/master 
Add MatadataChangeEvent processor to call seperate APIs
 2016-09-09 17:16:14 -07:00
jerrybai2009
2e428e9b28 Merge pull request #220 from jerrybai2009/master 
fix the issue that can not delete and edit the comments
 2016-09-09 16:56:06 -07:00
jbai
cd942e852c fix the issue that can not delete and edit the comments 2016-09-09 10:42:46 -07:00
Yi Wang
a69d9b109a fix a typo in dict_field_detail DDL 2016-09-08 09:34:29 -07:00
Yi Wang
5515cbdde9 Add MatadataChangeEvent processor to call seperate APIs 2016-09-06 16:41:50 -07:00
Yi (Alan) Wang
d04abe78c8 Merge pull request #212 from alyiwang/master 
Map scm repo owner to dataset owner table
 2016-09-06 11:19:24 -07:00
Yi Wang
4c500402fe Map repo owner fix, change 'main' to 'Producer' and reset sort id 2016-09-02 13:52:00 -07:00
jerrybai2009
9bbbf1a68e Merge pull request #213 from jerrybai2009/master 
fix the issue that action getSchema did not fired
 2016-09-02 11:13:22 -07:00
Eric Sun
0ac00e1af3 Update README.md 2016-09-02 09:35:13 -07:00
Douglas Moore
d44c194529 Backend service readme (#215) 
* Update README.md

* Update README

* Rename README to README.me

* Rename README.me to README.md

* Update README.md
 2016-09-02 09:31:52 -07:00
Yi Wang
a809b0ac47 Map repo owner fix to use dataset group mapping 2016-09-01 18:19:41 -07:00
jbai
f4f4538408 fix the issue that action getSchema did not fired 2016-08-31 16:50:20 -07:00
Yi Wang
81f891bfab Map scm repo owner to dataset owner table 2016-08-30 15:35:28 -07:00
Yi (Alan) Wang
b8e9ff5a7c Merge pull request #209 from alyiwang/master 
Update DatasetOwnerRecord to be compatible with linkedin branch
 2016-08-29 17:24:06 -07:00