yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-04 15:50:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,076 Commits 189 Branches 118 Tags
Commit Graph

12076 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yi (Alan) Wang
b4d949b4ba Merge pull request #227 from alyiwang/master 
Add MetadataInventoryEvent processor and API
 2016-09-15 14:02:22 -07:00
Yi Wang
ee01d7c6c7 rename DatasetPropertiesRecord to DatasetInventoryPropertiesRecord 2016-09-15 11:46:26 -07:00
Yi Wang
b136fc6c37 Add MetadataInventoryEvent processor and API 2016-09-15 09:22:42 -07:00
jerrybai2009
f7878cdfe4 fix the elastic search index out of gc issue (#223) 2016-09-13 16:43:48 -07:00
Eric Sun
86bf71499f Reformat the ETL job info message in log. (#222) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule

* continue process if hive sql parsor encounters error

* reformat etl job log message
 2016-09-13 14:01:14 -07:00
Yi (Alan) Wang
9543d091dd Merge pull request #221 from alyiwang/master 
Modify HdfsLoad to improve speed, Add process_id and hostname to wh_etl_job_execution
 2016-09-12 16:28:33 -07:00
Yi Wang
5ce5a1425e Add hostname and process_id to wh_etl_job_execution 2016-09-12 16:09:33 -07:00
Yi Wang
33e592da14 Modify HdfsLoad to improve speed 2016-09-09 17:41:13 -07:00
Yi (Alan) Wang
34fe3b26a2 Merge pull request #216 from alyiwang/master 
Add MatadataChangeEvent processor to call seperate APIs
 2016-09-09 17:16:14 -07:00
jerrybai2009
2e428e9b28 Merge pull request #220 from jerrybai2009/master 
fix the issue that can not delete and edit the comments
 2016-09-09 16:56:06 -07:00
jbai
cd942e852c fix the issue that can not delete and edit the comments 2016-09-09 10:42:46 -07:00
Yi Wang
a69d9b109a fix a typo in dict_field_detail DDL 2016-09-08 09:34:29 -07:00
Yi Wang
5515cbdde9 Add MatadataChangeEvent processor to call seperate APIs 2016-09-06 16:41:50 -07:00
Yi (Alan) Wang
d04abe78c8 Merge pull request #212 from alyiwang/master 
Map scm repo owner to dataset owner table
 2016-09-06 11:19:24 -07:00
Yi Wang
4c500402fe Map repo owner fix, change 'main' to 'Producer' and reset sort id 2016-09-02 13:52:00 -07:00
jerrybai2009
9bbbf1a68e Merge pull request #213 from jerrybai2009/master 
fix the issue that action getSchema did not fired
 2016-09-02 11:13:22 -07:00
Eric Sun
0ac00e1af3 Update README.md 2016-09-02 09:35:13 -07:00
Douglas Moore
d44c194529 Backend service readme (#215) 
* Update README.md

* Update README

* Rename README to README.me

* Rename README.me to README.md

* Update README.md
 2016-09-02 09:31:52 -07:00
Yi Wang
a809b0ac47 Map repo owner fix to use dataset group mapping 2016-09-01 18:19:41 -07:00
jbai
f4f4538408 fix the issue that action getSchema did not fired 2016-08-31 16:50:20 -07:00
Yi Wang
81f891bfab Map scm repo owner to dataset owner table 2016-08-30 15:35:28 -07:00
Yi (Alan) Wang
b8e9ff5a7c Merge pull request #209 from alyiwang/master 
Update DatasetOwnerRecord to be compatible with linkedin branch
 2016-08-29 17:24:06 -07:00
Yi Wang
e2b42d2ccb Update DatasetOwnerRecord to be compatible with linkedin branch 2016-08-25 09:12:31 -07:00
Yi Wang
183a9dcb6d Merge branch 'master' of https://github.com/linkedin/WhereHows 2016-08-24 09:20:27 -07:00
Yi (Alan) Wang
579b8fc9d7 Add metadataChangeEvent APIs to backend-service (#205) 
* Add multiproduct and git repo metadata etl job

* Extract commit hash use it when querying acl

* Use FileWriter to write records into CSV file

* Remove unnecessary log entries from kafka processor

* Fix the incompatibility between integer repo_id in db and string field in record

* merge API tables to existing dataset owner and schema field table

* Add confidential and recursive column to dict_dataset_field
 2016-08-24 09:10:35 -07:00
Yi Wang
7cbda15b5a Add confidential and recursive column to dict_dataset_field 2016-08-23 15:50:30 -07:00
Yi Wang
d46a9d8b8e merge API tables to existing dataset owner and schema field table 2016-08-22 17:06:20 -07:00
Yi Wang
46871face6 Add metadataChangeEvent APIs to backend-service 2016-08-16 18:47:53 -07:00
jerrybai2009
066c5f2ca5 Merge pull request #204 from jerrybai2009/master 
add the ui test for javascript code
 2016-08-16 15:47:33 -07:00
jbai
2389b6034a add the ui test for javascript code 2016-08-15 23:35:37 -07:00
Yi Wang
38ae6a0276 Merge branch 'master' of https://github.com/linkedin/WhereHows 2016-08-12 12:27:20 -07:00
Yi (Alan) Wang
078e90e8bd Add multiproduct and git repo metadata etl job (#202) 
* Add multiproduct and git repo metadata etl job

* implement the dataset availability section

* Extract commit hash use it when querying acl

* Use FileWriter to write records into CSV file

* Remove unnecessary log entries from kafka processor

* Fix the incompatibility between integer repo_id in db and string field in record
 2016-08-12 12:26:55 -07:00
Yi Wang
44807f5f7e Fix the incompatibility between integer repo_id in db and string field in record 2016-08-10 17:24:03 -07:00
Yi Wang
11158e0b9f Remove unnecessary log entries from kafka processor 2016-08-10 11:23:39 -07:00
Yi Wang
bc276274ff Use FileWriter to write records into CSV file 2016-08-10 11:20:31 -07:00
Yi Wang
83834e4e88 Add fetching acl owner info from svn, also change some property names. 2016-08-10 09:11:37 -07:00
jerrybai2009
162892a9e8 Merge pull request #203 from jerrybai2009/master 
implement the dataset accessiblities section
 2016-08-09 16:25:57 -07:00
Yi Wang
4689160dbb Extract commit hash use it when querying acl 2016-08-09 13:05:59 -07:00
jbai
45c528c9d9 implement the dataset accessiblities section 2016-08-09 11:34:46 -07:00
Yi Wang
830413e122 Add multiproduct and git repo metadata etl job 2016-08-08 21:28:37 -07:00
Eric Sun
cd4853d0a5 Use ProcessBuilder and redirected log file for HDFS Extract (#198) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule
 2016-08-08 14:02:34 -07:00
jerrybai2009
39cec22e25 Merge pull request #197 from jerrybai2009/master 
upgrade ember from 1.12 to 2.6.2
 2016-08-05 09:52:03 -07:00
jbai
5305124d8c fix the bind-attr and wrong flow link in search result issues 2016-08-04 18:46:33 -07:00
jbai
23910971a9 upgrade ember from 1.12 to 2.6.2 2016-08-04 17:44:47 -07:00
Eric Sun
c4d1605a0c Merge pull request #196 from alyiwang/master 
Modify Kafka Master to handle more than one Kafka connection configurations
Add additional error handling when starting the service

With this change in place.
- each Kafka Zookeeper requires a corresponding entry defined in wh_etl_job
- the connection info (such as Zookeeper, SchemaRegistery, topic to staging table mapping...) are configured in wh_etl_job_property
- kafka.consumer.etl.jobid in application.conf will determine if such Kafka job will be launched when the backend-service starts
 2016-08-04 14:26:59 -07:00
Yi Wang
c0cfe1f5ca Modify KafkaConsumerMaster to handle more than one kafka config, add error handling 2016-08-04 13:07:19 -07:00
Eric Sun
ef584552be Merge pull request #194 from alyiwang/master 
Get cluster info from cfg_cluster and format kafka events cluster field
 2016-08-03 20:24:07 -07:00
Yi Wang
3d3b2a8075 Get kafka job id from applicatoin.conf and then get ref_id and configs from DB 2016-08-03 18:55:07 -07:00
Yi Wang
dbbdb6e2fb Modify Oracle metadata ETL job, use Json dumps and remove unnecessary quotes 2016-08-03 18:49:00 -07:00
jerrybai2009
b4a718efd0 Merge pull request #195 from ericsun2/master 
temp fix for hdfs_schema_crawler getRuntime().exec() hangs problem
 2016-08-03 18:15:43 -07:00