Yi Wang
8ab5c824b0
Update MultiproductLoad to reflect new Owner types
2016-10-03 18:39:21 -07:00
Yi Wang
b74d58a33f
Update MetadataChangeEvent APIs according to schema change
2016-10-03 10:56:23 -07:00
camelliazhang
fe1e698b8a
remove hive instance hardcode cluster name ( #236 )
2016-09-30 17:15:43 -07:00
Na Zhang
10339690a9
Update HiveTransform and HiveLoad, remove hardcoded cluster name
2016-09-30 16:59:59 -07:00
Eric Sun
fd3b4baef8
avoid loop in LDAP org hierarchy ( #242 )
2016-09-30 16:45:38 -07:00
jerrybai2009
5f0426ea6b
using the dynamic cursor to reduce the memory usage ( #241 )
2016-09-30 16:45:17 -07:00
jerrybai2009
2f1b31df35
Merge pull request #240 from jerrybai2009/master
...
tracking the GobblinTrackingEvent_autit to get owner information
2016-09-29 16:13:14 -07:00
jbai
a11e4908dc
tracking the GobblinTrackingEvent_autit to get owner information
2016-09-29 15:01:32 -07:00
jerrybai2009
137d48a9b0
Merge pull request #237 from alyiwang/master
...
Update Kafka processor casting Object to String
2016-09-26 16:44:39 -07:00
Yi Wang
ac34eb683f
Update Kafka processor casting Object to String, also add debug info if can't fetch schema from Registery
2016-09-26 15:06:33 -07:00
Na Zhang
5c76f47313
remove hive instance hardcode cluster name
2016-09-26 15:06:30 -07:00
Yi (Alan) Wang
bc06c1a882
Merge pull request #234 from alyiwang/master
...
logback redirect ETL job logs into corresponding files
2016-09-24 14:43:34 -07:00
Yi Wang
1ad2b1528e
logback redirect ETL job logs into corresponding files
2016-09-23 16:54:52 -07:00
Yi (Alan) Wang
753de7de7c
Merge pull request #233 from alyiwang/master
...
Update backend APIs to cast SQL results back to Java record then to Json
2016-09-21 08:59:04 -07:00
Eric Sun
89ff794ddf
Add api to get dependents of a dataset ( #232 )
...
* Use ProcessBuilder and redirected log file for HDFS Extract
* relax urn validation rule
* continue process if hive sql parsor encounters error
* reformat etl job log message
* add API to find dataset dependents, such as which hive tables are based on an hdfs path
2016-09-21 08:55:44 -07:00
Yi Wang
be65efb0cc
Update backend APIs to cast SQL results back to Java record then serialize to Json reply
2016-09-20 18:56:49 -07:00
jerrybai2009
9db27f7048
Merge pull request #229 from jerrybai2009/master
...
update the dataset ownership information
2016-09-20 14:02:35 -07:00
Yi (Alan) Wang
c9a9d3a330
Merge pull request #231 from alyiwang/master
...
Add REST proxy for Security API from backend to web
2016-09-19 21:36:26 -07:00
Yi Wang(Data Infrastructure)
1171e00097
Add REST proxy for Security API from backend to web
2016-09-19 18:14:10 -07:00
jbai
44df5dd08a
update the dataset ownership information
2016-09-19 16:10:23 -07:00
Daniel Grove
937637c3ea
Add Database to create_tables readme ( #225 )
...
* Add Database to create_tables readme
* Add -D to signify database
2016-09-16 09:18:49 -07:00
Yi (Alan) Wang
b4d949b4ba
Merge pull request #227 from alyiwang/master
...
Add MetadataInventoryEvent processor and API
2016-09-15 14:02:22 -07:00
Yi Wang
ee01d7c6c7
rename DatasetPropertiesRecord to DatasetInventoryPropertiesRecord
2016-09-15 11:46:26 -07:00
Yi Wang
b136fc6c37
Add MetadataInventoryEvent processor and API
2016-09-15 09:22:42 -07:00
jerrybai2009
f7878cdfe4
fix the elastic search index out of gc issue ( #223 )
2016-09-13 16:43:48 -07:00
Eric Sun
86bf71499f
Reformat the ETL job info message in log. ( #222 )
...
* Use ProcessBuilder and redirected log file for HDFS Extract
* relax urn validation rule
* continue process if hive sql parsor encounters error
* reformat etl job log message
2016-09-13 14:01:14 -07:00
Yi (Alan) Wang
9543d091dd
Merge pull request #221 from alyiwang/master
...
Modify HdfsLoad to improve speed, Add process_id and hostname to wh_etl_job_execution
2016-09-12 16:28:33 -07:00
Yi Wang
5ce5a1425e
Add hostname and process_id to wh_etl_job_execution
2016-09-12 16:09:33 -07:00
Yi Wang
33e592da14
Modify HdfsLoad to improve speed
2016-09-09 17:41:13 -07:00
Yi (Alan) Wang
34fe3b26a2
Merge pull request #216 from alyiwang/master
...
Add MatadataChangeEvent processor to call seperate APIs
2016-09-09 17:16:14 -07:00
jerrybai2009
2e428e9b28
Merge pull request #220 from jerrybai2009/master
...
fix the issue that can not delete and edit the comments
2016-09-09 16:56:06 -07:00
jbai
cd942e852c
fix the issue that can not delete and edit the comments
2016-09-09 10:42:46 -07:00
Yi Wang
a69d9b109a
fix a typo in dict_field_detail DDL
2016-09-08 09:34:29 -07:00
Yi Wang
5515cbdde9
Add MatadataChangeEvent processor to call seperate APIs
2016-09-06 16:41:50 -07:00
Yi (Alan) Wang
d04abe78c8
Merge pull request #212 from alyiwang/master
...
Map scm repo owner to dataset owner table
2016-09-06 11:19:24 -07:00
Yi Wang
4c500402fe
Map repo owner fix, change 'main' to 'Producer' and reset sort id
2016-09-02 13:52:00 -07:00
jerrybai2009
9bbbf1a68e
Merge pull request #213 from jerrybai2009/master
...
fix the issue that action getSchema did not fired
2016-09-02 11:13:22 -07:00
Eric Sun
0ac00e1af3
Update README.md
2016-09-02 09:35:13 -07:00
Douglas Moore
d44c194529
Backend service readme ( #215 )
...
* Update README.md
* Update README
* Rename README to README.me
* Rename README.me to README.md
* Update README.md
2016-09-02 09:31:52 -07:00
Yi Wang
a809b0ac47
Map repo owner fix to use dataset group mapping
2016-09-01 18:19:41 -07:00
jbai
f4f4538408
fix the issue that action getSchema did not fired
2016-08-31 16:50:20 -07:00
Yi Wang
81f891bfab
Map scm repo owner to dataset owner table
2016-08-30 15:35:28 -07:00
Yi (Alan) Wang
b8e9ff5a7c
Merge pull request #209 from alyiwang/master
...
Update DatasetOwnerRecord to be compatible with linkedin branch
2016-08-29 17:24:06 -07:00
Yi Wang
e2b42d2ccb
Update DatasetOwnerRecord to be compatible with linkedin branch
2016-08-25 09:12:31 -07:00
Yi Wang
183a9dcb6d
Merge branch 'master' of https://github.com/linkedin/WhereHows
2016-08-24 09:20:27 -07:00
Yi (Alan) Wang
579b8fc9d7
Add metadataChangeEvent APIs to backend-service ( #205 )
...
* Add multiproduct and git repo metadata etl job
* Extract commit hash use it when querying acl
* Use FileWriter to write records into CSV file
* Remove unnecessary log entries from kafka processor
* Fix the incompatibility between integer repo_id in db and string field in record
* merge API tables to existing dataset owner and schema field table
* Add confidential and recursive column to dict_dataset_field
2016-08-24 09:10:35 -07:00
Yi Wang
7cbda15b5a
Add confidential and recursive column to dict_dataset_field
2016-08-23 15:50:30 -07:00
Yi Wang
d46a9d8b8e
merge API tables to existing dataset owner and schema field table
2016-08-22 17:06:20 -07:00
Yi Wang
46871face6
Add metadataChangeEvent APIs to backend-service
2016-08-16 18:47:53 -07:00
jerrybai2009
066c5f2ca5
Merge pull request #204 from jerrybai2009/master
...
add the ui test for javascript code
2016-08-16 15:47:33 -07:00
