16 Commits

Author SHA1 Message Date
Yi (Alan) Wang
e07306b51e Update MetadataChangeEvent, separate privacy compliance from security (#275) 2016-11-11 17:25:41 -08:00
Yi Wang
b4f5e438e2 Add JobExecutionLineageEvent and kafka processor 2016-11-08 19:11:37 -08:00
Yi Wang
664e4072bb Upgrade to play 2.4.8 2016-10-19 17:42:28 -07:00
Yi Wang
fcd6cf149e Update MetastoreAuditProcessor to reduce storage, also refactor some code 2016-10-11 11:26:36 -07:00
Yi (Alan) Wang
753de7de7c Merge pull request #233 from alyiwang/master
Update backend APIs to cast SQL results back to Java record then to Json
2016-09-21 08:59:04 -07:00
Eric Sun
89ff794ddf Add api to get dependents of a dataset (#232)
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule

* continue process if hive sql parsor encounters error

* reformat etl job log message

* add API to find dataset dependents, such as which hive tables are based on an hdfs path
2016-09-21 08:55:44 -07:00
Yi Wang
be65efb0cc Update backend APIs to cast SQL results back to Java record then serialize to Json reply 2016-09-20 18:56:49 -07:00
Yi Wang(Data Infrastructure)
1171e00097 Add REST proxy for Security API from backend to web 2016-09-19 18:14:10 -07:00
Yi Wang
b136fc6c37 Add MetadataInventoryEvent processor and API 2016-09-15 09:22:42 -07:00
Yi (Alan) Wang
579b8fc9d7 Add metadataChangeEvent APIs to backend-service (#205)
* Add multiproduct and git repo metadata etl job

* Extract commit hash use it when querying acl

* Use FileWriter to write records into CSV file

* Remove unnecessary log entries from kafka processor

* Fix the incompatibility between integer repo_id in db and string field in record

* merge API tables to existing dataset owner and schema field table

* Add confidential and recursive column to dict_dataset_field
2016-08-24 09:10:35 -07:00
jbai
7a77aba4b7 merge the pull request 165 to master branch 2016-07-21 10:38:36 -07:00
Naga Srinivas Vemuri
97370ed2e1 Query Dataset properties to retrieve datasetUrns 2016-07-21 11:54:47 +05:30
jbai
6af54658d6 merge Fetching dataset watchers via get /dataset/watchers to main branch 2016-06-30 10:20:54 -07:00
jbai
9705a07ad8 provide the dataset dependency api 2016-06-14 16:17:24 -07:00
SunZhaonan
5e9ae37952 Change to multi processing instead of multi thread. Fix hive ETL bug 2016-02-29 16:37:03 -08:00
SunZhaonan
d5c3d87d00 Initial commit 2015-11-19 14:39:21 -08:00