yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-23 17:39:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,664 Commits 205 Branches 123 Tags
Commit Graph

117 Commits

Author SHA1 Message Date
Eric Sun
a3504fa57f Fix jsonpath after upgrading com.jayway.jsonpath to 2.2 (#299) 
* use schema_url_helper to fetch avro schema from hdfs or http location

* trim space

* add dfs.namenode.kerberos.principal.pattern; include htrace for SchemaUrlHelper

* fix jsonpath for job history log parser; do not throw exception if kerberos config files are missing for job history http connection

* avoid null return value for sepCommaString(); fix a typo
 2016-12-13 21:14:52 -08:00
Yi (Alan) Wang
e07306b51e Update MetadataChangeEvent, separate privacy compliance from security (#275) 2016-11-11 17:25:41 -08:00
Yi Wang
b4f5e438e2 Add JobExecutionLineageEvent and kafka processor 2016-11-08 19:11:37 -08:00
Eric Sun
7b36d09b58 Add get_schema_literal_from_url() to fetch schema literal based on schema url (#268) 
* use schema_url_helper to fetch avro schema from hdfs or http location

* trim space

* add dfs.namenode.kerberos.principal.pattern; include htrace for SchemaUrlHelper
 2016-11-07 08:14:45 -08:00
Yi Wang
664e4072bb Upgrade to play 2.4.8 2016-10-19 17:42:28 -07:00
Na Zhang
043dc25e89 Get owners for espresso and oracle, and fix a bug for teradata 2016-10-19 11:13:32 -07:00
Yi Wang
5049c847fa Update Kafka consumer actors to reduce memory usage 2016-10-10 14:49:14 -07:00
Yi Wang
c9f4f18d9c Update Azkaban_Execution job to fetch cronExpression in flow scheduling 2016-10-06 13:43:10 -07:00
Yi (Alan) Wang
c9dfb637af Update MetadataChangeEvent APIs according to schema change (#243) 
* Update MetadataChangeEvent APIs according to schema change

* Update MultiproductLoad to reflect new Owner types

* Add comments for Owner_type precedence (priority) and compliance
 2016-10-06 13:33:45 -07:00
Yi Wang
0356497124 Add comments for Owner_type precedence (priority) and compliance 2016-10-06 13:24:29 -07:00
Yi Wang
b74d58a33f Update MetadataChangeEvent APIs according to schema change 2016-10-03 10:56:23 -07:00
jbai
a11e4908dc tracking the GobblinTrackingEvent_autit to get owner information 2016-09-29 15:01:32 -07:00
Yi Wang
ac34eb683f Update Kafka processor casting Object to String, also add debug info if can't fetch schema from Registery 2016-09-26 15:06:33 -07:00
Yi (Alan) Wang
753de7de7c Merge pull request #233 from alyiwang/master 
Update backend APIs to cast SQL results back to Java record then to Json
 2016-09-21 08:59:04 -07:00
Eric Sun
89ff794ddf Add api to get dependents of a dataset (#232) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule

* continue process if hive sql parsor encounters error

* reformat etl job log message

* add API to find dataset dependents, such as which hive tables are based on an hdfs path
 2016-09-21 08:55:44 -07:00
Yi Wang
be65efb0cc Update backend APIs to cast SQL results back to Java record then serialize to Json reply 2016-09-20 18:56:49 -07:00
Yi Wang(Data Infrastructure)
1171e00097 Add REST proxy for Security API from backend to web 2016-09-19 18:14:10 -07:00
Yi Wang
ee01d7c6c7 rename DatasetPropertiesRecord to DatasetInventoryPropertiesRecord 2016-09-15 11:46:26 -07:00
Yi Wang
b136fc6c37 Add MetadataInventoryEvent processor and API 2016-09-15 09:22:42 -07:00
Eric Sun
86bf71499f Reformat the ETL job info message in log. (#222) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule

* continue process if hive sql parsor encounters error

* reformat etl job log message
 2016-09-13 14:01:14 -07:00
Yi Wang
5515cbdde9 Add MatadataChangeEvent processor to call seperate APIs 2016-09-06 16:41:50 -07:00
Yi Wang
81f891bfab Map scm repo owner to dataset owner table 2016-08-30 15:35:28 -07:00
Yi Wang
e2b42d2ccb Update DatasetOwnerRecord to be compatible with linkedin branch 2016-08-25 09:12:31 -07:00
Yi Wang
7cbda15b5a Add confidential and recursive column to dict_dataset_field 2016-08-23 15:50:30 -07:00
Yi Wang
d46a9d8b8e merge API tables to existing dataset owner and schema field table 2016-08-22 17:06:20 -07:00
Yi Wang
46871face6 Add metadataChangeEvent APIs to backend-service 2016-08-16 18:47:53 -07:00
Yi (Alan) Wang
078e90e8bd Add multiproduct and git repo metadata etl job (#202) 
* Add multiproduct and git repo metadata etl job

* implement the dataset availability section

* Extract commit hash use it when querying acl

* Use FileWriter to write records into CSV file

* Remove unnecessary log entries from kafka processor

* Fix the incompatibility between integer repo_id in db and string field in record
 2016-08-12 12:26:55 -07:00
Yi Wang
44807f5f7e Fix the incompatibility between integer repo_id in db and string field in record 2016-08-10 17:24:03 -07:00
Yi Wang
bc276274ff Use FileWriter to write records into CSV file 2016-08-10 11:20:31 -07:00
Yi Wang
83834e4e88 Add fetching acl owner info from svn, also change some property names. 2016-08-10 09:11:37 -07:00
Yi Wang
830413e122 Add multiproduct and git repo metadata etl job 2016-08-08 21:28:37 -07:00
Yi Wang
3d3b2a8075 Get kafka job id from applicatoin.conf and then get ref_id and configs from DB 2016-08-03 18:55:07 -07:00
Eric Sun
9d2c803f0c Merge pull request #187 from ericsun2/master 
Add datacenter, deploymenttier, cluster info to better describe dataset instance
 2016-07-28 17:22:32 -07:00
Eric Sun
f745642212 add datacenter, deploymenttier, cluster to describe dataset instance 2016-07-28 16:38:03 -07:00
Yi Wang
74ed769bab add Oracle dataset metadata ETL job 2016-07-28 14:07:07 -07:00
Yi Wang
6d4706bc62 Ingest Gobblin tracking events into wherehows using Kafka consumer client 2016-07-25 15:03:29 -07:00
jbai
0f5124579c fix the issue of datasetSchemaRecord expected 11 args but got 9 2016-07-21 17:39:22 -07:00
jbai
7a77aba4b7 merge the pull request 165 to master branch 2016-07-21 10:38:36 -07:00
jbai
9fb5b09bd2 update dependency property name and fix the duplicated key issue when update cfg_object_name_map table 2016-07-20 19:07:16 -07:00
jbai
33b05cde4b tracking the dalids schema and expanded text by versions 2016-07-20 15:59:11 -07:00
jbai
9705a07ad8 provide the dataset dependency api 2016-06-14 16:17:24 -07:00
jbai
e5880cf81a fix the merge conflict 2016-06-07 11:31:00 -07:00
jbai
af976c3d5e Dali Metadata integration - combine dali versions into one node 2016-06-02 18:29:44 -07:00
SunZhaonan
bec1c5cee0 Add local mode for hdfs extract 2016-05-31 14:44:32 -07:00
jbai
a2e42d60f3 add the elasticsearch index build and update file 2016-05-23 17:58:37 -07:00
Zhaonan Sun
a7187a42bf Merge pull request #108 from SunZhaonan/master 
Innodb engine DDL. Add config for timeout and load sample.
 2016-04-06 15:07:44 -07:00
Arkadiusz Osinski
4d9f1681f0 missing letter in property name hive.metastore.username 2016-04-06 08:46:33 +02:00
SunZhaonan
b202832741 Innodb engine DDL. Add config for timeout and load sample. 2016-04-05 12:43:02 -07:00
SunZhaonan
4a4894a192 Use Kerberos login 2016-03-17 12:31:58 -07:00
SunZhaonan
a0b7cb9d57 Fix process hanging bug. Add hive field ETL process. 2016-03-16 19:12:21 -07:00