Yi Wang
81f891bfab
Map scm repo owner to dataset owner table
2016-08-30 15:35:28 -07:00
Yi Wang
e2b42d2ccb
Update DatasetOwnerRecord to be compatible with linkedin branch
2016-08-25 09:12:31 -07:00
Yi Wang
7cbda15b5a
Add confidential and recursive column to dict_dataset_field
2016-08-23 15:50:30 -07:00
Yi Wang
d46a9d8b8e
merge API tables to existing dataset owner and schema field table
2016-08-22 17:06:20 -07:00
Yi Wang
46871face6
Add metadataChangeEvent APIs to backend-service
2016-08-16 18:47:53 -07:00
Yi (Alan) Wang
078e90e8bd
Add multiproduct and git repo metadata etl job ( #202 )
...
* Add multiproduct and git repo metadata etl job
* implement the dataset availability section
* Extract commit hash use it when querying acl
* Use FileWriter to write records into CSV file
* Remove unnecessary log entries from kafka processor
* Fix the incompatibility between integer repo_id in db and string field in record
2016-08-12 12:26:55 -07:00
Yi Wang
44807f5f7e
Fix the incompatibility between integer repo_id in db and string field in record
2016-08-10 17:24:03 -07:00
Yi Wang
bc276274ff
Use FileWriter to write records into CSV file
2016-08-10 11:20:31 -07:00
Yi Wang
83834e4e88
Add fetching acl owner info from svn, also change some property names.
2016-08-10 09:11:37 -07:00
Yi Wang
830413e122
Add multiproduct and git repo metadata etl job
2016-08-08 21:28:37 -07:00
Yi Wang
3d3b2a8075
Get kafka job id from applicatoin.conf and then get ref_id and configs from DB
2016-08-03 18:55:07 -07:00
Eric Sun
9d2c803f0c
Merge pull request #187 from ericsun2/master
...
Add datacenter, deploymenttier, cluster info to better describe dataset instance
2016-07-28 17:22:32 -07:00
Eric Sun
f745642212
add datacenter, deploymenttier, cluster to describe dataset instance
2016-07-28 16:38:03 -07:00
Yi Wang
74ed769bab
add Oracle dataset metadata ETL job
2016-07-28 14:07:07 -07:00
Yi Wang
6d4706bc62
Ingest Gobblin tracking events into wherehows using Kafka consumer client
2016-07-25 15:03:29 -07:00
jbai
0f5124579c
fix the issue of datasetSchemaRecord expected 11 args but got 9
2016-07-21 17:39:22 -07:00
jbai
7a77aba4b7
merge the pull request 165 to master branch
2016-07-21 10:38:36 -07:00
jbai
9fb5b09bd2
update dependency property name and fix the duplicated key issue when update cfg_object_name_map table
2016-07-20 19:07:16 -07:00
jbai
33b05cde4b
tracking the dalids schema and expanded text by versions
2016-07-20 15:59:11 -07:00
jbai
9705a07ad8
provide the dataset dependency api
2016-06-14 16:17:24 -07:00
jbai
e5880cf81a
fix the merge conflict
2016-06-07 11:31:00 -07:00
jbai
af976c3d5e
Dali Metadata integration - combine dali versions into one node
2016-06-02 18:29:44 -07:00
SunZhaonan
bec1c5cee0
Add local mode for hdfs extract
2016-05-31 14:44:32 -07:00
jbai
a2e42d60f3
add the elasticsearch index build and update file
2016-05-23 17:58:37 -07:00
Zhaonan Sun
a7187a42bf
Merge pull request #108 from SunZhaonan/master
...
Innodb engine DDL. Add config for timeout and load sample.
2016-04-06 15:07:44 -07:00
Arkadiusz Osinski
4d9f1681f0
missing letter in property name hive.metastore.username
2016-04-06 08:46:33 +02:00
SunZhaonan
b202832741
Innodb engine DDL. Add config for timeout and load sample.
2016-04-05 12:43:02 -07:00
SunZhaonan
4a4894a192
Use Kerberos login
2016-03-17 12:31:58 -07:00
SunZhaonan
a0b7cb9d57
Fix process hanging bug. Add hive field ETL process.
2016-03-16 19:12:21 -07:00
SunZhaonan
6b024196cd
Fix Hive extract disorder bug. Add Hive database optional whitelist params
2016-03-16 19:09:52 -07:00
SunZhaonan
4574e89de9
Fix bug of sample data schema inconsistant. Add clean up. Parameterize number of Actor
2016-02-22 17:40:07 -08:00
SunZhaonan
dfeefba213
Parameterize dataset source derived process.
2016-02-17 16:06:24 -08:00
SunZhaonan
de4d4cd0c1
Add documentation on important Constants and Classes
2016-02-12 16:57:12 -08:00
SunZhaonan
db593bb3bd
Add some sample properties in template
2016-02-09 16:27:31 -08:00
SunZhaonan
b5d7c38b7d
Eclipse integration. Resolve circular dependency. wherehows-common-test configure.
2016-02-09 15:50:49 -08:00
SunZhaonan
32e2547035
Add license to the new files
2015-12-21 14:10:30 -08:00
Zhen Chen
7fca60a527
add ref flow id for sub flow jobs
2015-12-17 16:26:15 -08:00
Zhen Chen
d6f77a6f06
fix ldap org hierachy is empty bug and minor changes
2015-12-16 17:01:55 -08:00
SunZhaonan
cd44daba5d
Merge with master
2015-12-16 15:54:50 -08:00
Zhen Chen
3c51365e12
parse the xml format of gitorious project page instead of the html page
2015-12-15 11:44:05 -08:00
SunZhaonan
07c46304b5
Fix bug of duplicate field loading. Fix bug of subflow process in azkaban lineage ETL.
2015-12-11 19:46:35 -08:00
Zhen Chen
af04ff6efc
add git file commit history etl
2015-12-11 11:02:29 -08:00
Zhen Chen
ebbf9ec629
add ldap user and group metadata etl
2015-12-10 16:26:57 -08:00
Zhen Chen
5a08134b8d
add dataset owner metadata etl
2015-12-07 15:17:01 -08:00
SunZhaonan
b83c0a4322
Exclude tests that need configurations
2015-11-19 14:56:20 -08:00
SunZhaonan
d5c3d87d00
Initial commit
2015-11-19 14:39:21 -08:00
