yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-19 07:30:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
937 Commits 200 Branches 121 Tags
Commit Graph

66 Commits

Author SHA1 Message Date
Mars Lan
d57bce2c0b Redirect ETL job's stderr & stdout to files to make debugging easier. (#465) 2017-07-10 13:42:54 -07:00
Mars Lan
f5a7e0c9ec Make sure all intermediate directories are created for ETL job property files. (#450) 2017-07-10 13:42:51 -07:00
Mars Lan
fda572dd8a Allow the logback directory for ETL jobs to be overridden using system property (#448) 
* Allow the logback directory for ETL jobs to be overridden using system property.

See https://logback.qos.ch/manual/configuration.html#variableSubstitution for more details.

* Add WHZ_ETL_TEMP_DIR env var and play config to control where the ETL job logs & temp files to be saved.

This enables us to move away from the default /var/tmp/wherehows directory.
 2017-07-10 13:42:16 -07:00
Mars Lan
c75fa5e6dc Use environmental variables to set ETL & Kafka job IDs. (#418) 
This will allow us to set different job IDs in staging & production via cfg2.
 2017-07-10 09:57:51 -07:00
Mars Lan
cf4e157813 Read master key from environmental variable instead of from local fil… (#417) 
* Read master key from environmental variable instead of from local file. This would allow us to pass it in via cfg2 ultimiately.

* Move the env var name to Constant.java
 2017-07-10 09:55:16 -07:00
Mars Lan
11d6186fe6 Add healthcheck endpoint for frontend & backend. (#388) 2017-07-10 09:55:11 -07:00
Yi (Alan) Wang
8ede6f3314 Move logback.xml, modify etl job command generation (#364) 
- Move logback.xml in metadata-etl to etl_logback.xml under backend/conf to avoid multiple logback config in classpath. ETL jobs are able to write to their own log file again.
- Replace generated single string command with String[] and invoke Runtime.getRuntime().exec(String[])
 2017-07-10 09:54:20 -07:00
Yi (Alan) Wang
3360fe79cc Modify genearate java command to solve classpath issue (#362) 
Remove the single quote around classpath.
 2017-07-10 09:54:20 -07:00
Naga Srinivas Vemuri
803e3added Modify /dataset POST method to perform INSERT or UPDATE of the DatasetRecord 2017-07-10 09:53:25 -07:00
Yi (Alan) Wang
66a8eea21b Fix issues from Oracle MetadataChangeEvent integration (#336) 
* Fix issues from Oracle MetadataChangeEvent integration
 2017-03-14 17:19:30 -07:00
Yi (Alan) Wang
4f873a919a Fix bugs found by AppCheck in issue #328 (#335) 2017-02-24 14:20:56 -08:00
Yi Wang
14824c06bb Change sleep to 10s after etl job init error 2017-01-30 09:27:42 -08:00
Yi (Alan) Wang
665a5dbded Add retry for ETL jobs failed at initialization (#308) 2017-01-27 11:17:38 -08:00
Yi Wang
ea8f6e8551 Add retry for ETL jobs failed at initialization 2017-01-20 14:11:45 -08:00
Yi (Alan) Wang
e07306b51e Update MetadataChangeEvent, separate privacy compliance from security (#275) 2016-11-11 17:25:41 -08:00
Yi Wang
b4f5e438e2 Add JobExecutionLineageEvent and kafka processor 2016-11-08 19:11:37 -08:00
Yi Wang
664e4072bb Upgrade to play 2.4.8 2016-10-19 17:42:28 -07:00
Yi Wang
3227412339 Login authentication support multiple LDAP servers, add login history 2016-10-13 14:30:43 -07:00
Yi Wang
fcd6cf149e Update MetastoreAuditProcessor to reduce storage, also refactor some code 2016-10-11 11:26:36 -07:00
Yi Wang
5049c847fa Update Kafka consumer actors to reduce memory usage 2016-10-10 14:49:14 -07:00
Yi (Alan) Wang
c9dfb637af Update MetadataChangeEvent APIs according to schema change (#243) 
* Update MetadataChangeEvent APIs according to schema change

* Update MultiproductLoad to reflect new Owner types

* Add comments for Owner_type precedence (priority) and compliance
 2016-10-06 13:33:45 -07:00
jbai
a11e4908dc tracking the GobblinTrackingEvent_autit to get owner information 2016-09-29 15:01:32 -07:00
Yi Wang
ac34eb683f Update Kafka processor casting Object to String, also add debug info if can't fetch schema from Registery 2016-09-26 15:06:33 -07:00
Yi Wang
1ad2b1528e logback redirect ETL job logs into corresponding files 2016-09-23 16:54:52 -07:00
Yi (Alan) Wang
753de7de7c Merge pull request #233 from alyiwang/master 
Update backend APIs to cast SQL results back to Java record then to Json
 2016-09-21 08:59:04 -07:00
Eric Sun
89ff794ddf Add api to get dependents of a dataset (#232) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule

* continue process if hive sql parsor encounters error

* reformat etl job log message

* add API to find dataset dependents, such as which hive tables are based on an hdfs path
 2016-09-21 08:55:44 -07:00
Yi Wang
be65efb0cc Update backend APIs to cast SQL results back to Java record then serialize to Json reply 2016-09-20 18:56:49 -07:00
Yi Wang(Data Infrastructure)
1171e00097 Add REST proxy for Security API from backend to web 2016-09-19 18:14:10 -07:00
Yi Wang
b136fc6c37 Add MetadataInventoryEvent processor and API 2016-09-15 09:22:42 -07:00
Eric Sun
86bf71499f Reformat the ETL job info message in log. (#222) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule

* continue process if hive sql parsor encounters error

* reformat etl job log message
 2016-09-13 14:01:14 -07:00
Yi Wang
5ce5a1425e Add hostname and process_id to wh_etl_job_execution 2016-09-12 16:09:33 -07:00
Yi Wang
5515cbdde9 Add MatadataChangeEvent processor to call seperate APIs 2016-09-06 16:41:50 -07:00
Yi (Alan) Wang
579b8fc9d7 Add metadataChangeEvent APIs to backend-service (#205) 
* Add multiproduct and git repo metadata etl job

* Extract commit hash use it when querying acl

* Use FileWriter to write records into CSV file

* Remove unnecessary log entries from kafka processor

* Fix the incompatibility between integer repo_id in db and string field in record

* merge API tables to existing dataset owner and schema field table

* Add confidential and recursive column to dict_dataset_field
 2016-08-24 09:10:35 -07:00
Eric Sun
cd4853d0a5 Use ProcessBuilder and redirected log file for HDFS Extract (#198) 
* Use ProcessBuilder and redirected log file for HDFS Extract

* relax urn validation rule
 2016-08-08 14:02:34 -07:00
Yi Wang
c0cfe1f5ca Modify KafkaConsumerMaster to handle more than one kafka config, add error handling 2016-08-04 13:07:19 -07:00
Yi Wang
3d3b2a8075 Get kafka job id from applicatoin.conf and then get ref_id and configs from DB 2016-08-03 18:55:07 -07:00
Eric Sun
8c9cb99ba4 primary_dataset_type for cfg_database 2016-08-01 13:20:04 -07:00
Eric Sun
9d2c803f0c Merge pull request #187 from ericsun2/master 
Add datacenter, deploymenttier, cluster info to better describe dataset instance
 2016-07-28 17:22:32 -07:00
Eric Sun
f745642212 add datacenter, deploymenttier, cluster to describe dataset instance 2016-07-28 16:38:03 -07:00
Yi Wang
74ed769bab add Oracle dataset metadata ETL job 2016-07-28 14:07:07 -07:00
Yi Wang
7edacc9a9f get kafka config from wh_etl_job_property 2016-07-26 12:16:34 -07:00
Yi Wang
6d4706bc62 Ingest Gobblin tracking events into wherehows using Kafka consumer client 2016-07-25 15:03:29 -07:00
jbai
7a77aba4b7 merge the pull request 165 to master branch 2016-07-21 10:38:36 -07:00
Naga Srinivas Vemuri
97370ed2e1 Query Dataset properties to retrieve datasetUrns 2016-07-21 11:54:47 +05:30
jbai
6af54658d6 merge Fetching dataset watchers via get /dataset/watchers to main branch 2016-06-30 10:20:54 -07:00
jbai
6974ae26ae fix the gradlew check failed issue and make the cluster name input is mandatory 2016-06-15 11:12:20 -07:00
jbai
9705a07ad8 provide the dataset dependency api 2016-06-14 16:17:24 -07:00
jbai
e7885f28db comment out the flow tree builder since UI does not use it anymore 2016-06-13 10:15:32 -07:00
Rafal Kluszczynski
d06bfbfbd0 fix: use correctly directory path from properties when executing etl job 2016-05-30 08:59:05 +02:00
SunZhaonan
0b5c421311 Fix Hive column parser parent path bug 2016-05-19 16:36:30 -07:00