yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-10-30 18:26:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
563 Commits 395 Branches 129 Tags
Commit Graph

65 Commits

Author SHA1 Message Date
jbai
75b4d10526 add the table job attempt source code 2017-07-10 09:53:26 -07:00
Matthew Ropp
b0626fe957 myql57_pk_not_null - primary key columns must be not null (#378) 2017-03-30 08:44:18 -07:00
Yi (Alan) Wang
91926f9540 Fix table creation DDL bug for MySQL 5.7 (#375) 
Primary key fields needs to be "NOT NULL".
 2017-03-29 15:57:34 -07:00
Yi (Alan) Wang
d9438dc5bc Fix SQL query issues when upgrading to MySQL 5.7 (#345) 2017-03-15 09:22:46 -07:00
Yi (Alan) Wang
66a8eea21b Fix issues from Oracle MetadataChangeEvent integration (#336) 
* Fix issues from Oracle MetadataChangeEvent integration
 2017-03-14 17:19:30 -07:00
Yi (Alan) Wang
89c3b93d6f Fix MySQL DDL error for stg_dict_dataset_instance (#314) 2017-01-27 11:35:41 -08:00
camelliazhang
724f754f03 clean and refactor elastic serach ETL job (#300) 2016-12-14 21:22:30 -08:00
Yi Wang
51f911b400 Map git repo and owners to Oracle/espresso/dali datasets 2016-11-22 10:51:00 -08:00
Yi Wang
6310d6c0e0 Update etl_config_tables DDL, add default timestamps 2016-11-14 15:23:55 -08:00
Yi (Alan) Wang
e07306b51e Update MetadataChangeEvent, separate privacy compliance from security (#275) 2016-11-11 17:25:41 -08:00
Yi Wang
7a76bc5f59 Update MetadataChangeEvent, separate privacy compliance from security 2016-11-11 17:10:17 -08:00
Yi Wang
b4f5e438e2 Add JobExecutionLineageEvent and kafka processor 2016-11-08 19:11:37 -08:00
Stephane Maarek
a32a0e4b7f last_modified column has a default for the table cfg_database (#269) 
@alyiwang will also touch a few other cfg_??? tables for the default values
 2016-11-08 17:45:27 -08:00
Na Zhang
217b7d9d09 search ranking improvement with static boosting 2016-11-08 15:18:51 -08:00
Na Zhang
043dc25e89 Get owners for espresso and oracle, and fix a bug for teradata 2016-10-19 11:13:32 -07:00
Yi Wang
3227412339 Login authentication support multiple LDAP servers, add login history 2016-10-13 14:30:43 -07:00
Yi Wang
c9f4f18d9c Update Azkaban_Execution job to fetch cronExpression in flow scheduling 2016-10-06 13:43:10 -07:00
Yi (Alan) Wang
c9dfb637af Update MetadataChangeEvent APIs according to schema change (#243) 
* Update MetadataChangeEvent APIs according to schema change

* Update MultiproductLoad to reflect new Owner types

* Add comments for Owner_type precedence (priority) and compliance
 2016-10-06 13:33:45 -07:00
Yi Wang
0356497124 Add comments for Owner_type precedence (priority) and compliance 2016-10-06 13:24:29 -07:00
Yi Wang
8ab5c824b0 Update MultiproductLoad to reflect new Owner types 2016-10-03 18:39:21 -07:00
Yi Wang
b74d58a33f Update MetadataChangeEvent APIs according to schema change 2016-10-03 10:56:23 -07:00
Yi Wang
be65efb0cc Update backend APIs to cast SQL results back to Java record then serialize to Json reply 2016-09-20 18:56:49 -07:00
jbai
44df5dd08a update the dataset ownership information 2016-09-19 16:10:23 -07:00
Daniel Grove
937637c3ea Add Database to create_tables readme (#225) 
* Add Database to create_tables readme

* Add -D to signify database
 2016-09-16 09:18:49 -07:00
Yi Wang
b136fc6c37 Add MetadataInventoryEvent processor and API 2016-09-15 09:22:42 -07:00
Yi Wang
5ce5a1425e Add hostname and process_id to wh_etl_job_execution 2016-09-12 16:09:33 -07:00
Yi Wang
a69d9b109a fix a typo in dict_field_detail DDL 2016-09-08 09:34:29 -07:00
Yi Wang
5515cbdde9 Add MatadataChangeEvent processor to call seperate APIs 2016-09-06 16:41:50 -07:00
Yi Wang
81f891bfab Map scm repo owner to dataset owner table 2016-08-30 15:35:28 -07:00
Yi (Alan) Wang
579b8fc9d7 Add metadataChangeEvent APIs to backend-service (#205) 
* Add multiproduct and git repo metadata etl job

* Extract commit hash use it when querying acl

* Use FileWriter to write records into CSV file

* Remove unnecessary log entries from kafka processor

* Fix the incompatibility between integer repo_id in db and string field in record

* merge API tables to existing dataset owner and schema field table

* Add confidential and recursive column to dict_dataset_field
 2016-08-24 09:10:35 -07:00
Yi (Alan) Wang
078e90e8bd Add multiproduct and git repo metadata etl job (#202) 
* Add multiproduct and git repo metadata etl job

* implement the dataset availability section

* Extract commit hash use it when querying acl

* Use FileWriter to write records into CSV file

* Remove unnecessary log entries from kafka processor

* Fix the incompatibility between integer repo_id in db and string field in record
 2016-08-12 12:26:55 -07:00
Eric Sun
67479eac58 log_dataset_instance_load_status 2016-08-01 13:00:39 -07:00
Eric Sun
50c222d478 Avro schema for Kafka topics 2016-07-28 16:55:47 -07:00
Eric Sun
f745642212 add datacenter, deploymenttier, cluster to describe dataset instance 2016-07-28 16:38:03 -07:00
Yi Wang
6d4706bc62 Ingest Gobblin tracking events into wherehows using Kafka consumer client 2016-07-25 15:03:29 -07:00
jbai
f3c299480f update the column names from schema to schema_text and view_expanded_text to ddl_text 2016-07-20 18:01:25 -07:00
jbai
33b05cde4b tracking the dalids schema and expanded text by versions 2016-07-20 15:59:11 -07:00
jbai
c1b96db103 make the column version_sort_id of table dict_dataset_instance it not null and default value is 0 2016-06-29 23:00:18 -07:00
jbai
9166db7563 update the dict_dataset_instance data loading sql since table key changed 2016-06-29 18:00:10 -07:00
Eric Sun
807e59330e travis mysql setup 2016-06-22 21:21:19 -07:00
Eric Sun
56cebd0d9c add dict_dataset_instance and cfg_object_name_map to track dataset on multiple clusters and their replication dependency and view dependency 2016-06-22 19:40:05 -07:00
jerrybai2009
8a9eeb1bb8 Merge pull request #126 from jerrybai2009/master 
support the elasticsearch as search engine
 2016-05-23 18:12:55 -07:00
jbai
a2e42d60f3 add the elasticsearch index build and update file 2016-05-23 17:58:37 -07:00
Rafal Kluszczynski
8501b1d7b8 fix: update DDL for job_execution_ext_reference tables to work with mysql 5.7.x 2016-05-23 10:37:30 +02:00
Eric Sun
5ff4d35f51 add cfg_object_name_map and job_execution_ext_reference 2016-05-19 12:39:06 -07:00
SunZhaonan
31de21ddcf pass parameter through file. 2016-05-03 16:25:56 -07:00
SunZhaonan
b202832741 Innodb engine DDL. Add config for timeout and load sample. 2016-04-05 12:43:02 -07:00
SunZhaonan
6077f6565b Fix DDL db_type_id default to 0 2016-04-04 13:40:03 -07:00
SunZhaonan
c4671d2579 Add field comments ETL 
Fix API bug of tech_matrix_id

Add key in comment table
 2016-03-14 14:23:33 -07:00
SunZhaonan
5e9ae37952 Change to multi processing instead of multi thread. Fix hive ETL bug 2016-02-29 16:37:03 -08:00