Yi Wang
b136fc6c37
Add MetadataInventoryEvent processor and API
2016-09-15 09:22:42 -07:00
Yi Wang
5ce5a1425e
Add hostname and process_id to wh_etl_job_execution
2016-09-12 16:09:33 -07:00
Yi Wang
a69d9b109a
fix a typo in dict_field_detail DDL
2016-09-08 09:34:29 -07:00
Yi Wang
5515cbdde9
Add MatadataChangeEvent processor to call seperate APIs
2016-09-06 16:41:50 -07:00
Yi Wang
81f891bfab
Map scm repo owner to dataset owner table
2016-08-30 15:35:28 -07:00
Yi (Alan) Wang
579b8fc9d7
Add metadataChangeEvent APIs to backend-service ( #205 )
...
* Add multiproduct and git repo metadata etl job
* Extract commit hash use it when querying acl
* Use FileWriter to write records into CSV file
* Remove unnecessary log entries from kafka processor
* Fix the incompatibility between integer repo_id in db and string field in record
* merge API tables to existing dataset owner and schema field table
* Add confidential and recursive column to dict_dataset_field
2016-08-24 09:10:35 -07:00
Yi (Alan) Wang
078e90e8bd
Add multiproduct and git repo metadata etl job ( #202 )
...
* Add multiproduct and git repo metadata etl job
* implement the dataset availability section
* Extract commit hash use it when querying acl
* Use FileWriter to write records into CSV file
* Remove unnecessary log entries from kafka processor
* Fix the incompatibility between integer repo_id in db and string field in record
2016-08-12 12:26:55 -07:00
Eric Sun
67479eac58
log_dataset_instance_load_status
2016-08-01 13:00:39 -07:00
Eric Sun
50c222d478
Avro schema for Kafka topics
2016-07-28 16:55:47 -07:00
Eric Sun
f745642212
add datacenter, deploymenttier, cluster to describe dataset instance
2016-07-28 16:38:03 -07:00
Yi Wang
6d4706bc62
Ingest Gobblin tracking events into wherehows using Kafka consumer client
2016-07-25 15:03:29 -07:00
jbai
f3c299480f
update the column names from schema to schema_text and view_expanded_text to ddl_text
2016-07-20 18:01:25 -07:00
jbai
33b05cde4b
tracking the dalids schema and expanded text by versions
2016-07-20 15:59:11 -07:00
jbai
c1b96db103
make the column version_sort_id of table dict_dataset_instance it not null and default value is 0
2016-06-29 23:00:18 -07:00
jbai
9166db7563
update the dict_dataset_instance data loading sql since table key changed
2016-06-29 18:00:10 -07:00
Eric Sun
807e59330e
travis mysql setup
2016-06-22 21:21:19 -07:00
Eric Sun
56cebd0d9c
add dict_dataset_instance and cfg_object_name_map to track dataset on multiple clusters and their replication dependency and view dependency
2016-06-22 19:40:05 -07:00
jerrybai2009
8a9eeb1bb8
Merge pull request #126 from jerrybai2009/master
...
support the elasticsearch as search engine
2016-05-23 18:12:55 -07:00
jbai
a2e42d60f3
add the elasticsearch index build and update file
2016-05-23 17:58:37 -07:00
Rafal Kluszczynski
8501b1d7b8
fix: update DDL for job_execution_ext_reference tables to work with mysql 5.7.x
2016-05-23 10:37:30 +02:00
Eric Sun
5ff4d35f51
add cfg_object_name_map and job_execution_ext_reference
2016-05-19 12:39:06 -07:00
SunZhaonan
31de21ddcf
pass parameter through file.
2016-05-03 16:25:56 -07:00
SunZhaonan
b202832741
Innodb engine DDL. Add config for timeout and load sample.
2016-04-05 12:43:02 -07:00
SunZhaonan
6077f6565b
Fix DDL db_type_id default to 0
2016-04-04 13:40:03 -07:00
SunZhaonan
c4671d2579
Add field comments ETL
...
Fix API bug of tech_matrix_id
Add key in comment table
2016-03-14 14:23:33 -07:00
SunZhaonan
5e9ae37952
Change to multi processing instead of multi thread. Fix hive ETL bug
2016-02-29 16:37:03 -08:00
Eric Sun
f183d8e6da
clean up: dict_business_metric (product_page_key_group_sk => sub_category)
2016-02-29 00:13:40 -08:00
jbai
bae310e715
add the license header to fix the build error
2016-02-25 11:26:57 -08:00
Eric Sun
470a9f8984
Create readme.MD
2016-02-16 16:36:47 -08:00
Eric Sun
7b334e89f1
Create create_all_tables_wrapper.sql
2016-02-16 16:31:29 -08:00
Eric Sun
7971d5b692
Update users.sql
2016-02-16 00:41:07 -08:00
SunZhaonan
f5f3871c94
Fix UI bugs of duplicate node and urn null, urn lowercase. Add missing DDL.
2016-02-12 11:59:14 -08:00
SunZhaonan
c3da00003e
Fix bugs. Reenforce logging. Format jython scripts. Add missing table DDL.
2016-02-03 19:22:18 -08:00
Zhen Chen
7fca60a527
add ref flow id for sub flow jobs
2015-12-17 16:26:15 -08:00
Zhen Chen
d6f77a6f06
fix ldap org hierachy is empty bug and minor changes
2015-12-16 17:01:55 -08:00
SunZhaonan
cd44daba5d
Merge with master
2015-12-16 15:54:50 -08:00
SunZhaonan
07c46304b5
Fix bug of duplicate field loading. Fix bug of subflow process in azkaban lineage ETL.
2015-12-11 19:46:35 -08:00
Zhen Chen
af04ff6efc
add git file commit history etl
2015-12-11 11:02:29 -08:00
Zhen Chen
ebbf9ec629
add ldap user and group metadata etl
2015-12-10 16:26:57 -08:00
Zhen Chen
5a08134b8d
add dataset owner metadata etl
2015-12-07 15:17:01 -08:00
SunZhaonan
d5c3d87d00
Initial commit
2015-11-19 14:39:21 -08:00
