Mars Lan
5987ab5ed0
Update MCE schema and test ( #954 )
2018-02-01 11:12:59 -08:00
Yi (Alan) Wang
9dead1831e
Add MIE and MRE schema ( #951 )
2018-01-25 17:20:37 -08:00
Yi (Alan) Wang
aebb2797d5
Add refresh the dataset Property in MCE, and fix tests ( #947 )
2018-01-24 15:05:24 -08:00
Yi (Alan) Wang
b1f237393e
Modify list datasets API to return pagination info ( #936 )
2018-01-18 08:31:31 -08:00
Yi (Alan) Wang
0cd9d68258
Use changeAuditStamp for MCE/MLE whitelist validation ( #928 )
2018-01-05 11:21:56 -08:00
Yi (Alan) Wang
a12b7dfbe1
Fix MCE processor now that Schema may have two types ( #885 )
2017-11-29 17:07:49 -08:00
Yi (Alan) Wang
a3b11f6fea
Modify the return value for owners when record not found ( #868 )
2017-11-16 11:33:09 -08:00
na zhang
ae801c8d24
Merge pull request #834 from camelliazhang/master
...
refacoring elastic search query template by building json objects
2017-11-09 10:52:54 -08:00
Mars Lan
4c8c150b8d
Update MCE & MLE schema ( #847 )
2017-11-08 17:49:24 -08:00
Andrew Park
c870d598ce
Added a metadata lineage processor and a MetadataLineageEvent.avsc ( #841 )
...
* adding MLE processor and rely on operators
* Added Lineage Processor impl. and Kafka Event avro schema
2017-11-08 11:06:22 -08:00
Kiran
601cd258e1
Failed to open file 'ETL_DDL/kafka_tracking.sql ( #844 )
...
ERROR at line 26: Failed to open file 'ETL_DDL/kafka_tracking.sql', error: 2
2017-11-07 11:31:57 -08:00
Mars Lan
d31832fd18
Divert failed-to-process MCEs to a different topic ( #837 )
2017-11-03 11:12:54 -07:00
na zhang
04d6dc5973
reading json files for query templates
2017-11-02 21:51:31 -07:00
Yi (Alan) Wang
a6b726d26c
Remove kafka Gobblin event and processor ( #835 )
2017-11-01 15:32:19 -07:00
Yi (Alan) Wang
3cde8a5597
Fix issue with docker ports and DDL ( #822 )
2017-10-26 10:41:12 -07:00
na zhang
54237ff5ca
support elasticsearch auto re-index with zero downtime via alias switch
2017-10-25 15:50:03 -07:00
Yi (Alan) Wang
d466cb8cb3
Update compliance purge in MCE ( #817 )
2017-10-24 16:05:49 -07:00
Yi (Alan) Wang
776f83e893
Modify compliance API to access purge note ( #807 )
2017-10-19 10:26:45 -07:00
Yi (Alan) Wang
ae6937af41
Increase urn length to 500 in DDL ( #804 )
2017-10-18 14:57:58 -07:00
Yi (Alan) Wang
ddc9171eba
Add binary_json to dataset table storage type ( #802 )
2017-10-17 12:48:42 -07:00
Mars Lan
4b3f7b8935
Remove OS-specific build logic for wherehows-data-model ( #773 )
2017-09-29 10:00:14 -07:00
Yi (Alan) Wang
fce70ebe25
Update MCE processor ( #764 )
2017-09-25 17:22:04 -07:00
Na Zhang
4388be0b03
introduce customized analyzer for wherehows es index
2017-09-21 13:42:55 -07:00
Yi (Alan) Wang
55e26e8508
Remove unused avro schemas, move avro resolution ( #735 )
2017-09-11 16:37:32 -07:00
Yi (Alan) Wang
d2a3fe58db
Add schema and generated java to data model, refactor Gobblin audit processor ( #732 )
2017-09-11 15:26:06 -07:00
Na Zhang
ca34cd920d
ump metric ETL
2017-08-16 16:43:46 -07:00
Mars Lan
6153565968
Add missing DDLs for LDAP-related ETL. ( #664 )
2017-08-11 11:41:40 -07:00
Yi (Alan) Wang
068acf0a46
Remove deprecated compliance and security related frontend API ( #654 )
2017-08-09 17:10:58 -07:00
Andrew Park
7ccec38f21
- Added Elasticsearch Docker provisioning and composer
...
- Fixed local mysql port conflict
- Added temp files cleaning up
- Changed ES doc name to Readme.md
2017-07-27 17:44:16 -07:00
Yi (Alan) Wang
553b032346
Updated Metadata Catalog kafka event schema ( #625 )
2017-07-27 17:42:27 -07:00
Yi (Alan) Wang
d52edaf3e1
Allow ETL jobs to be disabled at run time ( #622 )
2017-07-26 21:13:56 -07:00
Mars Lan
67e6986c68
Add license header
2017-07-25 21:55:53 -07:00
Mars Lan
d161a072a2
Add DDL for log_jira__hdfs_directory_to_owner_map table.
2017-07-25 18:09:49 -07:00
Mars Lan
04f61e13f1
Add DDL for job_execution_script.
2017-07-25 14:19:47 -07:00
Yi (Alan) Wang
6a8fe66ef9
Add active, deprecated flag to dataset, modify ETL to update is_active ( #604 )
2017-07-19 17:07:28 -07:00
Mars Lan
3887c71c37
Rename readme.MD to README.md
2017-07-18 15:44:53 -07:00
Mars Lan
ec0b5ac309
Fix and significatly simplify the docker build process. ( #593 )
2017-07-13 09:39:59 -07:00
Mars Lan
bb5f483be9
Clean up comments and DDL for obsolete tables. ( #586 )
2017-07-10 13:44:35 -07:00
Yi (Alan) Wang
98ffbfb65f
track last modified person of compliance info ( #554 )
2017-07-10 13:44:33 -07:00
Mars Lan
53a30d5a77
Major refactoring of ETL scheduling & configuration ( #542 )
...
* Major refactoring
- Move job-spcific properties from wh_etl_job_property table to .job files
- Use the job file name instead of numeric IDs to identify ETL jobs
- Use reflection to create ETL job class at run time instead of relying hard-coded enums
- Drop ETL job-related APIs as they're no longer needed
- Drop wh_etl_job, wh_etl_job_property, wh_etl_job_execution tables
- Add wh_etl_schedule & wh_etl_history tables
2017-07-10 13:44:33 -07:00
Yi (Alan) Wang
445a574a87
Change compliance column type to MEDIUMTEXT ( #530 )
2017-07-10 13:43:00 -07:00
Yi (Alan) Wang
b2736e281b
Create dataset_compliance table to combine compliance and security ( #512 )
2017-07-10 13:42:58 -07:00
Mars Lan
d75ae54b4b
Rename data-model to wherehows-data-model. ( #492 )
2017-07-10 13:42:57 -07:00
