10 Commits

Author SHA1 Message Date
Yi (Alan) Wang
d444eb0122 Update Multiproduct and CodeSearch ETL job to purge non-exist dataset owner (#456) 2017-07-10 13:42:52 -07:00
Mars Lan
35c0781f59 Use WH_APP_FOLDER property as the base directory for temp files for various ETL jobs (#451)
* Use WH_APP_FOLDER property as the base directory for temp files in Oracle ETL, instead of the full path defined in wh_etl_job_property, which is often /var/tmp/{something}.

* Move common code to FileUtil.py and move Voldamort's temp files too.

* Move Kafaka ETL temp files.

* Move Espresso ETL temp files.

* Move Multiproduct ETL temp files.

* Move CodeSearch ETL temp files.

* Move teradata ETL temp files.
2017-07-10 13:42:51 -07:00
Yi (Alan) Wang
fe0d2ea5f9 Modify multiproduct ETL to set is_active for owner, return is_active to frontend ownership API (#437) 2017-07-10 13:42:15 -07:00
Yi Wang
5ff2c85fee Remove ownership auto confirm by SCM 2017-07-10 09:55:15 -07:00
Yi Wang
51f911b400 Map git repo and owners to Oracle/espresso/dali datasets 2016-11-22 10:51:00 -08:00
Yi (Alan) Wang
c9dfb637af Update MetadataChangeEvent APIs according to schema change (#243)
* Update MetadataChangeEvent APIs according to schema change

* Update MultiproductLoad to reflect new Owner types

* Add comments for Owner_type precedence (priority) and compliance
2016-10-06 13:33:45 -07:00
Yi Wang
4c500402fe Map repo owner fix, change 'main' to 'Producer' and reset sort id 2016-09-02 13:52:00 -07:00
Yi Wang
a809b0ac47 Map repo owner fix to use dataset group mapping 2016-09-01 18:19:41 -07:00
Yi Wang
81f891bfab Map scm repo owner to dataset owner table 2016-08-30 15:35:28 -07:00
Yi (Alan) Wang
078e90e8bd Add multiproduct and git repo metadata etl job (#202)
* Add multiproduct and git repo metadata etl job

* implement the dataset availability section

* Extract commit hash use it when querying acl

* Use FileWriter to write records into CSV file

* Remove unnecessary log entries from kafka processor

* Fix the incompatibility between integer repo_id in db and string field in record
2016-08-12 12:26:55 -07:00