yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-09 02:02:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
954 Commits 189 Branches 118 Tags
Commit Graph

954 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yi (Alan) Wang
d444eb0122 Update Multiproduct and CodeSearch ETL job to purge non-exist dataset owner (#456) 2017-07-10 13:42:52 -07:00
Seyi Adebajo
f957e71966 adds styles for browse entity navigation 2017-07-10 13:42:52 -07:00
Seyi Adebajo
7ed0d0f037 updates routes in browse route. adds todo. reformats search.js 2017-07-10 13:42:52 -07:00
Seyi Adebajo
36a82cbc31 updates reducers utils with mapUrnsToIds helper 2017-07-10 13:42:52 -07:00
Seyi Adebajo
339c10542d updates datasets, flows, and metrics actions: removes select[entity] 2017-07-10 13:42:52 -07:00
Seyi Adebajo
92ca92d4e0 adds fetchNodes to entities actions index 2017-07-10 13:42:52 -07:00
Seyi Adebajo
e24a09f9c3 refactors createLazyRequest to use asyncExecutor for better abstraction 2017-07-10 13:42:52 -07:00
Seyi Adebajo
06917ea4f4 adds actions for browse.entity to request nodes and child entities in view. 2017-07-10 13:42:52 -07:00
Seyi Adebajo
756cd8f072 adds state handling for dataset urn child nodes and/or child datasets 2017-07-10 13:42:52 -07:00
Seyi Adebajo
c1c27bb6e5 adds default tests. updates local reference to nodes in reducers.entities.entities 2017-07-10 13:42:52 -07:00
Seyi Adebajo
2c85fc05ed adds utility functions for entites data request 2017-07-10 13:42:52 -07:00
Seyi Adebajo
63ce289cf8 adds validity check to browse action 2017-07-10 13:42:52 -07:00
Seyi Adebajo
90c8a04ea2 adds a nodelist action to browse.entity routes to fetch nodes for the current entity and urn 2017-07-10 13:42:51 -07:00
Seyi Adebajo
e18a947527 updates reducers for flows with entity reducers abstraction. correct typo in metrics reducer. 2017-07-10 13:42:51 -07:00
Seyi Adebajo
d77bf7bd2a utility function to map urns to ids 2017-07-10 13:42:51 -07:00
Seyi Adebajo
13e0fa72cb refactors datasets metrics and abstracted entity reducer functions by removing use of combineReducers function: improper switching between actions caused shared state bug 2017-07-10 13:42:51 -07:00
Seyi Adebajo
83a9509795 refactors metrics reducer to use combinedReducer from entities reducer and wrapping aggregate reducer. this will be factored out into single aggregate reducer for all entities. 2017-07-10 13:42:51 -07:00
Seyi Adebajo
15dd63bb79 updates the datasets root reducer to use a combinedReducer from entities containing reducers abstracted for each entitiy: metrics,flows,datasets. updates datasets.page route to use async thunk lazyRequestPagedDatasets 2017-07-10 13:42:51 -07:00
Mars Lan
35c0781f59 Use WH_APP_FOLDER property as the base directory for temp files for various ETL jobs (#451) 
* Use WH_APP_FOLDER property as the base directory for temp files in Oracle ETL, instead of the full path defined in wh_etl_job_property, which is often /var/tmp/{something}.

* Move common code to FileUtil.py and move Voldamort's temp files too.

* Move Kafaka ETL temp files.

* Move Espresso ETL temp files.

* Move Multiproduct ETL temp files.

* Move CodeSearch ETL temp files.

* Move teradata ETL temp files.
 2017-07-10 13:42:51 -07:00
Yi (Alan) Wang
27d12973e0 Update Nuage Load to purge non-exist dataset owners (#453) 2017-07-10 13:42:51 -07:00
Seyi Adebajo
3f569dc4c2 adds additional patterns request_header and mobile_header to tracking field regular expression. extracts tracking field pattern check to utils 2017-07-10 13:42:51 -07:00
Seyi Adebajo
566e9038cd adds browse ui features: entity navigation, styles. routes for browse.entity. initial tests 2017-07-10 13:42:51 -07:00
Seyi Adebajo
3a28e294bb updates datasets-list to integrate with the updated redux store keys 2017-07-10 13:42:51 -07:00
Mars Lan
f5a7e0c9ec Make sure all intermediate directories are created for ETL job property files. (#450) 2017-07-10 13:42:51 -07:00
Seyi Adebajo
b610385542 abstracts common functionality between datasets/flows/metrics into entities modules for actions and reducers. 2017-07-10 13:42:50 -07:00
Yi (Alan) Wang
dbebb9fa4b add owner_source to primary key in dataset_owner table schema (#449) 2017-07-10 13:42:50 -07:00
Seyi Adebajo
674b37aa38 adds actions for browse feature. abstracts reducers for datasets/metrics/flowsp into entities reducers. adds route for browse feature. 2017-07-10 13:42:50 -07:00
Seyi Adebajo
a3a135372e refactors await use for proper case 2017-07-10 13:42:50 -07:00
Seyi Adebajo
b7e1a9d3aa integrate ember-redux with wherehows. creates reducers and initial actions for datasets using FSA. creates container and presentational components for datasets-list. adds fetch polyfill for fetch api. updates yarn.lock. creates action-set to ensure action rules and conventions 2017-07-10 13:42:50 -07:00
Seyi Adebajo
33c4c472e5 adds ember-redux package as dep. removes exception handler in search route 2017-07-10 13:42:50 -07:00
Seyi Adebajo
c71d970846 sort bower json 2017-07-10 13:42:50 -07:00
Mars Lan
2499ee0116 Unify the open-source application.conf with internal ones so we don't need to maintain both. 2017-07-10 13:42:50 -07:00
Mars Lan
fda572dd8a Allow the logback directory for ETL jobs to be overridden using system property (#448) 
* Allow the logback directory for ETL jobs to be overridden using system property.

See https://logback.qos.ch/manual/configuration.html#variableSubstitution for more details.

* Add WHZ_ETL_TEMP_DIR env var and play config to control where the ETL job logs & temp files to be saved.

This enables us to move away from the default /var/tmp/wherehows directory.
 2017-07-10 13:42:16 -07:00
Mars Lan
b2f0eaaf9d Add app.version play config and have the value exposed via the config API. (#447) 2017-07-10 13:42:16 -07:00
Yi (Alan) Wang
9e95091ca7 return modified time in dataset ownership API (#446) 2017-07-10 13:42:16 -07:00
Seyi Adebajo
9d564d6a91 prefilled tracking fields are not rendered on the confidential classification tab 2017-07-10 13:42:16 -07:00
Seyi Adebajo
afd5e9f115 prefilled tracking fields are not rendered on the compliance classification tab 2017-07-10 13:42:16 -07:00
Mars Lan
32d8dd8ef9 Set the correct system property for krb5 config flie. (#443) 
Clean up ember build properly
 2017-07-10 13:42:16 -07:00
Na Zhang
973fb93167 Elastic search completion suggester for type ahead 2017-07-10 13:42:16 -07:00
Yi (Alan) Wang
674e6f728f Add dataset level confidentiality field in security API and DB (#441) 2017-07-10 13:42:16 -07:00
Yi Wang
338fe52f3c Add check for user not in session, at get current user API 2017-07-10 13:42:15 -07:00
Yi (Alan) Wang
6facc830cc Check Linkedin interanl flag before fetching schema from Metastore (#440) 
Also during column data mapping, check if native data type and description exists in the returned schema fields from Metadata store.
 2017-07-10 13:42:15 -07:00
Yi (Alan) Wang
fe0d2ea5f9 Modify multiproduct ETL to set is_active for owner, return is_active to frontend ownership API (#437) 2017-07-10 13:42:15 -07:00
Seyi Adebajo
0199474138 updates placeholder text in dataset-table instances 2017-07-10 13:42:15 -07:00
Mars Lan
9b5cd71581 Change config tracking.piwik.siteid from type int to string. (#436) 
This allows it to be set via env var (hence cfg2).
 2017-07-10 13:42:15 -07:00
Seyi Adebajo
9939149adc updates network request handlers to integrate with api changes for /compliance and /security endpoints 2017-07-10 13:42:15 -07:00
Yi (Alan) Wang
3d6c8188b1 Increase privacy and security table field size to save larger data (#434) 2017-07-10 13:42:15 -07:00
Seyi Adebajo
c20ab5a054 creates top-level application error substate 2017-07-10 13:42:15 -07:00
Seyi Adebajo
e7b151dd48 updates list of types for field format on confidential tab 2017-07-10 13:42:15 -07:00
Seyi Adebajo
8bfae8ca6f fixes stream closed exception 2017-07-10 13:42:15 -07:00