12076 Commits

Author SHA1 Message Date
Seyi Adebajo
63d87d127a DSS-5869 Adds legacy third-party libraries 2017-07-10 09:54:11 -07:00
Seyi Adebajo
59bb7687d2 DSS-5869 Adds ember-cli boostrap and bower 2017-07-10 09:54:11 -07:00
Seyi Adebajo
a674d99665 DSS-5869 Adds wildcard route for unknown pages 2017-07-10 09:54:11 -07:00
Seyi Adebajo
451f5c72b0 Moves legacy-app out of build pipeline. Fixes reference to legacy wherehows.css in app.scss 2017-07-10 09:54:11 -07:00
Seyi Adebajo
b17514c797 Imports legacy styles into sass source 2017-07-10 09:54:10 -07:00
Seyi Adebajo
710ff4d0d9 Moves old sources into legacy-app folder accessible to ember-cli app structure 2017-07-10 09:54:10 -07:00
Seyi Adebajo
06650c008d Sets cli sass config to remove Bootstrap's JS libs since not needed initially 2017-07-10 09:54:09 -07:00
Seyi Adebajo
5ff23586d4 Adds sass support and bootstrap to ember-cli build config 2017-07-10 09:54:09 -07:00
Seyi Adebajo
e834c7d210 Uncomments options for logging in development environment 2017-07-10 09:54:09 -07:00
Seyi Adebajo
c99e6b2c09 Initial commit transition to Ember CLI 2017-07-10 09:54:09 -07:00
Seyi Adebajo
5d14cc9223 Updates .gitignore with dependency dirs, compiled output dirs, and misc files: test logs, output logs, etcetera 2017-07-10 09:54:09 -07:00
jbai
402c541fac DSS-4092, add metric into search and advanced search 2017-07-10 09:54:09 -07:00
Na Zhang
aacb722e4e switch from avro schema to espresso schema format 2017-07-10 09:54:09 -07:00
Eric Sun
eee2fbf230 only add "key:" prefix to fields[] but not to combined_schema (#360)
The combined_schema.keySchema will not need the "key:" prefix for Espresso
2017-07-10 09:54:09 -07:00
Mars Lan
1efa4eef16 Add hooks for LI-specific adaptation of the build script. 2017-07-10 09:54:09 -07:00
Yi (Alan) Wang
48aa36fa79 Frontend dataset columns get API to fetch data from Metadata store (#358) 2017-07-10 09:54:08 -07:00
Eric Sun
976e4e5d49 - add "key:" prefix to Espresso keySchema (e.g. "key:fieldName01")\n- support input json with empty "documentSchema" (#356) 2017-07-10 09:54:08 -07:00
Mars Lan
0f51ef231e Fix coverall reporting. (#353)
Play gradle plugin puts files in non-standard locations
2017-07-10 09:54:08 -07:00
Mars Lan
a589abbd76 Split the root build script into multiple scripts. (#348)
Split the root build script into multiple scripts
Add coveralls support.
2017-07-10 09:54:08 -07:00
Mars Lan
6b7609918e Replace sbt build with native Gradle Play plugin and update the docs. (#352)
Benefits
1. Simpler setup - no need to download activator in order to build & run
2. Faster build - See https://engineering.linkedin.com/play/developing-play-applications-using-gradle
3. Streamlined dependency management - Everything defined in build.gradle, instead of build.gradle + build.sbt
4. Better integration with gradle lifecycle tasks - build, test, dist, clean all work as expected

Changes
1. Location of staging & distribution files moved from target to build
2. Use ./gradle -t runPlayBinary to run app with hot reload support
3. The generated start scripts are quite different from those generated by sbt
2017-07-10 09:54:08 -07:00
na zhang
5f6fffde57 Restli Client for populating espresso/oracle datasets and schema metadata (#349)
* add dali view owner etl

* add idpc ui

* add the internal flag to switch linkedin internal features

* add idpc ui

* add the internal flag to switch linkedin internal features

* DSS-3495, implement the UI for IDPC JIRA part

* DSS-4076, update the metric view since data model changed

* DSS-4092, add metric into search and advanced search

* update metric database table name and fix the refId and refIdType issue

* remove duplicated idpc entry and javascript log

* Add fetch_owner hive script

* support Appworx flow and job definition and execution

* implement the Appworx log parser

* bring the script finder back

* update the script finder source table name

* add the flow_path into lineage and extract the script info

* fix the appwors flow job and lineage extract issues

* bring the git location back to lineage script node

* sort the script finder lineage info by type

* bring the script info back for lineage job tab

* fix the master branch merge issue

* fix the oracle unixtime calculating issue

* shorten the flow&job extract interval time to 2 hours instead of 1 day

* shorten the appworx refresh time

* add license header; include RUNNING chains from SO_JOB_QUEUE for Appworx

* implement the list view for metrics

* Modify /dataset POST method to perform INSERT or UPDATE of the DatasetRecord

* apply the list view css change to metric

* upgrade idpc and script finder to ember 2.6.2

* metadata dashboard confidential field data collecting

* implement the confidential fields of metadata dashboard

* metadata dashboard dataset description collecting

* update the final table name

* update the final table name for other load function

* exchange the source target of cfg_object_name_map

* implement the description tab for metadata dashboard

* add the load dataset and field comments function

* implemented the bar and pie chart for description

* implement the ownership section for metadata dashboard

* fix the issue that appworx lineage job running too long

* add the table job attempt source code

* implemented the idpc compliance section

* Security Compliance Tab UI (#246)

* Add back WhereHows internal tracking (#251)

* DSS-5178 DSS-5277: Implements Compliance and Confidential Spec
Adds 'logs/' to ignored files

Updates EmberSelectorComponent to handle a list of string options or list of options with value and label, flags the currently selected option, and bubble change actions with 'selectionDidChange' action

DSS-5178: Removes previous updates to search.js: moving jQuery + DOM heavy imperative implementation to Ember component

DSS-5178: Adds templates and components DropRegion and DraggableItem

DSS-5178: Adds getSecuritySpec action and compliance types to Dataset controller, cleans up Datasets route and removes inline securitySpec fetch from route

DSS-5178: Updates templates for compliance spec

DSS-5178: Adds compliance component and updates template

Adds .DS_Store to gitignore

DSS-5277: Adds dataset-confidential component to DOM, Creates DatasetConfidential component, refactors out data handling from component

DSS-5277: Moves data fetching to Dataset Route model and set model data on controller, Adds template for confidential spec component

DSS-5178: Moves view related complianceTypes to component

DSS-5277 DSS-5178: Adds styling for tab content

* DSS-5277 DSS-5178: Adds support for modifying compliancePurgeEntities that don't currently have identifierFields persisted on the remote, PurgeableEntityFieldIdentifierType enum is sourced in client

* DSS-5178 DSS-5277: Adds dataType field to UI for schema field name search result. Refactors processSchema into parseSchema to get fields and types

* DSS-5277 Fixes bug with missing params property on controller depending on route entry point

* DSS-5543: Fixes rendering of datasets in detailview navigating from sidebar/ treeview (#259)

* DSS-5677: Changes component from block syntax to inline. Add property for creating a new PrivacyCompliancePolicy and SecuritySpecification for statasets without either

* DSS-5677: Adds ability to create a new PrivacyCompliancePolicy and SecuritySpecification from the client UI. Also fixes issue with matching fields and data type properties on schema with inconsistent shapes

* DSS-5677: Add create banner for datasets without Privacy policy or Security specification

* DSS-5677: Updates UI to more closely match spec, changes search input behaviour to filter from search

* ADD ESPRESSO_DATASET_METADATA_ETL job to fetch Espresso metadata from Nuage

* Update Nuage load process, fix owner subtype and source

* Add VOLDEMORT ETL job to fetch datasets from Nuage

* Add KAFKA ETL job to fetch topics from Nuage

* skip KAFKA topics starting with 'test' when fetching from Nuage

* Merges front-end changes from master -> DSS-5178 DSS-5577 DSS-5677 DSS-5277 DSS-5677

* DSS-5784: Fixes issue with AdvancedSearch and ScriptFinder URL queries being RFC-3986 incompliant

* ScriptFinder Controller add URL decoding for Json fields (#290)

* DSS-5888 Adds configuration support for Piwik environment tracking. Setting the 'tracking.piwik.siteid' to a value will get rendered in the template and consumed by the tracking initializer

* DSS-5888 DSS-5875 Adds tracking for users. Adds client side tracking for keyword and init for Piwik script module

* Fixes mismatch with compliance api property name: privacyCompliancePolicy != privacyCompliance

* DSS-5888 Fixes tracking userId for noscript tag

* DSS-5865 Removes spinner on metadata/dashboard/idpc-compliance fail

* DSS-6177 Removed unused links in Metric Detail page

* Update Appworx Execution and Lineage jobs (#321)

* DSS-6197: Adds default value for classification property on security specification if not defined

* DSS-6198: Fixes issue with nested fields not getting rendered in the schema for compliance and confidential tabs

* DSS-6018 Adds ui feature to track feedback on user search results relevance using a up/down voting mechanism

* Make unit tests buildable again for backend and web (#325)

* Make unit tests buildable again for backend and web.

* Add back fest dependency so the tests can stay more of less the same as before.

* Generate code coverage reports (#334)

* Add playCoverage task to run code coverage using JaCoco for backend and web.

* Add jacocoTestReport task to run code coverage for testNG-based tests in wherehows-common & metadata-etl.

* Add data platform filter for dashboard APIs (#322)

* Add data platform filter for dashboard APIs

* Add exception handling for Espresso and Kafka ETL job

* restli client to populate espresso and oracle metadata
2017-07-10 09:54:08 -07:00
Yi (Alan) Wang
fb433973fc Add frontend API for datasetOwnedBy (#351) 2017-07-10 09:54:08 -07:00
Mars Lan
5a999f29b1 Revert "Split the root build script into multiple scripts."
This reverts commit 4b8a6f86577739209b09ec8cc8cb09c2808f4aa7.
2017-07-10 09:54:08 -07:00
Mars Lan
df40e643d5 Update travis configuration to generate coverage report for coveralls. 2017-07-10 09:54:08 -07:00
Mars Lan
edf5c54de3 Split the root build script into multiple scripts.
Add support for coveralls.
2017-07-10 09:54:08 -07:00
Mars Lan
162a753e10 Clean up external dependency and README file for metadata-etl. 2017-07-10 09:53:29 -07:00
Seyi Adebajo
a0dec6c015 DSS-6349 Refactors metadata dashboard data retrieval methods to support platform parameter. 2017-07-10 09:53:29 -07:00
Seyi Adebajo
4767d46d63 DSS-6349 Add Template styles. 2017-07-10 09:53:29 -07:00
Seyi Adebajo
69d41503c3 DSS-6349 Add HTMLBars template for platform filters. Adds Capitalize template helper 2017-07-10 09:53:29 -07:00
Seyi Adebajo
eef8439943 DSS-6385 Fixes issue with compliance and confidential tabs in dataset detail view 2017-07-10 09:53:28 -07:00
Yi (Alan) Wang
59b0d92692 Add data platform filter for dashboard APIs (#322)
* Add data platform filter for dashboard APIs

* Add exception handling for Espresso and Kafka ETL job
2017-07-10 09:53:28 -07:00
Mars Lan
e36a40cd65 Generate code coverage reports (#334)
* Add playCoverage task to run code coverage using JaCoco for backend and web.

* Add jacocoTestReport task to run code coverage for testNG-based tests in wherehows-common & metadata-etl.
2017-07-10 09:53:28 -07:00
Mars Lan
bcc3cd9f76 Make unit tests buildable again for backend and web (#325)
* Make unit tests buildable again for backend and web.

* Add back fest dependency so the tests can stay more of less the same as before.
2017-07-10 09:53:28 -07:00
Seyi Adebajo
7b987c0138 DSS-6018 Adds ui feature to track feedback on user search results relevance using a up/down voting mechanism 2017-07-10 09:53:28 -07:00
Seyi Adebajo
1c50aafe92 DSS-6198: Fixes issue with nested fields not getting rendered in the schema for compliance and confidential tabs 2017-07-10 09:53:28 -07:00
Seyi Adebajo
c71775a034 DSS-6197: Adds default value for classification property on security specification if not defined 2017-07-10 09:53:28 -07:00
Yi (Alan) Wang
5219d26b8e Update Appworx Execution and Lineage jobs (#321) 2017-07-10 09:53:28 -07:00
Seyi Adebajo
87719c03af DSS-6177 Removed unused links in Metric Detail page 2017-07-10 09:53:28 -07:00
Seyi Adebajo
aa9bb3263a DSS-5865 Removes spinner on metadata/dashboard/idpc-compliance fail 2017-07-10 09:53:28 -07:00
Seyi Adebajo
6fb996d4b9 DSS-5888 Fixes tracking userId for noscript tag 2017-07-10 09:53:28 -07:00
Seyi Adebajo
fd4c3c414e Fixes mismatch with compliance api property name: privacyCompliancePolicy != privacyCompliance 2017-07-10 09:53:28 -07:00
Seyi Adebajo
a0d5cdf755 DSS-5888 DSS-5875 Adds tracking for users. Adds client side tracking for keyword and init for Piwik script module 2017-07-10 09:53:28 -07:00
Seyi Adebajo
946d92dcbb DSS-5888 Adds configuration support for Piwik environment tracking. Setting the 'tracking.piwik.siteid' to a value will get rendered in the template and consumed by the tracking initializer 2017-07-10 09:53:27 -07:00
Yi (Alan) Wang
fa124be1e1 ScriptFinder Controller add URL decoding for Json fields (#290) 2017-07-10 09:53:27 -07:00
Seyi Adebajo
5628d70e48 DSS-5784: Fixes issue with AdvancedSearch and ScriptFinder URL queries being RFC-3986 incompliant 2017-07-10 09:53:27 -07:00
Seyi Adebajo
02cbf8377a Merges front-end changes from master -> DSS-5178 DSS-5577 DSS-5677 DSS-5277 DSS-5677 2017-07-10 09:53:27 -07:00
Yi Wang
3c6415390a skip KAFKA topics starting with 'test' when fetching from Nuage 2017-07-10 09:53:27 -07:00
Yi Wang
7d6bb9fac9 Add KAFKA ETL job to fetch topics from Nuage 2017-07-10 09:53:27 -07:00
Yi Wang
a9335bc49a Add VOLDEMORT ETL job to fetch datasets from Nuage 2017-07-10 09:53:27 -07:00
Yi Wang
d4b1182aca Update Nuage load process, fix owner subtype and source 2017-07-10 09:53:27 -07:00