1a974c2d38
Adds 'logs/' to ignored files Updates EmberSelectorComponent to handle a list of string options or list of options with value and label, flags the currently selected option, and bubble change actions with 'selectionDidChange' action DSS-5178: Removes previous updates to search.js: moving jQuery + DOM heavy imperative implementation to Ember component DSS-5178: Adds templates and components DropRegion and DraggableItem DSS-5178: Adds getSecuritySpec action and compliance types to Dataset controller, cleans up Datasets route and removes inline securitySpec fetch from route DSS-5178: Updates templates for compliance spec DSS-5178: Adds compliance component and updates template Adds .DS_Store to gitignore DSS-5277: Adds dataset-confidential component to DOM, Creates DatasetConfidential component, refactors out data handling from component DSS-5277: Moves data fetching to Dataset Route model and set model data on controller, Adds template for confidential spec component DSS-5178: Moves view related complianceTypes to component DSS-5277 DSS-5178: Adds styling for tab content
WhereHows 
WhereHows is a data discovery and lineage tool built at LinkedIn. It integrates with all the major data processing systems and collects both catalog and operational metadata from them.
Within the central metadata repository, WhereHows curates, associates, and surfaces the metadata information through two interfaces:
- a web application that enables data & linage discovery, and community collaboration
- an API endpoint that empowers automation of data processes/applications
WhereHows serves as the single platform that:
- links data objects with people and processes
- enables crowdsourcing for data knowledge
- provides data governance and provenance based on ownership and lineage
Documentation
The detailed information can be found in the Wiki
Examples in VM
There is a pre-built vmware image (about 11GB) to quickly demonstrate the functionality of WhereHows. Check out the VM Guide
Getting Started
New to Wherehows? Check out the Getting Started Guide
Preparation
First, please get Play Framework (Activator) in place.
# Download Activator
wget https://downloads.typesafe.com/typesafe-activator/1.3.11/typesafe-activator-1.3.11-minimal.zip
# Unzip, Remove zipped folder, move play folder to $HOME
unzip -q typesafe-activator-1.3.11-minimal.zip && rm typesafe-activator-1.3.11-minimal.zip && mv activator-1.3.11-minimal $HOME/
# Add ACTIVATOR_HOME, GRADLE_HOME. Update Path to include new gradle, alias to counteract issues
echo 'export ACTIVATOR_HOME="$HOME/activator-1.3.11-minimal"' >> ~/.bashrc
source ~/.bashrc
You need to increase the SBT build tool max heap size for building web module
echo 'export SBT_OPTS="-Xms1G -Xmx1G -Xss2M"' >> ~/.bashrc
source ~/.bashrc
Second, please setup the metadata repository in MySQL.
CREATE DATABASE wherehows
DEFAULT CHARACTER SET utf8
DEFAULT COLLATE utf8_general_ci;
CREATE USER 'wherehows';
SET PASSWORD FOR 'wherehows' = PASSWORD('wherehows');
GRANT ALL ON wherehows.* TO 'wherehows'
Execute the DDL files to create the required repository tables in wherehows database.
Build
- Get the source code:
git clone https://github.com/linkedin/WhereHows.git - Put a few 3rd-party jar files to metadata-etl/extralibs directory. Some of these jar files may not be available in Maven Central or Artifactory. See the download instrucitons for more detail.
cd WhereHows/metadata-etl/extralibs - Go back to the WhereHows root directory and build all the modules:
./gradlew build - Go back to the WhereHows root directory and start the metadata ETL and API service:
cd backend-service ; $ACTIVATOR_HOME/bin/activator run - Go back to the WhereHows root directory and start the web front-end:
cd web ; $ACTIVATOR_HOME/bin/activator runThen WhereHows UI is available at http://localhost:9000 by default. For example,$ACTIVATOR_HOME/bin/activator run -Dhttp.port=19001will use port 19001 to serve UI.
Contribute
Want to contribute? Check out the Contributors Guide
Community
Want help? Check out the Gitter chat room and Google Groups