Data Hub

Introduction

Data Hub is Linkedin's generalized metadata search & discovery tool. To learn more about Data Hub, check out our Linkedin blog post and Strata presentation. This repository contains the complete source code to be able to build Data Hub's frontend & backend services.

Quickstart

1. Install docker and docker-compose.
2. Clone this repo and make sure you are at the datahub branch.
3. Run below command to download and run all Docker containers in your local:

cd docker/quickstart && docker-compose pull && docker-compose up --build

4. After you have all Docker containers running in your machine, run below command to ingest provided sample data to Data Hub:

./gradlew :metadata-events:mxe-schemas:build && cd metadata-ingestion/mce-cli && sudo pip install -r requirements.txt && python mce_cli.py produce -d bootstrap_mce.dat

5. Finally, you can start Data Hub by typing http://localhost:9001 in your browser. You can sign in with datahub as username and password.

Quicklinks

• Docker Images
• Frontend App
• Generalized Metadata Store
• Metadata Consumer Jobs
• Metadata Ingestion

Roadmap

1. Add Neo4J graph query support
2. Add user profile page
3. Deploy Data Hub to Azure Cloud