DataHub

Introduction

DataHub is LinkedIn's generalized metadata search & discovery tool. To learn more about DataHub, check out our LinkedIn blog post and Strata presentation. You should also visit DataHub Architecture to get a better understanding of how DataHub is implemented and DataHub Onboarding Guide to understand how to extend DataHub for your own use case. This repository contains the complete source code to be able to build DataHub's frontend & backend services.

Quickstart

1. Install docker and docker-compose.
2. Clone this repo.
3. Run below command to download and run all Docker containers in your local:

cd docker/quickstart && docker-compose pull && docker-compose up --build

4. After you have all Docker containers running in your machine, run below command to ingest provided sample data to DataHub:

./gradlew :metadata-events:mxe-schemas:build && cd metadata-ingestion/mce-cli && pip install --user -r requirements.txt && python mce_cli.py produce -d bootstrap_mce.dat

Note: Make sure that you're using Java 8, we have a strict dependency to Java 8 for build.

5. Finally, you can start DataHub by typing http://localhost:9001 in your browser. You can sign in with datahub as username and password.

Refer to debugging guide if you have issues in any of the above steps.

Quicklinks

• DataHub Architecture
• DataHub Onboarding Guide
• Docker Images
• Frontend App
• Generalized Metadata Service
• Metadata Consumer Jobs
• Metadata Ingestion

Releases

• 2019/09/21: v0.1.0-alpha
• 2019/12/05: v0.2.0-alpha

Roadmap

1. Kubernetes for container orchestration
2. Deploy DataHub to Azure Cloud