b8e18b0b5d
* Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this.
DataHub: A Generalized Metadata Search & Discovery Tool
Quickstart | Documentation | Features | Roadmap | Adoption | FAQ | Town Hall
📣 Next DataHub town hall meeting on August 28th, 9am-10am PDT (convert to your local time)
- Signup sheet & questions
- VC link (NEW we're using zoom!): https://linkedin.zoom.us/j/95617940722
- Meeting details & past recordings
✨ Latest Update:
Introduction
DataHub is LinkedIn's generalized metadata search & discovery tool. To learn more about DataHub, check out our LinkedIn Engineering blog post and Strata presentation. You should also visit DataHub Architecture to get a better understanding of how DataHub is implemented and DataHub Onboarding Guide to understand how to extend DataHub for your own use cases.
This repository contains the complete source code for both DataHub's frontend & backend. You can also read about how we sync the changes between our internal fork and GitHub.
Quickstart
Please follow the DataHub Quickstart Guide to get a copy of DataHub up & running locally using Docker. As the guide assumes some basic knowledge of Docker, we'd recommend you to go through the "Hello World" example of A Docker Tutorial for Beginners if Docker is completely foreign to you.
Documentation
- DataHub Developer's Guide
- DataHub Architecture
- DataHub Onboarding Guide
- Docker Images
- Frontend
- Web App
- Generalized Metadata Service
- Metadata Ingestion
- Metadata Processing Jobs
Releases
See Releases page for more details. We follow the SemVer Specification when versioning the releases and adopt the Keep a Changelog convention for the changelog format.
FAQs
Frequently Asked Questions about DataHub can be found here.
Features & Roadmap
Check out DataHub's Features & Roadmap.
Contributing
We welcome contributions from the community. Please refer to our Contributing Guidelines for more details. We also have a contrib directory for incubating experimental features.
Community
Join our slack workspace for discussions and important announcements. You can also find out more about our upcoming town hall meetings and view past recordings.
Adoption
Here are the companies that have officially adopted DataHub. Please feel free to add yours to the list if we missed it.
Here is a list of companies that are currently building POC or seriously evaluating DataHub.
- Booking.com
- Experian
- Geotab
- Instructure
- Inventec
- Microsoft
- Morgan Stanley
- Orange Telecom
- REEF Technology
- SpotHero
- Sysco AS
- ThoughtWorks
- University of Phoenix
- Vectice
- Viasat
- Weee!
Select Articles & Talks
- DataHub: A Generalized Metadata Search & Discovery Tool
- Open sourcing DataHub: LinkedIn’s metadata search and discovery platform
- The evolution of metadata: LinkedIn’s story @ Strata Data Conference 2019
- Journey of metadata at LinkedIn @ Crunch Data Conference 2019
- DataHub Journey with Expedia Group by Arun Vasudevan
- Data Catalogue — Knowing your data
- LinkedIn DataHub Application Architecture Quick Understanding
- 25 Hot New Data Tools and What They DON’T Do
See the full list here.