0527e483bc
* added intial version of Business Glossary RFC * slight updates to Business Glossary RFC * slight updates to Business Glossary RFC * updated the sample business glossary table * updated the sample business glossary content * updated the sample business glossary content * Update README.md updated the attribute of Business Term * incorporated feedback around DatasetFieldUrn and relationships * Update README.md updated the alternatives section * Update README.md incorporated the feedback provided by Nagarjuna * Update README.md Incorporated feedback a) removal of redundant uri attribute from BusinessTerm definition b) rolled back changes to SchemaField (as per feedback) and will continue to use fieldPath is unique identified for a field in a given Dataset * Incrporated the review comments and Discussion Feedck * Fix teh RFC link * Fix teh RFC link
Introduction
DataHub is LinkedIn's generalized metadata search & discovery tool. To learn more about DataHub, check out our LinkedIn blog post and Strata presentation. You should also visit DataHub Architecture to get a better understanding of how DataHub is implemented and DataHub Onboarding Guide to understand how to extend DataHub for your own use case.
In general, Datahub has two types of users in mind. One has metadata, and use tools provided by Datahub to ingest metadata into Datahub; The other is to use Datahub to discover metadatas available within Datahub. Datahub provides intuitive UI, full text search capablitity, and graph relationship presentation to make the metadata discover and understanding much eaiser.
The following sequence diagram highlights the key features Datahub has, and how the two types of users - metadata ingestion engineers and metadata discover users, can take full advantage of the Datahub.
- It starts with ingesting your metadata into datahub. We provide a collection of sample Python scripts for you. Those scripts work with the popular relationship databases, find metadata of the data source, and publish metadata in Avro data format to MetadataChangeEvent(MCE) Kafka topic.
- A MetadataChangeEvent (MCE) processor consumes Kafka message with given topic, and make necessary transformation, send to Generalized Metadata Service (GMS), and GMS persists the metadata to a relational database of your choice. Currently we support MySQL, PostgreSQL and MariaDB.
- GMS also checks the received metadata to find out whether there is a previous version. If so, it will publish the difference to Kafka’s MetadataAuditEvent (MAE) topic.
- MAE processor consumes MetadataAuditEvent message from Kafka, and persist to Neo4j & Elastic Search (ES).
- The frontend of Datahub talks to the metadata restful API services of GMS. The metadata discovering users can browse, search metadatas, get the details of metadata such as the owner, the lineage and other customer tags.