yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-01 19:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update features.md

Browse Source
This commit is contained in:
Mars Lan 2020-03-11 04:56:22 -07:00 committed by GitHub
parent 7a0443cc4d
commit 990b3453c1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 24 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
docs/features.md
View File

@ -1,37 +1,47 @@
# Features of DataHub
DataHub is composed of a [generic backend infra](what/gma.md) and a [Ember-based UI](../datahub-web). Original DataHub
[blog post](https://engineering.linkedin.com/blog/2019/data-hub) extensively talks about the design and mentions some of
DataHub is made up of a [generic backend](what/gma.md) and a [Ember-based UI](../datahub-web). Original DataHub
[blog post](https://engineering.linkedin.com/blog/2019/data-hub) talks about the design extensively and mentions some of
the features of DataHub. Our open sourcing [blog post](https://engineering.linkedin.com/blog/2020/open-sourcing-datahub--linkedins-metadata-search-and-discovery-p)
also provides a comparison of some features between LinkedIn production DataHub vs open source DataHub. Although, these
are good references, we'll list down all available (also WIP) features of DataHub.
also provides a comparison of some features between LinkedIn production DataHub vs open source DataHub. Below is a list of the latest features that are available in DataHub, as well as features that will soon become available.
## Data Constructs (Entities)
Currently, open source DataHub only supports datasets, users and groups data constructs.
### Datasets
- **Search**: full-text & advanced search, search ranking
- **Browse**: browsing through a fixed hierarchy
- **Browse**: browsing through a configurable hierarchy
- **Schema**: table & document schema in tabular and JSON format
- **Coarse grain lineage**: support for lineage at the dataset level, tabular & graphical visualization of downstreams/upstreams
- **Ownership**: surfacing owners of a dataset, viewing datasets you own
- **Dataset life-cycle management**: deprecate/undeprecate, surface removed datasets and tag it with "removed"
- **Institutional knowledge**: support for adding free form doc to any dataset
- **Fine grain lineage**: support for lineage at the field level [*Not available yet*]
- **Social actions**: likes, follows, bookmarks [*Not available yet*]
- **Compliance management**: field level tag based compliance editing [*Not available yet*]
- **Top users**: frequent users of a dataset [*Not available yet*]
- **Fine grain lineage**: support for lineage at the field level [*available soon*]
- **Social actions**: likes, follows, bookmarks [*available soon*]
- **Compliance management**: field level tag based compliance editing [*available soon*]
- **Top users**: frequent users of a dataset [*available soon*]
### Users
- **Search**: full-text & advanced search, search ranking
- **Browse**: browsing through a configurable hierarchy [*available soon*]
- **Profile editing**: LinkedIn style professional profile editing such as summary, skills
### Metrics [*available soon*]
- **search**: full-text & advanced search, search ranking
- **Browse**: browsing through a configurable hierarchy
- **Basic information**: ownershp, dimensions, formula, input & output datasets, dashboards
- **Institutional knowledge**: support for adding free form doc to any metric
### Dashboards [*available soon*]
- **search**: full-text & advanced search, search ranking
- **Basic information**: ownership, location
- **Institutional knowledge**: support for adding free form doc to any dashboards
## Metadata Sources
You can integrate any data platform to DataHub easily. As long as you have a way of *E*xtracting metadata from the platform and
*T*ransform that into our standard [MCE](what/mxe.md) format, you're free to *L*oad/ingest metadata to DataHub from any available platform.
We have provided [ETL ingestion](architecture/metadata-ingestion.md) pipelines for:
You can integrate any data platform to DataHub easily. As long as you have a way of *Extracting* metadata from the platform and *Transform* that into our standard [MCE](what/mxe.md) format, you're free to *Load*/ingest metadata to DataHub from any available platform.
We have provided example [ETL ingestion](architecture/metadata-ingestion.md) scripts for:
- Hive
- Kafka
- RDBMS
- MySQL
- LDAP
- LDAP