datahub/docs/roadmap.md

# DataHub Roadmap

Below is DataHub's roadmap for the short, medium and long term. We welcome suggestions from the community.

ETAs are revisted on a regular basis and are subject to change. If you would like to see something prioritized, please reach out to us on [Slack](https://join.slack.com/t/datahubspace/shared_invite/zt-dkzbxfck-dzNl96vBzB06pJpbRwP6RA) or attend the [town hall](townhalls.md) to discuss!

## Short term (3 months) [ETA October 2020]
### Dashboards as entities 
- Models + UI
### Jobs & Flows as entities
- Link datasets to jobs & flows
### AI models as entities
- Models + UI
### Strongly consistent secondary index (SCSI)
- Add query-after-write capability to local DAO 
### Gremlin-based Query DAO
- Support majority of gremlin-compatible graph DBs
### Integration tests
- Add docker-based integration tests
### Kubernetes migration
- Migration from docker-compose to [Kubernetes](https://kubernetes.io/) for Docker container orchestration

## Medium term (3 - 6 months) [ETA January 2021]
### Aspect-specific MCE & MAE
- Split up unified events to improve scalability & modularity
### Dataset field-level lineage
- Models + impact analysis
### Data Concepts as an entity
- Models + UI
### Metrics as entities
- Models + UI
### Schemas as an entity
- Make schemas searchable
- Support GraphQL schemas
### Entity Insights
- UI to highlight high value information about Entities within Search and Entity Pages
### Data privacy management for datasets
- Simple tag-based data privacy metadata
### Social features
- Users will be able to like and follow entities
- Dataset & field-level commenting
### Templatized UI
- Config-driven UI
- Generate TypeScript types from Pegasus 
### Add GraphQL endpoint to GMS
- Use GraphQL exclusively for frontend queries
### Adopt Redux
- Use Redux exclusively for UI state management
### JNoSQL-based Local DAO
- Support a wide range of document stores
### Ownership Transfer
- Donate code to a foundation, e.g. Apache, Linux Foundation.
### Azure deployment
- Run DataHub in [Azure](https://azure.microsoft.com/en-us/) and provide how-to guides

## Long term (6 months - 1 year)
### Operational metadata
- Indexing in OLAP store ([Pinot](https://github.com/apache/incubator-pinot)) with TTL
### Microservices as an entity
- Initially focus on rest.li services & GraphQL integration
### AWS & GCP deployment
- Run DataHub in [AWS](https://aws.amazon.com/) & [GCP](https://cloud.google.com/gcp) and provide how-to guides

## Visionary Goals (1 year+)
### Rewrite midtier in Node
- TypeScript-only frontend development
### gRPC + protobuf
- Modeling in protobuf + serving in gRPC
### UI for metadata graph exploration
Update roadmap.md 2020-02-11 14:14:57 -08:00			`# DataHub Roadmap`
Add roadmap doc 2020-02-11 12:25:33 -08:00
Update roadmap.md 2020-07-26 07:27:55 -07:00			`Below is DataHub's roadmap for the short, medium and long term. We welcome suggestions from the community.`
Add roadmap doc 2020-02-11 12:25:33 -08:00
Update roadmap.md 2020-07-26 07:27:55 -07:00			`ETAs are revisted on a regular basis and are subject to change. If you would like to see something prioritized, please reach out to us on [Slack](https://join.slack.com/t/datahubspace/shared_invite/zt-dkzbxfck-dzNl96vBzB06pJpbRwP6RA) or attend the [town hall](townhalls.md) to discuss!`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00
			`## Short term (3 months) [ETA October 2020]`
			`### Dashboards as entities`
Update roadmap.md 2020-03-02 07:15:00 -08:00			`- Models + UI`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Jobs & Flows as entities`
Update roadmap.md 2020-02-16 06:25:41 -08:00			`- Link datasets to jobs & flows`
Update roadmap.md 2020-07-26 07:27:55 -07:00			`### AI models as entities`
			`- Models + UI`
			`### Strongly consistent secondary index (SCSI)`
Update roadmap.md 2020-02-15 16:38:48 -08:00			`- Add query-after-write capability to local DAO`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Gremlin-based Query DAO`
Update roadmap.md 2020-02-15 06:48:53 -08:00			`- Support majority of gremlin-compatible graph DBs`
Update roadmap.md 2020-07-22 07:03:06 -07:00			`### Integration tests`
			`- Add docker-based integration tests`
Add roadmap doc 2020-02-11 12:25:33 -08:00			`### Kubernetes migration`
Update roadmap.md 2020-02-15 06:48:53 -08:00			`- Migration from docker-compose to [Kubernetes](https://kubernetes.io/) for Docker container orchestration`
Add roadmap doc 2020-02-11 12:25:33 -08:00
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`## Medium term (3 - 6 months) [ETA January 2021]`
			`### Aspect-specific MCE & MAE`
			`- Split up unified events to improve scalability & modularity`
Update roadmap.md 2020-03-17 12:28:14 -07:00			`### Dataset field-level lineage`
Update roadmap.md 2020-02-16 06:25:41 -08:00			`- Models + impact analysis`
Update roadmap.md 2020-07-26 07:27:55 -07:00			`### Data Concepts as an entity`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`- Models + UI`
			`### Metrics as entities`
			`- Models + UI`
			`### Schemas as an entity`
			`- Make schemas searchable`
			`- Support GraphQL schemas`
			`### Entity Insights`
			`- UI to highlight high value information about Entities within Search and Entity Pages`
			`### Data privacy management for datasets`
			`- Simple tag-based data privacy metadata`
			`### Social features`
Update roadmap.md 2020-02-15 06:48:53 -08:00			`- Users will be able to like and follow entities`
Update roadmap.md 2020-02-16 06:25:41 -08:00			`- Dataset & field-level commenting`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Templatized UI`
			`- Config-driven UI`
			`- Generate TypeScript types from Pegasus`
Update roadmap.md 2020-02-21 21:34:45 -08:00			`### Add GraphQL endpoint to GMS`
			`- Use GraphQL exclusively for frontend queries`
Update roadmap.md 2020-02-16 13:28:33 -08:00			`### Adopt Redux`
Update roadmap.md 2020-02-15 14:54:25 -08:00			`- Use Redux exclusively for UI state management`
Update roadmap.md 2020-07-26 07:27:55 -07:00			`### JNoSQL-based Local DAO`
			`- Support a wide range of document stores`
			`### Ownership Transfer`
			`- Donate code to a foundation, e.g. Apache, Linux Foundation.`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Azure deployment`
			`- Run DataHub in [Azure](https://azure.microsoft.com/en-us/) and provide how-to guides`

			`## Long term (6 months - 1 year)`
Update roadmap.md 2020-07-26 07:27:55 -07:00			`### Operational metadata`
			`- Indexing in OLAP store ([Pinot](https://github.com/apache/incubator-pinot)) with TTL`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Microservices as an entity`
			`- Initially focus on rest.li services & GraphQL integration`
Update roadmap.md 2020-07-22 07:03:06 -07:00			`### AWS & GCP deployment`
			`- Run DataHub in [AWS](https://aws.amazon.com/) & [GCP](https://cloud.google.com/gcp) and provide how-to guides`
Update roadmap.md 2020-02-15 06:26:37 -08:00
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`## Visionary Goals (1 year+)`
Update roadmap.md 2020-02-15 06:26:37 -08:00			`### Rewrite midtier in Node`
Update roadmap.md 2020-02-16 06:25:41 -08:00			`- TypeScript-only frontend development`
Update roadmap.md 2020-02-15 14:56:09 -08:00			`### gRPC + protobuf`
Fix typos in docs 2020-02-21 15:42:53 -08:00			`- Modeling in protobuf + serving in gRPC`
Update roadmap.md 2020-02-18 14:11:06 -08:00			`### UI for metadata graph exploration`