datahub/docs/roadmap.md

# DataHub Roadmap

Below is DataHub's roadmap for the short and medium term. We'll revise this on a regular basis and welcome suggestions from the communities.

ETAs are revisted on a regular basis and are subject to change. If you would like to see anything prioritized higher than currently is on this list, please reach out to us and we can discuss it!

## Short term (3 months) [ETA October 2020]
### Dashboards as entities 
- Models + UI
### Jobs & Flows as entities
- Link datasets to jobs & flows
### Strongly consistent local index
- Add query-after-write capability to local DAO 
### Gremlin-based Query DAO
- Support majority of gremlin-compatible graph DBs
### Integration tests
- Add docker-based integration tests
### Kubernetes migration
- Migration from docker-compose to [Kubernetes](https://kubernetes.io/) for Docker container orchestration

## Medium term (3 - 6 months) [ETA January 2021]
### Aspect-specific MCE & MAE
- Split up unified events to improve scalability & modularity
### Dataset field-level lineage
- Models + impact analysis
### Data Concepts
- Models + UI
### Metrics as entities
- Models + UI
### Schemas as an entity
- Make schemas searchable
- Support GraphQL schemas
### Entity Insights
- UI to highlight high value information about Entities within Search and Entity Pages
### Data privacy management for datasets
- Simple tag-based data privacy metadata
### Operational metadata
- Indexing in OLAP store ([Pinot](https://github.com/apache/incubator-pinot)) with TTL
### Social features
- Users will be able to like and follow entities
- Dataset & field-level commenting
### Templatized UI
- Config-driven UI
- Generate TypeScript types from Pegasus 
### Add GraphQL endpoint to GMS
- Use GraphQL exclusively for frontend queries
### Adopt Redux
- Use Redux exclusively for UI state management
### Apache incubation
- Donate code to Apache foundation
### Azure deployment
- Run DataHub in [Azure](https://azure.microsoft.com/en-us/) and provide how-to guides

## Long term (6 months - 1 year)
### Microservices as an entity
- Initially focus on rest.li services & GraphQL integration
### JNoSQL-based Local DAO
- Support a wide range of document stores
### AWS & GCP deployment
- Run DataHub in [AWS](https://aws.amazon.com/) & [GCP](https://cloud.google.com/gcp) and provide how-to guides

## Visionary Goals (1 year+)
### Rewrite midtier in Node
- TypeScript-only frontend development
### gRPC + protobuf
- Modeling in protobuf + serving in gRPC
### UI for metadata graph exploration
Update roadmap.md 2020-02-11 14:14:57 -08:00			`# DataHub Roadmap`
Add roadmap doc 2020-02-11 12:25:33 -08:00
Update roadmap.md 2020-02-11 14:14:57 -08:00			`Below is DataHub's roadmap for the short and medium term. We'll revise this on a regular basis and welcome suggestions from the communities.`
Add roadmap doc 2020-02-11 12:25:33 -08:00
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`ETAs are revisted on a regular basis and are subject to change. If you would like to see anything prioritized higher than currently is on this list, please reach out to us and we can discuss it!`

			`## Short term (3 months) [ETA October 2020]`
			`### Dashboards as entities`
Update roadmap.md 2020-03-02 07:15:00 -08:00			`- Models + UI`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Jobs & Flows as entities`
Update roadmap.md 2020-02-16 06:25:41 -08:00			`- Link datasets to jobs & flows`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Strongly consistent local index`
Update roadmap.md 2020-02-15 16:38:48 -08:00			`- Add query-after-write capability to local DAO`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Gremlin-based Query DAO`
Update roadmap.md 2020-02-15 06:48:53 -08:00			`- Support majority of gremlin-compatible graph DBs`
Update roadmap.md 2020-07-22 07:03:06 -07:00			`### Integration tests`
			`- Add docker-based integration tests`
Add roadmap doc 2020-02-11 12:25:33 -08:00			`### Kubernetes migration`
Update roadmap.md 2020-02-15 06:48:53 -08:00			`- Migration from docker-compose to [Kubernetes](https://kubernetes.io/) for Docker container orchestration`
Add roadmap doc 2020-02-11 12:25:33 -08:00
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`## Medium term (3 - 6 months) [ETA January 2021]`
			`### Aspect-specific MCE & MAE`
			`- Split up unified events to improve scalability & modularity`
Update roadmap.md 2020-03-17 12:28:14 -07:00			`### Dataset field-level lineage`
Update roadmap.md 2020-02-16 06:25:41 -08:00			`- Models + impact analysis`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Data Concepts`
			`- Models + UI`
			`### Metrics as entities`
			`- Models + UI`
			`### Schemas as an entity`
			`- Make schemas searchable`
			`- Support GraphQL schemas`
			`### Entity Insights`
			`- UI to highlight high value information about Entities within Search and Entity Pages`
			`### Data privacy management for datasets`
			`- Simple tag-based data privacy metadata`
			`### Operational metadata`
Update roadmap.md 2020-02-16 06:27:21 -08:00			`- Indexing in OLAP store ([Pinot](https://github.com/apache/incubator-pinot)) with TTL`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Social features`
Update roadmap.md 2020-02-15 06:48:53 -08:00			`- Users will be able to like and follow entities`
Update roadmap.md 2020-02-16 06:25:41 -08:00			`- Dataset & field-level commenting`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Templatized UI`
			`- Config-driven UI`
			`- Generate TypeScript types from Pegasus`
Update roadmap.md 2020-02-21 21:34:45 -08:00			`### Add GraphQL endpoint to GMS`
			`- Use GraphQL exclusively for frontend queries`
Update roadmap.md 2020-02-16 13:28:33 -08:00			`### Adopt Redux`
Update roadmap.md 2020-02-15 14:54:25 -08:00			`- Use Redux exclusively for UI state management`
Update roadmap.md 2020-02-15 06:49:47 -08:00			`### Apache incubation`
Update roadmap.md 2020-02-21 21:34:45 -08:00			`- Donate code to Apache foundation`
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`### Azure deployment`
			`- Run DataHub in [Azure](https://azure.microsoft.com/en-us/) and provide how-to guides`

			`## Long term (6 months - 1 year)`
			`### Microservices as an entity`
			`- Initially focus on rest.li services & GraphQL integration`
			`### JNoSQL-based Local DAO`
			`- Support a wide range of document stores`
Update roadmap.md 2020-07-22 07:03:06 -07:00			`### AWS & GCP deployment`
			`- Run DataHub in [AWS](https://aws.amazon.com/) & [GCP](https://cloud.google.com/gcp) and provide how-to guides`
Update roadmap.md 2020-02-15 06:26:37 -08:00
Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> 2020-07-22 07:01:53 -07:00			`## Visionary Goals (1 year+)`
Update roadmap.md 2020-02-15 06:26:37 -08:00			`### Rewrite midtier in Node`
Update roadmap.md 2020-02-16 06:25:41 -08:00			`- TypeScript-only frontend development`
Update roadmap.md 2020-02-15 14:56:09 -08:00			`### gRPC + protobuf`
Fix typos in docs 2020-02-21 15:42:53 -08:00			`- Modeling in protobuf + serving in gRPC`
Update roadmap.md 2020-02-18 14:11:06 -08:00			`### UI for metadata graph exploration`