2020-02-11 14:14:57 -08:00
|
|
|
# DataHub Roadmap
|
2020-02-11 12:25:33 -08:00
|
|
|
|
2020-02-11 14:14:57 -08:00
|
|
|
Below is DataHub's roadmap for the short and medium term. We'll revise this on a regular basis and welcome suggestions from the communities.
|
2020-02-11 12:25:33 -08:00
|
|
|
|
2020-02-11 13:36:38 -08:00
|
|
|
## Short term (3-6 months)
|
2020-03-20 07:44:59 -07:00
|
|
|
### Replace PDSC with [PDL](https://linkedin.github.io/rest.li/pdl_schema) [*WIP*]
|
2020-03-20 07:41:03 -07:00
|
|
|
- More readable, Java-like syntax + code-gen based on annotations
|
2020-03-11 05:26:05 -07:00
|
|
|
### Aspect-specific MCE & MAE [*WIP*]
|
2020-03-06 13:46:27 -08:00
|
|
|
- Split up unified events to improve scalability & modularity
|
2020-03-11 05:30:40 -07:00
|
|
|
### Metrics as entities [*LinkedIn-internal, waiting to open source*]
|
2020-03-02 07:20:29 -08:00
|
|
|
- Models + UI
|
2020-03-11 05:30:40 -07:00
|
|
|
### Dashboards as entities [*LinkedIn-internal, waiting to open source*]
|
2020-03-02 07:15:00 -08:00
|
|
|
- Models + UI
|
2020-03-10 23:32:50 -07:00
|
|
|
### Jobs & Flows as entities [*WIP*]
|
2020-02-16 06:25:41 -08:00
|
|
|
- Link datasets to jobs & flows
|
2020-03-10 23:32:50 -07:00
|
|
|
### Schemas as an entity [*WIP*]
|
2020-02-14 13:26:53 -08:00
|
|
|
- Make schemas searchable
|
2020-02-21 15:42:53 -08:00
|
|
|
- Support GraphQL schemas
|
2020-03-11 05:30:40 -07:00
|
|
|
### Data privacy management for datasets [*LinkedIn-internal, waiting to open source*]
|
|
|
|
|
- Simple tag-based data privacy metadata
|
2020-03-10 23:32:50 -07:00
|
|
|
### Strongly consistent local index [*WIP*]
|
2020-02-15 16:38:48 -08:00
|
|
|
- Add query-after-write capability to local DAO
|
2020-03-10 23:32:50 -07:00
|
|
|
### Gremlin-based Query DAO [*WIP*]
|
2020-02-15 06:48:53 -08:00
|
|
|
- Support majority of gremlin-compatible graph DBs
|
2020-03-10 23:32:50 -07:00
|
|
|
### Templatized UI [*WIP*]
|
2020-02-15 14:54:25 -08:00
|
|
|
- Config-driven UI
|
|
|
|
|
- Generate TypeScript types from Pegasus
|
2020-03-10 23:32:50 -07:00
|
|
|
### Entity Insights [*WIP*]
|
2020-03-06 13:46:27 -08:00
|
|
|
- UI to highlight high value information about Entities within Search and Entity Pages
|
2020-02-11 12:25:33 -08:00
|
|
|
### Kubernetes migration
|
2020-02-15 06:48:53 -08:00
|
|
|
- Migration from docker-compose to [Kubernetes](https://kubernetes.io/) for Docker container orchestration
|
2020-02-11 12:25:33 -08:00
|
|
|
### Azure deployment
|
2020-02-15 14:57:51 -08:00
|
|
|
- Run DataHub in [Azure](https://azure.microsoft.com/en-us/) and provide how-to guides
|
2020-02-11 12:25:33 -08:00
|
|
|
|
2020-02-15 06:26:37 -08:00
|
|
|
## Medium term (6 months - 1 year)
|
2020-03-17 12:28:14 -07:00
|
|
|
### Dataset field-level lineage
|
2020-02-16 06:25:41 -08:00
|
|
|
- Models + impact analysis
|
2020-03-11 05:30:40 -07:00
|
|
|
### Operational metadata [*WIP*]
|
2020-02-16 06:27:21 -08:00
|
|
|
- Indexing in OLAP store ([Pinot](https://github.com/apache/incubator-pinot)) with TTL
|
2020-03-10 23:32:50 -07:00
|
|
|
### Social features [*WIP*]
|
2020-02-15 06:48:53 -08:00
|
|
|
- Users will be able to like and follow entities
|
2020-02-16 06:25:41 -08:00
|
|
|
- Dataset & field-level commenting
|
2020-03-11 05:30:40 -07:00
|
|
|
### Microservices as an entity [*WIP*]
|
2020-02-16 06:26:27 -08:00
|
|
|
- Initially focus on rest.li services & GraphQL integration
|
2020-02-15 06:48:53 -08:00
|
|
|
### Kundera-based Local DAO
|
|
|
|
|
- Support a wide range of document stores
|
2020-02-21 21:34:45 -08:00
|
|
|
### Add GraphQL endpoint to GMS
|
|
|
|
|
- Use GraphQL exclusively for frontend queries
|
2020-02-16 13:28:33 -08:00
|
|
|
### Adopt Redux
|
2020-02-15 14:54:25 -08:00
|
|
|
- Use Redux exclusively for UI state management
|
2020-02-11 12:25:33 -08:00
|
|
|
### Integration tests
|
2020-02-15 14:54:25 -08:00
|
|
|
- Add docker-based integration tests
|
2020-02-15 14:57:51 -08:00
|
|
|
### AWS & GCP deployment
|
|
|
|
|
- Run DataHub in [AWS](https://aws.amazon.com/) & [GCP](https://cloud.google.com/gcp) and provide how-to guides
|
2020-02-15 06:49:47 -08:00
|
|
|
### Apache incubation
|
2020-02-21 21:34:45 -08:00
|
|
|
- Donate code to Apache foundation
|
2020-02-15 06:26:37 -08:00
|
|
|
|
|
|
|
|
## Long term (1+ year)
|
|
|
|
|
### Rewrite midtier in Node
|
2020-02-16 06:25:41 -08:00
|
|
|
- TypeScript-only frontend development
|
2020-02-15 14:56:09 -08:00
|
|
|
### gRPC + protobuf
|
2020-02-21 15:42:53 -08:00
|
|
|
- Modeling in protobuf + serving in gRPC
|
2020-02-18 14:11:06 -08:00
|
|
|
### UI for metadata graph exploration
|
