* Update the roadmap
- Make short term more like what we're doing this quarter
- Medium term is next quarter
- Long term is 2 or 3 quarters from now
- Visionary is even beyond that
Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far.
* Update roadmap.md
Co-authored-by: Mars Lan <mars.th.lan@gmail.com>
* fix: modify the etl script dependency
* fix: Correct the way to catch the exception
* fix: Compatible with the following kafka cluster when the Kafka Topic message Key cannot be empty
* fix: Adjust the kafka message key; Improve the comment of field
* fix: Avro schema required for key
Co-authored-by: Cobolbaby <Zhang.Xing-Long@inventec.com>
* implement search feature
* add test for dataprocessIndexBuilder; refactor code based on feedback
* update based on PR feedback
* Update DataProcessDocument.pdl
fixed typo wording.
* add not null check for data process info
The current helm command is not compatible with the
docker image. Instead, rely on the docker image's CMD.
Co-authored-by: jsotelo <javier.sotelo@viasat.com>
* add job info as aspect of a dataset
* add job urn def., aspect and entity
* job entity with upstream and downstream lineage
* use job urn in upstream & downstream
* add Job entity rest APIs
* rest.li api, impl and factory for job entity
* code cleanup
* use pdl; onboard data process entity
* add es index json
* fix gradlew build ignored tasks
* add a comment about data process info field
* fix style warning issues
* update content based on PR
* checked in generated snapshot json
* updated based on PR feedback
* update data process data format
* updated based on code review feedback
* revert back gms & mce-job docker image
* delete temp files
* update based pr feedback
* file name and a typo
* format with linkedin style
Co-authored-by: Liangjun <liajiang@expediagroup.com>
* build(docker): refactor docker build scripts
- add "build" option to docker-compose files to simplify rebuilding of images
- create "start.sh" script so it's easier to override "command" in the quickstart's docker-compose file
- use dockerize to wait for requisite services to start up
- add a dedicated Dockerfile for kafka-setup
This fixes https://github.com/linkedin/datahub/issues/1549 & https://github.com/linkedin/datahub/issues/1550
