16 Commits

Author SHA1 Message Date
Liangjun Jiang
5d078aa617
Implemented data process search feature (#1706)
* implement search feature

* add test for dataprocessIndexBuilder; refactor code based on feedback

* update based on PR feedback

* Update DataProcessDocument.pdl

fixed typo wording.

* add not null check for data process info
2020-06-29 10:20:22 -07:00
Liangjun Jiang
92c4a3689e
Data process entity (#1680)
* add job info as aspect of a dataset

* add job urn def., aspect and entity

* job entity with upstream and downstream lineage

* use job urn in upstream & downstream

* add Job entity rest APIs

* rest.li api, impl and factory for job entity

* code cleanup

* use pdl; onboard data process entity

* add es index json

* fix gradlew build ignored tasks

* add a comment about data process info field

* fix style warning issues

* update content based on PR

* checked in generated snapshot json

* updated based on PR feedback

* update data process data format

* updated based on code review feedback

* revert back gms & mce-job docker image

* delete temp files

* update based pr feedback

* file name and a typo

* format with linkedin style

Co-authored-by: Liangjun <liajiang@expediagroup.com>
2020-06-09 15:42:08 -07:00
Mars Lan
4f221f9a12
build(docker): refactor docker build scripts (#1687)
* build(docker): refactor docker build scripts

- add "build" option to docker-compose files to simplify rebuilding of images
- create "start.sh" script so it's easier to override "command" in the quickstart's docker-compose file
- use dockerize to wait for requisite services to start up
- add a dedicated Dockerfile for kafka-setup

This fixes https://github.com/linkedin/datahub/issues/1549 & https://github.com/linkedin/datahub/issues/1550
2020-06-08 13:37:14 -07:00
Kerem Sahin
c009326bae Fix lowercase_keyword analyzer settings for people entity 2020-02-06 01:39:05 -08:00
Kerem Sahin
e56b6a2871 Add forward slash escape for Elasticsearch queries 2020-02-05 19:05:49 -08:00
Kerem Sahin
165d4aef95 Documentation update part-1 2019-12-18 18:57:18 -08:00
Kerem Sahin
4500e9ce7b Set Elasticsearch Docker container max heap size to 1GB and increase timeout to 120s for Elasticsearch to be ready 2019-11-12 18:13:37 -08:00
Kerem Sahin
bb38a9467a Add --build flag when using docker-compose up to always build elasticsearch-setup image 2019-10-05 15:28:54 -07:00
Kerem Sahin
c65a65c0b7 Update corp user search index mapping 2019-10-03 19:26:28 -07:00
Kerem Sahin
ea15628912 Add upstreams field to dataset search index mapping 2019-09-26 20:54:03 -07:00
Kerem Sahin
28b876f323 Update docs: No need for running initialization script for creating search indices 2019-09-12 19:06:35 -07:00
Kerem Sahin
1a3ddff4a4 Creating an image for elasticsearch-setup to automatically create indices 2019-09-12 19:04:29 -07:00
Kerem Sahin
1bf0ff72b4 Doc update: adding docker-compose pull before up to always get the latest version of images 2019-09-10 17:51:17 -07:00
Kerem Sahin
0dc5bd9fe0 Add documentation 2019-09-08 20:25:58 -07:00
Kerem Sahin
3f4048e2a7 Add docker compose for GMS and update other docker compose files 2019-09-02 16:44:34 -07:00
Kerem Sahin
23339df23a Initial commit for Data Hub 2019-08-31 20:51:14 -07:00