12076 Commits

Author SHA1 Message Date
John Plaisted
48b47a7a59
Update for October 2020-09-25 12:12:21 -07:00
John Plaisted
ab5d409206
Add September 2020-09-25 12:10:52 -07:00
Jyoti Wadhwani
dc9c877984 [scsi] preserve the order of urns 2020-09-24 16:02:12 -07:00
John Plaisted
5e70f3648c Fix build after sync.
- Add build files for new module.
- Correctly edit TestUtils since it isn't synced.
- Reference new test utils.
- Delete duplicate pdl files.

SYNC=metadata-models_100.0.1
2020-09-24 16:02:12 -07:00
John Plaisted
542ae67cb1 Add support for customizing topic names via a convention.
Requested by a few people in OS. See https://github.com/linkedin/datahub/issues/1840.

Companies need full customization over the topic name. This new class should be easily customizable by using a spring factory.

TODO to finish the implmentation for v5. For right now v5 is LI only and unfinished. Getting this in for v4 so it is useful to other companies now.

TODO AFTER OPEN SOURCE PUSH - make configurable via spring
TODO AFTER SUBMIT - see where else we can use this (jobs, where else?)
2020-09-24 16:02:12 -07:00
Jyoti Wadhwani
b944910b0e extend filter finder method to get metadata from SCSI 2020-09-24 16:02:12 -07:00
Jyoti Wadhwani
c3c52cf8f6 add support for getWithExtraInfo in BaseLocalDAO 2020-09-24 16:02:12 -07:00
John Plaisted
8d536a54d3 Break dependency of metadata-dao on metadata-models.
This also breaks a few others transitively.
2020-09-24 16:02:12 -07:00
John Plaisted
e4ce0376d2 Fix open source build.
Do not use internal URN API.
2020-09-24 16:02:12 -07:00
John Plaisted
96da83033c Break dependency of metadata-test-utils on metadata-models. 2020-09-24 16:02:12 -07:00
Jyoti Wadhwani
f07420a434 change finder return type for listing urns from SCSI 2020-09-24 16:02:12 -07:00
Jyoti Wadhwani
9d402574e2 use finder for list urns from index 2020-09-24 16:02:12 -07:00
Jyoti Wadhwani
caf22dc921 add support for conditions other than equals 2020-09-24 16:02:12 -07:00
Kerem Sahin
e042289b81
Update graph-onboarding.md 2020-09-23 14:58:35 -07:00
John Plaisted
1881b06838
Update agenda with times and people links 2020-09-23 10:05:56 -07:00
Fredrik Sannholm
d50b9c01b4
fix (docker): Fix install of Chrome in frontend Dockerimage (#1889)
* fix (docker): Fix install of Chrome in frontend Dockerimage

Retry installing Chrome after dependencies have been installed

* fix (docker): Install Chrome with apt-get

Install Chrome and dependencies at the same time, using apt-get
2020-09-22 12:02:37 -07:00
Kerem Sahin
27baa10004
Update townhall-history.md 2020-09-21 11:42:55 -07:00
John Plaisted
d6c2b5f56e
Update with September Agenda 2020-09-21 11:24:06 -07:00
Mars Lan
a897f7a3bf
build: add failure notification on push (#1873)
* build: add failure notification on push

* build: fix build (#1872)

* build: fix build

`kafkaSerializers` is already included transitively via `kafkaAvroSerde`

* Update build.gradle
2020-09-21 10:29:24 -07:00
Charlie Tran
57fdc5c00c
Adds ability for midtier to serve custom dataset properties from aspect (#1881) 2020-09-20 11:04:51 -07:00
Mars Lan
205a6c920b
Update debugging.md 2020-09-18 04:21:56 -07:00
Charlie Tran
3b7eb08284
Fixes current user entity not being populated correctly (#1880) 2020-09-17 23:03:15 -07:00
Jyoti Wadhwani
4f809fb11b
more changes on faceted search 2020-09-17 16:51:17 -07:00
Jyoti Wadhwani
c9deda8075
update the doc for facet field 2020-09-17 16:22:38 -07:00
Mars Lan
c07a3a9b8f
Delete saxo_bank_data_workbench.pdf 2020-09-15 11:09:42 -07:00
Mars Lan
6ba13469ae
Update townhall-history.md 2020-09-15 11:09:24 -07:00
Mars Lan
1f76ca7453
Update links.md 2020-09-15 11:08:44 -07:00
Mars Lan
2732a58590
Update README.md 2020-09-15 11:08:12 -07:00
Charlie Tran
84efa73098
fix (frontend): Partially fixes lineage issues and dataset API handling (#1874)
* Initial fix for lineage graph

* Fixes broken test by lineage fix commit and returns empty response instead of 500 for upstreams
2020-09-14 19:09:02 -07:00
Kerem Sahin
7c7de50be8
feat(dashboard): Dashboard metadata models (#1775)
* Dashboard metadata models

* Update pairings in Contains and DownstreamOf relationships

* Adding ChartDataSourceType

* Extend chart types

* Make chart query an aspect & add chartIds to dashboard search document

* Add urn validation for chart & dashboard urns
2020-09-14 12:01:54 -07:00
Mars Lan
e66a1be8fb
build: fix build (#1872)
* build: fix build

`kafkaSerializers` is already included transitively via `kafkaAvroSerde`

* Update build.gradle
2020-09-14 10:15:41 -07:00
John Plaisted
fbb52304e8
doc(rfc): Add requirements / non requirements section to RFC. (#1818)
This seems helpful to reviewers (and probably also the author). Motivation is too high level to be able to properly review a design. We need a clear set of requirements to review the design against.
2020-09-13 18:41:11 -07:00
John Plaisted
6ece2d6469
Start adding java ETL examples, starting with kafka etl. (#1805)
Start adding java ETL examples, starting with kafka etl.

We've had a few requests to start providing Java examples rather than Python due to type safety.

I've also started to add these to metadata-ingestion-examples to make it clearer these are *examples*. They can be used directly or as a basis for other things.

As we port to Java we'll move examples to contrib.
2020-09-11 13:04:21 -07:00
Jyoti Wadhwani
91486a2ffd
feat(ML models): RFC for ML models (#1812)
* Create RFC for ML Model

* Add ml model graph file

* add link to graph

* Add high level design for ML model

* ai->ml

* Rename 000-ml_models.md to README.md

* Add link to RFC PR

* Update README.md

* address some comments

* Delete ml_model_graph.png

* add ML Model graph

* Address reviewer comments

* Address more comments

* Update README.md

* Rename docs/rfc/active/000-ml_models/README.md to docs/rfc/active/1812-ml_models/README.md

* Delete high_level_design.png

* Delete ml_model_graph.png

* Add files via upload

Co-authored-by: Mars Lan <mars.th.lan@gmail.com>
2020-09-11 11:32:58 -07:00
John Plaisted
8223cdcbdb Fix build after merge:
- Add commonsLang to build file.
- Add emails field to CorpUserInfoDocument (either this should be synced or the index builder not synced in the future).
- Fix EbeanLocalDAOTest which used internal Urn API.
- Fix BaseSearchableEntityResource "backfill" override return types (and regenerate snapshots).
- EbeanlocalDAO's constructor changed; now requires URN class.
- Add restli resource module as dependency of :gms:api as it now contains a needed PDL model.
2020-09-11 09:15:56 -07:00
Kerem Sahin
411e0a8fee Implementing a truly free-form query interface on Neo4j query DAO 2020-09-11 09:15:56 -07:00
Kerem Sahin
46d8f4f05d Update backfill API to allow for backfill modes 2020-09-11 09:15:56 -07:00
John Plaisted
6b9a053f6e ROLL FORWARD: Add new style checks and fix issues.
- Upgrade to checkstyle 8
- Copy javadoc checks from Google
- Disable missing class and method checks for now, too many warnings. I'll have to figure out how to suppress them instead.
- Fix other issues, which are mostly missing periods at the end of sentences and lack of paragraph tags.

Revert "Reverting the commit range: 8dfdb73ac6c73581ef56c0d81c21a2a92e8a1a02..194bd6f57f4a4d075d2ea1f442397d1139080f7a."

This reverts commit ab178ec1469fa72c0c339f0b842e7ff0850e7c74.
2020-09-11 09:15:56 -07:00
Jyoti Wadhwani
f277a49dd0 [scsi] add resilience to EbeanLocalDAO 2020-09-11 09:15:56 -07:00
Na Zhang
5fcbd4cdc7 add new entity union and validation 2020-09-11 09:15:56 -07:00
Jyoti Wadhwani
a3a892f5ec change listUrns to return typed urns 2020-09-11 09:15:56 -07:00
Chris Lee
679069e16f Made the log to WARN for the v5 producer early termination. 2020-09-11 09:15:56 -07:00
John Plaisted
6ac7622af6 Reverting the commit range: 8dfdb73ac6c73581ef56c0d81c21a2a92e8a1a02..194bd6f57f4a4d075d2ea1f442397d1139080f7a.
REVERTED RB=99999 PCVALIDATIONOVERRIDE I18NOVERRIDE CIOVERRIDE TRUNKBLOCKERFIX

See https://crt.prod.linkedin.com/#/testing/executions/77e10182-d60f-4c8d-9e55-599bdc4384e0/execution for more details.
2020-09-11 09:15:56 -07:00
John Plaisted
b9f11ae21b Add new style checks and fix issues.
- Upgrade to checkstyle 8
- Copy javadoc checks from Google
- Disable missing class and method checks for now, too many warnings. I'll have to figure out how to suppress them instead.
- Fix other issues, which are mostly missing periods at the end of sentences and lack of paragraph tags.
2020-09-11 09:15:56 -07:00
John Plaisted
08f2ce61d9 Free form API for getPaths
Context:
We will need to perform customized query to get info from Neo4j such as get paths with only unremoved dataset, get paths for field-path lineage.

What are changed:
- Adding a new API in Neo4jQueryDAO so that we could perform a customized query from API user side(GMS)
2020-09-11 09:15:56 -07:00
Jyoti Wadhwani
02ffe16624 [scsi] support for multiple filters 2020-09-11 09:15:56 -07:00
Jyoti Wadhwani
8813ed9c86 save aspect fields to index table 2020-09-11 09:15:56 -07:00
Kerem Sahin
64e5160365 Implement backfill API which uses SCSI 2020-09-11 09:15:56 -07:00
John Plaisted
bc7a29802d Add user email to the search index.
ldap and email are technically different in a few ways. Email not only includes the domain (@linkedin.com), but the user part of it may be different than ldap. Generally emails are username@domain; at LinkedIn ldaps are generally usernames truncated to 8 characters.

For the sake of being technically correct; also index emails so that if clients want to search email, they can search email rather than ldap.
2020-09-11 09:15:56 -07:00
John Plaisted
55344330a1 [metadata-models] support range filter for ESSearchDAO 2020-09-11 09:15:56 -07:00