* (refactor): Convert dataPlatforms to GMA aspect and associated resource to GMA resource.
BREAKING CHANGE: /datasets/dataPlatforms API is now changed to become GMA resource.
* Change documentation style
Move code to linkedin/datahub-gma.
"GMA" (Generalized Metadata Architecture) is the backend of DataHub, and has been moved to its own repository.
This deletes the code that was moved and uses jars that GMA publishes to bintray to load it.
Note that not all of GMA was moved, but most of it. We may still move more things to the other repository in the future.
The coupling was between the static path extractor API. This broken by making a new `UrnPathExtractor` interface, and adding an overload of `EbeanLocalDAO`'s constructor to accept one (no breaking constructor change). The old constructors default to an `EmptyPathExtractor`, which does nothing (which is a breaking behavioral change, see below).
BREAKING: `DatasetUrnPathExtractor` was deleted. No one should've been depending on this directly. However, downstreams that were relying on it being there at runtime (dataset GMS) need to copy `DatasetUrnPathExtractor` and create their `EbeanLocalDAO` with one. Note that this is a little dangerous becasue it is a runtime behavioral change only. Potential impact is that SCSI suddenly stops working as intended.
SYNC=metadata-models_101.0.0
Requested by a few people in OS. See https://github.com/linkedin/datahub/issues/1840.
Companies need full customization over the topic name. This new class should be easily customizable by using a spring factory.
TODO to finish the implmentation for v5. For right now v5 is LI only and unfinished. Getting this in for v4 so it is useful to other companies now.
TODO AFTER OPEN SOURCE PUSH - make configurable via spring
TODO AFTER SUBMIT - see where else we can use this (jobs, where else?)
- Add commonsLang to build file.
- Add emails field to CorpUserInfoDocument (either this should be synced or the index builder not synced in the future).
- Fix EbeanLocalDAOTest which used internal Urn API.
- Fix BaseSearchableEntityResource "backfill" override return types (and regenerate snapshots).
- EbeanlocalDAO's constructor changed; now requires URN class.
- Add restli resource module as dependency of :gms:api as it now contains a needed PDL model.
- Upgrade to checkstyle 8
- Copy javadoc checks from Google
- Disable missing class and method checks for now, too many warnings. I'll have to figure out how to suppress them instead.
- Fix other issues, which are mostly missing periods at the end of sentences and lack of paragraph tags.
Revert "Reverting the commit range: 8dfdb73ac6c73581ef56c0d81c21a2a92e8a1a02..194bd6f57f4a4d075d2ea1f442397d1139080f7a."
This reverts commit ab178ec1469fa72c0c339f0b842e7ff0850e7c74.
- Upgrade to checkstyle 8
- Copy javadoc checks from Google
- Disable missing class and method checks for now, too many warnings. I'll have to figure out how to suppress them instead.
- Fix other issues, which are mostly missing periods at the end of sentences and lack of paragraph tags.
Context:
We will need to perform customized query to get info from Neo4j such as get paths with only unremoved dataset, get paths for field-path lineage.
What are changed:
- Adding a new API in Neo4jQueryDAO so that we could perform a customized query from API user side(GMS)
metadata-models 80.0.0 -> 90.0.13:
90.0.13: Roll forward: Fix the open source build by avoiding URN method that isn't part of the open source URN.
90.0.2: Refactor listUrnsFromIndex method
90.0.0: Start distinguishing between [] aspects vs null aspects input param
89.0.4: Fix the open source build by avoiding URN method that isn't part of the open source URN.
89.0.2: fix some test case name
89.0.0: META-12686: Made the MXE_v5 topics become strictly ACL'ed to avoid the wildcard write ACL as "MetadataXEvent.+"
88.0.6: change DAO to take Storage Config as input
88.0.3: Add a comment on lack of avro generation for MXEv5 + add MXEv5 to the pegasus validation task.
87.0.15: META-12651: Integrate the metadata-models-ext with metadata-models
87.0.13: add StorageConfig to Local DAO
87.0.3: Treat empty aspect vs optional aspect same until all clients are migrated
87.0.2: Treat empty aspect vs optional aspect differently
87.0.1: META-12533: Skip processing unregistered aspect specific MAE.
83.0.6: action method to return list of urns from strong consistent index
83.0.4: Change input param type for batch backfill
83.0.3: Implement batch backfill
83.0.1: Implement support for OR filter in browse query
82.0.10: Throw UnsupportedOperationException for unsupported condition types in search filter
82.0.6: Implement local secondary backfilling index as part of backfill method
82.0.5: [strongly consistent index] implement getUrns method
82.0.4: Add indexing urn fields to the local secondary index
82.0.0: Render Delta fiels in the MCE_v5.
81.0.1: Add pegasus to avro conversion for FMCE
80.0.4: add get all support for BaseSingleAspectEntitySimpleKeyResource
80.0.2: Add a BaseSearchWriterDAO with an ESBulkWriterDAO implementation.
80.0.1: META-12254: Produce aspect specific MAE with always emit option
80.0.0: Convert getNodesInTraversedPath to getSubgraph to return complete view of the subgraph (nodes+edges)
* add job info as aspect of a dataset
* add job urn def., aspect and entity
* job entity with upstream and downstream lineage
* use job urn in upstream & downstream
* add Job entity rest APIs
* rest.li api, impl and factory for job entity
* code cleanup
* use pdl; onboard data process entity
* add es index json
* fix gradlew build ignored tasks
* add a comment about data process info field
* fix style warning issues
* update content based on PR
* checked in generated snapshot json
* updated based on PR feedback
* update data process data format
* updated based on code review feedback
* revert back gms & mce-job docker image
* delete temp files
* update based pr feedback
* file name and a typo
* format with linkedin style
Co-authored-by: Liangjun <liajiang@expediagroup.com>
Expose EbeanServer from EbeanLocalDAO
Create test URNs and use those from entity models
Implement interface for browse
MP_VERSION=metadata-models:62.0.3
MP_VERSION=wherehows-samza:1.0.56
Changes include:
Remove all keys that can be moved back to respective GMS
Add support for <, <=, >, >= conditions for the filter API
Update Conditions model for <, <=, >, >= conditions
MP_VERSION=metadata-models:58.0.1
MP_VERSION=wherehows-samza:1.0.56
58.0.1: Remove all keys that can be moved back to respective GMS
58.0.0: Revert "Reverting the commit range: f0c894b490d3df047837cf2fb7b9911c86188cae..4b5f31ed8844f818d7db0880d30c8dc8c7ac0087."
57.0.16: Reverting the commit range: f0c894b490d3df047837cf2fb7b9911c86188cae..4b5f31ed8844f818d7db0880d30c8dc8c7ac0087.
57.0.15: Disable filtering removed entities in browse until META-10900 is solved
57.0.14: (resubmit) add graph index builder for ai-metadata entities and relationships
57.0.13: Reverting the commit range: 830e63b4b40cf701db216952c34d731a7a82ea1d..4255871452062c2fd14651cb4fffb7d337bad300.
57.0.12: add graph index builder for ai-metadata entities and relationships
57.0.11: Fix bug which sets removed field to always true while building DatasetDocument
57.0.10: Change p12 file name to new ina group name
57.0.9: Add removal field in field compliance to flag the proposal as removal or not.
57.0.8: Adding action Builder for DatasetInstance entity
57.0.7: Adding GMA entities and relations for GridWorkflow and GridWorkflowExecution
57.0.6: Adding dataType and dataClassification to the search document
57.0.5: Rename graph entity MlTrainedModel to MlTrainedModelEntity
57.0.4: Code to form the FollowedBy Graph based on the Follow Aspect
57.0.3: add graph entity and relationship models for ai-metadata
57.0.2: Refactor incorrect use of mock in variable names
57.0.1: Add support for <, <=, >, >= conditions for the filter API
57.0.0: Update Conditions model for <, <=, >, >= conditions
56.0.5: update version of pegasus metadata plugin
56.0.4: update container dependency
56.0.3: Move mlFeatures from SnapshotRequestBuilders to ActionRequestbuilder
56.0.2: Adding reserved versions aspect
56.0.1: Create search filter for compliance pending review proposal.
56.0.0: Add Likes aspect resource in metadata restli utils
55.0.6: Fix a bug with getAll API
55.0.5: Move applicable metadata-store SnapshotRequestBuilders to ActionRequestbuilder
55.0.4: EspressoDAO: Updated to expect a separator between entityType and aspectName for config mapping keys
55.0.3: Added EspressoRecordSerializer and EspressoDAOUtils
55.0.2: Rewrote EspressoLocalDAOTest with a mocked EspressoAccessor
55.0.1: Migrate metric-gms SnapshotRequestBuilders to ActionRequestBuilder
55.0.0: [Wormhole] Deprecate Holdem-centric locations in favor of the more general CORP locations, which contain Holdem.
54.0.1: Migrate job-gms SnapshotRequestBuilders to ActionRequestBuilder
wherehows-samza 1.0.56 -> 1.0.56:
1.0.56: Gradle5 migration
MP_VERSION=metadata-models:58.0.1
MP_VERSION=wherehows-samza:1.0.56
This commit is automatically generated by li-opensource tool.
54.0.0: Filter removed documents during browse
53.0.15: Throwing 404, when no aspects found in DB for a given entity
53.0.14: add node label when updating relationship
53.0.8: Handle * character in the directory path for browse
53.0.4: apply label for add/update graph node&edge
53.0.0: META-10395: Don't package KafkaAuditHeader and UUID classes in mxe-avro
52.0.7: Add API in search DAO to support query filters
52.0.5: META-10073: Refactor remote DAO to use the new Ingest action method
51.0.16: allow query dao use default order by from neo4j
51.0.15: enable dataset indexing in graph
51.0.12: Move EMPTY_FILTER to RestliConstants
51.0.11: Add KafkaEventProducerFactory to utils
51.0.10: Create in-memory Neo4j in Neo4jDriverFactory if integration config is set
51.0.9: Generalized add() in BaseLocalDAO and EbeanLocalDAO
51.0.4: Move Neo4jTestServerBuilder to metadata test utils
51.0.3: Move makeRelationshipFilter to neo4j utils
50.0.7: Implement Neo4jDriverFactory
MP_VERSION=metadata-models:54.0.1
MP_VERSION=wherehows-samza:1.0.56
