Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Brian <brianwebtek@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Merges in changes from our ES7 branch, and drops support for ES5.
This is a breaking change due to the upgrade, we have a ES5 branch at the commit before this.
* (refactor): Convert dataPlatforms to GMA aspect and associated resource to GMA resource.
BREAKING CHANGE: /datasets/dataPlatforms API is now changed to become GMA resource.
* Change documentation style
These tests verify that, given an index settings and mappings, data can be written to the index, and read from it with a query_all query. These are very simple sanity tests.
We can, and should, write more complex tests that specific to each index in the future.
* refactor(gms): use BaseLocalDAO as the interface in factories & rest.li resources
Fixes#1974
* Revert yarn.lock change
Co-authored-by: Mars Lan <mars@trayminder.com>
Move code to linkedin/datahub-gma.
"GMA" (Generalized Metadata Architecture) is the backend of DataHub, and has been moved to its own repository.
This deletes the code that was moved and uses jars that GMA publishes to bintray to load it.
Note that not all of GMA was moved, but most of it. We may still move more things to the other repository in the future.
- Add commonsLang to build file.
- Add emails field to CorpUserInfoDocument (either this should be synced or the index builder not synced in the future).
- Fix EbeanLocalDAOTest which used internal Urn API.
- Fix BaseSearchableEntityResource "backfill" override return types (and regenerate snapshots).
- EbeanlocalDAO's constructor changed; now requires URN class.
- Add restli resource module as dependency of :gms:api as it now contains a needed PDL model.
fix: Update defaults of aspectNames params.
The last PR to sync internal code broke the external GMS, as code was now expected aspectNames to be null rather than empty by default. This preventing me logging into DataHub as the corp user request would fail (it assumed I asked for no aspects rather than all aspects).
TESTED: Built locally, launched with docker/dev.sh (so used latest frontend, but whatever). Verified I can now log into DataHub, browse and search for datasets, and view my profile.
metadata-models 80.0.0 -> 90.0.13:
90.0.13: Roll forward: Fix the open source build by avoiding URN method that isn't part of the open source URN.
90.0.2: Refactor listUrnsFromIndex method
90.0.0: Start distinguishing between [] aspects vs null aspects input param
89.0.4: Fix the open source build by avoiding URN method that isn't part of the open source URN.
89.0.2: fix some test case name
89.0.0: META-12686: Made the MXE_v5 topics become strictly ACL'ed to avoid the wildcard write ACL as "MetadataXEvent.+"
88.0.6: change DAO to take Storage Config as input
88.0.3: Add a comment on lack of avro generation for MXEv5 + add MXEv5 to the pegasus validation task.
87.0.15: META-12651: Integrate the metadata-models-ext with metadata-models
87.0.13: add StorageConfig to Local DAO
87.0.3: Treat empty aspect vs optional aspect same until all clients are migrated
87.0.2: Treat empty aspect vs optional aspect differently
87.0.1: META-12533: Skip processing unregistered aspect specific MAE.
83.0.6: action method to return list of urns from strong consistent index
83.0.4: Change input param type for batch backfill
83.0.3: Implement batch backfill
83.0.1: Implement support for OR filter in browse query
82.0.10: Throw UnsupportedOperationException for unsupported condition types in search filter
82.0.6: Implement local secondary backfilling index as part of backfill method
82.0.5: [strongly consistent index] implement getUrns method
82.0.4: Add indexing urn fields to the local secondary index
82.0.0: Render Delta fiels in the MCE_v5.
81.0.1: Add pegasus to avro conversion for FMCE
80.0.4: add get all support for BaseSingleAspectEntitySimpleKeyResource
80.0.2: Add a BaseSearchWriterDAO with an ESBulkWriterDAO implementation.
80.0.1: META-12254: Produce aspect specific MAE with always emit option
80.0.0: Convert getNodesInTraversedPath to getSubgraph to return complete view of the subgraph (nodes+edges)
* implement search feature
* add test for dataprocessIndexBuilder; refactor code based on feedback
* update based on PR feedback
* Update DataProcessDocument.pdl
fixed typo wording.
* add not null check for data process info
* add job info as aspect of a dataset
* add job urn def., aspect and entity
* job entity with upstream and downstream lineage
* use job urn in upstream & downstream
* add Job entity rest APIs
* rest.li api, impl and factory for job entity
* code cleanup
* use pdl; onboard data process entity
* add es index json
* fix gradlew build ignored tasks
* add a comment about data process info field
* fix style warning issues
* update content based on PR
* checked in generated snapshot json
* updated based on PR feedback
* update data process data format
* updated based on code review feedback
* revert back gms & mce-job docker image
* delete temp files
* update based pr feedback
* file name and a typo
* format with linkedin style
Co-authored-by: Liangjun <liajiang@expediagroup.com>
