4 Commits

Author SHA1 Message Date
Kevin Hu
ebdaa0e359
feat(ingest): Feast ingestion integration (#2605)
* Add feast testing setup

* Init Feast test script

* Add feast to dependencies

* Update feast descriptors

* Sort integrations

* Working feast pytest

* Clean up feast docker-compose file

* Expand Feast tests

* Setup feast classes

* Add continuous and bytes data to feature types

* Update field type mapping

* Add PDLs

* Add MLFeatureSetUrn.java

* Comment out feast setup

* Add snapshot file and update inits

* Init Feast golden files generation

* Clean up Feast ingest

* Feast testing comments

* Yield Feature snapshots

* Fix Feature URN naming

* Update feast MCE

* Update Feature URN prefix

* Add MLEntity

* Update golden files with entities

* Specify feast sources

* Add feast source configs

* Working feast docker ingestion

* List entities and features before adding tables

* Add featureset names

* Remove unused

* Rename feast image

* Update README

* Add env to feast URNs

* Fix URN naming

* Remove redundant URN names

* Fix enum backcompatibility

* Move feast testing to docker

* Move URN generators to mce_builder

* Add source for features

* Switch TypeClass -> enum_type

* Rename source -> sourceDataset

* Add local Feast ingest image builds

* Rename Entity -> MLPrimaryKey

* Restore features and keys for each featureset

* Do not json encode source configs

* Remove old source properties from feature sets

* Regenerate golden file

* Fix race condition with Feast tests

* Exclude unknown source

* Update feature datatype enum

* Update README and fix typos

* Fix Entity typo

* Fix path to local docker image

* Specify feast config and version

* Fix feast env variables

* PR fixes

* Refactor feast ingest constants

* Make feature sources optional for back-compatibility

* Remove unused GCP files

* adding docker publish workflow

* Simplify name+namespace in PrimaryKeys

* adding docker publish workflow

* debug

* final attempt

* final final attempt

* final final final commit

* Switch to published ingestion image

* Update name and namespace in java files

* Rename FeatureSet -> FeatureTable

* Regenerate codegen

* Fix initial generation errors

* Update snapshot jsons

* Regenerated schemas

* Fix URN formats

* Revise builds

* Clean up feast URN builders

* Fix naming typos

* Fix Feature Set -> Feature Table

* Fix comments

* PR fixes

* All you need is Urn

* Regenerate snapshots and update validation

* Add UNKNOWN data type

* URNs for source types

* Add note on docker requirement

* Fix typo

* Reorder aspect unions

* Refactor feast ingest functions

* Update snapshot jsons

* Rebuild

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-09 15:07:04 -07:00
John Plaisted
6f59a91865
Bump GMA to latest (#2007)
Involves fixing some types. Bright side is that ModelValidation actually checks aspects now (it did not before because aspects are not records but unions!).
2020-11-18 14:07:48 -08:00
Kerem Sahin
5bf797b216 corp-identity-gms 1.0.0 -> 1.0.25:
1.0.24: Corp user search across teams and skills
    1.0.21: Make /corpGroups /gridUsers /gridGroups extend BaseEntityResource
    1.0.17: Use correct util function to load resource file
    1.0.16: Add ingest, backfill & getSnapshot action methods to all top-level resources in corp-identity-gms
    1.0.13: Onboard search query templates on corp-identity-gms
     1.0.9: Fix batch get and add client for batch get
     1.0.8: Change package name for corpuser search config
     1.0.7: Use search config to get autocomplete field
     1.0.6: Implement searchable client
     1.0.5: Auto-complete backend support
     1.0.3: Add search API

metadata-models 24.0.0 -> 38.1.6:
    38.1.3: Index active status to corp user search index
    38.1.2: Change update response to create response for create API
   38.0.10: Mark BaseAspectResource as deprecated
    38.0.9: Allow TYPEREF items which have primitive types for arrays in models
    38.0.7: Add get-and-set-if-absent function to Local DAO
    38.0.6: Add find entities with one relationship in query dao
    38.0.4: Fix the inconsistency use of constants and urn params in query dao
    38.0.2: Parse source map to obtain the urn
    38.0.1: Search document validator in Index Builder
    38.0.0: Add urns to search result metadata
    37.0.7: Refactor the query dao
    37.0.6: Use test models in neo4j dao
    37.0.5: Drop metadata model structural assumptions made in neo4j DAOs
    37.0.2: Return empty list from getBrowsePaths if browsePaths field doesn't exist
    36.0.3: Drop elasticsearch-dao's metadata-models dependency
   35.0.10: ES Search DAO to handle null values
    35.0.5: Ebean local Dao query string match
    35.0.4: Drop all search & browse configs that have been moved to individual GMS
    35.0.3: Add ReportTo relationship model
    35.0.0: Load resource file properly
    34.0.9: Make RestliAuditor injectable
    34.0.8: Use encoded query in the test resource
    34.0.4: Handle empty aspects param correctly for backfill & getSnapshot actions
    34.0.1: Remove corp user specific files from metadata-models
    34.0.0: Add backfill & getSnapshot actions to BaseEntityResource
   32.0.16: Merge data template classes into metadata-dao's main artifact
   32.0.14: Replace "update" method with "ingest" action in BaseEntityResource
   32.0.13: Make fliter & sortCriteria parameters optional as they should have been
   32.0.12: Move AspectVersion & SnapshotKey back to their original namespaces
   32.0.11: Break metadata-dao's dependency on metadata-models
   32.0.10: Move model validators to a separate module
    32.0.9: Extract principal from the request context for user AuditStamp
    32.0.8: Fixing nullability annotations for search/autocomplete/browse resources & daos
    32.0.7: Move DAO-specific models to metadata-dao module
    32.0.4: Fix search finder not returning total search results count
    32.0.3: Implement get_all using search index
    32.0.2: Add missing nullability annotation
    32.0.1: Use more consistent naming for the test models
    31.0.1: Use test-specific metadata models in metadata-dao
    31.0.0: Add sort order to Search Dao
    30.0.2: Rename Aspect test model to AspectUnion to avoid confusion
    30.0.1: Committing migration for metadata-models.
    30.0.0: Add default autocomplete field in search config
   29.0.16: Modify testcase to account for empty filters
   29.0.15: Add searchable interface that clients can use
   29.0.14: Use test-specific metadata models in ebean-dao
   29.0.12: Move TestUtils to metadata-test-models module
   29.0.11: Refactor all tests in metadata-restli to use test models
   29.0.10: Move li-metadata-test-utils, metadata-test-models, metadata-test-utils into a new metadata-testing directory to improve code organization.
            Note that this is a backward compatible change as this doesn't alter the produced artifacts.
    29.0.9: Move test-specific models to a stand-alone module
    29.0.2: Refactor the rest of validators
    28.0.3: Refactor validateSchema for aspect
    28.0.2: Implement searchDao for CorpUserInfo.
   27.0.16: Refactor for ModelValidation tests
   27.0.10: Add new relationship union to model utils
    27.0.9: Add plugin to rest client factory
    27.0.6: Add rest high level factory
    27.0.5: Fix a test bug when reviewing the code
    27.0.4: Add create via lambda API to BaseVersionedAspectResource
    27.0.2: Change return type of search finder to capture search result metadata in BaseSearchableEntityResource
    27.0.1: Drop the unnecessary get method from BaseEntityResource
    27.0.0: Add BaseBrowsableEntityResource
   26.0.15: Add autocomplete action to BaseSearchableEntityResource
   26.0.14: Add BaseSearchableEntityResource
   26.0.13: Add getUrnFromDocument & urnClassForDocument util methods that are needed in future RBs
   26.0.11: Add BaseVersionedAspectResource
    26.0.9: Index signals associated with dataset relevance
    26.0.4: Support namespace for ID generation
    26.0.1: Fix inconsistent instance variable naming in SearchResult
    25.0.6: Add entity-snapshot conversion
    25.0.5: Use test-specific metadata models in metadata-restli
    25.0.3: Add aspect filtering to BaseEntityResource
    24.0.9: Add update method to BaseEntityResource
    24.0.7: Fix for parameter types of getBrowsePaths action method

MP_VERSION=corp-identity-gms:1.0.25
MP_VERSION=metadata-models:38.1.6
MP_VERSION=wherehows-samza:1.0.29
2019-10-02 11:13:44 -07:00
Kerem Sahin
23339df23a Initial commit for Data Hub 2019-08-31 20:51:14 -07:00