Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Merges in changes from our ES7 branch, and drops support for ES5.
This is a breaking change due to the upgrade, we have a ES5 branch at the commit before this.
* Switch GMA dep from bintray to artifactory.
Bintray is set for deprecation and EoL in a few months.
We still need bintray for pegasus until restli team migrates off of it.
* tick version to version that should only be on artifactory
* Productionalizing GraphQL Part 1: Dataset Query support + introducing common datahub-graphql-core module.
Co-authored-by: John Joyce <jjoyce0510@gmail.com>
These tests verify that, given an index settings and mappings, data can be written to the index, and read from it with a query_all query. These are very simple sanity tests.
We can, and should, write more complex tests that specific to each index in the future.
Involves fixing some types. Bright side is that ModelValidation actually checks aspects now (it did not before because aspects are not records but unions!).
Move code to linkedin/datahub-gma.
"GMA" (Generalized Metadata Architecture) is the backend of DataHub, and has been moved to its own repository.
This deletes the code that was moved and uses jars that GMA publishes to bintray to load it.
Note that not all of GMA was moved, but most of it. We may still move more things to the other repository in the future.
* Bump up kafkaAvroSerde to support security config for Confluent Schema Registry
* Support certs in secrets
* Extra Spring config (e.g. security)
* Optional values
* Clarify log warnings
* Update faq.md
Co-authored-by: Lars Nielsen <Lars.Nielsen@kindredgroup.com>
Co-authored-by: Mars Lan <mars.th.lan@gmail.com>
Start adding java ETL examples, starting with kafka etl.
We've had a few requests to start providing Java examples rather than Python due to type safety.
I've also started to add these to metadata-ingestion-examples to make it clearer these are *examples*. They can be used directly or as a basis for other things.
As we port to Java we'll move examples to contrib.
metadata-models 80.0.0 -> 90.0.13:
90.0.13: Roll forward: Fix the open source build by avoiding URN method that isn't part of the open source URN.
90.0.2: Refactor listUrnsFromIndex method
90.0.0: Start distinguishing between [] aspects vs null aspects input param
89.0.4: Fix the open source build by avoiding URN method that isn't part of the open source URN.
89.0.2: fix some test case name
89.0.0: META-12686: Made the MXE_v5 topics become strictly ACL'ed to avoid the wildcard write ACL as "MetadataXEvent.+"
88.0.6: change DAO to take Storage Config as input
88.0.3: Add a comment on lack of avro generation for MXEv5 + add MXEv5 to the pegasus validation task.
87.0.15: META-12651: Integrate the metadata-models-ext with metadata-models
87.0.13: add StorageConfig to Local DAO
87.0.3: Treat empty aspect vs optional aspect same until all clients are migrated
87.0.2: Treat empty aspect vs optional aspect differently
87.0.1: META-12533: Skip processing unregistered aspect specific MAE.
83.0.6: action method to return list of urns from strong consistent index
83.0.4: Change input param type for batch backfill
83.0.3: Implement batch backfill
83.0.1: Implement support for OR filter in browse query
82.0.10: Throw UnsupportedOperationException for unsupported condition types in search filter
82.0.6: Implement local secondary backfilling index as part of backfill method
82.0.5: [strongly consistent index] implement getUrns method
82.0.4: Add indexing urn fields to the local secondary index
82.0.0: Render Delta fiels in the MCE_v5.
81.0.1: Add pegasus to avro conversion for FMCE
80.0.4: add get all support for BaseSingleAspectEntitySimpleKeyResource
80.0.2: Add a BaseSearchWriterDAO with an ESBulkWriterDAO implementation.
80.0.1: META-12254: Produce aspect specific MAE with always emit option
80.0.0: Convert getNodesInTraversedPath to getSubgraph to return complete view of the subgraph (nodes+edges)
