proposal: New EmbeddingRetriever for Haystack 2.0 (#3558)

* Add EmbeddingRetriever proposal

* Update with Sara's feedback

* Consistent naming

proposals/text/3558-embedding_retriever.md

@@ -0,0 +1,129 @@
+- Start Date: 2022-11-11
+- Proposal PR: https://github.com/deepset-ai/haystack/pull/3558
+- Github Issue: 
+  # Summary  
+    
+- Current EmbeddingRetriever doesn't allow Haystack users to provide new embedding methods and is 
+  currently constricted to farm, transformers, sentence transformers, OpenAI and Cohere based 
+  embedding approaches. Any new encoding methods need to be explicitly added to Haystack 
+  and registered with the EmbeddingRetriever.  
+ 
+
+- We should allow users to easily plug-in new embedding methods to EmbeddingRetriever. For example, a Haystack user should be able to 
+  add custom embeddings without having to commit additional code to Haystack repository.
+
+  # Basic example  
+    EmbeddingRetriever is instantiated with:  
+
+  ``` python
+	  retriever = EmbeddingRetriever(
+	      document_store=document_store,
+	      embedding_model="sentence-transformers/multi-qa-mpnet-base-dot-v1",
+	      model_format="sentence_transformers",
+	  )
+  ```
+- The current approach doesn't provide a pluggable abstraction point of composition but 
+  rather attempts to satisfy various embedding methodologies by having a lot of 
+  parameters which keep ever expanding.
+
+
+- The new approach allows creation of the underlying embedding mechanism (EmbeddingEncoder) 
+  which is then in turn plugged into EmbeddingRetriever.  For example:  
+ 
+  ``` python
+	  encoder = SomeNewFancyEmbeddingEncoder(api_key="asdfklklja", 
+                                          query_model="text-search-query",
+                                          doc_model="text-search-doc")
+  ```
+
+- EmbeddingEncoder is then used for the creation of EmbeddingRetriever. EmbeddingRetriever 
+  init method doesn't get polluted with additional parameters as all of the peculiarities 
+  of a particular encoder methodology are contained on in its abstraction layer.  
+
+  ``` python
+	  retriever = EmbeddingRetriever(
+	      document_store=document_store,
+	      encoder=encoder
+	  )
+  ```
+  
+  # Motivation
+
+- Why are we doing this? What use cases does it support? What is the expected outcome?  
+    
+  We could certainly keep the current solution as is; it does implement a decent level 
+  of composition/decoration to lower coupling between EmbeddingRetriever and the underlying 
+  mechanism of embedding (sentence transformers, OpenAI, etc). However, the current mechanism 
+  in place basically hard-codes available embedding implementations and prevents our users from 
+  adding new embedding mechanism by themselves outside of Haystack repository. We also might 
+  want to have a non-public dC embedding mechanism in the future. In the current design a non-public 
+  dC embedding mechanism would be impractical. In addition, the more underlying implementations we 
+  add we'll continue to "pollute" EmbeddingRetriever init method with more and more parameters. 
+  This is certainly less than ideal long term.  
+
+
+- EmbeddingEncoder classes should be subclasses of BaseComponent! As subclasses of BaseComponent,
+  we can use them outside the EmbeddingRetriever context in indexing pipelines, generating the
+  embeddings. We are currently employing a kludge of using Retrievers which is quite counter-intuitive
+  and confusing for our users.
+
+
+- EmbeddingEncoder classes might sound overly complicated, especially with a distinguishing mechanism
+  name pre-appended (i.e CohereEmbeddingEncoder). Therefore, we'll adopt <specific>Embedder
+  naming scheme, i.e. CohereEmbedder, SentenceTransformerEmbedder and so on.
+
+  # Detailed design  
+
+- Our new EmbeddingRetriever would still wrap the underlying encoding mechanism in the form of 
+  _BaseEmbedder. _BaseEmbedder still needs to implement methods:
+	- embed_queries  
+	- embed_documents 
+
+
+- The new design approach differs is in the creation of EmbeddingRetriever - rather than hiding the underlying encoding 
+  mechanism one could simply create the EmbeddingRetriever with a specific encoder directly. For example:  
+
+  ```
+	  retriever = EmbeddingRetriever(
+	      document_store=document_store,
+	      encoder=OpenAIEmbedder(api_key="asdfklklja", model="ada"),
+	      #additional EmbeddingRetriever-abstraction-level parameters
+	  )
+  ```
+ 
+- If the "two-step approach" of EmbeddingRetriever initialization is no longer the ideal solution (issues with current 
+  schema generation and loading/saving via YAML pipelines) we might simply add the EmbeddingRetriever 
+  class for every supported encoding approach. For example, we could have OpenAIEmbeddingRetriever, CohereEmbeddingRetriever, 
+  SentenceTransformerEmbeddingRetriever and so on. Each of these retrievers will delegate the bulk of the work to an 
+  existing EmbeddingRetriever with a per-class-specific Embedder set in the class constructor (for that custom
+  encoding part). We'll get the best of both worlds. Each <Specific>EmeddingRetriever will have only the relevant primitives 
+  parameters for the **init()** constructor; the underlying EmbeddingRetriever attribute in <Specific>EmeddingRetriever 
+  will handle most of the business logic of retrieving, yet each retriever will use an appropriate per-class-specific 
+  Embedder for the custom encoding part.
+
+
+ 
+  # Drawbacks  
+- The main shortcoming are:  
+	- The "two-step approach" in EmbeddingRetriever initialization  
+	- Likely be an issue for the current schema generation and loading/saving via YAML pipelines (see solution above)  
+	- It is a API breaking change so it'll require code update for all EmbeddingRetriever usage both in our codebase and for Haystack users  
+	- Can only be done in major release along with other breaking changes  
+
+  # Alternatives  
+    
+  We could certainly keep everything as is :-)
+
+  # Adoption strategy  
+- As it is a breaking change, we should implement it for the next major release.  
+ 
+  # How do we teach this?  
+- This change would require only a minor change in documentation.  
+- The concept of embedding retriever remains, just the mechanics are slightly changed  
+- All docs and tutorials need to be updated  
+- Haystack users are informed about a possibility to create and use their own embedders for embedding retriever.
+- # Unresolved questions  
+    
+  Optional, but suggested for first drafts. What parts of the design are still  
+  TBD?  
+