mirror of
https://github.com/deepset-ai/haystack.git
synced 2026-01-24 05:34:05 +00:00
126 lines
5.3 KiB
Plaintext
126 lines
5.3 KiB
Plaintext
---
|
||
title: "JinaDocumentEmbedder"
|
||
id: jinadocumentembedder
|
||
slug: "/jinadocumentembedder"
|
||
description: "This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Jina AI Embeddings models. The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents."
|
||
---
|
||
|
||
# JinaDocumentEmbedder
|
||
|
||
This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Jina AI Embeddings models. The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents.
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||
| **Mandatory init variables** | "api_key": The Jina API key. Can be set with `JINA_API_KEY` env var. |
|
||
| **Mandatory run variables** | “documents”: A list of documents |
|
||
| **Output variables** | “documents”: A list of documents (enriched with embeddings) <br /> <br />”meta”: A dictionary of metadata |
|
||
| **API reference** | [Jina](/reference/integrations-jina) |
|
||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/jina |
|
||
|
||
## Overview
|
||
|
||
`JinaDocumentEmbedder` enriches the metadata of documents with an embedding of their content. To embed a string, you should use the [`JinaTextEmbedder`](jinatextembedder.mdx). To see the list of compatible Jina Embeddings models, head to Jina AI’s [website](https://jina.ai/embeddings/). The default model for `JinaDocumentEmbedder` is `jina-embeddings-v2-base-en`.
|
||
|
||
To start using this integration with Haystack, install the package with:
|
||
|
||
```shell
|
||
pip install jina-haystack
|
||
```
|
||
|
||
The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||
|
||
```python
|
||
embedder = JinaDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||
```
|
||
|
||
To get a Jina Embeddings API key, head to https://jina.ai/embeddings/.
|
||
|
||
### Embedding Metadata
|
||
|
||
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
|
||
|
||
You can do this easily by using the Document Embedder:
|
||
|
||
```python
|
||
from haystack import Document
|
||
from haystack_integrations.components.embedders.jina import JinaDocumentEmbedder
|
||
|
||
doc = Document(content="some text",
|
||
meta={"title": "relevant title",
|
||
"page number": 18})
|
||
|
||
embedder = JinaDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"), meta_fields_to_embed=["title"])
|
||
|
||
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
|
||
|
||
```
|
||
|
||
## Usage
|
||
|
||
### On its own
|
||
|
||
Here is how you can use the component on its own:
|
||
|
||
```python
|
||
from haystack_integrations.components.embedders.jina import JinaDocumentEmbedder
|
||
|
||
doc = Document(content="I love pizza!")
|
||
|
||
document_embedder = JinaDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||
|
||
result = document_embedder.run([doc])
|
||
print(result['documents'][0].embedding)
|
||
|
||
## [0.017020374536514282, -0.023255806416273117, ...]
|
||
|
||
```
|
||
|
||
:::note
|
||
We recommend setting JINA_API_KEY as an environment variable instead of setting it as a parameter.
|
||
|
||
:::
|
||
|
||
### In a pipeline
|
||
|
||
```python
|
||
from haystack import Pipeline
|
||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||
from haystack_integrations.components.embedders.jina import JinaDocumentEmbedder
|
||
from haystack_integrations.components.embedders.jina import JinaTextEmbedder
|
||
from haystack.components.writers import DocumentWriter
|
||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||
|
||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||
|
||
documents = [Document(content="My name is Wolfgang and I live in Berlin"),
|
||
Document(content="I saw a black horse running"),
|
||
Document(content="Germany has many big cities")]
|
||
|
||
indexing_pipeline = Pipeline()
|
||
indexing_pipeline.add_component("embedder", JinaDocumentEmbedder(api_key=Secret.from_token("<your-api-key>")))
|
||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||
indexing_pipeline.connect("embedder", "writer")
|
||
|
||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||
|
||
query_pipeline = Pipeline()
|
||
query_pipeline.add_component("text_embedder", JinaTextEmbedder(api_key=Secret.from_token("<your-api-key>")))
|
||
query_pipeline.add_component("retriever", InMemoryEmbeddingRetriever(document_store=document_store))
|
||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||
|
||
query = "Who lives in Berlin?"
|
||
|
||
result = query_pipeline.run({"text_embedder":{"text": query}})
|
||
|
||
print(result['retriever']['documents'][0])
|
||
|
||
## Document(id=..., mimetype: 'text/plain',
|
||
## text: 'My name is Wolfgang and I live in Berlin')
|
||
|
||
```
|
||
|
||
## Additional References
|
||
|
||
🧑🍳 Cookbook: [Using the Jina-embeddings-v2-base-en model in a Haystack RAG pipeline for legal document analysis](https://haystack.deepset.ai/cookbook/jina-embeddings-v2-legal-analysis-rag)
|