mirror of
https://github.com/deepset-ai/haystack.git
synced 2026-02-06 06:52:53 +00:00
97 lines
4.3 KiB
Plaintext
97 lines
4.3 KiB
Plaintext
---
|
|
title: "WatsonxGenerator"
|
|
id: watsonxgenerator
|
|
slug: "/watsonxgenerator"
|
|
description: "Use this component with IBM watsonx models like `granite-3-2b-instruct` for simple text generation tasks."
|
|
---
|
|
|
|
# WatsonxGenerator
|
|
|
|
Use this component with IBM watsonx models like `granite-3-2b-instruct` for simple text generation tasks.
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Most common position in a pipeline** | After a [PromptBuilder](/docs/pipeline-components/builders/promptbuilder.mdx) |
|
|
| **Mandatory init variables** | "api_key": An IBM Cloud API key. Can be set with `WATSONX_API_KEY` env var. <br /> <br />"project_id": An IBM Cloud project ID. Can be set with `WATSONX_PROJECT_ID` env var. |
|
|
| **Mandatory run variables** | "prompt": A string containing the prompt for the LLM |
|
|
| **Output variables** | "replies": A list of strings with all the replies generated by the LLM <br /> <br />"meta": A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
|
| **API reference** | [Watsonx](/reference/integrations-watsonx) |
|
|
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/watsonx |
|
|
|
|
## Overview
|
|
|
|
This integration supports IBM watsonx.ai foundation models such as `ibm/granite-13b-chat-v2`, `ibm/llama-2-70b-chat`, `ibm/llama-3-70b-instruct`, and similar. These models provide high-quality text generation capabilities through IBM's cloud platform. Check out the most recent full list in the [IBM watsonx.ai documentation](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-ibm.html?context=wx).
|
|
|
|
### Parameters
|
|
|
|
`WatsonxGenerator` needs IBM Cloud credentials to work. You can provide these in:
|
|
|
|
- The `WATSONX_API_KEY` environment variable (recommended)
|
|
- The `WATSONX_PROJECT_ID` environment variable (recommended)
|
|
- The `api_key` and `project_id` init parameters using Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")`
|
|
|
|
Set your preferred IBM watsonx.ai model in the `model` parameter when initializing the component. The default model is `ibm/granite-3-2b-instruct`.
|
|
|
|
`WatsonxGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the IBM watsonx.ai API directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the IBM watsonx.ai API, see [IBM watsonx.ai documentation](https://cloud.ibm.com/apidocs/watsonx-ai).
|
|
|
|
The component also supports system prompts that can be set at initialization or passed during runtime to provide context or instructions for the generation.
|
|
|
|
Finally, the component run method requires a single string prompt to generate text.
|
|
|
|
### Streaming
|
|
|
|
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
|
|
|
## Usage
|
|
|
|
Install the `watsonx-haystack` package to use the `WatsonxGenerator`:
|
|
|
|
```shell
|
|
pip install watsonx-haystack
|
|
```
|
|
|
|
### On its own
|
|
|
|
```python
|
|
from haystack_integrations.components.generators.watsonx.generator import WatsonxGenerator
|
|
from haystack.utils import Secret
|
|
|
|
generator = WatsonxGenerator(
|
|
api_key=Secret.from_env_var("WATSONX_API_KEY"),
|
|
project_id=Secret.from_env_var("WATSONX_PROJECT_ID")
|
|
)
|
|
|
|
print(generator.run("What's Natural Language Processing? Be brief."))
|
|
```
|
|
|
|
### In a pipeline
|
|
|
|
You can also use `WatsonxGenerator` with the IBM watsonx.ai models in your pipeline.
|
|
|
|
```python
|
|
from haystack import Pipeline
|
|
from haystack.components.builders import PromptBuilder
|
|
from haystack_integrations.components.generators.watsonx.generator import WatsonxGenerator
|
|
from haystack.utils import Secret
|
|
|
|
template = """
|
|
You are an assistant giving out valuable information to language learners.
|
|
Answer this question, be brief.
|
|
|
|
Question: {{ query }}?
|
|
"""
|
|
|
|
pipe = Pipeline()
|
|
pipe.add_component("prompt_builder", PromptBuilder(template))
|
|
pipe.add_component("llm", WatsonxGenerator(
|
|
api_key=Secret.from_env_var("WATSONX_API_KEY"),
|
|
project_id=Secret.from_env_var("WATSONX_PROJECT_ID")
|
|
))
|
|
pipe.connect("prompt_builder", "llm")
|
|
|
|
query = "What language is spoken in Germany?"
|
|
res = pipe.run(data={"prompt_builder": {"query": query}})
|
|
|
|
print(res)
|
|
```
|