mirror of
https://github.com/deepset-ai/haystack.git
synced 2026-01-01 09:37:36 +00:00
94 lines
4.1 KiB
Plaintext
94 lines
4.1 KiB
Plaintext
---
|
||
title: "STACKITChatGenerator"
|
||
id: stackitchatgenerator
|
||
slug: "/stackitchatgenerator"
|
||
description: "This component enables chat completions using the STACKIT API."
|
||
---
|
||
|
||
# STACKITChatGenerator
|
||
|
||
This component enables chat completions using the STACKIT API.
|
||
|
||
<div className="key-value-table">
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) |
|
||
| **Mandatory init variables** | `model`: The model used through the STACKIT API |
|
||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply (such as token count, finish reason, and so on) |
|
||
| **API reference** | [STACKIT](/reference/integrations-stackit) |
|
||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/stackit |
|
||
|
||
</div>
|
||
|
||
## Overview
|
||
|
||
`STACKITChatGenerator` enables text generation models served by STACKIT through their API.
|
||
|
||
### Parameters
|
||
|
||
To use the `STACKITChatGenerator`, ensure you have set a `STACKIT_API_KEY` as an environment variable. Alternatively, provide the API key as another environment variable or a token by setting
|
||
`api_key` and using Haystack’s [secret management](../../concepts/secret-management.mdx).
|
||
|
||
Set your preferred supported model with the `model` parameter when initializing the component. See the full list of all supported models on the [STACKIT website](https://docs.stackit.cloud/stackit/en/models-licenses-319914532.html).
|
||
|
||
Optionally, you can change the default `api_base_url`, which is `"https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1"`.
|
||
|
||
You can pass any text generation parameters valid for the STACKIT Chat Completion API directly to this component with the `generation_kwargs` parameter in the init or run methods.
|
||
|
||
The component needs a list of `ChatMessage` objects to run. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. Find out more about it [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx).
|
||
|
||
### Streaming
|
||
|
||
This ChatGenerator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly into the output. To do so, pass a function to the `streaming_callback` init parameter.
|
||
|
||
## Usage
|
||
|
||
Install the `stackit-haystack` package to use the `STACKITChatGenerator`:
|
||
|
||
```shell
|
||
pip install stackit-haystack
|
||
```
|
||
|
||
### On its own
|
||
|
||
```python
|
||
from haystack_integrations.components.generators.stackit import STACKITChatGenerator
|
||
from haystack.dataclasses import ChatMessage
|
||
|
||
generator = STACKITChatGenerator(model="neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8")
|
||
|
||
result = generator.run([ChatMessage.from_user("Tell me a joke.")])
|
||
print(result)
|
||
```
|
||
|
||
### In a pipeline
|
||
|
||
You can also use `STACKITChatGenerator` in your pipeline.
|
||
|
||
```python
|
||
from haystack import Pipeline
|
||
from haystack.components.builders import ChatPromptBuilder
|
||
from haystack.dataclasses import ChatMessage
|
||
|
||
from haystack_integrations.components.generators.stackit import STACKITChatGenerator
|
||
|
||
prompt_builder = ChatPromptBuilder()
|
||
llm = STACKITChatGenerator(model="neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8")
|
||
|
||
messages = [ChatMessage.from_user("Question: {{question}} \\n")]
|
||
|
||
pipeline = Pipeline()
|
||
pipeline.add_component("prompt_builder", prompt_builder)
|
||
pipeline.add_component("llm", llm)
|
||
|
||
pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||
|
||
result = pipeline.run({"prompt_builder": {"template_variables": {"question": "Tell me a joke."}, "template": messages}})
|
||
|
||
print(result)
|
||
```
|
||
|
||
For an example of streaming in a pipeline, refer to the examples in the STACKIT integration [repository](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/stackit/examples) and on its dedicated [integration page](https://haystack.deepset.ai/integrations/stackit).
|