mirror of
https://github.com/microsoft/autogen.git
synced 2025-07-19 15:01:52 +00:00
904b293aa4
* make retry_time configurable, add doc * in seconds * retry_wait_time * bump version to 0.1.4 * remove .json * rename * time
112 lines
4.4 KiB
Markdown
112 lines
4.4 KiB
Markdown
# Frequently Asked Questions
|
|
|
|
## Set your API endpoints
|
|
|
|
There are multiple ways to construct a list of configurations for LLM inference.
|
|
|
|
### Option 1: Load a list of endpoints from json
|
|
|
|
The [`config_list_from_json`](/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file.
|
|
|
|
For example,
|
|
|
|
```python
|
|
import autogen
|
|
config_list = autogen.config_list_from_json(
|
|
"OAI_CONFIG_LIST",
|
|
file_location=".",
|
|
filter_dict={
|
|
"model": {
|
|
"gpt-4",
|
|
"gpt-3.5-turbo",
|
|
}
|
|
}
|
|
)
|
|
```
|
|
|
|
It first looks for environment variable "OAI_CONFIG_LIST" which needs to be a valid json string. If that variable is not found, it then looks for a json file named "OAI_CONFIG_LIST" under the specified `file_location`. It then filters the configs by models (you can filter by other keys as well).
|
|
|
|
The `OAI_CONFIG_LIST` var or file content looks like the following:
|
|
```json
|
|
[
|
|
{
|
|
"model": "gpt-4",
|
|
"api_key": "<your OpenAI API key here>"
|
|
},
|
|
{
|
|
"model": "gpt-4",
|
|
"api_key": "<your Azure OpenAI API key here>",
|
|
"api_base": "<your Azure OpenAI API base here>",
|
|
"api_type": "azure",
|
|
"api_version": "2023-07-01-preview"
|
|
},
|
|
{
|
|
"model": "gpt-3.5-turbo",
|
|
"api_key": "<your Azure OpenAI API key here>",
|
|
"api_base": "<your Azure OpenAI API base here>",
|
|
"api_type": "azure",
|
|
"api_version": "2023-07-01-preview"
|
|
}
|
|
]
|
|
```
|
|
|
|
### Option 2: Construct a list of endpoints for OpenAI or Azure OpenAI
|
|
|
|
The [`config_list_from_models`](/docs/reference/oai/openai_utils#config_list_from_models) function tries to create a list of configurations using Azure OpenAI endpoints and OpenAI endpoints for the provided list of models. It assumes the api keys and api bases are stored in the corresponding environment variables or local txt files:
|
|
|
|
- OpenAI API key: os.environ["OPENAI_API_KEY"] or `openai_api_key_file="key_openai.txt"`.
|
|
- Azure OpenAI API key: os.environ["AZURE_OPENAI_API_KEY"] or `aoai_api_key_file="key_aoai.txt"`. Multiple keys can be stored, one per line.
|
|
- Azure OpenAI API base: os.environ["AZURE_OPENAI_API_BASE"] or `aoai_api_base_file="base_aoai.txt"`. Multiple bases can be stored, one per line.
|
|
|
|
It's OK to have only the OpenAI API key, or only the Azure OpenAI API key + base.
|
|
|
|
```python
|
|
import autogen
|
|
config_list = autogen.config_list_from_models(model_list=["gpt-4", "gpt-3.5-turbo", "gpt-3.5-turbo-16k"])
|
|
```
|
|
|
|
The config list looks like the following, if only OpenAI API key is available:
|
|
```python
|
|
config_list = [
|
|
{
|
|
'model': 'gpt-4',
|
|
'api_key': '<your OpenAI API key here>',
|
|
}, # OpenAI API endpoint for gpt-4
|
|
{
|
|
'model': 'gpt-3.5-turbo',
|
|
'api_key': '<your OpenAI API key here>',
|
|
}, # OpenAI API endpoint for gpt-3.5-turbo
|
|
{
|
|
'model': 'gpt-3.5-turbo-16k',
|
|
'api_key': '<your OpenAI API key here>',
|
|
}, # OpenAI API endpoint for gpt-3.5-turbo-16k
|
|
]
|
|
```
|
|
|
|
### Use the constructed configuration list in agents
|
|
|
|
Make sure the "config_list" is included in the `llm_config` in the constructor of the LLM-based agent. For example,
|
|
```python
|
|
assistant = autogen.AssistantAgent(
|
|
name="assistant",
|
|
llm_config={"config_list": config_list}
|
|
)
|
|
```
|
|
|
|
The `llm_config` is used in the [`create`](/docs/reference/oai/completion#create) function for LLM inference.
|
|
When `llm_config` is not provided, the agent will rely on other openai settings such as `openai.api_key` or the environment variable `OPENAI_API_KEY`, which can also work when you'd like to use a single endpoint.
|
|
You can also explicitly specify that by:
|
|
```python
|
|
assistant = autogen.AssistantAgent(name="assistant", llm_config={"api_key": ...})
|
|
```
|
|
|
|
## Handle Rate Limit Error and Timeout Error
|
|
|
|
You can set `retry_wait_time` and `max_retry_period` to handle rate limit error. And you can set `request_timeout` to handle timeout error. They can all be specified in `llm_config` for an agent, which will be used in the [`create`](/docs/reference/oai/completion#create) function for LLM inference.
|
|
|
|
- `retry_wait_time` (int): the time interval to wait (in seconds) before retrying a failed request.
|
|
- `max_retry_period` (int): the total timeout (in seconds) allowed for retrying failed requests.
|
|
- `request_timeout` (int): the timeout (in seconds) sent with a single request.
|
|
|
|
Please refer to the [documentation](/docs/Use-Cases/enhanced_inference#runtime-error) for more info.
|