
* LMM Code added * LLaVA notebook update * Test cases and Notebook modified for OpenAI v1 * Move LMM into contrib To resolve test issues and deploy issues In the future, we can install pillow by default, and then move back LMM agents into agentchat * LMM test setup update * try...except... clause for LMM tests * disable patch for llava agent test To resolve dependencies issue for build * Add LMM Blog * Change docstring for LMM agents * Docstring update patch * llava: insert reply at position 1 now So, it can still handle human_input_mode and max_consecutive_reply * Resolve comments Fixing: typos, blogs, yml, and add OpenAIWrapper * Signature typo fix for LMM agent: system_message * Update LMM "content" from latest OpenAI release Reference https://platform.openai.com/docs/guides/vision * update LMM test according to latest OpenAI release * Fully support GPT-4V now 1. Add a notebook for GPT-4V. LLava notebook also updated. 2. img_utils updated 3. GPT-4V formatter now return base64 image with mime type 4. Infer mime type directly from b64 image content (while loading without suffix) 5. Test cases modified according to all the related changes. * GPT-4V link updated in blog --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com>
5.7 KiB
Installation
Setup Virtual Environment
When not using a docker container, we recommend using a virtual environment to install AutoGen. This will ensure that the dependencies for AutoGen are isolated from the rest of your system.
Option 1: venv
You can create a virtual environment with venv
as below:
python3 -m venv pyautogen
source pyautogen/bin/activate
The following command will deactivate the current venv
environment:
deactivate
Option 2: conda
Another option is with Conda
, Conda works better at solving dependency conflicts than pip. You can install it by following this doc,
and then create a virtual environment as below:
conda create -n pyautogen python=3.10 # python 3.10 is recommended as it's stable and not too old
conda activate pyautogen
The following command will deactivate the current conda
environment:
conda deactivate
Now, you're ready to install AutoGen in the virtual environment you've just created.
Python
AutoGen requires Python version >= 3.8, < 3.12. It can be installed from pip:
pip install pyautogen
pyautogen<0.2
requires openai<1
. Starting from pyautogen v0.2, openai>=1
is required.
Migration guide to v0.2
openai v1 is a total rewrite of the library with many breaking changes. For example, the inference requires instantiating a client, instead of using a global class method.
Therefore, some changes are required for users of pyautogen<0.2
.
api_base
->base_url
,request_timeout
->timeout
inllm_config
andconfig_list
.max_retry_period
andretry_wait_time
are deprecated.max_retries
can be set for each client.- MathChat, TeachableAgent are unsupported until they are tested in future release.
autogen.Completion
andautogen.ChatCompletion
are deprecated. The essential functionalities are moved toautogen.OpenAIWrapper
:
from autogen import OpenAIWrapper
client = OpenAIWrapper(config_list=config_list)
response = client.create(messages=[{"role": "user", "content": "2+2="}])
print(client.extract_text_or_function_call(response))
- Inference parameter tuning and inference logging features are currently unavailable in
OpenAIWrapper
. Logging will be added in a future release. Inference parameter tuning can be done viaflaml.tune
. use_cache
is removed as a kwarg inOpenAIWrapper.create()
for being automatically decided byseed
: int | None.
Optional Dependencies
-
docker
For the best user experience and seamless code execution, we highly recommend using Docker with AutoGen. Docker is a containerization platform that simplifies the setup and execution of your code. Developing in a docker container, such as GitHub Codespace, also makes the development convenient.
When running AutoGen out of a docker container, to use docker for code execution, you also need to install the python package docker
:
pip install docker
-
blendsearch
pyautogen<0.2
offers a cost-effective hyperparameter optimization technique EcoOptiGen for tuning Large Language Models. Please install with the [blendsearch] option to use it.
pip install "pyautogen[blendsearch]<0.2"
Example notebooks:
-
retrievechat
pyautogen<0.2
supports retrieval-augmented generation tasks such as question answering and code generation with RAG agents. Please install with the [retrievechat] option to use it.
pip install "pyautogen[retrievechat]<0.2"
RetrieveChat can handle various types of documents. By default, it can process
plain text and PDF files, including formats such as 'txt', 'json', 'csv', 'tsv',
'md', 'html', 'htm', 'rtf', 'rst', 'jsonl', 'log', 'xml', 'yaml', 'yml' and 'pdf'.
If you install unstructured
(pip install "unstructured[all-docs]"
), additional document types such as 'docx',
'doc', 'odt', 'pptx', 'ppt', 'xlsx', 'eml', 'msg', 'epub' will also be supported.
You can find a list of all supported document types by using autogen.retrieve_utils.TEXT_FORMATS
.
Example notebooks:
Automated Code Generation and Question Answering with Retrieval Augmented Agents
Group Chat with Retrieval Augmented Generation (with 5 group member agents and 1 manager agent)
Automated Code Generation and Question Answering with Qdrant based Retrieval Augmented Agents
-
Large Multimodal Model (LMM) Agents
We offered Multimodal Conversable Agent and LLaVA Agent. Please install with the [lmm] option to use it.
pip install "pyautogen[lmm]"
Example notebooks:
-
mathchat
pyautogen<0.2
offers an experimental agent for math problem solving. Please install with the [mathchat] option to use it.
pip install "pyautogen[mathchat]<0.2"
Example notebooks: