yujunjun/autogen
1
0
Fork 0
mirror of https://github.com/microsoft/autogen.git synced 2025-09-26 00:24:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

cleanup

Browse Source
This commit is contained in:
Chi Wang 2023-09-16 16:34:16 +00:00
parent 99f2f82fa3
commit a4b9297719
35 changed files with 205 additions and 1750 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
notebook/autogen_agentchat_MathChat.ipynb
View File

@ -15,15 +15,15 @@
"source": [
"# Auto Generated Agent Chat: Using MathChat to Solve Math Problems\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"MathChat is an experimental convesational framework for math problem solving. In this notebook, we demonstrate how to use MathChat to solve math problems. MathChat uses the `AssistantAgent` and `MathUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `MathUserProxyAgent` implements a different auto reply mechanism corresponding to the MathChat prompts. You can find more details in the paper [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337) or the [blogpost](https://microsoft.github.io/FLAML/blog/2023/06/28/MathChat).\n",
"MathChat is an experimental convesational framework for math problem solving. In this notebook, we demonstrate how to use MathChat to solve math problems. MathChat uses the `AssistantAgent` and `MathUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `MathUserProxyAgent` implements a different auto reply mechanism corresponding to the MathChat prompts. You can find more details in the paper [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337) or the [blogpost](https://microsoft.github.io/autogen/blog/2023/06/28/MathChat).\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [mathchat] option.\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install the [mathchat] option.\n",
"```bash\n",
"pip install flaml[mathchat]\n",
"pip install \"pyautogen[mathchat]\"\n",
"```"
]
},
@ -33,7 +33,7 @@
"metadata": {},
"outputs": [],
"source": [
"# %pip install flaml[mathchat]~=2.0.0"
"# %pip install \"pyautogen[mathchat]~=0.1.0\""
]
},
{
@ -52,7 +52,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list = autogen.config_list_from_json(\n",
" \"OAI_CONFIG_LIST\",\n",
@ -126,7 +126,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml.autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n",
"from autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n",
"\n",
"autogen.ChatCompletion.start_logging()\n",
"\n",

1333
notebook/autogen_agentchat_RetrieveChat.ipynb
View File

File diff suppressed because it is too large Load Diff

16
notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -19,16 +19,16 @@
"source": [
"# Auto Generated Agent Chat: Task Solving with Code Generation, Execution & Debugging\n",
"\n",
"FLAML offers conversable LLM agents, which can be used to solve various tasks with human or automatic feedback, including tasks that require using tools via code.\n",
"AutoGen offers conversable LLM agents, which can be used to solve various tasks with human or automatic feedback, including tasks that require using tools via code.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to write code and execute the code. Here `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for the human user to execute the code written by `AssistantAgent`, or automatically execute the code. Depending on the setting of `human_input_mode` and `max_consecutive_auto_reply`, the `UserProxyAgent` either solicits feedback from the human user or returns auto-feedback based on the result of code execution (success or failure and corresponding outputs) to `AssistantAgent`. `AssistantAgent` will debug the code and suggest new code if the result contains error. The two agents keep communicating to each other until the task is done.\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[autogen]\n",
"pip install pyautogen\n",
"```"
]
},
@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"# %pip install flaml[autogen]~=2.0.2"
"# %pip install pyautogen~=0.1.0"
]
},
{
@ -64,7 +64,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list = autogen.config_list_from_json(\n",
" \"OAI_CONFIG_LIST\",\n",
@ -117,7 +117,7 @@
"source": [
"## Example Task: Check Stock Price Change\n",
"\n",
"In the example below, let's see how to use the agents in FLAML to write a python script and execute the script. This process involves constructing a `AssistantAgent` to serve as the assistant, along with a `UserProxyAgent` that acts as a proxy for the human user. In this example demonstrated below, when constructing the `UserProxyAgent`, we select the `human_input_mode` to \"NEVER\". This means that the `UserProxyAgent` will not solicit feedback from the human user. It stops replying when the limit defined by `max_consecutive_auto_reply` is reached, or when `is_termination_msg()` returns true for the received message."
"In the example below, let's see how to use the agents in AutoGen to write a python script and execute the script. This process involves constructing a `AssistantAgent` to serve as the assistant, along with a `UserProxyAgent` that acts as a proxy for the human user. In this example demonstrated below, when constructing the `UserProxyAgent`, we select the `human_input_mode` to \"NEVER\". This means that the `UserProxyAgent` will not solicit feedback from the human user. It stops replying when the limit defined by `max_consecutive_auto_reply` is reached, or when `is_termination_msg()` returns true for the received message."
]
},
{
@ -360,7 +360,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The example above involves code execution. In FLAML, code execution is triggered automatically by the `UserProxyAgent` when it detects an executable code block in a received message and no human user input is provided. This process occurs in a designated working directory, using a Docker container by default. Unless a specific directory is specified, FLAML defaults to the `flaml/autogen/extensions` directory. Users have the option to specify a different working directory by setting the `work_dir` argument when constructing a new instance of the `UserProxyAgent`.\n",
"The example above involves code execution. In AutoGen, code execution is triggered automatically by the `UserProxyAgent` when it detects an executable code block in a received message and no human user input is provided. This process occurs in a designated working directory, using a Docker container by default. Unless a specific directory is specified, AutoGen defaults to the `autogen/extensions` directory. Users have the option to specify a different working directory by setting the `work_dir` argument when constructing a new instance of the `UserProxyAgent`.\n",
"\n",
"The whole chat is auto generated."
]

12
notebook/autogen_agentchat_chess.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_chess.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_chess.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -15,16 +15,16 @@
"source": [
"# Auto Generated Agent Chat: Chess Game Playing While Chitchatting by GPT-4 Agents\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"This notebook is modified based on https://github.com/ekzhu/FLAML/blob/evaluation/evaluation/chess/play_chess.ipynb\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[autogen]\n",
"pip install pyautogen\n",
"```"
]
},
@ -35,7 +35,7 @@
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"# %pip install flaml[autogen]~=2.1.0\n",
"# %pip install pyautogen~=0.1.0\n",
"%pip install chess -U"
]
},
@ -65,7 +65,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list_gpt4 = autogen.config_list_from_json(\n",
" \"OAI_CONFIG_LIST\",\n",

14
notebook/autogen_agentchat_function_call.ipynb
View File

@ -6,7 +6,7 @@
"id": "ae1f50ec",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_function_call.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_function_call.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -17,15 +17,15 @@
"source": [
"# Auto Generated Agent Chat: Task Solving with Provided Tools as Functions\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to make function calls with the new feature of OpenAI models (in model version 0613). A specified prompt and function configs need to be passed to `AssistantAgent` to initialize the agent. The corresponding functions need to be passed to `UserProxyAgent`, which will be responsible for executing any function calls made by `AssistantAgent`. Besides this requirement of matching descriptions with functions, we recommend checking the system message in the `AssistantAgent` to make sure the instructions align with the function call descriptions.\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [mathchat] option since we will import functions from `MathUserProxyAgent`:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install the [mathchat] option since we will import functions from `MathUserProxyAgent`:\n",
"```bash\n",
"pip install flaml[mathchat]\n",
"pip install \"pyautogen[mathchat]\"\n",
"```"
]
},
@ -36,7 +36,7 @@
"metadata": {},
"outputs": [],
"source": [
"# %pip install flaml[mathchat]~=2.0.0"
"# %pip install \"pyautogen[mathchat]~=0.1.0\""
]
},
{
@ -66,7 +66,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list = autogen.config_list_from_models(model_list=[\"gpt-4\", \"gpt-3.5-turbo\", \"gpt-3.5-turbo-16k\"], exclude=\"aoai\")"
]
@ -372,7 +372,7 @@
],
"source": [
"import os\n",
"from flaml.autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n",
"from autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n",
"\n",
"# you need to provide a wolfram alpha appid to run this example\n",
"if not os.environ.get(\"WOLFRAM_ALPHA_APPID\"):\n",

12
notebook/autogen_agentchat_groupchat.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -15,16 +15,16 @@
"source": [
"# Auto Generated Agent Chat: Group Chat\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"This notebook is modified based on https://github.com/microsoft/FLAML/blob/4ea686af5c3e8ff24d9076a7a626c8b28ab5b1d7/notebook/autogen_multiagent_roleplay_chat.ipynb\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[autogen]\n",
"pip install pyautogen\n",
"```"
]
},
@ -35,7 +35,7 @@
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"# %pip install flaml[autogen]~=2.0.2"
"# %pip install pyautogen~=0.1.0"
]
},
{
@ -54,7 +54,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list_gpt4 = autogen.config_list_from_json(\n",
" \"OAI_CONFIG_LIST\",\n",

12
notebook/autogen_agentchat_groupchat_research.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -15,14 +15,14 @@
"source": [
"# Auto Generated Agent Chat: Performs Research with Multi-Agent Group Chat\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[autogen]\n",
"pip install pyautogen\n",
"```"
]
},
@ -33,7 +33,7 @@
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"# %pip install flaml[autogen]~=2.0.3"
"# %pip install pyautogen~=0.1.0"
]
},
{
@ -52,7 +52,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list_gpt4 = autogen.config_list_from_json(\n",
" \"OAI_CONFIG_LIST\",\n",

12
notebook/autogen_agentchat_groupchat_vis.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -15,14 +15,14 @@
"source": [
"# Auto Generated Agent Chat: Group Chat with Coder and Visualization Critic\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[autogen]\n",
"pip install pyautogen\n",
"```"
]
},
@ -33,7 +33,7 @@
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"# %pip install flaml[autogen]~=2.0.3"
"# %pip install pyautogen~=0.1.0"
]
},
{
@ -52,7 +52,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list_gpt4 = autogen.config_list_from_json(\n",
" \"OAI_CONFIG_LIST\",\n",

12
notebook/autogen_agentchat_human_feedback.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_human_feedback.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_human_feedback.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -19,16 +19,16 @@
"source": [
"# Auto Generated Agent Chat: Task Solving with Code Generation, Execution, Debugging & Human Feedback\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to solve a challenging math problem with human feedback. Here `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. By setting `human_input_mode` properly, the `UserProxyAgent` can also prompt the user for feedback to `AssistantAgent`. For example, when `human_input_mode` is set to \"ALWAYS\", the `UserProxyAgent` will always prompt the user for feedback. When user feedback is provided, the `UserProxyAgent` will directly pass the feedback to `AssistantAgent`. When no user feedback is provided, the `UserProxyAgent` will execute the code written by `AssistantAgent` and return the execution results (success or failure and corresponding outputs) to `AssistantAgent`.\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[autogen]\n",
"pip install pyautogen\n",
"```"
]
},
@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"# %pip install flaml[autogen]~=2.0.2"
"# %pip install pyautogen~=0.1.0"
]
},
{
@ -64,7 +64,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list = autogen.config_list_from_json(\"OAI_CONFIG_LIST\")"
]

12
notebook/autogen_agentchat_planning.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_planning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_planning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -19,16 +19,16 @@
"source": [
"# Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"In this notebook, we demonstrate how to use multiple agents to work together and accomplish a task which requires finding info from the web and coding. `AssistantAgent` is an LLM-based agent that can write and debug Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. We further create a planning agent for the assistant agent to consult. The planning agent is a variation of the LLM-based `AssistantAgent` with a different system message.\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option and docker:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install pyautogen and docker:\n",
"```bash\n",
"pip install flaml[autogen] docker\n",
"pip install pyautogen docker\n",
"```"
]
},
@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"# %pip install flaml[autogen]~=2.0.2 docker"
"# %pip install pyautogen~=0.1.0 docker"
]
},
{
@ -70,7 +70,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list = autogen.config_list_from_json(\n",
" \"OAI_CONFIG_LIST\",\n",

14
notebook/autogen_agentchat_stream.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_stream.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_stream.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -19,16 +19,16 @@
"source": [
"# Interactive LLM Agent Dealing with Data Stream\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"In this notebook, we demonstrate how to use customized agents to continuously acquires news from the web and ask for investment suggestions.\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[autogen]\n",
"pip install pyautogen\n",
"```"
]
},
@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"# %pip install flaml[autogen]~=2.1.0"
"# %pip install pyautogen~=0.1.0"
]
},
{
@ -64,7 +64,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list = autogen.config_list_from_json(\"OAI_CONFIG_LIST\")"
]
@ -222,7 +222,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"# create an AssistantAgent instance named \"assistant\"\n",
"assistant = autogen.AssistantAgent(\n",

12
notebook/autogen_agentchat_web_info.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_web_info.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_agentchat_web_info.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -19,7 +19,7 @@
"source": [
"# Auto Generated Agent Chat: Solving Tasks Requiring Web Info\n",
"\n",
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to perform tasks which require acquiring info from the web:\n",
@ -30,9 +30,9 @@
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option and docker:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install pyautogen and docker:\n",
"```bash\n",
"pip install flaml[autogen]\n",
"pip install pyautogen docker\n",
"```"
]
},
@ -49,7 +49,7 @@
},
"outputs": [],
"source": [
"# %pip install flaml[autogen]~=2.0.2 docker"
"# %pip install pyautogen~=0.1.0 docker"
]
},
{
@ -68,7 +68,7 @@
"metadata": {},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"config_list = autogen.config_list_from_json(\n",
" \"OAI_CONFIG_LIST\",\n",

278
notebook/autogen_chatgpt_gpt4.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_chatgpt_gpt4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_chatgpt_gpt4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -21,20 +21,20 @@
"\n",
"Licensed under the MIT License.\n",
"\n",
"# Use FLAML to Tune ChatGPT\n",
"# Use AutoGen to Tune ChatGPT\n",
"\n",
"`flaml.autogen` offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
"AutoGen offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
"Please find documentation about this feature [here](/docs/Use-Cases/AutoGen#enhanced-inference).\n",
"\n",
"In this notebook, we tune OpenAI ChatGPT (both GPT-3.5 and GPT-4) models for math problem solving. We use [the MATH benchmark](https://crfm.stanford.edu/helm/latest/?group=math_chain_of_thought) for measuring mathematical problem solving on competition math problems with chain-of-thoughts style reasoning.\n",
"\n",
"Related link: [Blogpost](https://microsoft.github.io/FLAML/blog/2023/04/21/LLM-tuning-math) based on this experiment.\n",
"Related link: [Blogpost](https://microsoft.github.io/autogen/blog/2023/04/21/LLM-tuning-math) based on this experiment.\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [openai,blendsearch] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install with the [blendsearch] option:\n",
"```bash\n",
"pip install flaml[openai,blendsearch]\n",
"pip install \"pyautogen[blendsearch]\"\n",
"```"
]
},
@ -51,7 +51,7 @@
},
"outputs": [],
"source": [
"# %pip install flaml[openai,blendsearch] datasets"
"# %pip install \"pyautogen[blendsearch]\" datasets"
]
},
{
@ -59,7 +59,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"FLAML has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `autogen.ChatCompletion.tune` and to make a request with the tuned config: `autogen.ChatCompletion.create`. First, we import autogen from flaml:"
"AutoGen has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `autogen.ChatCompletion.tune` and to make a request with the tuned config: `autogen.ChatCompletion.create`. First, we import autogen:"
]
},
{
@ -78,7 +78,7 @@
},
"outputs": [],
"source": [
"from flaml import autogen"
"import autogen"
]
},
{
@ -293,7 +293,7 @@
},
"outputs": [],
"source": [
"from flaml.autogen.math_utils import eval_math_responses"
"from autogen.math_utils import eval_math_responses"
]
},
{
@ -376,7 +376,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:56.115383Z",
@ -385,44 +385,7 @@
"shell.execute_reply": "2023-02-13T23:41:55.044973Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32m[I 2023-08-01 22:38:01,549]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[flaml.tune.tune: 08-01 22:38:01] {805} INFO - trial 1 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.36280922847807595}, 'max_tokens': 347, 'n': 10, 'prompt': 0, 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:41:21] {197} INFO - result: {'expected_success': 0.89828529287, 'success': 0.9, 'success_vote': 0.8, 'voted_answer': 'We can write the given sequence as $3^4 \\\\cdot 225, 3^4 \\\\cdot 75, 3^4 \\\\cdot 25, \\\\ldots$. The exponents of 3 are $4, 4, 4, \\\\ldots$, so the sequence consists of the numbers $3^4 \\\\cdot 225, 3^4 \\\\cdot 75, 3^4 \\\\cdot 25, \\\\ldots, 3^4 \\\\cdot 3^0 \\\\cdot 225$. There are $\\\\boxed{4}$ integers in this sequence.', 'votes': 9.1, 'total_cost': 0.07211000000000002, 'cost': 0.07211000000000002, 'inference_cost': 0.0032122500000000003, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.36280922847807595}, 'max_tokens': 347, 'n': 10, 'prompt': 0, 'allow_format_str_template': True}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'top_p': 0.36280922847807595}, 'config/max_tokens': 347, 'config/n': 10, 'config/prompt': 0, 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 200.14745712280273}\n",
"[flaml.tune.tune: 08-01 22:41:21] {805} INFO - trial 2 config: {'temperature_or_top_p': {'temperature': 1.2672964698525508}, 'max_tokens': 470, 'n': 50, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:41:53] {197} INFO - result: {'success_vote': 0, 'total_cost': 0.10703200000000002, 'cost': 0.034922, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.2672964698525508}, 'max_tokens': 470, 'n': 50, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.2672964698525508}, 'config/max_tokens': 470, 'config/n': 50, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 31.767715454101562}\n",
"[flaml.tune.tune: 08-01 22:41:53] {805} INFO - trial 3 config: {'temperature_or_top_p': {'temperature': 1.5210614243979175}, 'max_tokens': 82, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:42:49] {197} INFO - result: {'expected_success': 0.15989600488062986, 'success': 0.2, 'success_vote': 0.2, 'voted_answer': 'Note that to get from 6075 to 2025 or from 2025 to 675, we must divide by 3. Thus the sequence in question, which ends in an ellipsis or a couple of periods dots indicating more members come next, also begins with the labeling \"700, 300.\" Recall from arithmetic pattern insight to sports like basketball and from looking in our answer choices section', 'votes': 0.7, 'total_cost': 0.13852200000000003, 'cost': 0.031490000000000004, 'inference_cost': 0.0015442499999999998, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.5210614243979175}, 'max_tokens': 82, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.5210614243979175}, 'config/max_tokens': 82, 'config/n': 9, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 55.53780817985535}\n",
"[flaml.tune.tune: 08-01 22:42:49] {805} INFO - trial 4 config: {'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'max_tokens': 231, 'n': 81, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:43:07] {197} INFO - result: {'success_vote': 0, 'total_cost': 0.18828000000000003, 'cost': 0.04975800000000001, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'max_tokens': 231, 'n': 81, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'top_p': 0.003948266327914451}, 'config/max_tokens': 231, 'config/n': 81, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 18.070116996765137}\n",
"[flaml.tune.tune: 08-01 22:43:07] {805} INFO - trial 5 config: {'temperature_or_top_p': {'top_p': 0.29187606817063316}, 'max_tokens': 781, 'n': 71, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:43:07] {197} INFO - result: {'inference_cost': inf, 'success_vote': -inf, 'cost': 0, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.29187606817063316}, 'max_tokens': 781, 'n': 71, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'top_p': 0.29187606817063316}, 'config/max_tokens': 781, 'config/n': 71, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 0.0005497932434082031}\n",
"[flaml.tune.tune: 08-01 22:43:07] {805} INFO - trial 6 config: {'temperature_or_top_p': {'temperature': 0.7466815201029384}, 'max_tokens': 375, 'n': 44, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:46:28] {197} INFO - result: {'expected_success': 0.9818164607828072, 'success': 1.0, 'success_vote': 0.95, 'voted_answer': 'To find the number of integers in the sequence, we need to find when each term becomes less than 1. \\n\\nStarting with 6075, we divide by 3 to get $\\\\frac{6075}{3} = 2025$. Since 2025 is an integer, it is included in the sequence.\\n\\nDividing 2025 by 3, we get $\\\\frac{2025}{3} = 675$. Again, 675 is an integer, so it is included in the sequence.\\n\\nIf we divide 675 by 3, we get $\\\\frac{675}{3} = 225$. 225 is an integer, so it is included in the sequence.\\n\\nDividing 225 by 3, we get $\\\\frac{225}{3} = 75$. 75 is an integer, so it is included in the sequence.\\n\\nDividing 75 by 3, we get $\\\\frac{75}{3} = 25$. 25 is an integer, so it is included in the sequence.\\n\\nIf we divide 25 by 3, we get $\\\\frac{25}{3} \\\\approx 8.3333$, which is not an integer. Thus, 25 is the last integer in the sequence.\\n\\nThere are a total of $\\\\boxed{6}$ integers in the sequence.', 'votes': 34.85, 'total_cost': 0.463802, 'cost': 0.27552199999999993, 'inference_cost': 0.01310685, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 0.7466815201029384}, 'max_tokens': 375, 'n': 44, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 0.7466815201029384}, 'config/max_tokens': 375, 'config/n': 44, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 201.2768588066101}\n",
"[flaml.tune.tune: 08-01 22:46:28] {805} INFO - trial 7 config: {'temperature_or_top_p': {'top_p': 0.5131382425543909}, 'max_tokens': 350, 'n': 60, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:47:09] {197} INFO - result: {'success_vote': 0, 'total_cost': 0.52441, 'cost': 0.060607999999999995, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.5131382425543909}, 'max_tokens': 350, 'n': 60, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'top_p': 0.5131382425543909}, 'config/max_tokens': 350, 'config/n': 60, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 41.3958899974823}\n",
"[flaml.tune.tune: 08-01 22:47:09] {805} INFO - trial 8 config: {'temperature_or_top_p': {'temperature': 1.8172977616173365}, 'max_tokens': 129, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:48:34] {197} INFO - result: {'expected_success': 0.06535605838853817, 'success': 0.1, 'success_vote': 0.1, 'voted_answer': 'To find out how many integers are in this sequence, we must determine the number of times 3 is a factor being successively divisible cases.\\n\\n\\nFor modern thought:\\nThe ultimate disaster approach ,\\nwill hit eighty year compound,\\ncos thirty pieces, successful trip necessitate; pounds prove evenly\\nHot before four boxes accumulate closely superior statistics prove Yet pale-eyed visionary spite.\\n\\n\\n\\n\\n\\nAnalyzer-based cipher elements yielded intervals This outcome integers.A reason.Brief Inspection Of available objects imply Par near Often Reason via options \\n\\nThe Ratio sum leaves ten; Five.\\n\\nReal Analy access tells not answer right I vary combinations&find divisions Prompt are strongSo inspection Replace Reverse', 'votes': 0.35, 'total_cost': 0.5708920000000002, 'cost': 0.046482, 'inference_cost': 0.00229385, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.8172977616173365}, 'max_tokens': 129, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.8172977616173365}, 'config/max_tokens': 129, 'config/n': 9, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 84.15163469314575}\n",
"[flaml.tune.tune: 08-01 22:48:34] {805} INFO - trial 9 config: {'temperature_or_top_p': {'temperature': 1.6573626526153533}, 'max_tokens': 57, 'n': 63, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:49:36] {197} INFO - result: {'expected_success': 0.12519255101013854, 'success': 0.15, 'success_vote': 0.15, 'voted_answer': 'Let the original term of the sequence be $x$. There are $796-43= \\\\boxed{753}$ sequences/pro-edits until term the when you divide the sequence becomes less vo/volume of OR counting totals that =prime-number?(-)+lifeisticment real!', 'votes': 1.1, 'total_cost': 0.71616, 'cost': 0.145268, 'inference_cost': 0.007233149999999999, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.6573626526153533}, 'max_tokens': 57, 'n': 63, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.6573626526153533}, 'config/max_tokens': 57, 'config/n': 63, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 62.10028266906738}\n",
"[flaml.tune.tune: 08-01 22:49:36] {805} INFO - trial 10 config: {'temperature_or_top_p': {'top_p': 0.1989475396788123}, 'max_tokens': 650, 'n': 35, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:51:50] {197} INFO - result: {'expected_success': 0.8499999999999934, 'success': 0.85, 'success_vote': 0.85, 'voted_answer': 'We can write the given sequence as $3^4 \\\\cdot 5^2, 3^4 \\\\cdot 5^1, 3^4 \\\\cdot 5^0, \\\\ldots$. We want to find the number of integers in this sequence. \\n\\nNotice that the exponent of 3 stays constant at 4, while the exponent of 5 decreases by 1 each time. We want to find the largest integer $n$ such that $3^4 \\\\cdot 5^n$ is an integer. \\n\\nSince $3^4$ is an integer, we only need to consider the exponent of 5. We want $5^n$ to be an integer, so $n$ must be nonnegative. However, we also want $5^n$ to be a factor of $3^4$, so $n$ must be less than or equal to 4. \\n\\nTherefore, the possible values of $n$ are 0, 1, 2, 3, and 4. There are $\\\\boxed{5}$ integers in the sequence.', 'votes': 33.8, 'total_cost': 0.9523240000000001, 'cost': 0.23616399999999999, 'inference_cost': 0.010300450000000001, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.1989475396788123}, 'max_tokens': 650, 'n': 35, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'top_p': 0.1989475396788123}, 'config/max_tokens': 650, 'config/n': 35, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 134.67861104011536}\n",
"[flaml.tune.tune: 08-01 22:51:50] {805} INFO - trial 11 config: {'temperature_or_top_p': {'temperature': 1.7678729591223725}, 'max_tokens': 132, 'n': 17, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n",
"[flaml.tune.tune: 08-01 22:52:44] {197} INFO - result: {'success_vote': 0, 'total_cost': 1.000234, 'cost': 0.04791, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.7678729591223725}, 'max_tokens': 132, 'n': 17, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.7678729591223725}, 'config/max_tokens': 132, 'config/n': 17, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 53.21780872344971}\n",
"[flaml.tune.tune: 08-01 22:52:44] {828} WARNING - fail to sample a trial for 100 times in a row, stopping.\n"
]
}
],
"outputs": [],
"source": [
"import logging\n",
"\n",
@ -454,7 +417,7 @@
"source": [
"### Output tuning results\n",
"\n",
"After the tuning, we can print out the config and the result found by FLAML:"
"After the tuning, we can print out the config and the result found by AutoGen, which uses flaml for tuning."
]
},
{
@ -1723,7 +1686,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:41:56.042764Z",
@ -1732,216 +1695,7 @@
"shell.execute_reply": "2023-02-13T23:53:05.596603Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[flaml.autogen.oai.completion: 08-01 22:55:55] {916} INFO - evaluating data instance 0\n",
"[flaml.autogen.oai.completion: 08-01 22:56:09] {916} INFO - evaluating data instance 1\n",
"[flaml.autogen.oai.completion: 08-01 22:56:20] {916} INFO - evaluating data instance 2\n",
"[flaml.autogen.oai.completion: 08-01 22:56:28] {916} INFO - evaluating data instance 3\n",
"[flaml.autogen.oai.completion: 08-01 22:56:34] {916} INFO - evaluating data instance 4\n",
"[flaml.autogen.oai.completion: 08-01 22:56:44] {916} INFO - evaluating data instance 5\n",
"[flaml.autogen.oai.completion: 08-01 22:56:57] {916} INFO - evaluating data instance 6\n",
"[flaml.autogen.oai.completion: 08-01 22:57:12] {916} INFO - evaluating data instance 7\n",
"[flaml.autogen.oai.completion: 08-01 22:57:20] {916} INFO - evaluating data instance 8\n",
"[flaml.autogen.oai.completion: 08-01 22:57:24] {916} INFO - evaluating data instance 9\n",
"[flaml.autogen.oai.completion: 08-01 22:57:34] {916} INFO - evaluating data instance 10\n",
"[flaml.autogen.oai.completion: 08-01 22:57:43] {916} INFO - evaluating data instance 11\n",
"[flaml.autogen.oai.completion: 08-01 22:57:52] {916} INFO - evaluating data instance 12\n",
"[flaml.autogen.oai.completion: 08-01 22:58:00] {916} INFO - evaluating data instance 13\n",
"[flaml.autogen.oai.completion: 08-01 22:58:08] {916} INFO - evaluating data instance 14\n",
"[flaml.autogen.oai.completion: 08-01 22:58:14] {916} INFO - evaluating data instance 15\n",
"[flaml.autogen.oai.completion: 08-01 22:58:22] {916} INFO - evaluating data instance 16\n",
"[flaml.autogen.oai.completion: 08-01 22:58:29] {916} INFO - evaluating data instance 17\n",
"[flaml.autogen.oai.completion: 08-01 22:58:40] {916} INFO - evaluating data instance 18\n",
"[flaml.autogen.oai.completion: 08-01 22:58:48] {916} INFO - evaluating data instance 19\n",
"[flaml.autogen.oai.completion: 08-01 22:58:57] {916} INFO - evaluating data instance 20\n",
"[flaml.autogen.oai.completion: 08-01 22:59:15] {916} INFO - evaluating data instance 21\n",
"[flaml.autogen.oai.completion: 08-01 22:59:29] {916} INFO - evaluating data instance 22\n",
"[flaml.autogen.oai.completion: 08-01 22:59:41] {916} INFO - evaluating data instance 23\n",
"[flaml.autogen.oai.completion: 08-01 22:59:54] {916} INFO - evaluating data instance 24\n",
"[flaml.autogen.oai.completion: 08-01 23:00:07] {916} INFO - evaluating data instance 25\n",
"[flaml.autogen.oai.completion: 08-01 23:00:24] {916} INFO - evaluating data instance 26\n",
"[flaml.autogen.oai.completion: 08-01 23:00:39] {916} INFO - evaluating data instance 27\n",
"[flaml.autogen.oai.completion: 08-01 23:00:55] {916} INFO - evaluating data instance 28\n",
"[flaml.autogen.oai.completion: 08-01 23:01:11] {916} INFO - evaluating data instance 29\n",
"[flaml.autogen.oai.completion: 08-01 23:01:26] {916} INFO - evaluating data instance 30\n",
"[flaml.autogen.oai.completion: 08-01 23:01:35] {916} INFO - evaluating data instance 31\n",
"[flaml.autogen.oai.completion: 08-01 23:01:46] {916} INFO - evaluating data instance 32\n",
"[flaml.autogen.oai.completion: 08-01 23:01:54] {916} INFO - evaluating data instance 33\n",
"[flaml.autogen.oai.completion: 08-01 23:02:03] {916} INFO - evaluating data instance 34\n",
"[flaml.autogen.oai.completion: 08-01 23:02:11] {916} INFO - evaluating data instance 35\n",
"[flaml.autogen.oai.completion: 08-01 23:02:27] {916} INFO - evaluating data instance 36\n",
"[flaml.autogen.oai.completion: 08-01 23:02:40] {916} INFO - evaluating data instance 37\n",
"[flaml.autogen.oai.completion: 08-01 23:02:46] {916} INFO - evaluating data instance 38\n",
"[flaml.autogen.oai.completion: 08-01 23:02:56] {916} INFO - evaluating data instance 39\n",
"[flaml.autogen.oai.completion: 08-01 23:03:06] {916} INFO - evaluating data instance 40\n",
"[flaml.autogen.oai.completion: 08-01 23:03:15] {916} INFO - evaluating data instance 41\n",
"[flaml.autogen.oai.completion: 08-01 23:03:23] {916} INFO - evaluating data instance 42\n",
"[flaml.autogen.oai.completion: 08-01 23:03:30] {916} INFO - evaluating data instance 43\n",
"[flaml.autogen.oai.completion: 08-01 23:03:38] {916} INFO - evaluating data instance 44\n",
"[flaml.autogen.oai.completion: 08-01 23:03:49] {916} INFO - evaluating data instance 45\n",
"[flaml.autogen.oai.completion: 08-01 23:03:55] {916} INFO - evaluating data instance 46\n",
"[flaml.autogen.oai.completion: 08-01 23:04:02] {916} INFO - evaluating data instance 47\n",
"[flaml.autogen.oai.completion: 08-01 23:04:14] {916} INFO - evaluating data instance 48\n",
"[flaml.autogen.oai.completion: 08-01 23:04:30] {916} INFO - evaluating data instance 49\n",
"[flaml.autogen.oai.completion: 08-01 23:04:42] {916} INFO - evaluating data instance 50\n",
"[flaml.autogen.oai.completion: 08-01 23:04:53] {916} INFO - evaluating data instance 51\n",
"[flaml.autogen.oai.completion: 08-01 23:05:05] {916} INFO - evaluating data instance 52\n",
"[flaml.autogen.oai.completion: 08-01 23:05:10] {916} INFO - evaluating data instance 53\n",
"[flaml.autogen.oai.completion: 08-01 23:05:22] {916} INFO - evaluating data instance 54\n",
"[flaml.autogen.oai.completion: 08-01 23:05:31] {916} INFO - evaluating data instance 55\n",
"[flaml.autogen.oai.completion: 08-01 23:05:43] {916} INFO - evaluating data instance 56\n",
"[flaml.autogen.oai.completion: 08-01 23:05:49] {916} INFO - evaluating data instance 57\n",
"[flaml.autogen.oai.completion: 08-01 23:05:59] {916} INFO - evaluating data instance 58\n",
"[flaml.autogen.oai.completion: 08-01 23:06:12] {916} INFO - evaluating data instance 59\n",
"[flaml.autogen.oai.completion: 08-01 23:06:20] {916} INFO - evaluating data instance 60\n",
"[flaml.autogen.oai.completion: 08-01 23:06:32] {916} INFO - evaluating data instance 61\n",
"[flaml.autogen.oai.completion: 08-01 23:06:42] {916} INFO - evaluating data instance 62\n",
"[flaml.autogen.oai.completion: 08-01 23:06:54] {916} INFO - evaluating data instance 63\n",
"[flaml.autogen.oai.completion: 08-01 23:07:08] {916} INFO - evaluating data instance 64\n",
"[flaml.autogen.oai.completion: 08-01 23:07:22] {916} INFO - evaluating data instance 65\n",
"[flaml.autogen.oai.completion: 08-01 23:07:34] {916} INFO - evaluating data instance 66\n",
"[flaml.autogen.oai.completion: 08-01 23:07:43] {916} INFO - evaluating data instance 67\n",
"[flaml.autogen.oai.completion: 08-01 23:07:49] {916} INFO - evaluating data instance 68\n",
"[flaml.autogen.oai.completion: 08-01 23:08:00] {916} INFO - evaluating data instance 69\n",
"[flaml.autogen.oai.completion: 08-01 23:08:12] {916} INFO - evaluating data instance 70\n",
"[flaml.autogen.oai.completion: 08-01 23:08:27] {916} INFO - evaluating data instance 71\n",
"[flaml.autogen.oai.completion: 08-01 23:08:36] {916} INFO - evaluating data instance 72\n",
"[flaml.autogen.oai.completion: 08-01 23:08:50] {916} INFO - evaluating data instance 73\n",
"[flaml.autogen.oai.completion: 08-01 23:08:58] {916} INFO - evaluating data instance 74\n",
"[flaml.autogen.oai.completion: 08-01 23:09:10] {916} INFO - evaluating data instance 75\n",
"[flaml.autogen.oai.completion: 08-01 23:09:19] {916} INFO - evaluating data instance 76\n",
"[flaml.autogen.oai.completion: 08-01 23:09:30] {916} INFO - evaluating data instance 77\n",
"[flaml.autogen.oai.completion: 08-01 23:09:38] {916} INFO - evaluating data instance 78\n",
"[flaml.autogen.oai.completion: 08-01 23:09:48] {916} INFO - evaluating data instance 79\n",
"[flaml.autogen.oai.completion: 08-01 23:09:58] {916} INFO - evaluating data instance 80\n",
"[flaml.autogen.oai.completion: 08-01 23:10:08] {916} INFO - evaluating data instance 81\n",
"[flaml.autogen.oai.completion: 08-01 23:10:19] {916} INFO - evaluating data instance 82\n",
"[flaml.autogen.oai.completion: 08-01 23:10:32] {916} INFO - evaluating data instance 83\n",
"[flaml.autogen.oai.completion: 08-01 23:10:37] {916} INFO - evaluating data instance 84\n",
"[flaml.autogen.oai.completion: 08-01 23:10:52] {916} INFO - evaluating data instance 85\n",
"[flaml.autogen.oai.completion: 08-01 23:11:07] {916} INFO - evaluating data instance 86\n",
"[flaml.autogen.oai.completion: 08-01 23:11:22] {916} INFO - evaluating data instance 87\n",
"[flaml.autogen.oai.completion: 08-01 23:11:33] {916} INFO - evaluating data instance 88\n",
"[flaml.autogen.oai.completion: 08-01 23:11:48] {916} INFO - evaluating data instance 89\n",
"[flaml.autogen.oai.completion: 08-01 23:11:55] {916} INFO - evaluating data instance 90\n",
"[flaml.autogen.oai.completion: 08-01 23:12:04] {916} INFO - evaluating data instance 91\n",
"[flaml.autogen.oai.completion: 08-01 23:12:15] {916} INFO - evaluating data instance 92\n",
"[flaml.autogen.oai.completion: 08-01 23:12:27] {916} INFO - evaluating data instance 93\n",
"[flaml.autogen.oai.completion: 08-01 23:12:39] {916} INFO - evaluating data instance 94\n",
"[flaml.autogen.oai.completion: 08-01 23:12:55] {916} INFO - evaluating data instance 95\n",
"[flaml.autogen.oai.completion: 08-01 23:13:05] {916} INFO - evaluating data instance 96\n",
"[flaml.autogen.oai.completion: 08-01 23:13:17] {916} INFO - evaluating data instance 97\n",
"[flaml.autogen.oai.completion: 08-01 23:13:30] {916} INFO - evaluating data instance 98\n",
"[flaml.autogen.oai.completion: 08-01 23:13:43] {916} INFO - evaluating data instance 99\n",
"[flaml.autogen.oai.completion: 08-01 23:13:51] {916} INFO - evaluating data instance 100\n",
"[flaml.autogen.oai.completion: 08-01 23:14:04] {916} INFO - evaluating data instance 101\n",
"[flaml.autogen.oai.completion: 08-01 23:14:09] {916} INFO - evaluating data instance 102\n",
"[flaml.autogen.oai.completion: 08-01 23:14:20] {916} INFO - evaluating data instance 103\n",
"[flaml.autogen.oai.completion: 08-01 23:14:32] {916} INFO - evaluating data instance 104\n",
"[flaml.autogen.oai.completion: 08-01 23:14:46] {916} INFO - evaluating data instance 105\n",
"[flaml.autogen.oai.completion: 08-01 23:14:59] {916} INFO - evaluating data instance 106\n",
"[flaml.autogen.oai.completion: 08-01 23:15:13] {916} INFO - evaluating data instance 107\n",
"[flaml.autogen.oai.completion: 08-01 23:15:23] {916} INFO - evaluating data instance 108\n",
"[flaml.autogen.oai.completion: 08-01 23:15:34] {916} INFO - evaluating data instance 109\n",
"[flaml.autogen.oai.completion: 08-01 23:15:46] {916} INFO - evaluating data instance 110\n",
"[flaml.autogen.oai.completion: 08-01 23:15:56] {916} INFO - evaluating data instance 111\n",
"[flaml.autogen.oai.completion: 08-01 23:16:10] {916} INFO - evaluating data instance 112\n",
"[flaml.autogen.oai.completion: 08-01 23:16:15] {916} INFO - evaluating data instance 113\n",
"[flaml.autogen.oai.completion: 08-01 23:16:27] {916} INFO - evaluating data instance 114\n",
"[flaml.autogen.oai.completion: 08-01 23:16:35] {916} INFO - evaluating data instance 115\n",
"[flaml.autogen.oai.completion: 08-01 23:16:48] {916} INFO - evaluating data instance 116\n",
"[flaml.autogen.oai.completion: 08-01 23:17:02] {916} INFO - evaluating data instance 117\n",
"[flaml.autogen.oai.completion: 08-01 23:17:14] {916} INFO - evaluating data instance 118\n",
"[flaml.autogen.oai.completion: 08-01 23:17:18] {916} INFO - evaluating data instance 119\n",
"[flaml.autogen.oai.completion: 08-01 23:17:31] {916} INFO - evaluating data instance 120\n",
"[flaml.autogen.oai.completion: 08-01 23:17:37] {916} INFO - evaluating data instance 121\n",
"[flaml.autogen.oai.completion: 08-01 23:17:46] {916} INFO - evaluating data instance 122\n",
"[flaml.autogen.oai.completion: 08-01 23:17:53] {916} INFO - evaluating data instance 123\n",
"[flaml.autogen.oai.completion: 08-01 23:18:00] {916} INFO - evaluating data instance 124\n",
"[flaml.autogen.oai.completion: 08-01 23:18:11] {916} INFO - evaluating data instance 125\n",
"[flaml.autogen.oai.completion: 08-01 23:18:17] {916} INFO - evaluating data instance 126\n",
"[flaml.autogen.oai.completion: 08-01 23:18:27] {916} INFO - evaluating data instance 127\n",
"[flaml.autogen.oai.completion: 08-01 23:18:30] {916} INFO - evaluating data instance 128\n",
"[flaml.autogen.oai.completion: 08-01 23:18:45] {916} INFO - evaluating data instance 129\n",
"[flaml.autogen.oai.completion: 08-01 23:18:53] {916} INFO - evaluating data instance 130\n",
"[flaml.autogen.oai.completion: 08-01 23:19:03] {916} INFO - evaluating data instance 131\n",
"[flaml.autogen.oai.completion: 08-01 23:19:07] {916} INFO - evaluating data instance 132\n",
"[flaml.autogen.oai.completion: 08-01 23:19:15] {916} INFO - evaluating data instance 133\n",
"[flaml.autogen.oai.completion: 08-01 23:19:29] {916} INFO - evaluating data instance 134\n",
"[flaml.autogen.oai.completion: 08-01 23:19:44] {916} INFO - evaluating data instance 135\n",
"[flaml.autogen.oai.completion: 08-01 23:19:55] {916} INFO - evaluating data instance 136\n",
"[flaml.autogen.oai.completion: 08-01 23:20:02] {916} INFO - evaluating data instance 137\n",
"[flaml.autogen.oai.completion: 08-01 23:20:15] {916} INFO - evaluating data instance 138\n",
"[flaml.autogen.oai.completion: 08-01 23:20:24] {916} INFO - evaluating data instance 139\n",
"[flaml.autogen.oai.completion: 08-01 23:20:34] {916} INFO - evaluating data instance 140\n",
"[flaml.autogen.oai.completion: 08-01 23:20:40] {916} INFO - evaluating data instance 141\n",
"[flaml.autogen.oai.completion: 08-01 23:20:49] {916} INFO - evaluating data instance 142\n",
"[flaml.autogen.oai.completion: 08-01 23:20:55] {916} INFO - evaluating data instance 143\n",
"[flaml.autogen.oai.completion: 08-01 23:21:05] {916} INFO - evaluating data instance 144\n",
"[flaml.autogen.oai.completion: 08-01 23:21:10] {916} INFO - evaluating data instance 145\n",
"[flaml.autogen.oai.completion: 08-01 23:21:17] {916} INFO - evaluating data instance 146\n",
"[flaml.autogen.oai.completion: 08-01 23:21:25] {916} INFO - evaluating data instance 147\n",
"[flaml.autogen.oai.completion: 08-01 23:21:38] {916} INFO - evaluating data instance 148\n",
"[flaml.autogen.oai.completion: 08-01 23:21:54] {916} INFO - evaluating data instance 149\n",
"[flaml.autogen.oai.completion: 08-01 23:22:05] {916} INFO - evaluating data instance 150\n",
"[flaml.autogen.oai.completion: 08-01 23:22:13] {916} INFO - evaluating data instance 151\n",
"[flaml.autogen.oai.completion: 08-01 23:22:24] {916} INFO - evaluating data instance 152\n",
"[flaml.autogen.oai.completion: 08-01 23:22:35] {916} INFO - evaluating data instance 153\n",
"[flaml.autogen.oai.completion: 08-01 23:22:44] {916} INFO - evaluating data instance 154\n",
"[flaml.autogen.oai.completion: 08-01 23:22:53] {916} INFO - evaluating data instance 155\n",
"[flaml.autogen.oai.completion: 08-01 23:23:01] {916} INFO - evaluating data instance 156\n",
"[flaml.autogen.oai.completion: 08-01 23:23:16] {916} INFO - evaluating data instance 157\n",
"[flaml.autogen.oai.completion: 08-01 23:23:23] {916} INFO - evaluating data instance 158\n",
"[flaml.autogen.oai.completion: 08-01 23:23:31] {916} INFO - evaluating data instance 159\n",
"[flaml.autogen.oai.completion: 08-01 23:23:44] {916} INFO - evaluating data instance 160\n",
"[flaml.autogen.oai.completion: 08-01 23:23:57] {916} INFO - evaluating data instance 161\n",
"[flaml.autogen.oai.completion: 08-01 23:24:03] {916} INFO - evaluating data instance 162\n",
"[flaml.autogen.oai.completion: 08-01 23:24:09] {916} INFO - evaluating data instance 163\n",
"[flaml.autogen.oai.completion: 08-01 23:24:16] {916} INFO - evaluating data instance 164\n",
"[flaml.autogen.oai.completion: 08-01 23:24:28] {916} INFO - evaluating data instance 165\n",
"[flaml.autogen.oai.completion: 08-01 23:24:39] {916} INFO - evaluating data instance 166\n",
"[flaml.autogen.oai.completion: 08-01 23:24:55] {916} INFO - evaluating data instance 167\n",
"[flaml.autogen.oai.completion: 08-01 23:25:00] {916} INFO - evaluating data instance 168\n",
"[flaml.autogen.oai.completion: 08-01 23:25:16] {916} INFO - evaluating data instance 169\n",
"[flaml.autogen.oai.completion: 08-01 23:25:23] {916} INFO - evaluating data instance 170\n",
"[flaml.autogen.oai.completion: 08-01 23:25:31] {916} INFO - evaluating data instance 171\n",
"[flaml.autogen.oai.completion: 08-01 23:25:36] {916} INFO - evaluating data instance 172\n",
"[flaml.autogen.oai.completion: 08-01 23:25:44] {916} INFO - evaluating data instance 173\n",
"[flaml.autogen.oai.completion: 08-01 23:25:56] {916} INFO - evaluating data instance 174\n",
"[flaml.autogen.oai.completion: 08-01 23:26:07] {916} INFO - evaluating data instance 175\n",
"[flaml.autogen.oai.completion: 08-01 23:26:21] {916} INFO - evaluating data instance 176\n",
"[flaml.autogen.oai.completion: 08-01 23:26:27] {916} INFO - evaluating data instance 177\n",
"[flaml.autogen.oai.completion: 08-01 23:26:34] {916} INFO - evaluating data instance 178\n",
"[flaml.autogen.oai.completion: 08-01 23:26:47] {916} INFO - evaluating data instance 179\n",
"[flaml.autogen.oai.completion: 08-01 23:27:01] {916} INFO - evaluating data instance 180\n",
"[flaml.autogen.oai.completion: 08-01 23:27:15] {916} INFO - evaluating data instance 181\n",
"[flaml.autogen.oai.completion: 08-01 23:27:22] {916} INFO - evaluating data instance 182\n",
"[flaml.autogen.oai.completion: 08-01 23:27:29] {916} INFO - evaluating data instance 183\n",
"[flaml.autogen.oai.completion: 08-01 23:27:40] {916} INFO - evaluating data instance 184\n",
"[flaml.autogen.oai.completion: 08-01 23:27:49] {916} INFO - evaluating data instance 185\n",
"[flaml.autogen.oai.completion: 08-01 23:27:55] {916} INFO - evaluating data instance 186\n",
"[flaml.autogen.oai.completion: 08-01 23:28:02] {916} INFO - evaluating data instance 187\n",
"[flaml.autogen.oai.completion: 08-01 23:28:06] {916} INFO - evaluating data instance 188\n",
"[flaml.autogen.oai.completion: 08-01 23:28:18] {916} INFO - evaluating data instance 189\n",
"[flaml.autogen.oai.completion: 08-01 23:28:27] {916} INFO - evaluating data instance 190\n",
"[flaml.autogen.oai.completion: 08-01 23:28:37] {916} INFO - evaluating data instance 191\n",
"[flaml.autogen.oai.completion: 08-01 23:28:49] {916} INFO - evaluating data instance 192\n",
"[flaml.autogen.oai.completion: 08-01 23:29:01] {916} INFO - evaluating data instance 193\n",
"[flaml.autogen.oai.completion: 08-01 23:29:14] {916} INFO - evaluating data instance 194\n",
"[flaml.autogen.oai.completion: 08-01 23:29:21] {916} INFO - evaluating data instance 195\n",
"[flaml.autogen.oai.completion: 08-01 23:29:30] {916} INFO - evaluating data instance 196\n",
"[flaml.autogen.oai.completion: 08-01 23:29:42] {916} INFO - evaluating data instance 197\n",
"[flaml.autogen.oai.completion: 08-01 23:29:56] {916} INFO - evaluating data instance 198\n",
"[flaml.autogen.oai.completion: 08-01 23:30:04] {916} INFO - evaluating data instance 199\n",
"[flaml.autogen.oai.completion: 08-01 23:30:20] {916} INFO - evaluating data instance 200\n",
"performance on test data with the tuned config: {'expected_success': 0.9914855260776184, 'success': 0.9950248756218906, 'success_vote': 0.9203980099502488, 'votes': 31.582089552238806, 'cost': 2.697486000000001, 'inference_cost': 0.01342032835820896}\n"
]
}
],
"outputs": [],
"source": [
"# result = autogen.ChatCompletion.test(test_data, logging_level=logging.INFO, config_list=config_list, **config)\n",
"# print(\"performance on test data with the tuned config:\", result)"
@ -2044,7 +1798,7 @@
"source": [
"We find that the 'success_vote' metric is increased at the cost of exceeding the inference budget. But the tuned configuration has both higher 'success_vote' (91% vs. 87%) and lower average inference cost ($0.015 vs. $0.037 per instance).\n",
"\n",
"A developer could use flaml to tune the configuration to satisfy the target inference budget while maximizing the value out of it."
"A developer could use AutoGen to tune the configuration to satisfy the target inference budget while maximizing the value out of it."
]
}
],

16
notebook/autogen_openai_completion.ipynb
View File

@ -5,7 +5,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_openai_completion.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_openai_completion.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
@ -21,18 +21,18 @@
"\n",
"Licensed under the MIT License.\n",
"\n",
"# Use FLAML to Tune OpenAI Models\n",
"# Use AutoGen to Tune OpenAI Models\n",
"\n",
"`flaml.autogen` offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The research study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
"AutoGen offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The research study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
"Please find documentation about this feature [here](/docs/Use-Cases/AutoGen#enhanced-inference).\n",
"\n",
"In this notebook, we tune OpenAI models for code generation. We use [the HumanEval benchmark](https://huggingface.co/datasets/openai_humaneval) released by OpenAI for synthesizing programs from docstrings.\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen,blendsearch] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install with the [blendsearch] option:\n",
"```bash\n",
"pip install flaml[autogen,blendsearch]\n",
"pip install pyautogen[blendsearch]\n",
"```"
]
},
@ -49,7 +49,7 @@
},
"outputs": [],
"source": [
"# %pip install flaml[autogen,blendsearch]~=2.0.0 datasets"
"# %pip install \"pyautogen[blendsearch]~=0.1.0\" datasets"
]
},
{
@ -81,7 +81,7 @@
},
"outputs": [],
"source": [
"from flaml import autogen\n",
"import autogen\n",
"\n",
"endpoint_list = autogen.config_list_openai_aoai()\n",
"# the endpoint_list looks like this:\n",
@ -362,7 +362,7 @@
"\n",
"## Use the tuning data to find a good configuration\n",
"\n",
"FLAML has provided an API for hyperparameter optimization of OpenAI models: `autogen.Completion.tune` and to make a request with the tuned config: `autogen.Completion.create`.\n",
"AutoGen has provided an API for hyperparameter optimization of OpenAI models: `autogen.Completion.tune` and to make a request with the tuned config: `autogen.Completion.create`.\n",
"\n",
"For (local) reproducibility and cost efficiency, we cache responses from OpenAI with a controllable seed."
]

6
notebook/research/autogen_code.ipynb
View File

@ -13,15 +13,15 @@
"\n",
"Licensed under the MIT License.\n",
"\n",
"# Use FLAML to Optimize Code Generation Performance\n",
"# Use AutoGen to Optimize Code Generation Performance\n",
"\n",
"In this notebook, we optimize OpenAI models for code generation. We use [the HumanEval benchmark](https://huggingface.co/datasets/openai_humaneval) released by OpenAI for synthesizing programs from docstrings.\n",
"\n",
"Related link: [Blogpost](https://microsoft.github.io/FLAML/blog/2023/05/18/GPT-adaptive-humaneval) based on this experiment.\n",
"Related link: [Blogpost](https://microsoft.github.io/autogen/blog/2023/05/18/GPT-adaptive-humaneval) based on this experiment.\n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [autogen] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[autogen]==1.2.2\n",
"```"

2
notebook/research/math_level5counting.ipynb
View File

@ -19,7 +19,7 @@
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [openai] option:\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n",
"```bash\n",
"pip install flaml[openai]==1.2.2\n",
"```"

4
test/agentchat/test_assistant_agent.py
View File

@ -1,8 +1,8 @@
import os
import sys
import pytest
from flaml import autogen
from flaml.autogen.agentchat import AssistantAgent, UserProxyAgent
import autogen
from autogen.agentchat import AssistantAgent, UserProxyAgent
KEY_LOC = "notebook"
OAI_CONFIG_LIST = "OAI_CONFIG_LIST"

2
test/agentchat/test_async.py
View File

@ -1,5 +1,5 @@
import asyncio
from flaml import autogen
import autogen
from test_assistant_agent import KEY_LOC, OAI_CONFIG_LIST

2
test/agentchat/test_conversable_agent.py
View File

@ -1,5 +1,5 @@
import pytest
from flaml.autogen.agentchat import ConversableAgent
from autogen.agentchat import ConversableAgent
def test_trigger():

2
test/agentchat/test_groupchat.py
View File

@ -1,4 +1,4 @@
from flaml import autogen
import autogen
def test_chat_manager():

6
test/agentchat/test_math_user_proxy_agent.py
View File

@ -1,7 +1,7 @@
import pytest
import sys
from flaml import autogen
from flaml.autogen.agentchat.contrib.math_user_proxy_agent import (
import autogen
from autogen.agentchat.contrib.math_user_proxy_agent import (
MathUserProxyAgent,
_remove_print,
_add_print_to_last_line,
@ -19,7 +19,7 @@ def test_math_user_proxy_agent():
except ImportError:
return
from flaml.autogen.agentchat.assistant_agent import AssistantAgent
from autogen.agentchat.assistant_agent import AssistantAgent
conversations = {}
autogen.ChatCompletion.start_logging(conversations)

16
test/agentchat/test_retrievechat.py
View File

@ -1,16 +1,16 @@
import pytest
import sys
from flaml import autogen
import autogen
from test_assistant_agent import KEY_LOC, OAI_CONFIG_LIST
try:
from flaml.autogen.agentchat.contrib.retrieve_assistant_agent import (
from autogen.agentchat.contrib.retrieve_assistant_agent import (
RetrieveAssistantAgent,
)
from flaml.autogen.agentchat.contrib.retrieve_user_proxy_agent import (
from autogen.agentchat.contrib.retrieve_user_proxy_agent import (
RetrieveUserProxyAgent,
)
from flaml.autogen.retrieve_utils import create_vector_db_from_dir, query_vector_db
from autogen.retrieve_utils import create_vector_db_from_dir, query_vector_db
import chromadb
skip_test = False
@ -70,15 +70,15 @@ def test_retrievechat():
)
def test_retrieve_utils():
client = chromadb.PersistentClient(path="/tmp/chromadb")
create_vector_db_from_dir(dir_path="./website/docs", client=client, collection_name="flaml-docs")
create_vector_db_from_dir(dir_path="./website/docs", client=client, collection_name="autogen-docs")
results = query_vector_db(
query_texts=[
"How can I use FLAML UserProxyAgent and AssistantAgent to do code generation?",
"How can I use AutoGen UserProxyAgent and AssistantAgent to do code generation?",
],
n_results=4,
client=client,
collection_name="flaml-docs",
search_string="FLAML",
collection_name="autogen-docs",
search_string="AutoGen",
)
print(results["ids"][0])
assert len(results["ids"][0]) == 4

6
test/oai/test_completion.py
View File

@ -5,14 +5,14 @@ import pytest
from functools import partial
import os
import json
from flaml import autogen
from flaml.autogen.code_utils import (
import autogen
from autogen.code_utils import (
eval_function_completions,
generate_assertions,
implement,
generate_code,
)
from flaml.autogen.math_utils import eval_math_responses, solve_problem
from autogen.math_utils import eval_math_responses, solve_problem
KEY_LOC = "notebook"
OAI_CONFIG_LIST = "OAI_CONFIG_LIST"

2
test/oai/test_utils.py
View File

@ -1,6 +1,6 @@
import json
import os
from flaml import autogen
import autogen
from test_completion import KEY_LOC, OAI_CONFIG_LIST

12
test/test_code.py
View File

@ -1,8 +1,8 @@
import sys
import os
import pytest
from flaml import autogen
from flaml.autogen.code_utils import (
import autogen
from autogen.code_utils import (
UNKNOWN,
extract_code,
execute_code,
@ -149,7 +149,7 @@ here = os.path.abspath(os.path.dirname(__file__))
def test_infer_lang():
assert infer_lang("print('hello world')") == "python"
assert infer_lang("pip install flaml") == "sh"
assert infer_lang("pip install autogen") == "sh"
def test_extract_code():
@ -238,7 +238,7 @@ def test_improve():
return
config_list = autogen.config_list_openai_aoai(KEY_LOC)
improved, _ = improve_function(
"flaml/autogen/math_utils.py",
"autogen/math_utils.py",
"solve_problem",
"Solve math problems accurately, by avoiding calculation errors and reduce reasoning errors.",
config_list=config_list,
@ -246,13 +246,13 @@ def test_improve():
with open(f"{here}/math_utils.py.improved", "w") as f:
f.write(improved)
suggestion, _ = improve_code(
["flaml/autogen/code_utils.py", "flaml/autogen/math_utils.py"],
["autogen/code_utils.py", "autogen/math_utils.py"],
"leverage generative AI smartly and cost-effectively",
config_list=config_list,
)
print(suggestion)
improvement, cost = improve_code(
["flaml/autogen/code_utils.py", "flaml/autogen/math_utils.py"],
["autogen/code_utils.py", "autogen/math_utils.py"],
"leverage generative AI smartly and cost-effectively",
suggest_only=False,
config_list=config_list,

8
test/test_function_call.py
View File

@ -4,8 +4,8 @@ except ImportError:
openai = None
import pytest
import json
from flaml import autogen
from flaml.autogen.math_utils import eval_math_responses
import autogen
from autogen.math_utils import eval_math_responses
from test_code import KEY_LOC
@ -61,7 +61,7 @@ def test_eval_math_responses():
def test_json_extraction():
from flaml.autogen.agentchat import UserProxyAgent
from autogen.agentchat import UserProxyAgent
user = UserProxyAgent(name="test", code_execution_config={"use_docker": False})
@ -76,7 +76,7 @@ def test_json_extraction():
def test_execute_function():
from flaml.autogen.agentchat import UserProxyAgent
from autogen.agentchat import UserProxyAgent
# 1. test calling a simple function
def add_num(num_to_be_added):

16
website/blog/2023-04-21-LLM-tuning-math/index.mdx
View File

@ -9,20 +9,20 @@ tags: [LLM, GPT, research]
**TL;DR:**
* **Just by tuning the inference parameters like model, number of responses, temperature etc. without changing any model weights or prompt, the baseline accuracy of untuned gpt-4 can be improved by 20% in high school math competition problems.**
* **For easy problems, the tuned gpt-3.5-turbo model vastly outperformed untuned gpt-4 in accuracy (e.g., 90% vs. 70%) and cost efficiency. For hard problems, the tuned gpt-4 is much more accurate (e.g., 35% vs. 20%) and less expensive than untuned gpt-4.**
* **FLAML can help with model selection, parameter tuning, and cost-saving in LLM applications.**
* **AutoGen can help with model selection, parameter tuning, and cost-saving in LLM applications.**
Large language models (LLMs) are powerful tools that can generate natural language texts for various applications, such as chatbots, summarization, translation, and more. GPT-4 is currently the state of the art LLM in the world. Is model selection irrelevant? What about inference parameters?
In this blog post, we will explore how model and inference parameter matter in LLM applications, using a case study for [MATH](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html), a benchmark for evaluating LLMs on advanced mathematical problem solving. MATH consists of 12K math competition problems from AMC-10, AMC-12 and AIME. Each problem is accompanied by a step-by-step solution.
We will use the new subpackage [`flaml.autogen`](docs/Use-Cases/Autogen) to automatically find the best model and inference parameter for LLMs on a given task and dataset given an inference budget, using a novel low-cost search & pruning strategy. FLAML currently supports all the LLMs from OpenAI, such as GPT-3.5 and GPT-4.
We will use AutoGen to automatically find the best model and inference parameter for LLMs on a given task and dataset given an inference budget, using a novel low-cost search & pruning strategy. AutoGen currently supports all the LLMs from OpenAI, such as GPT-3.5 and GPT-4.
We will use FLAML to perform model selection and inference parameter tuning. Then we compare the performance and inference cost on solving algebra problems with the untuned gpt-4. We will also analyze how different difficulty levels affect the results.
We will use AutoGen to perform model selection and inference parameter tuning. Then we compare the performance and inference cost on solving algebra problems with the untuned gpt-4. We will also analyze how different difficulty levels affect the results.
## Experiment Setup
We use FLAML to select between the following models with a target inference budget $0.02 per instance:
We use AutoGen to select between the following models with a target inference budget $0.02 per instance:
- gpt-3.5-turbo, a relatively cheap model that powers the popular ChatGPT app
- gpt-4, the state of the art LLM that costs more than 10 times of gpt-3.5-turbo
@ -54,21 +54,21 @@ On level 5 the result is similar.
![level 5 algebra](img/level5algebra.png)
We can see that FLAML has found different optimal model and inference parameters for each subset of a particular level, which shows that these parameters matter in cost-sensitive LLM applications and need to be carefully tuned or adapted.
We can see that AutoGen has found different optimal model and inference parameters for each subset of a particular level, which shows that these parameters matter in cost-sensitive LLM applications and need to be carefully tuned or adapted.
An example notebook to run these experiments can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/autogen_chatgpt.ipynb
An example notebook to run these experiments can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/autogen_chatgpt.ipynb. The experiments were run when AutoGen was a subpackage in FLAML.
## Analysis and Discussion
While gpt-3.5-turbo demonstrates competitive accuracy with voted answers in relatively easy algebra problems under the same inference budget, gpt-4 is a better choice for the most difficult problems. In general, through parameter tuning and model selection, we can identify the opportunity to save the expensive model for more challenging tasks, and improve the overall effectiveness of a budget-constrained system.
There are many other alternative ways of solving math problems, which we have not covered in this blog post. When there are choices beyond the inference parameters, they can be generally tuned via [`flaml.tune`](docs/Use-Cases/Tune-User-Defined-Function).
There are many other alternative ways of solving math problems, which we have not covered in this blog post. When there are choices beyond the inference parameters, they can be generally tuned via [`flaml.tune`](https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function).
The need for model selection, parameter tuning and cost saving is not specific to the math problems. The [Auto-GPT](https://github.com/Significant-Gravitas/Auto-GPT) project is an example where high cost can easily prevent a generic complex task to be accomplished as it needs many LLM inference calls.
## For Further Reading
* [Research paper about the tuning technique](https://arxiv.org/abs/2303.04673)
* [Documentation about `flaml.autogen`](https://microsoft.github.io/autogen/)
* [Documentation about AutoGen](https://microsoft.github.io/autogen/)
*Do you have any experience to share about LLM applications? Do you like to see more support or research of LLM optimization or automation? Please join our [Discord](https://discord.gg/Cppx2vSPVP) server for discussion.*

10
website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx
View File

@ -56,7 +56,7 @@ Combining these observations, we can design a solution with two intuitive ideas:
This solution works adaptively without knowing or predicting which task fits which configuration. It simply tries multiple configurations one by one, starting from the cheapest configuration. Note that one configuration can generate multiple responses (by setting the inference parameter n larger than 1). And different configurations can use the same model and different inference parameters such as n and temperature. Only one response is returned and evaluated per task.
An implementation of this solution is provided in [flaml.autogen](/docs/reference/autogen/code_utils#implement). It uses the following sequence of configurations:
An implementation of this solution is provided in [autogen](/docs/reference/code_utils#implement). It uses the following sequence of configurations:
1. GPT-3.5-Turbo, n=1, temperature=0
1. GPT-3.5-Turbo, n=7, temperature=1, stop=["\nclass", "\ndef", "\nif", "\nprint"]
@ -140,11 +140,11 @@ The last problem is an example with wrong example test cases in the original def
It is worth noting that the reduced inference cost is the amortized cost over all the tasks. For each individual task, the cost can be either larger or smaller than directly using GPT-4. This is the nature of the adaptive solution: The cost is in general larger for difficult tasks than that for easy tasks.
An example notebook to run this experiment can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/research/autogen_code.ipynb
An example notebook to run this experiment can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/research/autogen_code.ipynb. The experiment was run when AutoGen was a subpackage in FLAML.
## Discussion
Our solution is quite simple to implement using a generic interface offered in [`flaml.autogen`](https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#logic-error), yet the result is quite encouraging.
Our solution is quite simple to implement using a generic interface offered in [`autogen`](https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#logic-error), yet the result is quite encouraging.
While the specific way of generating assertions is application-specific, the main ideas are general in LLM operations:
* Generate multiple responses to select - especially useful when selecting a good response is relatively easier than generating a good response at one shot.
@ -153,7 +153,7 @@ While the specific way of generating assertions is application-specific, the mai
- Different configurations have complementary effect.
A [previous blog post](/blog/2023/04/21/LLM-tuning-math) provides evidence that these ideas are relevant in solving math problems too.
`flaml.autogen` uses a technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) to support inference parameter tuning and model selection.
`autogen` uses a technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) to support inference parameter tuning and model selection.
There are many directions of extensions in research and development:
* Generalize the way to provide feedback.
@ -164,5 +164,5 @@ There are many directions of extensions in research and development:
## For Further Reading
* [Documentation](https://microsoft.github.io/autogen/) about `flaml.autogen` and [Research paper](https://arxiv.org/abs/2303.04673).
* [Documentation](https://microsoft.github.io/autogen/) about `autogen` and [Research paper](https://arxiv.org/abs/2303.04673).
* [Blog post](/blog/2023/04/21/LLM-tuning-math) about a related study for math.

4
website/blog/2023-06-28-MathChat/index.mdx
View File

@ -75,7 +75,7 @@ We found that compared to basic prompting, which demonstrates the innate capabil
For categories like Algebra and Prealgebra, PoT and PS showed little improvement, and in some instances, even led to a decrease in accuracy. However, MathChat was able to enhance total accuracy by around 6% compared to PoT and PS, showing competitive performance across all categories. Remarkably, MathChat improved accuracy in the Algebra category by about 15% over other methods. Note that categories like Intermediate Algebra and Precalculus remained challenging for all methods, with only about 20% of problems solved accurately.
The code for experiments can be found at this [repository](https://github.com/kevin666aa/FLAML/tree/gpt_math_solver/flaml/autogen/math).
We now provide an implementation of MathChat using the interactive agents in FLAML. See this [notebook](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_MathChat.ipynb) for example usage.
We now provide an implementation of MathChat using the interactive agents in AutoGen. See this [notebook](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_MathChat.ipynb) for example usage.
## Future Directions
@ -89,6 +89,6 @@ Further work can be done to enhance this framework or math problem-solving in ge
## For Further Reading
* [Research paper of MathChat](https://arxiv.org/abs/2306.01337)
* [Documentation about `flaml.autogen`](https://microsoft.github.io/autogen/)
* [Documentation about `autogen`](https://microsoft.github.io/autogen/)
*Are you working on applications that involve math problem-solving? Would you appreciate additional research or support on the application of LLM-based agents for math problem-solving? Please join our [Discord](https://discord.gg/Cppx2vSPVP) server for discussion.*

14
website/blog/2023-07-14-Local-LLMs/index.mdx
View File

@ -1,10 +1,10 @@
---
title: Use flaml.autogen for Local LLMs
title: Use AutoGen for Local LLMs
authors: jialeliu
tags: [LLM, FLAMLv2]
tags: [LLM]
---
**TL;DR:**
We demonstrate how to use flaml.autogen for local LLM application. As an example, we will initiate an endpoint using [FastChat](https://github.com/lm-sys/FastChat) and perform inference on [ChatGLMv2-6b](https://github.com/THUDM/ChatGLM2-6B).
We demonstrate how to use autogen for local LLM application. As an example, we will initiate an endpoint using [FastChat](https://github.com/lm-sys/FastChat) and perform inference on [ChatGLMv2-6b](https://github.com/THUDM/ChatGLM2-6B).
## Preparations
@ -66,11 +66,11 @@ class CompletionResponseStreamChoice(BaseModel):
## Interact with model using `oai.Completion`
Now the models can be directly accessed through openai-python library as well as `flaml.oai.Completion` and `flaml.oai.ChatCompletion`.
Now the models can be directly accessed through openai-python library as well as `autogen.oai.Completion` and `autogen.oai.ChatCompletion`.
```python
from flaml import oai
from autogen import oai
# create a text completion request
response = oai.Completion.create(
@ -118,7 +118,7 @@ python -m fastchat.serve.multi_model_worker \
The inference code would be:
```python
from flaml import oai
from autogen import oai
# create a chat completion request
response = oai.ChatCompletion.create(
@ -143,5 +143,5 @@ print(response)
## For Further Reading
* [Documentation](https://microsoft.github.io/autogen/) about `flaml.autogen`
* [Documentation](https://microsoft.github.io/autogen/) about `autogen`
* [Documentation](https://github.com/lm-sys/FastChat) about FastChat.

22
website/docs/Examples/AutoGen-AgentChat.md
View File

@ -4,14 +4,14 @@ AutoGen offers conversable agents powered by LLM, tool or human, which can be us
Please find documentation about this feature [here](/docs/Use-Cases/agent_chat).
Links to notebook examples:
* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)
* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_human_feedback.ipynb)
* [Solve Tasks Requiring Web Info](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_web_info.ipynb)
* [Use Provided Tools as Functions](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_function_call.ipynb)
* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_planning.ipynb)
* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_two_users.ipynb)
* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_chess.ipynb)
* [Automated Task Solving by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat.ipynb)
* [Automated Data Visualization by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb)
* [Automated Complex Task Solving by Group Chat (with 6 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb)
* [Automated Continual Learning from New Data](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_stream.ipynb)
* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)
* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_human_feedback.ipynb)
* [Solve Tasks Requiring Web Info](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_web_info.ipynb)
* [Use Provided Tools as Functions](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_function_call.ipynb)
* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_planning.ipynb)
* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_two_users.ipynb)
* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_chess.ipynb)
* [Automated Task Solving by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat.ipynb)
* [Automated Data Visualization by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb)
* [Automated Complex Task Solving by Group Chat (with 6 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb)
* [Automated Continual Learning from New Data](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_stream.ipynb)

4
website/docs/Examples/AutoGen-OpenAI.md
View File

@ -4,5 +4,5 @@ AutoGen also offers a cost-effective hyperparameter optimization technique [EcoO
Please find documentation about this feature [here](/docs/Use-Cases/enhanced_inference).
Links to notebook examples:
* [Optimize for Code Generation](https://github.com/microsoft/flaml/blob/main/notebook/autogen_openai_completion.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/flaml/blob/main/notebook/autogen_openai_completion.ipynb)
* [Optimize for Math](https://github.com/microsoft/flaml/blob/main/notebook/autogen_chatgpt_gpt4.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/flaml/blob/main/notebook/autogen_chatgpt_gpt4.ipynb)
* [Optimize for Code Generation](https://github.com/microsoft/autogen/blob/main/notebook/autogen_openai_completion.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_openai_completion.ipynb)
* [Optimize for Math](https://github.com/microsoft/autogen/blob/main/notebook/autogen_chatgpt_gpt4.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_chatgpt_gpt4.ipynb)

10
website/docs/Getting-Started.md
View File

@ -20,14 +20,14 @@ AutoGen is powered by collaborative [research studies](/docs/Research) from Micr
### Quickstart
Install AutoGen from pip: `pip install "flaml[autogen]"`. Find more options in [Installation](/docs/Installation).
Install from pip: `pip install pyautogen`. Find more options in [Installation](/docs/Installation).
#### Multi-Agent Conversation Framework
Autogen enables the next-gen LLM applications with a generic multi-agent conversation framework. It offers customizable and conversable agents which integrate LLMs, tools and human.
By automating chat among multiple capable agents, one can easily make them collectively perform tasks autonomously or with human feedback, including tasks that require using tools via code. For example,
```python
from flaml.autogen import AssistantAgent, UserProxyAgent
from autogen import AssistantAgent, UserProxyAgent
assistant = AssistantAgent("assistant")
user_proxy = UserProxyAgent("user_proxy")
user_proxy.initiate_chat(assistant, message="Plot a chart of META and TESLA stock price change YTD.")
@ -59,10 +59,10 @@ response = autogen.Completion.create(context=test_instance, **config)
* Understand the use cases for [multi-agent conversation](/docs/Use-Cases/agent_chat).
* Understand the use cases for [enhanced LLM inference](/docs/Use-Cases/enhanced_inference).
* Find code examples from [Examples](/docs/Examples/AutoGen-AgentChat).
* Read [SDK](https://microsoft.github.io/FLAML/docs/reference/autogen/agentchat/responsive_agent/).
* Read [SDK](/docs/reference/autogen/agentchat/conversable_agent/).
* Learn about [research](/docs/Research) around AutoGen.
* Chat on [Discord](https://discord.gg/Cppx2vSPVP).
If you like our project, please give it a [star](https://github.com/microsoft/FLAML/stargazers) on GitHub. If you are interested in contributing, please read [Contributor's Guide](/docs/Contribute).
If you like our project, please give it a [star](https://github.com/microsoft/autogen/stargazers) on GitHub. If you are interested in contributing, please read [Contributor's Guide](/docs/Contribute).
<!-- <iframe src="https://ghbtns.com/github-btn.html?user=microsoft&amp;repo=FLAML&amp;type=star&amp;count=true&amp;size=large" frameborder="0" scrolling="0" width="170" height="30" title="GitHub"></iframe> -->
<!-- <iframe src="https://ghbtns.com/github-btn.html?user=microsoft&amp;repo=autogen&amp;type=star&amp;count=true&amp;size=large" frameborder="0" scrolling="0" width="170" height="30" title="GitHub"></iframe> -->

28
website/docs/Use-Cases/agent_chat.md
View File

@ -31,7 +31,7 @@ One can also easily extend it by registering reply functions with the `register_
In the following code, we create an `AssistantAgent` named "assistant" to serve as the assistant and a `UserProxyAgent` named "user_proxy" to serve as a proxy for the human user. We will later employ these two agents to solve a task.
```python
from flaml.autogen import AssistantAgent, UserProxyAgent
from autogen import AssistantAgent, UserProxyAgent
# create an AssistantAgent instance named "assistant"
assistant = AssistantAgent(name="assistant")
@ -69,10 +69,10 @@ On the one hand, one can achieve fully autonomous conversations after an initial
By adopting the conversation-driven control with both programming language and natural language, AutoGen inherently allows dynamic conversation. Dynamic conversation allows the agent topology to change depending on the actual flow of conversation under different input problem instances, while the flow of a static conversation always follows a pre-defined topology. The dynamic conversation pattern is useful in complex applications where the patterns of interaction cannot be predetermined in advance. AutoGen provides two general approaches to achieving dynamic conversation:
- Registered auto-reply. With the pluggable auto-reply function, one can choose to invoke conversations with other agents depending on the content of the current message and context. A working system demonstrating this type of dynamic conversation can be found in this code example, demonstrating a [dynamic group chat](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_groupchat.ipynb). In the system, we register an auto-reply function in the group chat manager, which lets LLM decide who will the next speaker be in a group chat setting.
- Registered auto-reply. With the pluggable auto-reply function, one can choose to invoke conversations with other agents depending on the content of the current message and context. A working system demonstrating this type of dynamic conversation can be found in this code example, demonstrating a [dynamic group chat](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat.ipynb). In the system, we register an auto-reply function in the group chat manager, which lets LLM decide who will the next speaker be in a group chat setting.
- LLM-based function call. In this approach, LLM decides whether or not to call a particular function depending on the conversation status in each inference call.
By messaging additional agents in the called functions, the LLM can drive dynamic multi-agent conversation. A working system showcasing this type of dynamic conversation can be found in the [multi-user math problem solving scenario](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_two_users.ipynb), where a student assistant would automatically resort to an expert using function calls.
By messaging additional agents in the called functions, the LLM can drive dynamic multi-agent conversation. A working system showcasing this type of dynamic conversation can be found in the [multi-user math problem solving scenario](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_two_users.ipynb), where a student assistant would automatically resort to an expert using function calls.
### Diverse Applications Implemented with AutoGen
@ -80,17 +80,17 @@ By messaging additional agents in the called functions, the LLM can drive dynami
The figure below shows six examples of applications built using AutoGen.
![Applications](images/app.png)
* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)
* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_human_feedback.ipynb)
* [Solve Tasks Requiring Web Info](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_web_info.ipynb)
* [Use Provided Tools as Functions](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_function_call.ipynb)
* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_planning.ipynb)
* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_two_users.ipynb)
* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_chess.ipynb)
* [Automated Task Solving by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat.ipynb)
* [Automated Data Visualization by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb)
* [Automated Complex Task Solving by Group Chat (with 6 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb)
* [Automated Continual Learning from New Data](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_stream.ipynb)
* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)
* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_human_feedback.ipynb)
* [Solve Tasks Requiring Web Info](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_web_info.ipynb)
* [Use Provided Tools as Functions](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_function_call.ipynb)
* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_planning.ipynb)
* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_two_users.ipynb)
* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_chess.ipynb)
* [Automated Task Solving by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat.ipynb)
* [Automated Data Visualization by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb)
* [Automated Complex Task Solving by Group Chat (with 6 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb)
* [Automated Continual Learning from New Data](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_stream.ipynb)

10
website/docs/Use-Cases/enhanced_inference.md
View File

@ -6,8 +6,8 @@ There are a number of benefits of using `autogen` to perform inference: performa
## Tune Inference Parameters
*Links to notebook examples:*
* [Optimize for Code Generation](https://github.com/microsoft/flaml/blob/main/notebook/autogen_openai_completion.ipynb)
* [Optimize for Math](https://github.com/microsoft/flaml/blob/main/notebook/autogen_chatgpt_gpt4.ipynb)
* [Optimize for Code Generation](https://github.com/microsoft/autogen/blob/main/notebook/autogen_openai_completion.ipynb)
* [Optimize for Math](https://github.com/microsoft/autogen/blob/main/notebook/autogen_chatgpt_gpt4.ipynb)
### Choices to optimize
@ -30,7 +30,7 @@ There are also complex interactions among subsets of the hyperparameters. For ex
the temperature and top_p are not recommended to be altered from their default values together because they both control the randomness of the generated text, and changing both at the same time can result in conflicting effects; n and best_of are rarely tuned together because if the application can process multiple outputs, filtering on the server side causes unnecessary information loss; both n and max_tokens will affect the total number of tokens generated, which in turn will affect the cost of the request.
These interactions and trade-offs make it difficult to manually determine the optimal hyperparameter settings for a given text generation task.
*Do the choices matter? Check this [blogpost](https://microsoft.github.io/FLAML/blog/2023/04/21/LLM-tuning-math) to find example tuning results about gpt-3.5-turbo and gpt-4.*
*Do the choices matter? Check this [blogpost](https://microsoft.github.io/autogen/blog/2023/04/21/LLM-tuning-math) to find example tuning results about gpt-3.5-turbo and gpt-4.*
With AutoGen, the tuning can be performed with the following information:
@ -111,7 +111,7 @@ The tuend config can be used to perform inference.
`autogen.Completion.create` is compatible with both `openai.Completion.create` and `openai.ChatCompletion.create`, and both OpenAI API and Azure OpenAI API. So models such as "text-davinci-003", "gpt-3.5-turbo" and "gpt-4" can share a common API.
When chat models are used and `prompt` is given as the input to `autogen.Completion.create`, the prompt will be automatically converted into `messages` to fit the chat completion API requirement. One advantage is that one can experiment with both chat and non-chat models for the same prompt in a unified API.
For local LLMs, one can spin up an endpoint using a package like [FastChat](https://github.com/lm-sys/FastChat), and then use the same API to send a request. See [here](https://microsoft.github.io/FLAML/blog/2023/07/14/Local-LLMs) for examples on how to make inference with local LLMs.
For local LLMs, one can spin up an endpoint using a package like [FastChat](https://github.com/lm-sys/FastChat), and then use the same API to send a request. See [here](https://microsoft.github.io/autogen/blog/2023/07/14/Local-LLMs) for examples on how to make inference with local LLMs.
When only working with the chat-based models, `autogen.ChatCompletion` can be used. It also does automatic conversion from prompt to messages, if prompt is provided instead of messages.
@ -183,7 +183,7 @@ response = autogen.Completion.create(
The example above will try to use text-ada-001, gpt-3.5-turbo, and text-davinci-003 iteratively, until a valid json string is returned or the last config is used. One can also repeat the same model in the list for multiple times to try one model multiple times for increasing the robustness of the final response.
*Advanced use case: Check this [blogpost](https://microsoft.github.io/FLAML/blog/2023/05/18/GPT-adaptive-humaneval) to find how to improve GPT-4's coding performance from 68% to 90% while reducing the inference cost.*
*Advanced use case: Check this [blogpost](https://microsoft.github.io/autogen/blog/2023/05/18/GPT-adaptive-humaneval) to find how to improve GPT-4's coding performance from 68% to 90% while reducing the inference cost.*
## Templating