"<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_teachability.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Making OpenAI Assistants Teachable\n",
"\n",
"Conversational assistants based on LLMs can remember the current chat with the user, and can even demonstrate in-context learning of things that the user teaches the assistant during the chat. But these memories and learnings are lost once the chat is over, or when a single chat grows too long for the LLM to handle effectively. In subsequent chats, the user is forced to repeat any necessary instructions over and over.\n",
"\n",
"The optional agent capability called `Teachability` addresses these limitations by persisting user teachings across chat boundaries in long-term memory (a vector database). Memories (called memos) are created and saved to disk throughout a conversation, then loaded from disk later. Instead of copying all the memos into the context window, which would eat up valuable space, individual memos are retrieved into context only as needed. This allows the user to teach many facts, preferences and skills to the teachable agent just once, and have it remember them in later chats.\n",
"\n",
"In making decisions about memo storage and retrieval, `Teachability` calls an instance of `TextAnalyzerAgent` to analyze pieces of text in several different ways. This adds extra LLM calls involving a relatively small number of tokens. These calls can add a few seconds to the time a user waits for a response.\n",
"\n",
"This notebook demonstrates how `Teachability` can be added to instances of `GPTAssistantAgent`\n",
"so that they can learn facts, preferences, and skills from users. As explained [here](https://microsoft.github.io/autogen/blog/2023/11/13/OAI-assistants), each instance of `GPTAssistantAgent` wraps an OpenAI Assistant that can be given a set of tools including functions, code interpreter, and retrieval. Assistants with these tools are demonstrated in separate standalone sections below, which can be run independently.\n",
"\n",
"## Requirements\n",
"\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install the [teachable] option.\n",
"```bash\n",
"pip install \"pyautogen[teachable]\"\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"# %pip install \"pyautogen[teachable]\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set your API Endpoint\n",
"\n",
"The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file."
"It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well). After application of the filter shown above, only the gpt-4 models are considered.\n",
"\n",
"The config list may look like the following:\n",
"```python\n",
"config_list = [\n",
" {\n",
" 'model': 'gpt-4-1106-preview',\n",
" 'api_key': '<your OpenAI API key here>',\n",
" },\n",
" {\n",
" 'model': 'gpt-4',\n",
" 'api_key': '<your OpenAI API key here>',\n",
" },\n",
" {\n",
" 'model': 'gpt-4',\n",
" 'api_key': '<your Azure OpenAI API key here>',\n",
" 'base_url': '<your Azure OpenAI API base here>',\n",
"If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
"\n",
"You can set the value of config_list in other ways if you prefer, e.g., loading from a YAML file."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Teachable OpenAI Assistant with Functions\n",
"This section is based on [agentchat_oai_assistant_function_call.ipynb](https://github.com/microsoft/autogen/blob/teach_cap/notebook/agentchat_oai_assistant_function_call.ipynb), which demonstrates an assistant accomplishing a task through **function calling**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Function schema and implementation\n",
"This example leverages OSS Insight (Open Source Software Insight) for advanced GitHub data analysis."
" \"description\": \"This is an API endpoint allowing users (analysts) to input question about GitHub in text format to retrieve the realted and structured data.\",\n",
" \"Hello, Open Source Project Analyst. You'll conduct comprehensive evaluations of open source projects or organizations on the GitHub platform, \"\n",
" \"analyzing project trajectories, contributor engagements, open source trends, and other vital parameters. \"\n",
" \"Please carefully read the context of the conversation to identify the current analysis question or problem that needs addressing.\"\n",
" ),\n",
" llm_config=llm_config,\n",
" verbose=True,\n",
")\n",
"oss_analyst.register_function(\n",
" function_map={\n",
" \"ossinsight_data_api\": get_ossinsight,\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Give the assistant a task involving function calls"
" }, # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.\n",
"Finally, let's clear the chat history to see whether the assistant can remember to use our preferred formatting in a new chat without having to be told again."
"user_proxy.initiate_chat(oss_analyst, message=\"List the top 10 developers with the most followers.\", clear_history=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Delete the teachable assistant\n",
"All OpenAI Assistants can be created, viewed and deleted manually through OpenAI's [website](https://platform.openai.com/assistants). They can also be deleted through the `GPTAssistantAgent` instance."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Permanently deleting assistant...\n"
]
}
],
"source": [
"oss_analyst.delete_assistant()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Teachable OpenAI Assistant with Code Interpreter\n",
"This section is based on [agentchat_oai_code_interpreter.ipynb](https://github.com/microsoft/autogen/blob/teach_cap/notebook/agentchat_oai_code_interpreter.ipynb), which demonstrates an assistant accomplishing a task by **writing code and executing it** in a sandbox."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create the OpenAI Assistant with code interpreter as a tool"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"GPT Assistant only supports one OpenAI client. Using the first client in the list.\n",
"No matching assistant found, creating a new assistant\n"
]
}
],
"source": [
"# Initiate an agent equipped with code interpreter\n",
" instructions=\"You are an expert at solving math questions. Write code and run it to solve math problems. Reply TERMINATE when the task is solved and there is no problem.\",\n",
")\n",
"\n",
"user_proxy = UserProxyAgent(\n",
" name=\"user_proxy\",\n",
" is_termination_msg=lambda msg: \"TERMINATE\" in msg[\"content\"],\n",
" \"use_docker\": False, # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.\n",
"If $725x + 727y = 1500$ and $729x+ 731y = 1508$, what is the value of $x - y$ ? After finding the values of variables, always explain how to find the solution.\n",
"To find the value of $x - y$ given the system of linear equations:\n",
"\n",
"\\[\n",
"\\begin{align*}\n",
"725x + 727y &= 1500 \\\\\n",
"729x + 731y &= 1508\n",
"\\end{align*}\n",
"\\]\n",
"\n",
"we will use the method of elimination to solve for $x$ and $y$. The steps are as follows:\n",
"\n",
"1. Multiply each equation by a suitable number to make the coefficients of either $x$ or $y$ the same in both equations.\n",
"2. Subtract one equation from the other to eliminate one of the variables.\n",
"3. Solve the resulting equation for the remaining variable.\n",
"4. Substitute the value of the found variable back into one of the original equations to find the value of the other variable.\n",
"5. Calculate $x - y$ using the values of $x$ and $y$ you've found.\n",
"\n",
"Let's apply these steps to our given system of equations. We'll start by multiplying the first equation by $731$ and the second equation by $727$ in order to have the same coefficients for $y$ in both equations and then subtract the second resulting equation from the first.\n",
"\n",
"\n",
"It looks like I made a mistake while trying to multiply the entire equation by a constant. In SymPy, to perform arithmetic operations on both sides of an equation, we should operate on the left-hand side (lhs) and right-hand side (rhs) separately, not on the Eq object directly. Let's correct this by multiplying both the lhs and rhs of each equation by the appropriate constants and try again.\n",
"\n",
"\n",
"I encountered another error here. The solution `solve` function returns for `x` is actually a list, not a dictionary, so I should not be trying to index into it with a symbol. Instead, I need to extract the first (and only) element from the list of solutions for `x` before using it to solve for `y`. Let's correct this and solve it step by step.\n",
"\n",
"First, I will correct the calculation of the resulting equation after eliminating $y$. Then I'll solve this equation for $x$, and afterwards I'll solve for $y$ using the value of $x$ we get. Then we'll calculate $x - y$.\n",
"\n",
"\n",
"The solution to the system of equations is $x = -23$ and $y = 25$. Therefore, the value of $x - y$ is $-48$.\n",
"\n",
"To summarize the solution:\n",
"\n",
"1. We multiplied the first equation by $731$ and the second equation by $727$ to make the coefficients of $y$ the same in both equations.\n",
"2. We then subtracted the second equation from the first to eliminate $y$ and obtained a single equation in terms of $x$.\n",
"3. We solved this resulting equation to find the value of $x$.\n",
"4. With the value of $x$ known, we substituted it back into one of the original equations to solve for $y$.\n",
"5. Finally, we computed $x - y$ using the values of $x$ and $y$ we found.\n",
" message=\"If $725x + 727y = 1500$ and $729x+ 731y = 1508$, what is the value of $x - y$ ? After finding the values of variables, always explain how to find the solution.\",\n",
"These equations describe a system of linear equations. To find the value of $x - y$, we can use the method of elimination or substitution. \n",
"\n",
"First, let's write down the given system:\n",
"\\[\\begin{align*}\n",
"725x + 727y & = 1500 \\quad (1)\\\\\n",
"729x + 731y & = 1508 \\quad (2)\\\\\n",
"\\end{align*}\\]\n",
"\n",
"To eliminate one of the variables, we can subtract equation (1) from equation (2), which gives us:\n",
"\\[4x + 4y = 8\\]\n",
"\n",
"This simplifies to:\n",
"\\[x + y = 2\\quad (3)\\]\n",
"\n",
"Now we have a simplified equation with both $x$ and $y$. We can solve this system of equations (1) and (3) to find the values of $x$ and $y$, and then calculate $x - y$. Let's do this next.\n",
"\n",
"\n",
"The value of \\( x - y \\) is \\(-48\\). \n",
"\n",
"Here's a brief explanation of how we found the solution:\n",
"\n",
"1. We started with two equations given in the problem statement:\n",
"\\[\\begin{align*}\n",
"725x + 727y &= 1500 \\quad (1)\\\\\n",
"729x + 731y &= 1508 \\quad (2)\\\\\n",
"\\end{align*}\\]\n",
"\n",
"2. We subtracted equation (1) from (2) to get a new equation with \\(4x + 4y\\), noticing that the subtraction would create a pair of coefficients that are the same for each variable:\n",
"3. Simplifying the new equation gave us the equation \\(x + y = 2\\), which we label as equation (3).\n",
"\n",
"4. With equation (3) and the original equation (1), we formed a system of two equations with two unknowns. We then solved this system to find values for \\(x\\) and \\(y\\).\n",
"\n",
"5. After solving for \\(x\\) and \\(y\\), we found that \\(x - y = -48\\).\n",
"\n",
"Therefore, the answer to the provided problem is that \\(x - y\\) equals \\(-48\\).\n",
"All OpenAI Assistants can be created, viewed and deleted manually through OpenAI's [website](https://platform.openai.com/assistants). They can also be deleted through the `GPTAssistantAgent` instance."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Permanently deleting assistant...\n"
]
}
],
"source": [
"coder_assistant.delete_assistant()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Teachable OpenAI Assistant with Retrieval\n",
"This section is based on [agentchat_oai_assistant_retrieval.ipynb](https://github.com/microsoft/autogen/blob/teach_cap/notebook/agentchat_oai_assistant_retrieval.ipynb), which demonstrates an assistant accomplishing a task through **retrieval augmented generation (RAG)**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create the OpenAI Assistant with retrieval as a tool\n",
"For this example, first upload the [conversable_agent.py](https://github.com/microsoft/autogen/blob/main/autogen/agentchat/conversable_agent.py) file to your OpenAI API account. This can be done manually through the [website](https://platform.openai.com/assistants). Then find the uploaded File ID on the [Files page](https://platform.openai.com/files), and paste that ID into the `file_ids` list in the code below."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"GPT Assistant only supports one OpenAI client. Using the first client in the list.\n",
"No matching assistant found, creating a new assistant\n"
"The file you provided does not contain any obvious references to bugs, errors, or exceptions outside what would be expected in a normal programming context. Below are a couple of instances where exceptions are handled or mentioned but not indicative of existing bugs within the file:\n",
"\n",
"1. There's code designed to catch and handle `Exception`s by reporting an error with the message `f\"Error: {e}\"` which indicates resilience in error handling but not necessarily a bug【1†source】【2†source】.\n",
"2. An `AssertionError` is raised with an error message if certain conditions are not met, which is a form of input validation to ensure the state of the application is as expected【3†source】.\n",
"3. Another example is raising a `RuntimeError` if an attempt is made to register a function for LLM (Language Learning Model) without a proper configuration set up, again more of a failsafe than a bug【5†source】.\n",
"4. A `ValueError` is raised if a message cannot be converted into a valid ChatCompletion message, which seems to be a part of the normal validation process when generating messages【6†source】.\n",
"\n",
"These instances are typical of robust software that is designed to handle unexpected conditions gracefully. However, the absence of a reference to the word \"bug\" in the search does not guarantee that the file is bug-free, as bugs may not be explicitly commented or easy to detect without a deep analysis or testing of the code. If you wish for a deeper code review, that would require a detailed analysis of the code logic and execution, including potentially running the code in question.\n",
"What is the name of the class of agents in the file I gave you? When you answer a question based on a file, always report your confidence in the answer as a percentage.\n",
"The class of agents defined in the file is called `ConversableAgent`. This class is for generic conversable agents which can be configured as an assistant or user proxy and has several methods for customizing its behavior.\n",
"Determining the presence of bugs in a file requires detailed code analysis which includes understanding the code logic, testing, and reviewing by an expert. Due to the limitations of the tools available, I cannot definitively check for logical bugs or runtime errors in the code. My search in the available portion of the file has not shown explicit instances of the term \"bug,\" but this does not guarantee that the code is bug-free.\n",
"\n",
"There are several assertions in the code, which suggests that the code includes certain checks to ensure expected behaviors or values. However, the mere presence of assertions does not necessarily indicate that there are bugs; rather, it might indicate robustness as the developer probably has placed checks to prevent the code from running under incorrect assumptions.\n",
"\n",
"Based on the information provided and without running or testing the code, my confidence level in stating the presence or absence of bugs cannot be more than a hypothesis. Therefore, my confidence in saying there are no evident bugs based on the static analysis of the code is about 50%, which essentially means that without additional testing or code review, it is unclear if bugs exist.\n",
"\n",
"If you can provide more specific criteria for what should be considered a bug or more information about the kind of issues you are expecting, I could potentially perform a more targeted search.\n",
" message=\"What is the name of the class of agents in the file I gave you? When you answer a question based on a file, always report your confidence in the answer as a percentage.\",\n",
"Finally, let's clear the chat history to see whether the assistant can remember to report its confidence in a new chat without having to be told again."
"The name of the class of agents in the file is `ConversableAgent` [0]. My confidence in this answer is 100%.\n",
" super().__init__(name)pChatManager) to decide when to call upon this agent. (Default: system_message)reply is generated.ecution. Default to 1. If set to 'auto', it will scan backwards through all messages arriving since the agent last spoke (typically this is the last time execution was attempted).\n",
"Based on the search results I obtained, there are no explicit mentions of bugs in the source file provided. Typically, for bugs to be identified in a source file, one would need to run the code and observe its behavior or have a detailed inspection report that specifies identified bugs.\n",
"\n",
"Since we are not able to run actual code or have such a report, the only way to indirectly infer potential issues would be to look for common error handling patterns or mentions of unexpected behavior in comments, neither of which seems to be present in the search results.\n",
"\n",
"Thus, at this stage, I cannot confirm the presence of any bugs with confidence. If you would like, I can attempt a more thorough search by examining more of the file content or look for specific programming concepts that can sometimes be associated with buggy behavior (like exception handling, for instance).\n",
"\n",
"My confidence in stating that there are no explicitly mentioned bugs in the parts of the file I have searched is high, roughly around 90%. However, my confidence in stating that the file is bug-free overall cannot be as high due to the limitations of this method of analysis.\n",
"All OpenAI Assistants can be created, viewed and deleted manually through OpenAI's [website](https://platform.openai.com/assistants). They can also be deleted through the `GPTAssistantAgent` instance."
