"FLAML offers an experimental feature of interactive LLM agents, which can be used to solve various tasks with human or automatic feedback, including tasks that require using tools via code.\n",
"In this notebook, we demonstrate how to use multiple agents to work together and accomplish a task which requires finding info from the web and coding. `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. We further create a planning agent for the assistant agent to consult. The planning agent is a variation of the `AssistantAgent`.\n",
"* The [`config_list_openai_aoai`](https://microsoft.github.io/FLAML/docs/reference/autogen/oai/openai_utils#config_list_openai_aoai) function tries to create a list of configurations using Azure OpenAI endpoints and OpenAI endpoints. \n",
"\n",
"It assumes the api keys and api bases are stored in the corresponding environment variables or local txt files:\n",
"\n",
"- OpenAI API key: os.environ[\"OPENAI_API_KEY\"] or `openai_api_key_file=\"key_openai.txt\"`.\n",
"- Azure OpenAI API key: os.environ[\"AZURE_OPENAI_API_KEY\"] or `aoai_api_key_file=\"key_aoai.txt\"`. Multiple keys can be stored, one per line.\n",
"- Azure OpenAI API base: os.environ[\"AZURE_OPENAI_API_BASE\"] or `aoai_api_base_file=\"base_aoai.txt\"`. Multiple bases can be stored, one per line.\n",
"It's OK to have only the OpenAI API key, or only the Azure OpenAI API key + base. If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n"
"We construct the planning agent named \"planner\" and a user proxy agent for the planner named \"planner_user\". We specify `human_input_mode` as \"TERMINATE\" in the user proxy agent, which will never ask for human feedback. We define `ask_planner` function to send a message to planner and return the suggestion from the planner."
" system_message=\"You are a helpful AI assistant. You suggest coding and reasoning steps for another AI assistant to accomplish a task. Do not suggest concrete code. For any action beyond writing code or reasoning, convert it to a step which can be implemented by writing code. For example, the action of browsing the web can be implemented by writing code which reads and prints the content of a web page. Finally, inspect the execution result. If the plan is not good, suggest a better plan. If the execution is wrong, analyze the error and suggest a fix.\"\n",
"We construct the assistant agent and the user proxy agent. We specify `human_input_mode` as \"TERMINATE\" in the user proxy agent, which will ask for feedback when it receives a \"TERMINATE\" signal from the assistant agent. We set the `functions` in `AssistantAgent` and `function_map` in UserProxyAgent to use the created `ask_planner` function."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# create an AssistantAgent instance named \"assistant\"\n",
"assistant = AssistantAgent(\n",
" name=\"assistant\",\n",
" request_timeout=600,\n",
" seed=42,\n",
" # Excluding azure openai endpoints from the config list.\n",
" # Change to `exclude=\"openai\"` to exclude openai endpoints, or remove the `exclude` argument to include both.\n",
" \"description\": \"question to ask planner. Make sure the question include enough context, such as the code and the execution result. The planner does not know the conversation between you and the user, unless you share the conversation with the planner.\",\n",
"We invoke the `receive()` method of the coding agent to start the conversation. When you run the cell below, you will be prompted to provide feedback after the coding agent sends a \"TERMINATE\" signal in the end of the message. If you don't provide any feedback (by pressing Enter directly), the conversation will finish. Before the \"TERMINATE\" signal, the user proxy agent will try to execute the code suggested by the coding agent on behalf of the user."
"To suggest a fix to an open good first issue of FLAML, we first need to fetch the list of open issues labeled as \"good first issue\" from the FLAML GitHub repository. We can do this using the GitHub API. Here is a Python script that fetches and prints the list of such issues.\n",
"Please save this script as `fetch_issues.py` and run it. It will print the list of open issues labeled as \"good first issue\" in the FLAML repository.\n",
" \"message\": \"Here are the open good first issues of FLAML: \\nIssue #1120: use_label_encoder warning with xgboost\\nIssue #1099: Do not print data split method when user provides X_val in AutoML\\nIssue #1034: Use appropriate wait time for retry based on the error message.\\nIssue #1029: Issues with Adding Custom APIs in Auto Generation\\nIssue #981: Running flaml[tune] using \\\"-O\\\" flag for python interpreter (optimization - disables assertions) crashes\\nIssue #903: Conditional parameter flow2 crash\\nIssue #884: indentation space\\nIssue #882: Check if openml version is required\\nIssue #834: Adjust the indent\\nIssue #821: pip install flaml FAIL\\nIssue #807: Isolate the ensemble part and expose it to users\\nIssue #805: how to pass categorical features names or indices to learner\\nIssue #785: Flaml/LightGBM - Shouldn't I found better/faster or equal results from FLAML than direct LightGBM?\\nIssue #764: Add an announcement of the discord channel\\nIssue #748: Documentation about small budget\\nIssue #737: Make zero-shot automl more discoverable\\nIssue #509: New HCrystalBall release\\nIssue #429: samples about conversion to ONNX\\nIssue #413: support anomaly detection\\nIssue #304: CatBoost Fails with Keyword 'groups'\\n\\nPlease suggest a fix for one of these issues.\"\n",
"To provide appropriate suggestions for fixes, we first need to isolate a problem, understand it, and then propose a solution. Let's consider this issue as an example: \n",
"1. Understand the Source of the Problem: Study the part of the code where the issue originates. From the issue title, it seems like in the FLAML AutoML component, a data split method is being printed even when a user has provided a pre-split validation dataset.\n",
"2. Analyze the Code: If the user has already provided a validation data set (X_val), there would be no requirement for another data split. Therefore, it involves understanding why, nonetheless, a data split is printed. To solve this, the AI might have to go through the process where the software splits the data or loads the validation dataset, if available.\n",
"3. Propose a Solution: The solution could involve incorporating a conditional statement that checks if X_val is not null. If it isn't, then avoid printing the data split method. \n",
"4. Test the Solution: Once the code is implemented, it is important to test the solution by providing a validation dataset and ensuring that the data split information is no longer printed.\n",
"5. Inspect the result: If the proposed plan doesn't work as expected, analyze the issue and try to develop a better plan. If the execution is wrong, try to understand the error and adapt the approach accordingly.\n",
"To provide appropriate suggestions for fixes, we first need to isolate a problem, understand it, and then propose a solution. Let's consider this issue as an example: \n",
"1. Understand the Source of the Problem: Study the part of the code where the issue originates. From the issue title, it seems like in the FLAML AutoML component, a data split method is being printed even when a user has provided a pre-split validation dataset.\n",
"2. Analyze the Code: If the user has already provided a validation data set (X_val), there would be no requirement for another data split. Therefore, it involves understanding why, nonetheless, a data split is printed. To solve this, the AI might have to go through the process where the software splits the data or loads the validation dataset, if available.\n",
"3. Propose a Solution: The solution could involve incorporating a conditional statement that checks if X_val is not null. If it isn't, then avoid printing the data split method. \n",
"4. Test the Solution: Once the code is implemented, it is important to test the solution by providing a validation dataset and ensuring that the data split information is no longer printed.\n",
"5. Inspect the result: If the proposed plan doesn't work as expected, analyze the issue and try to develop a better plan. If the execution is wrong, try to understand the error and adapt the approach accordingly.\n",
"Based on the plan, here is a Python script that you can use to fix the issue #1099. This script assumes that the data split method is printed in a function called `split_data` in a file called `automl.py` (you need to replace these with the actual function and file names). \n",
"Please replace the function name and file name with the actual ones, save this script as `fix_issue_1099.py`, and run it. This script will fix the issue by not printing the data split method when validation data is provided.\n",
"***** Suggested function Call: ask_planner *****\n",
"Arguments: \n",
"{\n",
" \"message\": \"The user has executed the suggested code for fixing issue #1099 in FLAML. The code executed successfully without any output. Please suggest the next steps.\"\n",
"The user has executed the suggested code for fixing issue #1099 in FLAML. The code executed successfully without any output. Please suggest the next steps.\n",
"1. Code Review: Ensure the code is conforming to the overall structure, style, and standards of the FLAML project. Check if the changes are minimal, only related to the specific issue, and not affecting other parts of the codebase. \n",
" A. Provide a test when X_val is not provided. The code should behave in the same way as before your changes, including printing data split information.\n",
"4. Documentation: If changes made to the code are substantial or might affect the way users interact with FLAML, properly document these changes. Make sure to clarify that if 'X_val' is provided, the data split method will not be printed.\n",
"5. Submit the Code: When you're sure the changes fix the issue and don't introduce new ones, it would be the time to submit your code for review. Depending on the project guidelines, you may need to fork the repository, commit your changes, push the commit(s) to your fork, and then open a pull request.\n",
"6. Wait for Feedback: Once the code is submitted, wait for the maintainers' feedback. They might suggest some changes, enhancements, or approve the changes right away. Keep an open communication channel and respond to the comments, making the suggested changes. You might need to go through a few iterations of this process until the code is accepted and the issue is closed.\n",
"1. Code Review: Ensure the code is conforming to the overall structure, style, and standards of the FLAML project. Check if the changes are minimal, only related to the specific issue, and not affecting other parts of the codebase. \n",
" A. Provide a test when X_val is not provided. The code should behave in the same way as before your changes, including printing data split information.\n",
"4. Documentation: If changes made to the code are substantial or might affect the way users interact with FLAML, properly document these changes. Make sure to clarify that if 'X_val' is provided, the data split method will not be printed.\n",
"5. Submit the Code: When you're sure the changes fix the issue and don't introduce new ones, it would be the time to submit your code for review. Depending on the project guidelines, you may need to fork the repository, commit your changes, push the commit(s) to your fork, and then open a pull request.\n",
"6. Wait for Feedback: Once the code is submitted, wait for the maintainers' feedback. They might suggest some changes, enhancements, or approve the changes right away. Keep an open communication channel and respond to the comments, making the suggested changes. You might need to go through a few iterations of this process until the code is accepted and the issue is closed.\n",
"When the assistant needs to consult the planner, it suggests a function call to `ask_planner`. When this happens, a line like the following will be displayed:\n",
"\n",
"***** Suggested function Call: ask_planner *****\n"
