"AutoGen offers conversable agents powered by LLM, tool, or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"In this notebook, we introduce a new class, `AgentBuilder`, to help user build an automatic task solving process powered by multi-agent system. Specifically, in `build()`, we prompt a LLM to create multiple participant agent and initialize a group chat, and specify whether this task need programming to solve. AgentBuilder also support open-source LLMs by [vLLM](https://docs.vllm.ai/en/latest/index.html) and [Fastchat](https://github.com/lm-sys/FastChat). Check the supported model list [here](https://docs.vllm.ai/en/latest/models/supported_models.html)."
]
},
{
"cell_type": "markdown",
"id": "ec78dda8e3826d8a",
"metadata": {
"collapsed": false
},
"source": [
"## Requirement\n",
"\n",
"AutoBuild require `pyautogen[autobuild]`, which can be installed by the following command:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e8e9ae50658be975",
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%pip install pyautogen[autobuild]"
]
},
{
"cell_type": "markdown",
"id": "7d0e63ab3604bdb9",
"metadata": {
"collapsed": false
},
"source": [
"## Step 1: prepare configuration and some useful functions\n",
"Prepare a `config_file_or_env` for assistant agent to limit the choice of LLM you want to use in this task. This config can be a path of json file or a name of environment variable. A `default_llm_config` is also required for initialize the specific config of LLMs like seed, temperature, etc..."
"Create a `AgentBuilder` with the specified `config_path_or_env`. AgentBuilder will use `gpt-4` in default to complete the whole process, you can specify the `builder_model` and `agent_model` to other OpenAI model to match your task. \n",
"You can also specify an open-source LLM supporting by vLLM and FastChat, see blog for more details."
"building_task = \"Generate some agents that can find papers on arxiv by programming and analyzing them in specific domains related to computer science and medical science.\""
]
},
{
"cell_type": "markdown",
"id": "5782dd5ecb6c217a",
"metadata": {
"collapsed": false
},
"source": [
"## Step 4: build group chat agents\n",
"Use `build()` to let build manager (the specified `builder_model`) complete the group chat agents generation. If you think coding is necessary in your task, you can use `coding=True` to add a user proxy (an automatic code interpreter) into the agent list, like: \n",
"There is no python code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
"To find a recent paper about GPT-4 on arXiv and explore its potential applications in software, we can utilize the arXiv API to search for papers related to \"GPT-4\". I can write a Python script to fetch this information. Let's proceed with that.\n",
"This script will fetch the titles and summaries of papers related to GPT-4 from arXiv. We can then analyze these summaries to identify potential applications in software. Shall I proceed to execute this script?\n",
"The search results from arXiv provide a diverse range of papers discussing the capabilities and applications of GPT-4. Here are some potential applications in software based on the summaries:\n",
"\n",
"1. **Dementia Diagnosis**: The first paper discusses the use of GPT-4 in dementia diagnosis, comparing its performance with traditional AI tools. Although it currently does not surpass traditional methods, it shows potential for future advancements in medical diagnostics.\n",
"2. **Logical Reasoning**: The third paper evaluates GPT-4's performance on logical reasoning tasks. It highlights that while GPT-4 shows improvements over previous models, it still struggles with out-of-distribution datasets. This suggests applications in enhancing reasoning capabilities in software systems that require robust decision-making.\n",
"3. **Data Analysis**: The paper titled \"Is GPT-4 a Good Data Analyst?\" explores GPT-4's capabilities in performing end-to-end data analysis. This indicates potential applications in software tools for data analytics, where GPT-4 could assist or augment human data analysts.\n",
"4. **Graph Neural Architecture Search**: The integration of GPT-4 in designing graph neural networks, as discussed in the \"Graph Neural Architecture Search with GPT-4\" paper, showcases its application in automating and optimizing the design of complex network architectures in software.\n",
"5. **Math Word Problems**: The paper on solving challenging math word problems using GPT-4's code interpreter suggests applications in educational software, particularly in developing tools that assist in learning and solving mathematical problems.\n",
"These applications demonstrate GPT-4's potential to enhance various aspects of software, from improving diagnostic tools in healthcare to optimizing data analysis and network design in technical fields.\n",
"The applications outlined by the AI_ComputerScience_Expert indeed highlight the versatility and potential of GPT-4 in various software domains. To further validate these applications, we could consider setting up experiments or simulations that specifically test GPT-4's performance in these areas. For instance, in the context of dementia diagnosis, we could simulate a diagnostic process using GPT-4 and compare its accuracy and efficiency against traditional AI tools. Similarly, for data analysis and graph neural architecture search, we could benchmark GPT-4 against current state-of-the-art methods to quantitatively assess its improvements or shortcomings.\n",
"These practical evaluations would provide a more concrete understanding of how GPT-4 can be integrated into software solutions and its potential impact on improving functionalities and user experiences. If needed, I can assist in designing these experiments or simulations to ensure they are robust and provide meaningful insights.\n",
"Absolutely, setting up experiments or simulations to test GPT-4's performance in specific applications would be a crucial step in validating its practical utility and integration into software solutions. For the dementia diagnosis application, we could use a dataset of clinical cases to evaluate the model's diagnostic accuracy and compare it with traditional AI systems. This would involve not only accuracy but also examining aspects like false positives and negatives, which are critical in medical diagnostics.\n",
"For data analysis, we could design a set of tasks that mimic real-world data analysis scenarios. These tasks could include data cleaning, exploration, visualization, and predictive modeling. GPT-4's performance can be evaluated based on its accuracy, efficiency, and the insights it generates compared to human data analysts or other AI tools.\n",
"In the case of graph neural architecture search, we could use standard datasets and benchmarks in the field to test the effectiveness of the architectures designed by GPT-4. Metrics such as the time taken to design the architecture, performance of the designed network on test data, and comparison with architectures designed by human experts or other automated systems would be valuable.\n",
"These experiments would not only help in understanding GPT-4's capabilities but also in identifying areas where it might need further improvement. If you need assistance with the statistical analysis or the setup of these experiments, I can contribute with my expertise in programming and data analysis to ensure that the experiments are conducted efficiently and the results are analyzed correctly.\n",
"The proposed experimental setups by the AI_ComputerScience_Expert are well-thought-out and would indeed provide valuable insights into GPT-4's capabilities across different domains. To support these experiments, I can contribute by developing data mining scripts that efficiently gather and preprocess the necessary data from various sources. For instance, for the dementia diagnosis application, we can mine patient data, symptoms, and diagnostic results to create a comprehensive dataset for testing GPT-4.\n",
"\n",
"Additionally, for the data analysis tasks, I can help automate the process of data cleaning and preparation, which is crucial for ensuring the accuracy of the results. This involves handling missing data, normalizing datasets, and encoding categorical variables, which are common tasks in data analysis that can be automated using Python scripts.\n",
"\n",
"For the graph neural architecture search, I can assist in mining existing literature and datasets to find relevant benchmarks and performance metrics that can be used to evaluate the architectures designed by GPT-4. This would involve not only retrieving data but also analyzing it to extract meaningful patterns and insights that can guide the experimental setup.\n",
"\n",
"By combining our expertise in AI, data mining, and bioinformatics, we can ensure that the experiments are not only well-designed but also supported by robust data handling and analysis methodologies. This collaborative approach will enhance the reliability and validity of the findings, providing a solid foundation for assessing GPT-4's practical applications in software.\n",
"The collaborative approach outlined by the DataMining_Expert is essential for the success of these experiments. By leveraging our combined expertise in AI, data mining, and bioinformatics, we can ensure that the experimental setups are robust and the data used is of high quality and relevance.\n",
"\n",
"For the dementia diagnosis application, integrating comprehensive patient data and diagnostic results will allow us to simulate realistic scenarios where GPT-4's diagnostic capabilities can be rigorously tested. This will help in assessing not only its accuracy but also its reliability and potential as a supportive tool in medical diagnostics.\n",
"\n",
"In the data analysis tasks, automating the data preparation process will significantly enhance the efficiency of the experiments. It will allow us to focus on evaluating GPT-4's performance in generating insights and making predictions, which are critical aspects of data analysis.\n",
"\n",
"For the graph neural architecture search, having access to relevant benchmarks and performance metrics is crucial. The data mining efforts to gather and analyze existing literature and datasets will provide a solid basis for evaluating the effectiveness of the architectures designed by GPT-4.\n",
"\n",
"Overall, this collaborative effort will enable us to conduct comprehensive and meaningful experiments that will provide insights into GPT-4's capabilities and limitations. This will not only contribute to the academic and scientific community but also guide future developments and applications of AI in software solutions. If there are no further inputs or adjustments needed, we can proceed with the planning and execution of these experiments.\n",
"## Step 6 (Optional): clear all agents and prepare for the next task\n",
"You can clear all agents generated in this task by the following code if your task is completed or the next task is largely different from the current task. If the agent's backbone is an open-source LLM, this process will also shut down the endpoint server. If necessary, you can use `recycle_endpoint=False` to retain the previous open-source LLMs' endpoint server."
"You can save all necessary information of the built group chat agents. Here is a case for those agents generated in the above task:\n",
"```json\n",
"{\n",
" \"building_task\": \"Generate some agents that can find papers on arxiv by programming and analyzing them in specific domains related to computer science and medical science.\",\n",
" \"agent_configs\": [\n",
" {\n",
" \"name\": \"ArXiv_Data_Scraper_Developer\",\n",
" \"model\": \"gpt-4-1106-preview\",\n",
" \"system_message\": \"You are now in a group chat. You need to complete a task with other participants. As an ArXiv_Data_Scraper_Developer, your focus is to create and refine tools capable of intelligent search and data extraction from arXiv, honing in on topics within the realms of computer science and medical science. Utilize your proficiency in Python programming to design scripts that navigate, query, and parse information from the platform, generating valuable insights and datasets for analysis. \\n\\nDuring your mission, it\\u2019s not just about formulating queries; your role encompasses the optimization and precision of the data retrieval process, ensuring relevance and accuracy of the information extracted. If you encounter an issue with a script or a discrepancy in the expected output, you are encouraged to troubleshoot and offer revisions to the code you find in the group chat.\\n\\nWhen you reach a point where the existing codebase does not fulfill task requirements or if the operation of provided code is unclear, you should ask for help from the group chat manager. They will facilitate your advancement by providing guidance or appointing another participant to assist you. Your ability to adapt and enhance scripts based on peer feedback is critical, as the dynamic nature of data scraping demands ongoing refinement of techniques and approaches.\\n\\nWrap up your participation by confirming the user's need has been satisfied with the data scraping solutions you've provided. Indicate the completion of your task by replying \\\"TERMINATE\\\" in the group chat.\",\n",
" \"description\": \"ArXiv_Data_Scraper_Developer is a specialized software development role requiring proficiency in Python, including familiarity with web scraping libraries such as BeautifulSoup or Scrapy, and a solid understanding of APIs and data parsing. They must possess the ability to identify and correct errors in existing scripts and confidently engage in technical discussions to improve data retrieval processes. The role also involves a critical eye for troubleshooting and optimizing code to ensure efficient data extraction from the ArXiv platform for research and analysis purposes.\"\n",
" \"system_message\": \"You are now in a group chat. You need to complete a task with other participants. As a Computer Science Research Analyst, your objective is to utilize your analytical capabilities to identify and examine scholarly articles on arXiv, focusing on areas bridging computer science and medical science. Employ Python for automation where appropriate and leverage your expertise in the subject matter to draw insights from the research.\\n\\nEnsure that the information is acquired systematically; tap into online databases, interpret data sets, and perform literature reviews to pinpoint relevant findings. Should you encounter a complex problem or if you find your progress stalled, feel free to question the existing approaches discussed in the chat or contribute an improved method or analysis.\\n\\nIf the task proves to be beyond your current means or if you face uncertainty at any stage, seek assistance from the group chat manager. The manager is available to provide guidance or to involve another expert if necessary to move forward effectively.\\n\\nYour contributions are crucial, and it is important to communicate your findings and conclusions clearly. Once you believe the task is complete and the group's need has been satisfied, please affirm the completion by replying \\\"TERMINATE\\\".\",\n",
" \"description\": \"Computer_Science_Research_Analyst is a role requiring strong analytical skills, a deep understanding of computer science concepts, and proficiency in Python for data analysis and automation. This position should have the ability to critically assess the validity of information, challenge assumptions, and provide evidence-based corrections or alternatives. They should also have excellent communication skills to articulate their findings and suggestions effectively within the group chat.\"\n",
" \"system_message\": \"You are now in a group chat. You need to complete a task with other participants. As a Medical_Science_Research_Analyst, your function is to harness your analytical strengths and understanding of medical research to source and evaluate pertinent papers from the arXiv database, focusing on the intersection of computer science and medical science. Utilize your Python programming skills to automate data retrieval and analysis tasks. Engage in systematic data mining to extract relevant content, then apply your analytical expertise to interpret the findings qualitatively. \\n\\nWhen there is a requirement to gather information, employ Python scripts to automate the aggregation process. This could include scraping web data, retrieving and processing documents, and performing content analyses. When these scripts produce outputs, use your subject matter expertise to evaluate the results. \\n\\nProgress through your task step by step. When an explicit plan is absent, present a structured outline of your intended methodology. Clarify which segments of the task are handled through automation, and which necessitate your interpretative skills. \\n\\nIn the event code is utilized, the script type must be specified. You are expected to execute the scripts provided without making changes. Scripts are to be complete and functionally standalone. Should you encounter an error upon execution, critically review the output, and if needed, present a revised script for the task at hand. \\n\\nFor tasks that require saving and executing scripts, indicate the intended filename at the beginning of the script. \\n\\nMaintain clear communication of the results by harnessing the 'print' function where applicable. If an error arises or a task remains unsolved after successful code execution, regroup to collect additional information, reassess your approach, and explore alternative strategies. \\n\\nUpon reaching a conclusion, substantiate your findings with credible evidence where possible.\\n\\nConclude your participation by confirming the task's completion with a \\\"TERMINATE\\\" response.\\n\\nShould uncertainty arise at any point, seek guidance from the group chat manager for further directives or reassignment of the task.\",\n",
" \"description\": \"The Medical Science Research Analyst is a professionally trained individual with strong analytical skills, specializing in interpreting and evaluating scientific research within the medical field. They should possess expertise in data analysis, likely with proficiency in Python for analyzing datasets, and have the ability to critically assess the validity and relevance of previous messages or findings relayed in the group chat. This role requires a solid foundation in medical knowledge to provide accurate and evidence-based corrections or insights.\"\n",
" },\n",
" {\n",
" \"name\": \"Data_Analysis_Engineer\",\n",
" \"model\": \"gpt-4-1106-preview\",\n",
" \"system_message\": \"You are now in a group chat. You need to complete a task with other participants. As a Data Analysis Engineer, your role involves leveraging your analytical skills to gather, process, and analyze large datasets. You will employ various data analysis techniques and tools, particularly Python for scripting, to extract insights from the data related to computer science and medical science domains on arxiv.\\n\\nIn scenarios where information needs to be collected or analyzed, you will develop Python scripts to automate the data retrieval and processing tasks. For example, you may write scripts to scrape the arXiv website, parse metadata of research papers, filter content based on specific criteria, and perform statistical analysis or data visualization. \\n\\nYour workflow will include the following steps:\\n\\n1. Use your Python coding abilities to design scripts for data extraction and analysis. This can involve browsing or searching the web, downloading and reading files, or printing the content of web pages or files relevant to the given domains.\\n2. After gathering the necessary data, apply your data analysis expertise to derive meaningful insights or patterns present in the data. This should be done methodically, making the most of your Python skills for data manipulation and interpretation.\\n3. Communicate your findings clearly to the group chat. Ensure the results are straightforward for others to understand and act upon.\\n4. If any issues arise from executing the code, such as lack of output or unexpected results, you can question the previous messages or code in the group chat and attempt to provide a corrected script or analysis.\\n5. When uncertain or facing a complex problem that you cannot solve alone, ask for assistance from the group chat manager. They can either provide guidance or assign another participant to help you.\\n\\nOnce you believe the task is completed satisfactorily, and you have fulfilled the user's need, respond with \\\"TERMINATE\\\" to signify the end of your contribution to the task. Remember, while technical proficiency in Python is essential for this role, the ability to work collaboratively within the group chat, communicate effectively, and adapt to challenges is equally important.\",\n",
" \"description\": \"Data_Analysis_Engineer is a professional adept in collecting, analyzing, and interpreting large datasets, using statistical tools and machine learning techniques to provide actionable insights. They should possess strong Python coding skills for data manipulation and analysis, an understanding of database management, as well as the ability to communicate complex results effectively to non-technical stakeholders. This position should be allowed to speak when data-driven clarity is needed or when existing analyses or methodologies are called into question.\"\n",
" \"system_message\": \"You are now in a group chat. You need to complete a task with other participants. As an ML_Paper_Summarization_Specialist, your role entails leveraging machine learning techniques to extract and analyze academic papers from arXiv, focusing on domains that intersect computer science and medical science. Utilize your expertise in natural language processing and data analysis to identify relevant papers, extract key insights, and generate summaries that accurately reflect the advancements and findings within those papers.\\n\\nYou are expected to apply your deep understanding of machine learning algorithms, data mining, and information retrieval to construct models and systems that can efficiently process and interpret scientific literature.\\n\\nIf you encounter any challenges in accessing papers, parsing content, or algorithmic processing, you may seek assistance by presenting your issue to the group chat. Should there be a disagreement regarding the efficacy of a method or the accuracy of a summarization, you are encouraged to critically evaluate previous messages or outputs and offer improved solutions to enhance the group's task performance.\\n\\nShould confusion arise during the task, rather than relying on coding scripts, please request guidance from the group chat manager, and allow them to facilitate the necessary support by inviting another participant who can aid in overcoming the current obstacle.\\n\\nRemember, your primary duty is to synthesize complex academic content into concise, accessible summaries that will serve as a valuable resource for researchers and professionals seeking to stay abreast of the latest developments in their respective fields. \\n\\nOnce you believe your task is completed and the summaries provided meet the necessary standards of accuracy and comprehensiveness, reply \\\"TERMINATE\\\" to signal the end of your contribution to the group's task.\",\n",
" \"description\": \"The ML_Paper_Summarization_Specialist is a professional adept in machine learning concepts and current research trends, with strong analytical skills to critically evaluate information, synthesizing knowledge from academic papers into digestible summaries. This specialist should be proficient in Python for text processing and have the ability to provide constructive feedback on technical discussions, guide effective implementation, and correct misconceptions or errors related to machine learning theory and practice in the chat. They should be a reliable resource for clarifying complex information and ensuring accurate application of machine learning techniques within the group chat context.\"\n",
" }\n",
" ],\n",
" \"coding\": true,\n",
" \"default_llm_config\": {\n",
" \"temperature\": 0\n",
" },\n",
" \"code_execution_config\": {\n",
" \"work_dir\": \"groupchat\",\n",
" \"use_docker\": false,\n",
" \"timeout\": 60,\n",
" \"last_n_messages\": 2\n",
" }\n",
"}\n",
"```\n",
"These information will be saved in JSON format. You can provide a specific filename, otherwise, AgentBuilder will save config to the current path with a generated filename 'save_config_TASK_MD5.json'."
"After that, you can load the saved config and skip the building process. AgentBuilder will create agents with those information without prompting the builder manager."
"There is no python code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
"To assist with the request on finding a recent paper about LLaVA on arXiv and exploring its potential applications in computer vision, I will perform a search on arXiv for the most recent papers related to LLaVA and analyze any mentioned applications in the field of computer vision.\n",
"\n",
"Let's start by searching for the most recent papers on this topic. I will write a Python script to query the arXiv API for papers related to \"LLaVA\" and \"computer vision\". Let's proceed with that.\n",
"This script will retrieve the top 5 most relevant papers from arXiv that mention both LLaVA and computer vision. We can analyze these papers to identify potential applications in computer vision.\n",
"The search has returned several interesting papers related to LLaVA and its applications in computer vision. Here are the summaries of the top papers:\n",
" - **Summary:** This paper introduces LLaVA-Interactive, a multimodal human-AI interaction system capable of multi-turn dialogues with human users by taking multiple inputs including images. It demonstrates applications in image chat, segmentation, generation, and editing.\n",
"2. **LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents**\n",
" - **Authors:** Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, Chunyuan Li\n",
" - **Published:** 2023-11-09\n",
" - **Summary:** LLaVA-Plus expands the capabilities of large multimodal models, maintaining a skill repository of pre-trained models for various tasks including visual tasks.\n",
" - **Summary:** This paper presents a novel approach to enhance image-to-image generation by leveraging the multimodal capabilities of LLaVA, focusing on improving visual content generation.\n",
"These papers highlight the versatility of LLaVA in handling various aspects of computer vision, such as image segmentation, generation, and editing. The applications are quite broad, impacting areas like multimodal human-AI interaction, enhancing image-to-image generation, and creating multimodal agents capable of performing visual tasks. These capabilities are crucial for advancing the field of computer vision, providing tools that can better understand and interact with visual data in a more human-like manner.\n",
"The summaries provided indeed highlight the potential applications of LLaVA in computer vision. The capabilities of LLaVA in handling tasks such as image segmentation, generation, and editing are particularly noteworthy. These functionalities can be extremely useful in various practical applications, such as enhancing visual content for media, improving interfaces for human-computer interaction, and even aiding in educational tools where visual aids are crucial.\n",
"Given the detailed information from the papers, it seems that LLaVA's integration into computer vision tasks could lead to significant advancements in how machines process and understand visual information, making them more efficient and effective in tasks that require a deep understanding of visual contexts.\n",
"It would be beneficial to further explore how these capabilities can be integrated into existing systems or used to develop new applications in fields that heavily rely on visual data.\n",
