From a4b9297719e19c27c8f7043b834db2847d5e9d2f Mon Sep 17 00:00:00 2001 From: Chi Wang Date: Sat, 16 Sep 2023 16:34:16 +0000 Subject: [PATCH] cleanup --- notebook/autogen_agentchat_MathChat.ipynb | 14 +- notebook/autogen_agentchat_RetrieveChat.ipynb | 1333 +---------------- ...at_auto_feedback_from_code_execution.ipynb | 16 +- notebook/autogen_agentchat_chess.ipynb | 12 +- .../autogen_agentchat_function_call.ipynb | 14 +- notebook/autogen_agentchat_groupchat.ipynb | 12 +- ...autogen_agentchat_groupchat_research.ipynb | 12 +- .../autogen_agentchat_groupchat_vis.ipynb | 12 +- .../autogen_agentchat_human_feedback.ipynb | 12 +- notebook/autogen_agentchat_planning.ipynb | 12 +- notebook/autogen_agentchat_stream.ipynb | 14 +- notebook/autogen_agentchat_web_info.ipynb | 12 +- notebook/autogen_chatgpt_gpt4.ipynb | 278 +--- notebook/autogen_openai_completion.ipynb | 16 +- notebook/research/autogen_code.ipynb | 6 +- notebook/research/math_level5counting.ipynb | 2 +- test/agentchat/test_assistant_agent.py | 4 +- test/agentchat/test_async.py | 2 +- test/agentchat/test_conversable_agent.py | 2 +- test/agentchat/test_groupchat.py | 2 +- test/agentchat/test_math_user_proxy_agent.py | 6 +- test/agentchat/test_retrievechat.py | 16 +- test/oai/test_completion.py | 6 +- test/oai/test_utils.py | 2 +- test/test_code.py | 12 +- test/test_function_call.py | 8 +- .../blog/2023-04-21-LLM-tuning-math/index.mdx | 16 +- .../index.mdx | 10 +- website/blog/2023-06-28-MathChat/index.mdx | 4 +- website/blog/2023-07-14-Local-LLMs/index.mdx | 14 +- website/docs/Examples/AutoGen-AgentChat.md | 22 +- website/docs/Examples/AutoGen-OpenAI.md | 4 +- website/docs/Getting-Started.md | 10 +- website/docs/Use-Cases/agent_chat.md | 28 +- website/docs/Use-Cases/enhanced_inference.md | 10 +- 35 files changed, 205 insertions(+), 1750 deletions(-) diff --git a/notebook/autogen_agentchat_MathChat.ipynb b/notebook/autogen_agentchat_MathChat.ipynb index 480474a0e..29709716a 100644 --- a/notebook/autogen_agentchat_MathChat.ipynb +++ b/notebook/autogen_agentchat_MathChat.ipynb @@ -15,15 +15,15 @@ "source": [ "# Auto Generated Agent Chat: Using MathChat to Solve Math Problems\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", - "MathChat is an experimental convesational framework for math problem solving. In this notebook, we demonstrate how to use MathChat to solve math problems. MathChat uses the `AssistantAgent` and `MathUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `MathUserProxyAgent` implements a different auto reply mechanism corresponding to the MathChat prompts. You can find more details in the paper [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337) or the [blogpost](https://microsoft.github.io/FLAML/blog/2023/06/28/MathChat).\n", + "MathChat is an experimental convesational framework for math problem solving. In this notebook, we demonstrate how to use MathChat to solve math problems. MathChat uses the `AssistantAgent` and `MathUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `MathUserProxyAgent` implements a different auto reply mechanism corresponding to the MathChat prompts. You can find more details in the paper [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337) or the [blogpost](https://microsoft.github.io/autogen/blog/2023/06/28/MathChat).\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [mathchat] option.\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install the [mathchat] option.\n", "```bash\n", - "pip install flaml[mathchat]\n", + "pip install \"pyautogen[mathchat]\"\n", "```" ] }, @@ -33,7 +33,7 @@ "metadata": {}, "outputs": [], "source": [ - "# %pip install flaml[mathchat]~=2.0.0" + "# %pip install \"pyautogen[mathchat]~=0.1.0\"" ] }, { @@ -52,7 +52,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list = autogen.config_list_from_json(\n", " \"OAI_CONFIG_LIST\",\n", @@ -126,7 +126,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml.autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n", + "from autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n", "\n", "autogen.ChatCompletion.start_logging()\n", "\n", diff --git a/notebook/autogen_agentchat_RetrieveChat.ipynb b/notebook/autogen_agentchat_RetrieveChat.ipynb index 7c326658d..eaa24931e 100644 --- a/notebook/autogen_agentchat_RetrieveChat.ipynb +++ b/notebook/autogen_agentchat_RetrieveChat.ipynb @@ -1,5 +1,12 @@ { "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Open" + ] + }, { "attachments": {}, "cell_type": "markdown", @@ -8,10 +15,10 @@ "\n", "# Auto Generated Agent Chat: Using RetrieveChat for Retrieve Augmented Code Generation and Question Answering\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", - "RetrieveChat is a convesational system for retrieve augmented code generation and question answering. In this notebook, we demonstrate how to utilize RetrieveChat to generate code and answer questions based on customized documentations that are not present in the LLM's training dataset. RetrieveChat uses the `RetrieveAssistantAgent` and `RetrieveUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `RetrieveAssistantAgent` and `RetrieveUserProxyAgent` implement a different auto-reply mechanism corresponding to the RetrieveChat prompts.\n", + "RetrieveChat is a convesational system for retrieve augmented code generation and question answering. In this notebook, we demonstrate how to utilize RetrieveChat to generate code and answer questions based on customized documentations that are not present in the LLM's training dataset. RetrieveChat uses the `RetrieveAssistantAgent` and `RetrieveUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `RetrieveAssistantAgent` and `RetrieveUserProxyAgent` implement a different auto-reply mechanism corresponding to the RetrieveChat prompts.\n", "\n", "## Table of Contents\n", "We'll demonstrates five examples of using RetrieveChat for code generation and question answering:\n", @@ -29,9 +36,9 @@ "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [retrievechat] option.\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install the [retrievechat] option and flaml.\n", "```bash\n", - "pip install flaml[retrievechat]\n", + "pip install \"pyautogen[retrievechat]\" flaml\n", "```" ] }, @@ -41,7 +48,7 @@ "metadata": {}, "outputs": [], "source": [ - "# %pip install flaml[retrievechat]~=2.0.0" + "# %pip install \"pyautogen[retrievechat]~=0.1.0\" flaml" ] }, { @@ -68,7 +75,7 @@ } ], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list = autogen.config_list_from_json(\n", " env_or_file=\".config.local\",\n", @@ -141,8 +148,8 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml.autogen.agentchat.contrib.retrieve_assistant_agent import RetrieveAssistantAgent\n", - "from flaml.autogen.agentchat.contrib.retrieve_user_proxy_agent import RetrieveUserProxyAgent\n", + "from autogen.agentchat.contrib.retrieve_assistant_agent import RetrieveAssistantAgent\n", + "from autogen.agentchat.contrib.retrieve_user_proxy_agent import RetrieveUserProxyAgent\n", "import chromadb\n", "\n", "autogen.ChatCompletion.start_logging()\n", @@ -1098,1315 +1105,9 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "doc_ids: [['doc_36', 'doc_58', 'doc_40', 'doc_51', 'doc_4', 'doc_23', 'doc_52', 'doc_15', 'doc_14', 'doc_59', 'doc_2', 'doc_7', 'doc_29', 'doc_56', 'doc_30', 'doc_3', 'doc_55', 'doc_44', 'doc_20', 'doc_33']]\n", - "\u001b[32mAdding doc_id doc_36 to context.\u001b[0m\n", - "\u001b[32mAdding doc_id doc_58 to context.\u001b[0m\n", - "\u001b[32mAdding doc_id doc_40 to context.\u001b[0m\n", - "\u001b[32mAdding doc_id doc_51 to context.\u001b[0m\n", - "\u001b[33mragproxyagent\u001b[0m (to assistant):\n", - "\n", - "You're a retrieve augmented coding assistant. You answer user's questions based on your own knowledge and the\n", - "context provided by the user.\n", - "If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.\n", - "For code generation, you must obey the following rules:\n", - "Rule 1. You MUST NOT install any packages because all the packages needed are already installed.\n", - "Rule 2. You must follow the formats below to write your code:\n", - "```language\n", - "# your code\n", - "```\n", - "\n", - "User's question is: Who is the author of FLAML?\n", - "\n", - "Context is: \n", - "- `seed` - int or None, default=None | The random seed for hpo.\n", - "- `n_concurrent_trials` - [Experimental] int, default=1 | The number of\n", - " concurrent trials. When n_concurrent_trials > 1, flaml performes\n", - " [parallel tuning](../../Use-Cases/Task-Oriented-AutoML#parallel-tuning)\n", - " and installation of ray or spark is required: `pip install flaml[ray]`\n", - " or `pip install flaml[spark]`. Please check\n", - " [here](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)\n", - " for more details about installing Spark.\n", - "- `keep_search_state` - boolean, default=False | Whether to keep data needed\n", - " for model search after fit(). By default the state is deleted for\n", - " space saving.\n", - "- `preserve_checkpoint` - boolean, default=True | Whether to preserve the saved checkpoint\n", - " on disk when deleting automl. By default the checkpoint is preserved.\n", - "- `early_stop` - boolean, default=False | Whether to stop early if the\n", - " search is considered to converge.\n", - "- `force_cancel` - boolean, default=False | Whether to forcely cancel Spark jobs if the\n", - " search time exceeded the time budget.\n", - "- `append_log` - boolean, default=False | Whetehr to directly append the log\n", - " records to the input log file if it exists.\n", - "- `auto_augment` - boolean, default=True | Whether to automatically\n", - " augment rare classes.\n", - "- `min_sample_size` - int, default=MIN_SAMPLE_TRAIN | the minimal sample\n", - " size when sample=True.\n", - "- `use_ray` - boolean or dict.\n", - " If boolean: default=False | Whether to use ray to run the training\n", - " in separate processes. This can be used to prevent OOM for large\n", - " datasets, but will incur more overhead in time.\n", - " If dict: the dict contains the keywords arguments to be passed to\n", - " [ray.tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html).\n", - "- `use_spark` - boolean, default=False | Whether to use spark to run the training\n", - " in parallel spark jobs. This can be used to accelerate training on large models\n", - " and large datasets, but will incur more overhead in time and thus slow down\n", - " training in some cases. GPU training is not supported yet when use_spark is True.\n", - " For Spark clusters, by default, we will launch one trial per executor. However,\n", - " sometimes we want to launch more trials than the number of executors (e.g., local mode).\n", - " In this case, we can set the environment variable `FLAML_MAX_CONCURRENT` to override\n", - " the detected `num_executors`. The final number of concurrent trials will be the minimum\n", - " of `n_concurrent_trials` and `num_executors`.\n", - "- `free_mem_ratio` - float between 0 and 1, default=0. The free memory ratio to keep during training.\n", - "- `metric_constraints` - list, default=[] | The list of metric constraints.\n", - " Each element in this list is a 3-tuple, which shall be expressed\n", - " in the following format: the first element of the 3-tuple is the name of the\n", - " metric, the second element is the inequality sign chosen from \">=\" and \"<=\",\n", - " and the third element is the constraint value. E.g., `('val_loss', '<=', 0.1)`.\n", - " Note that all the metric names in metric_constraints need to be reported via\n", - " the metrics_to_log dictionary returned by a customized metric function.\n", - " The customized metric function shall be provided via the `metric` key word\n", - " argument of the fit() function or the automl constructor.\n", - " Find an example in the 4th constraint type in this [doc](../../Use-Cases/Task-Oriented-AutoML#constraint).\n", - " If `pred_time_limit` is provided as one of keyword arguments to fit() function or\n", - " the automl constructor, flaml will automatically (and under the hood)\n", - " add it as an additional element in the metric_constraints. Essentially 'pred_time_limit'\n", - " specifies a constraint about the prediction latency constraint in seconds.\n", - "- `custom_hp` - dict, default=None | The custom search space specified by user.\n", - " It is a nested dict with keys being the estimator names, and values being dicts\n", - " per estimator search space. In the per estimator search space dict,\n", - " the keys are the hyperparameter names, and values are dicts of info (\"domain\",\n", - " \"init_value\", and \"low_cost_init_value\") about the search space associated with\n", - " the hyperparameter (i.e., per hyperparameter search space dict). When custom_hp\n", - " is provided, the built-in search space which is also a nested dict of per estimator\n", - " search space dict, will be updated with custom_hp. Note that during this nested dict update,\n", - " the per hyperparameter search space dicts will be replaced (instead of updated) by the ones\n", - " provided in custom_hp. Note that the value for \"domain\" can either be a constant\n", - " or a sample.Domain object.\n", - " e.g.,\n", - " \n", - "```python\n", - "custom_hp = {\n", - " \"transformer_ms\": {\n", - " \"model_path\": {\n", - " \"domain\": \"albert-base-v2\",\n", - " },\n", - " \"learning_rate\": {\n", - " \"domain\": tune.choice([1e-4, 1e-5]),\n", - " }\n", - " }\n", - " }\n", - "```\n", - "- `skip_transform` - boolean, default=False | Whether to pre-process data prior to modeling.\n", - "- `fit_kwargs_by_estimator` - dict, default=None | The user specified keywords arguments, grouped by estimator name.\n", - " e.g.,\n", - " \n", - "```python\n", - "fit_kwargs_by_estimator = {\n", - " \"transformer\": {\n", - " \"output_dir\": \"test/data/output/\",\n", - " \"fp16\": False,\n", - " }\n", - "}\n", - "```\n", - "- `mlflow_logging` - boolean, default=True | Whether to log the training results to mlflow.\n", - " This requires mlflow to be installed and to have an active mlflow run.\n", - " FLAML will create nested runs.\n", - "\n", - "#### config\\_history\n", - "\n", - "```python\n", - "@property\n", - "def config_history() -> dict\n", - "```\n", - "\n", - "A dictionary of iter->(estimator, config, time),\n", - "storing the best estimator, config, and the time when the best\n", - "model is updated each time.\n", - "\n", - "#### model\n", - "\n", - "```python\n", - "@property\n", - "def model()\n", - "```\n", - "\n", - "An object with `predict()` and `predict_proba()` method (for\n", - "classification), storing the best trained model.\n", - "\n", - "#### best\\_model\\_for\\_estimator\n", - "\n", - "```python\n", - "def best_model_for_estimator(estimator_name: str)\n", - "```\n", - "\n", - "Return the best model found for a particular estimator.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `estimator_name` - a str of the estimator's name.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " An object storing the best model for estimator_name.\n", - " If `model_history` was set to False during fit(), then the returned model\n", - " is untrained unless estimator_name is the best estimator.\n", - " If `model_history` was set to True, then the returned model is trained.\n", - "\n", - "#### best\\_estimator\n", - "\n", - "```python\n", - "@property\n", - "def best_estimator()\n", - "```\n", - "\n", - "A string indicating the best estimator found.\n", - "\n", - "#### best\\_iteration\n", - "\n", - "```python\n", - "@property\n", - "def best_iteration()\n", - "```\n", - "\n", - "An integer of the iteration number where the best\n", - "config is found.\n", - "\n", - "#### best\\_config\n", - "\n", - "```python\n", - "@property\n", - "def best_config()\n", - "```\n", - "\n", - "A dictionary of the best configuration.\n", - "\n", - "#### best\\_config\\_per\\_estimator\n", - "\n", - "```python\n", - "@property\n", - "def best_config_per_estimator()\n", - "```\n", - "\n", - "A dictionary of all estimators' best configuration.\n", - "\n", - "#### best\\_loss\\_per\\_estimator\n", - "\n", - "```python\n", - "@property\n", - "def best_loss_per_estimator()\n", - "```\n", - "\n", - "A dictionary of all estimators' best loss.\n", - "\n", - "#### best\\_loss\n", - "\n", - "```python\n", - "@property\n", - "def best_loss()\n", - "```\n", - "\n", - "A float of the best loss found.\n", - "\n", - "#### best\\_result\n", - "\n", - "```python\n", - "@property\n", - "def best_result()\n", - "```\n", - "\n", - "Result dictionary for model trained with the best config.\n", - "\n", - "#### metrics\\_for\\_best\\_config\n", - "\n", - "```python\n", - "@property\n", - "def metrics_for_best_config()\n", - "```\n", - "\n", - "Returns a float of the best loss, and a dictionary of the auxiliary metrics to log\n", - "associated with the best config. These two objects correspond to the returned\n", - "objects by the customized metric function for the config with the best loss.\n", - "\n", - "#### best\\_config\\_train\\_time\n", - "---\n", - "sidebar_label: estimator\n", - "title: default.estimator\n", - "---\n", - "\n", - "#### flamlize\\_estimator\n", - "\n", - "```python\n", - "def flamlize_estimator(super_class, name: str, task: str, alternatives=None)\n", - "```\n", - "\n", - "Enhance an estimator class with flaml's data-dependent default hyperparameter settings.\n", - "\n", - "**Example**:\n", - "\n", - " \n", - "```python\n", - "import sklearn.ensemble as ensemble\n", - "RandomForestRegressor = flamlize_estimator(\n", - " ensemble.RandomForestRegressor, \"rf\", \"regression\"\n", - ")\n", - "```\n", - " \n", - "\n", - "**Arguments**:\n", - "\n", - "- `super_class` - an scikit-learn compatible estimator class.\n", - "- `name` - a str of the estimator's name.\n", - "- `task` - a str of the task type.\n", - "- `alternatives` - (Optional) a list for alternative estimator names. For example,\n", - " ```[(\"max_depth\", 0, \"xgboost\")]``` means if the \"max_depth\" is set to 0\n", - " in the constructor, then look for the learned defaults for estimator \"xgboost\".\n", - "\n", - "\n", - " \n", - "- `seed` - int or None, default=None | The random seed for hpo.\n", - "- `n_concurrent_trials` - [Experimental] int, default=1 | The number of\n", - " concurrent trials. When n_concurrent_trials > 1, flaml performes\n", - " [parallel tuning](../../Use-Cases/Task-Oriented-AutoML#parallel-tuning)\n", - " and installation of ray or spark is required: `pip install flaml[ray]`\n", - " or `pip install flaml[spark]`. Please check\n", - " [here](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)\n", - " for more details about installing Spark.\n", - "- `keep_search_state` - boolean, default=False | Whether to keep data needed\n", - " for model search after fit(). By default the state is deleted for\n", - " space saving.\n", - "- `preserve_checkpoint` - boolean, default=True | Whether to preserve the saved checkpoint\n", - " on disk when deleting automl. By default the checkpoint is preserved.\n", - "- `early_stop` - boolean, default=False | Whether to stop early if the\n", - " search is considered to converge.\n", - "- `force_cancel` - boolean, default=False | Whether to forcely cancel the PySpark job if overtime.\n", - "- `append_log` - boolean, default=False | Whetehr to directly append the log\n", - " records to the input log file if it exists.\n", - "- `auto_augment` - boolean, default=True | Whether to automatically\n", - " augment rare classes.\n", - "- `min_sample_size` - int, default=MIN_SAMPLE_TRAIN | the minimal sample\n", - " size when sample=True.\n", - "- `use_ray` - boolean or dict.\n", - " If boolean: default=False | Whether to use ray to run the training\n", - " in separate processes. This can be used to prevent OOM for large\n", - " datasets, but will incur more overhead in time.\n", - " If dict: the dict contains the keywords arguments to be passed to\n", - " [ray.tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html).\n", - "- `use_spark` - boolean, default=False | Whether to use spark to run the training\n", - " in parallel spark jobs. This can be used to accelerate training on large models\n", - " and large datasets, but will incur more overhead in time and thus slow down\n", - " training in some cases.\n", - "- `free_mem_ratio` - float between 0 and 1, default=0. The free memory ratio to keep during training.\n", - "- `metric_constraints` - list, default=[] | The list of metric constraints.\n", - " Each element in this list is a 3-tuple, which shall be expressed\n", - " in the following format: the first element of the 3-tuple is the name of the\n", - " metric, the second element is the inequality sign chosen from \">=\" and \"<=\",\n", - " and the third element is the constraint value. E.g., `('precision', '>=', 0.9)`.\n", - " Note that all the metric names in metric_constraints need to be reported via\n", - " the metrics_to_log dictionary returned by a customized metric function.\n", - " The customized metric function shall be provided via the `metric` key word argument\n", - " of the fit() function or the automl constructor.\n", - " Find examples in this [test](https://github.com/microsoft/FLAML/tree/main/test/automl/test_constraints.py).\n", - " If `pred_time_limit` is provided as one of keyword arguments to fit() function or\n", - " the automl constructor, flaml will automatically (and under the hood)\n", - " add it as an additional element in the metric_constraints. Essentially 'pred_time_limit'\n", - " specifies a constraint about the prediction latency constraint in seconds.\n", - "- `custom_hp` - dict, default=None | The custom search space specified by user\n", - " Each key is the estimator name, each value is a dict of the custom search space for that estimator. Notice the\n", - " domain of the custom search space can either be a value of a sample.Domain object.\n", - " \n", - " \n", - " \n", - "```python\n", - "custom_hp = {\n", - " \"transformer_ms\": {\n", - " \"model_path\": {\n", - " \"domain\": \"albert-base-v2\",\n", - " },\n", - " \"learning_rate\": {\n", - " \"domain\": tune.choice([1e-4, 1e-5]),\n", - " }\n", - " }\n", - "}\n", - "```\n", - "- `time_col` - for a time series task, name of the column containing the timestamps. If not\n", - " provided, defaults to the first column of X_train/X_val\n", - " \n", - "- `cv_score_agg_func` - customized cross-validation scores aggregate function. Default to average metrics across folds. If specificed, this function needs to\n", - " have the following input arguments:\n", - " \n", - " * val_loss_folds: list of floats, the loss scores of each fold;\n", - " * log_metrics_folds: list of dicts/floats, the metrics of each fold to log.\n", - " \n", - " This function should return the final aggregate result of all folds. A float number of the minimization objective, and a dictionary as the metrics to log or None.\n", - " E.g.,\n", - " \n", - "```python\n", - "def cv_score_agg_func(val_loss_folds, log_metrics_folds):\n", - " metric_to_minimize = sum(val_loss_folds)/len(val_loss_folds)\n", - " metrics_to_log = None\n", - " for single_fold in log_metrics_folds:\n", - " if metrics_to_log is None:\n", - " metrics_to_log = single_fold\n", - " elif isinstance(metrics_to_log, dict):\n", - " metrics_to_log = {k: metrics_to_log[k] + v for k, v in single_fold.items()}\n", - " else:\n", - " metrics_to_log += single_fold\n", - " if metrics_to_log:\n", - " n = len(val_loss_folds)\n", - " metrics_to_log = (\n", - " {k: v / n for k, v in metrics_to_log.items()}\n", - " if isinstance(metrics_to_log, dict)\n", - " else metrics_to_log / n\n", - " )\n", - " return metric_to_minimize, metrics_to_log\n", - "```\n", - " \n", - "- `skip_transform` - boolean, default=False | Whether to pre-process data prior to modeling.\n", - "- `mlflow_logging` - boolean, default=None | Whether to log the training results to mlflow.\n", - " Default value is None, which means the logging decision is made based on\n", - " AutoML.__init__'s mlflow_logging argument.\n", - " This requires mlflow to be installed and to have an active mlflow run.\n", - " FLAML will create nested runs.\n", - "- `fit_kwargs_by_estimator` - dict, default=None | The user specified keywords arguments, grouped by estimator name.\n", - " For TransformersEstimator, available fit_kwargs can be found from\n", - " [TrainingArgumentsForAuto](nlp/huggingface/training_args).\n", - " e.g.,\n", - " \n", - "```python\n", - "fit_kwargs_by_estimator = {\n", - " \"transformer\": {\n", - " \"output_dir\": \"test/data/output/\",\n", - " \"fp16\": False,\n", - " },\n", - " \"tft\": {\n", - " \"max_encoder_length\": 1,\n", - " \"min_encoder_length\": 1,\n", - " \"static_categoricals\": [],\n", - " \"static_reals\": [],\n", - " \"time_varying_known_categoricals\": [],\n", - " \"time_varying_known_reals\": [],\n", - " \"time_varying_unknown_categoricals\": [],\n", - " \"time_varying_unknown_reals\": [],\n", - " \"variable_groups\": {},\n", - " \"lags\": {},\n", - " }\n", - "}\n", - "```\n", - " \n", - "- `**fit_kwargs` - Other key word arguments to pass to fit() function of\n", - " the searched learners, such as sample_weight. Below are a few examples of\n", - " estimator-specific parameters:\n", - "- `period` - int | forecast horizon for all time series forecast tasks.\n", - "- `gpu_per_trial` - float, default = 0 | A float of the number of gpus per trial,\n", - " only used by TransformersEstimator, XGBoostSklearnEstimator, and\n", - " TemporalFusionTransformerEstimator.\n", - "- `group_ids` - list of strings of column names identifying a time series, only\n", - " used by TemporalFusionTransformerEstimator, required for\n", - " 'ts_forecast_panel' task. `group_ids` is a parameter for TimeSeriesDataSet object\n", - " from PyTorchForecasting.\n", - " For other parameters to describe your dataset, refer to\n", - " [TimeSeriesDataSet PyTorchForecasting](https://pytorch-forecasting.readthedocs.io/en/stable/api/pytorch_forecasting.data.timeseries.TimeSeriesDataSet.html).\n", - " To specify your variables, use `static_categoricals`, `static_reals`,\n", - " `time_varying_known_categoricals`, `time_varying_known_reals`,\n", - " `time_varying_unknown_categoricals`, `time_varying_unknown_reals`,\n", - " `variable_groups`. To provide more information on your data, use\n", - " `max_encoder_length`, `min_encoder_length`, `lags`.\n", - "- `log_dir` - str, default = \"lightning_logs\" | Folder into which to log results\n", - " for tensorboard, only used by TemporalFusionTransformerEstimator.\n", - "- `max_epochs` - int, default = 20 | Maximum number of epochs to run training,\n", - " only used by TemporalFusionTransformerEstimator.\n", - "- `batch_size` - int, default = 64 | Batch size for training model, only\n", - " used by TemporalFusionTransformerEstimator.\n", - "\n", - "\n", - "---\n", - "sidebar_label: task\n", - "title: automl.task.task\n", - "---\n", - "\n", - "## Task Objects\n", - "\n", - "```python\n", - "class Task(ABC)\n", - "```\n", - "\n", - "Abstract base class for a machine learning task.\n", - "\n", - "Class definitions should implement abstract methods and provide a non-empty dictionary of estimator classes.\n", - "A Task can be suitable to be used for multiple machine-learning tasks (e.g. classification or regression) or be\n", - "implemented specifically for a single one depending on the generality of data validation and model evaluation methods\n", - "implemented. The implementation of a Task may optionally use the training data and labels to determine data and task\n", - "specific details, such as in determining if a problem is single-label or multi-label.\n", - "\n", - "FLAML evaluates at runtime how to behave exactly, relying on the task instance to provide implementations of\n", - "operations which vary between tasks.\n", - "\n", - "#### \\_\\_init\\_\\_\n", - "\n", - "```python\n", - "def __init__(task_name: str, X_train: Optional[Union[np.ndarray, DataFrame, psDataFrame]] = None, y_train: Optional[Union[np.ndarray, DataFrame, Series, psSeries]] = None)\n", - "```\n", - "\n", - "Constructor.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `task_name` - String name for this type of task. Used when the Task can be generic and implement a number of\n", - " types of sub-task.\n", - "- `X_train` - Optional. Some Task types may use the data shape or features to determine details of their usage,\n", - " such as in binary vs multilabel classification.\n", - "- `y_train` - Optional. Some Task types may use the data shape or features to determine details of their usage,\n", - " such as in binary vs multilabel classification.\n", - "\n", - "#### \\_\\_str\\_\\_\n", - "\n", - "```python\n", - "def __str__() -> str\n", - "```\n", - "\n", - "Name of this task type.\n", - "\n", - "#### evaluate\\_model\\_CV\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def evaluate_model_CV(config: dict, estimator: \"flaml.automl.ml.BaseEstimator\", X_train_all: Union[np.ndarray, DataFrame, psDataFrame], y_train_all: Union[np.ndarray, DataFrame, Series, psSeries], budget: int, kf, eval_metric: str, best_val_loss: float, log_training_metric: bool = False, fit_kwargs: Optional[dict] = {}) -> Tuple[float, float, float, float]\n", - "```\n", - "\n", - "Evaluate the model using cross-validation.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `config` - configuration used in the evaluation of the metric.\n", - "- `estimator` - Estimator class of the model.\n", - "- `X_train_all` - Complete training feature data.\n", - "- `y_train_all` - Complete training target data.\n", - "- `budget` - Training time budget.\n", - "- `kf` - Cross-validation index generator.\n", - "- `eval_metric` - Metric name to be used for evaluation.\n", - "- `best_val_loss` - Best current validation-set loss.\n", - "- `log_training_metric` - Bool defaults False. Enables logging of the training metric.\n", - "- `fit_kwargs` - Additional kwargs passed to the estimator's fit method.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " validation loss, metric value, train time, prediction time\n", - "\n", - "#### validate\\_data\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def validate_data(automl: \"flaml.automl.automl.AutoML\", state: \"flaml.automl.state.AutoMLState\", X_train_all: Union[np.ndarray, DataFrame, psDataFrame, None], y_train_all: Union[np.ndarray, DataFrame, Series, psSeries, None], dataframe: Union[DataFrame, None], label: str, X_val: Optional[Union[np.ndarray, DataFrame, psDataFrame]] = None, y_val: Optional[Union[np.ndarray, DataFrame, Series, psSeries]] = None, groups_val: Optional[List[str]] = None, groups: Optional[List[str]] = None)\n", - "```\n", - "\n", - "Validate that the data is suitable for this task type.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `automl` - The AutoML instance from which this task has been constructed.\n", - "- `state` - The AutoMLState instance for this run.\n", - "- `X_train_all` - The complete data set or None if dataframe is supplied.\n", - "- `y_train_all` - The complete target set or None if dataframe is supplied.\n", - "- `dataframe` - A dataframe constaining the complete data set with targets.\n", - "- `label` - The name of the target column in dataframe.\n", - "- `X_val` - Optional. A data set for validation.\n", - "- `y_val` - Optional. A target vector corresponding to X_val for validation.\n", - "- `groups_val` - Group labels (with matching length to y_val) or group counts (with sum equal to length of y_val)\n", - " for validation data. Need to be consistent with groups.\n", - "- `groups` - Group labels (with matching length to y_train) or groups counts (with sum equal to length of y_train)\n", - " for training data.\n", - " \n", - "\n", - "**Raises**:\n", - "\n", - "- `AssertionError` - The data provided is invalid for this task type and configuration.\n", - "\n", - "#### prepare\\_data\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def prepare_data(state: \"flaml.automl.state.AutoMLState\", X_train_all: Union[np.ndarray, DataFrame, psDataFrame], y_train_all: Union[np.ndarray, DataFrame, Series, psSeries, None], auto_augment: bool, eval_method: str, split_type: str, split_ratio: float, n_splits: int, data_is_df: bool, sample_weight_full: Optional[List[float]] = None)\n", - "```\n", - "\n", - "Prepare the data for fitting or inference.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `automl` - The AutoML instance from which this task has been constructed.\n", - "- `state` - The AutoMLState instance for this run.\n", - "- `X_train_all` - The complete data set or None if dataframe is supplied. Must\n", - " contain the target if y_train_all is None\n", - "- `y_train_all` - The complete target set or None if supplied in X_train_all.\n", - "- `auto_augment` - If true, task-specific data augmentations will be applied.\n", - "- `eval_method` - A string of resampling strategy, one of ['auto', 'cv', 'holdout'].\n", - "- `split_type` - str or splitter object, default=\"auto\" | the data split type.\n", - " * A valid splitter object is an instance of a derived class of scikit-learn\n", - " [KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold)\n", - " and have ``split`` and ``get_n_splits`` methods with the same signatures.\n", - " Set eval_method to \"cv\" to use the splitter object.\n", - " * Valid str options depend on different tasks.\n", - " For classification tasks, valid choices are\n", - " [\"auto\", 'stratified', 'uniform', 'time', 'group']. \"auto\" -> stratified.\n", - " For regression tasks, valid choices are [\"auto\", 'uniform', 'time'].\n", - " \"auto\" -> uniform.\n", - " For time series forecast tasks, must be \"auto\" or 'time'.\n", - " For ranking task, must be \"auto\" or 'group'.\n", - "- `split_ratio` - A float of the valiation data percentage for holdout.\n", - "- `n_splits` - An integer of the number of folds for cross - validation.\n", - "- `data_is_df` - True if the data was provided as a DataFrame else False.\n", - "- `sample_weight_full` - A 1d arraylike of the sample weight.\n", - " \n", - "\n", - "**Raises**:\n", - "\n", - "- `AssertionError` - The configuration provided is invalid for this task type and data.\n", - "\n", - "#### decide\\_split\\_type\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def decide_split_type(split_type: str, y_train_all: Union[np.ndarray, DataFrame, Series, psSeries, None], fit_kwargs: dict, groups: Optional[List[str]] = None) -> str\n", - "```\n", - "\n", - "Choose an appropriate data split type for this data and task.\n", - "\n", - "If split_type is 'auto' then this is determined based on the task type and data.\n", - "If a specific split_type is requested then the choice is validated to be appropriate.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `split_type` - Either 'auto' or a task appropriate split type.\n", - "- `y_train_all` - The complete set of targets.\n", - "- `fit_kwargs` - Additional kwargs passed to the estimator's fit method.\n", - "- `groups` - Optional. Group labels (with matching length to y_train) or groups counts (with sum equal to length\n", - " of y_train) for training data.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " The determined appropriate split type.\n", - " \n", - "\n", - "**Raises**:\n", - "\n", - "- `AssertionError` - The requested split_type is invalid for this task, configuration and data.\n", - "\n", - "#### preprocess\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def preprocess(X: Union[np.ndarray, DataFrame, psDataFrame], transformer: Optional[\"flaml.automl.data.DataTransformer\"] = None) -> Union[np.ndarray, DataFrame]\n", - "```\n", - "\n", - "Preprocess the data ready for fitting or inference with this task type.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `X` - The data set to process.\n", - "- `transformer` - A DataTransformer instance to be used in processing.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " The preprocessed data set having the same type as the input.\n", - "\n", - "#### default\\_estimator\\_list\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def default_estimator_list(estimator_list: Union[List[str], str] = \"auto\", is_spark_dataframe: bool = False) -> List[str]\n", - "```\n", - "\n", - "Return the list of default estimators registered for this task type.\n", - "\n", - "If 'auto' is provided then the default list is returned, else the provided list will be validated given this task\n", - "type.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `estimator_list` - Either 'auto' or a list of estimator names to be validated.\n", - "- `is_spark_dataframe` - True if the data is a spark dataframe.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " A list of valid estimator names for this task type.\n", - "\n", - "#### default\\_metric\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def default_metric(metric: str) -> str\n", - "```\n", - "\n", - "Return the default metric for this task type.\n", - "\n", - "\n", - "\n", - "--------------------------------------------------------------------------------\n", - "\u001b[32mAdding doc_id doc_58 to context.\u001b[0m\n", - "\u001b[32mAdding doc_id doc_40 to context.\u001b[0m\n", - "\u001b[32mAdding doc_id doc_51 to context.\u001b[0m\n", - "\u001b[33mragproxyagent\u001b[0m (to assistant):\n", - "\n", - "You're a retrieve augmented coding assistant. You answer user's questions based on your own knowledge and the\n", - "context provided by the user.\n", - "If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.\n", - "For code generation, you must obey the following rules:\n", - "Rule 1. You MUST NOT install any packages because all the packages needed are already installed.\n", - "Rule 2. You must follow the formats below to write your code:\n", - "```language\n", - "# your code\n", - "```\n", - "\n", - "User's question is: Who is the author of FLAML?\n", - "\n", - "Context is: \n", - "- `seed` - int or None, default=None | The random seed for hpo.\n", - "- `n_concurrent_trials` - [Experimental] int, default=1 | The number of\n", - " concurrent trials. When n_concurrent_trials > 1, flaml performes\n", - " [parallel tuning](../../Use-Cases/Task-Oriented-AutoML#parallel-tuning)\n", - " and installation of ray or spark is required: `pip install flaml[ray]`\n", - " or `pip install flaml[spark]`. Please check\n", - " [here](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)\n", - " for more details about installing Spark.\n", - "- `keep_search_state` - boolean, default=False | Whether to keep data needed\n", - " for model search after fit(). By default the state is deleted for\n", - " space saving.\n", - "- `preserve_checkpoint` - boolean, default=True | Whether to preserve the saved checkpoint\n", - " on disk when deleting automl. By default the checkpoint is preserved.\n", - "- `early_stop` - boolean, default=False | Whether to stop early if the\n", - " search is considered to converge.\n", - "- `force_cancel` - boolean, default=False | Whether to forcely cancel Spark jobs if the\n", - " search time exceeded the time budget.\n", - "- `append_log` - boolean, default=False | Whetehr to directly append the log\n", - " records to the input log file if it exists.\n", - "- `auto_augment` - boolean, default=True | Whether to automatically\n", - " augment rare classes.\n", - "- `min_sample_size` - int, default=MIN_SAMPLE_TRAIN | the minimal sample\n", - " size when sample=True.\n", - "- `use_ray` - boolean or dict.\n", - " If boolean: default=False | Whether to use ray to run the training\n", - " in separate processes. This can be used to prevent OOM for large\n", - " datasets, but will incur more overhead in time.\n", - " If dict: the dict contains the keywords arguments to be passed to\n", - " [ray.tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html).\n", - "- `use_spark` - boolean, default=False | Whether to use spark to run the training\n", - " in parallel spark jobs. This can be used to accelerate training on large models\n", - " and large datasets, but will incur more overhead in time and thus slow down\n", - " training in some cases. GPU training is not supported yet when use_spark is True.\n", - " For Spark clusters, by default, we will launch one trial per executor. However,\n", - " sometimes we want to launch more trials than the number of executors (e.g., local mode).\n", - " In this case, we can set the environment variable `FLAML_MAX_CONCURRENT` to override\n", - " the detected `num_executors`. The final number of concurrent trials will be the minimum\n", - " of `n_concurrent_trials` and `num_executors`.\n", - "- `free_mem_ratio` - float between 0 and 1, default=0. The free memory ratio to keep during training.\n", - "- `metric_constraints` - list, default=[] | The list of metric constraints.\n", - " Each element in this list is a 3-tuple, which shall be expressed\n", - " in the following format: the first element of the 3-tuple is the name of the\n", - " metric, the second element is the inequality sign chosen from \">=\" and \"<=\",\n", - " and the third element is the constraint value. E.g., `('val_loss', '<=', 0.1)`.\n", - " Note that all the metric names in metric_constraints need to be reported via\n", - " the metrics_to_log dictionary returned by a customized metric function.\n", - " The customized metric function shall be provided via the `metric` key word\n", - " argument of the fit() function or the automl constructor.\n", - " Find an example in the 4th constraint type in this [doc](../../Use-Cases/Task-Oriented-AutoML#constraint).\n", - " If `pred_time_limit` is provided as one of keyword arguments to fit() function or\n", - " the automl constructor, flaml will automatically (and under the hood)\n", - " add it as an additional element in the metric_constraints. Essentially 'pred_time_limit'\n", - " specifies a constraint about the prediction latency constraint in seconds.\n", - "- `custom_hp` - dict, default=None | The custom search space specified by user.\n", - " It is a nested dict with keys being the estimator names, and values being dicts\n", - " per estimator search space. In the per estimator search space dict,\n", - " the keys are the hyperparameter names, and values are dicts of info (\"domain\",\n", - " \"init_value\", and \"low_cost_init_value\") about the search space associated with\n", - " the hyperparameter (i.e., per hyperparameter search space dict). When custom_hp\n", - " is provided, the built-in search space which is also a nested dict of per estimator\n", - " search space dict, will be updated with custom_hp. Note that during this nested dict update,\n", - " the per hyperparameter search space dicts will be replaced (instead of updated) by the ones\n", - " provided in custom_hp. Note that the value for \"domain\" can either be a constant\n", - " or a sample.Domain object.\n", - " e.g.,\n", - " \n", - "```python\n", - "custom_hp = {\n", - " \"transformer_ms\": {\n", - " \"model_path\": {\n", - " \"domain\": \"albert-base-v2\",\n", - " },\n", - " \"learning_rate\": {\n", - " \"domain\": tune.choice([1e-4, 1e-5]),\n", - " }\n", - " }\n", - " }\n", - "```\n", - "- `skip_transform` - boolean, default=False | Whether to pre-process data prior to modeling.\n", - "- `fit_kwargs_by_estimator` - dict, default=None | The user specified keywords arguments, grouped by estimator name.\n", - " e.g.,\n", - " \n", - "```python\n", - "fit_kwargs_by_estimator = {\n", - " \"transformer\": {\n", - " \"output_dir\": \"test/data/output/\",\n", - " \"fp16\": False,\n", - " }\n", - "}\n", - "```\n", - "- `mlflow_logging` - boolean, default=True | Whether to log the training results to mlflow.\n", - " This requires mlflow to be installed and to have an active mlflow run.\n", - " FLAML will create nested runs.\n", - "\n", - "#### config\\_history\n", - "\n", - "```python\n", - "@property\n", - "def config_history() -> dict\n", - "```\n", - "\n", - "A dictionary of iter->(estimator, config, time),\n", - "storing the best estimator, config, and the time when the best\n", - "model is updated each time.\n", - "\n", - "#### model\n", - "\n", - "```python\n", - "@property\n", - "def model()\n", - "```\n", - "\n", - "An object with `predict()` and `predict_proba()` method (for\n", - "classification), storing the best trained model.\n", - "\n", - "#### best\\_model\\_for\\_estimator\n", - "\n", - "```python\n", - "def best_model_for_estimator(estimator_name: str)\n", - "```\n", - "\n", - "Return the best model found for a particular estimator.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `estimator_name` - a str of the estimator's name.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " An object storing the best model for estimator_name.\n", - " If `model_history` was set to False during fit(), then the returned model\n", - " is untrained unless estimator_name is the best estimator.\n", - " If `model_history` was set to True, then the returned model is trained.\n", - "\n", - "#### best\\_estimator\n", - "\n", - "```python\n", - "@property\n", - "def best_estimator()\n", - "```\n", - "\n", - "A string indicating the best estimator found.\n", - "\n", - "#### best\\_iteration\n", - "\n", - "```python\n", - "@property\n", - "def best_iteration()\n", - "```\n", - "\n", - "An integer of the iteration number where the best\n", - "config is found.\n", - "\n", - "#### best\\_config\n", - "\n", - "```python\n", - "@property\n", - "def best_config()\n", - "```\n", - "\n", - "A dictionary of the best configuration.\n", - "\n", - "#### best\\_config\\_per\\_estimator\n", - "\n", - "```python\n", - "@property\n", - "def best_config_per_estimator()\n", - "```\n", - "\n", - "A dictionary of all estimators' best configuration.\n", - "\n", - "#### best\\_loss\\_per\\_estimator\n", - "\n", - "```python\n", - "@property\n", - "def best_loss_per_estimator()\n", - "```\n", - "\n", - "A dictionary of all estimators' best loss.\n", - "\n", - "#### best\\_loss\n", - "\n", - "```python\n", - "@property\n", - "def best_loss()\n", - "```\n", - "\n", - "A float of the best loss found.\n", - "\n", - "#### best\\_result\n", - "\n", - "```python\n", - "@property\n", - "def best_result()\n", - "```\n", - "\n", - "Result dictionary for model trained with the best config.\n", - "\n", - "#### metrics\\_for\\_best\\_config\n", - "\n", - "```python\n", - "@property\n", - "def metrics_for_best_config()\n", - "```\n", - "\n", - "Returns a float of the best loss, and a dictionary of the auxiliary metrics to log\n", - "associated with the best config. These two objects correspond to the returned\n", - "objects by the customized metric function for the config with the best loss.\n", - "\n", - "#### best\\_config\\_train\\_time\n", - "---\n", - "sidebar_label: estimator\n", - "title: default.estimator\n", - "---\n", - "\n", - "#### flamlize\\_estimator\n", - "\n", - "```python\n", - "def flamlize_estimator(super_class, name: str, task: str, alternatives=None)\n", - "```\n", - "\n", - "Enhance an estimator class with flaml's data-dependent default hyperparameter settings.\n", - "\n", - "**Example**:\n", - "\n", - " \n", - "```python\n", - "import sklearn.ensemble as ensemble\n", - "RandomForestRegressor = flamlize_estimator(\n", - " ensemble.RandomForestRegressor, \"rf\", \"regression\"\n", - ")\n", - "```\n", - " \n", - "\n", - "**Arguments**:\n", - "\n", - "- `super_class` - an scikit-learn compatible estimator class.\n", - "- `name` - a str of the estimator's name.\n", - "- `task` - a str of the task type.\n", - "- `alternatives` - (Optional) a list for alternative estimator names. For example,\n", - " ```[(\"max_depth\", 0, \"xgboost\")]``` means if the \"max_depth\" is set to 0\n", - " in the constructor, then look for the learned defaults for estimator \"xgboost\".\n", - "\n", - "\n", - " \n", - "- `seed` - int or None, default=None | The random seed for hpo.\n", - "- `n_concurrent_trials` - [Experimental] int, default=1 | The number of\n", - " concurrent trials. When n_concurrent_trials > 1, flaml performes\n", - " [parallel tuning](../../Use-Cases/Task-Oriented-AutoML#parallel-tuning)\n", - " and installation of ray or spark is required: `pip install flaml[ray]`\n", - " or `pip install flaml[spark]`. Please check\n", - " [here](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)\n", - " for more details about installing Spark.\n", - "- `keep_search_state` - boolean, default=False | Whether to keep data needed\n", - " for model search after fit(). By default the state is deleted for\n", - " space saving.\n", - "- `preserve_checkpoint` - boolean, default=True | Whether to preserve the saved checkpoint\n", - " on disk when deleting automl. By default the checkpoint is preserved.\n", - "- `early_stop` - boolean, default=False | Whether to stop early if the\n", - " search is considered to converge.\n", - "- `force_cancel` - boolean, default=False | Whether to forcely cancel the PySpark job if overtime.\n", - "- `append_log` - boolean, default=False | Whetehr to directly append the log\n", - " records to the input log file if it exists.\n", - "- `auto_augment` - boolean, default=True | Whether to automatically\n", - " augment rare classes.\n", - "- `min_sample_size` - int, default=MIN_SAMPLE_TRAIN | the minimal sample\n", - " size when sample=True.\n", - "- `use_ray` - boolean or dict.\n", - " If boolean: default=False | Whether to use ray to run the training\n", - " in separate processes. This can be used to prevent OOM for large\n", - " datasets, but will incur more overhead in time.\n", - " If dict: the dict contains the keywords arguments to be passed to\n", - " [ray.tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html).\n", - "- `use_spark` - boolean, default=False | Whether to use spark to run the training\n", - " in parallel spark jobs. This can be used to accelerate training on large models\n", - " and large datasets, but will incur more overhead in time and thus slow down\n", - " training in some cases.\n", - "- `free_mem_ratio` - float between 0 and 1, default=0. The free memory ratio to keep during training.\n", - "- `metric_constraints` - list, default=[] | The list of metric constraints.\n", - " Each element in this list is a 3-tuple, which shall be expressed\n", - " in the following format: the first element of the 3-tuple is the name of the\n", - " metric, the second element is the inequality sign chosen from \">=\" and \"<=\",\n", - " and the third element is the constraint value. E.g., `('precision', '>=', 0.9)`.\n", - " Note that all the metric names in metric_constraints need to be reported via\n", - " the metrics_to_log dictionary returned by a customized metric function.\n", - " The customized metric function shall be provided via the `metric` key word argument\n", - " of the fit() function or the automl constructor.\n", - " Find examples in this [test](https://github.com/microsoft/FLAML/tree/main/test/automl/test_constraints.py).\n", - " If `pred_time_limit` is provided as one of keyword arguments to fit() function or\n", - " the automl constructor, flaml will automatically (and under the hood)\n", - " add it as an additional element in the metric_constraints. Essentially 'pred_time_limit'\n", - " specifies a constraint about the prediction latency constraint in seconds.\n", - "- `custom_hp` - dict, default=None | The custom search space specified by user\n", - " Each key is the estimator name, each value is a dict of the custom search space for that estimator. Notice the\n", - " domain of the custom search space can either be a value of a sample.Domain object.\n", - " \n", - " \n", - " \n", - "```python\n", - "custom_hp = {\n", - " \"transformer_ms\": {\n", - " \"model_path\": {\n", - " \"domain\": \"albert-base-v2\",\n", - " },\n", - " \"learning_rate\": {\n", - " \"domain\": tune.choice([1e-4, 1e-5]),\n", - " }\n", - " }\n", - "}\n", - "```\n", - "- `time_col` - for a time series task, name of the column containing the timestamps. If not\n", - " provided, defaults to the first column of X_train/X_val\n", - " \n", - "- `cv_score_agg_func` - customized cross-validation scores aggregate function. Default to average metrics across folds. If specificed, this function needs to\n", - " have the following input arguments:\n", - " \n", - " * val_loss_folds: list of floats, the loss scores of each fold;\n", - " * log_metrics_folds: list of dicts/floats, the metrics of each fold to log.\n", - " \n", - " This function should return the final aggregate result of all folds. A float number of the minimization objective, and a dictionary as the metrics to log or None.\n", - " E.g.,\n", - " \n", - "```python\n", - "def cv_score_agg_func(val_loss_folds, log_metrics_folds):\n", - " metric_to_minimize = sum(val_loss_folds)/len(val_loss_folds)\n", - " metrics_to_log = None\n", - " for single_fold in log_metrics_folds:\n", - " if metrics_to_log is None:\n", - " metrics_to_log = single_fold\n", - " elif isinstance(metrics_to_log, dict):\n", - " metrics_to_log = {k: metrics_to_log[k] + v for k, v in single_fold.items()}\n", - " else:\n", - " metrics_to_log += single_fold\n", - " if metrics_to_log:\n", - " n = len(val_loss_folds)\n", - " metrics_to_log = (\n", - " {k: v / n for k, v in metrics_to_log.items()}\n", - " if isinstance(metrics_to_log, dict)\n", - " else metrics_to_log / n\n", - " )\n", - " return metric_to_minimize, metrics_to_log\n", - "```\n", - " \n", - "- `skip_transform` - boolean, default=False | Whether to pre-process data prior to modeling.\n", - "- `mlflow_logging` - boolean, default=None | Whether to log the training results to mlflow.\n", - " Default value is None, which means the logging decision is made based on\n", - " AutoML.__init__'s mlflow_logging argument.\n", - " This requires mlflow to be installed and to have an active mlflow run.\n", - " FLAML will create nested runs.\n", - "- `fit_kwargs_by_estimator` - dict, default=None | The user specified keywords arguments, grouped by estimator name.\n", - " For TransformersEstimator, available fit_kwargs can be found from\n", - " [TrainingArgumentsForAuto](nlp/huggingface/training_args).\n", - " e.g.,\n", - " \n", - "```python\n", - "fit_kwargs_by_estimator = {\n", - " \"transformer\": {\n", - " \"output_dir\": \"test/data/output/\",\n", - " \"fp16\": False,\n", - " },\n", - " \"tft\": {\n", - " \"max_encoder_length\": 1,\n", - " \"min_encoder_length\": 1,\n", - " \"static_categoricals\": [],\n", - " \"static_reals\": [],\n", - " \"time_varying_known_categoricals\": [],\n", - " \"time_varying_known_reals\": [],\n", - " \"time_varying_unknown_categoricals\": [],\n", - " \"time_varying_unknown_reals\": [],\n", - " \"variable_groups\": {},\n", - " \"lags\": {},\n", - " }\n", - "}\n", - "```\n", - " \n", - "- `**fit_kwargs` - Other key word arguments to pass to fit() function of\n", - " the searched learners, such as sample_weight. Below are a few examples of\n", - " estimator-specific parameters:\n", - "- `period` - int | forecast horizon for all time series forecast tasks.\n", - "- `gpu_per_trial` - float, default = 0 | A float of the number of gpus per trial,\n", - " only used by TransformersEstimator, XGBoostSklearnEstimator, and\n", - " TemporalFusionTransformerEstimator.\n", - "- `group_ids` - list of strings of column names identifying a time series, only\n", - " used by TemporalFusionTransformerEstimator, required for\n", - " 'ts_forecast_panel' task. `group_ids` is a parameter for TimeSeriesDataSet object\n", - " from PyTorchForecasting.\n", - " For other parameters to describe your dataset, refer to\n", - " [TimeSeriesDataSet PyTorchForecasting](https://pytorch-forecasting.readthedocs.io/en/stable/api/pytorch_forecasting.data.timeseries.TimeSeriesDataSet.html).\n", - " To specify your variables, use `static_categoricals`, `static_reals`,\n", - " `time_varying_known_categoricals`, `time_varying_known_reals`,\n", - " `time_varying_unknown_categoricals`, `time_varying_unknown_reals`,\n", - " `variable_groups`. To provide more information on your data, use\n", - " `max_encoder_length`, `min_encoder_length`, `lags`.\n", - "- `log_dir` - str, default = \"lightning_logs\" | Folder into which to log results\n", - " for tensorboard, only used by TemporalFusionTransformerEstimator.\n", - "- `max_epochs` - int, default = 20 | Maximum number of epochs to run training,\n", - " only used by TemporalFusionTransformerEstimator.\n", - "- `batch_size` - int, default = 64 | Batch size for training model, only\n", - " used by TemporalFusionTransformerEstimator.\n", - "\n", - "\n", - "---\n", - "sidebar_label: task\n", - "title: automl.task.task\n", - "---\n", - "\n", - "## Task Objects\n", - "\n", - "```python\n", - "class Task(ABC)\n", - "```\n", - "\n", - "Abstract base class for a machine learning task.\n", - "\n", - "Class definitions should implement abstract methods and provide a non-empty dictionary of estimator classes.\n", - "A Task can be suitable to be used for multiple machine-learning tasks (e.g. classification or regression) or be\n", - "implemented specifically for a single one depending on the generality of data validation and model evaluation methods\n", - "implemented. The implementation of a Task may optionally use the training data and labels to determine data and task\n", - "specific details, such as in determining if a problem is single-label or multi-label.\n", - "\n", - "FLAML evaluates at runtime how to behave exactly, relying on the task instance to provide implementations of\n", - "operations which vary between tasks.\n", - "\n", - "#### \\_\\_init\\_\\_\n", - "\n", - "```python\n", - "def __init__(task_name: str, X_train: Optional[Union[np.ndarray, DataFrame, psDataFrame]] = None, y_train: Optional[Union[np.ndarray, DataFrame, Series, psSeries]] = None)\n", - "```\n", - "\n", - "Constructor.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `task_name` - String name for this type of task. Used when the Task can be generic and implement a number of\n", - " types of sub-task.\n", - "- `X_train` - Optional. Some Task types may use the data shape or features to determine details of their usage,\n", - " such as in binary vs multilabel classification.\n", - "- `y_train` - Optional. Some Task types may use the data shape or features to determine details of their usage,\n", - " such as in binary vs multilabel classification.\n", - "\n", - "#### \\_\\_str\\_\\_\n", - "\n", - "```python\n", - "def __str__() -> str\n", - "```\n", - "\n", - "Name of this task type.\n", - "\n", - "#### evaluate\\_model\\_CV\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def evaluate_model_CV(config: dict, estimator: \"flaml.automl.ml.BaseEstimator\", X_train_all: Union[np.ndarray, DataFrame, psDataFrame], y_train_all: Union[np.ndarray, DataFrame, Series, psSeries], budget: int, kf, eval_metric: str, best_val_loss: float, log_training_metric: bool = False, fit_kwargs: Optional[dict] = {}) -> Tuple[float, float, float, float]\n", - "```\n", - "\n", - "Evaluate the model using cross-validation.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `config` - configuration used in the evaluation of the metric.\n", - "- `estimator` - Estimator class of the model.\n", - "- `X_train_all` - Complete training feature data.\n", - "- `y_train_all` - Complete training target data.\n", - "- `budget` - Training time budget.\n", - "- `kf` - Cross-validation index generator.\n", - "- `eval_metric` - Metric name to be used for evaluation.\n", - "- `best_val_loss` - Best current validation-set loss.\n", - "- `log_training_metric` - Bool defaults False. Enables logging of the training metric.\n", - "- `fit_kwargs` - Additional kwargs passed to the estimator's fit method.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " validation loss, metric value, train time, prediction time\n", - "\n", - "#### validate\\_data\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def validate_data(automl: \"flaml.automl.automl.AutoML\", state: \"flaml.automl.state.AutoMLState\", X_train_all: Union[np.ndarray, DataFrame, psDataFrame, None], y_train_all: Union[np.ndarray, DataFrame, Series, psSeries, None], dataframe: Union[DataFrame, None], label: str, X_val: Optional[Union[np.ndarray, DataFrame, psDataFrame]] = None, y_val: Optional[Union[np.ndarray, DataFrame, Series, psSeries]] = None, groups_val: Optional[List[str]] = None, groups: Optional[List[str]] = None)\n", - "```\n", - "\n", - "Validate that the data is suitable for this task type.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `automl` - The AutoML instance from which this task has been constructed.\n", - "- `state` - The AutoMLState instance for this run.\n", - "- `X_train_all` - The complete data set or None if dataframe is supplied.\n", - "- `y_train_all` - The complete target set or None if dataframe is supplied.\n", - "- `dataframe` - A dataframe constaining the complete data set with targets.\n", - "- `label` - The name of the target column in dataframe.\n", - "- `X_val` - Optional. A data set for validation.\n", - "- `y_val` - Optional. A target vector corresponding to X_val for validation.\n", - "- `groups_val` - Group labels (with matching length to y_val) or group counts (with sum equal to length of y_val)\n", - " for validation data. Need to be consistent with groups.\n", - "- `groups` - Group labels (with matching length to y_train) or groups counts (with sum equal to length of y_train)\n", - " for training data.\n", - " \n", - "\n", - "**Raises**:\n", - "\n", - "- `AssertionError` - The data provided is invalid for this task type and configuration.\n", - "\n", - "#### prepare\\_data\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def prepare_data(state: \"flaml.automl.state.AutoMLState\", X_train_all: Union[np.ndarray, DataFrame, psDataFrame], y_train_all: Union[np.ndarray, DataFrame, Series, psSeries, None], auto_augment: bool, eval_method: str, split_type: str, split_ratio: float, n_splits: int, data_is_df: bool, sample_weight_full: Optional[List[float]] = None)\n", - "```\n", - "\n", - "Prepare the data for fitting or inference.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `automl` - The AutoML instance from which this task has been constructed.\n", - "- `state` - The AutoMLState instance for this run.\n", - "- `X_train_all` - The complete data set or None if dataframe is supplied. Must\n", - " contain the target if y_train_all is None\n", - "- `y_train_all` - The complete target set or None if supplied in X_train_all.\n", - "- `auto_augment` - If true, task-specific data augmentations will be applied.\n", - "- `eval_method` - A string of resampling strategy, one of ['auto', 'cv', 'holdout'].\n", - "- `split_type` - str or splitter object, default=\"auto\" | the data split type.\n", - " * A valid splitter object is an instance of a derived class of scikit-learn\n", - " [KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold)\n", - " and have ``split`` and ``get_n_splits`` methods with the same signatures.\n", - " Set eval_method to \"cv\" to use the splitter object.\n", - " * Valid str options depend on different tasks.\n", - " For classification tasks, valid choices are\n", - " [\"auto\", 'stratified', 'uniform', 'time', 'group']. \"auto\" -> stratified.\n", - " For regression tasks, valid choices are [\"auto\", 'uniform', 'time'].\n", - " \"auto\" -> uniform.\n", - " For time series forecast tasks, must be \"auto\" or 'time'.\n", - " For ranking task, must be \"auto\" or 'group'.\n", - "- `split_ratio` - A float of the valiation data percentage for holdout.\n", - "- `n_splits` - An integer of the number of folds for cross - validation.\n", - "- `data_is_df` - True if the data was provided as a DataFrame else False.\n", - "- `sample_weight_full` - A 1d arraylike of the sample weight.\n", - " \n", - "\n", - "**Raises**:\n", - "\n", - "- `AssertionError` - The configuration provided is invalid for this task type and data.\n", - "\n", - "#### decide\\_split\\_type\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def decide_split_type(split_type: str, y_train_all: Union[np.ndarray, DataFrame, Series, psSeries, None], fit_kwargs: dict, groups: Optional[List[str]] = None) -> str\n", - "```\n", - "\n", - "Choose an appropriate data split type for this data and task.\n", - "\n", - "If split_type is 'auto' then this is determined based on the task type and data.\n", - "If a specific split_type is requested then the choice is validated to be appropriate.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `split_type` - Either 'auto' or a task appropriate split type.\n", - "- `y_train_all` - The complete set of targets.\n", - "- `fit_kwargs` - Additional kwargs passed to the estimator's fit method.\n", - "- `groups` - Optional. Group labels (with matching length to y_train) or groups counts (with sum equal to length\n", - " of y_train) for training data.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " The determined appropriate split type.\n", - " \n", - "\n", - "**Raises**:\n", - "\n", - "- `AssertionError` - The requested split_type is invalid for this task, configuration and data.\n", - "\n", - "#### preprocess\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def preprocess(X: Union[np.ndarray, DataFrame, psDataFrame], transformer: Optional[\"flaml.automl.data.DataTransformer\"] = None) -> Union[np.ndarray, DataFrame]\n", - "```\n", - "\n", - "Preprocess the data ready for fitting or inference with this task type.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `X` - The data set to process.\n", - "- `transformer` - A DataTransformer instance to be used in processing.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " The preprocessed data set having the same type as the input.\n", - "\n", - "#### default\\_estimator\\_list\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def default_estimator_list(estimator_list: Union[List[str], str] = \"auto\", is_spark_dataframe: bool = False) -> List[str]\n", - "```\n", - "\n", - "Return the list of default estimators registered for this task type.\n", - "\n", - "If 'auto' is provided then the default list is returned, else the provided list will be validated given this task\n", - "type.\n", - "\n", - "**Arguments**:\n", - "\n", - "- `estimator_list` - Either 'auto' or a list of estimator names to be validated.\n", - "- `is_spark_dataframe` - True if the data is a spark dataframe.\n", - " \n", - "\n", - "**Returns**:\n", - "\n", - " A list of valid estimator names for this task type.\n", - "\n", - "#### default\\_metric\n", - "\n", - "```python\n", - "@abstractmethod\n", - "def default_metric(metric: str) -> str\n", - "```\n", - "\n", - "Return the default metric for this task type.\n", - "\n", - "\n", - "\n", - "--------------------------------------------------------------------------------\n", - "\u001b[33massistant\u001b[0m (to ragproxyagent):\n", - "\n", - "The authors of FLAML (Fast, Lightweight, and AutoML) are Chi Wang, Qiang Yang, and Huan Liu. They introduced this AutoML library with the goal to efficiently and automatically determine the best machine learning models and their hyperparameter configurations for a given dataset and task.\n", - "\n", - "--------------------------------------------------------------------------------\n" - ] - } - ], + "outputs": [], "source": [ "# reset the assistant. Always reset the assistant before starting a new conversation.\n", "assistant.reset()\n", diff --git a/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb b/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb index 1cf8047e1..b64360bb4 100644 --- a/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb +++ b/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -19,16 +19,16 @@ "source": [ "# Auto Generated Agent Chat: Task Solving with Code Generation, Execution & Debugging\n", "\n", - "FLAML offers conversable LLM agents, which can be used to solve various tasks with human or automatic feedback, including tasks that require using tools via code.\n", + "AutoGen offers conversable LLM agents, which can be used to solve various tasks with human or automatic feedback, including tasks that require using tools via code.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to write code and execute the code. Here `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for the human user to execute the code written by `AssistantAgent`, or automatically execute the code. Depending on the setting of `human_input_mode` and `max_consecutive_auto_reply`, the `UserProxyAgent` either solicits feedback from the human user or returns auto-feedback based on the result of code execution (success or failure and corresponding outputs) to `AssistantAgent`. `AssistantAgent` will debug the code and suggest new code if the result contains error. The two agents keep communicating to each other until the task is done.\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", - "pip install flaml[autogen]\n", + "pip install pyautogen\n", "```" ] }, @@ -45,7 +45,7 @@ }, "outputs": [], "source": [ - "# %pip install flaml[autogen]~=2.0.2" + "# %pip install pyautogen~=0.1.0" ] }, { @@ -64,7 +64,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list = autogen.config_list_from_json(\n", " \"OAI_CONFIG_LIST\",\n", @@ -117,7 +117,7 @@ "source": [ "## Example Task: Check Stock Price Change\n", "\n", - "In the example below, let's see how to use the agents in FLAML to write a python script and execute the script. This process involves constructing a `AssistantAgent` to serve as the assistant, along with a `UserProxyAgent` that acts as a proxy for the human user. In this example demonstrated below, when constructing the `UserProxyAgent`, we select the `human_input_mode` to \"NEVER\". This means that the `UserProxyAgent` will not solicit feedback from the human user. It stops replying when the limit defined by `max_consecutive_auto_reply` is reached, or when `is_termination_msg()` returns true for the received message." + "In the example below, let's see how to use the agents in AutoGen to write a python script and execute the script. This process involves constructing a `AssistantAgent` to serve as the assistant, along with a `UserProxyAgent` that acts as a proxy for the human user. In this example demonstrated below, when constructing the `UserProxyAgent`, we select the `human_input_mode` to \"NEVER\". This means that the `UserProxyAgent` will not solicit feedback from the human user. It stops replying when the limit defined by `max_consecutive_auto_reply` is reached, or when `is_termination_msg()` returns true for the received message." ] }, { @@ -360,7 +360,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The example above involves code execution. In FLAML, code execution is triggered automatically by the `UserProxyAgent` when it detects an executable code block in a received message and no human user input is provided. This process occurs in a designated working directory, using a Docker container by default. Unless a specific directory is specified, FLAML defaults to the `flaml/autogen/extensions` directory. Users have the option to specify a different working directory by setting the `work_dir` argument when constructing a new instance of the `UserProxyAgent`.\n", + "The example above involves code execution. In AutoGen, code execution is triggered automatically by the `UserProxyAgent` when it detects an executable code block in a received message and no human user input is provided. This process occurs in a designated working directory, using a Docker container by default. Unless a specific directory is specified, AutoGen defaults to the `autogen/extensions` directory. Users have the option to specify a different working directory by setting the `work_dir` argument when constructing a new instance of the `UserProxyAgent`.\n", "\n", "The whole chat is auto generated." ] diff --git a/notebook/autogen_agentchat_chess.ipynb b/notebook/autogen_agentchat_chess.ipynb index 465d4c366..253708a51 100644 --- a/notebook/autogen_agentchat_chess.ipynb +++ b/notebook/autogen_agentchat_chess.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -15,16 +15,16 @@ "source": [ "# Auto Generated Agent Chat: Chess Game Playing While Chitchatting by GPT-4 Agents\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "This notebook is modified based on https://github.com/ekzhu/FLAML/blob/evaluation/evaluation/chess/play_chess.ipynb\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", - "pip install flaml[autogen]\n", + "pip install pyautogen\n", "```" ] }, @@ -35,7 +35,7 @@ "outputs": [], "source": [ "%%capture --no-stderr\n", - "# %pip install flaml[autogen]~=2.1.0\n", + "# %pip install pyautogen~=0.1.0\n", "%pip install chess -U" ] }, @@ -65,7 +65,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list_gpt4 = autogen.config_list_from_json(\n", " \"OAI_CONFIG_LIST\",\n", diff --git a/notebook/autogen_agentchat_function_call.ipynb b/notebook/autogen_agentchat_function_call.ipynb index d8ce7ff82..36249153c 100644 --- a/notebook/autogen_agentchat_function_call.ipynb +++ b/notebook/autogen_agentchat_function_call.ipynb @@ -6,7 +6,7 @@ "id": "ae1f50ec", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -17,15 +17,15 @@ "source": [ "# Auto Generated Agent Chat: Task Solving with Provided Tools as Functions\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to make function calls with the new feature of OpenAI models (in model version 0613). A specified prompt and function configs need to be passed to `AssistantAgent` to initialize the agent. The corresponding functions need to be passed to `UserProxyAgent`, which will be responsible for executing any function calls made by `AssistantAgent`. Besides this requirement of matching descriptions with functions, we recommend checking the system message in the `AssistantAgent` to make sure the instructions align with the function call descriptions.\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [mathchat] option since we will import functions from `MathUserProxyAgent`:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install the [mathchat] option since we will import functions from `MathUserProxyAgent`:\n", "```bash\n", - "pip install flaml[mathchat]\n", + "pip install \"pyautogen[mathchat]\"\n", "```" ] }, @@ -36,7 +36,7 @@ "metadata": {}, "outputs": [], "source": [ - "# %pip install flaml[mathchat]~=2.0.0" + "# %pip install \"pyautogen[mathchat]~=0.1.0\"" ] }, { @@ -66,7 +66,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list = autogen.config_list_from_models(model_list=[\"gpt-4\", \"gpt-3.5-turbo\", \"gpt-3.5-turbo-16k\"], exclude=\"aoai\")" ] @@ -372,7 +372,7 @@ ], "source": [ "import os\n", - "from flaml.autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n", + "from autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n", "\n", "# you need to provide a wolfram alpha appid to run this example\n", "if not os.environ.get(\"WOLFRAM_ALPHA_APPID\"):\n", diff --git a/notebook/autogen_agentchat_groupchat.ipynb b/notebook/autogen_agentchat_groupchat.ipynb index 626c39c9e..f6e0d5188 100644 --- a/notebook/autogen_agentchat_groupchat.ipynb +++ b/notebook/autogen_agentchat_groupchat.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -15,16 +15,16 @@ "source": [ "# Auto Generated Agent Chat: Group Chat\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "This notebook is modified based on https://github.com/microsoft/FLAML/blob/4ea686af5c3e8ff24d9076a7a626c8b28ab5b1d7/notebook/autogen_multiagent_roleplay_chat.ipynb\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", - "pip install flaml[autogen]\n", + "pip install pyautogen\n", "```" ] }, @@ -35,7 +35,7 @@ "outputs": [], "source": [ "%%capture --no-stderr\n", - "# %pip install flaml[autogen]~=2.0.2" + "# %pip install pyautogen~=0.1.0" ] }, { @@ -54,7 +54,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list_gpt4 = autogen.config_list_from_json(\n", " \"OAI_CONFIG_LIST\",\n", diff --git a/notebook/autogen_agentchat_groupchat_research.ipynb b/notebook/autogen_agentchat_groupchat_research.ipynb index 8987c3291..6f01ddacd 100644 --- a/notebook/autogen_agentchat_groupchat_research.ipynb +++ b/notebook/autogen_agentchat_groupchat_research.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -15,14 +15,14 @@ "source": [ "# Auto Generated Agent Chat: Performs Research with Multi-Agent Group Chat\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", - "pip install flaml[autogen]\n", + "pip install pyautogen\n", "```" ] }, @@ -33,7 +33,7 @@ "outputs": [], "source": [ "%%capture --no-stderr\n", - "# %pip install flaml[autogen]~=2.0.3" + "# %pip install pyautogen~=0.1.0" ] }, { @@ -52,7 +52,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list_gpt4 = autogen.config_list_from_json(\n", " \"OAI_CONFIG_LIST\",\n", diff --git a/notebook/autogen_agentchat_groupchat_vis.ipynb b/notebook/autogen_agentchat_groupchat_vis.ipynb index 0e844cee4..c8be0ced0 100644 --- a/notebook/autogen_agentchat_groupchat_vis.ipynb +++ b/notebook/autogen_agentchat_groupchat_vis.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -15,14 +15,14 @@ "source": [ "# Auto Generated Agent Chat: Group Chat with Coder and Visualization Critic\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", - "pip install flaml[autogen]\n", + "pip install pyautogen\n", "```" ] }, @@ -33,7 +33,7 @@ "outputs": [], "source": [ "%%capture --no-stderr\n", - "# %pip install flaml[autogen]~=2.0.3" + "# %pip install pyautogen~=0.1.0" ] }, { @@ -52,7 +52,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list_gpt4 = autogen.config_list_from_json(\n", " \"OAI_CONFIG_LIST\",\n", diff --git a/notebook/autogen_agentchat_human_feedback.ipynb b/notebook/autogen_agentchat_human_feedback.ipynb index b876aa52e..2119eb5cf 100644 --- a/notebook/autogen_agentchat_human_feedback.ipynb +++ b/notebook/autogen_agentchat_human_feedback.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -19,16 +19,16 @@ "source": [ "# Auto Generated Agent Chat: Task Solving with Code Generation, Execution, Debugging & Human Feedback\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to solve a challenging math problem with human feedback. Here `AssistantAgent` is an LLM-based agent that can write Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. By setting `human_input_mode` properly, the `UserProxyAgent` can also prompt the user for feedback to `AssistantAgent`. For example, when `human_input_mode` is set to \"ALWAYS\", the `UserProxyAgent` will always prompt the user for feedback. When user feedback is provided, the `UserProxyAgent` will directly pass the feedback to `AssistantAgent`. When no user feedback is provided, the `UserProxyAgent` will execute the code written by `AssistantAgent` and return the execution results (success or failure and corresponding outputs) to `AssistantAgent`.\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", - "pip install flaml[autogen]\n", + "pip install pyautogen\n", "```" ] }, @@ -45,7 +45,7 @@ }, "outputs": [], "source": [ - "# %pip install flaml[autogen]~=2.0.2" + "# %pip install pyautogen~=0.1.0" ] }, { @@ -64,7 +64,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list = autogen.config_list_from_json(\"OAI_CONFIG_LIST\")" ] diff --git a/notebook/autogen_agentchat_planning.ipynb b/notebook/autogen_agentchat_planning.ipynb index c360da979..8622f0cf7 100644 --- a/notebook/autogen_agentchat_planning.ipynb +++ b/notebook/autogen_agentchat_planning.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -19,16 +19,16 @@ "source": [ "# Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "In this notebook, we demonstrate how to use multiple agents to work together and accomplish a task which requires finding info from the web and coding. `AssistantAgent` is an LLM-based agent that can write and debug Python code (in a Python coding block) for a user to execute for a given task. `UserProxyAgent` is an agent which serves as a proxy for a user to execute the code written by `AssistantAgent`. We further create a planning agent for the assistant agent to consult. The planning agent is a variation of the LLM-based `AssistantAgent` with a different system message.\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option and docker:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install pyautogen and docker:\n", "```bash\n", - "pip install flaml[autogen] docker\n", + "pip install pyautogen docker\n", "```" ] }, @@ -45,7 +45,7 @@ }, "outputs": [], "source": [ - "# %pip install flaml[autogen]~=2.0.2 docker" + "# %pip install pyautogen~=0.1.0 docker" ] }, { @@ -70,7 +70,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list = autogen.config_list_from_json(\n", " \"OAI_CONFIG_LIST\",\n", diff --git a/notebook/autogen_agentchat_stream.ipynb b/notebook/autogen_agentchat_stream.ipynb index 87305fb65..1c9c53b80 100644 --- a/notebook/autogen_agentchat_stream.ipynb +++ b/notebook/autogen_agentchat_stream.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -19,16 +19,16 @@ "source": [ "# Interactive LLM Agent Dealing with Data Stream\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "In this notebook, we demonstrate how to use customized agents to continuously acquires news from the web and ask for investment suggestions.\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", - "pip install flaml[autogen]\n", + "pip install pyautogen\n", "```" ] }, @@ -45,7 +45,7 @@ }, "outputs": [], "source": [ - "# %pip install flaml[autogen]~=2.1.0" + "# %pip install pyautogen~=0.1.0" ] }, { @@ -64,7 +64,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list = autogen.config_list_from_json(\"OAI_CONFIG_LIST\")" ] @@ -222,7 +222,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "# create an AssistantAgent instance named \"assistant\"\n", "assistant = autogen.AssistantAgent(\n", diff --git a/notebook/autogen_agentchat_web_info.ipynb b/notebook/autogen_agentchat_web_info.ipynb index 56a16eca4..d8cd9c21f 100644 --- a/notebook/autogen_agentchat_web_info.ipynb +++ b/notebook/autogen_agentchat_web_info.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -19,7 +19,7 @@ "source": [ "# Auto Generated Agent Chat: Solving Tasks Requiring Web Info\n", "\n", - "`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", + "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n", "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n", "\n", "In this notebook, we demonstrate how to use `AssistantAgent` and `UserProxyAgent` to perform tasks which require acquiring info from the web:\n", @@ -30,9 +30,9 @@ "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen] option and docker:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install pyautogen and docker:\n", "```bash\n", - "pip install flaml[autogen]\n", + "pip install pyautogen docker\n", "```" ] }, @@ -49,7 +49,7 @@ }, "outputs": [], "source": [ - "# %pip install flaml[autogen]~=2.0.2 docker" + "# %pip install pyautogen~=0.1.0 docker" ] }, { @@ -68,7 +68,7 @@ "metadata": {}, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "config_list = autogen.config_list_from_json(\n", " \"OAI_CONFIG_LIST\",\n", diff --git a/notebook/autogen_chatgpt_gpt4.ipynb b/notebook/autogen_chatgpt_gpt4.ipynb index 04007d33f..db07b23af 100644 --- a/notebook/autogen_chatgpt_gpt4.ipynb +++ b/notebook/autogen_chatgpt_gpt4.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -21,20 +21,20 @@ "\n", "Licensed under the MIT License.\n", "\n", - "# Use FLAML to Tune ChatGPT\n", + "# Use AutoGen to Tune ChatGPT\n", "\n", - "`flaml.autogen` offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n", + "AutoGen offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n", "Please find documentation about this feature [here](/docs/Use-Cases/AutoGen#enhanced-inference).\n", "\n", "In this notebook, we tune OpenAI ChatGPT (both GPT-3.5 and GPT-4) models for math problem solving. We use [the MATH benchmark](https://crfm.stanford.edu/helm/latest/?group=math_chain_of_thought) for measuring mathematical problem solving on competition math problems with chain-of-thoughts style reasoning.\n", "\n", - "Related link: [Blogpost](https://microsoft.github.io/FLAML/blog/2023/04/21/LLM-tuning-math) based on this experiment.\n", + "Related link: [Blogpost](https://microsoft.github.io/autogen/blog/2023/04/21/LLM-tuning-math) based on this experiment.\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [openai,blendsearch] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install with the [blendsearch] option:\n", "```bash\n", - "pip install flaml[openai,blendsearch]\n", + "pip install \"pyautogen[blendsearch]\"\n", "```" ] }, @@ -51,7 +51,7 @@ }, "outputs": [], "source": [ - "# %pip install flaml[openai,blendsearch] datasets" + "# %pip install \"pyautogen[blendsearch]\" datasets" ] }, { @@ -59,7 +59,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "FLAML has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `autogen.ChatCompletion.tune` and to make a request with the tuned config: `autogen.ChatCompletion.create`. First, we import autogen from flaml:" + "AutoGen has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `autogen.ChatCompletion.tune` and to make a request with the tuned config: `autogen.ChatCompletion.create`. First, we import autogen:" ] }, { @@ -78,7 +78,7 @@ }, "outputs": [], "source": [ - "from flaml import autogen" + "import autogen" ] }, { @@ -293,7 +293,7 @@ }, "outputs": [], "source": [ - "from flaml.autogen.math_utils import eval_math_responses" + "from autogen.math_utils import eval_math_responses" ] }, { @@ -376,7 +376,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": null, "metadata": { "execution": { "iopub.execute_input": "2023-02-13T23:40:56.115383Z", @@ -385,44 +385,7 @@ "shell.execute_reply": "2023-02-13T23:41:55.044973Z" } }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\u001b[32m[I 2023-08-01 22:38:01,549]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[flaml.tune.tune: 08-01 22:38:01] {805} INFO - trial 1 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.36280922847807595}, 'max_tokens': 347, 'n': 10, 'prompt': 0, 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:41:21] {197} INFO - result: {'expected_success': 0.89828529287, 'success': 0.9, 'success_vote': 0.8, 'voted_answer': 'We can write the given sequence as $3^4 \\\\cdot 225, 3^4 \\\\cdot 75, 3^4 \\\\cdot 25, \\\\ldots$. The exponents of 3 are $4, 4, 4, \\\\ldots$, so the sequence consists of the numbers $3^4 \\\\cdot 225, 3^4 \\\\cdot 75, 3^4 \\\\cdot 25, \\\\ldots, 3^4 \\\\cdot 3^0 \\\\cdot 225$. There are $\\\\boxed{4}$ integers in this sequence.', 'votes': 9.1, 'total_cost': 0.07211000000000002, 'cost': 0.07211000000000002, 'inference_cost': 0.0032122500000000003, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.36280922847807595}, 'max_tokens': 347, 'n': 10, 'prompt': 0, 'allow_format_str_template': True}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'top_p': 0.36280922847807595}, 'config/max_tokens': 347, 'config/n': 10, 'config/prompt': 0, 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 200.14745712280273}\n", - "[flaml.tune.tune: 08-01 22:41:21] {805} INFO - trial 2 config: {'temperature_or_top_p': {'temperature': 1.2672964698525508}, 'max_tokens': 470, 'n': 50, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:41:53] {197} INFO - result: {'success_vote': 0, 'total_cost': 0.10703200000000002, 'cost': 0.034922, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.2672964698525508}, 'max_tokens': 470, 'n': 50, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.2672964698525508}, 'config/max_tokens': 470, 'config/n': 50, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 31.767715454101562}\n", - "[flaml.tune.tune: 08-01 22:41:53] {805} INFO - trial 3 config: {'temperature_or_top_p': {'temperature': 1.5210614243979175}, 'max_tokens': 82, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:42:49] {197} INFO - result: {'expected_success': 0.15989600488062986, 'success': 0.2, 'success_vote': 0.2, 'voted_answer': 'Note that to get from 6075 to 2025 or from 2025 to 675, we must divide by 3. Thus the sequence in question, which ends in an ellipsis or a couple of periods dots indicating more members come next, also begins with the labeling \"700, 300.\" Recall from arithmetic pattern insight to sports like basketball and from looking in our answer choices section', 'votes': 0.7, 'total_cost': 0.13852200000000003, 'cost': 0.031490000000000004, 'inference_cost': 0.0015442499999999998, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.5210614243979175}, 'max_tokens': 82, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.5210614243979175}, 'config/max_tokens': 82, 'config/n': 9, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 55.53780817985535}\n", - "[flaml.tune.tune: 08-01 22:42:49] {805} INFO - trial 4 config: {'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'max_tokens': 231, 'n': 81, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:43:07] {197} INFO - result: {'success_vote': 0, 'total_cost': 0.18828000000000003, 'cost': 0.04975800000000001, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'max_tokens': 231, 'n': 81, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'top_p': 0.003948266327914451}, 'config/max_tokens': 231, 'config/n': 81, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 18.070116996765137}\n", - "[flaml.tune.tune: 08-01 22:43:07] {805} INFO - trial 5 config: {'temperature_or_top_p': {'top_p': 0.29187606817063316}, 'max_tokens': 781, 'n': 71, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:43:07] {197} INFO - result: {'inference_cost': inf, 'success_vote': -inf, 'cost': 0, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.29187606817063316}, 'max_tokens': 781, 'n': 71, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'top_p': 0.29187606817063316}, 'config/max_tokens': 781, 'config/n': 71, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 0.0005497932434082031}\n", - "[flaml.tune.tune: 08-01 22:43:07] {805} INFO - trial 6 config: {'temperature_or_top_p': {'temperature': 0.7466815201029384}, 'max_tokens': 375, 'n': 44, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:46:28] {197} INFO - result: {'expected_success': 0.9818164607828072, 'success': 1.0, 'success_vote': 0.95, 'voted_answer': 'To find the number of integers in the sequence, we need to find when each term becomes less than 1. \\n\\nStarting with 6075, we divide by 3 to get $\\\\frac{6075}{3} = 2025$. Since 2025 is an integer, it is included in the sequence.\\n\\nDividing 2025 by 3, we get $\\\\frac{2025}{3} = 675$. Again, 675 is an integer, so it is included in the sequence.\\n\\nIf we divide 675 by 3, we get $\\\\frac{675}{3} = 225$. 225 is an integer, so it is included in the sequence.\\n\\nDividing 225 by 3, we get $\\\\frac{225}{3} = 75$. 75 is an integer, so it is included in the sequence.\\n\\nDividing 75 by 3, we get $\\\\frac{75}{3} = 25$. 25 is an integer, so it is included in the sequence.\\n\\nIf we divide 25 by 3, we get $\\\\frac{25}{3} \\\\approx 8.3333$, which is not an integer. Thus, 25 is the last integer in the sequence.\\n\\nThere are a total of $\\\\boxed{6}$ integers in the sequence.', 'votes': 34.85, 'total_cost': 0.463802, 'cost': 0.27552199999999993, 'inference_cost': 0.01310685, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 0.7466815201029384}, 'max_tokens': 375, 'n': 44, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 0.7466815201029384}, 'config/max_tokens': 375, 'config/n': 44, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 201.2768588066101}\n", - "[flaml.tune.tune: 08-01 22:46:28] {805} INFO - trial 7 config: {'temperature_or_top_p': {'top_p': 0.5131382425543909}, 'max_tokens': 350, 'n': 60, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:47:09] {197} INFO - result: {'success_vote': 0, 'total_cost': 0.52441, 'cost': 0.060607999999999995, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.5131382425543909}, 'max_tokens': 350, 'n': 60, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'top_p': 0.5131382425543909}, 'config/max_tokens': 350, 'config/n': 60, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 41.3958899974823}\n", - "[flaml.tune.tune: 08-01 22:47:09] {805} INFO - trial 8 config: {'temperature_or_top_p': {'temperature': 1.8172977616173365}, 'max_tokens': 129, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:48:34] {197} INFO - result: {'expected_success': 0.06535605838853817, 'success': 0.1, 'success_vote': 0.1, 'voted_answer': 'To find out how many integers are in this sequence, we must determine the number of times 3 is a factor being successively divisible cases.\\n\\n\\nFor modern thought:\\nThe ultimate disaster approach ,\\nwill hit eighty year compound,\\ncos thirty pieces, successful trip necessitate; pounds prove evenly\\nHot before four boxes accumulate closely superior statistics prove Yet pale-eyed visionary spite.\\n\\n\\n\\n\\n\\nAnalyzer-based cipher elements yielded intervals This outcome integers.A reason.Brief Inspection Of available objects imply Par near Often Reason via options \\n\\nThe Ratio sum leaves ten; Five.\\n\\nReal Analy access tells not answer right I vary combinations&find divisions Prompt are strongSo inspection Replace Reverse', 'votes': 0.35, 'total_cost': 0.5708920000000002, 'cost': 0.046482, 'inference_cost': 0.00229385, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.8172977616173365}, 'max_tokens': 129, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.8172977616173365}, 'config/max_tokens': 129, 'config/n': 9, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 84.15163469314575}\n", - "[flaml.tune.tune: 08-01 22:48:34] {805} INFO - trial 9 config: {'temperature_or_top_p': {'temperature': 1.6573626526153533}, 'max_tokens': 57, 'n': 63, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:49:36] {197} INFO - result: {'expected_success': 0.12519255101013854, 'success': 0.15, 'success_vote': 0.15, 'voted_answer': 'Let the original term of the sequence be $x$. There are $796-43= \\\\boxed{753}$ sequences/pro-edits until term the when you divide the sequence becomes less vo/volume of OR counting totals that =prime-number?(-)+lifeisticment real!', 'votes': 1.1, 'total_cost': 0.71616, 'cost': 0.145268, 'inference_cost': 0.007233149999999999, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.6573626526153533}, 'max_tokens': 57, 'n': 63, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.6573626526153533}, 'config/max_tokens': 57, 'config/n': 63, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 62.10028266906738}\n", - "[flaml.tune.tune: 08-01 22:49:36] {805} INFO - trial 10 config: {'temperature_or_top_p': {'top_p': 0.1989475396788123}, 'max_tokens': 650, 'n': 35, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:51:50] {197} INFO - result: {'expected_success': 0.8499999999999934, 'success': 0.85, 'success_vote': 0.85, 'voted_answer': 'We can write the given sequence as $3^4 \\\\cdot 5^2, 3^4 \\\\cdot 5^1, 3^4 \\\\cdot 5^0, \\\\ldots$. We want to find the number of integers in this sequence. \\n\\nNotice that the exponent of 3 stays constant at 4, while the exponent of 5 decreases by 1 each time. We want to find the largest integer $n$ such that $3^4 \\\\cdot 5^n$ is an integer. \\n\\nSince $3^4$ is an integer, we only need to consider the exponent of 5. We want $5^n$ to be an integer, so $n$ must be nonnegative. However, we also want $5^n$ to be a factor of $3^4$, so $n$ must be less than or equal to 4. \\n\\nTherefore, the possible values of $n$ are 0, 1, 2, 3, and 4. There are $\\\\boxed{5}$ integers in the sequence.', 'votes': 33.8, 'total_cost': 0.9523240000000001, 'cost': 0.23616399999999999, 'inference_cost': 0.010300450000000001, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.1989475396788123}, 'max_tokens': 650, 'n': 35, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'top_p': 0.1989475396788123}, 'config/max_tokens': 650, 'config/n': 35, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 134.67861104011536}\n", - "[flaml.tune.tune: 08-01 22:51:50] {805} INFO - trial 11 config: {'temperature_or_top_p': {'temperature': 1.7678729591223725}, 'max_tokens': 132, 'n': 17, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}\n", - "[flaml.tune.tune: 08-01 22:52:44] {197} INFO - result: {'success_vote': 0, 'total_cost': 1.000234, 'cost': 0.04791, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.7678729591223725}, 'max_tokens': 132, 'n': 17, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.7678729591223725}, 'config/max_tokens': 132, 'config/n': 17, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 53.21780872344971}\n", - "[flaml.tune.tune: 08-01 22:52:44] {828} WARNING - fail to sample a trial for 100 times in a row, stopping.\n" - ] - } - ], + "outputs": [], "source": [ "import logging\n", "\n", @@ -454,7 +417,7 @@ "source": [ "### Output tuning results\n", "\n", - "After the tuning, we can print out the config and the result found by FLAML:" + "After the tuning, we can print out the config and the result found by AutoGen, which uses flaml for tuning." ] }, { @@ -1723,7 +1686,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": null, "metadata": { "execution": { "iopub.execute_input": "2023-02-13T23:41:56.042764Z", @@ -1732,216 +1695,7 @@ "shell.execute_reply": "2023-02-13T23:53:05.596603Z" } }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[flaml.autogen.oai.completion: 08-01 22:55:55] {916} INFO - evaluating data instance 0\n", - "[flaml.autogen.oai.completion: 08-01 22:56:09] {916} INFO - evaluating data instance 1\n", - "[flaml.autogen.oai.completion: 08-01 22:56:20] {916} INFO - evaluating data instance 2\n", - "[flaml.autogen.oai.completion: 08-01 22:56:28] {916} INFO - evaluating data instance 3\n", - "[flaml.autogen.oai.completion: 08-01 22:56:34] {916} INFO - evaluating data instance 4\n", - "[flaml.autogen.oai.completion: 08-01 22:56:44] {916} INFO - evaluating data instance 5\n", - "[flaml.autogen.oai.completion: 08-01 22:56:57] {916} INFO - evaluating data instance 6\n", - "[flaml.autogen.oai.completion: 08-01 22:57:12] {916} INFO - evaluating data instance 7\n", - "[flaml.autogen.oai.completion: 08-01 22:57:20] {916} INFO - evaluating data instance 8\n", - "[flaml.autogen.oai.completion: 08-01 22:57:24] {916} INFO - evaluating data instance 9\n", - "[flaml.autogen.oai.completion: 08-01 22:57:34] {916} INFO - evaluating data instance 10\n", - "[flaml.autogen.oai.completion: 08-01 22:57:43] {916} INFO - evaluating data instance 11\n", - "[flaml.autogen.oai.completion: 08-01 22:57:52] {916} INFO - evaluating data instance 12\n", - "[flaml.autogen.oai.completion: 08-01 22:58:00] {916} INFO - evaluating data instance 13\n", - "[flaml.autogen.oai.completion: 08-01 22:58:08] {916} INFO - evaluating data instance 14\n", - "[flaml.autogen.oai.completion: 08-01 22:58:14] {916} INFO - evaluating data instance 15\n", - "[flaml.autogen.oai.completion: 08-01 22:58:22] {916} INFO - evaluating data instance 16\n", - "[flaml.autogen.oai.completion: 08-01 22:58:29] {916} INFO - evaluating data instance 17\n", - "[flaml.autogen.oai.completion: 08-01 22:58:40] {916} INFO - evaluating data instance 18\n", - "[flaml.autogen.oai.completion: 08-01 22:58:48] {916} INFO - evaluating data instance 19\n", - "[flaml.autogen.oai.completion: 08-01 22:58:57] {916} INFO - evaluating data instance 20\n", - "[flaml.autogen.oai.completion: 08-01 22:59:15] {916} INFO - evaluating data instance 21\n", - "[flaml.autogen.oai.completion: 08-01 22:59:29] {916} INFO - evaluating data instance 22\n", - "[flaml.autogen.oai.completion: 08-01 22:59:41] {916} INFO - evaluating data instance 23\n", - "[flaml.autogen.oai.completion: 08-01 22:59:54] {916} INFO - evaluating data instance 24\n", - "[flaml.autogen.oai.completion: 08-01 23:00:07] {916} INFO - evaluating data instance 25\n", - "[flaml.autogen.oai.completion: 08-01 23:00:24] {916} INFO - evaluating data instance 26\n", - "[flaml.autogen.oai.completion: 08-01 23:00:39] {916} INFO - evaluating data instance 27\n", - "[flaml.autogen.oai.completion: 08-01 23:00:55] {916} INFO - evaluating data instance 28\n", - "[flaml.autogen.oai.completion: 08-01 23:01:11] {916} INFO - evaluating data instance 29\n", - "[flaml.autogen.oai.completion: 08-01 23:01:26] {916} INFO - evaluating data instance 30\n", - "[flaml.autogen.oai.completion: 08-01 23:01:35] {916} INFO - evaluating data instance 31\n", - "[flaml.autogen.oai.completion: 08-01 23:01:46] {916} INFO - evaluating data instance 32\n", - "[flaml.autogen.oai.completion: 08-01 23:01:54] {916} INFO - evaluating data instance 33\n", - "[flaml.autogen.oai.completion: 08-01 23:02:03] {916} INFO - evaluating data instance 34\n", - "[flaml.autogen.oai.completion: 08-01 23:02:11] {916} INFO - evaluating data instance 35\n", - "[flaml.autogen.oai.completion: 08-01 23:02:27] {916} INFO - evaluating data instance 36\n", - "[flaml.autogen.oai.completion: 08-01 23:02:40] {916} INFO - evaluating data instance 37\n", - "[flaml.autogen.oai.completion: 08-01 23:02:46] {916} INFO - evaluating data instance 38\n", - "[flaml.autogen.oai.completion: 08-01 23:02:56] {916} INFO - evaluating data instance 39\n", - "[flaml.autogen.oai.completion: 08-01 23:03:06] {916} INFO - evaluating data instance 40\n", - "[flaml.autogen.oai.completion: 08-01 23:03:15] {916} INFO - evaluating data instance 41\n", - "[flaml.autogen.oai.completion: 08-01 23:03:23] {916} INFO - evaluating data instance 42\n", - "[flaml.autogen.oai.completion: 08-01 23:03:30] {916} INFO - evaluating data instance 43\n", - "[flaml.autogen.oai.completion: 08-01 23:03:38] {916} INFO - evaluating data instance 44\n", - "[flaml.autogen.oai.completion: 08-01 23:03:49] {916} INFO - evaluating data instance 45\n", - "[flaml.autogen.oai.completion: 08-01 23:03:55] {916} INFO - evaluating data instance 46\n", - "[flaml.autogen.oai.completion: 08-01 23:04:02] {916} INFO - evaluating data instance 47\n", - "[flaml.autogen.oai.completion: 08-01 23:04:14] {916} INFO - evaluating data instance 48\n", - "[flaml.autogen.oai.completion: 08-01 23:04:30] {916} INFO - evaluating data instance 49\n", - "[flaml.autogen.oai.completion: 08-01 23:04:42] {916} INFO - evaluating data instance 50\n", - "[flaml.autogen.oai.completion: 08-01 23:04:53] {916} INFO - evaluating data instance 51\n", - "[flaml.autogen.oai.completion: 08-01 23:05:05] {916} INFO - evaluating data instance 52\n", - "[flaml.autogen.oai.completion: 08-01 23:05:10] {916} INFO - evaluating data instance 53\n", - "[flaml.autogen.oai.completion: 08-01 23:05:22] {916} INFO - evaluating data instance 54\n", - "[flaml.autogen.oai.completion: 08-01 23:05:31] {916} INFO - evaluating data instance 55\n", - "[flaml.autogen.oai.completion: 08-01 23:05:43] {916} INFO - evaluating data instance 56\n", - "[flaml.autogen.oai.completion: 08-01 23:05:49] {916} INFO - evaluating data instance 57\n", - "[flaml.autogen.oai.completion: 08-01 23:05:59] {916} INFO - evaluating data instance 58\n", - "[flaml.autogen.oai.completion: 08-01 23:06:12] {916} INFO - evaluating data instance 59\n", - "[flaml.autogen.oai.completion: 08-01 23:06:20] {916} INFO - evaluating data instance 60\n", - "[flaml.autogen.oai.completion: 08-01 23:06:32] {916} INFO - evaluating data instance 61\n", - "[flaml.autogen.oai.completion: 08-01 23:06:42] {916} INFO - evaluating data instance 62\n", - "[flaml.autogen.oai.completion: 08-01 23:06:54] {916} INFO - evaluating data instance 63\n", - "[flaml.autogen.oai.completion: 08-01 23:07:08] {916} INFO - evaluating data instance 64\n", - "[flaml.autogen.oai.completion: 08-01 23:07:22] {916} INFO - evaluating data instance 65\n", - "[flaml.autogen.oai.completion: 08-01 23:07:34] {916} INFO - evaluating data instance 66\n", - "[flaml.autogen.oai.completion: 08-01 23:07:43] {916} INFO - evaluating data instance 67\n", - "[flaml.autogen.oai.completion: 08-01 23:07:49] {916} INFO - evaluating data instance 68\n", - "[flaml.autogen.oai.completion: 08-01 23:08:00] {916} INFO - evaluating data instance 69\n", - "[flaml.autogen.oai.completion: 08-01 23:08:12] {916} INFO - evaluating data instance 70\n", - "[flaml.autogen.oai.completion: 08-01 23:08:27] {916} INFO - evaluating data instance 71\n", - "[flaml.autogen.oai.completion: 08-01 23:08:36] {916} INFO - evaluating data instance 72\n", - "[flaml.autogen.oai.completion: 08-01 23:08:50] {916} INFO - evaluating data instance 73\n", - "[flaml.autogen.oai.completion: 08-01 23:08:58] {916} INFO - evaluating data instance 74\n", - "[flaml.autogen.oai.completion: 08-01 23:09:10] {916} INFO - evaluating data instance 75\n", - "[flaml.autogen.oai.completion: 08-01 23:09:19] {916} INFO - evaluating data instance 76\n", - "[flaml.autogen.oai.completion: 08-01 23:09:30] {916} INFO - evaluating data instance 77\n", - "[flaml.autogen.oai.completion: 08-01 23:09:38] {916} INFO - evaluating data instance 78\n", - "[flaml.autogen.oai.completion: 08-01 23:09:48] {916} INFO - evaluating data instance 79\n", - "[flaml.autogen.oai.completion: 08-01 23:09:58] {916} INFO - evaluating data instance 80\n", - "[flaml.autogen.oai.completion: 08-01 23:10:08] {916} INFO - evaluating data instance 81\n", - "[flaml.autogen.oai.completion: 08-01 23:10:19] {916} INFO - evaluating data instance 82\n", - "[flaml.autogen.oai.completion: 08-01 23:10:32] {916} INFO - evaluating data instance 83\n", - "[flaml.autogen.oai.completion: 08-01 23:10:37] {916} INFO - evaluating data instance 84\n", - "[flaml.autogen.oai.completion: 08-01 23:10:52] {916} INFO - evaluating data instance 85\n", - "[flaml.autogen.oai.completion: 08-01 23:11:07] {916} INFO - evaluating data instance 86\n", - "[flaml.autogen.oai.completion: 08-01 23:11:22] {916} INFO - evaluating data instance 87\n", - "[flaml.autogen.oai.completion: 08-01 23:11:33] {916} INFO - evaluating data instance 88\n", - "[flaml.autogen.oai.completion: 08-01 23:11:48] {916} INFO - evaluating data instance 89\n", - "[flaml.autogen.oai.completion: 08-01 23:11:55] {916} INFO - evaluating data instance 90\n", - "[flaml.autogen.oai.completion: 08-01 23:12:04] {916} INFO - evaluating data instance 91\n", - "[flaml.autogen.oai.completion: 08-01 23:12:15] {916} INFO - evaluating data instance 92\n", - "[flaml.autogen.oai.completion: 08-01 23:12:27] {916} INFO - evaluating data instance 93\n", - "[flaml.autogen.oai.completion: 08-01 23:12:39] {916} INFO - evaluating data instance 94\n", - "[flaml.autogen.oai.completion: 08-01 23:12:55] {916} INFO - evaluating data instance 95\n", - "[flaml.autogen.oai.completion: 08-01 23:13:05] {916} INFO - evaluating data instance 96\n", - "[flaml.autogen.oai.completion: 08-01 23:13:17] {916} INFO - evaluating data instance 97\n", - "[flaml.autogen.oai.completion: 08-01 23:13:30] {916} INFO - evaluating data instance 98\n", - "[flaml.autogen.oai.completion: 08-01 23:13:43] {916} INFO - evaluating data instance 99\n", - "[flaml.autogen.oai.completion: 08-01 23:13:51] {916} INFO - evaluating data instance 100\n", - "[flaml.autogen.oai.completion: 08-01 23:14:04] {916} INFO - evaluating data instance 101\n", - "[flaml.autogen.oai.completion: 08-01 23:14:09] {916} INFO - evaluating data instance 102\n", - "[flaml.autogen.oai.completion: 08-01 23:14:20] {916} INFO - evaluating data instance 103\n", - "[flaml.autogen.oai.completion: 08-01 23:14:32] {916} INFO - evaluating data instance 104\n", - "[flaml.autogen.oai.completion: 08-01 23:14:46] {916} INFO - evaluating data instance 105\n", - "[flaml.autogen.oai.completion: 08-01 23:14:59] {916} INFO - evaluating data instance 106\n", - "[flaml.autogen.oai.completion: 08-01 23:15:13] {916} INFO - evaluating data instance 107\n", - "[flaml.autogen.oai.completion: 08-01 23:15:23] {916} INFO - evaluating data instance 108\n", - "[flaml.autogen.oai.completion: 08-01 23:15:34] {916} INFO - evaluating data instance 109\n", - "[flaml.autogen.oai.completion: 08-01 23:15:46] {916} INFO - evaluating data instance 110\n", - "[flaml.autogen.oai.completion: 08-01 23:15:56] {916} INFO - evaluating data instance 111\n", - "[flaml.autogen.oai.completion: 08-01 23:16:10] {916} INFO - evaluating data instance 112\n", - "[flaml.autogen.oai.completion: 08-01 23:16:15] {916} INFO - evaluating data instance 113\n", - "[flaml.autogen.oai.completion: 08-01 23:16:27] {916} INFO - evaluating data instance 114\n", - "[flaml.autogen.oai.completion: 08-01 23:16:35] {916} INFO - evaluating data instance 115\n", - "[flaml.autogen.oai.completion: 08-01 23:16:48] {916} INFO - evaluating data instance 116\n", - "[flaml.autogen.oai.completion: 08-01 23:17:02] {916} INFO - evaluating data instance 117\n", - "[flaml.autogen.oai.completion: 08-01 23:17:14] {916} INFO - evaluating data instance 118\n", - "[flaml.autogen.oai.completion: 08-01 23:17:18] {916} INFO - evaluating data instance 119\n", - "[flaml.autogen.oai.completion: 08-01 23:17:31] {916} INFO - evaluating data instance 120\n", - "[flaml.autogen.oai.completion: 08-01 23:17:37] {916} INFO - evaluating data instance 121\n", - "[flaml.autogen.oai.completion: 08-01 23:17:46] {916} INFO - evaluating data instance 122\n", - "[flaml.autogen.oai.completion: 08-01 23:17:53] {916} INFO - evaluating data instance 123\n", - "[flaml.autogen.oai.completion: 08-01 23:18:00] {916} INFO - evaluating data instance 124\n", - "[flaml.autogen.oai.completion: 08-01 23:18:11] {916} INFO - evaluating data instance 125\n", - "[flaml.autogen.oai.completion: 08-01 23:18:17] {916} INFO - evaluating data instance 126\n", - "[flaml.autogen.oai.completion: 08-01 23:18:27] {916} INFO - evaluating data instance 127\n", - "[flaml.autogen.oai.completion: 08-01 23:18:30] {916} INFO - evaluating data instance 128\n", - "[flaml.autogen.oai.completion: 08-01 23:18:45] {916} INFO - evaluating data instance 129\n", - "[flaml.autogen.oai.completion: 08-01 23:18:53] {916} INFO - evaluating data instance 130\n", - "[flaml.autogen.oai.completion: 08-01 23:19:03] {916} INFO - evaluating data instance 131\n", - "[flaml.autogen.oai.completion: 08-01 23:19:07] {916} INFO - evaluating data instance 132\n", - "[flaml.autogen.oai.completion: 08-01 23:19:15] {916} INFO - evaluating data instance 133\n", - "[flaml.autogen.oai.completion: 08-01 23:19:29] {916} INFO - evaluating data instance 134\n", - "[flaml.autogen.oai.completion: 08-01 23:19:44] {916} INFO - evaluating data instance 135\n", - "[flaml.autogen.oai.completion: 08-01 23:19:55] {916} INFO - evaluating data instance 136\n", - "[flaml.autogen.oai.completion: 08-01 23:20:02] {916} INFO - evaluating data instance 137\n", - "[flaml.autogen.oai.completion: 08-01 23:20:15] {916} INFO - evaluating data instance 138\n", - "[flaml.autogen.oai.completion: 08-01 23:20:24] {916} INFO - evaluating data instance 139\n", - "[flaml.autogen.oai.completion: 08-01 23:20:34] {916} INFO - evaluating data instance 140\n", - "[flaml.autogen.oai.completion: 08-01 23:20:40] {916} INFO - evaluating data instance 141\n", - "[flaml.autogen.oai.completion: 08-01 23:20:49] {916} INFO - evaluating data instance 142\n", - "[flaml.autogen.oai.completion: 08-01 23:20:55] {916} INFO - evaluating data instance 143\n", - "[flaml.autogen.oai.completion: 08-01 23:21:05] {916} INFO - evaluating data instance 144\n", - "[flaml.autogen.oai.completion: 08-01 23:21:10] {916} INFO - evaluating data instance 145\n", - "[flaml.autogen.oai.completion: 08-01 23:21:17] {916} INFO - evaluating data instance 146\n", - "[flaml.autogen.oai.completion: 08-01 23:21:25] {916} INFO - evaluating data instance 147\n", - "[flaml.autogen.oai.completion: 08-01 23:21:38] {916} INFO - evaluating data instance 148\n", - "[flaml.autogen.oai.completion: 08-01 23:21:54] {916} INFO - evaluating data instance 149\n", - "[flaml.autogen.oai.completion: 08-01 23:22:05] {916} INFO - evaluating data instance 150\n", - "[flaml.autogen.oai.completion: 08-01 23:22:13] {916} INFO - evaluating data instance 151\n", - "[flaml.autogen.oai.completion: 08-01 23:22:24] {916} INFO - evaluating data instance 152\n", - "[flaml.autogen.oai.completion: 08-01 23:22:35] {916} INFO - evaluating data instance 153\n", - "[flaml.autogen.oai.completion: 08-01 23:22:44] {916} INFO - evaluating data instance 154\n", - "[flaml.autogen.oai.completion: 08-01 23:22:53] {916} INFO - evaluating data instance 155\n", - "[flaml.autogen.oai.completion: 08-01 23:23:01] {916} INFO - evaluating data instance 156\n", - "[flaml.autogen.oai.completion: 08-01 23:23:16] {916} INFO - evaluating data instance 157\n", - "[flaml.autogen.oai.completion: 08-01 23:23:23] {916} INFO - evaluating data instance 158\n", - "[flaml.autogen.oai.completion: 08-01 23:23:31] {916} INFO - evaluating data instance 159\n", - "[flaml.autogen.oai.completion: 08-01 23:23:44] {916} INFO - evaluating data instance 160\n", - "[flaml.autogen.oai.completion: 08-01 23:23:57] {916} INFO - evaluating data instance 161\n", - "[flaml.autogen.oai.completion: 08-01 23:24:03] {916} INFO - evaluating data instance 162\n", - "[flaml.autogen.oai.completion: 08-01 23:24:09] {916} INFO - evaluating data instance 163\n", - "[flaml.autogen.oai.completion: 08-01 23:24:16] {916} INFO - evaluating data instance 164\n", - "[flaml.autogen.oai.completion: 08-01 23:24:28] {916} INFO - evaluating data instance 165\n", - "[flaml.autogen.oai.completion: 08-01 23:24:39] {916} INFO - evaluating data instance 166\n", - "[flaml.autogen.oai.completion: 08-01 23:24:55] {916} INFO - evaluating data instance 167\n", - "[flaml.autogen.oai.completion: 08-01 23:25:00] {916} INFO - evaluating data instance 168\n", - "[flaml.autogen.oai.completion: 08-01 23:25:16] {916} INFO - evaluating data instance 169\n", - "[flaml.autogen.oai.completion: 08-01 23:25:23] {916} INFO - evaluating data instance 170\n", - "[flaml.autogen.oai.completion: 08-01 23:25:31] {916} INFO - evaluating data instance 171\n", - "[flaml.autogen.oai.completion: 08-01 23:25:36] {916} INFO - evaluating data instance 172\n", - "[flaml.autogen.oai.completion: 08-01 23:25:44] {916} INFO - evaluating data instance 173\n", - "[flaml.autogen.oai.completion: 08-01 23:25:56] {916} INFO - evaluating data instance 174\n", - "[flaml.autogen.oai.completion: 08-01 23:26:07] {916} INFO - evaluating data instance 175\n", - "[flaml.autogen.oai.completion: 08-01 23:26:21] {916} INFO - evaluating data instance 176\n", - "[flaml.autogen.oai.completion: 08-01 23:26:27] {916} INFO - evaluating data instance 177\n", - "[flaml.autogen.oai.completion: 08-01 23:26:34] {916} INFO - evaluating data instance 178\n", - "[flaml.autogen.oai.completion: 08-01 23:26:47] {916} INFO - evaluating data instance 179\n", - "[flaml.autogen.oai.completion: 08-01 23:27:01] {916} INFO - evaluating data instance 180\n", - "[flaml.autogen.oai.completion: 08-01 23:27:15] {916} INFO - evaluating data instance 181\n", - "[flaml.autogen.oai.completion: 08-01 23:27:22] {916} INFO - evaluating data instance 182\n", - "[flaml.autogen.oai.completion: 08-01 23:27:29] {916} INFO - evaluating data instance 183\n", - "[flaml.autogen.oai.completion: 08-01 23:27:40] {916} INFO - evaluating data instance 184\n", - "[flaml.autogen.oai.completion: 08-01 23:27:49] {916} INFO - evaluating data instance 185\n", - "[flaml.autogen.oai.completion: 08-01 23:27:55] {916} INFO - evaluating data instance 186\n", - "[flaml.autogen.oai.completion: 08-01 23:28:02] {916} INFO - evaluating data instance 187\n", - "[flaml.autogen.oai.completion: 08-01 23:28:06] {916} INFO - evaluating data instance 188\n", - "[flaml.autogen.oai.completion: 08-01 23:28:18] {916} INFO - evaluating data instance 189\n", - "[flaml.autogen.oai.completion: 08-01 23:28:27] {916} INFO - evaluating data instance 190\n", - "[flaml.autogen.oai.completion: 08-01 23:28:37] {916} INFO - evaluating data instance 191\n", - "[flaml.autogen.oai.completion: 08-01 23:28:49] {916} INFO - evaluating data instance 192\n", - "[flaml.autogen.oai.completion: 08-01 23:29:01] {916} INFO - evaluating data instance 193\n", - "[flaml.autogen.oai.completion: 08-01 23:29:14] {916} INFO - evaluating data instance 194\n", - "[flaml.autogen.oai.completion: 08-01 23:29:21] {916} INFO - evaluating data instance 195\n", - "[flaml.autogen.oai.completion: 08-01 23:29:30] {916} INFO - evaluating data instance 196\n", - "[flaml.autogen.oai.completion: 08-01 23:29:42] {916} INFO - evaluating data instance 197\n", - "[flaml.autogen.oai.completion: 08-01 23:29:56] {916} INFO - evaluating data instance 198\n", - "[flaml.autogen.oai.completion: 08-01 23:30:04] {916} INFO - evaluating data instance 199\n", - "[flaml.autogen.oai.completion: 08-01 23:30:20] {916} INFO - evaluating data instance 200\n", - "performance on test data with the tuned config: {'expected_success': 0.9914855260776184, 'success': 0.9950248756218906, 'success_vote': 0.9203980099502488, 'votes': 31.582089552238806, 'cost': 2.697486000000001, 'inference_cost': 0.01342032835820896}\n" - ] - } - ], + "outputs": [], "source": [ "# result = autogen.ChatCompletion.test(test_data, logging_level=logging.INFO, config_list=config_list, **config)\n", "# print(\"performance on test data with the tuned config:\", result)" @@ -2044,7 +1798,7 @@ "source": [ "We find that the 'success_vote' metric is increased at the cost of exceeding the inference budget. But the tuned configuration has both higher 'success_vote' (91% vs. 87%) and lower average inference cost ($0.015 vs. $0.037 per instance).\n", "\n", - "A developer could use flaml to tune the configuration to satisfy the target inference budget while maximizing the value out of it." + "A developer could use AutoGen to tune the configuration to satisfy the target inference budget while maximizing the value out of it." ] } ], diff --git a/notebook/autogen_openai_completion.ipynb b/notebook/autogen_openai_completion.ipynb index 0c4b0d0ff..5ffde3cf0 100644 --- a/notebook/autogen_openai_completion.ipynb +++ b/notebook/autogen_openai_completion.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\"Open" + "\"Open" ] }, { @@ -21,18 +21,18 @@ "\n", "Licensed under the MIT License.\n", "\n", - "# Use FLAML to Tune OpenAI Models\n", + "# Use AutoGen to Tune OpenAI Models\n", "\n", - "`flaml.autogen` offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The research study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n", + "AutoGen offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. The research study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n", "Please find documentation about this feature [here](/docs/Use-Cases/AutoGen#enhanced-inference).\n", "\n", "In this notebook, we tune OpenAI models for code generation. We use [the HumanEval benchmark](https://huggingface.co/datasets/openai_humaneval) released by OpenAI for synthesizing programs from docstrings.\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [autogen,blendsearch] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install with the [blendsearch] option:\n", "```bash\n", - "pip install flaml[autogen,blendsearch]\n", + "pip install pyautogen[blendsearch]\n", "```" ] }, @@ -49,7 +49,7 @@ }, "outputs": [], "source": [ - "# %pip install flaml[autogen,blendsearch]~=2.0.0 datasets" + "# %pip install \"pyautogen[blendsearch]~=0.1.0\" datasets" ] }, { @@ -81,7 +81,7 @@ }, "outputs": [], "source": [ - "from flaml import autogen\n", + "import autogen\n", "\n", "endpoint_list = autogen.config_list_openai_aoai()\n", "# the endpoint_list looks like this:\n", @@ -362,7 +362,7 @@ "\n", "## Use the tuning data to find a good configuration\n", "\n", - "FLAML has provided an API for hyperparameter optimization of OpenAI models: `autogen.Completion.tune` and to make a request with the tuned config: `autogen.Completion.create`.\n", + "AutoGen has provided an API for hyperparameter optimization of OpenAI models: `autogen.Completion.tune` and to make a request with the tuned config: `autogen.Completion.create`.\n", "\n", "For (local) reproducibility and cost efficiency, we cache responses from OpenAI with a controllable seed." ] diff --git a/notebook/research/autogen_code.ipynb b/notebook/research/autogen_code.ipynb index 653bd9439..be6c31b3c 100644 --- a/notebook/research/autogen_code.ipynb +++ b/notebook/research/autogen_code.ipynb @@ -13,15 +13,15 @@ "\n", "Licensed under the MIT License.\n", "\n", - "# Use FLAML to Optimize Code Generation Performance\n", + "# Use AutoGen to Optimize Code Generation Performance\n", "\n", "In this notebook, we optimize OpenAI models for code generation. We use [the HumanEval benchmark](https://huggingface.co/datasets/openai_humaneval) released by OpenAI for synthesizing programs from docstrings.\n", "\n", - "Related link: [Blogpost](https://microsoft.github.io/FLAML/blog/2023/05/18/GPT-adaptive-humaneval) based on this experiment.\n", + "Related link: [Blogpost](https://microsoft.github.io/autogen/blog/2023/05/18/GPT-adaptive-humaneval) based on this experiment.\n", "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [autogen] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", "pip install flaml[autogen]==1.2.2\n", "```" diff --git a/notebook/research/math_level5counting.ipynb b/notebook/research/math_level5counting.ipynb index d929000c8..ab6f123ba 100644 --- a/notebook/research/math_level5counting.ipynb +++ b/notebook/research/math_level5counting.ipynb @@ -19,7 +19,7 @@ "\n", "## Requirements\n", "\n", - "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [openai] option:\n", + "AutoGen requires `Python>=3.8`. To run this notebook example, please install:\n", "```bash\n", "pip install flaml[openai]==1.2.2\n", "```" diff --git a/test/agentchat/test_assistant_agent.py b/test/agentchat/test_assistant_agent.py index afd5cd768..c36cc35e4 100644 --- a/test/agentchat/test_assistant_agent.py +++ b/test/agentchat/test_assistant_agent.py @@ -1,8 +1,8 @@ import os import sys import pytest -from flaml import autogen -from flaml.autogen.agentchat import AssistantAgent, UserProxyAgent +import autogen +from autogen.agentchat import AssistantAgent, UserProxyAgent KEY_LOC = "notebook" OAI_CONFIG_LIST = "OAI_CONFIG_LIST" diff --git a/test/agentchat/test_async.py b/test/agentchat/test_async.py index 8d523ecf8..9a806e6af 100644 --- a/test/agentchat/test_async.py +++ b/test/agentchat/test_async.py @@ -1,5 +1,5 @@ import asyncio -from flaml import autogen +import autogen from test_assistant_agent import KEY_LOC, OAI_CONFIG_LIST diff --git a/test/agentchat/test_conversable_agent.py b/test/agentchat/test_conversable_agent.py index 23f4a223c..d7ecc388f 100644 --- a/test/agentchat/test_conversable_agent.py +++ b/test/agentchat/test_conversable_agent.py @@ -1,5 +1,5 @@ import pytest -from flaml.autogen.agentchat import ConversableAgent +from autogen.agentchat import ConversableAgent def test_trigger(): diff --git a/test/agentchat/test_groupchat.py b/test/agentchat/test_groupchat.py index 51db4cb2d..5c5d3fb82 100644 --- a/test/agentchat/test_groupchat.py +++ b/test/agentchat/test_groupchat.py @@ -1,4 +1,4 @@ -from flaml import autogen +import autogen def test_chat_manager(): diff --git a/test/agentchat/test_math_user_proxy_agent.py b/test/agentchat/test_math_user_proxy_agent.py index 537779199..7874c2168 100644 --- a/test/agentchat/test_math_user_proxy_agent.py +++ b/test/agentchat/test_math_user_proxy_agent.py @@ -1,7 +1,7 @@ import pytest import sys -from flaml import autogen -from flaml.autogen.agentchat.contrib.math_user_proxy_agent import ( +import autogen +from autogen.agentchat.contrib.math_user_proxy_agent import ( MathUserProxyAgent, _remove_print, _add_print_to_last_line, @@ -19,7 +19,7 @@ def test_math_user_proxy_agent(): except ImportError: return - from flaml.autogen.agentchat.assistant_agent import AssistantAgent + from autogen.agentchat.assistant_agent import AssistantAgent conversations = {} autogen.ChatCompletion.start_logging(conversations) diff --git a/test/agentchat/test_retrievechat.py b/test/agentchat/test_retrievechat.py index 761665d57..f22cb2a66 100644 --- a/test/agentchat/test_retrievechat.py +++ b/test/agentchat/test_retrievechat.py @@ -1,16 +1,16 @@ import pytest import sys -from flaml import autogen +import autogen from test_assistant_agent import KEY_LOC, OAI_CONFIG_LIST try: - from flaml.autogen.agentchat.contrib.retrieve_assistant_agent import ( + from autogen.agentchat.contrib.retrieve_assistant_agent import ( RetrieveAssistantAgent, ) - from flaml.autogen.agentchat.contrib.retrieve_user_proxy_agent import ( + from autogen.agentchat.contrib.retrieve_user_proxy_agent import ( RetrieveUserProxyAgent, ) - from flaml.autogen.retrieve_utils import create_vector_db_from_dir, query_vector_db + from autogen.retrieve_utils import create_vector_db_from_dir, query_vector_db import chromadb skip_test = False @@ -70,15 +70,15 @@ def test_retrievechat(): ) def test_retrieve_utils(): client = chromadb.PersistentClient(path="/tmp/chromadb") - create_vector_db_from_dir(dir_path="./website/docs", client=client, collection_name="flaml-docs") + create_vector_db_from_dir(dir_path="./website/docs", client=client, collection_name="autogen-docs") results = query_vector_db( query_texts=[ - "How can I use FLAML UserProxyAgent and AssistantAgent to do code generation?", + "How can I use AutoGen UserProxyAgent and AssistantAgent to do code generation?", ], n_results=4, client=client, - collection_name="flaml-docs", - search_string="FLAML", + collection_name="autogen-docs", + search_string="AutoGen", ) print(results["ids"][0]) assert len(results["ids"][0]) == 4 diff --git a/test/oai/test_completion.py b/test/oai/test_completion.py index ad6ca3d27..7eb28bc3e 100644 --- a/test/oai/test_completion.py +++ b/test/oai/test_completion.py @@ -5,14 +5,14 @@ import pytest from functools import partial import os import json -from flaml import autogen -from flaml.autogen.code_utils import ( +import autogen +from autogen.code_utils import ( eval_function_completions, generate_assertions, implement, generate_code, ) -from flaml.autogen.math_utils import eval_math_responses, solve_problem +from autogen.math_utils import eval_math_responses, solve_problem KEY_LOC = "notebook" OAI_CONFIG_LIST = "OAI_CONFIG_LIST" diff --git a/test/oai/test_utils.py b/test/oai/test_utils.py index 17ed33d7f..685bcf904 100644 --- a/test/oai/test_utils.py +++ b/test/oai/test_utils.py @@ -1,6 +1,6 @@ import json import os -from flaml import autogen +import autogen from test_completion import KEY_LOC, OAI_CONFIG_LIST diff --git a/test/test_code.py b/test/test_code.py index 508143529..0bc35c49e 100644 --- a/test/test_code.py +++ b/test/test_code.py @@ -1,8 +1,8 @@ import sys import os import pytest -from flaml import autogen -from flaml.autogen.code_utils import ( +import autogen +from autogen.code_utils import ( UNKNOWN, extract_code, execute_code, @@ -149,7 +149,7 @@ here = os.path.abspath(os.path.dirname(__file__)) def test_infer_lang(): assert infer_lang("print('hello world')") == "python" - assert infer_lang("pip install flaml") == "sh" + assert infer_lang("pip install autogen") == "sh" def test_extract_code(): @@ -238,7 +238,7 @@ def test_improve(): return config_list = autogen.config_list_openai_aoai(KEY_LOC) improved, _ = improve_function( - "flaml/autogen/math_utils.py", + "autogen/math_utils.py", "solve_problem", "Solve math problems accurately, by avoiding calculation errors and reduce reasoning errors.", config_list=config_list, @@ -246,13 +246,13 @@ def test_improve(): with open(f"{here}/math_utils.py.improved", "w") as f: f.write(improved) suggestion, _ = improve_code( - ["flaml/autogen/code_utils.py", "flaml/autogen/math_utils.py"], + ["autogen/code_utils.py", "autogen/math_utils.py"], "leverage generative AI smartly and cost-effectively", config_list=config_list, ) print(suggestion) improvement, cost = improve_code( - ["flaml/autogen/code_utils.py", "flaml/autogen/math_utils.py"], + ["autogen/code_utils.py", "autogen/math_utils.py"], "leverage generative AI smartly and cost-effectively", suggest_only=False, config_list=config_list, diff --git a/test/test_function_call.py b/test/test_function_call.py index 2288a5d0a..9b026ca3c 100644 --- a/test/test_function_call.py +++ b/test/test_function_call.py @@ -4,8 +4,8 @@ except ImportError: openai = None import pytest import json -from flaml import autogen -from flaml.autogen.math_utils import eval_math_responses +import autogen +from autogen.math_utils import eval_math_responses from test_code import KEY_LOC @@ -61,7 +61,7 @@ def test_eval_math_responses(): def test_json_extraction(): - from flaml.autogen.agentchat import UserProxyAgent + from autogen.agentchat import UserProxyAgent user = UserProxyAgent(name="test", code_execution_config={"use_docker": False}) @@ -76,7 +76,7 @@ def test_json_extraction(): def test_execute_function(): - from flaml.autogen.agentchat import UserProxyAgent + from autogen.agentchat import UserProxyAgent # 1. test calling a simple function def add_num(num_to_be_added): diff --git a/website/blog/2023-04-21-LLM-tuning-math/index.mdx b/website/blog/2023-04-21-LLM-tuning-math/index.mdx index c8d01d1a4..a098c2e32 100644 --- a/website/blog/2023-04-21-LLM-tuning-math/index.mdx +++ b/website/blog/2023-04-21-LLM-tuning-math/index.mdx @@ -9,20 +9,20 @@ tags: [LLM, GPT, research] **TL;DR:** * **Just by tuning the inference parameters like model, number of responses, temperature etc. without changing any model weights or prompt, the baseline accuracy of untuned gpt-4 can be improved by 20% in high school math competition problems.** * **For easy problems, the tuned gpt-3.5-turbo model vastly outperformed untuned gpt-4 in accuracy (e.g., 90% vs. 70%) and cost efficiency. For hard problems, the tuned gpt-4 is much more accurate (e.g., 35% vs. 20%) and less expensive than untuned gpt-4.** -* **FLAML can help with model selection, parameter tuning, and cost-saving in LLM applications.** +* **AutoGen can help with model selection, parameter tuning, and cost-saving in LLM applications.** Large language models (LLMs) are powerful tools that can generate natural language texts for various applications, such as chatbots, summarization, translation, and more. GPT-4 is currently the state of the art LLM in the world. Is model selection irrelevant? What about inference parameters? In this blog post, we will explore how model and inference parameter matter in LLM applications, using a case study for [MATH](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html), a benchmark for evaluating LLMs on advanced mathematical problem solving. MATH consists of 12K math competition problems from AMC-10, AMC-12 and AIME. Each problem is accompanied by a step-by-step solution. -We will use the new subpackage [`flaml.autogen`](docs/Use-Cases/Autogen) to automatically find the best model and inference parameter for LLMs on a given task and dataset given an inference budget, using a novel low-cost search & pruning strategy. FLAML currently supports all the LLMs from OpenAI, such as GPT-3.5 and GPT-4. +We will use AutoGen to automatically find the best model and inference parameter for LLMs on a given task and dataset given an inference budget, using a novel low-cost search & pruning strategy. AutoGen currently supports all the LLMs from OpenAI, such as GPT-3.5 and GPT-4. -We will use FLAML to perform model selection and inference parameter tuning. Then we compare the performance and inference cost on solving algebra problems with the untuned gpt-4. We will also analyze how different difficulty levels affect the results. +We will use AutoGen to perform model selection and inference parameter tuning. Then we compare the performance and inference cost on solving algebra problems with the untuned gpt-4. We will also analyze how different difficulty levels affect the results. ## Experiment Setup -We use FLAML to select between the following models with a target inference budget $0.02 per instance: +We use AutoGen to select between the following models with a target inference budget $0.02 per instance: - gpt-3.5-turbo, a relatively cheap model that powers the popular ChatGPT app - gpt-4, the state of the art LLM that costs more than 10 times of gpt-3.5-turbo @@ -54,21 +54,21 @@ On level 5 the result is similar. ![level 5 algebra](img/level5algebra.png) -We can see that FLAML has found different optimal model and inference parameters for each subset of a particular level, which shows that these parameters matter in cost-sensitive LLM applications and need to be carefully tuned or adapted. +We can see that AutoGen has found different optimal model and inference parameters for each subset of a particular level, which shows that these parameters matter in cost-sensitive LLM applications and need to be carefully tuned or adapted. -An example notebook to run these experiments can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/autogen_chatgpt.ipynb +An example notebook to run these experiments can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/autogen_chatgpt.ipynb. The experiments were run when AutoGen was a subpackage in FLAML. ## Analysis and Discussion While gpt-3.5-turbo demonstrates competitive accuracy with voted answers in relatively easy algebra problems under the same inference budget, gpt-4 is a better choice for the most difficult problems. In general, through parameter tuning and model selection, we can identify the opportunity to save the expensive model for more challenging tasks, and improve the overall effectiveness of a budget-constrained system. -There are many other alternative ways of solving math problems, which we have not covered in this blog post. When there are choices beyond the inference parameters, they can be generally tuned via [`flaml.tune`](docs/Use-Cases/Tune-User-Defined-Function). +There are many other alternative ways of solving math problems, which we have not covered in this blog post. When there are choices beyond the inference parameters, they can be generally tuned via [`flaml.tune`](https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function). The need for model selection, parameter tuning and cost saving is not specific to the math problems. The [Auto-GPT](https://github.com/Significant-Gravitas/Auto-GPT) project is an example where high cost can easily prevent a generic complex task to be accomplished as it needs many LLM inference calls. ## For Further Reading * [Research paper about the tuning technique](https://arxiv.org/abs/2303.04673) -* [Documentation about `flaml.autogen`](https://microsoft.github.io/autogen/) +* [Documentation about AutoGen](https://microsoft.github.io/autogen/) *Do you have any experience to share about LLM applications? Do you like to see more support or research of LLM optimization or automation? Please join our [Discord](https://discord.gg/Cppx2vSPVP) server for discussion.* diff --git a/website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx b/website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx index 0c0500650..292923cab 100644 --- a/website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx +++ b/website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx @@ -56,7 +56,7 @@ Combining these observations, we can design a solution with two intuitive ideas: This solution works adaptively without knowing or predicting which task fits which configuration. It simply tries multiple configurations one by one, starting from the cheapest configuration. Note that one configuration can generate multiple responses (by setting the inference parameter n larger than 1). And different configurations can use the same model and different inference parameters such as n and temperature. Only one response is returned and evaluated per task. -An implementation of this solution is provided in [flaml.autogen](/docs/reference/autogen/code_utils#implement). It uses the following sequence of configurations: +An implementation of this solution is provided in [autogen](/docs/reference/code_utils#implement). It uses the following sequence of configurations: 1. GPT-3.5-Turbo, n=1, temperature=0 1. GPT-3.5-Turbo, n=7, temperature=1, stop=["\nclass", "\ndef", "\nif", "\nprint"] @@ -140,11 +140,11 @@ The last problem is an example with wrong example test cases in the original def It is worth noting that the reduced inference cost is the amortized cost over all the tasks. For each individual task, the cost can be either larger or smaller than directly using GPT-4. This is the nature of the adaptive solution: The cost is in general larger for difficult tasks than that for easy tasks. -An example notebook to run this experiment can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/research/autogen_code.ipynb +An example notebook to run this experiment can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/research/autogen_code.ipynb. The experiment was run when AutoGen was a subpackage in FLAML. ## Discussion -Our solution is quite simple to implement using a generic interface offered in [`flaml.autogen`](https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#logic-error), yet the result is quite encouraging. +Our solution is quite simple to implement using a generic interface offered in [`autogen`](https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#logic-error), yet the result is quite encouraging. While the specific way of generating assertions is application-specific, the main ideas are general in LLM operations: * Generate multiple responses to select - especially useful when selecting a good response is relatively easier than generating a good response at one shot. @@ -153,7 +153,7 @@ While the specific way of generating assertions is application-specific, the mai - Different configurations have complementary effect. A [previous blog post](/blog/2023/04/21/LLM-tuning-math) provides evidence that these ideas are relevant in solving math problems too. -`flaml.autogen` uses a technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) to support inference parameter tuning and model selection. +`autogen` uses a technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) to support inference parameter tuning and model selection. There are many directions of extensions in research and development: * Generalize the way to provide feedback. @@ -164,5 +164,5 @@ There are many directions of extensions in research and development: ## For Further Reading -* [Documentation](https://microsoft.github.io/autogen/) about `flaml.autogen` and [Research paper](https://arxiv.org/abs/2303.04673). +* [Documentation](https://microsoft.github.io/autogen/) about `autogen` and [Research paper](https://arxiv.org/abs/2303.04673). * [Blog post](/blog/2023/04/21/LLM-tuning-math) about a related study for math. diff --git a/website/blog/2023-06-28-MathChat/index.mdx b/website/blog/2023-06-28-MathChat/index.mdx index ce4b945a9..86fa122aa 100644 --- a/website/blog/2023-06-28-MathChat/index.mdx +++ b/website/blog/2023-06-28-MathChat/index.mdx @@ -75,7 +75,7 @@ We found that compared to basic prompting, which demonstrates the innate capabil For categories like Algebra and Prealgebra, PoT and PS showed little improvement, and in some instances, even led to a decrease in accuracy. However, MathChat was able to enhance total accuracy by around 6% compared to PoT and PS, showing competitive performance across all categories. Remarkably, MathChat improved accuracy in the Algebra category by about 15% over other methods. Note that categories like Intermediate Algebra and Precalculus remained challenging for all methods, with only about 20% of problems solved accurately. The code for experiments can be found at this [repository](https://github.com/kevin666aa/FLAML/tree/gpt_math_solver/flaml/autogen/math). -We now provide an implementation of MathChat using the interactive agents in FLAML. See this [notebook](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_MathChat.ipynb) for example usage. +We now provide an implementation of MathChat using the interactive agents in AutoGen. See this [notebook](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_MathChat.ipynb) for example usage. ## Future Directions @@ -89,6 +89,6 @@ Further work can be done to enhance this framework or math problem-solving in ge ## For Further Reading * [Research paper of MathChat](https://arxiv.org/abs/2306.01337) -* [Documentation about `flaml.autogen`](https://microsoft.github.io/autogen/) +* [Documentation about `autogen`](https://microsoft.github.io/autogen/) *Are you working on applications that involve math problem-solving? Would you appreciate additional research or support on the application of LLM-based agents for math problem-solving? Please join our [Discord](https://discord.gg/Cppx2vSPVP) server for discussion.* diff --git a/website/blog/2023-07-14-Local-LLMs/index.mdx b/website/blog/2023-07-14-Local-LLMs/index.mdx index 059205749..aca5b608b 100644 --- a/website/blog/2023-07-14-Local-LLMs/index.mdx +++ b/website/blog/2023-07-14-Local-LLMs/index.mdx @@ -1,10 +1,10 @@ --- -title: Use flaml.autogen for Local LLMs +title: Use AutoGen for Local LLMs authors: jialeliu -tags: [LLM, FLAMLv2] +tags: [LLM] --- **TL;DR:** -We demonstrate how to use flaml.autogen for local LLM application. As an example, we will initiate an endpoint using [FastChat](https://github.com/lm-sys/FastChat) and perform inference on [ChatGLMv2-6b](https://github.com/THUDM/ChatGLM2-6B). +We demonstrate how to use autogen for local LLM application. As an example, we will initiate an endpoint using [FastChat](https://github.com/lm-sys/FastChat) and perform inference on [ChatGLMv2-6b](https://github.com/THUDM/ChatGLM2-6B). ## Preparations @@ -66,11 +66,11 @@ class CompletionResponseStreamChoice(BaseModel): ## Interact with model using `oai.Completion` -Now the models can be directly accessed through openai-python library as well as `flaml.oai.Completion` and `flaml.oai.ChatCompletion`. +Now the models can be directly accessed through openai-python library as well as `autogen.oai.Completion` and `autogen.oai.ChatCompletion`. ```python -from flaml import oai +from autogen import oai # create a text completion request response = oai.Completion.create( @@ -118,7 +118,7 @@ python -m fastchat.serve.multi_model_worker \ The inference code would be: ```python -from flaml import oai +from autogen import oai # create a chat completion request response = oai.ChatCompletion.create( @@ -143,5 +143,5 @@ print(response) ## For Further Reading -* [Documentation](https://microsoft.github.io/autogen/) about `flaml.autogen` +* [Documentation](https://microsoft.github.io/autogen/) about `autogen` * [Documentation](https://github.com/lm-sys/FastChat) about FastChat. diff --git a/website/docs/Examples/AutoGen-AgentChat.md b/website/docs/Examples/AutoGen-AgentChat.md index 866233c1d..e34d3cf70 100644 --- a/website/docs/Examples/AutoGen-AgentChat.md +++ b/website/docs/Examples/AutoGen-AgentChat.md @@ -4,14 +4,14 @@ AutoGen offers conversable agents powered by LLM, tool or human, which can be us Please find documentation about this feature [here](/docs/Use-Cases/agent_chat). Links to notebook examples: -* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb) -* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_human_feedback.ipynb) -* [Solve Tasks Requiring Web Info](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_web_info.ipynb) -* [Use Provided Tools as Functions](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_function_call.ipynb) -* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_planning.ipynb) -* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_two_users.ipynb) -* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_chess.ipynb) -* [Automated Task Solving by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat.ipynb) -* [Automated Data Visualization by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb) -* [Automated Complex Task Solving by Group Chat (with 6 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb) -* [Automated Continual Learning from New Data](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_stream.ipynb) +* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb) +* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_human_feedback.ipynb) +* [Solve Tasks Requiring Web Info](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_web_info.ipynb) +* [Use Provided Tools as Functions](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_function_call.ipynb) +* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_planning.ipynb) +* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_two_users.ipynb) +* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_chess.ipynb) +* [Automated Task Solving by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat.ipynb) +* [Automated Data Visualization by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb) +* [Automated Complex Task Solving by Group Chat (with 6 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb) +* [Automated Continual Learning from New Data](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_stream.ipynb) diff --git a/website/docs/Examples/AutoGen-OpenAI.md b/website/docs/Examples/AutoGen-OpenAI.md index 44d1124de..c44b77018 100644 --- a/website/docs/Examples/AutoGen-OpenAI.md +++ b/website/docs/Examples/AutoGen-OpenAI.md @@ -4,5 +4,5 @@ AutoGen also offers a cost-effective hyperparameter optimization technique [EcoO Please find documentation about this feature [here](/docs/Use-Cases/enhanced_inference). Links to notebook examples: -* [Optimize for Code Generation](https://github.com/microsoft/flaml/blob/main/notebook/autogen_openai_completion.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/flaml/blob/main/notebook/autogen_openai_completion.ipynb) -* [Optimize for Math](https://github.com/microsoft/flaml/blob/main/notebook/autogen_chatgpt_gpt4.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/flaml/blob/main/notebook/autogen_chatgpt_gpt4.ipynb) +* [Optimize for Code Generation](https://github.com/microsoft/autogen/blob/main/notebook/autogen_openai_completion.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_openai_completion.ipynb) +* [Optimize for Math](https://github.com/microsoft/autogen/blob/main/notebook/autogen_chatgpt_gpt4.ipynb) | [Open in colab](https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/autogen_chatgpt_gpt4.ipynb) diff --git a/website/docs/Getting-Started.md b/website/docs/Getting-Started.md index 7f83ec645..0441e8836 100644 --- a/website/docs/Getting-Started.md +++ b/website/docs/Getting-Started.md @@ -20,14 +20,14 @@ AutoGen is powered by collaborative [research studies](/docs/Research) from Micr ### Quickstart -Install AutoGen from pip: `pip install "flaml[autogen]"`. Find more options in [Installation](/docs/Installation). +Install from pip: `pip install pyautogen`. Find more options in [Installation](/docs/Installation). #### Multi-Agent Conversation Framework Autogen enables the next-gen LLM applications with a generic multi-agent conversation framework. It offers customizable and conversable agents which integrate LLMs, tools and human. By automating chat among multiple capable agents, one can easily make them collectively perform tasks autonomously or with human feedback, including tasks that require using tools via code. For example, ```python -from flaml.autogen import AssistantAgent, UserProxyAgent +from autogen import AssistantAgent, UserProxyAgent assistant = AssistantAgent("assistant") user_proxy = UserProxyAgent("user_proxy") user_proxy.initiate_chat(assistant, message="Plot a chart of META and TESLA stock price change YTD.") @@ -59,10 +59,10 @@ response = autogen.Completion.create(context=test_instance, **config) * Understand the use cases for [multi-agent conversation](/docs/Use-Cases/agent_chat). * Understand the use cases for [enhanced LLM inference](/docs/Use-Cases/enhanced_inference). * Find code examples from [Examples](/docs/Examples/AutoGen-AgentChat). -* Read [SDK](https://microsoft.github.io/FLAML/docs/reference/autogen/agentchat/responsive_agent/). +* Read [SDK](/docs/reference/autogen/agentchat/conversable_agent/). * Learn about [research](/docs/Research) around AutoGen. * Chat on [Discord](https://discord.gg/Cppx2vSPVP). -If you like our project, please give it a [star](https://github.com/microsoft/FLAML/stargazers) on GitHub. If you are interested in contributing, please read [Contributor's Guide](/docs/Contribute). +If you like our project, please give it a [star](https://github.com/microsoft/autogen/stargazers) on GitHub. If you are interested in contributing, please read [Contributor's Guide](/docs/Contribute). - + diff --git a/website/docs/Use-Cases/agent_chat.md b/website/docs/Use-Cases/agent_chat.md index a38ba4d8e..231d96f9f 100644 --- a/website/docs/Use-Cases/agent_chat.md +++ b/website/docs/Use-Cases/agent_chat.md @@ -31,7 +31,7 @@ One can also easily extend it by registering reply functions with the `register_ In the following code, we create an `AssistantAgent` named "assistant" to serve as the assistant and a `UserProxyAgent` named "user_proxy" to serve as a proxy for the human user. We will later employ these two agents to solve a task. ```python -from flaml.autogen import AssistantAgent, UserProxyAgent +from autogen import AssistantAgent, UserProxyAgent # create an AssistantAgent instance named "assistant" assistant = AssistantAgent(name="assistant") @@ -69,10 +69,10 @@ On the one hand, one can achieve fully autonomous conversations after an initial By adopting the conversation-driven control with both programming language and natural language, AutoGen inherently allows dynamic conversation. Dynamic conversation allows the agent topology to change depending on the actual flow of conversation under different input problem instances, while the flow of a static conversation always follows a pre-defined topology. The dynamic conversation pattern is useful in complex applications where the patterns of interaction cannot be predetermined in advance. AutoGen provides two general approaches to achieving dynamic conversation: -- Registered auto-reply. With the pluggable auto-reply function, one can choose to invoke conversations with other agents depending on the content of the current message and context. A working system demonstrating this type of dynamic conversation can be found in this code example, demonstrating a [dynamic group chat](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_groupchat.ipynb). In the system, we register an auto-reply function in the group chat manager, which lets LLM decide who will the next speaker be in a group chat setting. +- Registered auto-reply. With the pluggable auto-reply function, one can choose to invoke conversations with other agents depending on the content of the current message and context. A working system demonstrating this type of dynamic conversation can be found in this code example, demonstrating a [dynamic group chat](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat.ipynb). In the system, we register an auto-reply function in the group chat manager, which lets LLM decide who will the next speaker be in a group chat setting. - LLM-based function call. In this approach, LLM decides whether or not to call a particular function depending on the conversation status in each inference call. -By messaging additional agents in the called functions, the LLM can drive dynamic multi-agent conversation. A working system showcasing this type of dynamic conversation can be found in the [multi-user math problem solving scenario](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_two_users.ipynb), where a student assistant would automatically resort to an expert using function calls. +By messaging additional agents in the called functions, the LLM can drive dynamic multi-agent conversation. A working system showcasing this type of dynamic conversation can be found in the [multi-user math problem solving scenario](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_two_users.ipynb), where a student assistant would automatically resort to an expert using function calls. ### Diverse Applications Implemented with AutoGen @@ -80,17 +80,17 @@ By messaging additional agents in the called functions, the LLM can drive dynami The figure below shows six examples of applications built using AutoGen. ![Applications](images/app.png) -* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb) -* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_human_feedback.ipynb) -* [Solve Tasks Requiring Web Info](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_web_info.ipynb) -* [Use Provided Tools as Functions](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_function_call.ipynb) -* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_planning.ipynb) -* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_two_users.ipynb) -* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_chess.ipynb) -* [Automated Task Solving by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat.ipynb) -* [Automated Data Visualization by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb) -* [Automated Complex Task Solving by Group Chat (with 6 group member agents and 1 manager agent)](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb) -* [Automated Continual Learning from New Data](https://github.com/microsoft/flaml/blob/main/notebook/autogen_agentchat_stream.ipynb) +* [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb) +* [Auto Code Generation, Execution, Debugging and Human Feedback](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_human_feedback.ipynb) +* [Solve Tasks Requiring Web Info](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_web_info.ipynb) +* [Use Provided Tools as Functions](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_function_call.ipynb) +* [Automated Task Solving with Coding & Planning Agents](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_planning.ipynb) +* [Automated Task Solving with GPT-4 + Multiple Human Users](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_two_users.ipynb) +* [Automated Chess Game Playing & Chitchatting by GPT-4 Agents](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_chess.ipynb) +* [Automated Task Solving by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat.ipynb) +* [Automated Data Visualization by Group Chat (with 3 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_vis.ipynb) +* [Automated Complex Task Solving by Group Chat (with 6 group member agents and 1 manager agent)](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_groupchat_research.ipynb) +* [Automated Continual Learning from New Data](https://github.com/microsoft/autogen/blob/main/notebook/autogen_agentchat_stream.ipynb) diff --git a/website/docs/Use-Cases/enhanced_inference.md b/website/docs/Use-Cases/enhanced_inference.md index 90cd46be1..5de73003c 100644 --- a/website/docs/Use-Cases/enhanced_inference.md +++ b/website/docs/Use-Cases/enhanced_inference.md @@ -6,8 +6,8 @@ There are a number of benefits of using `autogen` to perform inference: performa ## Tune Inference Parameters *Links to notebook examples:* -* [Optimize for Code Generation](https://github.com/microsoft/flaml/blob/main/notebook/autogen_openai_completion.ipynb) -* [Optimize for Math](https://github.com/microsoft/flaml/blob/main/notebook/autogen_chatgpt_gpt4.ipynb) +* [Optimize for Code Generation](https://github.com/microsoft/autogen/blob/main/notebook/autogen_openai_completion.ipynb) +* [Optimize for Math](https://github.com/microsoft/autogen/blob/main/notebook/autogen_chatgpt_gpt4.ipynb) ### Choices to optimize @@ -30,7 +30,7 @@ There are also complex interactions among subsets of the hyperparameters. For ex the temperature and top_p are not recommended to be altered from their default values together because they both control the randomness of the generated text, and changing both at the same time can result in conflicting effects; n and best_of are rarely tuned together because if the application can process multiple outputs, filtering on the server side causes unnecessary information loss; both n and max_tokens will affect the total number of tokens generated, which in turn will affect the cost of the request. These interactions and trade-offs make it difficult to manually determine the optimal hyperparameter settings for a given text generation task. -*Do the choices matter? Check this [blogpost](https://microsoft.github.io/FLAML/blog/2023/04/21/LLM-tuning-math) to find example tuning results about gpt-3.5-turbo and gpt-4.* +*Do the choices matter? Check this [blogpost](https://microsoft.github.io/autogen/blog/2023/04/21/LLM-tuning-math) to find example tuning results about gpt-3.5-turbo and gpt-4.* With AutoGen, the tuning can be performed with the following information: @@ -111,7 +111,7 @@ The tuend config can be used to perform inference. `autogen.Completion.create` is compatible with both `openai.Completion.create` and `openai.ChatCompletion.create`, and both OpenAI API and Azure OpenAI API. So models such as "text-davinci-003", "gpt-3.5-turbo" and "gpt-4" can share a common API. When chat models are used and `prompt` is given as the input to `autogen.Completion.create`, the prompt will be automatically converted into `messages` to fit the chat completion API requirement. One advantage is that one can experiment with both chat and non-chat models for the same prompt in a unified API. -For local LLMs, one can spin up an endpoint using a package like [FastChat](https://github.com/lm-sys/FastChat), and then use the same API to send a request. See [here](https://microsoft.github.io/FLAML/blog/2023/07/14/Local-LLMs) for examples on how to make inference with local LLMs. +For local LLMs, one can spin up an endpoint using a package like [FastChat](https://github.com/lm-sys/FastChat), and then use the same API to send a request. See [here](https://microsoft.github.io/autogen/blog/2023/07/14/Local-LLMs) for examples on how to make inference with local LLMs. When only working with the chat-based models, `autogen.ChatCompletion` can be used. It also does automatic conversion from prompt to messages, if prompt is provided instead of messages. @@ -183,7 +183,7 @@ response = autogen.Completion.create( The example above will try to use text-ada-001, gpt-3.5-turbo, and text-davinci-003 iteratively, until a valid json string is returned or the last config is used. One can also repeat the same model in the list for multiple times to try one model multiple times for increasing the robustness of the final response. -*Advanced use case: Check this [blogpost](https://microsoft.github.io/FLAML/blog/2023/05/18/GPT-adaptive-humaneval) to find how to improve GPT-4's coding performance from 68% to 90% while reducing the inference cost.* +*Advanced use case: Check this [blogpost](https://microsoft.github.io/autogen/blog/2023/05/18/GPT-adaptive-humaneval) to find how to improve GPT-4's coding performance from 68% to 90% while reducing the inference cost.* ## Templating