{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using RetrieveChat with Qdrant for Retrieve Augmented Code Generation and Question Answering\n",
"\n",
"[Qdrant](https://qdrant.tech/) is a high-performance vector search engine/database.\n",
"\n",
"This notebook demonstrates the usage of `QdrantRetrieveUserProxyAgent` for RAG, based on [agentchat_RetrieveChat.ipynb](https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_RetrieveChat.ipynb).\n",
"\n",
"\n",
"RetrieveChat is a conversational system for retrieve augmented code generation and question answering. In this notebook, we demonstrate how to utilize RetrieveChat to generate code and answer questions based on customized documentations that are not present in the LLM's training dataset. RetrieveChat uses the `RetrieveAssistantAgent` and `QdrantRetrieveUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_auto_feedback_from_code_execution.ipynb)).\n",
"\n",
"We'll demonstrate usage of RetrieveChat with Qdrant for code generation and question answering w/ human feedback.\n",
"\n",
"````{=mdx}\n",
":::info Requirements\n",
"Some extra dependencies are needed for this notebook, which can be installed via pip:\n",
"\n",
"```bash\n",
"pip install \"pyautogen[retrievechat]>=0.2.3\" \"flaml[automl]\" \"qdrant_client[fastembed]\"\n",
"```\n",
"\n",
"For more information, please refer to the [installation guide](/docs/installation/).\n",
":::\n",
"````"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: pyautogen>=0.2.3 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen[retrievechat]>=0.2.3) (0.2.3)\n",
"Requirement already satisfied: flaml[automl] in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (2.1.1)\n",
"Requirement already satisfied: qdrant_client[fastembed] in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (1.7.0)\n",
"Requirement already satisfied: diskcache in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (5.6.3)\n",
"Requirement already satisfied: openai>=1.3 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (1.6.1)\n",
"Requirement already satisfied: pydantic<3,>=1.10 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (2.5.3)\n",
"Requirement already satisfied: python-dotenv in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (1.0.0)\n",
"Requirement already satisfied: termcolor in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (2.4.0)\n",
"Requirement already satisfied: tiktoken in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (0.5.2)\n",
"Requirement already satisfied: NumPy>=1.17.0rc1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from flaml[automl]) (1.26.2)\n",
"Requirement already satisfied: lightgbm>=2.3.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from flaml[automl]) (4.2.0)\n",
"Requirement already satisfied: xgboost>=0.90 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from flaml[automl]) (2.0.3)\n",
"Requirement already satisfied: scipy>=1.4.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from flaml[automl]) (1.11.4)\n",
"Requirement already satisfied: pandas>=1.1.4 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from flaml[automl]) (2.1.4)\n",
"Requirement already satisfied: scikit-learn>=0.24 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from flaml[automl]) (1.3.2)\n",
"Requirement already satisfied: fastembed==0.1.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from qdrant_client[fastembed]) (0.1.1)\n",
"Requirement already satisfied: grpcio>=1.41.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from qdrant_client[fastembed]) (1.60.0)\n",
"Requirement already satisfied: grpcio-tools>=1.41.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from qdrant_client[fastembed]) (1.60.0)\n",
"Requirement already satisfied: httpx>=0.14.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from httpx[http2]>=0.14.0->qdrant_client[fastembed]) (0.26.0)\n",
"Requirement already satisfied: portalocker<3.0.0,>=2.7.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from qdrant_client[fastembed]) (2.8.2)\n",
"Requirement already satisfied: urllib3<2.0.0,>=1.26.14 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from qdrant_client[fastembed]) (1.26.18)\n",
"Requirement already satisfied: onnx<2.0,>=1.11 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from fastembed==0.1.1->qdrant_client[fastembed]) (1.15.0)\n",
"Requirement already satisfied: onnxruntime<2.0,>=1.15 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from fastembed==0.1.1->qdrant_client[fastembed]) (1.16.3)\n",
"Requirement already satisfied: requests<3.0,>=2.31 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from fastembed==0.1.1->qdrant_client[fastembed]) (2.31.0)\n",
"Requirement already satisfied: tokenizers<0.14,>=0.13 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from fastembed==0.1.1->qdrant_client[fastembed]) (0.13.3)\n",
"Requirement already satisfied: tqdm<5.0,>=4.65 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from fastembed==0.1.1->qdrant_client[fastembed]) (4.66.1)\n",
"Requirement already satisfied: chromadb in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen[retrievechat]>=0.2.3) (0.4.21)\n",
"Requirement already satisfied: ipython in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen[retrievechat]>=0.2.3) (8.19.0)\n",
"Requirement already satisfied: pypdf in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen[retrievechat]>=0.2.3) (3.17.4)\n",
"Requirement already satisfied: sentence-transformers in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyautogen[retrievechat]>=0.2.3) (2.2.2)\n",
"Requirement already satisfied: protobuf<5.0dev,>=4.21.6 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from grpcio-tools>=1.41.0->qdrant_client[fastembed]) (4.25.1)\n",
"Requirement already satisfied: setuptools in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from grpcio-tools>=1.41.0->qdrant_client[fastembed]) (65.5.0)\n",
"Requirement already satisfied: anyio in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from httpx>=0.14.0->httpx[http2]>=0.14.0->qdrant_client[fastembed]) (4.2.0)\n",
"Requirement already satisfied: certifi in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from httpx>=0.14.0->httpx[http2]>=0.14.0->qdrant_client[fastembed]) (2023.11.17)\n",
"Requirement already satisfied: httpcore==1.* in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from httpx>=0.14.0->httpx[http2]>=0.14.0->qdrant_client[fastembed]) (1.0.2)\n",
"Requirement already satisfied: idna in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from httpx>=0.14.0->httpx[http2]>=0.14.0->qdrant_client[fastembed]) (3.6)\n",
"Requirement already satisfied: sniffio in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from httpx>=0.14.0->httpx[http2]>=0.14.0->qdrant_client[fastembed]) (1.3.0)\n",
"Requirement already satisfied: h11<0.15,>=0.13 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.14.0->httpx[http2]>=0.14.0->qdrant_client[fastembed]) (0.14.0)\n",
"Requirement already satisfied: h2<5,>=3 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from httpx[http2]>=0.14.0->qdrant_client[fastembed]) (4.1.0)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from openai>=1.3->pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (1.9.0)\n",
"Requirement already satisfied: typing-extensions<5,>=4.7 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from openai>=1.3->pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (4.9.0)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pandas>=1.1.4->flaml[automl]) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pandas>=1.1.4->flaml[automl]) (2023.3.post1)\n",
"Requirement already satisfied: tzdata>=2022.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pandas>=1.1.4->flaml[automl]) (2023.4)\n",
"Requirement already satisfied: annotated-types>=0.4.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pydantic<3,>=1.10->pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (0.6.0)\n",
"Requirement already satisfied: pydantic-core==2.14.6 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pydantic<3,>=1.10->pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (2.14.6)\n",
"Requirement already satisfied: joblib>=1.1.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from scikit-learn>=0.24->flaml[automl]) (1.3.2)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from scikit-learn>=0.24->flaml[automl]) (3.2.0)\n",
"Requirement already satisfied: chroma-hnswlib==0.7.3 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (0.7.3)\n",
"Requirement already satisfied: fastapi>=0.95.2 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (0.108.0)\n",
"Requirement already satisfied: uvicorn>=0.18.3 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from uvicorn[standard]>=0.18.3->chromadb->pyautogen[retrievechat]>=0.2.3) (0.25.0)\n",
"Requirement already satisfied: posthog>=2.4.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (3.1.0)\n",
"Requirement already satisfied: pulsar-client>=3.1.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (3.3.0)\n",
"Requirement already satisfied: opentelemetry-api>=1.2.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (1.22.0)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc>=1.2.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (1.22.0)\n",
"Requirement already satisfied: opentelemetry-instrumentation-fastapi>=0.41b0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (0.43b0)\n",
"Requirement already satisfied: opentelemetry-sdk>=1.2.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (1.22.0)\n",
"Requirement already satisfied: pypika>=0.48.9 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (0.48.9)\n",
"Requirement already satisfied: overrides>=7.3.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (7.4.0)\n",
"Requirement already satisfied: importlib-resources in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (6.1.1)\n",
"Requirement already satisfied: bcrypt>=4.0.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (4.1.2)\n",
"Requirement already satisfied: typer>=0.9.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (0.9.0)\n",
"Requirement already satisfied: kubernetes>=28.1.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (28.1.0)\n",
"Requirement already satisfied: tenacity>=8.2.3 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (8.2.3)\n",
"Requirement already satisfied: PyYAML>=6.0.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (6.0.1)\n",
"Requirement already satisfied: mmh3>=4.0.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from chromadb->pyautogen[retrievechat]>=0.2.3) (4.0.1)\n",
"Requirement already satisfied: decorator in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from ipython->pyautogen[retrievechat]>=0.2.3) (5.1.1)\n",
"Requirement already satisfied: jedi>=0.16 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from ipython->pyautogen[retrievechat]>=0.2.3) (0.19.1)\n",
"Requirement already satisfied: matplotlib-inline in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from ipython->pyautogen[retrievechat]>=0.2.3) (0.1.6)\n",
"Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from ipython->pyautogen[retrievechat]>=0.2.3) (3.0.43)\n",
"Requirement already satisfied: pygments>=2.4.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from ipython->pyautogen[retrievechat]>=0.2.3) (2.17.2)\n",
"Requirement already satisfied: stack-data in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from ipython->pyautogen[retrievechat]>=0.2.3) (0.6.3)\n",
"Requirement already satisfied: traitlets>=5 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from ipython->pyautogen[retrievechat]>=0.2.3) (5.14.1)\n",
"Requirement already satisfied: pexpect>4.3 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from ipython->pyautogen[retrievechat]>=0.2.3) (4.9.0)\n",
"Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from sentence-transformers->pyautogen[retrievechat]>=0.2.3) (4.33.3)\n",
"Requirement already satisfied: torch>=1.6.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from sentence-transformers->pyautogen[retrievechat]>=0.2.3) (2.1.2)\n",
"Requirement already satisfied: torchvision in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from sentence-transformers->pyautogen[retrievechat]>=0.2.3) (0.16.2)\n",
"Requirement already satisfied: nltk in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from sentence-transformers->pyautogen[retrievechat]>=0.2.3) (3.8.1)\n",
"Requirement already satisfied: sentencepiece in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from sentence-transformers->pyautogen[retrievechat]>=0.2.3) (0.1.99)\n",
"Requirement already satisfied: huggingface-hub>=0.4.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from sentence-transformers->pyautogen[retrievechat]>=0.2.3) (0.20.1)\n",
"Requirement already satisfied: regex>=2022.1.18 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from tiktoken->pyautogen>=0.2.3->pyautogen[retrievechat]>=0.2.3) (2023.12.25)\n",
"Requirement already satisfied: starlette<0.33.0,>=0.29.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from fastapi>=0.95.2->chromadb->pyautogen[retrievechat]>=0.2.3) (0.32.0.post1)\n",
"Requirement already satisfied: hyperframe<7,>=6.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from h2<5,>=3->httpx[http2]>=0.14.0->qdrant_client[fastembed]) (6.0.1)\n",
"Requirement already satisfied: hpack<5,>=4.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from h2<5,>=3->httpx[http2]>=0.14.0->qdrant_client[fastembed]) (4.0.0)\n",
"Requirement already satisfied: filelock in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from huggingface-hub>=0.4.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (3.13.1)\n",
"Requirement already satisfied: fsspec>=2023.5.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from huggingface-hub>=0.4.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (2023.12.2)\n",
"Requirement already satisfied: packaging>=20.9 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from huggingface-hub>=0.4.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (23.2)\n",
"Requirement already satisfied: parso<0.9.0,>=0.8.3 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from jedi>=0.16->ipython->pyautogen[retrievechat]>=0.2.3) (0.8.3)\n",
"Requirement already satisfied: six>=1.9.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.16.0)\n",
"Requirement already satisfied: google-auth>=1.0.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (2.25.2)\n",
"Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.7.0)\n",
"Requirement already satisfied: requests-oauthlib in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.3.1)\n",
"Requirement already satisfied: oauthlib>=3.2.2 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (3.2.2)\n",
"Requirement already satisfied: coloredlogs in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from onnxruntime<2.0,>=1.15->fastembed==0.1.1->qdrant_client[fastembed]) (15.0.1)\n",
"Requirement already satisfied: flatbuffers in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from onnxruntime<2.0,>=1.15->fastembed==0.1.1->qdrant_client[fastembed]) (23.5.26)\n",
"Requirement already satisfied: sympy in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from onnxruntime<2.0,>=1.15->fastembed==0.1.1->qdrant_client[fastembed]) (1.12)\n",
"Requirement already satisfied: deprecated>=1.2.6 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-api>=1.2.0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.2.14)\n",
"Requirement already satisfied: importlib-metadata<7.0,>=6.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-api>=1.2.0->chromadb->pyautogen[retrievechat]>=0.2.3) (6.11.0)\n",
"Requirement already satisfied: backoff<3.0.0,>=1.10.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb->pyautogen[retrievechat]>=0.2.3) (2.2.1)\n",
"Requirement already satisfied: googleapis-common-protos~=1.52 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.62.0)\n",
"Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.22.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.22.0)\n",
"Requirement already satisfied: opentelemetry-proto==1.22.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.22.0)\n",
"Requirement already satisfied: opentelemetry-instrumentation-asgi==0.43b0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb->pyautogen[retrievechat]>=0.2.3) (0.43b0)\n",
"Requirement already satisfied: opentelemetry-instrumentation==0.43b0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb->pyautogen[retrievechat]>=0.2.3) (0.43b0)\n",
"Requirement already satisfied: opentelemetry-semantic-conventions==0.43b0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb->pyautogen[retrievechat]>=0.2.3) (0.43b0)\n",
"Requirement already satisfied: opentelemetry-util-http==0.43b0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb->pyautogen[retrievechat]>=0.2.3) (0.43b0)\n",
"Requirement already satisfied: wrapt<2.0.0,>=1.0.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-instrumentation==0.43b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.16.0)\n",
"Requirement already satisfied: asgiref~=3.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from opentelemetry-instrumentation-asgi==0.43b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb->pyautogen[retrievechat]>=0.2.3) (3.7.2)\n",
"Requirement already satisfied: ptyprocess>=0.5 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pexpect>4.3->ipython->pyautogen[retrievechat]>=0.2.3) (0.7.0)\n",
"Requirement already satisfied: monotonic>=1.5 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from posthog>=2.4.0->chromadb->pyautogen[retrievechat]>=0.2.3) (1.6)\n",
"Requirement already satisfied: wcwidth in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython->pyautogen[retrievechat]>=0.2.3) (0.2.12)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from requests<3.0,>=2.31->fastembed==0.1.1->qdrant_client[fastembed]) (3.3.2)\n",
"Requirement already satisfied: networkx in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (3.2.1)\n",
"Requirement already satisfied: jinja2 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (3.1.2)\n",
"Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (12.1.105)\n",
"Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (8.9.2.26)\n",
"Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (12.1.3.1)\n",
"Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (11.0.2.54)\n",
"Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (10.3.2.106)\n",
"Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (11.4.5.107)\n",
"Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (12.1.0.106)\n",
"Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (2.18.1)\n",
"Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (12.1.105)\n",
"Requirement already satisfied: triton==2.1.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (2.1.0)\n",
"Requirement already satisfied: nvidia-nvjitlink-cu12 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (12.3.101)\n",
"Requirement already satisfied: safetensors>=0.3.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (0.4.1)\n",
"Requirement already satisfied: click<9.0.0,>=7.1.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from typer>=0.9.0->chromadb->pyautogen[retrievechat]>=0.2.3) (8.1.7)\n",
"Requirement already satisfied: httptools>=0.5.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from uvicorn[standard]>=0.18.3->chromadb->pyautogen[retrievechat]>=0.2.3) (0.6.1)\n",
"Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from uvicorn[standard]>=0.18.3->chromadb->pyautogen[retrievechat]>=0.2.3) (0.19.0)\n",
"Requirement already satisfied: watchfiles>=0.13 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from uvicorn[standard]>=0.18.3->chromadb->pyautogen[retrievechat]>=0.2.3) (0.21.0)\n",
"Requirement already satisfied: websockets>=10.4 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from uvicorn[standard]>=0.18.3->chromadb->pyautogen[retrievechat]>=0.2.3) (12.0)\n",
"Requirement already satisfied: executing>=1.2.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from stack-data->ipython->pyautogen[retrievechat]>=0.2.3) (2.0.1)\n",
"Requirement already satisfied: asttokens>=2.1.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from stack-data->ipython->pyautogen[retrievechat]>=0.2.3) (2.4.1)\n",
"Requirement already satisfied: pure-eval in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from stack-data->ipython->pyautogen[retrievechat]>=0.2.3) (0.2.2)\n",
"Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from torchvision->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (10.2.0)\n",
"Requirement already satisfied: cachetools<6.0,>=2.0.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (5.3.2)\n",
"Requirement already satisfied: pyasn1-modules>=0.2.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (0.3.0)\n",
"Requirement already satisfied: rsa<5,>=3.1.4 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (4.9)\n",
"Requirement already satisfied: zipp>=0.5 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from importlib-metadata<7.0,>=6.0->opentelemetry-api>=1.2.0->chromadb->pyautogen[retrievechat]>=0.2.3) (3.17.0)\n",
"Requirement already satisfied: humanfriendly>=9.1 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from coloredlogs->onnxruntime<2.0,>=1.15->fastembed==0.1.1->qdrant_client[fastembed]) (10.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from jinja2->torch>=1.6.0->sentence-transformers->pyautogen[retrievechat]>=0.2.3) (2.1.3)\n",
"Requirement already satisfied: mpmath>=0.19 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from sympy->onnxruntime<2.0,>=1.15->fastembed==0.1.1->qdrant_client[fastembed]) (1.3.0)\n",
"Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /workspaces/autogen/.venv-3.11/lib/python3.11/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=28.1.0->chromadb->pyautogen[retrievechat]>=0.2.3) (0.5.1)\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install \"pyautogen[retrievechat]>=0.2.3\" \"flaml[automl]\" \"qdrant_client[fastembed]\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set your API Endpoint\n",
"\n",
"The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"models to use: ['gpt-4-1106-preview', 'gpt-4-turbo-preview', 'gpt-4-0613', 'gpt-35-turbo-0613', 'gpt-35-turbo-1106']\n"
]
}
],
"source": [
"from qdrant_client import QdrantClient\n",
"\n",
"import autogen\n",
"from autogen.agentchat.contrib.qdrant_retrieve_user_proxy_agent import QdrantRetrieveUserProxyAgent\n",
"from autogen.agentchat.contrib.retrieve_assistant_agent import RetrieveAssistantAgent\n",
"\n",
"# Accepted file formats for that can be stored in\n",
"# a vector database instance\n",
"from autogen.retrieve_utils import TEXT_FORMATS\n",
"\n",
"config_list = autogen.config_list_from_json(\n",
" env_or_file=\"OAI_CONFIG_LIST\",\n",
" file_location=\".\",\n",
" filter_dict={\n",
" \"model\": {\n",
" \"gpt-4\",\n",
" \"gpt4\",\n",
" \"gpt-4-32k\",\n",
" \"gpt-4-32k-0314\",\n",
" \"gpt-35-turbo\",\n",
" \"gpt-3.5-turbo\",\n",
" }\n",
" },\n",
")\n",
"\n",
"assert len(config_list) > 0\n",
"print(\"models to use: \", [config_list[i][\"model\"] for i in range(len(config_list))])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"````{=mdx}\n",
":::tip\n",
"Learn more about configuring LLMs for agents [here](/docs/topics/llm_configuration).\n",
":::\n",
"````"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accepted file formats for `docs_path`:\n",
"['txt', 'json', 'csv', 'tsv', 'md', 'html', 'htm', 'rtf', 'rst', 'jsonl', 'log', 'xml', 'yaml', 'yml', 'pdf']\n"
]
}
],
"source": [
"print(\"Accepted file formats for `docs_path`:\")\n",
"print(TEXT_FORMATS)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Construct agents for RetrieveChat\n",
"\n",
"We start by initializing the `RetrieveAssistantAgent` and `QdrantRetrieveUserProxyAgent`. The system message needs to be set to \"You are a helpful assistant.\" for RetrieveAssistantAgent. The detailed instructions are given in the user message. Later we will use the `QdrantRetrieveUserProxyAgent.generate_init_prompt` to combine the instructions and a retrieval augmented generation task for an initial prompt to be sent to the LLM assistant.\n",
"\n",
"### You can find the list of all the embedding models supported by Qdrant [here](https://qdrant.github.io/fastembed/examples/Supported_Models/)."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"# 1. create an RetrieveAssistantAgent instance named \"assistant\"\n",
"assistant = RetrieveAssistantAgent(\n",
" name=\"assistant\",\n",
" system_message=\"You are a helpful assistant.\",\n",
" llm_config={\n",
" \"timeout\": 600,\n",
" \"cache_seed\": 42,\n",
" \"config_list\": config_list,\n",
" },\n",
")\n",
"\n",
"# 2. create the QdrantRetrieveUserProxyAgent instance named \"ragproxyagent\"\n",
"# By default, the human_input_mode is \"ALWAYS\", which means the agent will ask for human input at every step. We set it to \"NEVER\" here.\n",
"# `docs_path` is the path to the docs directory. It can also be the path to a single file, or the url to a single file. By default,\n",
"# it is set to None, which works only if the collection is already created.\n",
"#\n",
"# Here we generated the documentations from FLAML's docstrings. Not needed if you just want to try this notebook but not to reproduce the\n",
"# outputs. Clone the FLAML (https://github.com/microsoft/FLAML) repo and navigate to its website folder. Pip install and run `pydoc-markdown`\n",
"# and it will generate folder `reference` under `website/docs`.\n",
"#\n",
"# `task` indicates the kind of task we're working on. In this example, it's a `code` task.\n",
"# `chunk_token_size` is the chunk token size for the retrieve chat. By default, it is set to `max_tokens * 0.6`, here we set it to 2000.\n",
"# We use an in-memory QdrantClient instance here. Not recommended for production.\n",
"# Get the installation instructions here: https://qdrant.tech/documentation/guides/installation/\n",
"ragproxyagent = QdrantRetrieveUserProxyAgent(\n",
" name=\"ragproxyagent\",\n",
" human_input_mode=\"NEVER\",\n",
" max_consecutive_auto_reply=10,\n",
" retrieve_config={\n",
" \"task\": \"code\",\n",
" \"docs_path\": [\n",
" \"https://raw.githubusercontent.com/microsoft/flaml/main/README.md\",\n",
" \"https://raw.githubusercontent.com/microsoft/FLAML/main/website/docs/Research.md\",\n",
" ], # change this to your own path, such as https://raw.githubusercontent.com/microsoft/autogen/main/README.md\n",
" \"chunk_token_size\": 2000,\n",
" \"model\": config_list[0][\"model\"],\n",
" \"client\": QdrantClient(\":memory:\"),\n",
" \"embedding_model\": \"BAAI/bge-small-en-v1.5\",\n",
" },\n",
" # code_execution_config={\n",
" # \"use_docker\": False,}\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Example 1\n",
"\n",
"[back to top](#toc)\n",
"\n",
"Use RetrieveChat to answer a question and ask for human-in-loop feedbacks.\n",
"\n",
"Problem: Is there a function named `tune_automl` in FLAML?"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Trying to create collection.\n",
"\u001b[32mAdding doc_id 0 to context.\u001b[0m\n",
"\u001b[32mAdding doc_id 2 to context.\u001b[0m\n",
"\u001b[32mAdding doc_id 1 to context.\u001b[0m\n",
"\u001b[33mragproxyagent\u001b[0m (to assistant):\n",
"\n",
"You're a retrieve augmented coding assistant. You answer user's questions based on your own knowledge and the\n",
"context provided by the user.\n",
"If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.\n",
"For code generation, you must obey the following rules:\n",
"Rule 1. You MUST NOT install any packages because all the packages needed are already installed.\n",
"Rule 2. You must follow the formats below to write your code:\n",
"```language\n",
"# your code\n",
"```\n",
"\n",
"User's question is: Is there a function called tune_automl?\n",
"\n",
"Context is: [](https://badge.fury.io/py/FLAML)\n",
"\n",
"[](https://github.com/microsoft/FLAML/actions/workflows/python-package.yml)\n",
"\n",
"[](https://pepy.tech/project/flaml)\n",
"[](https://discord.gg/Cppx2vSPVP)\n",
"\n",
"\n",
"\n",
"# A Fast Library for Automated Machine Learning & Tuning\n",
"\n",
"
\n",
" \n",
" \n",
"
\n",
"\n",
":fire: Heads-up: We have migrated [AutoGen](https://microsoft.github.io/autogen/) into a dedicated [github repository](https://github.com/microsoft/autogen). Alongside this move, we have also launched a dedicated [Discord](https://discord.gg/pAbnFJrkgZ) server and a [website](https://microsoft.github.io/autogen/) for comprehensive documentation.\n",
"\n",
":fire: The automated multi-agent chat framework in [AutoGen](https://microsoft.github.io/autogen/) is in preview from v2.0.0.\n",
"\n",
":fire: FLAML is highlighted in OpenAI's [cookbook](https://github.com/openai/openai-cookbook#related-resources-from-around-the-web).\n",
"\n",
":fire: [autogen](https://microsoft.github.io/autogen/) is released with support for ChatGPT and GPT-4, based on [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673).\n",
"\n",
":fire: FLAML supports Code-First AutoML & Tuning – Private Preview in [Microsoft Fabric Data Science](https://learn.microsoft.com/en-us/fabric/data-science/).\n",
"\n",
"\n",
"## What is FLAML\n",
"FLAML is a lightweight Python library for efficient automation of machine\n",
"learning and AI operations. It automates workflow based on large language models, machine learning models, etc.\n",
"and optimizes their performance.\n",
"\n",
"* FLAML enables building next-gen GPT-X applications based on multi-agent conversations with minimal effort. It simplifies the orchestration, automation and optimization of a complex GPT-X workflow. It maximizes the performance of GPT-X models and augments their weakness.\n",
"* For common machine learning tasks like classification and regression, it quickly finds quality models for user-provided data with low computational resources. It is easy to customize or extend. Users can find their desired customizability from a smooth range.\n",
"* It supports fast and economical automatic tuning (e.g., inference hyperparameters for foundation models, configurations in MLOps/LMOps workflows, pipelines, mathematical/statistical models, algorithms, computing experiments, software configurations), capable of handling large search space with heterogeneous evaluation cost and complex constraints/guidance/early stopping.\n",
"\n",
"FLAML is powered by a series of [research studies](https://microsoft.github.io/FLAML/docs/Research/) from Microsoft Research and collaborators such as Penn State University, Stevens Institute of Technology, University of Washington, and University of Waterloo.\n",
"\n",
"FLAML has a .NET implementation in [ML.NET](http://dot.net/ml), an open-source, cross-platform machine learning framework for .NET.\n",
"\n",
"## Installation\n",
"\n",
"FLAML requires **Python version >= 3.8**. It can be installed from pip:\n",
"\n",
"```bash\n",
"pip install flaml\n",
"```\n",
"\n",
"Minimal dependencies are installed without extra options. You can install extra options based on the feature you need. For example, use the following to install the dependencies needed by the [`autogen`](https://microsoft.github.io/autogen/) package.\n",
"```bash\n",
"pip install \"flaml[autogen]\"\n",
"```\n",
"\n",
"Find more options in [Installation](https://microsoft.github.io/FLAML/docs/Installation).\n",
"Each of the [`notebook examples`](https://github.com/microsoft/FLAML/tree/main/notebook) may require a specific option to be installed.\n",
"\n",
"## Quickstart\n",
"\n",
"* (New) The [autogen](https://microsoft.github.io/autogen/) package enables the next-gen GPT-X applications with a generic multi-agent conversation framework.\n",
"It offers customizable and conversable agents which integrate LLMs, tools and human.\n",
"By automating chat among multiple capable agents, one can easily make them collectively perform tasks autonomously or with human feedback, including tasks that require using tools via code. For example,\n",
"```python\n",
"from flaml import autogen\n",
"assistant = autogen.AssistantAgent(\"assistant\")\n",
"user_proxy = autogen.UserProxyAgent(\"user_proxy\")\n",
"user_proxy.initiate_chat(assistant, message=\"Show me the YTD gain of 10 largest technology companies as of today.\")\n",
"# This initiates an automated chat between the two agents to solve the task\n",
"```\n",
"\n",
"Autogen also helps maximize the utility out of the expensive LLMs such as ChatGPT and GPT-4. It offers a drop-in replacement of `openai.Completion` or `openai.ChatCompletion` with powerful functionalites like tuning, caching, templating, filtering. For example, you can optimize generations by LLM with your own tuning data, success metrics and budgets.\n",
"```python\n",
"# perform tuning\n",
"config, analysis = autogen.Completion.tune(\n",
" data=tune_data,\n",
" metric=\"success\",\n",
" mode=\"max\",\n",
" eval_func=eval_func,\n",
" inference_budget=0.05,\n",
" optimization_budget=3,\n",
" num_samples=-1,\n",
")\n",
"# perform inference for a test instance\n",
"response = autogen.Completion.create(context=test_instance, **config)\n",
"```\n",
"* With three lines of code, you can start using this economical and fast\n",
"AutoML engine as a [scikit-learn style estimator](https://microsoft.github.io/FLAML/docs/Use-Cases/Task-Oriented-AutoML).\n",
"\n",
"```python\n",
"from flaml import AutoML\n",
"automl = AutoML()\n",
"automl.fit(X_train, y_train, task=\"classification\")\n",
"```\n",
"\n",
"* You can restrict the learners and use FLAML as a fast hyperparameter tuning\n",
"tool for XGBoost, LightGBM, Random Forest etc. or a [customized learner](https://microsoft.github.io/FLAML/docs/Use-Cases/Task-Oriented-AutoML#estimator-and-search-space).\n",
"\n",
"```python\n",
"automl.fit(X_train, y_train, task=\"classification\", estimator_list=[\"lgbm\"])\n",
"```\n",
"\n",
"* You can also run generic hyperparameter tuning for a [custom function](https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function).\n",
"\n",
"```python\n",
"from flaml import tune\n",
"tune.run(evaluation_function, config={…}, low_cost_partial_config={…}, time_budget_s=3600)\n",
"```\n",
"\n",
"* [Zero-shot AutoML](https://microsoft.github.io/FLAML/docs/Use-Cases/Zero-Shot-AutoML) allows using the existing training API from lightgbm, xgboost etc. while getting the benefit of AutoML in choosing high-performance hyperparameter configurations per task.\n",
"\n",
"```python\n",
"from flaml.default import LGBMRegressor\n",
"\n",
"# Use LGBMRegressor in the same way as you use lightgbm.LGBMRegressor.\n",
"estimator = LGBMRegressor()\n",
"# The hyperparameters are automatically set according to the training data.\n",
"estimator.fit(X_train, y_train)\n",
"```\n",
"\n",
"## Documentation\n",
"\n",
"You can find a detailed documentation about FLAML [here](https://microsoft.github.io/FLAML/).\n",
"\n",
"In addition, you can find:\n",
"\n",
"- [Research](https://microsoft.github.io/FLAML/docs/Research) and [blogposts](https://microsoft.github.io/FLAML/blog) around FLAML.\n",
"\n",
"- [Discord](https://discord.gg/Cppx2vSPVP).\n",
"\n",
"- [Contributing guide](https://microsoft.github.io/FLAML/docs/Contribute).\n",
"\n",
"- ML.NET documentation and tutorials for [Model Builder](https://learn.microsoft.com/dotnet/machine-learning/tutorials/predict-prices-with-model-builder), [ML.NET CLI](https://learn.microsoft.com/dotnet/machine-learning/tutorials/sentiment-analysis-cli), and [AutoML API](https://learn.microsoft.com/dotnet/machine-learning/how-to-guides/how-to-use-the-automl-api).\n",
"\n",
"## Contributing\n",
"\n",
"This project welcomes contributions and suggestions. Most contributions require you to agree to a\n",
"Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\n",
"the rights to use your contribution. For details, visit .\n",
"\n",
"If you are new to GitHub [here](https://help.github.com/categories/collaborating-with-issues-and-pull-requests/) is a detailed help source on getting involved with development on GitHub.\n",
"# Research\n",
"\n",
"For technical details, please check our research publications.\n",
"\n",
"* [FLAML: A Fast and Lightweight AutoML Library](https://www.microsoft.com/en-us/research/publication/flaml-a-fast-and-lightweight-automl-library/). Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu. MLSys 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{wang2021flaml,\n",
" title={FLAML: A Fast and Lightweight AutoML Library},\n",
" author={Chi Wang and Qingyun Wu and Markus Weimer and Erkang Zhu},\n",
" year={2021},\n",
" booktitle={MLSys},\n",
"}\n",
"```\n",
"\n",
"* [Frugal Optimization for Cost-related Hyperparameters](https://arxiv.org/abs/2005.01571). Qingyun Wu, Chi Wang, Silu Huang. AAAI 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{wu2021cfo,\n",
" title={Frugal Optimization for Cost-related Hyperparameters},\n",
" author={Qingyun Wu and Chi Wang and Silu Huang},\n",
" year={2021},\n",
" booktitle={AAAI},\n",
"}\n",
"```\n",
"\n",
"* [Economical Hyperparameter Optimization With Blended Search Strategy](https://www.microsoft.com/en-us/research/publication/economical-hyperparameter-optimization-with-blended-search-strategy/). Chi Wang, Qingyun Wu, Silu Huang, Amin Saied. ICLR 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{wang2021blendsearch,\n",
" title={Economical Hyperparameter Optimization With Blended Search Strategy},\n",
" author={Chi Wang and Qingyun Wu and Silu Huang and Amin Saied},\n",
" year={2021},\n",
" booktitle={ICLR},\n",
"}\n",
"```\n",
"\n",
"* [An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models](https://aclanthology.org/2021.acl-long.178.pdf). Susan Xueqing Liu, Chi Wang. ACL 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{liuwang2021hpolm,\n",
" title={An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models},\n",
" author={Susan Xueqing Liu and Chi Wang},\n",
" year={2021},\n",
" booktitle={ACL},\n",
"}\n",
"```\n",
"\n",
"* [ChaCha for Online AutoML](https://www.microsoft.com/en-us/research/publication/chacha-for-online-automl/). Qingyun Wu, Chi Wang, John Langford, Paul Mineiro and Marco Rossi. ICML 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{wu2021chacha,\n",
" title={ChaCha for Online AutoML},\n",
" author={Qingyun Wu and Chi Wang and John Langford and Paul Mineiro and Marco Rossi},\n",
" year={2021},\n",
" booktitle={ICML},\n",
"}\n",
"```\n",
"\n",
"* [Fair AutoML](https://arxiv.org/abs/2111.06495). Qingyun Wu, Chi Wang. ArXiv preprint arXiv:2111.06495 (2021).\n",
"\n",
"```bibtex\n",
"@inproceedings{wuwang2021fairautoml,\n",
" title={Fair AutoML},\n",
" author={Qingyun Wu and Chi Wang},\n",
" year={2021},\n",
" booktitle={ArXiv preprint arXiv:2111.06495},\n",
"}\n",
"```\n",
"\n",
"* [Mining Robust Default Configurations for Resource-constrained AutoML](https://arxiv.org/abs/2202.09927). Moe Kayali, Chi Wang. ArXiv preprint arXiv:2202.09927 (2022).\n",
"\n",
"```bibtex\n",
"@inproceedings{kayaliwang2022default,\n",
" title={Mining Robust Default Configurations for Resource-constrained AutoML},\n",
" author={Moe Kayali and Chi Wang},\n",
" year={2022},\n",
" booktitle={ArXiv preprint arXiv:2202.09927},\n",
"}\n",
"```\n",
"\n",
"* [Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives](https://openreview.net/forum?id=0Ij9_q567Ma). Shaokun Zhang, Feiran Jia, Chi Wang, Qingyun Wu. ICLR 2023 (notable-top-5%).\n",
"\n",
"```bibtex\n",
"@inproceedings{zhang2023targeted,\n",
" title={Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives},\n",
" author={Shaokun Zhang and Feiran Jia and Chi Wang and Qingyun Wu},\n",
" booktitle={International Conference on Learning Representations},\n",
" year={2023},\n",
" url={https://openreview.net/forum?id=0Ij9_q567Ma},\n",
"}\n",
"```\n",
"\n",
"* [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673). Chi Wang, Susan Xueqing Liu, Ahmed H. Awadallah. ArXiv preprint arXiv:2303.04673 (2023).\n",
"\n",
"```bibtex\n",
"@inproceedings{wang2023EcoOptiGen,\n",
" title={Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference},\n",
" author={Chi Wang and Susan Xueqing Liu and Ahmed H. Awadallah},\n",
" year={2023},\n",
" booktitle={ArXiv preprint arXiv:2303.04673},\n",
"}\n",
"```\n",
"\n",
"* [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337). Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, Chi Wang. ArXiv preprint arXiv:2306.01337 (2023).\n",
"\n",
"```bibtex\n",
"@inproceedings{wu2023empirical,\n",
" title={An Empirical Study on Challenging Math Problem Solving with GPT-4},\n",
" author={Yiran Wu and Feiran Jia and Shaokun Zhang and Hangyu Li and Erkang Zhu and Yue Wang and Yin Tat Lee and Richard Peng and Qingyun Wu and Chi Wang},\n",
" year={2023},\n",
" booktitle={ArXiv preprint arXiv:2306.01337},\n",
"}\n",
"```\n",
"\n",
"\n",
"When you submit a pull request, a CLA bot will automatically determine whether you need to provide\n",
"a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\n",
"provided by the bot. You will only need to do this once across all repos using our CLA.\n",
"\n",
"This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\n",
"For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\n",
"contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n",
"\n",
"\n",
"\n",
"\n",
"--------------------------------------------------------------------------------\n",
"\u001b[33massistant\u001b[0m (to ragproxyagent):\n",
"\n",
"Based on the context provided, which is about the FLAML library, there is no direct reference to a function specifically called `tune_automl`. However, FLAML does offer functionality for automated machine learning (AutoML) and hyperparameter tuning.\n",
"\n",
"The closest reference to an AutoML tuning operation in the given context is shown in the Quickstart section, which demonstrates how to use FLAML as a scikit-learn style estimator for machine learning tasks like classification and regression. It does talk about automated machine learning and tuning, but doesn't mention a function `tune_automl` by name.\n",
"\n",
"If you are looking for a way to perform tuning with FLAML, the context indicates you can use the `tune` module to run generic hyperparameter tuning for a custom function, as demonstrated in the Quickstart section:\n",
"\n",
"```python\n",
"from flaml import tune\n",
"tune.run(evaluation_function, config={…}, low_cost_partial_config={…}, time_budget_s=3600)\n",
"```\n",
"\n",
"This is not called `tune_automl` but rather just `tune.run`.\n",
"\n",
"If you need confirmation on whether a function called `tune_automl` specifically exists, the FLAML documentation or its API reference should be checked. If documentation is not enough to confirm and you require to look into the actual code or a structured list of functionalities provided by FLAML, that information isn't available in the given context.\n",
"\n",
"In that case, the instruction should be: `UPDATE CONTEXT`.\n",
"\n",
"--------------------------------------------------------------------------------\n",
"\u001b[32mUpdating context and resetting conversation.\u001b[0m\n",
"\u001b[32mNo more context, will terminate.\u001b[0m\n",
"\u001b[33mragproxyagent\u001b[0m (to assistant):\n",
"\n",
"TERMINATE\n",
"\n",
"--------------------------------------------------------------------------------\n"
]
},
{
"data": {
"text/plain": [
"ChatResult(chat_id=None, chat_history=[{'content': 'TERMINATE', 'role': 'assistant'}], summary='', cost=({'total_cost': 0.12719999999999998, 'gpt-4': {'cost': 0.12719999999999998, 'prompt_tokens': 3634, 'completion_tokens': 303, 'total_tokens': 3937}}, {'total_cost': 0.12719999999999998, 'gpt-4': {'cost': 0.12719999999999998, 'prompt_tokens': 3634, 'completion_tokens': 303, 'total_tokens': 3937}}), human_input=[])"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# reset the assistant. Always reset the assistant before starting a new conversation.\n",
"assistant.reset()\n",
"\n",
"qa_problem = \"Is there a function called tune_automl?\"\n",
"ragproxyagent.initiate_chat(assistant, message=ragproxyagent.message_generator, problem=qa_problem)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Example 2\n",
"\n",
"[back to top](#toc)\n",
"\n",
"Use RetrieveChat to answer a question that is not related to code generation.\n",
"\n",
"Problem: Who is the author of FLAML?"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[32mAdding doc_id 2 to context.\u001b[0m\n",
"\u001b[32mAdding doc_id 0 to context.\u001b[0m\n",
"\u001b[32mAdding doc_id 1 to context.\u001b[0m\n",
"\u001b[33mragproxyagent\u001b[0m (to assistant):\n",
"\n",
"You're a retrieve augmented coding assistant. You answer user's questions based on your own knowledge and the\n",
"context provided by the user.\n",
"If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.\n",
"For code generation, you must obey the following rules:\n",
"Rule 1. You MUST NOT install any packages because all the packages needed are already installed.\n",
"Rule 2. You must follow the formats below to write your code:\n",
"```language\n",
"# your code\n",
"```\n",
"\n",
"User's question is: Who is the author of FLAML?\n",
"\n",
"Context is: # Research\n",
"\n",
"For technical details, please check our research publications.\n",
"\n",
"* [FLAML: A Fast and Lightweight AutoML Library](https://www.microsoft.com/en-us/research/publication/flaml-a-fast-and-lightweight-automl-library/). Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu. MLSys 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{wang2021flaml,\n",
" title={FLAML: A Fast and Lightweight AutoML Library},\n",
" author={Chi Wang and Qingyun Wu and Markus Weimer and Erkang Zhu},\n",
" year={2021},\n",
" booktitle={MLSys},\n",
"}\n",
"```\n",
"\n",
"* [Frugal Optimization for Cost-related Hyperparameters](https://arxiv.org/abs/2005.01571). Qingyun Wu, Chi Wang, Silu Huang. AAAI 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{wu2021cfo,\n",
" title={Frugal Optimization for Cost-related Hyperparameters},\n",
" author={Qingyun Wu and Chi Wang and Silu Huang},\n",
" year={2021},\n",
" booktitle={AAAI},\n",
"}\n",
"```\n",
"\n",
"* [Economical Hyperparameter Optimization With Blended Search Strategy](https://www.microsoft.com/en-us/research/publication/economical-hyperparameter-optimization-with-blended-search-strategy/). Chi Wang, Qingyun Wu, Silu Huang, Amin Saied. ICLR 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{wang2021blendsearch,\n",
" title={Economical Hyperparameter Optimization With Blended Search Strategy},\n",
" author={Chi Wang and Qingyun Wu and Silu Huang and Amin Saied},\n",
" year={2021},\n",
" booktitle={ICLR},\n",
"}\n",
"```\n",
"\n",
"* [An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models](https://aclanthology.org/2021.acl-long.178.pdf). Susan Xueqing Liu, Chi Wang. ACL 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{liuwang2021hpolm,\n",
" title={An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models},\n",
" author={Susan Xueqing Liu and Chi Wang},\n",
" year={2021},\n",
" booktitle={ACL},\n",
"}\n",
"```\n",
"\n",
"* [ChaCha for Online AutoML](https://www.microsoft.com/en-us/research/publication/chacha-for-online-automl/). Qingyun Wu, Chi Wang, John Langford, Paul Mineiro and Marco Rossi. ICML 2021.\n",
"\n",
"```bibtex\n",
"@inproceedings{wu2021chacha,\n",
" title={ChaCha for Online AutoML},\n",
" author={Qingyun Wu and Chi Wang and John Langford and Paul Mineiro and Marco Rossi},\n",
" year={2021},\n",
" booktitle={ICML},\n",
"}\n",
"```\n",
"\n",
"* [Fair AutoML](https://arxiv.org/abs/2111.06495). Qingyun Wu, Chi Wang. ArXiv preprint arXiv:2111.06495 (2021).\n",
"\n",
"```bibtex\n",
"@inproceedings{wuwang2021fairautoml,\n",
" title={Fair AutoML},\n",
" author={Qingyun Wu and Chi Wang},\n",
" year={2021},\n",
" booktitle={ArXiv preprint arXiv:2111.06495},\n",
"}\n",
"```\n",
"\n",
"* [Mining Robust Default Configurations for Resource-constrained AutoML](https://arxiv.org/abs/2202.09927). Moe Kayali, Chi Wang. ArXiv preprint arXiv:2202.09927 (2022).\n",
"\n",
"```bibtex\n",
"@inproceedings{kayaliwang2022default,\n",
" title={Mining Robust Default Configurations for Resource-constrained AutoML},\n",
" author={Moe Kayali and Chi Wang},\n",
" year={2022},\n",
" booktitle={ArXiv preprint arXiv:2202.09927},\n",
"}\n",
"```\n",
"\n",
"* [Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives](https://openreview.net/forum?id=0Ij9_q567Ma). Shaokun Zhang, Feiran Jia, Chi Wang, Qingyun Wu. ICLR 2023 (notable-top-5%).\n",
"\n",
"```bibtex\n",
"@inproceedings{zhang2023targeted,\n",
" title={Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives},\n",
" author={Shaokun Zhang and Feiran Jia and Chi Wang and Qingyun Wu},\n",
" booktitle={International Conference on Learning Representations},\n",
" year={2023},\n",
" url={https://openreview.net/forum?id=0Ij9_q567Ma},\n",
"}\n",
"```\n",
"\n",
"* [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673). Chi Wang, Susan Xueqing Liu, Ahmed H. Awadallah. ArXiv preprint arXiv:2303.04673 (2023).\n",
"\n",
"```bibtex\n",
"@inproceedings{wang2023EcoOptiGen,\n",
" title={Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference},\n",
" author={Chi Wang and Susan Xueqing Liu and Ahmed H. Awadallah},\n",
" year={2023},\n",
" booktitle={ArXiv preprint arXiv:2303.04673},\n",
"}\n",
"```\n",
"\n",
"* [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337). Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, Chi Wang. ArXiv preprint arXiv:2306.01337 (2023).\n",
"\n",
"```bibtex\n",
"@inproceedings{wu2023empirical,\n",
" title={An Empirical Study on Challenging Math Problem Solving with GPT-4},\n",
" author={Yiran Wu and Feiran Jia and Shaokun Zhang and Hangyu Li and Erkang Zhu and Yue Wang and Yin Tat Lee and Richard Peng and Qingyun Wu and Chi Wang},\n",
" year={2023},\n",
" booktitle={ArXiv preprint arXiv:2306.01337},\n",
"}\n",
"```\n",
"\n",
"[](https://badge.fury.io/py/FLAML)\n",
"\n",
"[](https://github.com/microsoft/FLAML/actions/workflows/python-package.yml)\n",
"\n",
"[](https://pepy.tech/project/flaml)\n",
"[](https://discord.gg/Cppx2vSPVP)\n",
"\n",
"\n",
"\n",
"# A Fast Library for Automated Machine Learning & Tuning\n",
"\n",
"
\n",
" \n",
" \n",
"
\n",
"\n",
":fire: Heads-up: We have migrated [AutoGen](https://microsoft.github.io/autogen/) into a dedicated [github repository](https://github.com/microsoft/autogen). Alongside this move, we have also launched a dedicated [Discord](https://discord.gg/pAbnFJrkgZ) server and a [website](https://microsoft.github.io/autogen/) for comprehensive documentation.\n",
"\n",
":fire: The automated multi-agent chat framework in [AutoGen](https://microsoft.github.io/autogen/) is in preview from v2.0.0.\n",
"\n",
":fire: FLAML is highlighted in OpenAI's [cookbook](https://github.com/openai/openai-cookbook#related-resources-from-around-the-web).\n",
"\n",
":fire: [autogen](https://microsoft.github.io/autogen/) is released with support for ChatGPT and GPT-4, based on [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673).\n",
"\n",
":fire: FLAML supports Code-First AutoML & Tuning – Private Preview in [Microsoft Fabric Data Science](https://learn.microsoft.com/en-us/fabric/data-science/).\n",
"\n",
"\n",
"## What is FLAML\n",
"FLAML is a lightweight Python library for efficient automation of machine\n",
"learning and AI operations. It automates workflow based on large language models, machine learning models, etc.\n",
"and optimizes their performance.\n",
"\n",
"* FLAML enables building next-gen GPT-X applications based on multi-agent conversations with minimal effort. It simplifies the orchestration, automation and optimization of a complex GPT-X workflow. It maximizes the performance of GPT-X models and augments their weakness.\n",
"* For common machine learning tasks like classification and regression, it quickly finds quality models for user-provided data with low computational resources. It is easy to customize or extend. Users can find their desired customizability from a smooth range.\n",
"* It supports fast and economical automatic tuning (e.g., inference hyperparameters for foundation models, configurations in MLOps/LMOps workflows, pipelines, mathematical/statistical models, algorithms, computing experiments, software configurations), capable of handling large search space with heterogeneous evaluation cost and complex constraints/guidance/early stopping.\n",
"\n",
"FLAML is powered by a series of [research studies](https://microsoft.github.io/FLAML/docs/Research/) from Microsoft Research and collaborators such as Penn State University, Stevens Institute of Technology, University of Washington, and University of Waterloo.\n",
"\n",
"FLAML has a .NET implementation in [ML.NET](http://dot.net/ml), an open-source, cross-platform machine learning framework for .NET.\n",
"\n",
"## Installation\n",
"\n",
"FLAML requires **Python version >= 3.8**. It can be installed from pip:\n",
"\n",
"```bash\n",
"pip install flaml\n",
"```\n",
"\n",
"Minimal dependencies are installed without extra options. You can install extra options based on the feature you need. For example, use the following to install the dependencies needed by the [`autogen`](https://microsoft.github.io/autogen/) package.\n",
"```bash\n",
"pip install \"flaml[autogen]\"\n",
"```\n",
"\n",
"Find more options in [Installation](https://microsoft.github.io/FLAML/docs/Installation).\n",
"Each of the [`notebook examples`](https://github.com/microsoft/FLAML/tree/main/notebook) may require a specific option to be installed.\n",
"\n",
"## Quickstart\n",
"\n",
"* (New) The [autogen](https://microsoft.github.io/autogen/) package enables the next-gen GPT-X applications with a generic multi-agent conversation framework.\n",
"It offers customizable and conversable agents which integrate LLMs, tools and human.\n",
"By automating chat among multiple capable agents, one can easily make them collectively perform tasks autonomously or with human feedback, including tasks that require using tools via code. For example,\n",
"```python\n",
"from flaml import autogen\n",
"assistant = autogen.AssistantAgent(\"assistant\")\n",
"user_proxy = autogen.UserProxyAgent(\"user_proxy\")\n",
"user_proxy.initiate_chat(assistant, message=\"Show me the YTD gain of 10 largest technology companies as of today.\")\n",
"# This initiates an automated chat between the two agents to solve the task\n",
"```\n",
"\n",
"Autogen also helps maximize the utility out of the expensive LLMs such as ChatGPT and GPT-4. It offers a drop-in replacement of `openai.Completion` or `openai.ChatCompletion` with powerful functionalites like tuning, caching, templating, filtering. For example, you can optimize generations by LLM with your own tuning data, success metrics and budgets.\n",
"```python\n",
"# perform tuning\n",
"config, analysis = autogen.Completion.tune(\n",
" data=tune_data,\n",
" metric=\"success\",\n",
" mode=\"max\",\n",
" eval_func=eval_func,\n",
" inference_budget=0.05,\n",
" optimization_budget=3,\n",
" num_samples=-1,\n",
")\n",
"# perform inference for a test instance\n",
"response = autogen.Completion.create(context=test_instance, **config)\n",
"```\n",
"* With three lines of code, you can start using this economical and fast\n",
"AutoML engine as a [scikit-learn style estimator](https://microsoft.github.io/FLAML/docs/Use-Cases/Task-Oriented-AutoML).\n",
"\n",
"```python\n",
"from flaml import AutoML\n",
"automl = AutoML()\n",
"automl.fit(X_train, y_train, task=\"classification\")\n",
"```\n",
"\n",
"* You can restrict the learners and use FLAML as a fast hyperparameter tuning\n",
"tool for XGBoost, LightGBM, Random Forest etc. or a [customized learner](https://microsoft.github.io/FLAML/docs/Use-Cases/Task-Oriented-AutoML#estimator-and-search-space).\n",
"\n",
"```python\n",
"automl.fit(X_train, y_train, task=\"classification\", estimator_list=[\"lgbm\"])\n",
"```\n",
"\n",
"* You can also run generic hyperparameter tuning for a [custom function](https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function).\n",
"\n",
"```python\n",
"from flaml import tune\n",
"tune.run(evaluation_function, config={…}, low_cost_partial_config={…}, time_budget_s=3600)\n",
"```\n",
"\n",
"* [Zero-shot AutoML](https://microsoft.github.io/FLAML/docs/Use-Cases/Zero-Shot-AutoML) allows using the existing training API from lightgbm, xgboost etc. while getting the benefit of AutoML in choosing high-performance hyperparameter configurations per task.\n",
"\n",
"```python\n",
"from flaml.default import LGBMRegressor\n",
"\n",
"# Use LGBMRegressor in the same way as you use lightgbm.LGBMRegressor.\n",
"estimator = LGBMRegressor()\n",
"# The hyperparameters are automatically set according to the training data.\n",
"estimator.fit(X_train, y_train)\n",
"```\n",
"\n",
"## Documentation\n",
"\n",
"You can find a detailed documentation about FLAML [here](https://microsoft.github.io/FLAML/).\n",
"\n",
"In addition, you can find:\n",
"\n",
"- [Research](https://microsoft.github.io/FLAML/docs/Research) and [blogposts](https://microsoft.github.io/FLAML/blog) around FLAML.\n",
"\n",
"- [Discord](https://discord.gg/Cppx2vSPVP).\n",
"\n",
"- [Contributing guide](https://microsoft.github.io/FLAML/docs/Contribute).\n",
"\n",
"- ML.NET documentation and tutorials for [Model Builder](https://learn.microsoft.com/dotnet/machine-learning/tutorials/predict-prices-with-model-builder), [ML.NET CLI](https://learn.microsoft.com/dotnet/machine-learning/tutorials/sentiment-analysis-cli), and [AutoML API](https://learn.microsoft.com/dotnet/machine-learning/how-to-guides/how-to-use-the-automl-api).\n",
"\n",
"## Contributing\n",
"\n",
"This project welcomes contributions and suggestions. Most contributions require you to agree to a\n",
"Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\n",
"the rights to use your contribution. For details, visit .\n",
"\n",
"If you are new to GitHub [here](https://help.github.com/categories/collaborating-with-issues-and-pull-requests/) is a detailed help source on getting involved with development on GitHub.\n",
"\n",
"When you submit a pull request, a CLA bot will automatically determine whether you need to provide\n",
"a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\n",
"provided by the bot. You will only need to do this once across all repos using our CLA.\n",
"\n",
"This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\n",
"For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\n",
"contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n",
"\n",
"\n",
"\n",
"\n",
"--------------------------------------------------------------------------------\n",
"\u001b[33massistant\u001b[0m (to ragproxyagent):\n",
"\n",
"The author of FLAML is Chi Wang, along with other collaborators including Qingyun Wu, Markus Weimer, Erkang Zhu, Silu Huang, Amin Saied, Susan Xueqing Liu, John Langford, Paul Mineiro, Marco Rossi, Moe Kayali, Shaokun Zhang, Feiran Jia, Yiran Wu, Hangyu Li, Yue Wang, Yin Tat Lee, Richard Peng, and Ahmed H. Awadallah, as indicated in the provided references for FLAML's research publications.\n",
"\n",
"--------------------------------------------------------------------------------\n"
]
},
{
"data": {
"text/plain": [
"ChatResult(chat_id=None, chat_history=[{'content': 'You\\'re a retrieve augmented coding assistant. You answer user\\'s questions based on your own knowledge and the\\ncontext provided by the user.\\nIf you can\\'t answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.\\nFor code generation, you must obey the following rules:\\nRule 1. You MUST NOT install any packages because all the packages needed are already installed.\\nRule 2. You must follow the formats below to write your code:\\n```language\\n# your code\\n```\\n\\nUser\\'s question is: Who is the author of FLAML?\\n\\nContext is: # Research\\n\\nFor technical details, please check our research publications.\\n\\n* [FLAML: A Fast and Lightweight AutoML Library](https://www.microsoft.com/en-us/research/publication/flaml-a-fast-and-lightweight-automl-library/). Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu. MLSys 2021.\\n\\n```bibtex\\n@inproceedings{wang2021flaml,\\n title={FLAML: A Fast and Lightweight AutoML Library},\\n author={Chi Wang and Qingyun Wu and Markus Weimer and Erkang Zhu},\\n year={2021},\\n booktitle={MLSys},\\n}\\n```\\n\\n* [Frugal Optimization for Cost-related Hyperparameters](https://arxiv.org/abs/2005.01571). Qingyun Wu, Chi Wang, Silu Huang. AAAI 2021.\\n\\n```bibtex\\n@inproceedings{wu2021cfo,\\n title={Frugal Optimization for Cost-related Hyperparameters},\\n author={Qingyun Wu and Chi Wang and Silu Huang},\\n year={2021},\\n booktitle={AAAI},\\n}\\n```\\n\\n* [Economical Hyperparameter Optimization With Blended Search Strategy](https://www.microsoft.com/en-us/research/publication/economical-hyperparameter-optimization-with-blended-search-strategy/). Chi Wang, Qingyun Wu, Silu Huang, Amin Saied. ICLR 2021.\\n\\n```bibtex\\n@inproceedings{wang2021blendsearch,\\n title={Economical Hyperparameter Optimization With Blended Search Strategy},\\n author={Chi Wang and Qingyun Wu and Silu Huang and Amin Saied},\\n year={2021},\\n booktitle={ICLR},\\n}\\n```\\n\\n* [An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models](https://aclanthology.org/2021.acl-long.178.pdf). Susan Xueqing Liu, Chi Wang. ACL 2021.\\n\\n```bibtex\\n@inproceedings{liuwang2021hpolm,\\n title={An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models},\\n author={Susan Xueqing Liu and Chi Wang},\\n year={2021},\\n booktitle={ACL},\\n}\\n```\\n\\n* [ChaCha for Online AutoML](https://www.microsoft.com/en-us/research/publication/chacha-for-online-automl/). Qingyun Wu, Chi Wang, John Langford, Paul Mineiro and Marco Rossi. ICML 2021.\\n\\n```bibtex\\n@inproceedings{wu2021chacha,\\n title={ChaCha for Online AutoML},\\n author={Qingyun Wu and Chi Wang and John Langford and Paul Mineiro and Marco Rossi},\\n year={2021},\\n booktitle={ICML},\\n}\\n```\\n\\n* [Fair AutoML](https://arxiv.org/abs/2111.06495). Qingyun Wu, Chi Wang. ArXiv preprint arXiv:2111.06495 (2021).\\n\\n```bibtex\\n@inproceedings{wuwang2021fairautoml,\\n title={Fair AutoML},\\n author={Qingyun Wu and Chi Wang},\\n year={2021},\\n booktitle={ArXiv preprint arXiv:2111.06495},\\n}\\n```\\n\\n* [Mining Robust Default Configurations for Resource-constrained AutoML](https://arxiv.org/abs/2202.09927). Moe Kayali, Chi Wang. ArXiv preprint arXiv:2202.09927 (2022).\\n\\n```bibtex\\n@inproceedings{kayaliwang2022default,\\n title={Mining Robust Default Configurations for Resource-constrained AutoML},\\n author={Moe Kayali and Chi Wang},\\n year={2022},\\n booktitle={ArXiv preprint arXiv:2202.09927},\\n}\\n```\\n\\n* [Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives](https://openreview.net/forum?id=0Ij9_q567Ma). Shaokun Zhang, Feiran Jia, Chi Wang, Qingyun Wu. ICLR 2023 (notable-top-5%).\\n\\n```bibtex\\n@inproceedings{zhang2023targeted,\\n title={Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives},\\n author={Shaokun Zhang and Feiran Jia and Chi Wang and Qingyun Wu},\\n booktitle={International Conference on Learning Representations},\\n year={2023},\\n url={https://openreview.net/forum?id=0Ij9_q567Ma},\\n}\\n```\\n\\n* [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673). Chi Wang, Susan Xueqing Liu, Ahmed H. Awadallah. ArXiv preprint arXiv:2303.04673 (2023).\\n\\n```bibtex\\n@inproceedings{wang2023EcoOptiGen,\\n title={Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference},\\n author={Chi Wang and Susan Xueqing Liu and Ahmed H. Awadallah},\\n year={2023},\\n booktitle={ArXiv preprint arXiv:2303.04673},\\n}\\n```\\n\\n* [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337). Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, Chi Wang. ArXiv preprint arXiv:2306.01337 (2023).\\n\\n```bibtex\\n@inproceedings{wu2023empirical,\\n title={An Empirical Study on Challenging Math Problem Solving with GPT-4},\\n author={Yiran Wu and Feiran Jia and Shaokun Zhang and Hangyu Li and Erkang Zhu and Yue Wang and Yin Tat Lee and Richard Peng and Qingyun Wu and Chi Wang},\\n year={2023},\\n booktitle={ArXiv preprint arXiv:2306.01337},\\n}\\n```\\n\\n[](https://badge.fury.io/py/FLAML)\\n\\n[](https://github.com/microsoft/FLAML/actions/workflows/python-package.yml)\\n\\n[](https://pepy.tech/project/flaml)\\n[](https://discord.gg/Cppx2vSPVP)\\n\\n\\n\\n# A Fast Library for Automated Machine Learning & Tuning\\n\\n
\\n \\n \\n
\\n\\n:fire: Heads-up: We have migrated [AutoGen](https://microsoft.github.io/autogen/) into a dedicated [github repository](https://github.com/microsoft/autogen). Alongside this move, we have also launched a dedicated [Discord](https://discord.gg/pAbnFJrkgZ) server and a [website](https://microsoft.github.io/autogen/) for comprehensive documentation.\\n\\n:fire: The automated multi-agent chat framework in [AutoGen](https://microsoft.github.io/autogen/) is in preview from v2.0.0.\\n\\n:fire: FLAML is highlighted in OpenAI\\'s [cookbook](https://github.com/openai/openai-cookbook#related-resources-from-around-the-web).\\n\\n:fire: [autogen](https://microsoft.github.io/autogen/) is released with support for ChatGPT and GPT-4, based on [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://arxiv.org/abs/2303.04673).\\n\\n:fire: FLAML supports Code-First AutoML & Tuning – Private Preview in [Microsoft Fabric Data Science](https://learn.microsoft.com/en-us/fabric/data-science/).\\n\\n\\n## What is FLAML\\nFLAML is a lightweight Python library for efficient automation of machine\\nlearning and AI operations. It automates workflow based on large language models, machine learning models, etc.\\nand optimizes their performance.\\n\\n* FLAML enables building next-gen GPT-X applications based on multi-agent conversations with minimal effort. It simplifies the orchestration, automation and optimization of a complex GPT-X workflow. It maximizes the performance of GPT-X models and augments their weakness.\\n* For common machine learning tasks like classification and regression, it quickly finds quality models for user-provided data with low computational resources. It is easy to customize or extend. Users can find their desired customizability from a smooth range.\\n* It supports fast and economical automatic tuning (e.g., inference hyperparameters for foundation models, configurations in MLOps/LMOps workflows, pipelines, mathematical/statistical models, algorithms, computing experiments, software configurations), capable of handling large search space with heterogeneous evaluation cost and complex constraints/guidance/early stopping.\\n\\nFLAML is powered by a series of [research studies](https://microsoft.github.io/FLAML/docs/Research/) from Microsoft Research and collaborators such as Penn State University, Stevens Institute of Technology, University of Washington, and University of Waterloo.\\n\\nFLAML has a .NET implementation in [ML.NET](http://dot.net/ml), an open-source, cross-platform machine learning framework for .NET.\\n\\n## Installation\\n\\nFLAML requires **Python version >= 3.8**. It can be installed from pip:\\n\\n```bash\\npip install flaml\\n```\\n\\nMinimal dependencies are installed without extra options. You can install extra options based on the feature you need. For example, use the following to install the dependencies needed by the [`autogen`](https://microsoft.github.io/autogen/) package.\\n```bash\\npip install \"flaml[autogen]\"\\n```\\n\\nFind more options in [Installation](https://microsoft.github.io/FLAML/docs/Installation).\\nEach of the [`notebook examples`](https://github.com/microsoft/FLAML/tree/main/notebook) may require a specific option to be installed.\\n\\n## Quickstart\\n\\n* (New) The [autogen](https://microsoft.github.io/autogen/) package enables the next-gen GPT-X applications with a generic multi-agent conversation framework.\\nIt offers customizable and conversable agents which integrate LLMs, tools and human.\\nBy automating chat among multiple capable agents, one can easily make them collectively perform tasks autonomously or with human feedback, including tasks that require using tools via code. For example,\\n```python\\nfrom flaml import autogen\\nassistant = autogen.AssistantAgent(\"assistant\")\\nuser_proxy = autogen.UserProxyAgent(\"user_proxy\")\\nuser_proxy.initiate_chat(assistant, message=\"Show me the YTD gain of 10 largest technology companies as of today.\")\\n# This initiates an automated chat between the two agents to solve the task\\n```\\n\\nAutogen also helps maximize the utility out of the expensive LLMs such as ChatGPT and GPT-4. It offers a drop-in replacement of `openai.Completion` or `openai.ChatCompletion` with powerful functionalites like tuning, caching, templating, filtering. For example, you can optimize generations by LLM with your own tuning data, success metrics and budgets.\\n```python\\n# perform tuning\\nconfig, analysis = autogen.Completion.tune(\\n data=tune_data,\\n metric=\"success\",\\n mode=\"max\",\\n eval_func=eval_func,\\n inference_budget=0.05,\\n optimization_budget=3,\\n num_samples=-1,\\n)\\n# perform inference for a test instance\\nresponse = autogen.Completion.create(context=test_instance, **config)\\n```\\n* With three lines of code, you can start using this economical and fast\\nAutoML engine as a [scikit-learn style estimator](https://microsoft.github.io/FLAML/docs/Use-Cases/Task-Oriented-AutoML).\\n\\n```python\\nfrom flaml import AutoML\\nautoml = AutoML()\\nautoml.fit(X_train, y_train, task=\"classification\")\\n```\\n\\n* You can restrict the learners and use FLAML as a fast hyperparameter tuning\\ntool for XGBoost, LightGBM, Random Forest etc. or a [customized learner](https://microsoft.github.io/FLAML/docs/Use-Cases/Task-Oriented-AutoML#estimator-and-search-space).\\n\\n```python\\nautoml.fit(X_train, y_train, task=\"classification\", estimator_list=[\"lgbm\"])\\n```\\n\\n* You can also run generic hyperparameter tuning for a [custom function](https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function).\\n\\n```python\\nfrom flaml import tune\\ntune.run(evaluation_function, config={…}, low_cost_partial_config={…}, time_budget_s=3600)\\n```\\n\\n* [Zero-shot AutoML](https://microsoft.github.io/FLAML/docs/Use-Cases/Zero-Shot-AutoML) allows using the existing training API from lightgbm, xgboost etc. while getting the benefit of AutoML in choosing high-performance hyperparameter configurations per task.\\n\\n```python\\nfrom flaml.default import LGBMRegressor\\n\\n# Use LGBMRegressor in the same way as you use lightgbm.LGBMRegressor.\\nestimator = LGBMRegressor()\\n# The hyperparameters are automatically set according to the training data.\\nestimator.fit(X_train, y_train)\\n```\\n\\n## Documentation\\n\\nYou can find a detailed documentation about FLAML [here](https://microsoft.github.io/FLAML/).\\n\\nIn addition, you can find:\\n\\n- [Research](https://microsoft.github.io/FLAML/docs/Research) and [blogposts](https://microsoft.github.io/FLAML/blog) around FLAML.\\n\\n- [Discord](https://discord.gg/Cppx2vSPVP).\\n\\n- [Contributing guide](https://microsoft.github.io/FLAML/docs/Contribute).\\n\\n- ML.NET documentation and tutorials for [Model Builder](https://learn.microsoft.com/dotnet/machine-learning/tutorials/predict-prices-with-model-builder), [ML.NET CLI](https://learn.microsoft.com/dotnet/machine-learning/tutorials/sentiment-analysis-cli), and [AutoML API](https://learn.microsoft.com/dotnet/machine-learning/how-to-guides/how-to-use-the-automl-api).\\n\\n## Contributing\\n\\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\\nthe rights to use your contribution. For details, visit .\\n\\nIf you are new to GitHub [here](https://help.github.com/categories/collaborating-with-issues-and-pull-requests/) is a detailed help source on getting involved with development on GitHub.\\n\\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\\nprovided by the bot. You will only need to do this once across all repos using our CLA.\\n\\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\\n\\n\\n', 'role': 'assistant'}, {'content': \"The author of FLAML is Chi Wang, along with other collaborators including Qingyun Wu, Markus Weimer, Erkang Zhu, Silu Huang, Amin Saied, Susan Xueqing Liu, John Langford, Paul Mineiro, Marco Rossi, Moe Kayali, Shaokun Zhang, Feiran Jia, Yiran Wu, Hangyu Li, Yue Wang, Yin Tat Lee, Richard Peng, and Ahmed H. Awadallah, as indicated in the provided references for FLAML's research publications.\", 'role': 'user'}], summary=\"The author of FLAML is Chi Wang, along with other collaborators including Qingyun Wu, Markus Weimer, Erkang Zhu, Silu Huang, Amin Saied, Susan Xueqing Liu, John Langford, Paul Mineiro, Marco Rossi, Moe Kayali, Shaokun Zhang, Feiran Jia, Yiran Wu, Hangyu Li, Yue Wang, Yin Tat Lee, Richard Peng, and Ahmed H. Awadallah, as indicated in the provided references for FLAML's research publications.\", cost=({'total_cost': 0.11538, 'gpt-4': {'cost': 0.11538, 'prompt_tokens': 3632, 'completion_tokens': 107, 'total_tokens': 3739}}, {'total_cost': 0}), human_input=[])"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# reset the assistant. Always reset the assistant before starting a new conversation.\n",
"assistant.reset()\n",
"\n",
"qa_problem = \"Who is the author of FLAML?\"\n",
"ragproxyagent.initiate_chat(assistant, message=ragproxyagent.message_generator, problem=qa_problem)"
]
}
],
"metadata": {
"front_matter": {
"tags": ["rag"],
"description": "This notebook demonstrates the usage of QdrantRetrieveUserProxyAgent for RAG."
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
![](\"https://github.com/microsoft/FLAML/blob/main/website/static/img/flaml.svg\")
\n",
"