{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_chatgpt_gpt4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved. \n",
    "\n",
    "Licensed under the MIT License.\n",
    "\n",
    "# Use FLAML to Tune ChatGPT\n",
    "\n",
    "FLAML offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. Our study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
    "\n",
    "In this notebook, we tune OpenAI ChatGPT (both GPT-3.5 and GPT-4) models for math problem solving. We use [the MATH benchmark](https://crfm.stanford.edu/helm/latest/?group=math_chain_of_thought) for measuring mathematical problem solving on competition math problems with chain-of-thoughts style reasoning.\n",
    "\n",
    "Related link: [Blogpost](https://microsoft.github.io/FLAML/blog/2023/04/21/LLM-tuning-math) based on this experiment.\n",
    "\n",
    "## Requirements\n",
    "\n",
    "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [openai,blendsearch] option:\n",
    "```bash\n",
    "pip install flaml[openai,blendsearch]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
     "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
     "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
     "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
    }
   },
   "outputs": [],
   "source": [
    "# %pip install flaml[openai,blendsearch] datasets"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the oai and tune subpackages from flaml\n",
    "\n",
    "FLAML has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `oai.ChatCompletion.tune` and to make a request with the tuned config: `oai.ChatCompletion.create`. First, we import oai from flaml:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:54.634335Z",
     "iopub.status.busy": "2023-02-13T23:40:54.633929Z",
     "iopub.status.idle": "2023-02-13T23:40:56.105700Z",
     "shell.execute_reply": "2023-02-13T23:40:56.105085Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from flaml import oai"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set your API Endpoint\n",
    "\n",
    "The [`config_list_openai_aoai`](https://microsoft.github.io/FLAML/docs/reference/autogen/oai/openai_utils#config_list_openai_aoai) function tries to create a list of  Azure OpenAI endpoints and OpenAI endpoints. It assumes the api keys and api bases are stored in the corresponding environment variables or local txt files:\n",
    "\n",
    "- OpenAI API key: os.environ[\"OPENAI_API_KEY\"] or `openai_api_key_file=\"key_openai.txt\"`.\n",
    "- Azure OpenAI API key: os.environ[\"AZURE_OPENAI_API_KEY\"] or `aoai_api_key_file=\"key_aoai.txt\"`. Multiple keys can be stored, one per line.\n",
    "- Azure OpenAI API base: os.environ[\"AZURE_OPENAI_API_BASE\"] or `aoai_api_base_file=\"base_aoai.txt\"`. Multiple bases can be stored, one per line.\n",
    "\n",
    "It's OK to have only the OpenAI API key, or only the Azure OpenAI API key + base.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:52.324240Z",
     "iopub.status.busy": "2023-02-13T23:40:52.323783Z",
     "iopub.status.idle": "2023-02-13T23:40:52.330570Z",
     "shell.execute_reply": "2023-02-13T23:40:52.329750Z"
    }
   },
   "outputs": [],
   "source": [
    "config_list = oai.config_list_openai_aoai()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The config list looks like the following:\n",
    "```python\n",
    "config_list = [\n",
    "    {'api_key': '<your OpenAI API key here>'},  # only if OpenAI API key is found\n",
    "    {\n",
    "        'api_key': '<your first Azure OpenAI API key here>',\n",
    "        'api_base': '<your first Azure OpenAI API base here>',\n",
    "        'api_type': 'azure',\n",
    "        'api_version': '2023-03-15-preview',\n",
    "    },  # only if the at least one Azure OpenAI API key is found\n",
    "    {\n",
    "        'api_key': '<your second Azure OpenAI API key here>',\n",
    "        'api_base': '<your second Azure OpenAI API base here>',\n",
    "        'api_type': 'azure',\n",
    "        'api_version': '2023-03-15-preview',\n",
    "    },  # only if the second Azure OpenAI API key is found\n",
    "]\n",
    "```\n",
    "\n",
    "You can directly override it if the above function returns an empty list, i.e., it doesn't find the keys in the specified locations."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "We load the competition_math dataset. The dataset contains 201 \"Level 2\" Algebra examples. We use a random sample of 20 examples for tuning the generation hyperparameters and the remaining for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:52.339977Z",
     "iopub.status.busy": "2023-02-13T23:40:52.339556Z",
     "iopub.status.idle": "2023-02-13T23:40:54.603349Z",
     "shell.execute_reply": "2023-02-13T23:40:54.602630Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default\n",
      "Found cached dataset competition_math (/home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9e8a0c134dcc447db7395ad52f12f8bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b/cache-f1cfe8228271b121.arrow\n",
      "Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b/cache-d155a2d38c23bd53.arrow\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 201\n"
     ]
    }
   ],
   "source": [
    "import datasets\n",
    "\n",
    "seed = 41\n",
    "data = datasets.load_dataset(\"competition_math\")\n",
    "train_data = data[\"train\"].shuffle(seed=seed)\n",
    "test_data = data[\"test\"].shuffle(seed=seed)\n",
    "n_tune_data = 20\n",
    "tune_data = [\n",
    "    {\n",
    "        \"problem\": train_data[x][\"problem\"],\n",
    "        \"solution\": train_data[x][\"solution\"],\n",
    "    }\n",
    "    for x in range(len(train_data)) if train_data[x][\"level\"] == \"Level 2\" and train_data[x][\"type\"] == \"Algebra\"\n",
    "][:n_tune_data]\n",
    "test_data = [\n",
    "    {\n",
    "        \"problem\": test_data[x][\"problem\"],\n",
    "        \"solution\": test_data[x][\"solution\"],\n",
    "    }\n",
    "    for x in range(len(test_data)) if test_data[x][\"level\"] == \"Level 2\" and test_data[x][\"type\"] == \"Algebra\"\n",
    "]\n",
    "print(len(tune_data), len(test_data))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Check a tuning example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:54.607152Z",
     "iopub.status.busy": "2023-02-13T23:40:54.606441Z",
     "iopub.status.idle": "2023-02-13T23:40:54.610504Z",
     "shell.execute_reply": "2023-02-13T23:40:54.609759Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is the slope of a line parallel to the line $2x - 4y = 9$? Express your answer as a common fraction.\n"
     ]
    }
   ],
   "source": [
    "print(tune_data[1][\"problem\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is one example of the canonical solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:54.613590Z",
     "iopub.status.busy": "2023-02-13T23:40:54.613168Z",
     "iopub.status.idle": "2023-02-13T23:40:54.616873Z",
     "shell.execute_reply": "2023-02-13T23:40:54.616193Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slope-intercept form of the equation of a line is $y = m x + b$ where $m$ is the slope.  So if we get $y$ on the opposite side from $x$ and make it have a coefficient of 1, the slope of the line will be the coefficient of $x$.  Thus we add $4y$ to both sides and divide everything by 4 which makes the coefficient of $x$ equal to $\\boxed{\\frac{1}{2}}$.\n"
     ]
    }
   ],
   "source": [
    "print(tune_data[1][\"solution\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Success Metric\n",
    "\n",
    "Before we start tuning, we need to define the success metric we want to optimize. For each math task, we use voting to select a response with the most common answers out of all the generated responses. If it has an equivalent answer to the canonical solution, we consider the task as successfully solved. Then we can optimize the mean success rate of a collection of tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:54.626998Z",
     "iopub.status.busy": "2023-02-13T23:40:54.626593Z",
     "iopub.status.idle": "2023-02-13T23:40:54.631383Z",
     "shell.execute_reply": "2023-02-13T23:40:54.630770Z"
    }
   },
   "outputs": [],
   "source": [
    "from flaml.autogen.math_utils import eval_math_responses"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Use the tuning data to find a good configuration\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For (local) reproducibility and cost efficiency, we cache responses from OpenAI with a controllable seed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:56.109177Z",
     "iopub.status.busy": "2023-02-13T23:40:56.108624Z",
     "iopub.status.idle": "2023-02-13T23:40:56.112651Z",
     "shell.execute_reply": "2023-02-13T23:40:56.112076Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "oai.ChatCompletion.set_cache(seed)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create a disk cache in \".cache/{seed}\". You can change `cache_path_root` from \".cache\" to a different path in `set_cache()`. The cache for different seeds are stored separately.\n",
    "\n",
    "### Perform tuning\n",
    "\n",
    "The tuning will take a while to finish, depending on the optimization budget. The tuning will be performed under the specified optimization budgets.\n",
    "\n",
    "* `inference_budget` is the target average inference budget per instance in the benchmark. For example, 0.004 means the target inference budget is 0.004 dollars, which translates to 2000 tokens (input + output combined) if the gpt-3.5-turbo model is used.\n",
    "* `optimization_budget` is the total budget allowed to perform the tuning. For example, 1 means 1 dollars are allowed in total, which translates to 500K tokens for the gpt-3.5-turbo model.\n",
    "* `num_sumples` is the number of different hyperparameter configurations which is allowed to try. The tuning will stop after either num_samples trials or after optimization_budget dollars spent, whichever happens first. -1 means no hard restriction in the number of trials and the actual number is decided by `optimization_budget`.\n",
    "\n",
    "Users can specify tuning data, optimization metric, optimization mode, evaluation function, search spaces etc.. The default search space is:\n",
    "\n",
    "```python\n",
    "default_search_space = {\n",
    "    \"model\": tune.choice([\n",
    "        \"gpt-3.5-turbo\",\n",
    "        \"gpt-4\",\n",
    "    ]),\n",
    "    \"temperature_or_top_p\": tune.choice(\n",
    "        [\n",
    "            {\"temperature\": tune.uniform(0, 2)},\n",
    "            {\"top_p\": tune.uniform(0, 1)},\n",
    "        ]\n",
    "    ),\n",
    "    \"max_tokens\": tune.lograndint(50, 1000),\n",
    "    \"n\": tune.randint(1, 100),\n",
    "    \"prompt\": \"{prompt}\",\n",
    "}\n",
    "```\n",
    "\n",
    "The default search space can be overridden by users' input.\n",
    "For example, the following code specifies a fixed prompt template. For hyperparameters which don't appear in users' input, the default search space will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:56.115383Z",
     "iopub.status.busy": "2023-02-13T23:40:56.114975Z",
     "iopub.status.idle": "2023-02-13T23:41:55.045654Z",
     "shell.execute_reply": "2023-02-13T23:41:55.044973Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2023-05-20 22:12:09,860]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[flaml.tune.tune: 05-20 22:12:09] {773} INFO - trial 1 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.36280922847807595}, 'max_tokens': 347, 'n': 10, 'prompt': 0}\n",
      "[flaml.tune.tune: 05-20 22:12:10] {197} INFO - result: {'expected_success': 0.8043947377449999, 'success': 0.85, 'success_vote': 0.7, 'voted_answer': 'We have $(-27)^{5/3} = \\\\left[(-27)^{1/3}\\\\right]^5$. Since $(-3)^3 = -27$, we have $(-27)^{1/3} = -3$. Therefore, $\\\\left[(-27)^{1/3}\\\\right]^5 = (-3)^5 = \\\\boxed{-243}$.', 'votes': 8.7, 'total_cost': 0.05771399999999999, 'cost': 0.05771399999999999, 'inference_cost': 0.0025029, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.36280922847807595}, 'max_tokens': 347, 'n': 10, 'prompt': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'top_p': 0.36280922847807595}, 'config/max_tokens': 347, 'config/n': 10, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.22889995574951172}\n",
      "[flaml.tune.tune: 05-20 22:12:10] {773} INFO - trial 2 config: {'temperature_or_top_p': {'temperature': 1.2672964698525508}, 'max_tokens': 470, 'n': 50, 'prompt': 0, 'model': 'gpt-3.5-turbo'}\n",
      "[flaml.tune.tune: 05-20 22:13:52] {197} INFO - result: {'expected_success': 0.9999161685004145, 'success': 1.0, 'success_vote': 0.9, 'voted_answer': 'We have \\\\begin{align*}\\n(-27)^{5/3} &= ((-27)^{1/3})^{5}\\\\\\\\\\n&=(\\\\sqrt[3]{-27})^5 &&\\\\mbox{(by definition of cube root)}\\\\\\\\\\n&= (-3)^5 && \\\\mbox{(because $(-a)^{2n+1}= -a^{2n+1}$ for all real numbers $a$ and integers $n \\\\geq 0$)} \\\\\\\\\\n&= -243.\\n\\\\end{align*}Thus, $(-27)^{5/3} = \\\\boxed{-243}$.', 'votes': 33.8, 'total_cost': 0.37417999999999996, 'cost': 0.316466, 'inference_cost': 0.0149563, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.2672964698525508}, 'max_tokens': 470, 'n': 50, 'prompt': 0, 'model': 'gpt-3.5-turbo'}, 'config/temperature_or_top_p': {'temperature': 1.2672964698525508}, 'config/max_tokens': 470, 'config/n': 50, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'experiment_tag': 'exp', 'time_total_s': 102.0125892162323}\n",
      "[flaml.tune.tune: 05-20 22:13:52] {773} INFO - trial 3 config: {'temperature_or_top_p': {'temperature': 1.5210614243979175}, 'max_tokens': 82, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo'}\n",
      "[flaml.tune.tune: 05-20 22:14:04] {197} INFO - result: {'expected_success': 0.49315329667554064, 'success': 0.55, 'success_vote': 0.55, 'voted_answer': 'Note that $(-27)^{1/3}=-3$, so \\\\[(-27)^{5/3}=(-27)^{1/3\\\\cdot 5}=\\\\left[(-27)^{1/3}\\\\right]^5=(-3)^5=-243.\\\\]Thus, $(-27)^{5/3} = \\\\boxed{-243}$.', 'votes': 2.05, 'total_cost': 0.403738, 'cost': 0.029557999999999994, 'inference_cost': 0.0014779, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.5210614243979175}, 'max_tokens': 82, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo'}, 'config/temperature_or_top_p': {'temperature': 1.5210614243979175}, 'config/max_tokens': 82, 'config/n': 9, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'experiment_tag': 'exp', 'time_total_s': 12.128517150878906}\n",
      "[flaml.tune.tune: 05-20 22:14:04] {773} INFO - trial 4 config: {'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'max_tokens': 231, 'n': 81, 'prompt': 0, 'model': 'gpt-3.5-turbo'}\n",
      "[flaml.tune.tune: 05-20 22:15:10] {197} INFO - result: {'expected_success': 0.75, 'success': 0.75, 'success_vote': 0.75, 'voted_answer': 'We have $(-27)^{5/3} = \\\\left[(-27)^{1/3}\\\\right]^5$. Since $(-3)^3 = -27$, we have $(-27)^{1/3} = -3$. Therefore, $\\\\left[(-27)^{1/3}\\\\right]^5 = (-3)^5 = \\\\boxed{-243}$.', 'votes': 76.15, 'total_cost': 0.7355139999999999, 'cost': 0.331776, 'inference_cost': 0.0161767, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'max_tokens': 231, 'n': 81, 'prompt': 0, 'model': 'gpt-3.5-turbo'}, 'config/temperature_or_top_p': {'top_p': 0.003948266327914451}, 'config/max_tokens': 231, 'config/n': 81, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'experiment_tag': 'exp', 'time_total_s': 66.01478743553162}\n",
      "[flaml.tune.tune: 05-20 22:15:10] {773} INFO - trial 5 config: {'temperature_or_top_p': {'top_p': 0.29187606817063316}, 'max_tokens': 781, 'n': 71, 'prompt': 0, 'model': 'gpt-3.5-turbo'}\n",
      "[flaml.tune.tune: 05-20 22:16:50] {197} INFO - result: {'success_vote': 0, 'total_cost': 1.008044, 'cost': 0.27253000000000005, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'top_p': 0.29187606817063316}, 'max_tokens': 781, 'n': 71, 'prompt': 0, 'model': 'gpt-3.5-turbo'}, 'config/temperature_or_top_p': {'top_p': 0.29187606817063316}, 'config/max_tokens': 781, 'config/n': 71, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'experiment_tag': 'exp', 'time_total_s': 100.23262238502502}\n",
      "[flaml.tune.tune: 05-20 22:16:50] {796} WARNING - fail to sample a trial for 100 times in a row, stopping.\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "\n",
    "prompts = [\"{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\\\boxed{{}}.\"]\n",
    "config, analysis = oai.ChatCompletion.tune(\n",
    "    data=tune_data,  # the data for tuning\n",
    "    metric=\"success_vote\",  # the metric to optimize\n",
    "    mode=\"max\",  # the optimization mode\n",
    "    eval_func=eval_math_responses,  # the evaluation function to return the success metrics\n",
    "    # log_file_name=\"logs/math.log\",  # the log file name\n",
    "    inference_budget=0.02,  # the inference budget (dollar per instance)\n",
    "    optimization_budget=1,  # the optimization budget (dollar in total)\n",
    "    # num_samples can further limit the number of trials for different hyperparameter configurations;\n",
    "    # -1 means decided by the optimization budget only\n",
    "    num_samples=20,\n",
    "    model=\"gpt-3.5-turbo\",  # comment to tune both gpt-3.5-turbo and gpt-4\n",
    "    prompt=prompts,  # the prompt templates to choose from\n",
    "    # stop=\"###\",  # the stop sequence\n",
    "    config_list=config_list,  # the endpoint list\n",
    "    # logging_level=logging.INFO,  # the logging level\n",
    ")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output tuning results\n",
    "\n",
    "After the tuning, we can print out the config and the result found by FLAML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:41:55.049204Z",
     "iopub.status.busy": "2023-02-13T23:41:55.048871Z",
     "iopub.status.idle": "2023-02-13T23:41:55.053284Z",
     "shell.execute_reply": "2023-02-13T23:41:55.052574Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "optimized config {'max_tokens': 470, 'n': 50, 'prompt': '{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\\\boxed{{}}.', 'model': 'gpt-3.5-turbo', 'temperature': 1.2672964698525508}\n",
      "best result on tuning data {'expected_success': 0.9999161685004145, 'success': 1.0, 'success_vote': 0.9, 'voted_answer': 'We have \\\\begin{align*}\\n(-27)^{5/3} &= ((-27)^{1/3})^{5}\\\\\\\\\\n&=(\\\\sqrt[3]{-27})^5 &&\\\\mbox{(by definition of cube root)}\\\\\\\\\\n&= (-3)^5 && \\\\mbox{(because $(-a)^{2n+1}= -a^{2n+1}$ for all real numbers $a$ and integers $n \\\\geq 0$)} \\\\\\\\\\n&= -243.\\n\\\\end{align*}Thus, $(-27)^{5/3} = \\\\boxed{-243}$.', 'votes': 33.8, 'total_cost': 0.37417999999999996, 'cost': 0.316466, 'inference_cost': 0.0149563, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.2672964698525508}, 'max_tokens': 470, 'n': 50, 'prompt': 0, 'model': 'gpt-3.5-turbo'}, 'config/temperature_or_top_p': {'temperature': 1.2672964698525508}, 'config/max_tokens': 470, 'config/n': 50, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'experiment_tag': 'exp', 'time_total_s': 102.0125892162323}\n"
     ]
    }
   ],
   "source": [
    "print(\"optimized config\", config)\n",
    "print(\"best result on tuning data\", analysis.best_result)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Make a request with the tuned config\n",
    "\n",
    "We can apply the tuned config on the request for an example task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:41:55.056205Z",
     "iopub.status.busy": "2023-02-13T23:41:55.055631Z",
     "iopub.status.idle": "2023-02-13T23:41:56.039259Z",
     "shell.execute_reply": "2023-02-13T23:41:56.038427Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "response on an example data instance: {\n",
      "  \"choices\": [\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 0,\n",
      "      \"message\": {\n",
      "        \"content\": \"Given $2x-4y=9$. Putting this in slope-intercept form yields $-2y = -2x + 9 \\\\Rightarrow y=x-\\\\frac{9}{2}$, so the slope is $\\\\boxed{1}$. Parallel lines have equal slopes.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 1,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of a line parallel to $2x-4y=9,$ we need to find the slope of the line $2x-4y=9.$ Solving for $y$ in this first equation, we get $$\\\\begin{aligned} 2x-4y&=9\\n\\\\\\\\\\\\Rightarrow\\\\qquad 4y&=2x-9\\n\\\\\\\\\\\\Rightarrow\\\\qquad y&=\\\\frac{1}{2}x-\\\\frac{9}{4}.\\\\end{aligned}$$So the slope of this line is $\\\\frac{1}{2}.$ Therefore, the slope of any line parallel to this line will also be $\\\\boxed{\\\\frac{1}{2}}.$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 2,\n",
      "      \"message\": {\n",
      "        \"content\": \"We will first convert the equation $2x-4y=9$ into slope-intercept form, $y=mx+b$. Solving this equation for $y$, we get $$-4y=-2x+9\\\\Rightarrow y=\\\\dfrac12 x -\\\\dfrac94.$$Therefore, the slope of this line is $\\\\dfrac 12$.\\n\\nSince two lines are parallel if and only if they have the same slope, any line that is parallel to $2x-4y=9$ must also have a slope of $\\\\boxed{\\\\dfrac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 3,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of a line parallel to $2x-4y=9$, we first need to write the equation in slope-intercept form, $y=mx+c$, where $m$ is the slope and $c$ is the $y$-intercept.\\n\\nFirst let's move the $-4y$ to the right side of the equation by adding it to the both sides:\\n\\\\begin{align*}\\n2x-4y&=9 \\\\\\\\\\n2x&=4y+9 \\\\\\\\\\ny&=\\\\frac12x-\\\\frac94\\n\\\\end{align*}\\n\\nNow we find the slope by looking at the coefficient of $x$: $m=\\\\frac12$.\\n\\nTherefore, any line that is parallel to $2x-4y=9$ has the same slope of $\\\\frac12$, and our final answer is $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 4,\n",
      "      \"message\": {\n",
      "        \"content\": \"To figure out the slope, we simply need to put the line $2x - 4y = 9$ into slope-intercept form, $y = mx+b$, solve for $m$, and recognize that a line that is parallel to it will have the same slope.  We can get $y$ by moving $2x - 9$ to the other side and dividing by $-4$, to get $y = \\\\frac{1}{2} x - \\\\frac{9}{4}$.  Therefore, the slope of the line is $\\\\boxed{\\\\frac{1}{2}}$.\\n\\nAn alternative method is to recognize that parallel lines have the same slope.  We first put the given line in the slope-intercept form as above, so $y = \\\\frac{1}{2} x - \\\\frac{9}{4}$.  We now know the slope of a parallel line to this line is $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 5,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can determine the slope of the line $2x-4y=9$ by rearranging it to slope-intercept form, $y=\\\\frac{1}{2}x-\\\\frac{9}{4}$, which indicates that the slope is $\\\\frac{1}{2}$.\\nIf a line is parallel to this line, it will have the same slope of $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 6,\n",
      "      \"message\": {\n",
      "        \"content\": \"In order for two lines to be parallel, they must have the same slope. To find the slope of a line, we can put it into slope-intercept form ($y=mx+b$), where $m$ is the slope.\\n\\nStarting with $2x-4y=9$, we can isolate $y$ to get it into slope-intercept form: $$2x-4y=9$$ $$-4y=-2x+9$$ $$y=\\\\frac12x-\\\\frac94$$\\n\\nTherefore, the slope of the given line is $\\\\frac12$.\\n\\nSince we are looking for a line that is parallel, it will have the same slope. Thus, the slope of the desired line is also $\\\\frac12$.\\n\\nTherefore, the final answer is \\\\boxed{\\\\frac{1}{2}}.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 7,\n",
      "      \"message\": {\n",
      "        \"content\": \"A line parallel to $2x-4y=9$ will have the same slope. To find the slope, we can solve  $2x-4y=9$ for $y$ in terms of $x$. First, we add $2y$ to both sides, to get $2x-9=4y$. Then we divide by $4$ to get $y=\\\\frac{1}{4}x-\\\\frac{9}{4}$. Taking this in slope-intercept form we get that the slope of $2x - 4y = 9$ is $\\\\frac14$, so our final answer is $\\\\boxed{\\\\frac14}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 8,\n",
      "      \"message\": {\n",
      "        \"content\": \"Recall that parallel lines have equal slopes.\\n\\nFirst, put the equation $2x - 4y = 9$ in slope-intercept form $y = mx + b$, where $m$ is the slope.\\n\\n\\\\[2x - 4y = 9 \\\\qquad \\\\Rightarrow \\\\qquad  -4y = -2x + 9 \\\\qquad \\\\Rightarrow \\\\qquad y = \\\\frac{1}{2}x - \\\\frac{9}{4}\\\\]\\n\\nTherefore, the slope of the line $2x - 4y = 9$ is $\\\\frac{1}{2}$.\\n\\nSince the slope of any line parallel to this line also has to be $\\\\frac{1}{2}$, our answer is $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 9,\n",
      "      \"message\": {\n",
      "        \"content\": \"Recall that two lines are parallel if and only if they have the same slope. We need to find the slope of the line $2x-4y=9$ to use this fact. Solving this equation for $y$ yields $y=\\\\frac 12 x - \\\\frac 94$, so the slope of this line is $\\\\frac12$. Therefore, any line parallel to $2x-4y=9$ also has slope $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 10,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of the line, we must rewrite the equation in slope-intercept form, $y = mx + b$, where the coefficient of $x$, $m$, is the slope of the line.  Solving for $y$ in $2x - 4y = 9$, we have  \\\\[\\n-4y = -2x + 9\\\\quad \\\\Rightarrow\\\\quad y = \\\\frac12 x - \\\\frac94.\\n\\\\]So, a line parallel to this line will have a slope of $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 11,\n",
      "      \"message\": {\n",
      "        \"content\": \"To determine the slope of a line parallel to $2x-4y=9$, we need to find the slope of this line first. We can then use the fact that parallel lines have the same slope.\\n\\nRewriting this equation of the line in slope-intercept form, we get:\\n\\n\\\\begin{align*}\\n2x-4y&=9\\\\\\\\\\n-4y&=-2x+9\\\\\\\\\\ny&=\\\\frac{1}{2}x-\\\\frac{9}{4}\\n\\\\end{align*}\\n\\nSo the slope of this line is $\\\\frac{1}{2}$. Therefore, any line parallel to this line will also have a slope of $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 12,\n",
      "      \"message\": {\n",
      "        \"content\": \"To solve the problem at hand, we take note that parallel lines must have the same slope. So, to find the slope of the line in question, we just need to rearrange $2x - 4y = 9$ to be in the slope-intercept form. Adding $4y$ and then dividing by $2$, we get $y = \\\\frac12 x - \\\\frac94$. Therefore, the slope of the line is $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 13,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of $2x - 4y = 9$, we can put it into slope-intercept form $y = mx + b$ by solving for $y$, giving $-4y = -2x + 9 \\\\Rightarrow y = \\\\frac{1}{2}x - \\\\frac{9}{4}$. Therefore, the slope of the given line is $\\\\frac12$. \\n\\nParallel lines have the same slope, so any line parallel to this one has a slope of $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 14,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can begin by solving the equation $2x-4y=9$ for $y$ in terms of $x$ to find its slope: $$2x-4y=9\\\\Rightarrow -4y=-2x+9\\\\Rightarrow y=\\\\frac12x-\\\\frac94.$$We recognize $\\\\frac12$ as the slope of this line -- and therefore the slope of any line parallel to it. Thus, our final answer is $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 15,\n",
      "      \"message\": {\n",
      "        \"content\": \"Note that if two lines in the coordinate plane are parallel, then they have the same slope (indeed, take any two distinct points on a parallel line to get a unique slope, and note that this will hold for all pairs of distinct points on the line due to parallelism).\\n\\nWe now try to put the given line into slope-intercept form ($y = mx + b$). We can do this by solving for $y$:, should get $$2x - 4y = 9\\\\qquad\\\\Longrightarrow\\\\qquad y = \\\\frac{1}{2}x - \\\\frac{9}{4}.$$ Therefore, a line parallel to $2x - 4y = 9$ must have a slope of $\\\\boxed{\\\\frac{1}{2}}$, the same slope as $y = \\\\frac{1}{2}x - \\\\frac{9}{4}.$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 16,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of the line $2x - 4y = 9,$ we can rewrite it in slope-intercept form: \\\\begin{align*}\\n2x - 4y &= 9 \\\\\\\\\\n-4y &= -2x + 9 \\\\\\\\\\ny &= \\\\frac{1}{2} x - \\\\frac{9}{4}.\\n\\\\end{align*}Therefore, the slope of this line is $\\\\boxed{\\\\frac{1}{2}}.$\\n\\nRecall that if two lines are parallel, then they have the same slope.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 17,\n",
      "      \"message\": {\n",
      "        \"content\": \"A line parallel to any line must have the same slope of that line. Therefore, let's re-write $2x-4y = 9$ in slope-intercept form and then use the slope.$$\\n\\\\begin{array}{r r c r@{\\\\;\\\\;=\\\\;\\\\;}l}\\n2x-4y &=& 9 & 4y & 2x-9 \\\\\\\\\\n-4y & = & -2x+9 & y & \\\\displaystyle{\\\\frac{-2}{4}}x + \\\\displaystyle{\\\\frac{9}{4}}\\\\\\\\\\ny & = & \\\\displaystyle{\\\\frac{-1}{2}} x + \\\\displaystyle{\\\\frac{9}{4}}&\\n\\\\end{array}\\n$$We see that the slope is $\\\\boxed{\\\\frac{-1}{2}}$ as all other lines that our parallel to $2x-4y=9$ are of the type, in slope-intercept form,  $y = -\\\\frac{1}{2} x + b$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 18,\n",
      "      \"message\": {\n",
      "        \"content\": \"Recall that two lines are parallel if and only if they have the same slope. Therefore, we want to find the slope of the line $2x - 4y = 9$.\\n\\nOne way to do this is to put the equation in slope-intercept form: $$ 2x - 4y = 9 \\\\Rightarrow  y = \\\\frac{1}{2} x - \\\\frac{9}{4}.$$We can see that the slope of this line is $\\\\boxed{\\\\frac{1}{2}}$.\\n\\nAnother way to do this is to rewrite the equation in terms of $y$ as an explicit function of $x$, then take the derivative with respect to $x$ and evaluate the expression at any point. Doing this method is more advanced, but yields another way to check our work: $$ 2x - 4y = 9 \\\\Rightarrow y = \\\\frac{1}{2}x - \\\\frac{9}{4}.$$Differentiating with respect to $x$, we get $\\\\frac{dy}{dx} = \\\\frac{1}{2}$. Again, this confirms that the slope of the original line is $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 19,\n",
      "      \"message\": {\n",
      "        \"content\": \"The slope-intercept form of a line is $y=mx+b,$ where $m$ is the slope and $b$ is the $y$-intercept. To find the slope of a line from its equation in standard form $ax+by=c,$ solve for $y$ to get $y=-\\\\frac{a}{b}x+\\\\frac{c}{b}$ and we spot the slope of $-\\\\frac{a}{b}.$ We can find the slope-intercept form from $2x-4y=9$ as follows\\\\begin{align*}\\n2x-4y&=9 \\\\\\\\\\n-4y&=-2x+9 \\\\\\\\\\ny&=\\\\frac{1}{2}x-\\\\frac{9}{4}.\\n\\\\end{align*}The slope of this line is $\\\\frac{1}{2}.$\\n\\nEvery line parallel to this line has the same slope.  Thus, the slope of a line parallel to $2x - 4y = 9$ is $\\\\boxed{\\\\frac{1}{2}}.$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 20,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of the line defined by $2x-4y=9,$ we need to solve for $y$ so that the equation is in slope-intercept form, $y=mx+b,$ where $m$ is the slope. Solving for $y$ results in \\\\begin{align*}\\n2x-4y&=9\\\\\\\\\\n\\\\Rightarrow\\\\qquad y&=\\\\frac{1}{2}x-\\\\frac{9}{4}.\\n\\\\end{align*}Therefore, the slope of the line is $\\\\displaystyle\\\\frac{1}{2}.$\\n\\nTwo non-vertical lines are parallel if and only if they have the same slope. Thus, a line parallel to $2x-4y=9$ has a slope of $\\\\boxed{\\\\frac{1}{2}}.$  (Note that, if we solve for $x$ instead of $y$, we will obtain a different value.)\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 21,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of a line, we need to put the equation in slope-intercept form $y=mx+b$ where $m$ is the slope. To do this, we can start by solving the equation we're given for $y$:\\n\\n\\\\begin{align*}\\n2x - 4y &= 9\\\\\\\\\\n-4y &= -2x + 9\\\\\\\\\\ny &= \\\\frac{1}{2}x - \\\\frac{9}{4}.\\n\\\\end{align*}\\n\\nSo the slope of the line given is $\\\\frac{1}{2}$. \\n\\nFor a line to be parallel, it must have the same slope, so our answer is $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 22,\n",
      "      \"message\": {\n",
      "        \"content\": \"To obtain the slope of the given line $2x-4y=9$, we must rewrite the equation in slope-intercept form, i.e. in the form $y = mx + b$, where $m$ is the slope and $b$ is the $y$-intercept.\\n\\nSolving the given equation for $y$ yields: \\n\\n$$2x - 4y = 9 \\\\Rightarrow -4y = -2x + 9 \\\\Rightarrow y = \\\\frac{1}{2}x - \\\\frac{9}{4}$$ \\n\\nTherefore, the slope of the given line is $\\\\frac12$.\\n\\nParallel lines have the same slope. Thus, any line parallel to the given line also has a slope of $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 23,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can start this problem by putting the equation $2x - 4y = 9$ into slope-intercept form. Subtracting $2x$ from both sides gives $-4y = -2x + 9$, so dividing by $-4$ gives $y = \\\\frac{1}{2}x - \\\\frac{9}{4}$. Therefore, the slope of the line is $\\\\frac{1}{2}$.\\n\\nAny line that is parallel to this line will have the same slope of $\\\\frac{1}{2}$, since parallel lines have equal slopes. Therefore, the answer is $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 24,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can find the slope of the line by putting $2x - 4y = 9$ into slope-intercept form, which means we need to solve for $y$:\\\\begin{align*}\\n2x - 4y &= 9\\\\\\\\\\n\\\\Rightarrow\\\\qquad 4y&=2x-9\\\\\\\\\\n\\\\Rightarrow\\\\qquad y&=\\\\frac 12x-\\\\frac 94.\\n\\\\end{align*}This line has slope $\\\\frac 12$.  Any line perpendicular to this line therefore has slope the opposite reciprocal, or $-2$.  $\\\\boxed{-2}$ is the slope of any line parallel to the line $2x-4y=9$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 25,\n",
      "      \"message\": {\n",
      "        \"content\": \"To begin, we can rearrange the given equation $2x - 4y = 9$ by subtracting $2x$ from both sides to obtain $-4y = -2x + 9$. Then, we can divide both sides by $-4$ to solve for $y$ and obtain $y = \\\\frac{1}{2}x - \\\\frac{9}{4}$.\\n\\nWe see that any line parallel to this line must have the same slope, since parallel lines have the same slope. Thus, the desired slope is $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 26,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of a line parallel to $2x-4y=9,$ we need to find the slope of the given line first.  To do this, we can transform it into slope-intercept form by solving for $y.$\\n\\n\\n\\\\begin{align*}\\n2x-4y &= 9\\\\\\\\\\n-4y &= -2x + 9\\\\\\\\\\ny &= \\\\frac12 x - \\\\frac94.\\n\\\\end{align*}Therefore, the slope of the given line is $\\\\frac12.$  \\n\\nAny line parallel to this line will have the same slope, so our final answer is $\\\\boxed{\\\\frac{1}{2}}.$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 27,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of the given line, we can first rewrite it in slope-intercept form: $$2x-4y=9 \\\\Rightarrow -4y=-2x+9 \\\\Rightarrow y=\\\\frac{1}{2}x-\\\\frac{9}{4}.$$Therefore, the slope of the given line is $\\\\frac{1}{2}.$\\n\\nLines that are parallel have the same slope. Therefore, any line that is parallel to this line will have a slope of $\\\\boxed{\\\\frac{1}{2}}.$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 28,\n",
      "      \"message\": {\n",
      "        \"content\": \"To start, we solve for $y$:\\n\\n\\\\[\\\\begin{aligned} 2x - 4y &= 9 \\\\\\\\-4y &= -2x + 9 \\\\\\\\ y &= \\\\frac{1}{2}x -\\\\frac{9}{4}\\\\end{aligned}\\\\]\\n\\nThe slope of this line is $\\\\frac{1}{2}$. For lines to be parallel, they must have the same slope, so any line that is parallel to the given line also has a slope of $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 29,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of the line $2x - 4y = 9$, we need to rewrite the equation in slope-intercept form, $y = mx + b,$ which means solving for $y.$\\n\\nWe start by subtracting $2x$ from both sides of the equation to get $-4y = -2x + 9.$ Dividing both sides by $-4,$ we have\\n\\\\begin{align*}\\ny &= \\\\frac{-2x+9}{-4}\\\\\\\\\\n& = \\\\frac{1}{2} \\\\cdot \\\\frac{-2x+9}{-2}\\\\\\\\\\n& = \\\\frac{1}{2}x - \\\\frac{9}{8}.\\n\\\\end{align*}Thus, the slope of the original line is $m=-\\\\frac{1}{2}$.\\n\\nA line parallel to this line must have the same slope, $\\\\boxed{-\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 30,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of a line parallel to a given line, we must first find the slope of the given line. To do that, we solve the equation of the given line for $y$ in terms of $x$:\\n\\n\\\\begin{align*}\\n2x - 4y &= 9 \\\\\\\\\\n-4y &= -2x + 9 \\\\\\\\\\ny &= \\\\frac12 x - \\\\frac94 \\\\\\\\\\n\\\\end{align*}\\n\\nSo the slope of the given line is $\\\\frac12$, because the coefficient of $x$ in the equation $y = \\\\frac12 x - \\\\frac94$ is $\\\\frac12$.\\n\\nA line parallel to the given line must have the same slope. Therefore the slope of a line parallel to the given line is also $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 31,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can find the slope of the given line by solving for $y$: $$2x-4y=9 \\\\Rightarrow -4y=-2x+9 \\\\Rightarrow y=\\\\frac{1}{2}x-\\\\frac{9}{4}.$$Hence, the slope of the given line is $\\\\frac{1}{2}$.\\n\\nFor any line parallel to this line, the slope will be the same, namely $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 32,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can rearrange the given equation to get the slope-intercept form $y = \\\\frac{1}{2}x - \\\\frac{9}{4}$. We can see that any line parallel to this line will have the same slope of $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 33,\n",
      "      \"message\": {\n",
      "        \"content\": \"Converting the equation $2x - 4y = 9$ to slope-intercept form, we get $y = \\\\frac{1}{2}x - \\\\frac{9}{4}$.  Therefore, lines parallel to this one will also have slope $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 34,\n",
      "      \"message\": {\n",
      "        \"content\": \"Putting the equation in slope-intercept form, we have $-4y = -2x + 9 \\\\Rightarrow y = \\\\frac{1}{2}x - \\\\frac{9}{4}$.  Therefore, the slope of the original line is $\\\\frac{1}{2}$.  Every line that is parallel to this one will have the same slope of $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 35,\n",
      "      \"message\": {\n",
      "        \"content\": \"In order for two lines to be parallel, they must have the same slope. To find the slope of $2x-4y=9$, we can solve for $y$ in terms of $x$ and write the equation in slope-intercept form, $y=mx+b$, where $m$ is the slope.\\n\\nFirst, we subtract $2x$ from both sides and get $-4y=-2x+9$. Then, we divide both sides by $-4$ to solve for $y$: $$-4y=-2x+9 \\\\qquad \\\\Rightarrow \\\\qquad y=\\\\frac{1}{2}x-\\\\frac{9}{4}.$$ The slope of this line is $\\\\frac{1}{2}$.\\n\\nTherefore, a line parallel to $2x-4y=9$ will also have a slope of $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 36,\n",
      "      \"message\": {\n",
      "        \"content\": \"A line parallel to $2x- 4y =9$ will have the same slope, since parallel lines have equal slopes. To find the slope, we write the equation in slope-intercept form, $ y = m x +b$, solving for  $y$:  $y=\\\\frac12x-\\\\frac{9}{4}$, where $m=\\\\frac12$ is the slope (coefficient of $x$). Thus a parallel line will also have a slope of $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 37,\n",
      "      \"message\": {\n",
      "        \"content\": \"To determine the slope of the line given in point-slope form, we need to solve for $y$:\\n\\n$$2x - 4y = 9 \\\\Rightarrow -4y = -2x +9 \\\\Rightarrow y = \\\\frac{1}{2}x - \\\\frac{9}{4}$$\\n\\nTherefore, the line has a slope of $\\\\boxed{\\\\frac{1}{2}}$.\\n\\nTo find the slope of a line parallel to this one, note that parallel lines have the same slope. Therefore, any line parallel to the given line will also have a slope of $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 38,\n",
      "      \"message\": {\n",
      "        \"content\": \"Recall that two lines are parallel if and only if they have the same slope. To find the slope of the given line, we put it in slope-intercept form by solving for $y$: $$2x - 4y = 9.$$Isolating $y$, we get $$-4y = -2x + 9,$$which implies $$y = \\\\frac{1}{2}x - \\\\frac{9}{4}.$$Thus, the slope of the given line is $\\\\frac{1}{2}$. Any line parallel to the given line will have the same slope $\\\\frac{1}{2}$, so the answer is \\\\boxed{\\\\frac{1}{2}}.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 39,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by rearranging the given equation to the slope-intercept form: $$2x - 4y = 9 \\\\Rightarrow 2x - 9 = 4y \\\\Rightarrow y = \\\\frac12 x - \\\\frac94.$$The slope of the line is therefore $\\\\boxed{\\\\frac 12}$. \\n\\nRecall that all lines that are parallel have the same slope.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 40,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of a line parallel to a given line, we first need to find the slope of the given line. In order to do that, we will solve the given equation for $y$ in terms of $x$ (in other words, we will put it into slope-intercept form, $y=mx+b$).\\n\\n\\\\begin{align*}\\n2x -4y &= 9\\\\\\\\\\n-4y &= -2x + 9\\\\\\\\\\ny &= \\\\frac12 x - \\\\frac{9}{4}.\\n\\\\end{align*}\\n\\nWe now see that the slope of the given line is $\\\\frac12$. Since lines that are parallel have the same slope, any line parallel to this one will also have a slope of $\\\\frac12$. Therefore, our final answer is $\\\\boxed{\\\\frac12}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 41,\n",
      "      \"message\": {\n",
      "        \"content\": \"We convert $2x - 4y = 9$ to slope-intercept form.  Solving for $y$, we find that $y=\\\\frac{1}{2}x-\\\\frac{9}{4}$.  Hence, if a line is parallel to this line, it will have the same slope and thus an answer of $\\\\boxed{\\\\frac{1}{2}}$.\\n\\nIn general, two lines in the plane are parallel if and only if they have the same slope.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 42,\n",
      "      \"message\": {\n",
      "        \"content\": \"For a line to be parallel to another line, it must have the same slope. Thus, we need to find the slope of the line $2x - 4y = 9$.\\n\\nFirst, we can rearrange this equation to isolate $y$ by subtracting $2x$ from both sides and then dividing by $-4$:\\n\\n$$-4y=-2x+9$$\\n\\n$$y=\\\\frac{1}{2}x-\\\\frac{9}{4}$$\\n\\nNow we can see that the slope of this line is $\\\\frac{1}{2}$. Therefore, any line parallel to this one must also have a slope of $\\\\frac{1}{2}$. \\n\\nOur final answer is $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 43,\n",
      "      \"message\": {\n",
      "        \"content\": \"Solving for $y$ in $2x - 4y = 9$ gives $y = \\\\frac{1}{2}x - \\\\frac{9}{4}$. Thus, a line parallel to this one has slope $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 44,\n",
      "      \"message\": {\n",
      "        \"content\": \"We know that lines that are parallel have the same slope. To find the slope of $2x-4y=9$, we can rewrite the equation in slope-intercept form, $y=mx+b$, where $m$ is the slope. Solving for $y$, we get $$-4y=-2x+9\\\\qquad\\\\Rightarrow\\\\qquad y=\\\\frac{1}{2}x-\\\\frac{9}{4}.$$ Therefore, a line parallel to this line has slope $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 45,\n",
      "      \"message\": {\n",
      "        \"content\": \"First we want to get the given equation into slope-intercept form, $y=mx+b$, where $m$ is the slope. Solve this equation for $y$ explicitly to obtain $$ 2x-4y = 9 \\\\implies -4y = -2x+9 \\\\implies y = \\\\frac{1}{2}x - \\\\frac{9}{4}. $$\\n\\nComparing this equation to $y=mx+b$, we see that $m=\\\\boxed{\\\\frac{1}{2}}$ is the slope of this line. But the problem asks for the slope of a line \\\\emph{parallel} to this one. Since parallel lines have the same slope, we simply write $\\\\boxed{\\\\frac{1}{2}}$ as our answer.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 46,\n",
      "      \"message\": {\n",
      "        \"content\": \"To say that two lines in a plane are parallel is to say that they never intersect. In the $xy$-plane, two lines that never intersect are either (1) horizontal lines (they have the same $y$-intercept, i.e. they go through the $y$-axis at the same height) or (2) vertical lines (they have the same $x$-intercept, and are, it is usually assumed, substantially less exciting!)\\n\\nPutting the equation $2x-4y=9$ in slope-intercept form $y=mx+b$ gives $y=\\\\frac{1}{2}x-\\\\frac{9}{4}$.  Therefore, a parallel line would have a slope of $\\\\boxed{\\\\frac{1}{2}}$, and would have an equation of the type $y = \\\\frac{1}{2}x + b$ for some choice of $b$ (the $y$-intercept of the new line). Note that slope-intercept form identifies $m$ as the slope and $b$ as the vertical intercept of a line; try writing a few different lines in slope-intercept form and see if you agree that this form makes those geometrical concepts visually clear.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 47,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of a line parallel to $2x-4y=9$, we need to find the slope of the given line. Solving for $y$ gives $y=\\\\frac12 x - \\\\frac94$. Thus the slope of the given line is $\\\\frac12$.\\n\\nTwo lines are parallel if and only if they have the same slope, so the slope of any line parallel to $2x-4y=9$ is also $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 48,\n",
      "      \"message\": {\n",
      "        \"content\": \"To find the slope of this line, we need to put it into slope-intercept form: $$2x - 4y = 9$$ $$-4y = -2x +9$$ $$y = \\\\frac{1}{2}x - \\\\frac{9}{4}.$$ So the slope of the given line is $\\\\frac{1}{2}$. Since any line parallel to this line must have the same slope, our answer is $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 49,\n",
      "      \"message\": {\n",
      "        \"content\": \"We need to find the slope of a line that is parallel to the line $2x - 4y = 9$. First, we write this equation in slope-intercept form, $y = mx + b$. Solving for $y$ yields $y=\\\\frac{1}{2}x - \\\\frac{9}{4}$. Thus, the slope of the original line is $\\\\frac{1}{2}$.\\n\\nSince lines that are parallel have the same slope, the line we are looking for also has a slope of $\\\\frac{1}{2}$. Therefore, our answer is $\\\\boxed{\\\\frac{1}{2}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    }\n",
      "  ],\n",
      "  \"config_id\": 0,\n",
      "  \"cost\": 0.013798,\n",
      "  \"created\": 1684615715,\n",
      "  \"id\": \"chatcmpl-7INn9wZkp1kYxtJ9UVa1n0FC1NopJ\",\n",
      "  \"model\": \"gpt-3.5-turbo-0301\",\n",
      "  \"object\": \"chat.completion\",\n",
      "  \"usage\": {\n",
      "    \"completion_tokens\": 6838,\n",
      "    \"prompt_tokens\": 61,\n",
      "    \"total_tokens\": 6899\n",
      "  }\n",
      "}\n",
      "metric_results on the example data instance: {'expected_success': 1.0, 'success': True, 'success_vote': 1.0, 'voted_answer': 'To find the slope of a line parallel to $2x-4y=9,$ we need to find the slope of the line $2x-4y=9.$ Solving for $y$ in this first equation, we get $$\\\\begin{aligned} 2x-4y&=9\\n\\\\\\\\\\\\Rightarrow\\\\qquad 4y&=2x-9\\n\\\\\\\\\\\\Rightarrow\\\\qquad y&=\\\\frac{1}{2}x-\\\\frac{9}{4}.\\\\end{aligned}$$So the slope of this line is $\\\\frac{1}{2}.$ Therefore, the slope of any line parallel to this line will also be $\\\\boxed{\\\\frac{1}{2}}.$', 'votes': 45}\n"
     ]
    }
   ],
   "source": [
    "response = oai.ChatCompletion.create(context=tune_data[1], config_list=config_list, **config)\n",
    "metric_results = eval_math_responses(oai.ChatCompletion.extract_text(response), **tune_data[1])\n",
    "print(\"response on an example data instance:\", response)\n",
    "print(\"metric_results on the example data instance:\", metric_results)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the success rate on the test data\n",
    "\n",
    "You can use flaml's `oai.ChatCompletion.test` to evaluate the performance of an entire dataset with the tuned config. The following code will take a while (30 mins to 1 hour) to evaluate all the test data instances if uncommented and run. It will cost roughly $3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:41:56.042764Z",
     "iopub.status.busy": "2023-02-13T23:41:56.042086Z",
     "iopub.status.idle": "2023-02-13T23:53:05.597643Z",
     "shell.execute_reply": "2023-02-13T23:53:05.596603Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[flaml.autogen.oai.completion: 05-20 22:21:33] {927} INFO - evaluating data instance 0\n",
      "[flaml.autogen.oai.completion: 05-20 22:22:43] {927} INFO - evaluating data instance 1\n",
      "[flaml.autogen.oai.completion: 05-20 22:23:03] {927} INFO - evaluating data instance 2\n",
      "[flaml.autogen.oai.completion: 05-20 22:23:20] {927} INFO - evaluating data instance 3\n",
      "[flaml.autogen.oai.completion: 05-20 22:23:47] {927} INFO - evaluating data instance 4\n",
      "[flaml.autogen.oai.completion: 05-20 22:23:47] {927} INFO - evaluating data instance 5\n",
      "[flaml.autogen.oai.completion: 05-20 22:23:48] {927} INFO - evaluating data instance 6\n",
      "[flaml.autogen.oai.completion: 05-20 22:23:48] {927} INFO - evaluating data instance 7\n",
      "[flaml.autogen.oai.completion: 05-20 22:23:48] {927} INFO - evaluating data instance 8\n",
      "[flaml.autogen.oai.completion: 05-20 22:24:20] {927} INFO - evaluating data instance 9\n",
      "[flaml.autogen.oai.completion: 05-20 22:24:20] {927} INFO - evaluating data instance 10\n",
      "[flaml.autogen.oai.completion: 05-20 22:24:20] {927} INFO - evaluating data instance 11\n",
      "[flaml.autogen.oai.completion: 05-20 22:25:27] {927} INFO - evaluating data instance 12\n",
      "[flaml.autogen.oai.completion: 05-20 22:25:58] {927} INFO - evaluating data instance 13\n",
      "[flaml.autogen.oai.completion: 05-20 22:25:58] {927} INFO - evaluating data instance 14\n",
      "[flaml.autogen.oai.completion: 05-20 22:25:58] {927} INFO - evaluating data instance 15\n",
      "[flaml.autogen.oai.completion: 05-20 22:25:58] {927} INFO - evaluating data instance 16\n",
      "[flaml.autogen.oai.completion: 05-20 22:25:58] {927} INFO - evaluating data instance 17\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:21] {927} INFO - evaluating data instance 18\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:21] {927} INFO - evaluating data instance 19\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:21] {927} INFO - evaluating data instance 20\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:53] {927} INFO - evaluating data instance 21\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:53] {927} INFO - evaluating data instance 22\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:53] {927} INFO - evaluating data instance 23\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:53] {927} INFO - evaluating data instance 24\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:53] {927} INFO - evaluating data instance 25\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:53] {927} INFO - evaluating data instance 26\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:53] {927} INFO - evaluating data instance 27\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:53] {927} INFO - evaluating data instance 28\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:54] {927} INFO - evaluating data instance 29\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:54] {927} INFO - evaluating data instance 30\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:54] {927} INFO - evaluating data instance 31\n",
      "[flaml.autogen.oai.completion: 05-20 22:26:54] {927} INFO - evaluating data instance 32\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 33\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 34\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 35\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 36\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 37\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 38\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 39\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 40\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:07] {927} INFO - evaluating data instance 41\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:22] {927} INFO - evaluating data instance 42\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:34] {927} INFO - evaluating data instance 43\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:34] {927} INFO - evaluating data instance 44\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:34] {927} INFO - evaluating data instance 45\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:34] {927} INFO - evaluating data instance 46\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:34] {927} INFO - evaluating data instance 47\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:53] {927} INFO - evaluating data instance 48\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:53] {927} INFO - evaluating data instance 49\n",
      "[flaml.autogen.oai.completion: 05-20 22:28:53] {927} INFO - evaluating data instance 50\n",
      "[flaml.autogen.oai.completion: 05-20 22:29:25] {927} INFO - evaluating data instance 51\n",
      "[flaml.autogen.oai.completion: 05-20 22:29:25] {927} INFO - evaluating data instance 52\n",
      "[flaml.autogen.oai.completion: 05-20 22:29:46] {927} INFO - evaluating data instance 53\n",
      "[flaml.autogen.oai.completion: 05-20 22:29:46] {927} INFO - evaluating data instance 54\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:17] {927} INFO - evaluating data instance 55\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:17] {927} INFO - evaluating data instance 56\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:17] {927} INFO - evaluating data instance 57\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:17] {927} INFO - evaluating data instance 58\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:17] {927} INFO - evaluating data instance 59\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:17] {927} INFO - evaluating data instance 60\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:17] {927} INFO - evaluating data instance 61\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:41] {927} INFO - evaluating data instance 62\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:41] {927} INFO - evaluating data instance 63\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:41] {927} INFO - evaluating data instance 64\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:41] {927} INFO - evaluating data instance 65\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:41] {927} INFO - evaluating data instance 66\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:41] {927} INFO - evaluating data instance 67\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:57] {927} INFO - evaluating data instance 68\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:57] {927} INFO - evaluating data instance 69\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:57] {927} INFO - evaluating data instance 70\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:57] {927} INFO - evaluating data instance 71\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:57] {927} INFO - evaluating data instance 72\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:57] {927} INFO - evaluating data instance 73\n",
      "[flaml.autogen.oai.completion: 05-20 22:30:57] {927} INFO - evaluating data instance 74\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:03] {927} INFO - evaluating data instance 75\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:03] {927} INFO - evaluating data instance 76\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:20] {927} INFO - evaluating data instance 77\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:20] {927} INFO - evaluating data instance 78\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:20] {927} INFO - evaluating data instance 79\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:20] {927} INFO - evaluating data instance 80\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:20] {927} INFO - evaluating data instance 81\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:20] {927} INFO - evaluating data instance 82\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:20] {927} INFO - evaluating data instance 83\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:38] {927} INFO - evaluating data instance 84\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:38] {927} INFO - evaluating data instance 85\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:38] {927} INFO - evaluating data instance 86\n",
      "[flaml.autogen.oai.completion: 05-20 22:32:38] {927} INFO - evaluating data instance 87\n",
      "[flaml.autogen.oai.completion: 05-20 22:33:12] {927} INFO - evaluating data instance 88\n",
      "[flaml.autogen.oai.completion: 05-20 22:33:35] {927} INFO - evaluating data instance 89\n",
      "[flaml.autogen.oai.completion: 05-20 22:33:57] {927} INFO - evaluating data instance 90\n",
      "[flaml.autogen.oai.completion: 05-20 22:33:57] {927} INFO - evaluating data instance 91\n",
      "[flaml.autogen.oai.completion: 05-20 22:33:57] {927} INFO - evaluating data instance 92\n",
      "[flaml.autogen.oai.completion: 05-20 22:34:28] {927} INFO - evaluating data instance 93\n",
      "[flaml.autogen.oai.completion: 05-20 22:34:28] {927} INFO - evaluating data instance 94\n",
      "[flaml.autogen.oai.completion: 05-20 22:34:28] {927} INFO - evaluating data instance 95\n",
      "[flaml.autogen.oai.completion: 05-20 22:34:28] {927} INFO - evaluating data instance 96\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 97\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 98\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 99\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 100\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 101\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 102\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 103\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 104\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 105\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:01] {927} INFO - evaluating data instance 106\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:17] {927} INFO - evaluating data instance 107\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:17] {927} INFO - evaluating data instance 108\n",
      "[flaml.autogen.oai.completion: 05-20 22:35:43] {927} INFO - evaluating data instance 109\n",
      "[flaml.autogen.oai.completion: 05-20 22:36:16] {927} INFO - evaluating data instance 110\n",
      "[flaml.autogen.oai.completion: 05-20 22:36:17] {927} INFO - evaluating data instance 111\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:02] {927} INFO - evaluating data instance 112\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:03] {927} INFO - evaluating data instance 113\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:03] {927} INFO - evaluating data instance 114\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:03] {927} INFO - evaluating data instance 115\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:03] {927} INFO - evaluating data instance 116\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:03] {927} INFO - evaluating data instance 117\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:03] {927} INFO - evaluating data instance 118\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:29] {927} INFO - evaluating data instance 119\n",
      "[flaml.autogen.oai.completion: 05-20 22:37:51] {927} INFO - evaluating data instance 120\n",
      "[flaml.autogen.oai.completion: 05-20 22:40:09] {927} INFO - evaluating data instance 121\n",
      "[flaml.autogen.oai.completion: 05-20 22:40:40] {927} INFO - evaluating data instance 122\n",
      "[flaml.autogen.oai.completion: 05-20 22:40:48] {927} INFO - evaluating data instance 123\n",
      "[flaml.autogen.oai.completion: 05-20 22:40:59] {927} INFO - evaluating data instance 124\n",
      "[flaml.autogen.oai.completion: 05-20 22:41:12] {927} INFO - evaluating data instance 125\n",
      "[flaml.autogen.oai.completion: 05-20 22:41:27] {927} INFO - evaluating data instance 126\n",
      "[flaml.autogen.oai.completion: 05-20 22:41:58] {927} INFO - evaluating data instance 127\n",
      "[flaml.autogen.oai.completion: 05-20 22:42:29] {927} INFO - evaluating data instance 128\n",
      "[flaml.autogen.oai.completion: 05-20 22:42:53] {927} INFO - evaluating data instance 129\n",
      "[flaml.autogen.oai.completion: 05-20 22:43:06] {927} INFO - evaluating data instance 130\n",
      "[flaml.autogen.oai.completion: 05-20 22:44:13] {927} INFO - evaluating data instance 131\n",
      "[flaml.autogen.oai.completion: 05-20 22:44:45] {927} INFO - evaluating data instance 132\n",
      "[flaml.autogen.oai.completion: 05-20 22:45:16] {927} INFO - evaluating data instance 133\n",
      "[flaml.autogen.oai.completion: 05-20 22:45:35] {927} INFO - evaluating data instance 134\n",
      "[flaml.autogen.oai.completion: 05-20 22:45:51] {927} INFO - evaluating data instance 135\n",
      "[flaml.autogen.oai.completion: 05-20 22:46:22] {927} INFO - evaluating data instance 136\n",
      "[flaml.autogen.oai.completion: 05-20 22:47:33] {927} INFO - evaluating data instance 137\n",
      "[flaml.autogen.oai.completion: 05-20 22:48:38] {927} INFO - evaluating data instance 138\n",
      "[flaml.autogen.oai.completion: 05-20 22:49:52] {927} INFO - evaluating data instance 139\n",
      "[flaml.autogen.oai.completion: 05-20 22:50:23] {927} INFO - evaluating data instance 140\n",
      "[flaml.autogen.oai.completion: 05-20 22:50:35] {927} INFO - evaluating data instance 141\n",
      "[flaml.autogen.oai.completion: 05-20 22:51:49] {927} INFO - evaluating data instance 142\n",
      "[flaml.autogen.oai.completion: 05-20 22:52:08] {927} INFO - evaluating data instance 143\n",
      "[flaml.autogen.oai.completion: 05-20 22:53:14] {927} INFO - evaluating data instance 144\n",
      "[flaml.autogen.oai.completion: 05-20 22:53:26] {927} INFO - evaluating data instance 145\n",
      "[flaml.autogen.oai.completion: 05-20 22:53:58] {927} INFO - evaluating data instance 146\n",
      "[flaml.autogen.oai.completion: 05-20 22:54:30] {927} INFO - evaluating data instance 147\n",
      "[flaml.autogen.oai.completion: 05-20 22:55:40] {927} INFO - evaluating data instance 148\n",
      "[flaml.autogen.oai.completion: 05-20 22:56:11] {927} INFO - evaluating data instance 149\n",
      "[flaml.autogen.oai.completion: 05-20 22:56:24] {927} INFO - evaluating data instance 150\n",
      "[flaml.autogen.oai.completion: 05-20 22:56:31] {927} INFO - evaluating data instance 151\n",
      "[flaml.autogen.oai.completion: 05-20 22:57:44] {927} INFO - evaluating data instance 152\n",
      "[flaml.autogen.oai.completion: 05-20 22:58:54] {927} INFO - evaluating data instance 153\n",
      "[flaml.autogen.oai.completion: 05-20 22:59:26] {927} INFO - evaluating data instance 154\n",
      "[flaml.autogen.oai.completion: 05-20 22:59:37] {927} INFO - evaluating data instance 155\n",
      "[flaml.autogen.oai.completion: 05-20 22:59:47] {927} INFO - evaluating data instance 156\n",
      "[flaml.autogen.oai.completion: 05-20 23:00:52] {927} INFO - evaluating data instance 157\n",
      "[flaml.autogen.oai.completion: 05-20 23:02:02] {927} INFO - evaluating data instance 158\n",
      "[flaml.autogen.oai.completion: 05-20 23:02:33] {927} INFO - evaluating data instance 159\n",
      "[flaml.autogen.oai.completion: 05-20 23:03:01] {927} INFO - evaluating data instance 160\n",
      "[flaml.autogen.oai.completion: 05-20 23:03:16] {927} INFO - evaluating data instance 161\n",
      "[flaml.autogen.oai.completion: 05-20 23:03:25] {927} INFO - evaluating data instance 162\n",
      "[flaml.autogen.oai.completion: 05-20 23:04:34] {927} INFO - evaluating data instance 163\n",
      "[flaml.autogen.oai.completion: 05-20 23:05:05] {927} INFO - evaluating data instance 164\n",
      "[flaml.autogen.oai.completion: 05-20 23:06:17] {927} INFO - evaluating data instance 165\n",
      "[flaml.autogen.oai.completion: 05-20 23:06:38] {927} INFO - evaluating data instance 166\n",
      "[flaml.autogen.oai.completion: 05-20 23:07:02] {927} INFO - evaluating data instance 167\n",
      "[flaml.autogen.oai.completion: 05-20 23:07:31] {927} INFO - evaluating data instance 168\n",
      "[flaml.autogen.oai.completion: 05-20 23:07:49] {927} INFO - evaluating data instance 169\n",
      "[flaml.autogen.oai.completion: 05-20 23:08:02] {927} INFO - evaluating data instance 170\n",
      "[flaml.autogen.oai.completion: 05-20 23:09:15] {927} INFO - evaluating data instance 171\n",
      "[flaml.autogen.oai.completion: 05-20 23:09:39] {927} INFO - evaluating data instance 172\n",
      "[flaml.autogen.oai.completion: 05-20 23:10:54] {927} INFO - evaluating data instance 173\n",
      "[flaml.autogen.oai.completion: 05-20 23:11:13] {927} INFO - evaluating data instance 174\n",
      "[flaml.autogen.oai.completion: 05-20 23:12:27] {927} INFO - evaluating data instance 175\n",
      "[flaml.autogen.oai.completion: 05-20 23:13:36] {927} INFO - evaluating data instance 176\n",
      "[flaml.autogen.oai.completion: 05-20 23:13:57] {927} INFO - evaluating data instance 177\n",
      "[flaml.autogen.oai.completion: 05-20 23:14:27] {927} INFO - evaluating data instance 178\n",
      "[flaml.autogen.oai.completion: 05-20 23:14:48] {927} INFO - evaluating data instance 179\n",
      "[flaml.autogen.oai.completion: 05-20 23:15:09] {927} INFO - evaluating data instance 180\n",
      "[flaml.autogen.oai.completion: 05-20 23:15:26] {927} INFO - evaluating data instance 181\n",
      "[flaml.autogen.oai.completion: 05-20 23:16:41] {927} INFO - evaluating data instance 182\n",
      "[flaml.autogen.oai.completion: 05-20 23:17:01] {927} INFO - evaluating data instance 183\n",
      "[flaml.autogen.oai.completion: 05-20 23:18:08] {927} INFO - evaluating data instance 184\n",
      "[flaml.autogen.oai.completion: 05-20 23:18:23] {927} INFO - evaluating data instance 185\n",
      "[flaml.autogen.oai.completion: 05-20 23:18:44] {927} INFO - evaluating data instance 186\n",
      "[flaml.autogen.oai.completion: 05-20 23:19:12] {927} INFO - evaluating data instance 187\n",
      "[flaml.autogen.oai.completion: 05-20 23:20:19] {927} INFO - evaluating data instance 188\n",
      "[flaml.autogen.oai.completion: 05-20 23:20:37] {927} INFO - evaluating data instance 189\n",
      "[flaml.autogen.oai.completion: 05-20 23:21:43] {927} INFO - evaluating data instance 190\n",
      "[flaml.autogen.oai.completion: 05-20 23:22:14] {927} INFO - evaluating data instance 191\n",
      "[flaml.autogen.oai.completion: 05-20 23:22:30] {927} INFO - evaluating data instance 192\n",
      "[flaml.autogen.oai.completion: 05-20 23:22:48] {927} INFO - evaluating data instance 193\n",
      "[flaml.autogen.oai.completion: 05-20 23:23:18] {927} INFO - evaluating data instance 194\n",
      "[flaml.autogen.oai.completion: 05-20 23:23:48] {927} INFO - evaluating data instance 195\n",
      "[flaml.autogen.oai.completion: 05-20 23:24:04] {927} INFO - evaluating data instance 196\n",
      "[flaml.autogen.oai.completion: 05-20 23:25:19] {927} INFO - evaluating data instance 197\n",
      "[flaml.autogen.oai.completion: 05-20 23:25:45] {927} INFO - evaluating data instance 198\n",
      "[flaml.autogen.oai.completion: 05-20 23:26:06] {927} INFO - evaluating data instance 199\n",
      "[flaml.autogen.oai.completion: 05-20 23:26:19] {927} INFO - evaluating data instance 200\n",
      "performance on test data with the tuned config: {'expected_success': 0.9825768303568986, 'success': 0.9900497512437811, 'success_vote': 0.9054726368159204, 'votes': 32.114427860696516, 'cost': 3.163385999999999, 'inference_cost': 0.015738238805970146}\n"
     ]
    }
   ],
   "source": [
    "# result = oai.ChatCompletion.test(test_data, logging_level=logging.INFO, config_list=config_list, **config)\n",
    "# print(\"performance on test data with the tuned config:\", result)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about the default, untuned gpt-4 config (with the same prompt as the tuned config)? We can evaluate it and compare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "performance on test data from gpt-4 with a default config: {'expected_success': 0.6965174129353234, 'success': 0.6965174129353234, 'success_vote': 0.6965174129353234, 'votes': 1.0, 'cost': 1.9264799999999993, 'inference_cost': 0.009584477611940295}\n"
     ]
    }
   ],
   "source": [
    "# the following code will cost roughly $2 if uncommented and run.\n",
    "\n",
    "# default_config = {\"model\": 'gpt-4', \"prompt\": prompts[0]}\n",
    "# default_result = oai.ChatCompletion.test(test_data, config_list=config_list, **default_config)\n",
    "# print(\"performance on test data from gpt-4 with a default config:\", default_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tuned config succeeds in 90.5% test cases\n",
      "untuned config succeeds in 69.7% test cases\n"
     ]
    }
   ],
   "source": [
    "# print(\"tuned config succeeds in {:.1f}% test cases\".format(result[\"success_vote\"] * 100))\n",
    "# print(\"untuned config succeeds in {:.1f}% test cases\".format(default_result[\"success_vote\"] * 100))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default use of GPT-4 has a much lower accuracy. Note that the default config has a lower inference cost. What if we heuristically increase the number of responses n?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following evaluation costs $3 and longer than one hour if you uncomment it and run it.\n",
    "\n",
    "# config_n2 = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 2}\n",
    "# result_n2 = oai.ChatCompletion.test(test_data, config_list=config_list, **config_n2)\n",
    "# print(\"performance on test data from gpt-4 with a default config and n=2:\", result_n2)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inference cost is doubled and matches the tuned config. But the success rate doesn't improve much. What if we further increase the number of responses n to 5?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following evaluation costs $8 and longer than one hour if you uncomment it and run it.\n",
    "\n",
    "# config_n5 = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 5}\n",
    "# result_n5 = oai.ChatCompletion.test(test_data, config_list=config_list, **config_n5)\n",
    "# print(\"performance on test data from gpt-4 with a default config and n=5:\", result_n5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find that the 'success_vote' metric is increased at the cost of exceeding the inference budget. But the tuned configuration has both higher 'success_vote' (91% vs. 87%) and lower average inference cost ($0.015 vs. $0.037 per instance).\n",
    "\n",
    "A developer could use flaml to tune the configuration to satisfy the target inference budget while maximizing the value out of it."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "2d910cfd2d2a4fc49fc30fbbdc5576a7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "454146d0f7224f038689031002906e6f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
        "IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
        "IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
       ],
       "layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
       "tabbable": null,
       "tooltip": null
      }
     },
     "577e1e3cc4db4942b0883577b3b52755": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
       "tabbable": null,
       "tooltip": null,
       "value": 1
      }
     },
     "6086462a12d54bafa59d3c4566f06cb2": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "74a6ba0c3cbc4051be0a83e152fe1e62": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "7d3f3d9e15894d05a4d188ff4f466554": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "background": null,
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "b40bdfb1ac1d4cffb7cefcb870c64d45": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
       "placeholder": "​",
       "style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
       "tabbable": null,
       "tooltip": null,
       "value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
      }
     },
     "ca245376fd9f4354af6b2befe4af4466": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "background": null,
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "dc83c7bff2f241309537a8119dfc7555": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e4ae2b6f5a974fd4bafb6abb9d12ff26": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
       "placeholder": "​",
       "style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
       "tabbable": null,
       "tooltip": null,
       "value": "100%"
      }
     },
     "f1355871cc6f4dd4b50d9df5af20e5c8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}