{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved. \n",
    "\n",
    "Licensed under the MIT License.\n",
    "\n",
    "# Use FLAML to Tune ChatGPT\n",
    "\n",
    "FLAML offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. Our study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
    "\n",
    "In this notebook, we tune OpenAI ChatGPT (both GPT-3.5 and GPT-4) models for math problem solving. We use [the MATH benchmark](https://crfm.stanford.edu/helm/latest/?group=math_chain_of_thought) for measuring mathematical problem solving on competition math problems with chain-of-thoughts style reasoning. \n",
    "\n",
    "## Requirements\n",
    "\n",
    "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [openai,blendsearch] option:\n",
    "```bash\n",
    "pip install flaml[openai,blendsearch]==1.2.1\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
     "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
     "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
     "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
    }
   },
   "outputs": [],
   "source": [
    "# %pip install flaml[openai,blendsearch]==1.2.1 datasets"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set your OpenAI key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:52.324240Z",
     "iopub.status.busy": "2023-02-13T23:40:52.323783Z",
     "iopub.status.idle": "2023-02-13T23:40:52.330570Z",
     "shell.execute_reply": "2023-02-13T23:40:52.329750Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = \"<your OpenAI API key here>\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the following to use Azure OpenAI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:52.333547Z",
     "iopub.status.busy": "2023-02-13T23:40:52.333249Z",
     "iopub.status.idle": "2023-02-13T23:40:52.336508Z",
     "shell.execute_reply": "2023-02-13T23:40:52.335858Z"
    }
   },
   "outputs": [],
   "source": [
    "# import openai\n",
    "# openai.api_type = \"azure\"\n",
    "# openai.api_base = \"https://<your_endpoint>.openai.azure.com/\"\n",
    "# openai.api_version = \"2023-03-15-preview\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "First, we load the competition_math dataset. The dataset contains 201 \"Level 2\" Algebra examples. We use a random sample of 20 examples for tuning the generation hyperparameters and the remaining for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:52.339977Z",
     "iopub.status.busy": "2023-02-13T23:40:52.339556Z",
     "iopub.status.idle": "2023-02-13T23:40:54.603349Z",
     "shell.execute_reply": "2023-02-13T23:40:54.602630Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default\n",
      "Found cached dataset competition_math (/home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c23bfd043e284ea29f8a6b4de2974637",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b/cache-f1cfe8228271b121.arrow\n",
      "Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b/cache-d155a2d38c23bd53.arrow\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 201\n"
     ]
    }
   ],
   "source": [
    "import datasets\n",
    "\n",
    "seed = 41\n",
    "data = datasets.load_dataset(\"competition_math\")\n",
    "train_data = data[\"train\"].shuffle(seed=seed)\n",
    "test_data = data[\"test\"].shuffle(seed=seed)\n",
    "n_tune_data = 20\n",
    "tune_data = [\n",
    "    {\n",
    "        \"problem\": train_data[x][\"problem\"],\n",
    "        \"solution\": train_data[x][\"solution\"],\n",
    "    }\n",
    "    for x in range(len(train_data)) if train_data[x][\"level\"] == \"Level 2\" and train_data[x][\"type\"] == \"Algebra\"\n",
    "][:n_tune_data]\n",
    "test_data = [\n",
    "    {\n",
    "        \"problem\": test_data[x][\"problem\"],\n",
    "        \"solution\": test_data[x][\"solution\"],\n",
    "    }\n",
    "    for x in range(len(test_data)) if test_data[x][\"level\"] == \"Level 2\" and test_data[x][\"type\"] == \"Algebra\"\n",
    "]\n",
    "print(len(tune_data), len(test_data))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Check a tuning example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:54.607152Z",
     "iopub.status.busy": "2023-02-13T23:40:54.606441Z",
     "iopub.status.idle": "2023-02-13T23:40:54.610504Z",
     "shell.execute_reply": "2023-02-13T23:40:54.609759Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rationalize the denominator of $\\displaystyle\\frac{21}{\\sqrt{21}}$.\n"
     ]
    }
   ],
   "source": [
    "print(tune_data[1][\"problem\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is one example of the canonical solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:54.613590Z",
     "iopub.status.busy": "2023-02-13T23:40:54.613168Z",
     "iopub.status.idle": "2023-02-13T23:40:54.616873Z",
     "shell.execute_reply": "2023-02-13T23:40:54.616193Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$\\dfrac{21}{\\sqrt{21}} = \\dfrac{21}{\\sqrt{21}} \\cdot \\dfrac{\\sqrt{21}}{\\sqrt{21}} = \\dfrac{21\\sqrt{21}}{21} = \\boxed{\\!\\sqrt{21}}$.\n"
     ]
    }
   ],
   "source": [
    "print(tune_data[1][\"solution\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Success Metric\n",
    "\n",
    "Before we start tuning, we need to define the success metric we want to optimize. For each math task, we use voting to select a response with the most common answers out of all the generated responses. If it has an equivalent answer to the canonical solution, we consider the task as successfully solved. Then we can optimize the mean success rate of a collection of tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:54.626998Z",
     "iopub.status.busy": "2023-02-13T23:40:54.626593Z",
     "iopub.status.idle": "2023-02-13T23:40:54.631383Z",
     "shell.execute_reply": "2023-02-13T23:40:54.630770Z"
    }
   },
   "outputs": [],
   "source": [
    "from flaml.autogen.math_utils import eval_math_responses"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Use the tuning data to find a good configuration\n",
    "\n",
    "### Import the oai and tune subpackages from flaml.\n",
    "\n",
    "FLAML has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `oai.ChatCompletion.tune` and to make a request with the tuned config: `oai.ChatCompletion.create`. First, we import oai from flaml:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:54.634335Z",
     "iopub.status.busy": "2023-02-13T23:40:54.633929Z",
     "iopub.status.idle": "2023-02-13T23:40:56.105700Z",
     "shell.execute_reply": "2023-02-13T23:40:56.105085Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from flaml import oai"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For (local) reproducibility and cost efficiency, we cache responses from OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:56.109177Z",
     "iopub.status.busy": "2023-02-13T23:40:56.108624Z",
     "iopub.status.idle": "2023-02-13T23:40:56.112651Z",
     "shell.execute_reply": "2023-02-13T23:40:56.112076Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "oai.ChatCompletion.set_cache(seed)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create a disk cache in \".cache/{seed}\". You can change `cache_path` in `set_cache()`. The cache for different seeds are stored separately.\n",
    "\n",
    "### Perform tuning\n",
    "\n",
    "The tuning will take a while to finish, depending on the optimization budget. The tuning will be performed under the specified optimization budgets.\n",
    "\n",
    "* `inference_budget` is the target average inference budget per instance in the benchmark. For example, 0.004 means the target inference budget is 0.004 dollars, which translates to 2000 tokens (input + output combined) if the gpt-3.5-turbo model is used.\n",
    "* `optimization_budget` is the total budget allowed to perform the tuning. For example, 1 means 1 dollars are allowed in total, which translates to 500K tokens for the gpt-3.5-turbo model.\n",
    "* `num_sumples` is the number of different hyperparameter configurations which is allowed to try. The tuning will stop after either num_samples trials or after optimization_budget dollars spent, whichever happens first. -1 means no hard restriction in the number of trials and the actual number is decided by `optimization_budget`.\n",
    "\n",
    "Users can specify tuning data, optimization metric, optimization mode, evaluation function, search spaces etc.. The default search space is:\n",
    "\n",
    "```python\n",
    "default_search_space = {\n",
    "    \"model\": tune.choice([\n",
    "        \"gpt-3.5-turbo\",\n",
    "        \"gpt-4\",\n",
    "    ]),\n",
    "    \"temperature_or_top_p\": tune.choice(\n",
    "        [\n",
    "            {\"temperature\": tune.uniform(0, 1)},\n",
    "            {\"top_p\": tune.uniform(0, 1)},\n",
    "        ]\n",
    "    ),\n",
    "    \"max_tokens\": tune.lograndint(50, 1000),\n",
    "    \"n\": tune.randint(1, 100),\n",
    "    \"prompt\": \"{prompt}\",\n",
    "}\n",
    "```\n",
    "\n",
    "The default search space can be overridden by users' input.\n",
    "For example, the following code specifies a fixed prompt template. For hyperparameters which don't appear in users' input, the default search space will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:40:56.115383Z",
     "iopub.status.busy": "2023-02-13T23:40:56.114975Z",
     "iopub.status.idle": "2023-02-13T23:41:55.045654Z",
     "shell.execute_reply": "2023-02-13T23:41:55.044973Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2023-04-02 21:11:48,741]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n",
      "\u001b[32m[I 2023-04-02 21:11:48,744]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[flaml.tune.tune: 04-02 21:11:48] {832} INFO - trial 1 config: {'prompt': 0, 'subspace': {'model': 'gpt-4', 'max_tokens': 347, 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'n': 1}}\n",
      "[flaml.tune.tune: 04-02 21:11:48] {215} INFO - result: {'expected_success': 0.8, 'success': 0.8, 'success_vote': 0.8, 'voted_answer': 'We use the distance formula to find the distance between the points $(0,4)$ and $(3,0)$: $$\\\\sqrt{(3-0)^2+(0-4)^2}=\\\\sqrt{3^2+(-4)^2}=\\\\sqrt{9+16}=\\\\sqrt{25}=\\\\boxed{5}.$$', 'votes': 1.0, 'total_cost': 0.14595, 'cost': 0.14595, 'inference_cost': 0.007297499999999999, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'model': 'gpt-4', 'max_tokens': 347, 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'n': 1}}, 'config/prompt': 0, 'config/subspace': {'model': 'gpt-4', 'max_tokens': 347, 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'n': 1}, 'experiment_tag': 'exp', 'time_total_s': 0.005024433135986328}\n",
      "[flaml.tune.tune: 04-02 21:11:48] {832} INFO - trial 2 config: {'prompt': 0, 'subspace': {'model': 'gpt-3.5-turbo', 'max_tokens': 347, 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'n': 1}}\n",
      "[flaml.tune.tune: 04-02 21:11:48] {215} INFO - result: {'expected_success': 0.75, 'success': 0.75, 'success_vote': 0.75, 'voted_answer': 'Using the distance formula, we have:\\n\\n$$\\\\sqrt{(3-0)^2+(0-4)^2}=\\\\sqrt{9+16}=\\\\sqrt{25}=5$$\\n\\nTherefore, the distance between the points (0,4) and (3,0) is $\\\\boxed{5}$.', 'votes': 1.0, 'total_cost': 0.153144, 'cost': 0.007194, 'inference_cost': 0.0003577, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'model': 'gpt-3.5-turbo', 'max_tokens': 347, 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'n': 1}}, 'config/prompt': 0, 'config/subspace': {'model': 'gpt-3.5-turbo', 'max_tokens': 347, 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'n': 1}, 'experiment_tag': 'exp', 'time_total_s': 0.004560708999633789}\n",
      "[flaml.tune.tune: 04-02 21:11:48] {832} INFO - trial 3 config: {'prompt': 0, 'subspace': {'max_tokens': 332, 'temperature_or_top_p': {'temperature': 0.22479664553084766}, 'n': 20, 'model': 'gpt-3.5-turbo'}}\n",
      "[flaml.tune.tune: 04-02 21:14:17] {215} INFO - result: {'expected_success': 0.9135658788605563, 'success': 0.95, 'success_vote': 0.8, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}.$$ Plugging in the coordinates of the two points, we get: $$\\\\sqrt{(3-0)^2+(0-4)^2}.$$ Simplifying, we have: $$\\\\sqrt{9+16}.$$ This equals $\\\\sqrt{25}=5$. Therefore, the distance between the two points is $\\\\boxed{5}$.', 'votes': 15.45, 'total_cost': 0.261568, 'cost': 0.108424, 'inference_cost': 0.0054192, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'max_tokens': 332, 'temperature_or_top_p': {'temperature': 0.22479664553084766}, 'n': 20, 'model': 'gpt-3.5-turbo'}}, 'config/prompt': 0, 'config/subspace': {'max_tokens': 332, 'temperature_or_top_p': {'temperature': 0.22479664553084766}, 'n': 20, 'model': 'gpt-3.5-turbo'}, 'experiment_tag': 'exp', 'time_total_s': 148.96061491966248}\n",
      "[flaml.tune.tune: 04-02 21:14:17] {832} INFO - trial 4 config: {'prompt': 0, 'subspace': {'max_tokens': 65, 'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'n': 51, 'model': 'gpt-3.5-turbo'}}\n",
      "[flaml.tune.tune: 04-02 21:15:13] {215} INFO - result: {'expected_success': 0.25, 'success': 0.25, 'success_vote': 0.25, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}$$ Letting $(x_1,y_1)=(0,4)$ and $(x_2,y_2)=(3,0)$, we have: $$\\\\sqrt{(3', 'votes': 15.3, 'total_cost': 0.3915420000000002, 'cost': 0.12997399999999998, 'inference_cost': 0.006496699999999999, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'max_tokens': 65, 'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'n': 51, 'model': 'gpt-3.5-turbo'}}, 'config/prompt': 0, 'config/subspace': {'max_tokens': 65, 'temperature_or_top_p': {'top_p': 0.003948266327914451}, 'n': 51, 'model': 'gpt-3.5-turbo'}, 'experiment_tag': 'exp', 'time_total_s': 56.23015236854553}\n",
      "[flaml.tune.tune: 04-02 21:15:13] {832} INFO - trial 5 config: {'prompt': 0, 'subspace': {'max_tokens': 433, 'temperature_or_top_p': {'top_p': 0.7145757833976906}, 'n': 54, 'model': 'gpt-3.5-turbo'}}\n",
      "[flaml.tune.tune: 04-02 21:18:45] {215} INFO - result: {'expected_success': 0.9999912490789106, 'success': 1.0, 'success_vote': 0.9, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}.$$ Letting $(x_1,y_1)=(0,4)$ and $(x_2,y_2)=(3,0)$, we have: \\\\begin{align*}\\n\\\\sqrt{(3-0)^2+(0-4)^2} &= \\\\sqrt{9+16} \\\\\\\\\\n&= \\\\sqrt{25} \\\\\\\\\\n&= \\\\boxed{5}.\\n\\\\end{align*}', 'votes': 41.65, 'total_cost': 0.6993100000000001, 'cost': 0.30776799999999993, 'inference_cost': 0.014812200000000001, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'max_tokens': 433, 'temperature_or_top_p': {'top_p': 0.7145757833976906}, 'n': 54, 'model': 'gpt-3.5-turbo'}}, 'config/prompt': 0, 'config/subspace': {'max_tokens': 433, 'temperature_or_top_p': {'top_p': 0.7145757833976906}, 'n': 54, 'model': 'gpt-3.5-turbo'}, 'experiment_tag': 'exp', 'time_total_s': 211.06462264060974}\n",
      "[flaml.tune.tune: 04-02 21:18:45] {832} INFO - trial 6 config: {'prompt': 0, 'subspace': {'max_tokens': 375, 'temperature_or_top_p': {'temperature': 0.6177669784693172}, 'n': 51, 'model': 'gpt-4'}}\n",
      "[flaml.tune.tune: 04-02 21:19:09] {215} INFO - result: {'success_vote': 0, 'total_cost': 0.75685, 'cost': 0.057539999999999994, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'max_tokens': 375, 'temperature_or_top_p': {'temperature': 0.6177669784693172}, 'n': 51, 'model': 'gpt-4'}}, 'config/prompt': 0, 'config/subspace': {'max_tokens': 375, 'temperature_or_top_p': {'temperature': 0.6177669784693172}, 'n': 51, 'model': 'gpt-4'}, 'experiment_tag': 'exp', 'time_total_s': 24.581016063690186}\n",
      "[flaml.tune.tune: 04-02 21:19:09] {832} INFO - trial 7 config: {'prompt': 0, 'subspace': {'max_tokens': 557, 'temperature_or_top_p': {'top_p': 0.3192360889885453}, 'n': 9, 'model': 'gpt-3.5-turbo'}}\n",
      "[flaml.tune.tune: 04-02 21:20:54] {215} INFO - result: {'expected_success': 0.7499999998709412, 'success': 0.75, 'success_vote': 0.75, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(3-0)^2+(0-4)^2}=\\\\sqrt{9+16}=\\\\sqrt{25}=5.$$ Therefore, the distance between the points (0,4) and (3,0) is $\\\\boxed{5}$.', 'votes': 8.45, 'total_cost': 0.8058679999999998, 'cost': 0.049018, 'inference_cost': 0.0024489, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'max_tokens': 557, 'temperature_or_top_p': {'top_p': 0.3192360889885453}, 'n': 9, 'model': 'gpt-3.5-turbo'}}, 'config/prompt': 0, 'config/subspace': {'max_tokens': 557, 'temperature_or_top_p': {'top_p': 0.3192360889885453}, 'n': 9, 'model': 'gpt-3.5-turbo'}, 'experiment_tag': 'exp', 'time_total_s': 104.52026915550232}\n",
      "[flaml.tune.tune: 04-02 21:20:54] {832} INFO - trial 8 config: {'prompt': 0, 'subspace': {'max_tokens': 597, 'temperature_or_top_p': {'top_p': 0.5475861559192435}, 'n': 82, 'model': 'gpt-3.5-turbo'}}\n",
      "[flaml.tune.tune: 04-02 21:22:29] {215} INFO - result: {'success_vote': 0, 'total_cost': 0.9975759999999998, 'cost': 0.19170800000000002, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'max_tokens': 597, 'temperature_or_top_p': {'top_p': 0.5475861559192435}, 'n': 82, 'model': 'gpt-3.5-turbo'}}, 'config/prompt': 0, 'config/subspace': {'max_tokens': 597, 'temperature_or_top_p': {'top_p': 0.5475861559192435}, 'n': 82, 'model': 'gpt-3.5-turbo'}, 'experiment_tag': 'exp', 'time_total_s': 94.87062358856201}\n",
      "[flaml.tune.tune: 04-02 21:22:29] {832} INFO - trial 9 config: {'prompt': 0, 'subspace': {'max_tokens': 142, 'temperature_or_top_p': {'temperature': 0.8839364795611863}, 'n': 33, 'model': 'gpt-4'}}\n",
      "[flaml.tune.tune: 04-02 21:22:38] {215} INFO - result: {'success_vote': 0, 'total_cost': 1.0206759999999997, 'cost': 0.0231, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'max_tokens': 142, 'temperature_or_top_p': {'temperature': 0.8839364795611863}, 'n': 33, 'model': 'gpt-4'}}, 'config/prompt': 0, 'config/subspace': {'max_tokens': 142, 'temperature_or_top_p': {'temperature': 0.8839364795611863}, 'n': 33, 'model': 'gpt-4'}, 'experiment_tag': 'exp', 'time_total_s': 9.344996690750122}\n",
      "[flaml.tune.tune: 04-02 21:22:38] {855} WARNING - fail to sample a trial for 100 times in a row, stopping.\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "\n",
    "prompts = [\"{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\\\boxed{{}}.\"]\n",
    "config, analysis = oai.ChatCompletion.tune(\n",
    "    data=tune_data,  # the data for tuning\n",
    "    metric=\"success_vote\",  # the metric to optimize\n",
    "    mode=\"max\",  # the optimization mode\n",
    "    eval_func=eval_math_responses,  # the evaluation function to return the success metrics\n",
    "    # log_file_name=\"logs/math.log\",  # the log file name\n",
    "    inference_budget=0.02,  # the inference budget (dollar per instance)\n",
    "    optimization_budget=1,  # the optimization budget (dollar in total)\n",
    "    # num_samples can further limit the number of trials for different hyperparameter configurations;\n",
    "    # -1 means decided by the optimization budget only\n",
    "    num_samples=-1,\n",
    "    # model=\"gpt-3.5-turbo\",  # uncomment if you don't have access to gpt-4\n",
    "    prompt=prompts,  # the prompt templates to choose from\n",
    "    # stop=\"###\",  # the stop sequence\n",
    "    logging_level=logging.INFO,  # the logging level\n",
    ")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output tuning results\n",
    "\n",
    "After the tuning, we can print out the config and the result found by FLAML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:41:55.049204Z",
     "iopub.status.busy": "2023-02-13T23:41:55.048871Z",
     "iopub.status.idle": "2023-02-13T23:41:55.053284Z",
     "shell.execute_reply": "2023-02-13T23:41:55.052574Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "optimized config {'prompt': '{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\\\boxed{{}}.', 'max_tokens': 433, 'n': 54, 'model': 'gpt-3.5-turbo', 'stop': None, 'top_p': 0.7145757833976906}\n",
      "best result on tuning data {'expected_success': 0.9999912490789106, 'success': 1.0, 'success_vote': 0.9, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}.$$ Letting $(x_1,y_1)=(0,4)$ and $(x_2,y_2)=(3,0)$, we have: \\\\begin{align*}\\n\\\\sqrt{(3-0)^2+(0-4)^2} &= \\\\sqrt{9+16} \\\\\\\\\\n&= \\\\sqrt{25} \\\\\\\\\\n&= \\\\boxed{5}.\\n\\\\end{align*}', 'votes': 41.65, 'total_cost': 0.6993100000000001, 'cost': 0.30776799999999993, 'inference_cost': 0.014812200000000001, 'training_iteration': 0, 'config': {'prompt': 0, 'subspace': {'max_tokens': 433, 'temperature_or_top_p': {'top_p': 0.7145757833976906}, 'n': 54, 'model': 'gpt-3.5-turbo'}}, 'config/prompt': 0, 'config/subspace': {'max_tokens': 433, 'temperature_or_top_p': {'top_p': 0.7145757833976906}, 'n': 54, 'model': 'gpt-3.5-turbo'}, 'experiment_tag': 'exp', 'time_total_s': 211.06462264060974}\n"
     ]
    }
   ],
   "source": [
    "print(\"optimized config\", config)\n",
    "print(\"best result on tuning data\", analysis.best_result)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Make a request with the tuned config\n",
    "\n",
    "We can apply the tuned config on the request for an example task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:41:55.056205Z",
     "iopub.status.busy": "2023-02-13T23:41:55.055631Z",
     "iopub.status.idle": "2023-02-13T23:41:56.039259Z",
     "shell.execute_reply": "2023-02-13T23:41:56.038427Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "response on an example data instance: {\n",
      "  \"choices\": [\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 0,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to eliminate the radical in the denominator. We can do this by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$Therefore, the simplified expression is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 1,\n",
      "      \"message\": {\n",
      "        \"content\": \"We have $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 2,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify the expression by multiplying both the numerator and the denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 3,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\sqrt{21}$ by factoring it as $\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, we have \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}}&=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\\\\\\\n&=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{3}}{\\\\sqrt{3}}\\\\\\\\\\n&=\\\\frac{21\\\\sqrt{3}}{3\\\\sqrt{7}}\\\\\\\\\\n&=\\\\frac{7\\\\sqrt{3}}{\\\\sqrt{7}}\\\\\\\\\\n&=\\\\frac{7\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{7}\\\\\\\\\\n&=\\\\boxed{\\\\sqrt{21}}.\\n\\\\end{align*}\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 4,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we get: $$\\\\frac{21\\\\sqrt{21}}{21}=\\\\frac{\\\\cancelto{1}{21}\\\\cdot\\\\sqrt{21}}{\\\\cancelto{1}{21}}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 5,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can start by simplifying the fraction by canceling a factor of $\\\\sqrt{21}$ from the numerator and denominator: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}.$$ Therefore, the denominator is already rationalized, and the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 6,\n",
      "      \"message\": {\n",
      "        \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 7,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to eliminate the radical in the denominator. To do this, we multiply both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we get $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 8,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\sqrt{21}$ by factoring it as $\\\\sqrt{21} = \\\\sqrt{3\\\\cdot7} = \\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, we have \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\\\\\\\n&= \\\\frac{21}{\\\\sqrt{3}}\\\\cdot\\\\frac{1}{\\\\sqrt{7}}\\\\\\\\\\n&= \\\\frac{21}{\\\\sqrt{3}}\\\\cdot\\\\frac{\\\\sqrt{7}}{\\\\sqrt{7}\\\\cdot\\\\sqrt{7}}\\\\\\\\\\n&= \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3\\\\cdot7}}\\\\\\\\\\n&= \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{21}}.\\n\\\\end{align*}Thus, the denominator is rationalized, and the simplified answer is $\\\\boxed{\\\\frac{21\\\\sqrt{7}}{21}}$, which can be further simplified to $\\\\boxed{\\\\sqrt{7}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 9,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}$ by multiplying the numerator and denominator by $\\\\sqrt{21}$. This gives us: $$\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$ Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 10,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\sqrt{21}$ by factoring it as $\\\\sqrt{21} = \\\\sqrt{3 \\\\cdot 7} = \\\\sqrt{3} \\\\cdot \\\\sqrt{7}$. Therefore, we have \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7}} \\\\\\\\\\n&= \\\\frac{21}{\\\\sqrt{3}} \\\\cdot \\\\frac{1}{\\\\sqrt{7}} \\\\\\\\\\n&= \\\\frac{21}{\\\\sqrt{3}} \\\\cdot \\\\frac{\\\\sqrt{7}}{\\\\sqrt{7} \\\\cdot \\\\sqrt{7}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3 \\\\cdot 7}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{7}}{3\\\\sqrt{7}} \\\\\\\\\\n&= \\\\frac{21}{3} \\\\\\\\\\n&= \\\\boxed{7}.\\n\\\\end{align*}\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 11,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\frac{21}{\\\\sqrt{21}}$ by multiplying both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$Therefore, the rationalized form of the denominator is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 12,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the radical in the denominator, so we multiply both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying the fraction, we get $\\\\frac{\\\\cancel{21}\\\\sqrt{21}}{\\\\cancel{21}} = \\\\sqrt{21}$. Thus, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}} = \\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 13,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to eliminate the radical from the denominator. To do this, we multiply both the numerator and denominator by $\\\\sqrt{21}$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\boxed{\\\\sqrt{21}}.\\n\\\\end{align*}\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 14,\n",
      "      \"message\": {\n",
      "        \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$Therefore, the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 15,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$Simplifying, we have $\\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}$. Thus, the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 16,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\frac{21}{\\\\sqrt{21}}$ by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21\\\\cdot \\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot \\\\sqrt{21}} = \\\\frac{21\\\\cdot \\\\sqrt{21}}{21} = \\\\sqrt{21}.$$ Therefore, the rationalized form of the denominator is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 17,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we get $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 18,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the square root in the denominator, so we multiply the numerator and denominator by $\\\\sqrt{21}$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\cdot\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\boxed{\\\\sqrt{21}}.\\n\\\\end{align*}\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 19,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can start by simplifying the fraction $\\\\frac{21}{\\\\sqrt{21}}$ by dividing the numerator and denominator by the greatest perfect square that divides both of them, which is 3. This gives us:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{7\\\\cdot 3}{\\\\sqrt{7\\\\cdot 3}}=\\\\frac{7\\\\cdot 3}{\\\\sqrt{7}\\\\cdot \\\\sqrt{3}}$$\\n\\nNext, we can rationalize the denominator by multiplying the numerator and denominator by $\\\\sqrt{3}$:\\n\\n$$\\\\frac{7\\\\cdot 3}{\\\\sqrt{7}\\\\cdot \\\\sqrt{3}}=\\\\frac{7\\\\cdot 3\\\\cdot \\\\sqrt{3}}{\\\\sqrt{7}\\\\cdot \\\\sqrt{3}\\\\cdot \\\\sqrt{3}}=\\\\frac{21\\\\sqrt{3}}{3}$$\\n\\nFinally, we can simplify the fraction by dividing both the numerator and denominator by 3:\\n\\n$$\\\\frac{21\\\\sqrt{3}}{3}=7\\\\sqrt{3}$$\\n\\nTherefore, the final answer is $\\\\boxed{7\\\\sqrt{3}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 20,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying both the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 21,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can start by simplifying the denominator using the property $\\\\sqrt{a^2}=|a|$. Since $21$ is positive, $\\\\sqrt{21}=|21|=21$. Therefore, we have: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{21}=\\\\boxed{1}$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 22,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\frac{21}{\\\\sqrt{21}}$ by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21\\\\cdot \\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot \\\\sqrt{21}} = \\\\frac{21\\\\cdot \\\\sqrt{21}}{21}.$$ Simplifying, we get $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 23,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\sqrt{21}$ by finding its prime factorization: $$\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$$ So, we have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}$$ To rationalize the denominator, we need to get rid of the square root in the denominator by multiplying both the numerator and denominator by $\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$: $$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}=\\\\frac{21\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}=\\\\frac{3\\\\cdot7\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$$ Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 24,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21 \\\\cdot \\\\sqrt{21}}{\\\\sqrt{21} \\\\cdot \\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we get $$\\\\frac{21\\\\sqrt{21}}{21} = \\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 25,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\frac{21}{\\\\sqrt{21}}$ by multiplying both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying the fraction, we have $$\\\\frac{21\\\\sqrt{21}}{21} = \\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 26,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying the numerator and denominator, we have: $$\\\\frac{21\\\\sqrt{21}}{21}=\\\\frac{3\\\\cdot7\\\\cdot\\\\sqrt{3\\\\cdot7}}{3\\\\cdot7}=\\\\frac{\\\\sqrt{3\\\\cdot7}}{1}=\\\\sqrt{21}.$$ Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 27,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the square root in the denominator. One way to do this is to multiply the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Now we can simplify by canceling the common factor of 21: $$\\\\frac{21\\\\sqrt{21}}{21}=\\\\frac{\\\\cancel{21}\\\\sqrt{21}}{\\\\cancel{21}}=\\\\sqrt{21}.$$ Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 28,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}$ by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$ Therefore, the simplified expression is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 29,\n",
      "      \"message\": {\n",
      "        \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 30,\n",
      "      \"message\": {\n",
      "        \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$ Therefore, the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 31,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the radical in the denominator. To do this, we multiply the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we have $$\\\\frac{21\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 32,\n",
      "      \"message\": {\n",
      "        \"content\": \"We have $\\\\dfrac{21}{\\\\sqrt{21}} = \\\\dfrac{21}{\\\\sqrt{21}}\\\\cdot\\\\dfrac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\dfrac{21\\\\sqrt{21}}{21} = \\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 33,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can start by simplifying the square root in the denominator: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3\\\\cdot7}}=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}$$ To rationalize the denominator, we need to get rid of the square root in the denominator. We can do this by multiplying the numerator and denominator by $\\\\sqrt{7}$: $$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{7}}{\\\\sqrt{7}}=\\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}\\\\cdot\\\\sqrt{7}}=\\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3\\\\cdot7}}$$ Simplifying the denominator gives us: $$\\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3\\\\cdot7}}=\\\\frac{21\\\\sqrt{7}}{\\\\sqrt{21}}$$ Therefore, the simplified form of $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}$ is $\\\\boxed{\\\\frac{21\\\\sqrt{7}}{21}}$, or simply $\\\\boxed{\\\\sqrt{7}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 34,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying the numerator, we have $\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}$. Therefore, $\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 35,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the square root in the denominator. We can do this by multiplying both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we get $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 36,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can rationalize the denominator by multiplying the numerator and denominator by $\\\\sqrt{21}$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\sqrt{21}.\\n\\\\end{align*}Therefore, the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 37,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can rationalize the denominator by multiplying both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$ Therefore, the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 38,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can simplify the denominator by using the fact that $\\\\sqrt{21} = \\\\sqrt{3\\\\cdot7} = \\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.$$To rationalize the denominator, we multiply both the numerator and denominator by $\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$: $$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}} \\\\cdot \\\\frac{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}} = \\\\frac{21\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}.$$Simplifying, we get $$\\\\frac{21\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7} = \\\\frac{3\\\\cdot7\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7} = \\\\boxed{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 39,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can rationalize the denominator by multiplying both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$Therefore, the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 40,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the square root in the denominator. One way to do this is to multiply both the numerator and denominator by $\\\\sqrt{21}$. We get $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we get $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 41,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the radical in the denominator. To do this, we multiply the numerator and denominator by $\\\\sqrt{21}$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\cdot\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\boxed{\\\\sqrt{21}}.\\n\\\\end{align*} We simplified the expression as much as possible by canceling the factor of 21 in the numerator and denominator.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 42,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to eliminate the radical from the denominator. To do this, we can multiply the numerator and denominator by $\\\\sqrt{21}$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}\\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21}\\\\\\\\\\n&= \\\\sqrt{21}\\n\\\\end{align*}Therefore, the simplified expression is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 43,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the radical in the denominator. One way to do this is to multiply both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we get $\\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}$. Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}} = \\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 44,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to eliminate the radical from the denominator. To do this, we multiply the numerator and denominator by $\\\\sqrt{21}$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}\\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21}\\\\\\\\\\n&= \\\\sqrt{21}.\\n\\\\end{align*}Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}} = \\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 45,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying both the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ We can simplify this by canceling a factor of 21: $$\\\\frac{21\\\\sqrt{21}}{21}=\\\\frac{\\\\cancelto{1}{21}\\\\sqrt{21}}{\\\\cancelto{1}{21}}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 46,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying both the numerator and the denominator by $\\\\sqrt{21}$ to get rid of the radical in the denominator: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\cdot\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\frac{\\\\cancel{21}\\\\cdot\\\\sqrt{21}}{\\\\cancel{21}} \\\\\\\\\\n&= \\\\sqrt{21}.\\n\\\\end{align*}Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 47,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can start by simplifying the square root in the denominator: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3\\\\cdot7}}.$$ Since $\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$, we can rewrite the fraction as follows: $$\\\\frac{21}{\\\\sqrt{3\\\\cdot7}}=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.$$ Now, we can multiply the numerator and denominator by $\\\\sqrt{3}$ to get rid of the radical in the denominator: $$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{3}}{\\\\sqrt{3}}=\\\\frac{21\\\\sqrt{3}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{3\\\\cdot7}}.$$ Simplifying the denominator, we have $\\\\sqrt{3}\\\\cdot\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3^2\\\\cdot7}=3\\\\sqrt{7}$. Therefore, we can simplify the expression as follows: $$\\\\frac{21\\\\sqrt{3}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{3\\\\cdot7}}=\\\\frac{21\\\\sqrt{3}}{3\\\\sqrt{7}}=\\\\frac{7\\\\sqrt{3}}{\\\\sqrt{7}}.$$ To rationalize the denominator, we can multiply the numerator and denominator by $\\\\sqrt{7}$: $$\\\\frac{7\\\\sqrt{3}}{\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{7}}{\\\\sqrt{7}}=\\\\frac{7\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}.$$ Therefore, the final answer is $\\\\boxed{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 48,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the square root in the denominator. We can do this by multiplying both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$Simplifying, we get $\\\\frac{21\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 49,\n",
      "      \"message\": {\n",
      "        \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 50,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to get rid of the square root in the denominator. To do this, we can multiply both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$ Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 51,\n",
      "      \"message\": {\n",
      "        \"content\": \"We begin by multiplying both the numerator and the denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying the numerator, we have $\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}$. Thus, the rationalized form of $\\\\frac{21}{\\\\sqrt{21}}$ is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 52,\n",
      "      \"message\": {\n",
      "        \"content\": \"We want to eliminate the radical in the denominator. To do this, we can multiply both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we get $$\\\\frac{21\\\\sqrt{21}}{21}=\\\\frac{\\\\cancelto{1}{21}\\\\cdot\\\\sqrt{21}}{\\\\cancelto{1}{21}}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"finish_reason\": \"stop\",\n",
      "      \"index\": 53,\n",
      "      \"message\": {\n",
      "        \"content\": \"We can start by simplifying the denominator. Since $\\\\sqrt{21}$ can be written as $\\\\sqrt{3\\\\cdot7}$, we can simplify as follows: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3\\\\cdot7}} = \\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.$$ Now we can rationalize the denominator by multiplying the numerator and denominator by $\\\\sqrt{7}$: $$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{7}}{\\\\sqrt{7}} = \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}\\\\cdot\\\\sqrt{7}} = \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3\\\\cdot7}}.$$ Finally, we can simplify the denominator: $$\\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3\\\\cdot7}} = \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3}}.$$ Thus, the answer is $\\\\boxed{\\\\frac{21}{\\\\sqrt{3}}}$.\",\n",
      "        \"role\": \"assistant\"\n",
      "      }\n",
      "    }\n",
      "  ],\n",
      "  \"created\": 1680470558,\n",
      "  \"id\": \"chatcmpl-70zRmEgMXBlt1xSrJIWXTfYPYGoeU\",\n",
      "  \"model\": \"gpt-3.5-turbo-0301\",\n",
      "  \"object\": \"chat.completion\",\n",
      "  \"usage\": {\n",
      "    \"completion_tokens\": 6606,\n",
      "    \"prompt_tokens\": 50,\n",
      "    \"total_tokens\": 6656\n",
      "  }\n",
      "}\n",
      "metric_results on the example data instance: {'expected_success': 1.0, 'success': True, 'success_vote': 1.0, 'voted_answer': 'We want to eliminate the radical in the denominator. We can do this by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$Therefore, the simplified expression is $\\\\boxed{\\\\sqrt{21}}$.', 'votes': 45}\n"
     ]
    }
   ],
   "source": [
    "response = oai.ChatCompletion.create(context=tune_data[1], **config)\n",
    "metric_results = eval_math_responses(oai.ChatCompletion.extract_text(response), **tune_data[1])\n",
    "print(\"response on an example data instance:\", response)\n",
    "print(\"metric_results on the example data instance:\", metric_results)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the success rate on the test data\n",
    "\n",
    "You can use flaml's `oai.ChatCompletion.test` to evaluate the performance of an entire dataset with the tuned config. The following code will take a while (30 mins to 1 hour) to evaluate all the test data instances if uncommented and run. It will cost roughly $3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-13T23:41:56.042764Z",
     "iopub.status.busy": "2023-02-13T23:41:56.042086Z",
     "iopub.status.idle": "2023-02-13T23:53:05.597643Z",
     "shell.execute_reply": "2023-02-13T23:53:05.596603Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "performance on test data with the tuned config: {'expected_success': 0.9765082554311302, 'success': 0.9800995024875622, 'success_vote': 0.9104477611940298, 'votes': 39.87064676616915, 'cost': 3.0702499999999997, 'inference_cost': 0.015274875621890546}\n"
     ]
    }
   ],
   "source": [
    "# result = oai.Completion.test(test_data, config, eval_math_responses)\n",
    "# print(\"performance on test data with the tuned config:\", result)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about the default, untuned gpt-4 config (with the same prompt as the tuned config)? We can evaluate it and compare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "performance on test data from gpt-4 with a default config: {'expected_success': 0.6716417910447762, 'success': 0.6716417910447762, 'success_vote': 0.6716417910447762, 'votes': 0.9900497512437811, 'cost': 1.92477, 'inference_cost': 0.009575970149253732}\n"
     ]
    }
   ],
   "source": [
    "# assuming you have access to gpt-4; otherwise use gpt-3.5-turbo\n",
    "# the following code will cost roughly $2 if uncommented and run.\n",
    "\n",
    "# default_config = {\"model\": 'gpt-4', \"prompt\": prompts[0]}\n",
    "# default_result = oai.Completion.test(test_data, default_config, eval_math_responses)\n",
    "# print(\"performance on test data from gpt-4 with a default config:\", default_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tuned config succeeds in 91.0% test cases\n",
      "untuned config succeeds in 67.2% test cases\n"
     ]
    }
   ],
   "source": [
    "# print(\"tuned config succeeds in {:.1f}% test cases\".format(result[\"success_vote\"] * 100))\n",
    "# print(\"untuned config succeeds in {:.1f}% test cases\".format(default_result[\"success_vote\"] * 100))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default use of GPT-4 has a much lower accuracy. Note that the default config has a lower inference cost. What if we heuristically increase the number of responses n?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "performance on test data from gpt-4 with a default config and n=2: {'expected_success': 0.7699004975124378, 'success': 0.835820895522388, 'success_vote': 0.7064676616915423, 'votes': 1.5870646766169154, 'cost': 3.267810000000001, 'inference_cost': 0.016257761194029857}\n"
     ]
    }
   ],
   "source": [
    "# The following evaluation costs $3 and longer than one hour if you uncomment it and run it.\n",
    "\n",
    "# config_n2 = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 2}\n",
    "# result_n2 = oai.ChatCompletion.test(test_data, config_n2, eval_math_responses)\n",
    "# print(\"performance on test data from gpt-4 with a default config and n=2:\", result_n2)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inference cost is doubled and matches the tuned config. But the success rate doesn't improve much. What if we further increase the number of responses n to 5?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "performance on test data from gpt-4 with a default config and n=5: {'expected_success': 0.9197771144278607, 'success': 0.9552238805970149, 'success_vote': 0.8656716417910447, 'votes': 3.691542288557214, 'cost': 7.518809999999999, 'inference_cost': 0.03740701492537313}\n"
     ]
    }
   ],
   "source": [
    "# The following evaluation costs $8 and longer than one hour if you uncomment it and run it.\n",
    "\n",
    "# config_n5 = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 5}\n",
    "# result_n5 = oai.ChatCompletion.test(test_data, config_n5, eval_math_responses)\n",
    "# print(\"performance on test data from gpt-4 with a default config and n=5:\", result_n5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find that the 'success_vote' metric is increased at the cost of exceeding the inference budget. But the tuned configuration has both higher 'success_vote' (91% vs. 87%) and lower average inference cost ($0.015 vs. $0.037 per instance).\n",
    "\n",
    "A developer could use flaml to tune the configuration to satisfy the target inference budget while maximizing the value out of it."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "2d910cfd2d2a4fc49fc30fbbdc5576a7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "454146d0f7224f038689031002906e6f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
        "IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
        "IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
       ],
       "layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
       "tabbable": null,
       "tooltip": null
      }
     },
     "577e1e3cc4db4942b0883577b3b52755": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
       "tabbable": null,
       "tooltip": null,
       "value": 1
      }
     },
     "6086462a12d54bafa59d3c4566f06cb2": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "74a6ba0c3cbc4051be0a83e152fe1e62": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "7d3f3d9e15894d05a4d188ff4f466554": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "background": null,
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "b40bdfb1ac1d4cffb7cefcb870c64d45": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
       "placeholder": "​",
       "style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
       "tabbable": null,
       "tooltip": null,
       "value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
      }
     },
     "ca245376fd9f4354af6b2befe4af4466": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "background": null,
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "dc83c7bff2f241309537a8119dfc7555": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e4ae2b6f5a974fd4bafb6abb9d12ff26": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
       "placeholder": "​",
       "style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
       "tabbable": null,
       "tooltip": null,
       "value": "100%"
      }
     },
     "f1355871cc6f4dd4b50d9df5af20e5c8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}