yujunjun/autogen
1
0
Fork 0
mirror of https://github.com/microsoft/autogen.git synced 2025-09-29 01:59:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
autogen/notebook/integrate_chatgpt.ipynb

1796 lines
92 KiB
Plaintext
Raw Normal View History

gpt-4 support; openai workflow fix; model str; timeout; voting (#958) * workflow; model str; timeout * voting * notebook * pull request * recover workflow * voted answer * aoai * ignore None answer * default config * note * gpt-4 * n=5 * cleanup * config name * introduction * readme * avoid None * add output/ to gitignore * openai version * invalid var * comment long running cells
2023-03-26 10:13:06 -07:00
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
"\n",
"Licensed under the MIT License.\n",
"\n",
"# Use FLAML to Tune ChatGPT\n",
"\n",
"FLAML offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. Our study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
"\n",
"In this notebook, we tune OpenAI ChatGPT (both GPT-3.5 and GPT-4) models for math problem solving. We use [the MATH benchmark](https://crfm.stanford.edu/helm/latest/?group=math_chain_of_thought) for measuring mathematical problem solving on competition math problems with chain-of-thoughts style reasoning. \n",
"\n",
"## Requirements\n",
"\n",
"FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [openai] option:\n",
"```bash\n",
"pip install flaml[openai]==1.2.0\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:52.317406Z",
"iopub.status.busy": "2023-02-13T23:40:52.316561Z",
"iopub.status.idle": "2023-02-13T23:40:52.321193Z",
"shell.execute_reply": "2023-02-13T23:40:52.320628Z"
}
},
"outputs": [],
"source": [
"# %pip install flaml[openai]==1.2.0 datasets"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Set your OpenAI key:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:52.324240Z",
"iopub.status.busy": "2023-02-13T23:40:52.323783Z",
"iopub.status.idle": "2023-02-13T23:40:52.330570Z",
"shell.execute_reply": "2023-02-13T23:40:52.329750Z"
}
},
"outputs": [],
"source": [
"import os\n",
"\n",
"if \"OPENAI_API_KEY\" not in os.environ:\n",
" os.environ[\"OPENAI_API_KEY\"] = \"<your OpenAI API key here>\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Uncomment the following to use Azure OpenAI:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:52.333547Z",
"iopub.status.busy": "2023-02-13T23:40:52.333249Z",
"iopub.status.idle": "2023-02-13T23:40:52.336508Z",
"shell.execute_reply": "2023-02-13T23:40:52.335858Z"
}
},
"outputs": [],
"source": [
"# openai.api_type = \"azure\"\n",
"# openai.api_base = \"https://<your_endpoint>.openai.azure.com/\"\n",
"# openai.api_version = \"2022-12-01\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load dataset\n",
"\n",
"First, we load the competition_math dataset. The dataset contains 201 \"Level 2\" Algebra examples. We use a random sample of 20 examples for tuning the generation hyperparameters and the remaining for evaluation. We use one demonstration example in the prompt."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:52.339977Z",
"iopub.status.busy": "2023-02-13T23:40:52.339556Z",
"iopub.status.idle": "2023-02-13T23:40:54.603349Z",
"shell.execute_reply": "2023-02-13T23:40:54.602630Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using custom data configuration default\n",
"Found cached dataset competition_math (/home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b)\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "8358c4bf9cc44b99916c9b6cb1e3a279",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/2 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b/cache-f1cfe8228271b121.arrow\n",
"Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/competition_math/default/1.0.0/2a2a2995c2847186883ecd64f69be7d602b8a6f6b51950624d4dc2263f93333b/cache-d155a2d38c23bd53.arrow\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"20 201\n"
]
}
],
"source": [
"import datasets\n",
"\n",
"seed = 41\n",
"data = datasets.load_dataset(\"competition_math\")\n",
"train_data = data[\"train\"].shuffle(seed=seed)\n",
"test_data = data[\"test\"].shuffle(seed=seed)\n",
"n_tune_data = 20\n",
"tune_data = [\n",
" {\n",
" \"problem\": train_data[x][\"problem\"],\n",
" \"solution\": train_data[x][\"solution\"],\n",
" }\n",
" for x in range(len(train_data)) if train_data[x][\"level\"] == \"Level 2\" and train_data[x][\"type\"] == \"Algebra\"\n",
"][:n_tune_data]\n",
"test_data = [\n",
" {\n",
" \"problem\": test_data[x][\"problem\"],\n",
" \"solution\": test_data[x][\"solution\"],\n",
" }\n",
" for x in range(len(test_data)) if test_data[x][\"level\"] == \"Level 2\" and test_data[x][\"type\"] == \"Algebra\"\n",
"]\n",
"print(len(tune_data), len(test_data))\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Check a tuning example:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:54.607152Z",
"iopub.status.busy": "2023-02-13T23:40:54.606441Z",
"iopub.status.idle": "2023-02-13T23:40:54.610504Z",
"shell.execute_reply": "2023-02-13T23:40:54.609759Z"
},
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Rationalize the denominator of $\\displaystyle\\frac{21}{\\sqrt{21}}$.\n"
]
}
],
"source": [
"print(tune_data[1][\"problem\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is one example of the canonical solution:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:54.613590Z",
"iopub.status.busy": "2023-02-13T23:40:54.613168Z",
"iopub.status.idle": "2023-02-13T23:40:54.616873Z",
"shell.execute_reply": "2023-02-13T23:40:54.616193Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"$\\dfrac{21}{\\sqrt{21}} = \\dfrac{21}{\\sqrt{21}} \\cdot \\dfrac{\\sqrt{21}}{\\sqrt{21}} = \\dfrac{21\\sqrt{21}}{21} = \\boxed{\\!\\sqrt{21}}$.\n"
]
}
],
"source": [
"print(tune_data[1][\"solution\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define Success Metric\n",
"\n",
"Before we start tuning, we need to define the success metric we want to opotimize. For each math task, we use voting to select a response with the most common answers out of all the generated responses. If it has an equivalent answer to the canonical solution, we consider the task as successfully solved. Then we can optimize the mean success rate of a collection of tasks."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:54.626998Z",
"iopub.status.busy": "2023-02-13T23:40:54.626593Z",
"iopub.status.idle": "2023-02-13T23:40:54.631383Z",
"shell.execute_reply": "2023-02-13T23:40:54.630770Z"
}
},
"outputs": [],
"source": [
"from typing import Optional\n",
"\n",
"def remove_boxed(string: str) -> Optional[str]:\n",
" \"\"\"Source: https://github.com/hendrycks/math\n",
" Extract the text within a \\\\boxed{...} environment.\n",
" Example:\n",
" >>> remove_boxed(\\\\boxed{\\\\frac{2}{3}})\n",
" \\\\frac{2}{3}\n",
" \"\"\"\n",
" left = \"\\\\boxed{\"\n",
" try:\n",
" assert string[: len(left)] == left\n",
" assert string[-1] == \"}\"\n",
" return string[len(left) : -1]\n",
" except Exception:\n",
" return None\n",
"\n",
"\n",
"def last_boxed_only_string(string: str) -> Optional[str]:\n",
" \"\"\"Source: https://github.com/hendrycks/math\n",
" Extract the last \\\\boxed{...} or \\\\fbox{...} element from a string.\n",
" \"\"\"\n",
" idx = string.rfind(\"\\\\boxed\")\n",
" if idx < 0:\n",
" idx = string.rfind(\"\\\\fbox\")\n",
" if idx < 0:\n",
" return None\n",
"\n",
" i = idx\n",
" right_brace_idx = None\n",
" num_left_braces_open = 0\n",
" while i < len(string):\n",
" if string[i] == \"{\":\n",
" num_left_braces_open += 1\n",
" if string[i] == \"}\":\n",
" num_left_braces_open -= 1\n",
" if num_left_braces_open == 0:\n",
" right_brace_idx = i\n",
" break\n",
" i += 1\n",
"\n",
" if right_brace_idx is None:\n",
" retval = None\n",
" else:\n",
" retval = string[idx : right_brace_idx + 1]\n",
"\n",
" return retval\n",
"\n",
"\n",
"def _fix_fracs(string: str) -> str:\n",
" \"\"\"Source: https://github.com/hendrycks/math\n",
" Reformat fractions.\n",
" Examples:\n",
" >>> _fix_fracs(\"\\\\frac1b\")\n",
" \\frac{1}{b}\n",
" >>> _fix_fracs(\"\\\\frac12\")\n",
" \\frac{1}{2}\n",
" >>> _fix_fracs(\"\\\\frac1{72}\")\n",
" \\frac{1}{72}\n",
" \"\"\"\n",
" substrs = string.split(\"\\\\frac\")\n",
" new_str = substrs[0]\n",
" if len(substrs) > 1:\n",
" substrs = substrs[1:]\n",
" for substr in substrs:\n",
" new_str += \"\\\\frac\"\n",
" if substr[0] == \"{\":\n",
" new_str += substr\n",
" else:\n",
" try:\n",
" assert len(substr) >= 2\n",
" except Exception:\n",
" return string\n",
" a = substr[0]\n",
" b = substr[1]\n",
" if b != \"{\":\n",
" if len(substr) > 2:\n",
" post_substr = substr[2:]\n",
" new_str += \"{\" + a + \"}{\" + b + \"}\" + post_substr\n",
" else:\n",
" new_str += \"{\" + a + \"}{\" + b + \"}\"\n",
" else:\n",
" if len(substr) > 2:\n",
" post_substr = substr[2:]\n",
" new_str += \"{\" + a + \"}\" + b + post_substr\n",
" else:\n",
" new_str += \"{\" + a + \"}\" + b\n",
" string = new_str\n",
" return string\n",
"\n",
"\n",
"def _fix_a_slash_b(string: str) -> str:\n",
" \"\"\"Source: https://github.com/hendrycks/math\n",
" Reformat fractions formatted as a/b to \\\\frac{a}{b}.\n",
" Example:\n",
" >>> _fix_a_slash_b(\"2/3\")\n",
" \\frac{2}{3}\n",
" \"\"\"\n",
" if len(string.split(\"/\")) != 2:\n",
" return string\n",
" a_str = string.split(\"/\")[0]\n",
" b_str = string.split(\"/\")[1]\n",
" try:\n",
" a = int(a_str)\n",
" b = int(b_str)\n",
" assert string == \"{}/{}\".format(a, b)\n",
" new_string = \"\\\\frac{\" + str(a) + \"}{\" + str(b) + \"}\"\n",
" return new_string\n",
" except Exception:\n",
" return string\n",
"\n",
"\n",
"def _remove_right_units(string: str) -> str:\n",
" \"\"\"Source: https://github.com/hendrycks/math\n",
" Remove units (on the right).\n",
" \"\\\\text{ \" only ever occurs (at least in the val set) when describing units.\n",
" \"\"\"\n",
" if \"\\\\text{ \" in string:\n",
" splits = string.split(\"\\\\text{ \")\n",
" assert len(splits) == 2\n",
" return splits[0]\n",
" else:\n",
" return string\n",
"\n",
"\n",
"def _fix_sqrt(string: str) -> str:\n",
" \"\"\"Source: https://github.com/hendrycks/math\n",
" Reformat square roots.\n",
" Example:\n",
" >>> _fix_sqrt(\"\\\\sqrt3\")\n",
" \\sqrt{3}\n",
" \"\"\"\n",
" if \"\\\\sqrt\" not in string:\n",
" return string\n",
" splits = string.split(\"\\\\sqrt\")\n",
" new_string = splits[0]\n",
" for split in splits[1:]:\n",
" if split[0] != \"{\":\n",
" a = split[0]\n",
" new_substr = \"\\\\sqrt{\" + a + \"}\" + split[1:]\n",
" else:\n",
" new_substr = \"\\\\sqrt\" + split\n",
" new_string += new_substr\n",
" return new_string\n",
"\n",
"\n",
"def _strip_string(string: str) -> str:\n",
" \"\"\"Source: https://github.com/hendrycks/math\n",
" Apply the reformatting helper functions above.\n",
" \"\"\"\n",
" # linebreaks\n",
" string = string.replace(\"\\n\", \"\")\n",
" # print(string)\n",
"\n",
" # remove inverse spaces\n",
" string = string.replace(\"\\\\!\", \"\")\n",
" # print(string)\n",
"\n",
" # replace \\\\ with \\\n",
" string = string.replace(\"\\\\\\\\\", \"\\\\\")\n",
" # print(string)\n",
"\n",
" # replace tfrac and dfrac with frac\n",
" string = string.replace(\"tfrac\", \"frac\")\n",
" string = string.replace(\"dfrac\", \"frac\")\n",
" # print(string)\n",
"\n",
" # remove \\left and \\right\n",
" string = string.replace(\"\\\\left\", \"\")\n",
" string = string.replace(\"\\\\right\", \"\")\n",
" # print(string)\n",
"\n",
" # Remove circ (degrees)\n",
" string = string.replace(\"^{\\\\circ}\", \"\")\n",
" string = string.replace(\"^\\\\circ\", \"\")\n",
"\n",
" # remove dollar signs\n",
" string = string.replace(\"\\\\$\", \"\")\n",
"\n",
" # remove units (on the right)\n",
" string = _remove_right_units(string)\n",
"\n",
" # remove percentage\n",
" string = string.replace(\"\\\\%\", \"\")\n",
" string = string.replace(\"\\%\", \"\")\n",
"\n",
" # \" 0.\" equivalent to \" .\" and \"{0.\" equivalent to \"{.\" Alternatively, add \"0\" if \".\" is the start of the string\n",
" string = string.replace(\" .\", \" 0.\")\n",
" string = string.replace(\"{.\", \"{0.\")\n",
" # if empty, return empty string\n",
" if len(string) == 0:\n",
" return string\n",
" if string[0] == \".\":\n",
" string = \"0\" + string\n",
"\n",
" # to consider: get rid of e.g. \"k = \" or \"q = \" at beginning\n",
" if len(string.split(\"=\")) == 2:\n",
" if len(string.split(\"=\")[0]) <= 2:\n",
" string = string.split(\"=\")[1]\n",
"\n",
" # fix sqrt3 --> sqrt{3}\n",
" string = _fix_sqrt(string)\n",
"\n",
" # remove spaces\n",
" string = string.replace(\" \", \"\")\n",
"\n",
" # \\frac1b or \\frac12 --> \\frac{1}{b} and \\frac{1}{2}, etc.\n",
" # Even works with \\frac1{72} (but not \\frac{72}1).\n",
" # Also does a/b --> \\\\frac{a}{b}\n",
" string = _fix_fracs(string)\n",
"\n",
" # manually change 0.5 --> \\frac{1}{2}\n",
" if string == \"0.5\":\n",
" string = \"\\\\frac{1}{2}\"\n",
"\n",
" # NOTE: X/Y changed to \\frac{X}{Y} in dataset, but in simple cases fix in case the model output is X/Y\n",
" string = _fix_a_slash_b(string)\n",
"\n",
" return string\n",
"\n",
"\n",
"def get_answer(solution: Optional[str]) -> Optional[str]:\n",
" if solution is None:\n",
" return None\n",
" last_boxed = last_boxed_only_string(solution)\n",
" if last_boxed is None:\n",
" return None\n",
" answer = remove_boxed(last_boxed)\n",
" if answer is None:\n",
" return None\n",
" return answer\n",
"\n",
"\n",
"def is_equiv(str1: Optional[str], str2: Optional[str]) -> float:\n",
" \"\"\"Returns (as a float) whether two strings containing math are equivalent up to differences of formatting in\n",
" - units\n",
" - fractions\n",
" - square roots\n",
" - superfluous LaTeX.\n",
" Source: https://github.com/hendrycks/math\n",
" \"\"\"\n",
" if str1 is None and str2 is None:\n",
" print(\"WARNING: Both None\")\n",
" return 1.0\n",
" if str1 is None or str2 is None:\n",
" return 0.0\n",
"\n",
" try:\n",
" ss1 = _strip_string(str1)\n",
" ss2 = _strip_string(str2)\n",
" return float(ss1 == ss2)\n",
" except Exception:\n",
" return float(str1 == str2)\n",
"\n",
"\n",
"def is_equiv_chain_of_thought(str1: str, str2: str) -> float:\n",
" \"\"\"Strips the solution first before calling `is_equiv`.\"\"\"\n",
" ans1 = get_answer(str1)\n",
" ans2 = get_answer(str2)\n",
"\n",
" return is_equiv(ans1, ans2)\n",
"\n",
"\n",
"def success_metrics(responses, solution, **args):\n",
" \"\"\"Check if each response is correct.\n",
" \n",
" Args:\n",
" responses (list): The list of responses.\n",
" solution (str): The canonical solution.\n",
" \n",
" Returns:\n",
" dict: The success metrics.\n",
" \"\"\"\n",
" success_list = []\n",
" n = len(responses)\n",
" for i in range(n):\n",
" response = responses[i]\n",
" succeed = is_equiv_chain_of_thought(response, solution)\n",
" success_list.append(succeed)\n",
" # voting\n",
" answers = {}\n",
" for i in range(n):\n",
" equiv = i\n",
" if get_answer(responses[i]) is None:\n",
" # ignore None answers\n",
" continue\n",
" for j in answers:\n",
" if is_equiv_chain_of_thought(responses[i], responses[j]):\n",
" equiv = j\n",
" break\n",
" if equiv in answers:\n",
" answers[equiv] += 1\n",
" else:\n",
" answers[equiv] = 1\n",
" # find the answer with highest votes in answers\n",
" answer = max(answers.items(), key=lambda x: x[1], default=(0, 0))[0]\n",
" # check if the answer is correct\n",
" success_vote = is_equiv_chain_of_thought(responses[answer], solution)\n",
" return {\n",
" \"expected_success\": 1 - pow(1 - sum(success_list) / n, n),\n",
" \"success\": any(s for s in success_list),\n",
" \"success_vote\": success_vote,\n",
" \"voted_answer\": responses[answer],\n",
" }\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Use the tuning data to find a good configuration\n",
"\n",
"### Import the oai and tune subpackages from flaml.\n",
"\n",
"FLAML has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `oai.ChatCompletion.tune` and to make a request with the tuned config: `oai.ChatCompletion.create`. First, we import oai from flaml:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:54.634335Z",
"iopub.status.busy": "2023-02-13T23:40:54.633929Z",
"iopub.status.idle": "2023-02-13T23:40:56.105700Z",
"shell.execute_reply": "2023-02-13T23:40:56.105085Z"
},
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"from flaml import oai"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"For (local) reproducibility and cost efficiency, we cache responses from OpenAI."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:56.109177Z",
"iopub.status.busy": "2023-02-13T23:40:56.108624Z",
"iopub.status.idle": "2023-02-13T23:40:56.112651Z",
"shell.execute_reply": "2023-02-13T23:40:56.112076Z"
},
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"oai.ChatCompletion.set_cache(seed)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This will create a disk cache in \".cache/{seed}\". You can change `cache_path` in `set_cache()`. The cache for different seeds are stored separately.\n",
"\n",
"### Perform tuning\n",
"\n",
"The tuning will take a while to finish, depending on the optimization budget. The tuning will be performed under the specified optimization budgets.\n",
"\n",
"* `inference_budget` is the target average inference budget per instance in the benchmark. For example, 0.004 means the target inference budget is 0.004 dollars, which translates to 2000 tokens (input + output combined) if the gpt-3.5-turbo model is used.\n",
"* `optimization_budget` is the total budget allowed to perform the tuning. For example, 1 means 1 dollars are allowed in total, which translates to 500K tokens for the gpt-3.5-turbo model.\n",
"* `num_sumples` is the number of different hyperparameter configurations which is allowed to try. The tuning will stop after either num_samples trials or after optimization_budget dollars spent, whichever happens first. -1 means no hard restriction in the number of trials and the actual number is decided by `optimization_budget`.\n",
"\n",
"Users can specify tuning data, optimization metric, optimization mode, evaluation function, search spaces etc.. The default search space is:\n",
"\n",
"```python\n",
"default_search_space = {\n",
" \"model\": tune.choice([\n",
" \"gpt-3.5-turbo\",\n",
" \"gpt-4\",\n",
" ]),\n",
" \"temperature_or_top_p\": tune.choice(\n",
" [\n",
" {\"temperature\": tune.uniform(0, 1)},\n",
" {\"top_p\": tune.uniform(0, 1)},\n",
" ]\n",
" ),\n",
" \"max_tokens\": tune.lograndint(50, 1000),\n",
" \"n\": tune.randint(1, 100),\n",
" \"prompt\": \"{prompt}\",\n",
"}\n",
"```\n",
"\n",
"The default search space can be overridden by users' input.\n",
"For example, the following code specifies a fixed prompt template. For hyperparameters which don't appear in users' input, the default search space will be used."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:40:56.115383Z",
"iopub.status.busy": "2023-02-13T23:40:56.114975Z",
"iopub.status.idle": "2023-02-13T23:41:55.045654Z",
"shell.execute_reply": "2023-02-13T23:41:55.044973Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32m[I 2023-03-26 04:03:37,074]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n",
"\u001b[32m[I 2023-03-26 04:03:37,077]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[flaml.tune.tune: 03-26 04:03:37] {811} INFO - trial 1 config: {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 0}\n",
"[flaml.tune.tune: 03-26 04:03:37] {215} INFO - result: {'expected_success': 0.9, 'success': 0.9, 'success_vote': 0.9, 'voted_answer': 'We use the distance formula to find the distance between the two points: $\\\\sqrt{(3-0)^2+(0-4)^2}=\\\\sqrt{3^2+(-4)^2}=\\\\sqrt{9+16}=\\\\sqrt{25}=\\\\boxed{5}$.', 'total_cost': 0.13772999999999996, 'cost': 0.13772999999999996, 'inference_cost': 0.0068864999999999985, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 0}, 'config/model': 'gpt-4', 'config/temperature_or_top_p': {'temperature': 0.36865945026811975}, 'config/max_tokens': 347, 'config/n': 1, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.004978179931640625}\n",
"[flaml.tune.tune: 03-26 04:03:37] {811} INFO - trial 2 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 0}\n",
"[flaml.tune.tune: 03-26 04:03:37] {215} INFO - result: {'expected_success': 0.8, 'success': 0.8, 'success_vote': 0.8, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}$$ Letting $(x_1,y_1)=(0,4)$ and $(x_2,y_2)=(3,0)$, we have: $$\\\\sqrt{(3-0)^2+(0-4)^2}=\\\\sqrt{9+16}=\\\\sqrt{25}=5$$ Therefore, the distance between the points $(0,4)$ and $(3,0)$ is $\\\\boxed{5}$.', 'total_cost': 0.145722, 'cost': 0.007992, 'inference_cost': 0.00039759999999999996, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.36865945026811975}, 'config/max_tokens': 347, 'config/n': 1, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.0047664642333984375}\n",
"[flaml.tune.tune: 03-26 04:03:37] {811} INFO - trial 3 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.4985070123025904}, 'max_tokens': 97, 'n': 20, 'prompt': 0}\n",
"[flaml.tune.tune: 03-26 04:03:37] {215} INFO - result: {'expected_success': 0.5140933870421127, 'success': 0.55, 'success_vote': 0.5, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(3-0)^2+(0-4)^2}=\\\\sqrt{9+16}=\\\\sqrt{25}=5.$$ Therefore, the distance between the points (0,4) and (3,0) is $\\\\boxed{5}$.', 'total_cost': 0.21644799999999997, 'cost': 0.07072600000000001, 'inference_cost': 0.0035343, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.4985070123025904}, 'max_tokens': 97, 'n': 20, 'prompt': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'top_p': 0.4985070123025904}, 'config/max_tokens': 97, 'config/n': 20, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.010622501373291016}\n",
"[flaml.tune.tune: 03-26 04:03:37] {811} INFO - trial 4 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.9533933461949365}, 'max_tokens': 50, 'n': 51, 'prompt': 0}\n",
"[flaml.tune.tune: 03-26 04:03:37] {215} INFO - result: {'expected_success': 0.3386014997741698, 'success': 0.4, 'success_vote': 0.35, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(3-0)^2+(0-4)^2}=\\\\sqrt{9+16}=\\\\sqrt{25}=\\\\boxed{5}.$$', 'total_cost': 0.3192479999999999, 'cost': 0.10279999999999999, 'inference_cost': 0.005138, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.9533933461949365}, 'max_tokens': 50, 'n': 51, 'prompt': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'top_p': 0.9533933461949365}, 'config/max_tokens': 50, 'config/n': 51, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.015543699264526367}\n",
"[flaml.tune.tune: 03-26 04:03:37] {811} INFO - trial 5 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9177741225129434}, 'max_tokens': 424, 'n': 54, 'prompt': 0}\n",
"[flaml.tune.tune: 03-26 04:03:37] {215} INFO - result: {'expected_success': 0.9999998720099207, 'success': 1.0, 'success_vote': 0.95, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2},$$ where $(x_1,y_1)$ and $(x_2,y_2)$ are the given points. Plugging in the values $(x_1,y_1)=(0,4)$ and $(x_2,y_2)=(3,0),$ we have: $$\\\\sqrt{(3-0)^2+(0-4)^2} = \\\\sqrt{9+16} = \\\\sqrt{25}.$$ Therefore, the distance between the two points is $\\\\boxed{5}$.', 'total_cost': 0.6322379999999999, 'cost': 0.31299, 'inference_cost': 0.015323400000000001, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9177741225129434}, 'max_tokens': 424, 'n': 54, 'prompt': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.9177741225129434}, 'config/max_tokens': 424, 'config/n': 54, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.05237627029418945}\n",
"[flaml.tune.tune: 03-26 04:03:37] {811} INFO - trial 6 config: {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.4340139933332937}, 'max_tokens': 317, 'n': 51, 'prompt': 0}\n",
"[flaml.tune.tune: 03-26 04:03:37] {215} INFO - result: {'success_vote': 0, 'total_cost': 0.7246679999999999, 'cost': 0.09243, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.4340139933332937}, 'max_tokens': 317, 'n': 51, 'prompt': 0}, 'config/model': 'gpt-4', 'config/temperature_or_top_p': {'temperature': 0.4340139933332937}, 'config/max_tokens': 317, 'config/n': 51, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.001924753189086914}\n",
"[flaml.tune.tune: 03-26 04:03:37] {811} INFO - trial 7 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9086488808086682}, 'max_tokens': 129, 'n': 9, 'prompt': 0}\n",
"[flaml.tune.tune: 03-26 04:03:37] {215} INFO - result: {'expected_success': 0.7572581384563789, 'success': 0.8, 'success_vote': 0.8, 'voted_answer': 'We use the distance formula: \\\\begin{align*}\\n\\\\text{distance}&=\\\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}\\\\\\\\\\n&=\\\\sqrt{(3-0)^2+(0-4)^2}\\\\\\\\\\n&=\\\\sqrt{9+16}\\\\\\\\\\n&=\\\\sqrt{25}\\\\\\\\\\n&=\\\\boxed{5}.\\n\\\\end{align*}', 'total_cost': 0.7647499999999999, 'cost': 0.04008199999999999, 'inference_cost': 0.0020021, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9086488808086682}, 'max_tokens': 129, 'n': 9, 'prompt': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.9086488808086682}, 'config/max_tokens': 129, 'config/n': 9, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.007839441299438477}\n",
"[flaml.tune.tune: 03-26 04:03:37] {811} INFO - trial 8 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.6262871483113925}, 'max_tokens': 257, 'n': 82, 'prompt': 0}\n",
"[flaml.tune.tune: 03-26 04:03:37] {215} INFO - result: {'success_vote': 0, 'total_cost': 1.0214359999999998, 'cost': 0.25668599999999997, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.6262871483113925}, 'max_tokens': 257, 'n': 82, 'prompt': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.6262871483113925}, 'config/max_tokens': 257, 'config/n': 82, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.009511232376098633}\n",
"[flaml.tune.tune: 03-26 04:03:37] {834} WARNING - fail to sample a trial for 100 times in a row, stopping.\n"
]
}
],
"source": [
"import logging\n",
"\n",
"prompts = [\"{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\\\boxed{{}}.\"]\n",
"config, analysis = oai.ChatCompletion.tune(\n",
" data=tune_data, # the data for tuning\n",
" metric=\"success_vote\", # the metric to optimize\n",
" mode=\"max\", # the optimization mode\n",
" eval_func=success_metrics, # the evaluation function to return the success metrics\n",
" # log_file_name=\"logs/math.log\", # the log file name\n",
" inference_budget=0.03, # the inference budget (dollar)\n",
" optimization_budget=1, # the optimization budget (dollar)\n",
" # num_samples can further limit the number of trials for different hyperparameter configurations;\n",
" # -1 means decided by the optimization budget only\n",
" num_samples=-1,\n",
" # model=\"chatgpt-35-turbo-0301\", # uncomment if using Azure OpenAI\n",
Adding a test function for OpenAI completion in flaml (#951) * improve max_valid_n and doc * Update README.md Co-authored-by: Li Jiang <lijiang1@microsoft.com> * add support for chatgpt * notebook * newline at end of file * chatgpt notebook * ChatGPT in Azure * doc * math * warning, timeout, log file name * handle import error * doc update; default value * paper * doc * docstr * eval_func * add a test func in completion * update notebook * update math notebook * improve notebok * lint and handle exception * flake8 * exception in test * add agg_method * NameError * refactor * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * add example * merge files from oai_eval_test * Revert "merge files from oai_eval_test" This reverts commit 1e6a550f913bb94df6e9680934ccb7175d00702e. * merge * save results to notebook_output * update version and cache * update doc * save nb cell results to file * fix typo in model name * code improvements * improve docstr * docstr * docstr on the Returns of test --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Susan Xueqing Liu <liususan091219@users.noreply.github.com>
2023-04-02 12:14:11 -04:00
" # model=\"gpt-3.5-turbo\", # uncomment if you don't have access to gpt-4\n",
gpt-4 support; openai workflow fix; model str; timeout; voting (#958) * workflow; model str; timeout * voting * notebook * pull request * recover workflow * voted answer * aoai * ignore None answer * default config * note * gpt-4 * n=5 * cleanup * config name * introduction * readme * avoid None * add output/ to gitignore * openai version * invalid var * comment long running cells
2023-03-26 10:13:06 -07:00
" prompt=prompts, # the prompt templates to choose from\n",
" # stop=\"###\", # the stop sequence\n",
" logging_level=logging.INFO, # the logging level\n",
")\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Output tuning results\n",
"\n",
"After the tuning, we can print out the config and the result found by FLAML:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:41:55.049204Z",
"iopub.status.busy": "2023-02-13T23:41:55.048871Z",
"iopub.status.idle": "2023-02-13T23:41:55.053284Z",
"shell.execute_reply": "2023-02-13T23:41:55.052574Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"optimized config {'model': 'gpt-3.5-turbo', 'max_tokens': 424, 'n': 54, 'prompt': '{problem} Solve the problem carefully. Simplify your answer as much as possible. Put the final answer in \\\\boxed{{}}.', 'stop': None, 'temperature': 0.9177741225129434}\n",
"best result on tuning data {'expected_success': 0.9999998720099207, 'success': 1.0, 'success_vote': 0.95, 'voted_answer': 'We use the distance formula: $$\\\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2},$$ where $(x_1,y_1)$ and $(x_2,y_2)$ are the given points. Plugging in the values $(x_1,y_1)=(0,4)$ and $(x_2,y_2)=(3,0),$ we have: $$\\\\sqrt{(3-0)^2+(0-4)^2} = \\\\sqrt{9+16} = \\\\sqrt{25}.$$ Therefore, the distance between the two points is $\\\\boxed{5}$.', 'total_cost': 0.6322379999999999, 'cost': 0.31299, 'inference_cost': 0.015323400000000001, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9177741225129434}, 'max_tokens': 424, 'n': 54, 'prompt': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.9177741225129434}, 'config/max_tokens': 424, 'config/n': 54, 'config/prompt': 0, 'experiment_tag': 'exp', 'time_total_s': 0.05237627029418945}\n"
]
}
],
"source": [
"print(\"optimized config\", config)\n",
"print(\"best result on tuning data\", analysis.best_result)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Make a request with the tuned config\n",
"\n",
"We can apply the tuned config on the request for an example task:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:41:55.056205Z",
"iopub.status.busy": "2023-02-13T23:41:55.055631Z",
"iopub.status.idle": "2023-02-13T23:41:56.039259Z",
"shell.execute_reply": "2023-02-13T23:41:56.038427Z"
},
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"choices\": [\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 0,\n",
" \"message\": {\n",
" \"content\": \"We want to get rid of the square root in the denominator. We can do this by multiplying both the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$ Thus, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 1,\n",
" \"message\": {\n",
" \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}.$$Therefore, the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 2,\n",
" \"message\": {\n",
" \"content\": \"Using the definition of square roots, we see that $\\\\sqrt{21}\\\\cdot\\\\sqrt{21}=21$. Therefore, we can write $\\\\frac{21}{\\\\sqrt{21}}$ as $\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 3,\n",
" \"message\": {\n",
" \"content\": \"We start by multiplying both the numerator and the denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying the fraction, we get: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 4,\n",
" \"message\": {\n",
" \"content\": \"We can simplify $\\\\sqrt{21}$ by finding its prime factorization: $21=3\\\\cdot7$, so $\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, \\\\[\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.\\\\] To rationalize this denominator, we need to multiply both the numerator and denominator by $\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$, which gives \\\\[\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}=\\\\frac{21\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}\\\\] \\\\[=\\\\boxed{3\\\\sqrt{3}}.\\\\]\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 5,\n",
" \"message\": {\n",
" \"content\": \"We can start by simplifying the denominator. Since $\\\\sqrt{21}$ equals $\\\\sqrt{3} \\\\cdot \\\\sqrt{7}$, we can write:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7}}$$\\n\\nTo rationalize the denominator, we need to multiply both the numerator and denominator by $\\\\sqrt{3} \\\\cdot \\\\sqrt{7}$:\\n\\n\\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7}} \\\\cdot \\\\frac{\\\\sqrt{3} \\\\cdot \\\\sqrt{7}}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7}} &= \\\\frac{21 \\\\cdot \\\\sqrt{3} \\\\cdot \\\\sqrt{7}}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7} \\\\cdot \\\\sqrt{3} \\\\cdot \\\\sqrt{7}} \\\\\\\\\\n&= \\\\frac{21 \\\\cdot \\\\sqrt{3} \\\\cdot \\\\sqrt{7}}{3 \\\\cdot 7} \\\\\\\\\\n&= \\\\boxed{\\\\frac{3 \\\\sqrt{21}}{7}}\\n\\\\end{align*}\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 6,\n",
" \"message\": {\n",
" \"content\": \"We have $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}$. Therefore, $\\\\boxed{\\\\sqrt{21}}$ is our final answer.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 7,\n",
" \"message\": {\n",
" \"content\": \"We have $\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}$. Thus, our final answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 8,\n",
" \"message\": {\n",
" \"content\": \"We can begin by multiplying both the numerator and denominator of the fraction by $\\\\sqrt{21}$, since $\\\\sqrt{21}/\\\\sqrt{21} = 1$:\\n\\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}}&=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}\\\\\\\\\\n&=\\\\frac{21\\\\cdot\\\\sqrt{21}}{21}\\\\\\\\\\n&=\\\\sqrt{21}.\\n\\\\end{align*}\\nTherefore, the simplified expression is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 9,\n",
" \"message\": {\n",
" \"content\": \"We start by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{(\\\\sqrt{21})^2} = \\\\frac{21\\\\sqrt{21}}{21}$$ Simplifying, we get: $$\\\\frac{21\\\\sqrt{21}}{21}= \\\\sqrt{21}$$ Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 10,\n",
" \"message\": {\n",
" \"content\": \"We can simplify the denominator by rationalizing it, which means getting rid of the square root in the denominator. To do this, we can multiply both the numerator and denominator by $\\\\sqrt{21}$:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}$$\\n\\nTherefore, $\\\\frac{21}{\\\\sqrt{21}} = \\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 11,\n",
" \"message\": {\n",
" \"content\": \"We start by rationalizing the denominator. We want to get rid of the square root in the denominator, so we need to multiply both the numerator and denominator by something that will give us a perfect square in the denominator. We notice that $\\\\sqrt{21} = \\\\sqrt{3 \\\\cdot 7}$, so we can multiply both the numerator and denominator by $\\\\sqrt{3}$ to get:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{3}}{\\\\sqrt{3}} = \\\\frac{21\\\\sqrt{3}}{\\\\sqrt{21}\\\\sqrt{3}} = \\\\frac{21\\\\sqrt{3}}{\\\\sqrt{3 \\\\cdot 21}}$$\\n\\nNow we can simplify the denominator:\\n\\n$$\\\\frac{21\\\\sqrt{3}}{\\\\sqrt{3 \\\\cdot 21}} = \\\\frac{21\\\\sqrt{3}}{\\\\sqrt{3} \\\\cdot \\\\sqrt{21}} = \\\\frac{21\\\\sqrt{3}}{\\\\sqrt{3} \\\\cdot \\\\sqrt{3 \\\\cdot 7}} = \\\\frac{21\\\\sqrt{3}}{\\\\sqrt{3^2 \\\\cdot 7}} = \\\\boxed{\\\\frac{3\\\\sqrt{7}}{2}}$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 12,\n",
" \"message\": {\n",
" \"content\": \"We can simplify $\\\\sqrt{21}$ by recognizing that $21=3\\\\cdot7$ and $\\\\sqrt{3^2\\\\cdot7}=3\\\\sqrt{7}$. Therefore, we have $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3^2\\\\cdot7}} = \\\\frac{21}{3\\\\sqrt{7}} = \\\\frac{7\\\\cdot3}{3\\\\cdot\\\\sqrt{7}} = \\\\frac{7}{\\\\sqrt{7}}.$$We can rationalize this by multiplying top and bottom by $\\\\sqrt{7}$, giving us $$\\\\frac{7}{\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{7}}{\\\\sqrt{7}} = \\\\frac{7\\\\sqrt{7}}{7} = \\\\boxed{\\\\sqrt{7}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 13,\n",
" \"message\": {\n",
" \"content\": \"We want to get rid of the square root sign in the denominator. One way to do this is to multiply both the numerator and denominator by $\\\\sqrt{21}$. We get \\\\[\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21\\\\cdot\\\\sqrt{21}}{(\\\\sqrt{21})\\\\cdot(\\\\sqrt{21})} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\boxed{\\\\sqrt{21}}.\\\\]Note that we simplified by dividing $21$ by $21$ to get $1$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 14,\n",
" \"message\": {\n",
" \"content\": \"To rationalize the denominator, we need to eliminate the square root. We can do this by multiplying both the numerator and denominator by $\\\\sqrt{21}$:$$\\\\dfrac{21}{\\\\sqrt{21}}\\\\cdot\\\\dfrac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\dfrac{21\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 15,\n",
" \"message\": {\n",
" \"content\": \"To rationalize the denominator, we need to eliminate the radical from the denominator. We can do this by multiplying both the numerator and denominator by $\\\\sqrt{21}$. \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}&=\\\\frac{21\\\\sqrt{21}}{21}\\\\\\\\\\n&=\\\\frac{\\\\cancelto{1}{21}\\\\cdot \\\\cancel{\\\\sqrt{21}}\\\\cdot \\\\sqrt{21}}{\\\\cancel{21}}\\\\\\\\\\n&=\\\\sqrt{21}\\n\\\\end{align*}Therefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 16,\n",
" \"message\": {\n",
" \"content\": \"We begin by multiplying the top and bottom of the fraction by $\\\\sqrt{21}$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} &= \\\\frac{21\\\\sqrt{21}}{21}\\\\\\\\\\n&= \\\\frac{\\\\cancel{21}\\\\cdot\\\\sqrt{21}}{\\\\cancel{21}}\\\\\\\\\\n&= \\\\sqrt{\\\\boxed{21}}.\\n\\\\end{align*}\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 17,\n",
" \"message\": {\n",
" \"content\": \"We rationalize the denominator as follows: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21\\\\cdot \\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}\\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21}\\\\\\\\\\n&= \\\\boxed{\\\\sqrt{21}}.\\n\\\\end{align*}\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 18,\n",
" \"message\": {\n",
" \"content\": \"We can begin by simplifying the square root in the denominator. The largest perfect square that divides into 21 is 3, so we can break apart the square root as follows: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3\\\\cdot7}} = \\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}$$ Now we can use the property that $\\\\frac{1}{\\\\sqrt{x}} = \\\\frac{\\\\sqrt{x}}{x}$. Applying this to our expression, we have: $$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}} = \\\\frac{21\\\\cdot\\\\frac{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}\\\\cdot\\\\frac{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}} = \\\\frac{21\\\\sqrt{21}}{3\\\\cdot7} = \\\\frac{\\\\cancel{21}\\\\sqrt{21}}{\\\\cancel{3}\\\\cdot\\\\cancel{7}} = \\\\boxed{\\\\sqrt{21}}$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 19,\n",
" \"message\": {\n",
" \"content\": \"We start by writing $\\\\sqrt{21}$ as a product of simpler terms. Since $21=3\\\\cdot7$ and neither factor has a perfect square factor other than 1, we can write $\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, we have \\\\[\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.\\\\] To rationalize the denominator, we multiply both the numerator and denominator by $\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$: \\\\[\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}} =\\\\frac{21\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7} = \\\\boxed{\\\\frac{3\\\\sqrt{21}}{7}}.\\\\]\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 20,\n",
" \"message\": {\n",
" \"content\": \"We want to remove the radical from the denominator. To do this, we multiply the top and bottom of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{21}.$$ Simplifying the numerator, we have $21\\\\cdot\\\\sqrt{21}=21^{\\\\frac{3}{2}}$. Therefore, the simplified version of $\\\\frac{21}{\\\\sqrt{21}}$ is $\\\\boxed{21^{\\\\frac{3}{2}}}$ or $\\\\boxed{21\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 21,\n",
" \"message\": {\n",
" \"content\": \"We can simplify $\\\\frac{21}{\\\\sqrt{21}}$ by multiplying both the numerator and denominator by $\\\\sqrt{21}$:\\n\\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\cdot\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\boxed{\\\\sqrt{21}}.\\n\\\\end{align*}\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 22,\n",
" \"message\": {\n",
" \"content\": \"We can begin by simplifying the square root in the denominator. We notice that $\\\\sqrt{21}$ can be divided into $\\\\sqrt{3}$ and $\\\\sqrt{7}$. Specifically, $\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, we have \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}} \\\\\\\\\\n&= \\\\frac{21}{\\\\sqrt{3}}\\\\cdot\\\\frac{1}{\\\\sqrt{7}} \\\\\\\\\\n&= \\\\frac{21}{\\\\sqrt{3}}\\\\cdot\\\\frac{\\\\sqrt{7}}{7} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot 7} \\\\\\\\\\n&= \\\\boxed{\\\\frac{3\\\\sqrt{7}}{\\\\sqrt{3}}}.\\n\\\\end{align*}\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 23,\n",
" \"message\": {\n",
" \"content\": \"We start by noticing that $\\\\sqrt{21}$ can be simplified. The prime factorization of 21 is $3\\\\cdot7$, so $\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, we have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.$$ To rationalize the denominator, we need to get rid of the $\\\\sqrt{3}$ and $\\\\sqrt{7}$ in the denominator. We can do this by multiplying the numerator and denominator by $\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. We get $$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}=\\\\frac{21\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}=\\\\frac{21\\\\sqrt{21}}{3\\\\cdot7}=\\\\frac{3\\\\cdot7\\\\cdot\\\\sqrt{21}}{3\\\\cdot7}.$$ Cancelling the 3's and the 7's, we get $$\\\\frac{3\\\\cdot7\\\\cdot\\\\sqrt{21}}{3\\\\cdot7} = \\\\boxed{\\\\sqrt{21}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 24,\n",
" \"message\": {\n",
" \"content\": \"We want to get rid of the radical in the denominator. One way to do this is to multiply both the numerator and the denominator by the radical (or a simplified version of it). In this case, $\\\\sqrt{21}$ is already simplified, so we can just multiply top and bottom by $\\\\sqrt{21}$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}\\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\boxed{\\\\sqrt{21}}.\\n\\\\end{align*}We simplified $\\\\frac{21\\\\sqrt{21}}{21}$ by canceling the common factor of 21 and leaving only $\\\\sqrt{21}$ in the numerator.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 25,\n",
" \"message\": {\n",
" \"content\": \"We know that $\\\\sqrt{21}$ can be simplified as $\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, \\n\\n$$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.$$ \\n\\nWe can now rationalize the denominator by multiplying both the numerator and denominator by $\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$:\\n\\n$$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}=\\\\frac{21\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}=\\\\boxed{3\\\\sqrt{3}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 26,\n",
" \"message\": {\n",
" \"content\": \"Since the denominator has a radical, we want to get rid of it. We can do this by multiplying both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{21}$$\\nSimplifying, we see that the $21$s cancel: $$\\\\frac{21\\\\cdot\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 27,\n",
" \"message\": {\n",
" \"content\": \"We can begin by simplifying $\\\\sqrt{21}$ as a product of its prime factors: $\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, we have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.$$ To rationalize the denominator, we can multiply both the numerator and denominator by $\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$: $$\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}\\\\cdot\\\\frac{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}=\\\\frac{21\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}.$$ Simplifying the fraction in the numerator, we have $$\\\\frac{21\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}=\\\\frac{3\\\\cdot7\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot7}=\\\\boxed{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 28,\n",
" \"message\": {\n",
" \"content\": \"We want to get rid of the radical in the denominator. To do this, we can multiply the numerator and denominator by $\\\\sqrt{21}$, which is equal to $\\\\sqrt{21}\\\\cdot\\\\sqrt{21} = 21$: \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\sqrt{21}.\\n\\\\end{align*}Thus, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}} = \\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 29,\n",
" \"message\": {\n",
" \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 30,\n",
" \"message\": {\n",
" \"content\": \"We begin by multiplying both the numerator and denominator of the fraction $\\\\frac{21}{\\\\sqrt{21}}$ by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying the numerator by canceling $21$ from both the numerator and denominator, we get: $$\\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}.$$ Therefore, $\\\\frac{21}{\\\\sqrt{21}} = \\\\boxed{\\\\sqrt{21}}$ (after rationalizing the denominator).\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 31,\n",
" \"message\": {\n",
" \"content\": \"We start by writing $\\\\dfrac{21}{\\\\sqrt{21}}$ as $\\\\dfrac{21}{\\\\sqrt{21}} \\\\cdot \\\\dfrac{\\\\sqrt{21}}{\\\\sqrt{21}}$. This gives us \\\\begin{align*}\\n\\\\dfrac{21}{\\\\sqrt{21}} \\\\cdot \\\\dfrac{\\\\sqrt{21}}{\\\\sqrt{21}} &= \\\\dfrac{21\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot \\\\sqrt{21}} \\\\\\\\\\n&= \\\\dfrac{21\\\\sqrt{21}}{21} \\\\\\\\\\n&=\\\\boxed{\\\\sqrt{21}}.\\n\\\\end{align*}\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 32,\n",
" \"message\": {\n",
" \"content\": \"We can rationalize the denominator $\\\\sqrt{21}$ by multiplying both the numerator and denominator by $\\\\sqrt{21}$. \\n\\n\\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21 \\\\cdot \\\\sqrt{21}}{\\\\sqrt{21} \\\\cdot \\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\sqrt{21}\\n\\\\end{align*}\\n\\nTherefore, the simplified answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 33,\n",
" \"message\": {\n",
" \"content\": \"We can begin by multiplying the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$ Simplifying, we find that $\\\\frac{21\\\\sqrt{21}}{21} = \\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 34,\n",
" \"message\": {\n",
" \"content\": \"We can rationalize the denominator by multiplying the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot \\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot \\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\boxed{\\\\sqrt{21}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 35,\n",
" \"message\": {\n",
" \"content\": \"We can start by simplifying $\\\\sqrt{21}$. Since $21$ has no perfect square factors other than $1$, we have $\\\\sqrt{21} = \\\\sqrt{3\\\\cdot7}= \\\\sqrt{3}\\\\sqrt{7}$. Thus, we have:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3}\\\\sqrt{7}}$$\\n\\nTo rationalize the denominator, we want to get rid of the radical in the denominator by multiplying both the numerator and denominator by $\\\\sqrt{3}\\\\sqrt{7}$. This gives:\\n\\n$$\\\\frac{21}{\\\\sqrt{3}\\\\sqrt{7}} \\\\cdot \\\\frac{\\\\sqrt{3}\\\\sqrt{7}}{\\\\sqrt{3}\\\\sqrt{7}} = \\\\frac{21\\\\sqrt{3}\\\\sqrt{7}}{3\\\\cdot7} = \\\\frac{3\\\\sqrt{3}\\\\sqrt{7}}{1}$$\\n\\nTherefore, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}$ simplified with a rationalized denominator is $\\\\boxed{3\\\\sqrt{3}\\\\sqrt{7}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 36,\n",
" \"message\": {\n",
" \"content\": \"We start by multiplying the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}.$$ Now we can simplify by canceling the common factor of 21: $$\\\\frac{21\\\\sqrt{21}}{21} = \\\\boxed{\\\\sqrt{21}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 37,\n",
" \"message\": {\n",
" \"content\": \"We can simplify this fraction by rationalizing the denominator, which means getting rid of the square root in the denominator.\\n\\nTo do this, we can multiply both the numerator and denominator by $\\\\sqrt{21}$:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}$$\\n\\nSimplifying the fraction, we get:\\n\\n$$\\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}$$\\n\\nTherefore, $\\\\boxed{\\\\sqrt{21}}$ is the final, simplified answer.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 38,\n",
" \"message\": {\n",
" \"content\": \"We notice that $\\\\sqrt{21}=3\\\\sqrt{7}$. So, rationalizing the denominator of $\\\\frac{21}{\\\\sqrt{21}}$ is the same as multiplying it by $\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}$:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}= \\\\frac{3\\\\cdot7\\\\cdot\\\\sqrt{3\\\\cdot7}}{3\\\\cdot7} = \\\\frac{\\\\sqrt{3\\\\cdot7}}{1} = \\\\boxed{\\\\sqrt{21}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 39,\n",
" \"message\": {\n",
" \"content\": \"We start by noting that $\\\\sqrt{21} = \\\\sqrt{3\\\\cdot7} = \\\\sqrt{3}\\\\cdot\\\\sqrt{7}$. Therefore, $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}} = \\\\frac{21}{\\\\sqrt{3}}\\\\cdot\\\\frac{1}{\\\\sqrt{7}} = \\\\frac{21}{\\\\sqrt{3}}\\\\cdot\\\\frac{\\\\sqrt{7}}{7} = \\\\boxed{\\\\frac{3\\\\sqrt{7}}{\\\\sqrt{3}}}. $$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 40,\n",
" \"message\": {\n",
" \"content\": \"We can begin by simplifying the denominator. Recall that $\\\\sqrt{21}$ can be written as $21^{\\\\frac{1}{2}}$. So, we have:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{21^{\\\\frac{1}{2}}}$$\\n\\nTo rationalize the denominator, we need to get rid of the radical in the denominator. We can do this by multiplying both the numerator and the denominator by $\\\\sqrt{21}$. This gives:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21}$$\\n\\nNow we can simplify:\\n\\n$$\\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}$$\\n\\nTherefore, $$\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}} = \\\\sqrt{21}$$\\n\\nAnd our final answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 41,\n",
" \"message\": {\n",
" \"content\": \"We want to eliminate the radical in the denominator of the fraction $\\\\frac{21}{\\\\sqrt{21}}$. To do this, we can multiply both the numerator and denominator by $\\\\sqrt{21}$. This gives us \\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21 \\\\cdot \\\\sqrt{21}}{\\\\sqrt{21} \\\\cdot \\\\sqrt{21}}\\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21}\\\\\\\\\\n&= \\\\sqrt{21}.\\n\\\\end{align*} Therefore, $\\\\boxed{\\\\sqrt{21}}$ is our final answer.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 42,\n",
" \"message\": {\n",
" \"content\": \"We start by multiplying the numerator and denominator of the fraction by $\\\\sqrt{21}$:\\n\\\\begin{align*}\\n\\\\frac{21}{\\\\sqrt{21}} &= \\\\frac{21 \\\\cdot \\\\sqrt{21}}{\\\\sqrt{21} \\\\cdot \\\\sqrt{21}} \\\\\\\\\\n&= \\\\frac{21\\\\sqrt{21}}{21} \\\\\\\\\\n&= \\\\sqrt{21}\\n\\\\end{align*}\\nTherefore, $\\\\frac{21}{\\\\sqrt{21}} = \\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 43,\n",
" \"message\": {\n",
" \"content\": \"We can begin by simplifying the denominator using the property $\\\\sqrt{n^2}=n$ for any positive real number $n$. We have:\\n\\n$$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3^2\\\\cdot 7}}=\\\\frac{21}{3\\\\sqrt{7}}$$\\n\\nWe can simplify this fraction further by dividing both the numerator and denominator by the greatest common factor, which is 3. We obtain:\\n\\n$$\\\\frac{21}{3\\\\sqrt{7}}=\\\\frac{7}{\\\\sqrt{7}}=\\\\frac{7\\\\cdot\\\\sqrt{7}}{7}=\\\\boxed{\\\\sqrt{7}}$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 44,\n",
" \"message\": {\n",
" \"content\": \"We want to get rid of the square root in the denominator. To do this, we can multiply both the numerator and the denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{\\\\sqrt{21}\\\\cdot\\\\sqrt{21}}=\\\\frac{21\\\\cdot \\\\sqrt{21}}{21}.$$Simplifying the fraction, we have $\\\\frac{21}{21}=1$, so our final answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 45,\n",
" \"message\": {\n",
" \"content\": \"Since $\\\\sqrt{21}=\\\\sqrt{3\\\\cdot7}=\\\\sqrt{3}\\\\cdot\\\\sqrt{7}$, we have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}=\\\\frac{21}{\\\\sqrt{3}}\\\\cdot\\\\frac{1}{\\\\sqrt{7}}=\\\\boxed{3\\\\sqrt{7}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 46,\n",
" \"message\": {\n",
" \"content\": \"We have \\\\[\\n\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\cdot \\\\sqrt{21}}{21}=\\\\sqrt{21}.\\n\\\\] Thus, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}= \\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 47,\n",
" \"message\": {\n",
" \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\frac{\\\\cancel{21}\\\\sqrt{\\\\cancel{21}\\\\cdot3}}{\\\\cancel{21}}=\\\\boxed{\\\\sqrt{3}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 48,\n",
" \"message\": {\n",
" \"content\": \"We want to find a way to eliminate the radical in the denominator. One way to do this is to multiply the numerator and denominator by the radical, since $(\\\\sqrt{21})^2=21$. Doing this gives us: $$\\\\frac{21}{\\\\sqrt{21}} \\\\cdot \\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\frac{\\\\cancel{21}\\\\sqrt{3}\\\\cdot\\\\cancel{7}}{\\\\cancel{21}} = \\\\boxed{\\\\sqrt{3}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"length\",\n",
" \"index\": 49,\n",
" \"message\": {\n",
" \"content\": \"We begin by noticing that $\\\\sqrt{21}=21^{\\\\frac12}$, so we have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{21^{\\\\frac12}}.$$ Using the rule $a^{m}\\\\cdot a^{n}=a^{m+n}$, we can write $21=3\\\\cdot7$ as $$21^{\\\\frac12}=3^{\\\\frac12}\\\\cdot7^{\\\\frac12}.$$ Thus, we have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{21^{\\\\frac12}}=\\\\frac{21}{3^{\\\\frac12}\\\\cdot7^{\\\\frac12}}.$$ To rationalize the denominator, we multiply the top and bottom of the fraction by $3^{\\\\frac12}\\\\cdot7^{\\\\frac12}$: $$\\\\frac{21}{3^{\\\\frac12}\\\\cdot7^{\\\\frac12}}\\\\cdot\\\\frac{3^{\\\\frac12}\\\\cdot7^{\\\\frac12}}{3^{\\\\frac12}\\\\cdot7^{\\\\frac12}}=\\\\frac{21\\\\cdot3^{\\\\frac12}\\\\cdot7^{\\\\frac12}}{3\\\\cdot7}.$$ Simplifying the numerator, we have $$\\\\frac{21\\\\cdot3^{\\\\frac12}\\\\cdot7^{\\\\frac12}}{3\\\\cdot7}=\\\\frac{3^{\\\\frac12}\\\\cdot7^{\\\\frac32}}{3}.$$ Now, using the rule $a^{m}/a^{n}=a^{m-n}$, we simplify further: $$\\\\frac{3^{\\\\frac12}\\\\cdot7^{\\\\frac32}}{3}=3^{-\\\\frac12}\\\\cdot7^{\\\\frac32}=\\\\frac{7^{\\\\frac32}}{\\\\sqrt{3}}}.$$ Finally, we rationalize the denominator by multiplying top and bottom by $\\\\sqrt{3}$: $$\\\\frac{7^{\\\\frac32}}{\\\\sqrt{3}}\\\\cdot\\\\frac{\\\\sqrt{3}}{\\\\sqrt{3}}=\\\\boxed{\\\\\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 50,\n",
" \"message\": {\n",
" \"content\": \"To rationalize the denominator, we need to get rid of the square root. We can do this by multiplying both the numerator and denominator by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\sqrt{21}.$$ Therefore, the answer is $\\\\boxed{\\\\sqrt{21}}$.\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 51,\n",
" \"message\": {\n",
" \"content\": \"We start with $\\\\frac{21}{\\\\sqrt{21}}$. To rationalize the denominator, we want to get rid of the square root. We know that $\\\\sqrt{21}$ can be simplified as $\\\\sqrt{3 \\\\cdot 7} = \\\\sqrt{3} \\\\cdot \\\\sqrt{7}$. Therefore, we have \\\\[\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7}}.\\\\] We can now use the property $\\\\frac{a}{b \\\\cdot c} = \\\\frac{a}{b} \\\\cdot \\\\frac{1}{c}$ to split up the denominator: \\\\[\\\\frac{21}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7}} = \\\\frac{21}{\\\\sqrt{3}} \\\\cdot \\\\frac{1}{\\\\sqrt{7}} = \\\\frac{21\\\\sqrt{7}}{\\\\sqrt{3} \\\\cdot \\\\sqrt{7} \\\\cdot \\\\sqrt{7}} = \\\\frac{21\\\\sqrt{7}}{7\\\\sqrt{3}}.\\\\] Finally, we simplify by dividing both the numerator and the denominator by $7$: \\\\[\\\\frac{21\\\\sqrt{7}}{7\\\\sqrt{3}} = \\\\frac{3\\\\sqrt{7}}{\\\\sqrt{3}} = \\\\frac{3\\\\sqrt{7}}{\\\\sqrt{3}} \\\\cdot \\\\frac{\\\\sqrt{3}}{\\\\sqrt{3}} = \\\\boxed{3\\\\sqrt{21}}.\\\\]\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 52,\n",
" \"message\": {\n",
" \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}} = \\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}} = \\\\frac{21\\\\sqrt{21}}{21} = \\\\frac{3\\\\cdot 7\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot 7}.$$ Simplifying this, we get $$\\\\frac{3\\\\cdot 7\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{3\\\\cdot 7} = \\\\frac{\\\\cancel{3}\\\\cdot \\\\cancel{7}\\\\cdot\\\\sqrt{3}\\\\cdot\\\\sqrt{7}}{\\\\cancel{3}\\\\cdot \\\\cancel{7}} = \\\\sqrt{3}\\\\sqrt{7}.$$ Therefore, we have $$\\\\frac{21}{\\\\sqrt{21}} = \\\\boxed{\\\\sqrt{3}\\\\sqrt{7}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" },\n",
" {\n",
" \"finish_reason\": \"stop\",\n",
" \"index\": 53,\n",
" \"message\": {\n",
" \"content\": \"We have $$\\\\frac{21}{\\\\sqrt{21}}=\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\cdot\\\\sqrt{21}}{21}=\\\\frac{\\\\cancel{21}\\\\cdot\\\\sqrt{\\\\cancel{21}\\\\cdot 3}}{\\\\cancel{21}}=\\\\boxed{\\\\sqrt{3}}.$$\",\n",
" \"role\": \"assistant\"\n",
" }\n",
" }\n",
" ],\n",
" \"created\": 1679622917,\n",
" \"id\": \"chatcmpl-6xQw98DIHC3S1iQAacY3vjL6TLPRL\",\n",
" \"model\": \"gpt-3.5-turbo-0301\",\n",
" \"object\": \"chat.completion\",\n",
" \"usage\": {\n",
" \"completion_tokens\": 7762,\n",
" \"prompt_tokens\": 50,\n",
" \"total_tokens\": 7812\n",
" }\n",
"}\n",
"{'expected_success': 1.0, 'success': True, 'success_vote': 1.0, 'voted_answer': 'We want to get rid of the square root in the denominator. We can do this by multiplying both the numerator and denominator of the fraction by $\\\\sqrt{21}$: $$\\\\frac{21}{\\\\sqrt{21}}\\\\cdot\\\\frac{\\\\sqrt{21}}{\\\\sqrt{21}}=\\\\frac{21\\\\sqrt{21}}{21}=\\\\sqrt{21}.$$ Thus, $\\\\displaystyle\\\\frac{21}{\\\\sqrt{21}}=\\\\boxed{\\\\sqrt{21}}$.'}\n"
]
}
],
"source": [
"responses = oai.ChatCompletion.create(context=tune_data[1], **config)\n",
Adding a test function for OpenAI completion in flaml (#951) * improve max_valid_n and doc * Update README.md Co-authored-by: Li Jiang <lijiang1@microsoft.com> * add support for chatgpt * notebook * newline at end of file * chatgpt notebook * ChatGPT in Azure * doc * math * warning, timeout, log file name * handle import error * doc update; default value * paper * doc * docstr * eval_func * add a test func in completion * update notebook * update math notebook * improve notebok * lint and handle exception * flake8 * exception in test * add agg_method * NameError * refactor * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * add example * merge files from oai_eval_test * Revert "merge files from oai_eval_test" This reverts commit 1e6a550f913bb94df6e9680934ccb7175d00702e. * merge * save results to notebook_output * update version and cache * update doc * save nb cell results to file * fix typo in model name * code improvements * improve docstr * docstr * docstr on the Returns of test --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Susan Xueqing Liu <liususan091219@users.noreply.github.com>
2023-04-02 12:14:11 -04:00
"metric_results = success_metrics([response[\"message\"][\"content\"].rstrip() for response in responses[\"choices\"]], **tune_data[1])\n",
"print(\"response on an example data instance:\", responses)\n",
"print(\"metric_results on the example data instance:\", metric_results)\n"
gpt-4 support; openai workflow fix; model str; timeout; voting (#958) * workflow; model str; timeout * voting * notebook * pull request * recover workflow * voted answer * aoai * ignore None answer * default config * note * gpt-4 * n=5 * cleanup * config name * introduction * readme * avoid None * add output/ to gitignore * openai version * invalid var * comment long running cells
2023-03-26 10:13:06 -07:00
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Evaluate the success rate on the test data\n",
"\n",
Adding a test function for OpenAI completion in flaml (#951) * improve max_valid_n and doc * Update README.md Co-authored-by: Li Jiang <lijiang1@microsoft.com> * add support for chatgpt * notebook * newline at end of file * chatgpt notebook * ChatGPT in Azure * doc * math * warning, timeout, log file name * handle import error * doc update; default value * paper * doc * docstr * eval_func * add a test func in completion * update notebook * update math notebook * improve notebok * lint and handle exception * flake8 * exception in test * add agg_method * NameError * refactor * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * add example * merge files from oai_eval_test * Revert "merge files from oai_eval_test" This reverts commit 1e6a550f913bb94df6e9680934ccb7175d00702e. * merge * save results to notebook_output * update version and cache * update doc * save nb cell results to file * fix typo in model name * code improvements * improve docstr * docstr * docstr on the Returns of test --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Susan Xueqing Liu <liususan091219@users.noreply.github.com>
2023-04-02 12:14:11 -04:00
"You can use flaml's `oai.ChatCompletion.test` to evaluate the performance of an entire dataset with the tuned config. The following code will take a while (30 mins to 1 hour) to evaluate all the test data instances if uncommented and run. It will cost roughly $3. "
gpt-4 support; openai workflow fix; model str; timeout; voting (#958) * workflow; model str; timeout * voting * notebook * pull request * recover workflow * voted answer * aoai * ignore None answer * default config * note * gpt-4 * n=5 * cleanup * config name * introduction * readme * avoid None * add output/ to gitignore * openai version * invalid var * comment long running cells
2023-03-26 10:13:06 -07:00
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"execution": {
"iopub.execute_input": "2023-02-13T23:41:56.042764Z",
"iopub.status.busy": "2023-02-13T23:41:56.042086Z",
"iopub.status.idle": "2023-02-13T23:53:05.597643Z",
"shell.execute_reply": "2023-02-13T23:53:05.596603Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'expected_success': 0.9878128576084944, 'success': 0.9950248756218906, 'success_vote': 0.9203980099502488, 'voted_answer': \"We have that $1$ kilowatt is equivalent to $1.36$ horsepower. Therefore, we can set up the proportion $\\\\frac{1\\\\text{ kW}}{1.36\\\\text{ hp}} = \\\\frac{x\\\\text{ kW}}{500\\\\text{ hp}}$, where $x$ is the number of kilowatts that Eric's car's engine can generate. Solving for $x$, we get $x = \\\\frac{(1\\\\text{ kW})(500\\\\text{ hp})}{1.36\\\\text{ hp}} \\\\approx \\\\boxed{368 \\\\text{ kW}}$.\", 'total_cost': 4.194939999999996, 'cost': 3.1735039999999994, 'inference_cost': 0.01577204825870647}\n"
]
}
],
"source": [
Adding a test function for OpenAI completion in flaml (#951) * improve max_valid_n and doc * Update README.md Co-authored-by: Li Jiang <lijiang1@microsoft.com> * add support for chatgpt * notebook * newline at end of file * chatgpt notebook * ChatGPT in Azure * doc * math * warning, timeout, log file name * handle import error * doc update; default value * paper * doc * docstr * eval_func * add a test func in completion * update notebook * update math notebook * improve notebok * lint and handle exception * flake8 * exception in test * add agg_method * NameError * refactor * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * add example * merge files from oai_eval_test * Revert "merge files from oai_eval_test" This reverts commit 1e6a550f913bb94df6e9680934ccb7175d00702e. * merge * save results to notebook_output * update version and cache * update doc * save nb cell results to file * fix typo in model name * code improvements * improve docstr * docstr * docstr on the Returns of test --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Susan Xueqing Liu <liususan091219@users.noreply.github.com>
2023-04-02 12:14:11 -04:00
"# result = oai.Completion.test(test_data, config, success_metrics)\n",
"# print(\"performance on test data with the tuned config:\", result)"
gpt-4 support; openai workflow fix; model str; timeout; voting (#958) * workflow; model str; timeout * voting * notebook * pull request * recover workflow * voted answer * aoai * ignore None answer * default config * note * gpt-4 * n=5 * cleanup * config name * introduction * readme * avoid None * add output/ to gitignore * openai version * invalid var * comment long running cells
2023-03-26 10:13:06 -07:00
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"What about the default, untuned gpt-4 config (with the same prompt as the tuned config)? We can evaluate it and compare:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'expected_success': 0.6965174129353234, 'success': 0.6965174129353234, 'success_vote': 0.6965174129353234, 'voted_answer': \"If we let $x$ be the number of kilowatts, then we can set up the proportion $\\\\frac{x\\\\text{ kW}}{500\\\\text{ hp}}=\\\\frac{1\\\\text{ kW}}{1.36\\\\text{ hp}}$. Solving for $x$, we get $x=\\\\frac{500}{1.36} = 367.65$. Rounding to the nearest integer, we get that Eric's car's engine has $\\\\boxed{368}$ kilowatts.\", 'total_cost': 6.009489999999993, 'cost': 1.8145500000000006, 'inference_cost': 0.008809679104477611}\n"
]
}
],
"source": [
"# assuming you have access to gpt-4; otherwise use gpt-3.5-turbo\n",
"# the following code will cost roughly $2 if uncommented and run.\n",
"\n",
Adding a test function for OpenAI completion in flaml (#951) * improve max_valid_n and doc * Update README.md Co-authored-by: Li Jiang <lijiang1@microsoft.com> * add support for chatgpt * notebook * newline at end of file * chatgpt notebook * ChatGPT in Azure * doc * math * warning, timeout, log file name * handle import error * doc update; default value * paper * doc * docstr * eval_func * add a test func in completion * update notebook * update math notebook * improve notebok * lint and handle exception * flake8 * exception in test * add agg_method * NameError * refactor * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * add example * merge files from oai_eval_test * Revert "merge files from oai_eval_test" This reverts commit 1e6a550f913bb94df6e9680934ccb7175d00702e. * merge * save results to notebook_output * update version and cache * update doc * save nb cell results to file * fix typo in model name * code improvements * improve docstr * docstr * docstr on the Returns of test --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Susan Xueqing Liu <liususan091219@users.noreply.github.com>
2023-04-02 12:14:11 -04:00
"# default_config = {\"model\": 'gpt-4', \"prompt\": prompts[0]}\n",
"# default_result = oai.Completion.test(test_data, default_config, success_metrics)\n",
"# print(\"performance on test data from gpt-4 with a default config:\", default_result)"
gpt-4 support; openai workflow fix; model str; timeout; voting (#958) * workflow; model str; timeout * voting * notebook * pull request * recover workflow * voted answer * aoai * ignore None answer * default config * note * gpt-4 * n=5 * cleanup * config name * introduction * readme * avoid None * add output/ to gitignore * openai version * invalid var * comment long running cells
2023-03-26 10:13:06 -07:00
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tuned config succeeds in 92.0% test cases\n",
"untuned config succeeds in 69.7% test cases\n"
]
}
],
"source": [
"# print(\"tuned config succeeds in {:.1f}% test cases\".format(result[\"success_vote\"] * 100))\n",
"# print(\"untuned config succeeds in {:.1f}% test cases\".format(default_result[\"success_vote\"] * 100))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that the untuned config has a lower inference cost. What if we heuristically increase the number of responses n to 5?"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'expected_success': 0.9181755223880596, 'success': 0.9552238805970149, 'success_vote': 0.8756218905472637, 'voted_answer': \"To figure out how many kilowatts of power Eric's car can generate, we need to find the conversion factor for metric horsepower to kilowatts. To do this, we start by dividing the power in Eric's car in horsepower by the number of kilowatts per horsepower: $$\\\\frac{500\\\\text{ hp}}{1.36\\\\text{ hp/kW}}$$Now, to get to kilowatts, we divide by 1 hp, which gives us $$\\\\frac{500}{1.36}\\\\approx \\\\boxed{368}\\\\text{ kW}$$\", 'total_cost': 14.071600000000004, 'cost': 8.06211, 'inference_cost': 0.039892067164179104}\n"
]
}
],
"source": [
Adding a test function for OpenAI completion in flaml (#951) * improve max_valid_n and doc * Update README.md Co-authored-by: Li Jiang <lijiang1@microsoft.com> * add support for chatgpt * notebook * newline at end of file * chatgpt notebook * ChatGPT in Azure * doc * math * warning, timeout, log file name * handle import error * doc update; default value * paper * doc * docstr * eval_func * add a test func in completion * update notebook * update math notebook * improve notebok * lint and handle exception * flake8 * exception in test * add agg_method * NameError * refactor * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * add example * merge files from oai_eval_test * Revert "merge files from oai_eval_test" This reverts commit 1e6a550f913bb94df6e9680934ccb7175d00702e. * merge * save results to notebook_output * update version and cache * update doc * save nb cell results to file * fix typo in model name * code improvements * improve docstr * docstr * docstr on the Returns of test --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Susan Xueqing Liu <liususan091219@users.noreply.github.com>
2023-04-02 12:14:11 -04:00
"# config_larger = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 5}\n",
"# default_result = oai.ChatCompletion.test(test_data, config_larger, success_metrics)\n",
"# print(\"performance on test data from gpt-4 with a default config and n=5:\", default_result)"
gpt-4 support; openai workflow fix; model str; timeout; voting (#958) * workflow; model str; timeout * voting * notebook * pull request * recover workflow * voted answer * aoai * ignore None answer * default config * note * gpt-4 * n=5 * cleanup * config name * introduction * readme * avoid None * add output/ to gitignore * openai version * invalid var * comment long running cells
2023-03-26 10:13:06 -07:00
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We find that the 'success_vote' metric is increased at the cost of exceeding the inference budget. But the tuned configuration has both higher 'success_vote' (92% vs. 87%) and lower average inference cost ($0.016 vs. $0.04 per instance).\n",
"\n",
"A developer could use flaml to tune the configuration to satisfy the target inference budget while maximizing the value out of it."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
},
"vscode": {
"interpreter": {
"hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
}
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {
"2d910cfd2d2a4fc49fc30fbbdc5576a7": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"454146d0f7224f038689031002906e6f": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "2.0.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_e4ae2b6f5a974fd4bafb6abb9d12ff26",
"IPY_MODEL_577e1e3cc4db4942b0883577b3b52755",
"IPY_MODEL_b40bdfb1ac1d4cffb7cefcb870c64d45"
],
"layout": "IPY_MODEL_dc83c7bff2f241309537a8119dfc7555",
"tabbable": null,
"tooltip": null
}
},
"577e1e3cc4db4942b0883577b3b52755": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "2.0.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_allow_html": false,
"layout": "IPY_MODEL_2d910cfd2d2a4fc49fc30fbbdc5576a7",
"max": 1,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_74a6ba0c3cbc4051be0a83e152fe1e62",
"tabbable": null,
"tooltip": null,
"value": 1
}
},
"6086462a12d54bafa59d3c4566f06cb2": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"74a6ba0c3cbc4051be0a83e152fe1e62": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"7d3f3d9e15894d05a4d188ff4f466554": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HTMLStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HTMLStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "StyleView",
"background": null,
"description_width": "",
"font_size": null,
"text_color": null
}
},
"b40bdfb1ac1d4cffb7cefcb870c64d45": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "2.0.0",
"_view_name": "HTMLView",
"description": "",
"description_allow_html": false,
"layout": "IPY_MODEL_f1355871cc6f4dd4b50d9df5af20e5c8",
"placeholder": "​",
"style": "IPY_MODEL_ca245376fd9f4354af6b2befe4af4466",
"tabbable": null,
"tooltip": null,
"value": " 1/1 [00:00&lt;00:00, 44.69it/s]"
}
},
"ca245376fd9f4354af6b2befe4af4466": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HTMLStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HTMLStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "StyleView",
"background": null,
"description_width": "",
"font_size": null,
"text_color": null
}
},
"dc83c7bff2f241309537a8119dfc7555": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"e4ae2b6f5a974fd4bafb6abb9d12ff26": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "2.0.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "2.0.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "2.0.0",
"_view_name": "HTMLView",
"description": "",
"description_allow_html": false,
"layout": "IPY_MODEL_6086462a12d54bafa59d3c4566f06cb2",
"placeholder": "​",
"style": "IPY_MODEL_7d3f3d9e15894d05a4d188ff4f466554",
"tabbable": null,
"tooltip": null,
"value": "100%"
}
},
"f1355871cc6f4dd4b50d9df5af20e5c8": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
}
},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue Copy Permalink