mirror of
https://github.com/microsoft/autogen.git
synced 2025-09-26 08:34:10 +00:00
45641000c0
* improve max_valid_n and doc * Update README.md Co-authored-by: Li Jiang <lijiang1@microsoft.com> * add support for chatgpt * notebook * newline at end of file * chatgpt notebook * ChatGPT in Azure * doc * math * warning, timeout, log file name * handle import error * doc update; default value * paper * doc * docstr * eval_func * add a test func in completion * update notebook * update math notebook * improve notebok * lint and handle exception * flake8 * exception in test * add agg_method * NameError * refactor * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Update flaml/integrations/oai/completion.py Co-authored-by: Chi Wang <wang.chi@microsoft.com> * add example * merge files from oai_eval_test * Revert "merge files from oai_eval_test" This reverts commit 1e6a550f913bb94df6e9680934ccb7175d00702e. * merge * save results to notebook_output * update version and cache * update doc * save nb cell results to file * fix typo in model name * code improvements * improve docstr * docstr * docstr on the Returns of test --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Li Jiang <lijiang1@microsoft.com> Co-authored-by: Susan Xueqing Liu <liususan091219@users.noreply.github.com>
1233 lines
62 KiB
Plaintext
1233 lines
62 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"slideshow": {
|
||
"slide_type": "slide"
|
||
}
|
||
},
|
||
"source": [
|
||
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
|
||
"\n",
|
||
"Licensed under the MIT License.\n",
|
||
"\n",
|
||
"# Use FLAML to Tune OpenAI Models\n",
|
||
"\n",
|
||
"FLAML offers a cost-effective hyperparameter optimization technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) for tuning Large Language Models. Our study finds that tuning hyperparameters can significantly improve the utility of LLMs.\n",
|
||
"\n",
|
||
"In this notebook, we tune OpenAI models for code generation. We use [the HumanEval benchmark](https://huggingface.co/datasets/openai_humaneval) released by OpenAI for synthesizing programs from docstrings. \n",
|
||
"\n",
|
||
"## Requirements\n",
|
||
"\n",
|
||
"FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [openai] option:\n",
|
||
"```bash\n",
|
||
"pip install flaml[openai]==1.1.3\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:36.910966Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:36.910473Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:36.914554Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:36.914030Z"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# %pip install flaml[openai]==1.1.3 datasets"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Set your OpenAI key:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:36.917301Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:36.917011Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:36.923156Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:36.922619Z"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import os\n",
|
||
"\n",
|
||
"if \"OPENAI_API_KEY\" not in os.environ:\n",
|
||
" os.environ[\"OPENAI_API_KEY\"] = \"<your OpenAI API key here>\""
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"If you use Azure OpenAI, uncomment the following:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:36.925804Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:36.925423Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:36.928191Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:36.927673Z"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# openai.api_type = \"azure\"\n",
|
||
"# openai.api_base = \"https://<your_endpoint>.openai.azure.com/\"\n",
|
||
"# openai.api_version = \"2022-12-01\" # change if necessary"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Load dataset\n",
|
||
"\n",
|
||
"First, we load the humaneval dataset. The dataset contains 164 examples. We use the first 20 for tuning the generation hyperparameters and the remaining for evaluation. In each example, the \"prompt\" is the prompt string for eliciting the code generation, \"test\" is the Python code for unit test for the example, and \"entry_point\" is the function name to be tested."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:36.931255Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:36.930838Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:39.148799Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:39.148113Z"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Found cached dataset openai_humaneval (/home/vscode/.cache/huggingface/datasets/openai_humaneval/openai_humaneval/1.0.0/2955cebd73602e828fa8c0a424c594e5fab4ec863b316ca98f3d8fdb6a626e75)\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"application/vnd.jupyter.widget-view+json": {
|
||
"model_id": "d025d7cf0bc3438ba290e24d97855d8f",
|
||
"version_major": 2,
|
||
"version_minor": 0
|
||
},
|
||
"text/plain": [
|
||
" 0%| | 0/1 [00:00<?, ?it/s]"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/openai_humaneval/openai_humaneval/1.0.0/2955cebd73602e828fa8c0a424c594e5fab4ec863b316ca98f3d8fdb6a626e75/cache-1e8448101c1b32e8.arrow\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import datasets\n",
|
||
"\n",
|
||
"seed = 41\n",
|
||
"data = datasets.load_dataset(\"openai_humaneval\")[\"test\"].shuffle(seed=seed)\n",
|
||
"n_tune_data = 20\n",
|
||
"tune_data = [\n",
|
||
" {\n",
|
||
" \"prompt\": data[x][\"prompt\"],\n",
|
||
" \"test\": data[x][\"test\"],\n",
|
||
" \"entry_point\": data[x][\"entry_point\"],\n",
|
||
" }\n",
|
||
" for x in range(n_tune_data)\n",
|
||
"]\n",
|
||
"test_data = [\n",
|
||
" {\n",
|
||
" \"prompt\": data[x][\"prompt\"],\n",
|
||
" \"test\": data[x][\"test\"],\n",
|
||
" \"entry_point\": data[x][\"entry_point\"],\n",
|
||
" }\n",
|
||
" for x in range(n_tune_data, len(data))\n",
|
||
"]\n"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"slideshow": {
|
||
"slide_type": "slide"
|
||
}
|
||
},
|
||
"source": [
|
||
"Check a tuning example:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:39.152156Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:39.151531Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:39.155313Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:39.154731Z"
|
||
},
|
||
"slideshow": {
|
||
"slide_type": "subslide"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"def compare(game,guess):\n",
|
||
" \"\"\"I think we all remember that feeling when the result of some long-awaited\n",
|
||
" event is finally known. The feelings and thoughts you have at that moment are\n",
|
||
" definitely worth noting down and comparing.\n",
|
||
" Your task is to determine if a person correctly guessed the results of a number of matches.\n",
|
||
" You are given two arrays of scores and guesses of equal length, where each index shows a match. \n",
|
||
" Return an array of the same length denoting how far off each guess was. If they have guessed correctly,\n",
|
||
" the value is 0, and if not, the value is the absolute difference between the guess and the score.\n",
|
||
" \n",
|
||
" \n",
|
||
" example:\n",
|
||
"\n",
|
||
" compare([1,2,3,4,5,1],[1,2,3,4,2,-2]) -> [0,0,0,0,3,3]\n",
|
||
" compare([0,5,0,0,0,4],[4,1,1,0,0,-2]) -> [4,4,1,0,0,6]\n",
|
||
" \"\"\"\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(tune_data[1][\"prompt\"])"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here is one example of the unit test code for verifying the correctness of the generated code:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:39.158398Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:39.157766Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:39.161396Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:39.160797Z"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"def check(candidate):\n",
|
||
"\n",
|
||
" # Check some simple cases\n",
|
||
" assert candidate([1,2,3,4,5,1],[1,2,3,4,2,-2])==[0,0,0,0,3,3], \"This prints if this assert fails 1 (good for debugging!)\"\n",
|
||
" assert candidate([0,0,0,0,0,0],[0,0,0,0,0,0])==[0,0,0,0,0,0], \"This prints if this assert fails 1 (good for debugging!)\"\n",
|
||
" assert candidate([1,2,3],[-1,-2,-3])==[2,4,6], \"This prints if this assert fails 1 (good for debugging!)\"\n",
|
||
" assert candidate([1,2,3,5],[-1,2,3,4])==[2,0,0,1], \"This prints if this assert fails 1 (good for debugging!)\"\n",
|
||
"\n",
|
||
" # Check some edge cases that are easy to work out by hand.\n",
|
||
" assert True, \"This prints if this assert fails 2 (also good for debugging!)\"\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(tune_data[1][\"test\"])"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Define Success Metric\n",
|
||
"\n",
|
||
"Before we start tuning, we need to define the success metric we want to opotimize. For each code generation task, if one of the returned responses can pass the test, we consider the task as successfully solved. Then we can define the mean success rate of a collection of tasks.\n",
|
||
"\n",
|
||
"### Define a code executor\n",
|
||
"\n",
|
||
"First, we write a simple code executor. The code executor takes the generated code and the test code as the input, and execute them with a timer."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:39.164187Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:39.163867Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:39.169009Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:39.168427Z"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import signal\n",
|
||
"import subprocess\n",
|
||
"import sys\n",
|
||
"\n",
|
||
"def timeout_handler(signum, frame):\n",
|
||
" raise TimeoutError(\"Timed out!\")\n",
|
||
"\n",
|
||
"signal.signal(signal.SIGALRM, timeout_handler)\n",
|
||
"max_exec_time = 3 # seconds\n",
|
||
"\n",
|
||
"def execute_code(code):\n",
|
||
" code = code.strip()\n",
|
||
" with open(\"codetest.py\", \"w\") as fout:\n",
|
||
" fout.write(code)\n",
|
||
" try:\n",
|
||
" signal.alarm(max_exec_time)\n",
|
||
" result = subprocess.run(\n",
|
||
" [sys.executable, \"codetest.py\"],\n",
|
||
" stdout=subprocess.DEVNULL,\n",
|
||
" stderr=subprocess.PIPE,\n",
|
||
" )\n",
|
||
" signal.alarm(0)\n",
|
||
" except TimeoutError:\n",
|
||
" return 0\n",
|
||
" return int(result.returncode == 0)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"This function will create a temp file \"codetest.py\" and execute it in a separate process. It allows for 3 seconds to finish that code.\n",
|
||
"\n",
|
||
"### Define a function to evaluate the success for a given program synthesis task"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:39.171752Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:39.171347Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:39.176343Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:39.175510Z"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def success_metrics(responses, prompt, test, entry_point):\n",
|
||
" \"\"\"Check if the task is successful.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" responses (list): The list of responses.\n",
|
||
" prompt (str): The input prompt.\n",
|
||
" test (str): The test code.\n",
|
||
" entry_point (str): The name of the function.\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" dict: The success metrics.\n",
|
||
" \"\"\"\n",
|
||
" success_list = []\n",
|
||
" n = len(responses)\n",
|
||
" for i in range(n):\n",
|
||
" response = responses[i]\n",
|
||
" code = f\"{prompt}{response}\\n{test}\\ncheck({entry_point})\"\n",
|
||
" succeed = execute_code(code)\n",
|
||
" success_list.append(succeed)\n",
|
||
" return {\n",
|
||
" \"expected_success\": 1 - pow(1 - sum(success_list) / n, n),\n",
|
||
" \"success\": any(s for s in success_list),\n",
|
||
" }\n"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"slideshow": {
|
||
"slide_type": "slide"
|
||
}
|
||
},
|
||
"source": [
|
||
"## Use the tuning data to find a good configuration\n",
|
||
"\n",
|
||
"### Import the oai and tune subpackages from flaml.\n",
|
||
"\n",
|
||
"FLAML has provided an API for hyperparameter optimization of OpenAI models: `oai.Completion.tune` and to make a request with the tuned config: `oai.Completion.create`. First, we import oai from flaml:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:39.179030Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:39.178624Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:40.584410Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:40.583802Z"
|
||
},
|
||
"slideshow": {
|
||
"slide_type": "slide"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from flaml import oai, tune"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"For (local) reproducibility and cost efficiency, we cache responses from OpenAI."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:40.587815Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:40.587283Z",
|
||
"iopub.status.idle": "2023-02-24T23:25:40.590826Z",
|
||
"shell.execute_reply": "2023-02-24T23:25:40.590158Z"
|
||
},
|
||
"slideshow": {
|
||
"slide_type": "slide"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"oai.Completion.set_cache(seed)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"This will create a disk cache in \".cache/{seed}\". You can change `cache_path` in `set_cache()`. The cache for different seeds are stored separately.\n",
|
||
"\n",
|
||
"### Perform tuning\n",
|
||
"\n",
|
||
"The tuning will take a while to finish, depending on the optimization budget. The tuning will be performed under the specified optimization budgets.\n",
|
||
"\n",
|
||
"* `inference_budget` is the target average inference budget per instance in the benchmark. For example, 0.02 means the target inference budget is 0.02 dollars, which translates to 1000 tokens (input + output combined) if the text Davinci model is used.\n",
|
||
"* `optimization_budget` is the total budget allowed to perform the tuning. For example, 5 means 5 dollars are allowed in total, which translates to 250K tokens for the text Davinci model.\n",
|
||
"* `num_sumples` is the number of different hyperparameter configurations which is allowed to try. The tuning will stop after either num_samples trials or after optimization_budget dollars spent, whichever happens first. -1 means no hard restriction in the number of trials and the actual number is decided by `optimization_budget`.\n",
|
||
"\n",
|
||
"Users can specify tuning data, optimization metric, optimization mode, evaluation function, search spaces etc.. The default search space is:\n",
|
||
"\n",
|
||
"```python\n",
|
||
"default_search_space = {\n",
|
||
" \"model\": tune.choice([\n",
|
||
" \"text-ada-001\",\n",
|
||
" \"text-babbage-001\",\n",
|
||
" \"text-davinci-003\",\n",
|
||
" \"gpt-3.5-turbo\",\n",
|
||
" \"gpt-4\",\n",
|
||
" ]),\n",
|
||
" \"temperature_or_top_p\": tune.choice(\n",
|
||
" [\n",
|
||
" {\"temperature\": tune.uniform(0, 1)},\n",
|
||
" {\"top_p\": tune.uniform(0, 1)},\n",
|
||
" ]\n",
|
||
" ),\n",
|
||
" \"max_tokens\": tune.lograndint(50, 1000),\n",
|
||
" \"n\": tune.randint(1, 100),\n",
|
||
" \"prompt\": \"{prompt}\",\n",
|
||
"}\n",
|
||
"```\n",
|
||
"\n",
|
||
"The default search space can be overridden by users' input.\n",
|
||
"For example, the following code specifies four choices for the prompt and a fixed list of stop sequences. For hyperparameters which don't appear in users' input, the default search space will be used. If you don't have access to gpt-4 or would like to modify the choice of models, you can provide a different search space for model."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 21,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:25:40.593603Z",
|
||
"iopub.status.busy": "2023-02-24T23:25:40.593269Z",
|
||
"iopub.status.idle": "2023-02-24T23:26:38.349191Z",
|
||
"shell.execute_reply": "2023-02-24T23:26:38.348392Z"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\u001b[32m[I 2023-03-26 02:53:26,384]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n",
|
||
"\u001b[32m[I 2023-03-26 02:53:26,387]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"[flaml.tune.tune: 03-26 02:53:26] {811} INFO - trial 1 config: {'model': 'text-davinci-003', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:29] {215} INFO - result: {'expected_success': 0.6, 'success': 0.6, 'total_cost': 0.09264000000000001, 'cost': 0.09264000000000001, 'inference_cost': 0.004632, 'training_iteration': 0, 'config': {'model': 'text-davinci-003', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}, 'config/model': 'text-davinci-003', 'config/temperature_or_top_p': {'temperature': 0.36865945026811975}, 'config/max_tokens': 347, 'config/n': 1, 'config/prompt': 1, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 3.5772321224212646}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:29] {811} INFO - trial 2 config: {'model': 'text-ada-001', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:30] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 0.09429879999999999, 'cost': 0.0016588, 'inference_cost': 7.264e-05, 'training_iteration': 0, 'config': {'model': 'text-ada-001', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}, 'config/model': 'text-ada-001', 'config/temperature_or_top_p': {'temperature': 0.36865945026811975}, 'config/max_tokens': 347, 'config/n': 1, 'config/prompt': 1, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 0.5873167514801025}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:30] {811} INFO - trial 3 config: {'model': 'text-babbage-001', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:31] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 0.09782479999999999, 'cost': 0.003526, 'inference_cost': 0.00016342499999999997, 'training_iteration': 0, 'config': {'model': 'text-babbage-001', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}, 'config/model': 'text-babbage-001', 'config/temperature_or_top_p': {'temperature': 0.36865945026811975}, 'config/max_tokens': 347, 'config/n': 1, 'config/prompt': 1, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 0.6068365573883057}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:31] {811} INFO - trial 4 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:31] {215} INFO - result: {'expected_success': 0.2, 'success': 0.2, 'total_cost': 0.10643079999999999, 'cost': 0.008606, 'inference_cost': 0.0004394, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.36865945026811975}, 'config/max_tokens': 347, 'config/n': 1, 'config/prompt': 1, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 0.5878369808197021}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:31] {811} INFO - trial 5 config: {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:32] {215} INFO - result: {'expected_success': 0.8, 'success': 0.8, 'total_cost': 0.2603308, 'cost': 0.15389999999999998, 'inference_cost': 0.007861499999999999, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.36865945026811975}, 'max_tokens': 347, 'n': 1, 'prompt': 1, 'stop': 0}, 'config/model': 'gpt-4', 'config/temperature_or_top_p': {'temperature': 0.36865945026811975}, 'config/max_tokens': 347, 'config/n': 1, 'config/prompt': 1, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 0.6071126461029053}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:32] {811} INFO - trial 6 config: {'model': 'text-ada-001', 'temperature_or_top_p': {'temperature': 0.7605307121989587}, 'max_tokens': 82, 'n': 9, 'prompt': 1, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:37] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 0.2629064, 'cost': 0.0025756000000000004, 'inference_cost': 0.00011848, 'training_iteration': 0, 'config': {'model': 'text-ada-001', 'temperature_or_top_p': {'temperature': 0.7605307121989587}, 'max_tokens': 82, 'n': 9, 'prompt': 1, 'stop': 0}, 'config/model': 'text-ada-001', 'config/temperature_or_top_p': {'temperature': 0.7605307121989587}, 'config/max_tokens': 82, 'config/n': 9, 'config/prompt': 1, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 5.4761645793914795}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:37] {811} INFO - trial 7 config: {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.14217004760152696}, 'max_tokens': 152, 'n': 67, 'prompt': 2, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:37] {215} INFO - result: {'expected_success': 0, 'total_cost': 0.4132364, 'cost': 0.15033000000000002, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.14217004760152696}, 'max_tokens': 152, 'n': 67, 'prompt': 2, 'stop': 0}, 'config/model': 'gpt-4', 'config/temperature_or_top_p': {'temperature': 0.14217004760152696}, 'config/max_tokens': 152, 'config/n': 67, 'config/prompt': 2, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 0.0022079944610595703}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:37] {811} INFO - trial 8 config: {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.30070005663620336}, 'max_tokens': 70, 'n': 83, 'prompt': 3, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:37] {215} INFO - result: {'expected_success': 0, 'total_cost': 0.6260264, 'cost': 0.21278999999999998, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.30070005663620336}, 'max_tokens': 70, 'n': 83, 'prompt': 3, 'stop': 0}, 'config/model': 'gpt-4', 'config/temperature_or_top_p': {'temperature': 0.30070005663620336}, 'config/max_tokens': 70, 'config/n': 83, 'config/prompt': 3, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 0.0022161006927490234}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:37] {811} INFO - trial 9 config: {'model': 'text-babbage-001', 'temperature_or_top_p': {'temperature': 0.16501589771914849}, 'max_tokens': 161, 'n': 10, 'prompt': 3, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:43] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 0.6310854, 'cost': 0.005059, 'inference_cost': 0.00023457499999999997, 'training_iteration': 0, 'config': {'model': 'text-babbage-001', 'temperature_or_top_p': {'temperature': 0.16501589771914849}, 'max_tokens': 161, 'n': 10, 'prompt': 3, 'stop': 0}, 'config/model': 'text-babbage-001', 'config/temperature_or_top_p': {'temperature': 0.16501589771914849}, 'config/max_tokens': 161, 'config/n': 10, 'config/prompt': 3, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 5.868851661682129}\n",
|
||
"[flaml.tune.tune: 03-26 02:53:43] {811} INFO - trial 10 config: {'model': 'text-ada-001', 'temperature_or_top_p': {'temperature': 0.5902013629854229}, 'max_tokens': 56, 'n': 36, 'prompt': 3, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:54:05] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 0.6344234000000001, 'cost': 0.003338, 'inference_cost': 0.0001522, 'training_iteration': 0, 'config': {'model': 'text-ada-001', 'temperature_or_top_p': {'temperature': 0.5902013629854229}, 'max_tokens': 56, 'n': 36, 'prompt': 3, 'stop': 0}, 'config/model': 'text-ada-001', 'config/temperature_or_top_p': {'temperature': 0.5902013629854229}, 'config/max_tokens': 56, 'config/n': 36, 'config/prompt': 3, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 21.348156690597534}\n",
|
||
"[flaml.tune.tune: 03-26 02:54:05] {811} INFO - trial 11 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.763240587143681}, 'max_tokens': 693, 'n': 42, 'prompt': 0, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:54:33] {215} INFO - result: {'expected_success': 0.3476191678990812, 'success': 0.35, 'total_cost': 0.7530034000000003, 'cost': 0.11858000000000002, 'inference_cost': 0.005490999999999999, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.763240587143681}, 'max_tokens': 693, 'n': 42, 'prompt': 0, 'stop': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'top_p': 0.763240587143681}, 'config/max_tokens': 693, 'config/n': 42, 'config/prompt': 0, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 28.24349284172058}\n",
|
||
"[flaml.tune.tune: 03-26 02:54:33] {811} INFO - trial 12 config: {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.2927979762895091}, 'max_tokens': 60, 'n': 97, 'prompt': 2, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:54:33] {215} INFO - result: {'expected_success': 0, 'total_cost': 0.9340534000000004, 'cost': 0.18105, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'temperature_or_top_p': {'temperature': 0.2927979762895091}, 'max_tokens': 60, 'n': 97, 'prompt': 2, 'stop': 0}, 'config/model': 'gpt-4', 'config/temperature_or_top_p': {'temperature': 0.2927979762895091}, 'config/max_tokens': 60, 'config/n': 97, 'config/prompt': 2, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 0.002497434616088867}\n",
|
||
"[flaml.tune.tune: 03-26 02:54:33] {811} INFO - trial 13 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.7186028103822503}, 'max_tokens': 288, 'n': 4, 'prompt': 1, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:54:35] {215} INFO - result: {'expected_success': 0.28359375, 'success': 0.35, 'total_cost': 0.9496594000000004, 'cost': 0.015605999999999998, 'inference_cost': 0.0007894, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'top_p': 0.7186028103822503}, 'max_tokens': 288, 'n': 4, 'prompt': 1, 'stop': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'top_p': 0.7186028103822503}, 'config/max_tokens': 288, 'config/n': 4, 'config/prompt': 1, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 2.29030704498291}\n",
|
||
"[flaml.tune.tune: 03-26 02:54:35] {811} INFO - trial 14 config: {'model': 'text-ada-001', 'temperature_or_top_p': {'top_p': 0.3653649712141158}, 'max_tokens': 96, 'n': 75, 'prompt': 1, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:55:20] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 0.9550898000000005, 'cost': 0.0054304, 'inference_cost': 0.00026122, 'training_iteration': 0, 'config': {'model': 'text-ada-001', 'temperature_or_top_p': {'top_p': 0.3653649712141158}, 'max_tokens': 96, 'n': 75, 'prompt': 1, 'stop': 0}, 'config/model': 'text-ada-001', 'config/temperature_or_top_p': {'top_p': 0.3653649712141158}, 'config/max_tokens': 96, 'config/n': 75, 'config/prompt': 1, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 44.837317943573}\n",
|
||
"[flaml.tune.tune: 03-26 02:55:20] {811} INFO - trial 15 config: {'model': 'text-davinci-003', 'temperature_or_top_p': {'temperature': 0.3814115349046321}, 'max_tokens': 791, 'n': 92, 'prompt': 3, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:55:20] {215} INFO - result: {'expected_success': 0, 'total_cost': 1.0798498000000005, 'cost': 0.12475999999999998, 'training_iteration': 0, 'config': {'model': 'text-davinci-003', 'temperature_or_top_p': {'temperature': 0.3814115349046321}, 'max_tokens': 791, 'n': 92, 'prompt': 3, 'stop': 0}, 'config/model': 'text-davinci-003', 'config/temperature_or_top_p': {'temperature': 0.3814115349046321}, 'config/max_tokens': 791, 'config/n': 92, 'config/prompt': 3, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 0.0024149417877197266}\n",
|
||
"[flaml.tune.tune: 03-26 02:55:20] {811} INFO - trial 16 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.4284507389678964}, 'max_tokens': 398, 'n': 11, 'prompt': 3, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:55:29] {215} INFO - result: {'expected_success': 0.5484931390416686, 'success': 0.55, 'total_cost': 1.1118038000000003, 'cost': 0.031954, 'inference_cost': 0.0015885000000000003, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.4284507389678964}, 'max_tokens': 398, 'n': 11, 'prompt': 3, 'stop': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.4284507389678964}, 'config/max_tokens': 398, 'config/n': 11, 'config/prompt': 3, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 9.271101951599121}\n",
|
||
"[flaml.tune.tune: 03-26 02:55:29] {811} INFO - trial 17 config: {'model': 'gpt-4', 'max_tokens': 211, 'n': 13, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.25447895557126815}}\n",
|
||
"[flaml.tune.tune: 03-26 02:55:46] {215} INFO - result: {'expected_success': 0.8822303234803123, 'success': 0.9, 'total_cost': 2.1304238, 'cost': 1.0186199999999999, 'inference_cost': 0.0484995, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'max_tokens': 211, 'n': 13, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.25447895557126815}}, 'config/model': 'gpt-4', 'config/max_tokens': 211, 'config/n': 13, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.25447895557126815}, 'experiment_tag': 'exp', 'time_total_s': 16.604310512542725}\n",
|
||
"[flaml.tune.tune: 03-26 02:55:46] {811} INFO - trial 18 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9761031076386442}, 'max_tokens': 349, 'n': 23, 'prompt': 0, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:05] {215} INFO - result: {'expected_success': 0.3551828400470255, 'success': 0.4, 'total_cost': 2.1919698000000003, 'cost': 0.061546, 'inference_cost': 0.0030944, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9761031076386442}, 'max_tokens': 349, 'n': 23, 'prompt': 0, 'stop': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.9761031076386442}, 'config/max_tokens': 349, 'config/n': 23, 'config/prompt': 0, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 19.451276063919067}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:05] {811} INFO - trial 19 config: {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9822374507369328}, 'max_tokens': 393, 'n': 22, 'prompt': 0, 'stop': 0}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:18] {215} INFO - result: {'expected_success': 0.2898979473186428, 'success': 0.35, 'total_cost': 2.2507018000000003, 'cost': 0.058732, 'inference_cost': 0.0029537, 'training_iteration': 0, 'config': {'model': 'gpt-3.5-turbo', 'temperature_or_top_p': {'temperature': 0.9822374507369328}, 'max_tokens': 393, 'n': 22, 'prompt': 0, 'stop': 0}, 'config/model': 'gpt-3.5-turbo', 'config/temperature_or_top_p': {'temperature': 0.9822374507369328}, 'config/max_tokens': 393, 'config/n': 22, 'config/prompt': 0, 'config/stop': 0, 'experiment_tag': 'exp', 'time_total_s': 13.075204372406006}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:18] {811} INFO - trial 20 config: {'model': 'gpt-4', 'max_tokens': 348, 'n': 1, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.36865945026811975}}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:19] {215} INFO - result: {'expected_success': 0.75, 'success': 0.75, 'total_cost': 2.4012418000000006, 'cost': 0.15053999999999995, 'inference_cost': 0.007693499999999999, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'max_tokens': 348, 'n': 1, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.36865945026811975}}, 'config/model': 'gpt-4', 'config/max_tokens': 348, 'config/n': 1, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.36865945026811975}, 'experiment_tag': 'exp', 'time_total_s': 0.6143312454223633}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:19] {811} INFO - trial 21 config: {'model': 'text-ada-001', 'max_tokens': 130, 'n': 22, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.22084263211180838}}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:32] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 2.4043414000000016, 'cost': 0.0030996000000000005, 'inference_cost': 0.00014468, 'training_iteration': 0, 'config': {'model': 'text-ada-001', 'max_tokens': 130, 'n': 22, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.22084263211180838}}, 'config/model': 'text-ada-001', 'config/max_tokens': 130, 'config/n': 22, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.22084263211180838}, 'experiment_tag': 'exp', 'time_total_s': 13.137321710586548}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:32] {811} INFO - trial 22 config: {'model': 'text-ada-001', 'max_tokens': 342, 'n': 4, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.2881152790307279}}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:35] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 2.4061918000000024, 'cost': 0.0018504, 'inference_cost': 8.222e-05, 'training_iteration': 0, 'config': {'model': 'text-ada-001', 'max_tokens': 342, 'n': 4, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.2881152790307279}}, 'config/model': 'text-ada-001', 'config/max_tokens': 342, 'config/n': 4, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.2881152790307279}, 'experiment_tag': 'exp', 'time_total_s': 2.4484035968780518}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:35] {811} INFO - trial 23 config: {'model': 'gpt-4', 'max_tokens': 253, 'n': 23, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.41254458573656}}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:35] {215} INFO - result: {'expected_success': 0, 'total_cost': 2.618831800000003, 'cost': 0.21264, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'max_tokens': 253, 'n': 23, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.41254458573656}}, 'config/model': 'gpt-4', 'config/max_tokens': 253, 'config/n': 23, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.41254458573656}, 'experiment_tag': 'exp', 'time_total_s': 0.003139972686767578}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:35] {811} INFO - trial 24 config: {'model': 'gpt-4', 'max_tokens': 176, 'n': 3, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.0964133254059763}}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:36] {215} INFO - result: {'expected_success': 0.8185185185185185, 'success': 0.85, 'total_cost': 2.912231800000003, 'cost': 0.29339999999999994, 'inference_cost': 0.014836499999999999, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'max_tokens': 176, 'n': 3, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.0964133254059763}}, 'config/model': 'gpt-4', 'config/max_tokens': 176, 'config/n': 3, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.0964133254059763}, 'experiment_tag': 'exp', 'time_total_s': 1.8556303977966309}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:36] {811} INFO - trial 25 config: {'model': 'text-babbage-001', 'max_tokens': 343, 'n': 27, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.24286268913046594}}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:55] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 2.9569863000000023, 'cost': 0.04475450000000001, 'inference_cost': 0.00222485, 'training_iteration': 0, 'config': {'model': 'text-babbage-001', 'max_tokens': 343, 'n': 27, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.24286268913046594}}, 'config/model': 'text-babbage-001', 'config/max_tokens': 343, 'config/n': 27, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.24286268913046594}, 'experiment_tag': 'exp', 'time_total_s': 19.013901472091675}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:55] {811} INFO - trial 26 config: {'model': 'text-babbage-001', 'max_tokens': 130, 'n': 1, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.26609522201207036}}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:56] {215} INFO - result: {'expected_success': 0.0, 'success': 0.0, 'total_cost': 2.9595088000000023, 'cost': 0.0025224999999999996, 'inference_cost': 0.00011325, 'training_iteration': 0, 'config': {'model': 'text-babbage-001', 'max_tokens': 130, 'n': 1, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.26609522201207036}}, 'config/model': 'text-babbage-001', 'config/max_tokens': 130, 'config/n': 1, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.26609522201207036}, 'experiment_tag': 'exp', 'time_total_s': 0.5786199569702148}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:56] {811} INFO - trial 27 config: {'model': 'gpt-4', 'max_tokens': 212, 'n': 29, 'prompt': 0, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.24802150727233283}}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:56] {215} INFO - result: {'expected_success': 0, 'total_cost': 3.0123088000000022, 'cost': 0.05279999999999999, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'max_tokens': 212, 'n': 29, 'prompt': 0, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.24802150727233283}}, 'config/model': 'gpt-4', 'config/max_tokens': 212, 'config/n': 29, 'config/prompt': 0, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.24802150727233283}, 'experiment_tag': 'exp', 'time_total_s': 0.0019483566284179688}\n",
|
||
"[flaml.tune.tune: 03-26 02:56:56] {834} WARNING - fail to sample a trial for 100 times in a row, stopping.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"config, analysis = oai.Completion.tune(\n",
|
||
" data=tune_data, # the data for tuning\n",
|
||
" metric=\"expected_success\", # the metric to optimize\n",
|
||
" mode=\"max\", # the optimization mode\n",
|
||
" eval_func=success_metrics, # the evaluation function to return the success metrics\n",
|
||
" # log_file_name=\"logs/humaneval.log\", # the log file name\n",
|
||
" inference_budget=0.05, # the inference budget (dollar)\n",
|
||
" optimization_budget=3, # the optimization budget (dollar)\n",
|
||
" # num_samples can further limit the number of trials for different hyperparameter configurations;\n",
|
||
" # -1 means decided by the optimization budget only\n",
|
||
" num_samples=-1,\n",
|
||
" prompt=[\n",
|
||
" \"{prompt}\",\n",
|
||
" \"# Python 3{prompt}\",\n",
|
||
" \"Complete the following Python function:{prompt}\",\n",
|
||
" \"Complete the following Python function while including necessary import statements inside the function:{prompt}\",\n",
|
||
" ], # the prompt templates to choose from\n",
|
||
" stop=[\"\\nclass\", \"\\ndef\", \"\\nif\", \"\\nprint\"], # the stop sequence\n",
|
||
")\n"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Output tuning results\n",
|
||
"\n",
|
||
"After the tuning, we can print out the config and the result found by FLAML:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 22,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:26:38.352710Z",
|
||
"iopub.status.busy": "2023-02-24T23:26:38.352378Z",
|
||
"iopub.status.idle": "2023-02-24T23:26:38.356939Z",
|
||
"shell.execute_reply": "2023-02-24T23:26:38.356217Z"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"optimized config {'model': 'gpt-4', 'max_tokens': 211, 'n': 13, 'prompt': '# Python 3{prompt}', 'stop': ['\\nclass', '\\ndef', '\\nif', '\\nprint'], 'temperature': 0.25447895557126815}\n",
|
||
"best result on tuning data {'expected_success': 0.8822303234803123, 'success': 0.9, 'total_cost': 2.1304238, 'cost': 1.0186199999999999, 'inference_cost': 0.0484995, 'training_iteration': 0, 'config': {'model': 'gpt-4', 'max_tokens': 211, 'n': 13, 'prompt': 1, 'stop': 0, 'temperature_or_top_p': {'temperature': 0.25447895557126815}}, 'config/model': 'gpt-4', 'config/max_tokens': 211, 'config/n': 13, 'config/prompt': 1, 'config/stop': 0, 'config/temperature_or_top_p': {'temperature': 0.25447895557126815}, 'experiment_tag': 'exp', 'time_total_s': 16.604310512542725}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"optimized config\", config)\n",
|
||
"print(\"best result on tuning data\", analysis.best_result)\n",
|
||
"\n",
|
||
"# save results to notebook_output.txt\n",
|
||
"from flaml.version import __version__ as flaml_version\n",
|
||
"import datetime\n",
|
||
"results = {\"optimized config\": config, \"best result on tuning data\": analysis.best_result,}\n",
|
||
"result_info_dict = {\"result_name\": \"integrate_openai.ipynb + optimized config and best result on tuning data\",\n",
|
||
" \"flaml_version\": flaml_version, \n",
|
||
" \"time\": datetime.datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\"),\n",
|
||
" \"results\": results}\n",
|
||
"result_info = \"result name: {result_name}, flaml version: {flaml_version}, time: {time}, results: {results}\".format(**result_info_dict)\n",
|
||
"with open(\"notebook_output.txt\", \"a\") as f:\n",
|
||
" f.write(\"\\n\")\n",
|
||
" f.write(result_info)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"slideshow": {
|
||
"slide_type": "slide"
|
||
}
|
||
},
|
||
"source": [
|
||
"### Make a request with the tuned config\n",
|
||
"\n",
|
||
"We can apply the tuned config on the request for an example task:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 23,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:26:38.359902Z",
|
||
"iopub.status.busy": "2023-02-24T23:26:38.359506Z",
|
||
"iopub.status.idle": "2023-02-24T23:26:39.343921Z",
|
||
"shell.execute_reply": "2023-02-24T23:26:39.343051Z"
|
||
},
|
||
"slideshow": {
|
||
"slide_type": "subslide"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"{\n",
|
||
" \"choices\": [\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 0,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 1,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 2,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 3,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n return [abs(game[i] - guess[i]) for i in range(len(game))]\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 4,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 5,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 6,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n if game[i] == guess[i]:\\n result.append(0)\\n else:\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 7,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n return [abs(game[i] - guess[i]) for i in range(len(game))]\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 8,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n return [abs(a - b) for a, b in zip(game, guess)]\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 9,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 10,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 11,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"finish_reason\": \"stop\",\n",
|
||
" \"index\": 12,\n",
|
||
" \"message\": {\n",
|
||
" \"content\": \"def compare(game, guess):\\n result = []\\n for i in range(len(game)):\\n if game[i] == guess[i]:\\n result.append(0)\\n else:\\n result.append(abs(game[i] - guess[i]))\\n return result\\n\",\n",
|
||
" \"role\": \"assistant\"\n",
|
||
" }\n",
|
||
" }\n",
|
||
" ],\n",
|
||
" \"created\": 1679636800,\n",
|
||
" \"id\": \"chatcmpl-6xUY4niTRrpJ5UShayb9QncgjS8rg\",\n",
|
||
" \"model\": \"gpt-4-0314\",\n",
|
||
" \"object\": \"chat.completion\",\n",
|
||
" \"usage\": {\n",
|
||
" \"completion_tokens\": 440,\n",
|
||
" \"prompt_tokens\": 236,\n",
|
||
" \"total_tokens\": 676\n",
|
||
" }\n",
|
||
"}\n",
|
||
"{'expected_success': 1.0, 'success': True}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"responses = oai.Completion.create(context=tune_data[1], **config)\n",
|
||
"metric_results = success_metrics([response[\"message\"][\"content\"] if config[\"model\"] in oai.Completion.chat_models else response[\"text\"] for response in responses[\"choices\"]], **tune_data[1])\n",
|
||
"print(\"response on an example data instance:\", responses)\n",
|
||
"print(\"metric_results on the example data instance:\", metric_results)\n"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Evaluate the success rate on the test data\n",
|
||
"\n",
|
||
"You can use flaml's `oai.Completion.test` to evaluate the performance of an entire dataset with the tuned config. The following code will take a while to evaluate all the 144 test data instances. The cost is about $7 if you uncomment it and run it."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 25,
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2023-02-24T23:26:39.347295Z",
|
||
"iopub.status.busy": "2023-02-24T23:26:39.346994Z",
|
||
"iopub.status.idle": "2023-02-24T23:29:27.160335Z",
|
||
"shell.execute_reply": "2023-02-24T23:29:27.159519Z"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"{'expected_success': 0.8326778348739547, 'success': 0.8472222222222222, 'total_cost': 10.024478799999999, 'cost': 7.01217, 'inference_cost': 0.049131249999999994}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"result = oai.Completion.test(test_data, config, success_metrics)\n",
|
||
"print(\"performance on test data with the tuned config:\", result)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The result will vary with the inference budget and optimization budget.\n"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "tutorial",
|
||
"language": "python",
|
||
"name": "tutorial"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.9.7"
|
||
},
|
||
"vscode": {
|
||
"interpreter": {
|
||
"hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
|
||
}
|
||
},
|
||
"widgets": {
|
||
"application/vnd.jupyter.widget-state+json": {
|
||
"state": {
|
||
"24dd93300e0442788ee6cc1310e5bf14": {
|
||
"model_module": "@jupyter-widgets/controls",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "HTMLStyleModel",
|
||
"state": {
|
||
"_model_module": "@jupyter-widgets/controls",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "HTMLStyleModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/base",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "StyleView",
|
||
"background": null,
|
||
"description_width": "",
|
||
"font_size": null,
|
||
"text_color": null
|
||
}
|
||
},
|
||
"35cd066a31b242bb87b2c106ee72e5f2": {
|
||
"model_module": "@jupyter-widgets/controls",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "HBoxModel",
|
||
"state": {
|
||
"_dom_classes": [],
|
||
"_model_module": "@jupyter-widgets/controls",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "HBoxModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/controls",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "HBoxView",
|
||
"box_style": "",
|
||
"children": [
|
||
"IPY_MODEL_8e7ee7687a99410d88a98a74ecfcea99",
|
||
"IPY_MODEL_421e02a11a974b40b3ddb75382b3b640",
|
||
"IPY_MODEL_77db9797e78b49438d21c5c8da34b4cb"
|
||
],
|
||
"layout": "IPY_MODEL_47d3046236a54b0e8f9ae455a82c7e0b",
|
||
"tabbable": null,
|
||
"tooltip": null
|
||
}
|
||
},
|
||
"3d5d106a38954af2bb3bde5777702f4e": {
|
||
"model_module": "@jupyter-widgets/controls",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "HTMLStyleModel",
|
||
"state": {
|
||
"_model_module": "@jupyter-widgets/controls",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "HTMLStyleModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/base",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "StyleView",
|
||
"background": null,
|
||
"description_width": "",
|
||
"font_size": null,
|
||
"text_color": null
|
||
}
|
||
},
|
||
"3e1ebb31412443b0bca86a301cbdac11": {
|
||
"model_module": "@jupyter-widgets/controls",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "ProgressStyleModel",
|
||
"state": {
|
||
"_model_module": "@jupyter-widgets/controls",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "ProgressStyleModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/base",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "StyleView",
|
||
"bar_color": null,
|
||
"description_width": ""
|
||
}
|
||
},
|
||
"421e02a11a974b40b3ddb75382b3b640": {
|
||
"model_module": "@jupyter-widgets/controls",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "FloatProgressModel",
|
||
"state": {
|
||
"_dom_classes": [],
|
||
"_model_module": "@jupyter-widgets/controls",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "FloatProgressModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/controls",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "ProgressView",
|
||
"bar_style": "success",
|
||
"description": "",
|
||
"description_allow_html": false,
|
||
"layout": "IPY_MODEL_e6398d4027c9459a97965b9d91ae484f",
|
||
"max": 1,
|
||
"min": 0,
|
||
"orientation": "horizontal",
|
||
"style": "IPY_MODEL_3e1ebb31412443b0bca86a301cbdac11",
|
||
"tabbable": null,
|
||
"tooltip": null,
|
||
"value": 1
|
||
}
|
||
},
|
||
"47d3046236a54b0e8f9ae455a82c7e0b": {
|
||
"model_module": "@jupyter-widgets/base",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "LayoutModel",
|
||
"state": {
|
||
"_model_module": "@jupyter-widgets/base",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "LayoutModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/base",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "LayoutView",
|
||
"align_content": null,
|
||
"align_items": null,
|
||
"align_self": null,
|
||
"border_bottom": null,
|
||
"border_left": null,
|
||
"border_right": null,
|
||
"border_top": null,
|
||
"bottom": null,
|
||
"display": null,
|
||
"flex": null,
|
||
"flex_flow": null,
|
||
"grid_area": null,
|
||
"grid_auto_columns": null,
|
||
"grid_auto_flow": null,
|
||
"grid_auto_rows": null,
|
||
"grid_column": null,
|
||
"grid_gap": null,
|
||
"grid_row": null,
|
||
"grid_template_areas": null,
|
||
"grid_template_columns": null,
|
||
"grid_template_rows": null,
|
||
"height": null,
|
||
"justify_content": null,
|
||
"justify_items": null,
|
||
"left": null,
|
||
"margin": null,
|
||
"max_height": null,
|
||
"max_width": null,
|
||
"min_height": null,
|
||
"min_width": null,
|
||
"object_fit": null,
|
||
"object_position": null,
|
||
"order": null,
|
||
"overflow": null,
|
||
"padding": null,
|
||
"right": null,
|
||
"top": null,
|
||
"visibility": null,
|
||
"width": null
|
||
}
|
||
},
|
||
"754800f7feb04acea977696e4787d1ff": {
|
||
"model_module": "@jupyter-widgets/base",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "LayoutModel",
|
||
"state": {
|
||
"_model_module": "@jupyter-widgets/base",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "LayoutModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/base",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "LayoutView",
|
||
"align_content": null,
|
||
"align_items": null,
|
||
"align_self": null,
|
||
"border_bottom": null,
|
||
"border_left": null,
|
||
"border_right": null,
|
||
"border_top": null,
|
||
"bottom": null,
|
||
"display": null,
|
||
"flex": null,
|
||
"flex_flow": null,
|
||
"grid_area": null,
|
||
"grid_auto_columns": null,
|
||
"grid_auto_flow": null,
|
||
"grid_auto_rows": null,
|
||
"grid_column": null,
|
||
"grid_gap": null,
|
||
"grid_row": null,
|
||
"grid_template_areas": null,
|
||
"grid_template_columns": null,
|
||
"grid_template_rows": null,
|
||
"height": null,
|
||
"justify_content": null,
|
||
"justify_items": null,
|
||
"left": null,
|
||
"margin": null,
|
||
"max_height": null,
|
||
"max_width": null,
|
||
"min_height": null,
|
||
"min_width": null,
|
||
"object_fit": null,
|
||
"object_position": null,
|
||
"order": null,
|
||
"overflow": null,
|
||
"padding": null,
|
||
"right": null,
|
||
"top": null,
|
||
"visibility": null,
|
||
"width": null
|
||
}
|
||
},
|
||
"77db9797e78b49438d21c5c8da34b4cb": {
|
||
"model_module": "@jupyter-widgets/controls",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "HTMLModel",
|
||
"state": {
|
||
"_dom_classes": [],
|
||
"_model_module": "@jupyter-widgets/controls",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "HTMLModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/controls",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "HTMLView",
|
||
"description": "",
|
||
"description_allow_html": false,
|
||
"layout": "IPY_MODEL_7b6c4e1c11e249409a1edcd63be450d8",
|
||
"placeholder": "",
|
||
"style": "IPY_MODEL_3d5d106a38954af2bb3bde5777702f4e",
|
||
"tabbable": null,
|
||
"tooltip": null,
|
||
"value": " 1/1 [00:00<00:00, 44.40it/s]"
|
||
}
|
||
},
|
||
"7b6c4e1c11e249409a1edcd63be450d8": {
|
||
"model_module": "@jupyter-widgets/base",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "LayoutModel",
|
||
"state": {
|
||
"_model_module": "@jupyter-widgets/base",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "LayoutModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/base",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "LayoutView",
|
||
"align_content": null,
|
||
"align_items": null,
|
||
"align_self": null,
|
||
"border_bottom": null,
|
||
"border_left": null,
|
||
"border_right": null,
|
||
"border_top": null,
|
||
"bottom": null,
|
||
"display": null,
|
||
"flex": null,
|
||
"flex_flow": null,
|
||
"grid_area": null,
|
||
"grid_auto_columns": null,
|
||
"grid_auto_flow": null,
|
||
"grid_auto_rows": null,
|
||
"grid_column": null,
|
||
"grid_gap": null,
|
||
"grid_row": null,
|
||
"grid_template_areas": null,
|
||
"grid_template_columns": null,
|
||
"grid_template_rows": null,
|
||
"height": null,
|
||
"justify_content": null,
|
||
"justify_items": null,
|
||
"left": null,
|
||
"margin": null,
|
||
"max_height": null,
|
||
"max_width": null,
|
||
"min_height": null,
|
||
"min_width": null,
|
||
"object_fit": null,
|
||
"object_position": null,
|
||
"order": null,
|
||
"overflow": null,
|
||
"padding": null,
|
||
"right": null,
|
||
"top": null,
|
||
"visibility": null,
|
||
"width": null
|
||
}
|
||
},
|
||
"8e7ee7687a99410d88a98a74ecfcea99": {
|
||
"model_module": "@jupyter-widgets/controls",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "HTMLModel",
|
||
"state": {
|
||
"_dom_classes": [],
|
||
"_model_module": "@jupyter-widgets/controls",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "HTMLModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/controls",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "HTMLView",
|
||
"description": "",
|
||
"description_allow_html": false,
|
||
"layout": "IPY_MODEL_754800f7feb04acea977696e4787d1ff",
|
||
"placeholder": "",
|
||
"style": "IPY_MODEL_24dd93300e0442788ee6cc1310e5bf14",
|
||
"tabbable": null,
|
||
"tooltip": null,
|
||
"value": "100%"
|
||
}
|
||
},
|
||
"e6398d4027c9459a97965b9d91ae484f": {
|
||
"model_module": "@jupyter-widgets/base",
|
||
"model_module_version": "2.0.0",
|
||
"model_name": "LayoutModel",
|
||
"state": {
|
||
"_model_module": "@jupyter-widgets/base",
|
||
"_model_module_version": "2.0.0",
|
||
"_model_name": "LayoutModel",
|
||
"_view_count": null,
|
||
"_view_module": "@jupyter-widgets/base",
|
||
"_view_module_version": "2.0.0",
|
||
"_view_name": "LayoutView",
|
||
"align_content": null,
|
||
"align_items": null,
|
||
"align_self": null,
|
||
"border_bottom": null,
|
||
"border_left": null,
|
||
"border_right": null,
|
||
"border_top": null,
|
||
"bottom": null,
|
||
"display": null,
|
||
"flex": null,
|
||
"flex_flow": null,
|
||
"grid_area": null,
|
||
"grid_auto_columns": null,
|
||
"grid_auto_flow": null,
|
||
"grid_auto_rows": null,
|
||
"grid_column": null,
|
||
"grid_gap": null,
|
||
"grid_row": null,
|
||
"grid_template_areas": null,
|
||
"grid_template_columns": null,
|
||
"grid_template_rows": null,
|
||
"height": null,
|
||
"justify_content": null,
|
||
"justify_items": null,
|
||
"left": null,
|
||
"margin": null,
|
||
"max_height": null,
|
||
"max_width": null,
|
||
"min_height": null,
|
||
"min_width": null,
|
||
"object_fit": null,
|
||
"object_position": null,
|
||
"order": null,
|
||
"overflow": null,
|
||
"padding": null,
|
||
"right": null,
|
||
"top": null,
|
||
"visibility": null,
|
||
"width": null
|
||
}
|
||
}
|
||
},
|
||
"version_major": 2,
|
||
"version_minor": 0
|
||
}
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|