{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved. \n",
    "\n",
    "Licensed under the MIT License.\n",
    "\n",
    "# Use FLAML to Optimize Code Generation Performance\n",
    "\n",
    "In this notebook, we optimize OpenAI models for code generation. We use [the HumanEval benchmark](https://huggingface.co/datasets/openai_humaneval) released by OpenAI for synthesizing programs from docstrings. \n",
    "\n",
    "## Requirements\n",
    "\n",
    "FLAML requires `Python>=3.7`. To run this notebook example, please install flaml with the [autogen] option:\n",
    "```bash\n",
    "pip install flaml[autogen]==1.2.2\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-24T23:25:36.910966Z",
     "iopub.status.busy": "2023-02-24T23:25:36.910473Z",
     "iopub.status.idle": "2023-02-24T23:25:36.914554Z",
     "shell.execute_reply": "2023-02-24T23:25:36.914030Z"
    }
   },
   "outputs": [],
   "source": [
    "# %pip install flaml[autogen]==1.2.2 datasets"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set your OpenAI key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-24T23:25:36.917301Z",
     "iopub.status.busy": "2023-02-24T23:25:36.917011Z",
     "iopub.status.idle": "2023-02-24T23:25:36.923156Z",
     "shell.execute_reply": "2023-02-24T23:25:36.922619Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = \"<your OpenAI API key here>\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use Azure OpenAI, uncomment the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-24T23:25:36.925804Z",
     "iopub.status.busy": "2023-02-24T23:25:36.925423Z",
     "iopub.status.idle": "2023-02-24T23:25:36.928191Z",
     "shell.execute_reply": "2023-02-24T23:25:36.927673Z"
    }
   },
   "outputs": [],
   "source": [
    "# import openai\n",
    "# openai.api_type = \"azure\"\n",
    "# openai.api_base = \"https://<your_endpoint>.openai.azure.com/\"\n",
    "# openai.api_version = \"2023-03-15-preview\"  # change if necessary"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "First, we load the humaneval dataset. The dataset contains 164 examples. In each example, the \"prompt\" is the prompt string for eliciting the code generation (renamed into \"definition\"), \"test\" is the Python code for unit test for the example, and \"entry_point\" is the function name to be tested."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-24T23:25:36.931255Z",
     "iopub.status.busy": "2023-02-24T23:25:36.930838Z",
     "iopub.status.idle": "2023-02-24T23:25:39.148799Z",
     "shell.execute_reply": "2023-02-24T23:25:39.148113Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset openai_humaneval (/home/vscode/.cache/huggingface/datasets/openai_humaneval/openai_humaneval/1.0.0/2955cebd73602e828fa8c0a424c594e5fab4ec863b316ca98f3d8fdb6a626e75)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1fdc8853bf2a4aecaa2cd024ad99b5a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached shuffled indices for dataset at /home/vscode/.cache/huggingface/datasets/openai_humaneval/openai_humaneval/1.0.0/2955cebd73602e828fa8c0a424c594e5fab4ec863b316ca98f3d8fdb6a626e75/cache-1e8448101c1b32e8.arrow\n"
     ]
    }
   ],
   "source": [
    "import datasets\n",
    "\n",
    "seed = 41\n",
    "data = datasets.load_dataset(\"openai_humaneval\")[\"test\"].shuffle(seed=seed)\n",
    "data = data.select(range(len(data))).rename_column(\"prompt\", \"definition\").remove_columns([\"task_id\", \"canonical_solution\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-24T23:25:39.164187Z",
     "iopub.status.busy": "2023-02-24T23:25:39.163867Z",
     "iopub.status.idle": "2023-02-24T23:25:39.169009Z",
     "shell.execute_reply": "2023-02-24T23:25:39.168427Z"
    }
   },
   "outputs": [],
   "source": [
    "from flaml.autogen.code_utils import eval_function_completions, implement\n",
    "from flaml import oai"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The `implement` function will first generate assertion statements for a problem. Then, it uses the assertions to select the generated responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-24T23:25:39.179030Z",
     "iopub.status.busy": "2023-02-24T23:25:39.178624Z",
     "iopub.status.idle": "2023-02-24T23:25:40.584410Z",
     "shell.execute_reply": "2023-02-24T23:25:40.583802Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example 0, config 1, success 1\n",
      "Example 1, config 0, success 2\n",
      "Example 2, config 0, success 3\n",
      "Example 3, config 2, success 4\n",
      "Example 4, config 2, success 5\n",
      "Example 5, config 4, success 6\n",
      "Example 6, config 4, success 6\n",
      "Example 7, config 2, success 7\n",
      "Example 8, config 2, success 8\n",
      "Example 9, config 0, success 9\n",
      "Example 10, config 1, success 10\n",
      "Example 11, config 0, success 10\n",
      "Example 12, config 2, success 11\n",
      "Example 13, config 2, success 12\n",
      "Example 14, config 0, success 13\n",
      "Example 15, config 2, success 14\n",
      "Example 16, config 0, success 15\n",
      "Example 17, config 1, success 15\n",
      "Example 18, config 1, success 16\n",
      "Example 19, config 3, success 17\n",
      "Example 20, config 2, success 18\n",
      "Example 21, config 2, success 19\n",
      "Example 22, config 2, success 19\n",
      "Example 23, config 2, success 20\n",
      "Example 24, config 0, success 21\n",
      "Example 25, config 0, success 22\n",
      "Example 26, config 4, success 23\n",
      "Example 27, config 2, success 24\n",
      "Example 28, config 4, success 24\n",
      "Example 29, config 2, success 25\n",
      "Example 30, config 2, success 26\n",
      "Example 31, config 0, success 27\n",
      "Example 32, config 0, success 28\n",
      "Example 33, config 0, success 29\n",
      "Example 34, config 2, success 30\n",
      "Example 35, config 1, success 30\n",
      "Example 36, config 0, success 31\n",
      "Example 37, config 0, success 32\n",
      "Example 38, config 0, success 33\n",
      "Example 39, config 2, success 34\n",
      "Example 40, config 0, success 35\n",
      "Example 41, config 0, success 36\n",
      "Example 42, config 3, success 37\n",
      "Example 43, config 0, success 38\n",
      "Example 44, config 2, success 39\n",
      "Example 45, config 2, success 40\n",
      "Example 46, config 2, success 40\n",
      "Example 47, config 0, success 41\n",
      "Example 48, config 3, success 42\n",
      "Example 49, config 2, success 43\n",
      "Example 50, config 1, success 44\n",
      "Example 51, config 2, success 45\n",
      "Example 52, config 3, success 46\n",
      "Example 53, config 2, success 47\n",
      "Example 54, config 0, success 48\n",
      "Example 55, config 2, success 49\n",
      "Example 56, config 2, success 50\n",
      "Example 57, config 2, success 51\n",
      "Example 58, config 0, success 52\n",
      "Example 59, config 1, success 53\n",
      "Example 60, config 0, success 53\n",
      "Example 61, config 0, success 54\n",
      "Example 62, config 1, success 55\n",
      "Example 63, config 1, success 56\n",
      "Example 64, config 0, success 57\n",
      "Example 65, config 2, success 58\n",
      "Example 66, config 2, success 59\n",
      "Example 67, config 2, success 60\n",
      "Example 68, config 2, success 61\n",
      "Example 69, config 4, success 61\n",
      "Example 70, config 2, success 62\n",
      "Example 71, config 0, success 63\n",
      "Example 72, config 0, success 64\n",
      "Example 73, config 0, success 65\n",
      "Example 74, config 0, success 66\n",
      "Example 75, config 0, success 67\n",
      "Example 76, config 1, success 68\n",
      "Example 77, config 2, success 69\n",
      "Example 78, config 1, success 70\n",
      "Example 79, config 4, success 70\n",
      "Example 80, config 2, success 71\n",
      "Example 81, config 2, success 72\n",
      "Example 82, config 0, success 72\n",
      "Example 83, config 0, success 73\n",
      "Example 84, config 4, success 73\n",
      "Example 85, config 3, success 74\n",
      "Example 86, config 0, success 75\n",
      "Example 87, config 2, success 76\n",
      "Example 88, config 2, success 77\n",
      "Example 89, config 1, success 78\n",
      "Example 90, config 0, success 79\n",
      "Example 91, config 2, success 80\n",
      "Example 92, config 1, success 81\n",
      "Example 93, config 0, success 82\n",
      "Example 94, config 0, success 83\n",
      "Example 95, config 0, success 84\n",
      "Example 96, config 2, success 85\n",
      "Example 97, config 2, success 86\n",
      "Example 98, config 2, success 87\n",
      "Example 99, config 4, success 88\n",
      "Example 100, config 0, success 89\n",
      "Example 101, config 0, success 90\n",
      "Example 102, config 2, success 91\n",
      "Example 103, config 4, success 91\n",
      "Example 104, config 2, success 92\n",
      "Example 105, config 2, success 93\n",
      "Example 106, config 4, success 93\n",
      "Example 107, config 2, success 94\n",
      "Example 108, config 0, success 95\n",
      "Example 109, config 2, success 96\n",
      "Example 110, config 0, success 97\n",
      "Example 111, config 0, success 98\n",
      "Example 112, config 2, success 99\n",
      "Example 113, config 0, success 99\n",
      "Example 114, config 2, success 100\n",
      "Example 115, config 2, success 100\n",
      "Example 116, config 0, success 101\n",
      "Example 117, config 0, success 102\n",
      "Example 118, config 0, success 103\n",
      "Example 119, config 4, success 104\n",
      "Example 120, config 2, success 105\n",
      "Example 121, config 2, success 106\n",
      "Example 122, config 0, success 107\n",
      "Example 123, config 2, success 108\n",
      "Example 124, config 1, success 109\n",
      "Example 125, config 0, success 110\n",
      "Example 126, config 1, success 111\n",
      "Example 127, config 4, success 111\n",
      "Example 128, config 2, success 112\n",
      "Example 129, config 2, success 113\n",
      "Example 130, config 0, success 114\n",
      "Example 131, config 2, success 115\n",
      "Example 132, config 0, success 116\n",
      "Example 133, config 2, success 117\n",
      "Example 134, config 1, success 118\n",
      "Example 135, config 1, success 119\n",
      "Example 136, config 0, success 120\n",
      "Example 137, config 0, success 121\n",
      "Example 138, config 2, success 122\n",
      "Example 139, config 2, success 123\n",
      "Example 140, config 2, success 124\n",
      "Example 141, config 2, success 125\n",
      "Example 142, config 2, success 126\n",
      "Example 143, config 0, success 127\n",
      "Example 144, config 0, success 128\n",
      "Example 145, config 2, success 129\n",
      "Example 146, config 1, success 130\n",
      "Example 147, config 1, success 131\n",
      "Example 148, config 2, success 132\n",
      "Example 149, config 0, success 133\n",
      "Example 150, config 0, success 134\n",
      "Example 151, config 2, success 135\n",
      "Example 152, config 0, success 136\n",
      "Example 153, config 2, success 137\n",
      "Example 154, config 2, success 138\n",
      "Example 155, config 2, success 139\n",
      "Example 156, config 0, success 140\n",
      "Example 157, config 0, success 141\n",
      "Example 158, config 4, success 142\n",
      "Example 159, config 2, success 143\n",
      "Example 160, config 0, success 144\n",
      "Example 161, config 0, success 145\n",
      "Example 162, config 0, success 146\n",
      "Example 163, config 4, success 147\n",
      "Success rate: 0.896\n",
      "Average cost: 0.00818\n"
     ]
    }
   ],
   "source": [
    "prompt = \"# Python 3{definition}\"\n",
    "stops = [[\"\\nclass\", \"\\ndef\", \"\\nif\", \"\\nprint\"], None]\n",
    "configs = [{\"model\": 'gpt-3.5-turbo', \"prompt\": prompt, \"stop\": stops[1], \"temperature\": 0, \"seed\": 0}, {\"model\": 'gpt-3.5-turbo', \"prompt\": prompt, \"stop\": stops[0], \"n\": 7, \"seed\": 0}, {\"model\": 'gpt-4', \"prompt\": prompt, \"stop\": stops[1], \"temperature\": 0, \"seed\": 1}, {\"model\": 'gpt-4', \"prompt\": prompt, \"stop\": stops[0], \"n\": 2, \"seed\": 2}, {\"model\": 'gpt-4', \"prompt\": prompt, \"stop\": stops[0], \"n\": 1, \"seed\": 2}]\n",
    "oai.Completion.set_cache(0)\n",
    "oai.Completion.retry_timeout = 600\n",
    "cost = 0\n",
    "success = 0\n",
    "for i, d in enumerate(data):\n",
    "    response, cost_i, j = implement(d[\"definition\"], configs)\n",
    "    metrics = eval_function_completions(responses=[response], use_docker=False, **d)\n",
    "    success += metrics[\"success\"]\n",
    "    cost += cost_i\n",
    "    print(f\"Example {i}, config {j}, success {success}\")\n",
    "print(f\"Success rate: {success / len(data):.3f}\")\n",
    "print(f\"Average cost: {cost / len(data):.5f}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "24dd93300e0442788ee6cc1310e5bf14": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "background": null,
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "35cd066a31b242bb87b2c106ee72e5f2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_8e7ee7687a99410d88a98a74ecfcea99",
        "IPY_MODEL_421e02a11a974b40b3ddb75382b3b640",
        "IPY_MODEL_77db9797e78b49438d21c5c8da34b4cb"
       ],
       "layout": "IPY_MODEL_47d3046236a54b0e8f9ae455a82c7e0b",
       "tabbable": null,
       "tooltip": null
      }
     },
     "3d5d106a38954af2bb3bde5777702f4e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "background": null,
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "3e1ebb31412443b0bca86a301cbdac11": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "421e02a11a974b40b3ddb75382b3b640": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_e6398d4027c9459a97965b9d91ae484f",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_3e1ebb31412443b0bca86a301cbdac11",
       "tabbable": null,
       "tooltip": null,
       "value": 1
      }
     },
     "47d3046236a54b0e8f9ae455a82c7e0b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "754800f7feb04acea977696e4787d1ff": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "77db9797e78b49438d21c5c8da34b4cb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_7b6c4e1c11e249409a1edcd63be450d8",
       "placeholder": "​",
       "style": "IPY_MODEL_3d5d106a38954af2bb3bde5777702f4e",
       "tabbable": null,
       "tooltip": null,
       "value": " 1/1 [00:00&lt;00:00, 44.40it/s]"
      }
     },
     "7b6c4e1c11e249409a1edcd63be450d8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "8e7ee7687a99410d88a98a74ecfcea99": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "2.0.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "2.0.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_allow_html": false,
       "layout": "IPY_MODEL_754800f7feb04acea977696e4787d1ff",
       "placeholder": "​",
       "style": "IPY_MODEL_24dd93300e0442788ee6cc1310e5bf14",
       "tabbable": null,
       "tooltip": null,
       "value": "100%"
      }
     },
     "e6398d4027c9459a97965b9d91ae484f": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}