LLMs-from-scratch/appendix-D/01_main-chapter-code/appendix-D.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9a5936bd-af17-4a7e-a4d2-e910411708ea",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af53bcb1-ff9d-49c7-a0bc-5b8d32ff975b",
   "metadata": {},
   "source": [
    "## Appendix D: Adding Bells and Whistles to the Training Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f58c142-9434-49af-b33a-356b80a45b86",
   "metadata": {},
   "source": [
    "- In this appendix, we add a few more advanced features to the training function, which are used in typical pretraining and finetuning; finetuning is covered in chapters 6 and 7\n",
    "- The next three sections below discuss learning rate warmup, cosine decay, and gradient clipping\n",
    "- The final section adds these techniques to the training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "744def4f-c03f-42ee-97bb-5d7d5b89b723",
   "metadata": {},
   "source": [
    "- We start by initializing a model reusing the code from chapter 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8755bd5e-bc06-4e6e-9e63-c7c82b816cbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch version: 2.9.0\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "import torch\n",
    "\n",
    "print(\"torch version:\", version(\"torch\"))\n",
    "\n",
    "\n",
    "from previous_chapters import GPTModel\n",
    "# If the `previous_chapters.py` file is not available locally,\n",
    "# you can import it from the `llms-from-scratch` PyPI package.\n",
    "# For details, see: https://github.com/rasbt/LLMs-from-scratch/tree/main/pkg\n",
    "# E.g.,\n",
    "# from llms_from_scratch.ch04 import GPTModel\n",
    "\n",
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,   # Vocabulary size\n",
    "    \"context_length\": 256, # Shortened context length (orig: 1024)\n",
    "    \"emb_dim\": 768,        # Embedding dimension\n",
    "    \"n_heads\": 12,         # Number of attention heads\n",
    "    \"n_layers\": 12,        # Number of layers\n",
    "    \"drop_rate\": 0.1,      # Dropout rate\n",
    "    \"qkv_bias\": False      # Query-key-value bias\n",
    "}\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")\n",
    "elif torch.backends.mps.is_available():\n",
    "    # Use PyTorch 2.9 or newer for stable mps results\n",
    "    major, minor = map(int, torch.__version__.split(\".\")[:2])\n",
    "    if (major, minor) >= (2, 9):\n",
    "        device = torch.device(\"mps\")\n",
    "    else:\n",
    "        device = torch.device(\"cpu\")\n",
    "else:\n",
    "    device = torch.device(\"cpu\")\n",
    "\n",
    "print(\"Device:\", device)\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.eval();  # Disable dropout during inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51574e57-a098-412c-83e8-66dafa5a0b99",
   "metadata": {},
   "source": [
    "- Next, using the same code we used in chapter 5, we initialize the data loaders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "386ca110-2bb4-42f1-bd54-8836df80acaa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nimport os\\nimport urllib.request\\n\\nif not os.path.exists(file_path):\\n    with urllib.request.urlopen(url) as response:\\n        text_data = response.read().decode(\\'utf-8\\')\\n    with open(file_path, \"w\", encoding=\"utf-8\") as file:\\n        file.write(text_data)\\nelse:\\n    with open(file_path, \"r\", encoding=\"utf-8\") as file:\\n        text_data = file.read()\\n'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import requests\n",
    "\n",
    "file_path = \"the-verdict.txt\"\n",
    "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n",
    "\n",
    "if not os.path.exists(file_path):\n",
    "    response = requests.get(url, timeout=30)\n",
    "    response.raise_for_status()\n",
    "    text_data = response.text\n",
    "    with open(file_path, \"w\", encoding=\"utf-8\") as file:\n",
    "        file.write(text_data)\n",
    "else:\n",
    "    with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "        text_data = file.read()\n",
    "\n",
    "# The book originally used the following code below\n",
    "# However, urllib uses older protocol settings that\n",
    "# can cause problems for some readers using a VPN.\n",
    "# The `requests` version above is more robust\n",
    "# in that regard.\n",
    "\n",
    "\"\"\"\n",
    "import os\n",
    "import urllib.request\n",
    "\n",
    "if not os.path.exists(file_path):\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        text_data = response.read().decode('utf-8')\n",
    "    with open(file_path, \"w\", encoding=\"utf-8\") as file:\n",
    "        file.write(text_data)\n",
    "else:\n",
    "    with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "        text_data = file.read()\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae96992b-536a-4684-a924-658b9ffb7e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import create_dataloader_v1\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch02 import create_dataloader_v1\n",
    "\n",
    "\n",
    "# Train/validation ratio\n",
    "train_ratio = 0.90\n",
    "split_idx = int(train_ratio * len(text_data))\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = create_dataloader_v1(\n",
    "    text_data[:split_idx],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"context_length\"],\n",
    "    stride=GPT_CONFIG_124M[\"context_length\"],\n",
    "    drop_last=True,\n",
    "    shuffle=True,\n",
    "    num_workers=0\n",
    ")\n",
    "\n",
    "val_loader = create_dataloader_v1(\n",
    "    text_data[split_idx:],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"context_length\"],\n",
    "    stride=GPT_CONFIG_124M[\"context_length\"],\n",
    "    drop_last=False,\n",
    "    shuffle=False,\n",
    "    num_workers=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939c08d8-257a-41c6-b842-019f7897ac74",
   "metadata": {},
   "source": [
    "## D.1 Learning rate warmup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fafcd30-ddf7-4a9f-bcf4-b13c052b3133",
   "metadata": {},
   "source": [
    "- When training complex models like LLMs, implementing learning rate warmup can help stabilize the training\n",
    "- In learning rate warmup, we gradually increase the learning rate from a very low value (`initial_lr`) to a user-specified maximum (`peak_lr`)\n",
    "- This way, the model will start the training with small weight updates, which helps decrease the risk of large destabilizing updates during the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2bb4790b-b8b6-4e9e-adf4-704a04b31ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 15\n",
    "initial_lr = 0.0001\n",
    "peak_lr = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf3a8da-abc4-4b80-a5d8-f1cc1c7cc5f3",
   "metadata": {},
   "source": [
    "- Typically, the number of warmup steps is between 0.1% to 20% of the total number of steps\n",
    "- We can compute the increment as the difference between the `peak_lr` and `initial_lr` divided by the number of warmup steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5f6d083f-1b25-4c23-b46d-ef7783446690",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27\n"
     ]
    }
   ],
   "source": [
    "total_steps = len(train_loader) * n_epochs\n",
    "warmup_steps = int(0.2 * total_steps) # 20% warmup\n",
    "print(warmup_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b6bbdc8-0104-459e-a7ed-b08be8578709",
   "metadata": {},
   "source": [
    "- Note that the print book accidentally includes a leftover code line, `warmup_steps = 20`, which is not used and can be safely ignored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e075f80e-a398-4809-be1d-8019e1d31c90",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "global_step = -1\n",
    "track_lrs = []\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        if global_step < warmup_steps:\n",
    "            lr = initial_lr + global_step * lr_increment\n",
    "        else:\n",
    "            lr = peak_lr\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.param_groups[0][\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights\n",
    "        # ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cb6da121-eeed-4023-bdd8-3666c594b4ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAANRFJREFUeJzt3Ql4U1XaB/C3e6F0oS10YytQLVBoFWwt9htEliJlE5Xl4xNEhsWpyjIjyq5+OBWQkYFBgdEPnGdksY4i7RS0grJDWUqhlE0piy1doQst3ZL7Pe8pySSQ1hbS3nuT/+958iT35iQ5N2nz5pz7nnNsJEmSCAAAABTHVu4KAAAAgGkI0gAAAAqFIA0AAKBQCNIAAAAKhSANAACgUAjSAAAACoUgDQAAoFAI0gAAAAplL3cF1Eqr1VJ2dja5urqSjY2N3NUBAACZ8dxgpaWl5O/vT7a25mkDI0g/IA7Q7du3N8uHAAAAluP69evUrl07szwXgvQD4ha07sNwc3Mzy4cBAADqVVJSIhpvuvhgDgjSD0jXxc0BGkEaAAB0zHkKFIljAAAACoUgDQAAoFAI0gAAAAole5Beu3YtderUiZydnSkiIoJSUlLqLR8fH0/BwcGifM+ePSkpKcno/q+//poGDx5MXl5e4rzAqVOn7nuOiooKio2NFWVatWpFzz//POXm5pr92AAAAFQbpLdt20Zz5syhJUuW0MmTJyk0NJSio6MpLy/PZPlDhw7R+PHjacqUKZSamkqjRo0Sl/T0dH2ZsrIyioqKomXLltX5urNnz6aEhAQR8Pfu3SuGU40ePbpJjhEAAOBB2Ug8+lom3HJ+4okn6G9/+5t+ghBOX3/99dfp7bffvq/82LFjRRBOTEzU73vyyScpLCyM1q1bZ1T2ypUrFBgYKII5369TXFxMbdq0oc2bN9MLL7wg9p0/f566detGhw8fFs/X0FR7d3d38XzI7gYAgJImiAuyDcGqqqqiEydO0Lx58/T7eIaWgQMHimBpCu/nlrchbnlv3769wa/Lr1ldXS1eR4e7zzt06FBvkK6srBQXww8Dmo5WK9H7SefofA7eZwBoeva2tvT5K+GkNLIF6YKCAtJoNOTj42O0n7e5ZWtKTk6OyfK8v6G4rKOjI3l4eDTqeeLi4ujdd99t8OvAwzlyuZA+O5CJtxEAmoWjnewpWiZhMpMG4ha/YSteN7MMNI0dadni+pngtjQyzB9vMwA0KVuFrsEgW5D29vYmOzu7+7KqedvX19fkY3h/Y8rX9Rzc1V5UVGTUmv6t53FychIXaHpVNVramV7bq/H7qEDq29UbbzsAWCXZ2vfc5dy7d2/avXu3fh8njvF2ZGSkycfwfsPyLDk5uc7ypvBrOjg4GD3PhQsX6Nq1a416Hmg6+y/lU/Gdamrr6kQRnb3wVgOA1ZK1u5u7jydNmkR9+vSh8PBwWrVqlcjenjx5srh/4sSJFBAQIM4Hs5kzZ1K/fv1o5cqVFBMTQ1u3bqXjx4/Thg0b9M958+ZNEXB5WJUuADNuJfOFM+94CBe/tqenp8jA42xyDtANzeyG5unqjunlR3a2yuyCAgCw+CDNQ6ry8/Np8eLFImmLh0rt2rVLnxzGwdZwTc6+ffuKoVMLFy6k+fPnU1BQkMjsDgkJ0ZfZsWOHPsizcePGiWsei/3OO++I2x999JF4Xp7EhDO2OUP8448/bsYjh7rcqdJQckbtKY3hoTgXDQDWTdZx0mqGcdJNIyEtm17fkkrtPVvQvjf7m3U1GQAAtcUFZeacg1UHaTa8lz8CNABYPQRpUAxOFvvpQr64PQLDrgAAEKRBOb47m0NVGi0FtW1Fj/q4yl0dAADZoSUNiuvqHhGKrm4AAIYgDYpQcLuSDv1SKG4jqxsAoBaCNChC0pkbpNFK1KudO3XydpG7OgAAioAgDYrr6gYAgFoI0iC7rKI7dOzKLeIh0cN6IUgDAOggSIPsEu+2op/o5Em+7s5yVwcAQDEQpEExc3WjqxsAwBiCNMjql/zbdDa7hOxtbWhoTz98GgAABhCkQREJY1FB3uTp4ohPAwDAAII0yIbXdtlhMFc3AAAYQ5AG2WTcKKHL+WXkZG9Lg3vULk8KAAD/gSANstG1op8Jbkuuzg74JAAA7oEgDbLQaiVKTLshbmMaUAAA0xCkQRap12+JSUxaOdmLljQAANwPQRpkseNUbVf34O4+5Oxgh08BAMAEBGlodjUaLf37DLq6AQB+C4I0NLsjl29Swe0qat3SQYyPBgAA0xCkodntSMsS18/29CMHO/wJAgDUBd+Q0KwqazS0Mz1H3MZc3QAA9UOQhma190I+lVbUkI+bk1j1CgAA6oYgDc0q4XRtwhivG21na4N3HwCgHgjS0GzKq2roh4xccRtd3QAAvw1BGppNckYu3anWUEevltSrnTveeQCA34AgDc0mQTcNaC9/srFBVzcAwG9BkIZmUVxeTXsv5onbI8KwLCUAQEMgSEOz2HX2BlVrJHrUx5Ue8XHFuw4A0AAI0tCsXd1oRQMAqChIr127ljp16kTOzs4UERFBKSkp9ZaPj4+n4OBgUb5nz56UlJRkdL8kSbR48WLy8/OjFi1a0MCBA+nSpUtGZS5evEgjR44kb29vcnNzo6ioKPrxxx+b5PiAKK+0gg79UqA/Hw0AACoI0tu2baM5c+bQkiVL6OTJkxQaGkrR0dGUl1d77vJehw4dovHjx9OUKVMoNTWVRo0aJS7p6en6MsuXL6fVq1fTunXr6OjRo+Ti4iKes6KiQl9m2LBhVFNTQ3v27KETJ06I1+V9OTm1M2GBeSWdvkFaiSisvQd18GqJtxcAoKEkGYWHh0uxsbH6bY1GI/n7+0txcXEmy48ZM0aKiYkx2hcRESFNnz5d3NZqtZKvr6+0YsUK/f1FRUWSk5OTtGXLFrGdn58v8WHv27dPX6akpETsS05ObnDdi4uLxWP4Gur33NoDUse3EqVP91/GWwUAFqu4CeKCbC3pqqoq0Yrl7mgdW1tbsX348GGTj+H9huUZt5J15TMzM0Vr2LCMu7u76EbXlfHy8qJHH32U/vGPf1BZWZloUa9fv57atm1LvXv3bqKjtV7Xb5bTyWtFxCOuhvXyk7s6AACqYi/XCxcUFJBGoyEfHx+j/bx9/vx5k4/hAGyqvK6bWnddXxken/vDDz+IbnJXV1fxw4AD9K5du6h169Z11reyslJcdEpKShp9zNYo8e40oE8GepGPm7Pc1QEAUBXZE8eaGyeWxcbGisC8f/9+kajGAXv48OF040ZtQDElLi5OtMp1l/bt2zdrvdVqR1q2uB4eioQxAADVBGnOrLazs6Pc3Nq5nHV429fX1+RjeH995XXX9ZXhZLHExETaunUrPfXUU/T444/Txx9/LDLBP//88zrrO2/ePCouLtZfrl+//oBHbj1+ziulczdKyN7Whp4NMf2ZAgCAAoO0o6OjOAe8e/du/T6tViu2IyMjTT6G9xuWZ8nJyfrygYGBIhgbluFuac7y1pUpLy8X19zNbYi3+fXr4uTkJIZrGV6gfjvujo3+3SNtqLWLI94uAAC1nJNmPPxq0qRJ1KdPHwoPD6dVq1aJZK7JkyeL+ydOnEgBAQGiq5nNnDmT+vXrRytXrqSYmBjRGj5+/Dht2LBBf7551qxZtHTpUgoKChJBe9GiReTv7y+6tBkHaz73zK/L46m5Bf33v/9dJJ3xc4L5Tisk6Lu6kTAGAKC6ID127FjKz88XwZITu8LCwkQCly7x69q1a0Yt3r59+9LmzZtp4cKFNH/+fBGIt2/fTiEhIfoyc+fOFYF+2rRpVFRUJCYq4efkyU903ey8vWDBAnrmmWeourqaevToQd9++60YLw3mcTa7hDILysjJ3pYGdUdXNwDAg7DhcVh46xqPu9E5gYzPT6Pr+35/TjpHG/ZdppiefrR2wuP4EwMAi1fSBHHB6rK7oelptYZd3cjqBgB4UAjSYHbHr96iG8UV5OpkT08/2gbvMADAA0KQBrPTtaIH9/AlZwc7vMMAAA8IQRrMqkajpaQzWJYSAMAcEKTBrA7+UkiFZVXk6eJIfbt44d0FAHgICNLQJF3dQ3v6koMd/rwAAB4GvkXBbCqqNfRdeu1CJiNCA/DOAgA8JARpMJufLuRTaWUN+bk7U5+Oda8oBgAADYMgDWaTcLq2q5vXjba1tcE7CwDwkBCkwSzKKmto97na1cfQ1Q0AIGOQrqmpoR9++IHWr19PpaWlYl92djbdvn3bTNUCtUnOyKWKai0FertQSABWCAMAkGWBjatXr9KQIUPE4heVlZU0aNAgcnV1pWXLlontdevWmaVioC47dNOA9vITq5EBAIAMLWleLpKXlrx165ZY5lHnueeeu2+tZ7AOReVVtO9ivrg9IgxzdQMAyNaS3r9/Px06dIgcHR2N9nfq1ImysrLMVjFQj53pOVSjlaibnxt1besqd3UAAKy3Ja3Vakmj0dy3/9dffxXd3mB9dpzSrXjlJ3dVAACsO0gPHjyYVq1apd/m84+cMLZkyRIaOnSouesHCpdXUkFHMgvF7eG90NUNACBrd/fKlSspOjqaunfvThUVFfTf//3fdOnSJfL29qYtW7aYtXKgfImnb5AkET3ewYPae7aUuzoAANYdpNu1a0dpaWm0bds2cc2t6ClTptCECROMEsnAyrK6Q9GKBgCQPUjv27eP+vbtK4IyXwzHTvN9v/vd78xdR1Co6zfL6dT1IuLJxWJ64Xw0AIDs56T79+9PN2/evG9/cXGxuA+srxUd2cWL2ro6y10dAACL0+ggLUmSyckqCgsLycXFxVz1AhUtS4mEMQAAmbu7R48eLa45QL/88svk5OSkv4+HZJ0+fVp0g4N1uJhbSudzSsnBzoaeDUFXNwCArEHa3d1d35Lm8dCGSWI8scmTTz5JU6dObZJKgnJb0f0eaUPuLR3krg4AgHUH6Y0bN+pnFvvTn/6Erm0rxj/UkNUNAKDA7G6etASs2+lfi+lqYTk5O9jSwG4+clcHAMBiNTpIs6+++oq+/PJLsRJWVVWV0X0nT540V91A4V3dHKBdnB7oTwgAAJoiu3v16tU0efJk8vHxodTUVAoPDycvLy+6fPkyPfvss419OlAZrVYSs4yxEZjABABAWUH6448/pg0bNtCaNWtEwtjcuXMpOTmZ3njjDTFWGixbypWblFNSQa7O9tTv0TZyVwcAwKI1OkhzF7duqBVneJeWlorbL730EubutqKu7iE9fMnJ3k7u6gAAWLRGB2lfX1/9jGMdOnSgI0eOiNuZmZki6xcsV7VGS0ln7nZ1h2GubgAAxQXpZ555hnbs2CFu87np2bNn06BBg2js2LH03HPPNUUdQSEO/FxAt8qrybuVI0V29pK7OgAAFq/RQZrPRy9YsEDcjo2Npf/7v/+jbt260XvvvUeffPJJoyuwdu1aMfba2dmZIiIiKCUlpd7y8fHxFBwcLMr37NmTkpKSjO7n1vzixYvJz89PdMcPHDhQLKV5r3//+9/i9bhM69atadSoUY2uu7VJOFXb1T20px/Z2zX6TwcAABqpUd+0vNLV0qVLKScnR79v3LhxIuP79ddfF4lkjcHLXc6ZM0eMveahW6GhoWKt6ry8PJPlDx06ROPHjxdLY3JmOQdWvqSnp+vLLF++XNRn3bp1dPToUTHpCj8nr32t869//UucQ+eeAF5u8+DBg2JdbKhbRbWGvs/IFbeR1Q0A0EykRnJxcZEyMzMlcwgPD5diY2P12xqNRvL395fi4uJMlh8zZowUExNjtC8iIkKaPn26uK3VaiVfX19pxYoV+vuLiookJycnacuWLWK7urpaCggIkD799NOHqntxcTGfgBfX1iDpdLbU8a1EqW/cbkmj0cpdHQAAxWmKuNDoPssBAwbQ3r17H/rHAU+CcuLECdEdrWNrayu2Dx8+bPIxvN+wPONWsq48J69xK9+wDM85zt3aujLcYs/KyhKv9dhjj4lucR7fbdgah/vppgEd1suPbHkBaQAAaHKNni6KA9rbb79NZ86cod69e983h/eIESMa9DwFBQVi9SyeFMUQb58/f97kYzgAmyqv637XXddXhiddYe+88w795S9/EefDV65cSU8//TRdvHiRPD09Tb52ZWWluOiUlJSQtSitqKY952tPQQzHBCYAAMoN0n/4wx/ENQe4e/Eylhx4lUyr1YprTn57/vnn9YuHtGvXTiSlTZ8+3eTj4uLi6N133yVrlJyRS5U1WurcxoV6+LvJXR0AAKth+yBBrq5LYwK0t7c32dnZUW5ubTKSDm/zWGxTeH995XXX9ZXh7m3WvXt3/f28Nnbnzp3FRC11mTdvnphRTXe5fv06WQv9ile9/MUPMQAAaB6yjaPhTHDuLt+9e7d+Hwd63o6MjDT5GN5vWJ7xlKS68oGBgSIYG5bhbmnO8taV4dfkoHzhwgV9merqarpy5Qp17NixzvryY9zc3Iwu1uBmWRUduFQgbmMCEwCA5iXrEkY8/GrSpEnUp08fsVDHqlWrqKysTAyNYhMnTqSAgADR1cxmzpxJ/fr1E+eQY2JiaOvWrXT8+HExdptxK2/WrFlimFhQUJAI2osWLSJ/f3/9OGgOrjNmzBDDvtq3by8C84oVK8R9L774omzvhVLtTL9BNVpJdHN3adNK7uoAAFgVWYM0z1KWn58vJh/hxK6wsDDatWuXPvGLu585C1uH5wzfvHkzLVy4kObPny8C8fbt2ykkJERfhhf84EA/bdo0KioqoqioKPGcPPmJDgdle3t7MVb6zp07Ivt7z549YlITMLbj7gQmSBgDAGh+NjwOS4bXVT3uRufhXXx+2lK7vnOKKyjyg93EfyEH336GAjxayF0lAACriguY2xHqlHg6WwToPh1bI0ADAKihu7uu8cF8PpiTqxo7NSgof1lKJIwBAKgkSHt4eNQ7DIfHG7/88ssiMcvwfDKoy5WCMkr7tZh4crFnQ2qHrQEAgMKD9KZNm8REIByIOSOb8cpVn3/+uUjo4kSwDz/8ULSqObkL1NvVzZ7q6k1tXJ3krg4AgFVqdJDmYMxDoMaMGaPfN3z4cLFs5Pr168UY5Q4dOtD777+PIG0JE5hgGlAAANk0uj+al4vkhSnuxft0i1jwsKf6Zu8CZTufU0IXc2+To50tRfcwPfsbAAAoMEjzBCCfffbZfft5H9/HCgsLMebYAhLG+j3ahtxbOMhdHQAAq9Xo7m4+38wzc+3cuZOeeOIJsY9n/eKVq7766iuxfezYMTFRCagPD5tPSLshbo9AVzcAgLqCNC9FyQGZzz/z0o665St55i9e9pG9+uqr5q8pNItT14vo2s1yauFgRwO6tcW7DgCgtmlBeU7sDz74wPy1AcUkjA3q7kMtHWWdNRYAwOo90Lcwz4nNw67y8vL06zPr8KIYoE4arUT/Po2ubgAA1QbphIQEmjBhAt2+fVvMTWo4sQnfRpBWr6OZhZRXWkluzvb0u0fayF0dAACr1+js7j/+8Y/0yiuviCDNLepbt27pLzdv3rT6N9QSsrp5hjFHe8wWBwAgt0Z/E2dlZdEbb7xBLVu2bJoagSyqarS0Mz1H3MZc3QAAKg3S0dHRYsgVWJYDP+dTUXk1ebdyoic7e8ldHQAAeJBz0jExMfTmm29SRkaGmArUwcHhviFaoD47TtV2dQ/r5Ud2vKoGAACoL0hPnTpVXL/33nv33ceJYxqNxjw1g2Zzp0pDyRm54jbm6gYAUHGQvnfIFajfnvN5VFaloQCPFvR4Bw+5qwMAAHchhRdoR1qWvhVd31rhAACgwJb06tWradq0aeTs7Cxu14czv0E9Siqq6ccL+eI25uoGAFAWG4lXVGjANKCc0e3l5SVu1/lkNjZ0+fJlsgYlJSXk7u5OxcXFYlIXtfrqxK/0p/g06tq2FSXP/h1a0gAACooLDWpJZ2ZmmrwNljNXN7ei0dUNAKAsOCdtxQpvV9LBnwvEbWR1AwBYQHY3D7HatGkT7d692+QCG3v27DFn/aAJJaXniEU1ega4U6C3C95rAAC1B+mZM2eKIM2TmoSEhKCLVMUS7k5ggoQxAAALCdJbt26lL7/8koYOHdo0NYJmkV10h1Ku1C6IEtPLD+86AIAlnJN2dHSkrl27Nk1toNno1o0O7+RJ/h4t8M4DAFjKUpV//etfqQEjt0AFWd3Dw/zlrgoAAJiru/vAgQP0448/0s6dO6lHjx73LbDx9ddfN/YpoZllFpTRmaxisZDG0BBfvP8AAJYSpD08POi5555rmtpAs0i424p+qqs3ebVywrsOAGAJQbqmpob69+9PgwcPJl9ftMDUiE9TGE5gAgAAFnJO2t7enmbMmEGVlZVmrcTatWupU6dOYm7wiIgISklJqbd8fHw8BQcHi/K8pnVSUtJ9gWjx4sXk5+dHLVq0oIEDB9KlS5dMPhcfS1hYmBhKdurUKbJ0526U0s95t8nR3pYG9/CRuzoAAGDOxLHw8HBKTU0lc9m2bRvNmTOHlixZQidPnqTQ0FCKjo4WE6WYcujQIRo/fjxNmTJF1GPUqFHikp6eri+zfPlysRDIunXr6OjRo+Ti4iKes6Ki4r7nmzt3Lvn7W0+LUteK7v9oG3JzNs4nAAAAhZEaadu2bVLnzp2lNWvWSIcOHZLS0tKMLo0VHh4uxcbG6rc1Go3k7+8vxcXFmSw/ZswYKSYmxmhfRESENH36dHFbq9VKvr6+0ooVK/T3FxUVSU5OTtKWLVuMHpeUlCQFBwdLZ8+e5VR1KTU1tcH1Li4uFo/ha7Xg96Zv3G6p41uJUmJattzVAQCwKMVNEBcanTg2bty4+5ak5K5i7mLma542tKGqqqroxIkTNG/ePP0+W1tb0T19+PBhk4/h/dzyNsSt5O3bt+sXAMnJyRHPocOrknA3Oj9WV//c3FyaOnWqeFzLli1/s67cLW7Yzc+rnajNyWtFlFV0h1wc7eiZ4LZyVwcAAH5Do4O0OVfBKigoEEHdx8f43Chvnz9/3uRjOACbKs/7dffr9tVVhn9QvPzyy+L8ep8+fejKlSu/Wde4uDh69913yRKyugd196EWjnZyVwcAAMwdpDt27Ehqt2bNGiotLTVqwf8WLmvYgueWdPv27UkteCGNxLuzjI3ABCYAAJYZpHUyMjLo2rVrosva0IgRIxr8HN7e3mRnZye6ng3xdl1DvHh/feV117yPs7sNy3AWt26lLu76dnIyHiPMreoJEybQ559/ft/rctl7y6vJkcuFVHC7kjxaOlBU1zZyVwcAAJoiSF++fFlMZnLmzBn9uWjGt1ljzknzPOC9e/cWy15yhjbjpS95+7XXXjP5mMjISHH/rFmz9PuSk5PFfhYYGCgCNZfRBWVu9XKW96uvviq2OfN76dKl+sdnZ2eL89qcac7nri3RjrsrXj0b4iuGXwEAgIUuVcmBkIMgX/OY5sLCQjGn94cfftjoCnAX8qRJk0Qrlod3rVq1isrKymjy5Mni/okTJ1JAQIA4J6x7/X79+tHKlSvFcpm8Ktfx48dpw4YN+h8LHMA5CAcFBYk6Llq0SAyz0v0Q6NChg1EdWrVqJa67dOlC7dq1I0tTVaOlnem1Xd3DMYEJAIDlBmnuJubuYu6q5kxsvkRFRYkgyhnfjR1DPXbsWMrPzxeTj3BiF7d+d+3apU/84i51fg2dvn370ubNm2nhwoU0f/58EYg5Q5vXtjYc+8yBftq0aVRUVCTqx8/Jk59Yo30X86mkoobaujpRRKCX3NUBAIAGsuFxWNQIrVu3FpOOcAuVW56ffvqpmCr0l19+EbN/lZeXkzXgLnQe2lVcXExubm6kZG9sSRWTmLzyVCAtHt5d7uoAAFikkiaIC41uSXOLNS0tTQRpPn/Ls3vxuWXubu7cubNZKgXmU15VQ8kZtYl2w0P/k0gHAADK1+ggzd3M3JXM3nvvPRo2bBj913/9F3l5eYnEK1CW3efy6E61hjp4tqSw9h5yVwcAAJoySHMWtE7Xrl3FpCM3b94U3eC6DG9Q3lzd3IrG5wMAoC4PPBbn559/pu+++47u3LlDnp6e5q0VmEXxnWraeyFf3EZWNwCAFQRpHm41YMAAeuSRR2jo0KF040bt0B5elYqHYYFyfHc2h6o0WnrEpxUF+yo7uQ0AAMwQpGfPnk0ODg5iaJThwhQ8lIqHOYHy5uoegbHRAADWcU76+++/F93c9076weOVr169as66wUPIL62kgz8XiNvDelnPetkAAFbdkubMblNLO3LymJrntrY0PMOYViIKbedOnbxd5K4OAAA0R5Dm4Vb/+Mc/9NucMczzbfN4aZ7UBJQ1VzcSxgAArKi7m4MxJ47xfNm8AhZPwXn27FnRkj548GDT1BIaJavoDh2/eot4RBy6ugEArKglzTOOXbx4UcyHPXLkSNH9PXr0aDFnN08TCspJGAvv5Em+7tY5XzkAgNWuJ81zky5YsMBo36+//ioWtNCtRgUKyOoOQ8IYAICamW1hYR4//dlnn5nr6eAB/ZJ/m85ml5C9rQ09G4K5ugEA1MxsQRqUlTAWFeRNni6OclcHAAAeAoK0BeFVRxNOYwITAABLgSBtQbib+3J+GTnZ29Kg7j5yVwcAAJorcYwzuOtTVFT0sHUBMyWMPRPcllydHfB+AgBYS5DmjO7fun/ixInmqBM8AK1WosTTtYudYK5uAAArC9IbN25s2prAQzl57ZaYxKSVkz31D26LdxMAwALgnLSF2HG3q3twDx9ydrCTuzoAAGAGCNIWoEajpaQztV3dmKsbAMByIEhbgMOXC6ngdhW1bulAUV295a4OAACYCYK0BU1gMrSnHznY4SMFALAU+EZXucoaDe06myNuo6sbAMCyIEir3N4L+VRaUUO+bs5i1SsAALAcCNIWktU9rJcf2drayF0dAAAwIwRpFSurrKEfzuWK2+jqBgCwPAjSKsYBuqJaSx29WlKvdvXPCAcAAOqDIG0Bc3XzNKA2NujqBgCwNAjSKlVUXkV7L+aL25irGwDAMikiSK9du5Y6depEzs7OFBERQSkpKfWWj4+Pp+DgYFG+Z8+elJSUdN+6yosXLyY/Pz9q0aIFDRw4kC5duqS//8qVKzRlyhQKDAwU93fp0oWWLFlCVVVVpBa70nOoWiNRsK8rBfm4yl0dAACwxCC9bds2mjNnjgiSJ0+epNDQUIqOjqa8vDyT5Q8dOkTjx48XQTY1NZVGjRolLunp6foyy5cvp9WrV9O6devo6NGj5OLiIp6zoqJC3H/+/HnSarW0fv16Onv2LH300Uei7Pz580ktEk7XdnUjYQwAwIJJMgsPD5diY2P12xqNRvL395fi4uJMlh8zZowUExNjtC8iIkKaPn26uK3VaiVfX19pxYoV+vuLiookJycnacuWLXXWY/ny5VJgYGCD611cXCzx28fXzS235I4U+Hai1PGtROlaYVmzvz4AADRPXJC1Jc3dyydOnBDd0Tq2trZi+/DhwyYfw/sNyzNuJevKZ2ZmUk5OjlEZXuuau9Hrek5WXFxMnp7qmAzk36dvkFYiCmvvQe09W8pdHQAAkHs96aZQUFBAGo2GfHx8jPbzNndJm8IB2FR53q+7X7evrjL3+vnnn2nNmjX04Ycf1lnXyspKcdEpKSkhJWR1AwCA5ZL9nLTcsrKyaMiQIfTiiy/S1KlT6ywXFxcnWuS6S/v27UkO12+W08lrRcQjrniWMQAAsFyyBmlvb2+ys7Oj3NzaWbN0eNvX19fkY3h/feV11w15zuzsbOrfvz/17duXNmzYUG9d582bJ7rEdZfr16+TnAljTwZ6UVs3Z1nqAAAAVhCkHR0dqXfv3rR79279Ps665u3IyEiTj+H9huVZcnKyvjwPq+JgbFiGu6Y5y9vwObkF/fTTT4vX37hxozgXXh8nJydyc3MzusghIe2GuB4Rhq5uAABLJ+s5acbDryZNmkR9+vSh8PBwWrVqFZWVldHkyZPF/RMnTqSAgADR3cxmzpxJ/fr1o5UrV1JMTAxt3bqVjh8/rm8J88xbs2bNoqVLl1JQUJAI2osWLSJ/f38xVMswQHfs2FGch87Pr50UhNXVgleCn/NK6dyNEnKws6FnQ5RbTwAAsJAgPXbsWBEkefIRTuwKCwujXbt26RO/rl27ZtTK5a7pzZs308KFC8W4Zg7E27dvp5CQEH2ZuXPnikA/bdo0KioqoqioKPGcPPmJruXNyWJ8adeu3X0ToSjVjlO1Xd2/C2pDHi0d5a4OAAA0MRseh9XUL2KJuAudE8j4/HRzdH3zx9T/w5/oSmE5rRobRqMeC2jy1wQAAHnjgtVnd6tFelaJCNDODrY0qLvx8DIAALBMCNIqsSMtS1wP6OZDLk6yn6UAAIBmgCCtAlqtRImna7O6h/dCVjcAgLVAkFaB41dv0Y3iCnJ1sqenH20jd3UAAKCZIEirqKs7OsSXnB3s5K4OAAA0EwRphavWaCnpTO2c41iWEgDAuiBIK9yhXwrpZlkVebk40lNdvOSuDgAANCMEaYXTTWAytKcf2dvh4wIAsCb41lewimoNfX8WXd0AANYKQVrBfrqQR6WVNeTn7kx9OraWuzoAANDMEKQVTLfiFSeM2drayF0dAABoZgjSCnW7soZ+OFe7JvaIUExgAgBgjRCkFSo5I4cqa7QU6O1CPfzlWbsaAADkhSCtgq5uXiMbAACsD4K0At0qq6J9F/PF7RGhfnJXBwAAZIIgrUA703OoRitRNz836trWVe7qAACATBCkFSghrXYCEySMAQBYNwRphcktqaAjmYXi9rBe6OoGALBmCNIKw+tGSxLR4x08qL1nS7mrAwAAMkKQVpgd6OoGAIC7EKQV5FphOaVdLyKeXCymFyYwAQCwdgjSCpJwujZhrG8Xb2rj6iR3dQAAQGYI0gpclnI4xkYDAACCtHJcyCmlC7ml5GBnQ0N6IKsbAADQklbc2Oh+j7Ql95YOclcHAAAUAN3dCiBJkj6rG13dAACggyCtAKd/LaZrN8uphYMdDeruI3d1AABAIRCkFUDXih7Y3YdaOtrLXR0AAFAIBGmZabQSJd4dejUc04ACAIABBGmZpWTepNySSnJztqd+j7aRuzoAAKAgCNIKmcBkSIgvOdnbyV0dAABQEEUE6bVr11KnTp3I2dmZIiIiKCUlpd7y8fHxFBwcLMr37NmTkpKS7suWXrx4Mfn5+VGLFi1o4MCBdOnSJaMyN2/epAkTJpCbmxt5eHjQlClT6Pbt29ScqjVa2nnmhrg9IjSgWV8bAACUT/YgvW3bNpozZw4tWbKETp48SaGhoRQdHU15eXkmyx86dIjGjx8vgmpqaiqNGjVKXNLT0/Vlli9fTqtXr6Z169bR0aNHycXFRTxnRUWFvgwH6LNnz1JycjIlJibSvn37aNq0adScDlwqoFvl1eTdypGe7OzZrK8NAAAqIMksPDxcio2N1W9rNBrJ399fiouLM1l+zJgxUkxMjNG+iIgIafr06eK2VquVfH19pRUrVujvLyoqkpycnKQtW7aI7YyMDIkP/dixY/oyO3fulGxsbKSsrKwG1bu4uFg8B18/qNlbU6WObyVKi7efeeDnAAAAZTBHXLiXrC3pqqoqOnHihOiO1rG1tRXbhw8fNvkY3m9YnnErWVc+MzOTcnJyjMq4u7uLbnRdGb7mLu4+ffroy3B5fm1ueZtSWVlJJSUlRpeHNePpLhTbvwu90Lv9Qz8XAABYHlmDdEFBAWk0GvLxMZ7Ag7c50JrC++srr7v+rTJt27Y1ut/e3p48PT3rfN24uDgR7HWX9u0fPrA+4uNKb0YHU8927g/9XAAAYHlkPyetFvPmzaPi4mL95fr163JXCQAALJysQdrb25vs7OwoNzfXaD9v+/r6mnwM76+vvO76t8rcm5hWU1MjMr7rel0nJyeRCW54AQAAsNgg7ejoSL1796bdu3fr92m1WrEdGRlp8jG837A84wxtXfnAwEARaA3L8PljPtesK8PXRUVF4ny4zp49e8Rr87lrAAAARZBktnXrVpF5vWnTJpF1PW3aNMnDw0PKyckR97/00kvS22+/rS9/8OBByd7eXvrwww+lc+fOSUuWLJEcHBykM2f+kyH9wQcfiOf49ttvpdOnT0sjR46UAgMDpTt37ujLDBkyRHrssceko0ePSgcOHJCCgoKk8ePHy5rFBwAA6lXcBHFB9tUcxo4dS/n5+WLyEU7aCgsLo127dukTv65duyayrnX69u1LmzdvpoULF9L8+fMpKCiItm/fTiEhIfoyc+fOpbKyMjHumVvMUVFR4jl58hOdL774gl577TUaMGCAeP7nn39ejK0GAABQChuO1HJXQo24C52zvDmJDOenAQCgpAniArK7AQAAFEr27m610nVAmGNSEwAAUL+Su/HAnB3UCNIPqLS0VFybY1ITAACwrPjg7m6eSapwTvoB8XCt7OxscnV1JRsbmwf+1cVBnidGsZTz2jgmdcDnpA74nNT1OXGiM8cDf39/o4Tnh4GW9APiD6Bdu3Zm+RAscXIUHJM64HNSB3xO6sCtZ3N/lyNxDAAAQKEQpAEAABQKQVpGPB/4kiVLxLWlwDGpAz4ndcDnpA5N+TkhcQwAAECh0JIGAABQKARpAAAAhUKQBgAAUCgEaRmtXbuWOnXqJFbn4nWsU1JSSC3i4uLoiSeeEJO5tG3blkaNGkUXLlwwKlNRUUGxsbHk5eVFrVq1EiuN5ebmkhp88MEHYlKCWbNmqfp4srKy6H/+539EnVu0aEE9e/ak48eP6+/n6Qt5BTo/Pz9x/8CBA+nSpUukVBqNhhYtWiTWjef6dunShf73f//XaBpGpR/Tvn37aPjw4WLCC/4b41X8DDWk/jdv3qQJEyaIMbkeHh40ZcoUun37NinxmKqrq+mtt94Sf3suLi6izMSJE8VkUGo9pnvNmDFDlFm1apXZjwlBWibbtm2jOXPmiIzAkydPUmhoKEVHR1NeXh6pwd69e0XAOnLkCCUnJ4t/xMGDB4slQnVmz55NCQkJFB8fL8rzP+Xo0aNJ6Y4dO0br16+nXr16Ge1X2/HcunWLnnrqKXJwcKCdO3dSRkYGrVy5klq3bq0vs3z5crFE67p16+jo0aPiS5T/DvkHiRItW7aMPvnkE/rb3/5G586dE9t8DGvWrFHNMfH/CP+/8490UxpSf/7iP3v2rPjfS0xMFAGFl+ZV4jGVl5eL7zj+ccXXX3/9tfhBP2LECKNyajomQ9988434HuRgfi+zHJPZVqaGRgkPD5diY2P12xqNRvL395fi4uJU+U7m5eWJxc737t0rtouKiiQHBwcpPj5eX+bcuXOizOHDhyWlKi0tlYKCgqTk5GSpX79+0syZM1V7PG+99ZYUFRVV5/1arVby9fWVVqxYod/Hx+nk5CRt2bJFUqKYmBjplVdeMdo3evRoacKECao8Jv77+eabb/TbDal/RkaGeNyxY8f0ZXbu3CnZ2NhIWVlZktKOyZSUlBRR7urVq6o+pl9//VUKCAiQ0tPTpY4dO0offfSR/j5zHRNa0jKoqqqiEydOiG4sw2lGefvw4cOkRrx+KvP09BTXfHzcujY8xuDgYOrQoYOij5F7B2JiYozqrdbj2bFjB/Xp04defPFFcUriscceo7///e/6+zMzMyknJ8fomHhaQz71otRj6tu3L+3evZsuXrwottPS0ujAgQP07LPPqvaYDDWk/nzNXaf82epwef4O4Za3Wr4vuHuYj0Otx6TVaumll16iN998k3r06HHf/eY6JszdLYOCggJxbs3Hx8doP2+fP3+e1Ib/WPncLXethoSEiH38RePo6Kj/JzQ8Rr5PibZu3Sq647i7+15qPJ7Lly+LrmE+rTJ//nxxXG+88YY4jkmTJunrbervUKnH9Pbbb4vFDPgHkp2dnfg/ev/990W3IlPjMRlqSP35mn90GbK3txc/kNVwjNxtz+eox48fr5/nWo3HtGzZMlFH/p8yxVzHhCANZml9pqenixaNWvFKZDNnzhTnjjiRzxLwjyf+Ff/nP/9ZbHNLmj8nPtfJQVqNvvzyS/riiy9o8+bNovVy6tQp8QORzweq9ZisCfdGjRkzRiTH8Q9ItTpx4gT99a9/FT/qH3QVxIZCd7cMvL29RSvg3sxg3vb19SU1ee2110RCxI8//mi0KhgfB3frFxUVqeIY+Z+Ok/Yef/xx8WuXL5wcxgk8fJtbMmo6HsbZwd27dzfa161bN7GcHtPVW01/h9y1yK3pcePGiWxh7m7khD4ebaDWYzLUkPrz9b0JpjU1NSKTWMnHqAvQV69eFT+GDVeLUtsx7d+/X9SXT3fpvi/4uP74xz+KETvmPCYEaRlwd2Pv3r3FuTXDVg9vR0ZGkhrwL2EO0JzZuGfPHjEkxhAfH2cVGx4jZ3RygFDiMQ4YMIDOnDkjWma6C7dCuRtVd1tNx8P49MO9w+L4XG7Hjh3Fbf7M+MvC8Ji4K5nPlyn1mDhT+N51evkHL///qPWYDDWk/nzNPxb5h6UO/w/ye8DnrpUcoHko2Q8//CCGBBpS2zG99NJLdPr0aaPvC+7N4R+R3333nXmPyQyJb/AAtm7dKjI2N23aJLIAp02bJnl4eEg5OTmqeD9fffVVyd3dXfrpp5+kGzdu6C/l5eX6MjNmzJA6dOgg7dmzRzp+/LgUGRkpLmphmN2txuPhDFp7e3vp/fffly5duiR98cUXUsuWLaV//vOf+jIffPCB+Lv79ttvpdOnT0sjR46UAgMDpTt37khKNGnSJJFNm5iYKGVmZkpff/215O3tLc2dO1c1x8QjCFJTU8WFv4L/8pe/iNu6TOeG1H/IkCHSY489Jh09elQ6cOCAGJEwfvx4RR5TVVWVNGLECKldu3bSqVOnjL4vKisrVXlMptyb3W2uY0KQltGaNWvEl76jo6MYknXkyBFJLfiP1tRl48aN+jL8pfKHP/xBat26tQgOzz33nPjHVGuQVuPxJCQkSCEhIeIHYXBwsLRhwwaj+3nIz6JFiyQfHx9RZsCAAdKFCxckpSopKRGfCf/fODs7S507d5YWLFhg9GWv9GP68ccfTf7v8A+Qhta/sLBQfNm3atVKcnNzkyZPniyCihKPiX9M1fV9wY9T4zE1NEib45iwChYAAIBC4Zw0AACAQiFIAwAAKBSCNAAAgEIhSAMAACgUgjQAAIBCIUgDAAAoFII0AACAQiFIAwAAKBSCNAAAgEIhSAOAXn5+Pr366qtidR8nJyex2EN0dDQdPHhQ3M/L8m3fvh3vGEAzwXrSAKD3/PPPiyU5P//8c+rcubNYIpFXZCosLMS7BCADzN0NAAIvq9e6dWv66aefqF+/fve9K7xOLq+Zq8NLXl65ckXc/vbbb+ndd9+ljIwMsWTfpEmTaMGCBWKdXfFFY2NDH3/8Me3YsUM8P691vXz5cnrhhRfw7gPUA93dACC0atVKXLg7u7Ky8r535dixY+J648aNdOPGDf32/v37aeLEiTRz5kwRpNevX0+bNm2i999/3+jxixYtEi31tLQ0sU73uHHj6Ny5c3j3AeqBljQA6P3rX/+iqVOn0p07d+jxxx8XLWoOpr169ar9wrCxoW+++YZGjRqlf8zAgQNpwIABNG/ePP2+f/7znzR37lzKzs7WP27GjBn0ySef6Ms8+eST4jW4hQ0ApqElDQB63NLlwMrd0kOGDBFd0xxIuWVcF24Zv/fee/qWOF840HNru7y8XF8uMjLS6HG8jZY0QP2QOAYARpydnWnQoEHiwl3Uv//972nJkiX08ssvm3ynbt++Lc5Hjx492uRzAcCDQ0saAOrVvXt3KisrE7cdHBxIo9EY3c8t7QsXLlDXrl3vu9ja/ucr5siRI0aP4+1u3brh3QeoB1rSACDwMKsXX3yRXnnlFXEO2tXVlY4fPy6ysEeOHKnP8OYhWU899ZQYR83Z4IsXL6Zhw4aJsdWcrc2BmbvA09PTaenSpfp3Nz4+nvr06UNRUVH0xRdfUEpKCn322Wd49wHqgcQxABA4o/udd96h77//nn755Reqrq6m9u3bi8A9f/58atGiBSUkJNCcOXPE0KuAgAD9EKzvvvtOnJdOTU0Vre3g4GDRTc7npsUXjY0NrV27VmSO79u3TwzBWrZsGY0ZMwbvPkA9EKQBoMmZygoHgN+Gc9IAAAAKhSANAACgUEgcA4AmJ0kS3mWAB4CWNAAAgEIhSAMAACgUgjQAAIBCIUgDAAAoFII0AACAQiFIAwAAKBSCNAAAgEIhSAMAACgUgjQAAAAp0/8DaegWV0FT+LQAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "total_training_steps = len(train_loader) * n_epochs\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "plt.tight_layout(); plt.savefig(\"1.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3996b6-3f7a-420a-8584-c5760249f3d8",
   "metadata": {},
   "source": [
    "## D.2 Cosine decay"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5216214-de79-40cf-a733-b1049a73023c",
   "metadata": {},
   "source": [
    "- Another popular technique for training complex deep neural networks is cosine decay, which also adjusts the learning rate across training epochs\n",
    "- In cosine decay, the learning rate follows a cosine curve, decreasing from its initial value to near zero following a half-cosine cycle\n",
    "- This gradual reduction is designed to slow the pace of learning as the model begins to improve its weights; it reduces the risk of overshooting minima as the training progresses,  which is crucial for stabilizing the training in its later stages\n",
    "- Cosine decay is often preferred over linear decay for its smoother transition in learning rate adjustments, but linear decay is also used in practice (for example, [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4e8d2068-a057-4abf-b478-f02cc37191f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "min_lr = 0.1 * initial_lr\n",
    "track_lrs = []\n",
    "\n",
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "global_step = -1\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "        if global_step < warmup_steps:\n",
    "            # Linear warmup\n",
    "            lr = initial_lr + global_step * lr_increment  \n",
    "        else:\n",
    "            # Cosine annealing after warmup\n",
    "            progress = ((global_step - warmup_steps) / \n",
    "                        (total_training_steps - warmup_steps))\n",
    "            lr = min_lr + (peak_lr - min_lr) * 0.5 * (\n",
    "                1 + math.cos(math.pi * progress))\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.param_groups[0][\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0e779e33-8a44-4984-bb23-be0603dc4158",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAScBJREFUeJzt3Qd0VFXXBuCX9EIKJJBGC0UCJHQSAigiKCgg7ZciUiJKEZWi8oFSxE8/REARQYoiQQVBFGkCSpMaEjqEjnRIhxQC6fOvfWDGBAISSHJnMu+z1l0zc+dmcu4MZM85d599Sul0Oh2IiIjI6Fho3QAiIiLKH4M0ERGRkWKQJiIiMlIM0kREREaKQZqIiMhIMUgTEREZKQZpIiIiI8UgTUREZKSstG6AqcrJycHVq1fh5OSEUqVKad0cIiLSmNQGS0lJgbe3NywsCqcPzCD9iCRAV6xYsVA+BCIiKjkuXbqEChUqFMprMUg/IulB6z8MZ2fnQvkwiIjIdCUnJ6vOmz4+FAYG6UekH+KWAM0gTUREeoV5CZSJY0REREaKQZqIiMhIMUgTEREZKc2D9KxZs1ClShXY2dkhKCgIERERDzx+2bJl8PPzU8cHBARg7dq1eZ5fvnw5nnvuObi5uanrAgcPHrznNdLS0jB06FB1TOnSpdGtWzfExMQU+rkRERGZbJBeunQpRo4ciQkTJmD//v2oV68e2rZti9jY2HyP37VrF3r16oUBAwbgwIED6Ny5s9oiIyMNx6SmpqJFixaYPHnyfX/viBEjsHr1ahXwt27dqqZTde3atUjOkYiI6FGV0snsa41Iz7lJkyaYOXOmoUCIpK+/9dZbGD169D3H9+jRQwXhNWvWGPY1bdoU9evXx5w5c/Ice/78efj6+qpgLs/rJSUloVy5cli8eDH+7//+T+07ceIEatWqhbCwMPV6D5tq7+Liol6P2d1ERJRcBHFBsylYGRkZ2LdvH8aMGWPYJxVa2rRpo4JlfmS/9Lxzk573ihUrHvr3yu/MzMxUv0dPhs8rVar0wCCdnp6uttwfBhWdnBwdPll7HKdjb8DWygJ21pZ3bi3gaGuFcqVtUd7ZDuWdbOHhbAcPZ1s42HBGIRGVLJr9VYuPj0d2djY8PDzy7JfH0rPNT3R0dL7Hy/6HJcfa2NjA1dW1QK8zadIkTJw48aF/Dz2esLMJmL/jXIF+xtPZDjU8SqN6+dKoUd5J3a/l5YzStgzeRGSa+NfrIUmPP3cvXl9ZhorGqoNX1e3TNcvhudqeSM/KRlpmjrpNSctCbEo6YpLTEJeSjtjkNKRmZCM6OU1t20/HG17HohTg5+mMxlXKoFHl25uPqz3rrRORSdAsSLu7u8PS0vKerGp57Onpme/PyP6CHH+/15Ch9sTExDy96X97HVtbW7VR0ZNAvDYySt0f9FQ1BFdz+9efSbqViTOxN3AmNgWnY26oYfJTMSmISkrDsahktX0fdkEdK0Fagv8zfuXRrJo77G0si/yciIhMKkjLkHOjRo2wadMmlaGtTxyTx2+++Wa+PxMcHKyeHz58uGHfhg0b1P6HJb/T2tpavY5MvRInT57ExYsXC/Q6VHS2noxTvWUZvg70LftQP+Nib23oKecWnZSGvReuYd+F62o7ejUZVxJvYVH4RbXJde5m1dzwTC0PPO/vCffS/CJGRMZD0+FuGT7u168fGjdujMDAQEyfPl1lb4eEhKjn+/btCx8fH3U9WAwbNgwtW7bEtGnT0L59eyxZsgR79+7FvHnzDK957do1FXBlWpU+AAvpJcsmmXcyhUt+d9myZVUGnmSTS4B+2MxuKlorD93+7DrU9YKljFc/Bk8XO3So6602cTMjC2F/J2DziVhsORGLq0lp2HIyTm0frjqKJ2u4o3N9HzxXx4OJaERk3kFaplTFxcVh/PjxKmlLpkqtX7/ekBwmwTb3mpzNmjVTU6fGjh2L999/HzVq1FCZ3f7+/oZjVq1aZQjyomfPnupW5mJ/+OGH6v4XX3yhXld60pKxLRniX3/9dTGeOd3PjfQsbDp++5JGp/o+hf5GSQZ461oeapPZhydjUlTAXh8ZjcOXk/DXyTi1OdhY4rnaHujepCKCq94ujENEZFbzpE0Z50kXjd8OXMaIpYfg6+6Ize+0LNbg+HfcDaw8eBUrD17BhYSbhv01ypdGn+DK6NLAB0521sXWHiIyLUURFxikjejDICBkQYQaeh7WugZGPPuEJm+JfG89eCkRv+y7jN8OXMHNjGy139HGEl0bVkC/ZpVRvXzhrRdLRCVDMoO08WCQLnzXUjMQ+MlGZOXosOmdlqhWrjS0lpKWieX7r+D7sPP4Oy5V7ZPOfdvanhjaqjoCKrho3UQiMhIlquIY0d3WHolSAdrfx9koArSQ4e1+zaqgb3Bl7Po7AaG7zmPDsRisPxqttqeeKIehT1dDUNV/nyZGRFRQDNJkdAVMXqx3OxPbmMi18ebV3dUm869n//U3Vh26im2n4tQWWKUs3m1b86GnjBERmcRSlUTiauItRJy/poaSOxphkM7tCQ8nfNGjPra88zReDqoEG0sL1fbuc8MwIHQPTkanaN1EIiohGKTJKKy+Mze6SZWy8HKxhymo5OaA/3UJwLZRrVSwljndm07Eot2X2/DuskOqaAoR0eNgkCajIEPHolN94+5F369gigTrP0c8paqWyaRGyQxvNfUvTPnjhCqgQkT0KBikSXNSc1vKdVpZlMIL/l4wVZLsNvuVRlj+RjME+ZZFRlYOZm35G62nbcXvh6PU1C4iooJgkCaj6UVLpnQZRxuYuoaVymDJwKaY80ojtZiHLPIxdPF+vDI/XC0AQkT0sBikSVPSu1x18IrRZnU/TjZ4O39PbBzZEm+3rgEbKwvsPJOAdtO3Y9K640jLvF0ghYjoQRikSVNHriThfMJN2Flb4Nnat2u2lySyDObIZ5/AxhEt0aaWh5oHPnfrWTz/5XaEn03QunlEZOQYpElTUitbSABztC250/YlE/zbfo3xTd/G8HC2xbn4VPSYtxvjVkSqqmZERPlhkCbNZOfosObw1SJb8coYyWjBnyNaoldgRfX4h90X0PaLbdhyMlbrphGREWKQJs2En0tATHI6nO2s8NQT7mbzSbjYW2NS17pY/FoQKpV1UGtahyzYgw9+O8LpWkSUB4M0aV7A5Hl/L9haWZrdJ9GsujvWD38SIc2rqMeLwi+iw4wdOHQpUeumEZGRYJAmTcgc4rVHok22gElhcbCxwoSOdbDotSB4OtvhbHwqus3eha82nUZWdo7WzSMijTFIkyZkUYqkW5ko72TLFaQAtXCH9Krb1/VSGeDTNpxSiWWXrt3kv1AiM8YgTZpYeWeou0Ndb1XzmgBXBxvM7NUAX/SoBydbK+y7cB3tZ2xXS2MSkXlikKZil5qehY13As+LZjzUfb8iKF0aVMDaYU+iXkVXJKdl4fXv9+J/a48jk8PfRGaHQZqK3cbjMbiVmY3Kbg6oV8GFn0A+KpZ1wLJBwXi1ua96PG/bWfSYG6aW9CQi88EgTcVu1Z0CJlIGVHqOlD8pJTq+Y21VA9zJzgr7LybihRnb8RfnVBOZDQZpKlbXUzOw9VQczD2ruyCkBvjvbz2JAB8XJN7MREjoHszacoarahGZAQZpKlbrIqNV9nItL2dUL+/Ed78AZUV/GRKMXoGV1HrVU/44qVbWkuv7RFRyMUhTsVp5Z8Ur9qILTgq+TOoagP91CYC1ZSk1z7zr17twISG10D8nIjIODNJUbKKSbiHi/DV1v2MJWpayuL0cVEmtV13OyRYnY1LQ8asdhksIRFSyMEhTsVlzKEoN1TapUgY+rvZ85x9Do8plseatFmhQ6fY0rZAFEQjdeY7vKVEJwyBNxWbVoX+yuunxeTjbqR5198YVkKMDPlx9DONXRrKcKFEJwiBNxeJs3A0cuZKkqou9EODFd70Qr1NP7lYXY573g8xm+z7sAl5duBfJXKOaqERgkKZi7UW3qO4Ot9K2fNcLkcw1H9SymppPbW9tqeqid/t6F+t+E5UADNJU5HQ6naGACbO6i07bOp5YNjgYHs62OB17A51n7cSBi9eL8DcSUYkP0rNmzUKVKlVgZ2eHoKAgREREPPD4ZcuWwc/PTx0fEBCAtWvX3hMQxo8fDy8vL9jb26NNmzY4ffp0nmNOnTqFTp06wd3dHc7OzmjRogW2bNlSJOdHwNGryWoJRlsrCzxXx5NvSRHy93HByqEtUMfbGQmpGej1zW5sOs4FOohMlaZBeunSpRg5ciQmTJiA/fv3o169emjbti1iY2PzPX7Xrl3o1asXBgwYgAMHDqBz585qi4yMNBzz2WefYcaMGZgzZw7Cw8Ph6OioXjMtLc1wTIcOHZCVlYXNmzdj37596vfKvujo2+sbU9HMjW5TywOlba349hYxTxc7/DwoGC2fKIe0zBy1QMfi8It834lMkU5DgYGBuqFDhxoeZ2dn67y9vXWTJk3K9/ju3bvr2rdvn2dfUFCQbtCgQep+Tk6OztPTUzdlyhTD84mJiTpbW1vdTz/9pB7HxcXp5LS3bdtmOCY5OVnt27Bhw0O3PSkpSf2M3NL9ZWfn6Jr+b6Ou8n/W6NYdieJbVYwysrJ17/x8UL33sk3744T6P0JERaMo4oJmPemMjAzVi5XhaD0LCwv1OCwsLN+fkf25jxfSS9Yff+7cOdUbzn2Mi4uLGkbXH+Pm5oaaNWvi+++/R2pqqupRz507F+XLl0ejRo2K6GzN157z1xCVlKYWiHi6Zjmtm2NWrC0tMOX/6uLtZ6qrxzM2n8GoXw5zyUsiE6LZ2GN8fDyys7Ph4eGRZ788PnHiRL4/IwE4v+P1w9T62wcdI5mwGzduVMPkTk5O6ouBBOj169ejTJky921venq62vSSk5MLfM7maOWdrO52dTxhZ22pdXPMjvx7H/lcTXi42GHcikgs23cZ11IzMKt3Q34eRCZA88Sx4iaJZUOHDlWBefv27SpRTQJ2x44dERUVdd+fmzRpkuqV67eKFSsWa7tNUUZWDtYeuf2edqrvo3VzzFrvoMqY26exSt7bdCIW/RdE4AYX5yAyepoFacmstrS0RExM3sxTeezpmX8GsOx/0PH62wcdI8lia9aswZIlS9C8eXM0bNgQX3/9tcoEX7hw4X3bO2bMGCQlJRm2S5cuPeKZm48dZ+LU0orupW0RXM1N6+aYvWdre2Dhq4EqeW/32Wvo/c1utXQoERkvzYK0jY2Nuga8adMmw76cnBz1ODg4ON+fkf25jxcbNmwwHO/r66uCce5jZFhasrz1x9y8eVPdyjB3bvJYfv/92NraqulauTd6MP3c6A51vVSlMdJe06puWPx6EMo4WOPQ5SR0nxuGmOR/Zj4QkXHRdLhbpl998803qgd7/PhxDBkyRCVzhYSEqOf79u2rerB6w4YNU9eOp02bpq5bf/jhh9i7dy/efPNNw/W34cOH4+OPP8aqVatw5MgR9Rre3t5qSFtIsJZrz/369cOhQ4fUnOn33ntPJZ21b99eo3ei5LmVkY0/j90e0XixPmt1G5O6FVzVFC190ZP/m7MLFxNuf3klIuOiaZDu0aMHpk6dqoqP1K9fHwcPHlRBWJ/4dfHixTzXiZs1a4bFixdj3rx5am7zL7/8ghUrVsDf399wzKhRo/DWW29h4MCBaNKkCW7cuKFeU4qf6IfZ5bHsf+aZZ9C4cWPs2LEDK1euVK9JhWPj8RjczMhGxbL2aFDRlW+rkanh4YRfBjdDpbIOuHTtlgrUp2JStG4WEd2llMzDunsn/TsZRpcEMrk+zaHve722cK8K1ENbVcN7bf34T8pIxSan4ZX54TgVcwOuDtZYGBKIevxSRWQ0ccHssrup6CXdzMTWU7erxjGr27iVd7bD0oHBKjBLkt/L3+xG+NkErZtFRHcwSFOhWxcZhcxsHfw8nfCEhxPfYSNXxtEGi14LQnBVN6RmZKP/gj0I+5uBmsgYMEhTkS1LyYQx0yHTshaENMFTT5TDrcxshIRGYNeZeK2bRWT2GKSpUMl0nrA7w6Ud6zKr25RIRbh5fRqp8q2yMEdI6B7sOM1ATaQlBmkqVGsOR0FSERtVLoOKZR347ppgoJ7bpxGe8SuP9KwcDFi4B9tOxWndLCKzxSBNhWrVnWUpX6zHXrSpsrWyxOxXGqJNrduB+rXv9+Kvk/kvH0tERYtBmgrN+fhUVcVKqou9EODFd9bEA/XXvRupUqJSg33g9/uw5QQDNVFxY5CmQk8Ya1bNDeWcbPnOmjgbKwvMerkh2tbxQEZ2Dgb9sA+bjueti09ERYtBmgqF1MRZyaHuEhmoZ77cEM/7e6pAPfhHBmqi4sQgTYXiWFQy/o5LVX/U2/rnv4oZmSZrSwvM6NUA7QO81Pz3IT/uZzIZkTEH6aysLGzcuBFz585FSsrter9Xr15V9bDJvIe6n6lZHs521lo3h4ogUE/vWd8w9D3wh70seEJkjEH6woULCAgIQKdOnTB06FDExd2enjF58mS8++67RdFGMnI5OTqsvrMsZSeueFWiA/VXvRqq6Vkyj1qmZ+27cE3rZhGVaAUO0rJcpKwcdf36ddjb2xv2d+nS5Z61nsk87Lt4HVeT0lTVqlZ+5bVuDhUhuZzxde+GaFHdXa1y1v+7PTh8OZHvOZGxBOnt27dj7NixsLGxybO/SpUquHLl9hxZMi/6hLG2dTxVMQwyg8pkfRsh0LcsUtKz0Gd+BI5dTda6WUQlUoGDdE5ODrKzs+/Zf/nyZTg5cTEFc5OZnYO1R6LVfdbqNh8ONlb4rn8TNKjkiqRbmegzPxynuR41kfZB+rnnnsP06dMNj0uVKqUSxiZMmIAXXnihsNtHRm7HmXhcS82Am6MNmldz07o5VIzk8kZoSCD8fZyRkJqBl78Nx7n4VH4GRFoG6WnTpmHnzp2oXbs20tLS8PLLLxuGuiV5jMyLPmGsfV0vWFlyRp+5cbG3xg+vBqllSeNS0tV61Jeu3dS6WUQlRimdVKF4hClYS5cuxaFDh1QvumHDhujdu3eeRLKSLjk5GS4uLkhKSoKzszPM0a2MbDT+eINag/jXIcFoVLms1k0ijcTfSEePuWFqrnxlNwcsGxyM8k52/DzIrCQXQVwocJDetm0bmjVrBisrq3sC965du/DUU0/BHDBIA78fjsLQxfvh42qPHf9ppS59kHkvU9pt9i5cvn5L9ayXDgyGiwPnzJP5SC6CIF3g8clWrVrh2rV750ZKo+Q5Mh+GMqD1vRmgCR7Odlj0WpCq234iOgUhoRG4mZHFd4aoOIO0dLzz6zElJCTA0dHxcdpCJkQyev86ebuQDZelJL3Kbo74YUCgula9/2KiWpQjPeve2SBE9HDyjlk/QNeuXdWtBOj+/fvD1vafVY5kStbhw4fVMDiZhz8io1V5yCc8SquhTSI9P09nLAhpgle+Dcf20/EY9tNBzHy5ARMLiYqyJy3j7LJJT1rmQ+sfy+bp6YmBAwfixx9/fJQ2kAnX6pZeNK9F090aViqDeX0aw8bSAuuPRmPM8iOqfCwRFVFPesGCBepWpltJjW4ObZuv2JQ07Po7Xt1/sZ6P1s0hI9Wihjtm9KqPNxbtx7J9l+Fsb42x7WvxSx1RUV6TlqIlDNDmTbK6pVNUv6IrKrk5aN0cMmLt/L0wuVtddX/+jnP4avMZrZtEVDJ70rn98ssv+Pnnn3Hx4kVkZGTkeW7//v2F1TYyUiu54hUVwEuNKyI5LQv/XXMMn284hTIO1ugTXIXvIVFR9KRnzJiBkJAQeHh44MCBAwgMDISbmxvOnj2L559/vqAvRybmYsJNHLyUCItSt6uMET2MAS188XbrGur++FVH1WgMERVBkP76668xb948fPXVV2olrFGjRmHDhg14++231VxpKtlWHbo9N7pZNXdWlKICGdGmBl4OqgQpnzRi6UHsOnM7r4GICjFIyxC3fqqVlAFNSUlR9/v06YOffvqpoC9HJpzVTVQQMgvgv5380a6Op5q+N/CHfYi8wi/2RIUapGW6lb7iWKVKlbB79251/9y5c2p6FpVcJ6KTcSrmhppW09bfU+vmkAmytCiF6T3rI8i3LG6kZ6H/gj24kMCVs4gKLUg/88wzWLVqlbov16ZHjBiBZ599Fj169ECXLl0K+nKYNWuWmtZlZ2eHoKAgREREPPD4ZcuWwc/PTx0fEBCAtWvX5nleviiMHz8eXl5eqqffpk0bnD59+p7X+f3339Xvk2PKlCmDzp07F7jt5pow9nTNcqqiFNGjsLO2xDf9GqOWl7NamKPvdxFqBS0iKoQgLdejP/jgA3V/6NCh+O6771CrVi189NFHmD17doFeS1bSGjlypJrWJVnh9erVQ9u2bREbG5vv8bKAR69evTBgwACVtCaBVbbIyEjDMZ999plKbpszZw7Cw8PVdDF5TVlWU+/XX39Vw/PyJUNW8pKlN2XJTbo/+fKzypDVzbnR9Hic7ayxMKQJKpa1x4WEm+i/IAIpaZl8W4nupiuAzMxM3cSJE3WXLl3SFYbAwEDd0KFDDY+zs7N13t7eukmTJuV7fPfu3XXt27fPsy8oKEg3aNAgdT8nJ0fn6empmzJliuH5xMREna2tre6nn34ynIOPj4/u22+/fay2JyUlydi+ujUHe88n6Cr/Z42u9rh1ulsZWVo3h0qIs3E3dA0/+lP92+o1L0yXlsl/W2S6koogLhSoJy3LU0pPVZalfFwyv3rfvn1qOFrPwsJCPQ4LC8v3Z2R/7uOF9JL1x8t18ejo6DzHSNlSGdbWHyM99itXrqjf1aBBAzUsLlPHcvfG6V76XnTbOp5quJKoMPi6OyI0JBCONpbY9XcCRi49hGyWDyV69OHu1q1bY+vWrXhc8fHxamEOmW+dmzyWQJsf2f+g4/W3DzpG5nOLDz/8EGPHjsWaNWvUNemnn3463yU49dLT09Vaobk3c5GVnYPfj9ye19qxPrO6qXAFVHDB3D6NYW1ZSv07m7j6KJNQiR614pj0OkePHo0jR46gUaNG95QIffHFF2HMcnJy1K1cV+/WrZuhLnmFChVUUtqgQYPy/blJkyZh4sSJMEfSw4m/kYGyjjZoUd1d6+ZQCa3z/Xn3+nh7yQF8H3YB5Z1s8eYzt4ufEJmzAgfpN954Q91+/vnn+c6DlN7xw3B3d4elpSViYmLy7JfHMs0rP7L/Qcfrb2WfDGPnPqZ+/frqvn5/7dq1Dc/LsptVq1ZVc8DvZ8yYMSrJTU960hUrVoQ5ZXW/EOAJa8sCD74QPZSO9byRcCMdH64+hql/noJbaVv0CqzEd4/MmsWj9ETvtz1sgBZSrUx64ps2bcrz2vI4ODg435+R/bmPF1LtTH+8r6+vCtS5j5FgKlne+mPkd0pQPnnypOGYzMxMnD9/HpUrV75ve+VnnJ2d82zmIC0zG38cvX2pgFndVNT6N/fF0FbV1P0PfjuCTcfzfiknMjeadoukZ/rNN99g4cKFOH78OIYMGYLU1FQ1NUr07dtX9WD1hg0bhvXr12PatGk4ceKEuq68d+9evPnmm4ae/PDhw/Hxxx+rudwyJC+v4e3tbZgHLcF18ODBatrXn3/+qYK1/F7x0ksvafI+GLMtJ2JV0QlvFzs0qlRG6+aQGXj3uZro3riCWmntzcUHVK14InP1SKtgFRYpgBIXF6eKj0hilwxJSxDWJ37J8LNkYetJOdLFixerhK/3338fNWrUwIoVK+Dv7284RmqJS6AfOHAgEhMT0aJFC/WaUvxEb8qUKSpTXeZK37p1S2V/b968WSWQUf5lQCVhzEJW1SAqYvJl+5MuAYhJTsfWU3F4NXQPlg9phiruefNfiMxBKZmHpXUjTJEMo8v0LllUpKQOfSenZaLxxxuRkZWD399ugTreLlo3icxIanoWes7bjSNXklDZzQG/DmkG99K2WjeLqFjjArOA6L7+PBqjAnS1co6o7VUyv4iQ8XK0tcJ3/f+pSjYgdA9uZjx+jQYiU8IgTfe18uAVQ8KYDEESFbdyTraq2EkZB2scupykrlHLvH0ic1HgIH13QQ/9JktWShUxKhlkwQOZHy24LCVpqVq50vi2XxPYWllg84lYjFsZyWInZDYKHKRdXV1VgtXdm+yXFaVkGpNkTuuLhpBpWnskSpVnrFfBhQk7pLlGlctgRq8GkNzFnyIu4avNZ7RuEpFxBunQ0FA1pUmyqyWzWja57+Pjo1bBkqxqWYXq008/LZoWU/FmdddjGVAyDlI3fuKLddT9zzecws97L2ndJCLjm4Ilc5plnnL37t0N+zp27KjWdp47d64qJFKpUiV88sknKniT6bl07Sb2XbgOuQzNIE3GpE9wFVxNSsPsv/7GmOVHVPnQp2uW17pZRMbTk5Y1nWX1qLvJPv1KUzI3+UElNsm4rT58uxfd1NcNHs7/zC8nMgaj2tZElwY+6nLMG4v248jlJK2bRGQ8QVrqVc+fP/+e/bJPX8s6ISGBhUFKwLKUnbjiFRkhmWkwuVtdtdjLzYxshITuUaM/RCVRgYe7p06dqspnrlu3Dk2aNFH7pDSnlOn85Zdf1OM9e/aoamJkek5Gp+BEdIpaNvB5/38WKSEyJjZWFpj9SkN0n7sbx6OS0e+7CPwypJlaqY3IrHvSshSlBGRZslLWX5ZN7su+Dh06qGOkFnZ+q2SR8Vt16Pbc6JZPlIeLg7XWzSG6Lyc7a4SGNIGPqz3OxqfitYV7cCvj4Rf5ITIFLAv6iEpiWVCpEPvUlC24dO2Wmu7C+dFkCk7HpKDb7F1ITsvCc7U9MPuVRrBknXkqIXHhkRbYkIUrIiIiEBsbe898aFl1ikzTgUuJKkA72FiiTS1mzJJpqOHhpIqdvDI/HH8ei8GHq47io051WCWPSoQCB+nVq1ejd+/euHHjhvqmkLtcpNxnkDb9hLFna3vAwUbTBdKICiTQtyym96iPoYv344fdF+Dlaoc3nq7Od5HM75r0O++8g1dffVUFaelRX79+3bDJ9WkyTVIPec3hKHWfWd1kil4I8MK49rXV/c/Wn8RvBy5r3SSi4g/SV65cwdtvvw0HB4fH/+1kNHafvYb4G+lwdbBGi+rltG4O0SN5tYUvXn/SV91/b9lh7Dgdz3eSzCtIt23bVk25opK54pX0RmR6C5GpGvN8LVUpLytHh8E/7sPRqyx2QqarwBce27dvj/feew/Hjh1TpUCtra3vmaJFpiUtMxvrj0ar+8zoJlNnYVEKU1+qi7iUNDVCFLJgD5a/0QwVynD0j8xgCpaFxf17WZI4lp1tHvMUS9IUrPWR0arH4elsh12jn1F/5IhMXdKtTHSfE4aTMSmoVs4Rvw5pBlcHFjsh04oLBR7XlClX99vMJUCXNKsNK155MUBTieFib43QV5vAy8UOf8dJsZO9atSIyJTw4qOZS0nLxMbjMep+p/o+WjeHqFB5udgjNCQQTnZW2HvhOoYvOagW5iAqUdekZX1oWSfazs5O3X8Qyfwm07HhWAzSs3JQ1d0RdbxNe9ieKD81PZ3wTd/G6Ds/QuVe/HfNMUzoWJvFTqjkXJP29fVVGd1ubm7q/n1frFQpnD17FuagpFyTloUJtp6Kw/A2NTC8zRNaN4eoyKw5fBVvLj6g7o953g+DWlbju00loyzouXPn8r1Ppi3hRjp2nLk9j5RZ3VTSdajrjeikNHz8+3FMWncCni52vMRDRo/XpM3Y2iNR6vpcgI8LqpYrrXVziIrca09WxYAWt0cD3112CDvvfEklKjHzpCWDOzQ0FJs2bcp3gY3NmzcXZvuoCK26k9XNXjSZkw9eqIXo5DT8fjgKg37Yh58HBaM28zGopATpYcOGqSAtRU38/f2ZfGGiriTewp7z1yHro3So56V1c4iKjdQB+Lx7PcSnpCP83DWEhEZg+RvN1brURCYfpJcsWYKff/4ZL7zwQtG0iIp1bnRglbJqmgqRObG1ssS8vo3x0pxdOBVzQyVQ/jI4mMVOyPSvSdvY2KB6dS4BV1KWpeTcaDLrYichgarS3pnYGxj4/T4WO6GSsVTll19+iQJWEyUjciY2BceikmFlUQrP+3tq3RwizXi72quqZE62Vog4fw0jf2axEzLx4e4dO3Zgy5YtWLduHerUqXPPAhvLly8vzPZREfaiWz5RDmUcWcuYzJufpzPm9m2E/t/twdoj0SjvxGInZMJB2tXVFV26dCma1lCRkxGQlfqs7vrefMeJADSr5o6p3evh7Z8OIHTXeXi72mHgUyx2QiY23J2VlYVWrVph0qRJWLBgQb7bo5g1axaqVKmiyo4GBQUhIiLigccvW7YMfn5+6nhZLnPt2rX3BKLx48fDy8sL9vb2aNOmDU6fPp3va6Wnp6N+/foqS/3gwYMo6Q5fTsKFhJuwt7ZEm1oeWjeHyGjIVMSx7Wup+/9be8KwxjqRyQRpKysrDB48WAW2wrJ06VKMHDkSEyZMwP79+1GvXj20bdtWzcHOz65du9CrVy8MGDAABw4cQOfOndUWGRlpOOazzz5TNcbnzJmD8PBwODo6qtdMS0u75/VGjRoFb2/z6VGuvDPU3aa2BxxtCzyQQlTii5282vyfYie7WOyEtKYroJYtW+p+++03XWEJDAzUDR061PA4Oztb5+3trZs0aVK+x3fv3l3Xvn37PPuCgoJ0gwYNUvdzcnJ0np6euilTphieT0xM1Nna2up++umnPD+3du1anZ+fn+7o0aOSBac7cODAQ7c7KSlJ/Yzcmoqs7Bxdk4836Cr/Z41uw9ForZtDZJSys3N0byzap/6f+I9frzt21XT+j5O2iiIuFDi7+4033lAZ3jNnzkRYWBgOHz6cZyuIjIwM7Nu3Tw1H61lYWKjH8tr5kf25jxfSS9YfL7XFo6Oj8xwjBc9lGD33a8bExOD111/HDz/8AAcHh39tq4weSPH03JupCT+bgNiUdDX15KknymndHCKjLXYy7aV6CPQti5T0LPRfEKGK/xBpocDjnT179rxnSUq5nivXgeVWyoY+rPj4eHW8h0fea6Py+MSJE/n+jATg/I6X/frn9fvud4y0tX///mrovnHjxjh//vy/tlWuw0+cOBEloQyoTLuysWLZdqL7sbO2xDd9GuOlubeLnfSZH45fBjdDWc6GIGMP0iVhFayvvvoKKSkpGDNmzEP/jBwr1871pCddsWJFmIr0rGy1oIZgVjfRv3NxsMbCVwPR7etdOBuXipAFEVj8elPmcpBxB+nKlSsX2i93d3eHpaWlGnrOTR57euZfZEP2P+h4/a3sk+zu3MdIFrd+ERAZ+ra1tc3zOtKr7t27NxYuXHjP75Vj7z7elGw7FY/ktCyUd7JFkK+b1s0hMglSMvf7AUGqfOihy0kY/OM+zO/XhCNRVGweeczz2LFjWL9+PVatWpVnK2iJ0UaNGqkVtfRkVS15HBwcnO/PyP7cx4sNGzYYjvf19VWBOvcx0uuVLG/9MZL5fejQITXlSjb9FC7JNP/kk09QEumnk3Ss5w1Li1JaN4fIZFQvXxoLQgLhYGOJ7afjVVWynBxWXCQj7UmfPXtWFTM5cuSI4Vq0kPuiINekhQwh9+vXT/ViAwMDMX36dKSmpiIkJEQ937dvX/j4+KhrwvpVuFq2bIlp06aplbhkwY+9e/di3rx5hnYMHz4cH3/8MWrUqKGC9rhx49Q0K5mqJSpVqpSnDaVL315LuVq1aqhQoQJKmtT0LGw8fnv0gctSEhVc/YqumNunEV4N3YM1h6Pg5miDD1+sw1UAyfh60hIkJfDJPGbJij569Ci2bdumguxff/1V4Ab06NEDU6dOVcVHZDhaerbSQ9cnfl28eBFRUbevpYpmzZph8eLFKijLnOpffvkFK1asUMtm5p77/NZbb2HgwIFo0qQJbty4oV5Tip+Yow3HYpCWmYMqbg6oW8FF6+YQmaQna5TDtO5S+AhYGHYBX20+o3WTyAyUknlYBb2OLNd069atq6Y2SXWwmjVrqn0yNUsKjJgDGUKX809KSoKzszOMmXz733wiFm8/Ux0jn6updXOITFroznP4cPUxdf/jzv54pWnh5emQaUsugrhQ4J60DGc7OTkZAvbVq1cNCWUnT54slEZR4bmemoFtp+LUfWZ1Ez2+/s191RdeMW5lpGHWBJFRXJOWYWVJupIhbykQIiU4JQFMhp+rVq1aJI2kR7c2MgpZOTrU9nJG9fK3v1wR0eMZ8ewTiE/NwOLwixi+5KAqENS8ujvfVip0Be5Jjx07VmVgi48++kjNm37yySdVhrRkTZNx1uruxBWviAqNJKj+t5O/KgyUkZ2Dgd/vxZHLSXyHSftr0vm5du0aypQpY1aZjqZwTfpq4i00n7wZ8gnvHP0MfFzttW4SUYkiRYJCFuzBrr8TVMb3ssHBqFru9mwRMj/JxnBNWu/MmTP4448/cOvWLZQtW7ZQGkOFa83hqypAB1YpywBNVARsrSzV1Cx/H2ckpGagz/wI9eWYqLAUOEgnJCSgdevWeOKJJ/DCCy8YpkfJ0pGS3U3GV6u7I4e6iYqMk501QkMC4evuqBbieOXbcMTfKLzlfMm8FThIjxgxAtbW1mr+cu7Vo2S+s8xFJuPwd9wNRF5JhpVFKbQP+Kc8KhEVPvfStvjxtSB4u9jhbHyq6lEn3czkW03FH6T//PNPTJ48+Z7KXFLd68KFC4/fIioUq+4kjLWo4c6Ve4iKgeR8LHq9qQrYx6OSERIaoar9ERVrkJaSnfmtvyzJY6a8AEVJIrmA+qFuZnUTFR8Z8v5hQCCc7ayw/2IiBv6wF2mZBSuVTPRYQVqmW33//feGx5LRLVOyZL50q1atCvpyVARkmPtcfCpsrSzwbO38VxMjoqJRy8tZLXEpC3LsPJOAt346gMzs29NWiYq8mIkEY0kck0UtMjIyVJ1sqd8tPemdO3cWuAFUdCtetantgdK2Bf6IiegxNahUBt/2a4z+C/ao2vnvLjuEL7rXhwVXoKOi7klLxbFTp06hRYsW6NSpkxr+7tq1q6rZLatIkbayc3RYffj2UDdXvCLSTrNq7pjdu6FK3pSiQlJCtBDKUpCZeaRulkzW/uCDD/Lsu3z5slp1Sr9kJGkj4tw1xCSnw8nOCk/XLMePgUhDrWt54PMe9TFsyQEsCr8IR1srjHnez6wKP5FGxUzymz89f/78wno5ekT6hDEpVyiFFohIWzKiNalLgLo/b9tZTP3zJHvUVPxBmrSXkZVjWJGnU30frZtDRHf0DKyEDzvWVvdnbfkb0zee5ntDD4VBugTZfjoOSbcyUc7JFk2rumndHCK6a4nLse1rqftfbjqNmZsZqOnfMUiXwBWvOtT1giWzSImMzmtPVsV/2vmp+1P/PIU5W//WuklUUhLHJIP7QRITEwujPfSIbmZkqakeglndRMZryNPVkJWdg2kbTuHTdSdU9rcEb6LHCtKS0f1vz/ft2/dhX44KmQToW5nZqFTWAfUruvL9JTJib7WugawcnRr2/vj347C2tEC/ZlW0bhaZcpBesGBB0baEHsvqO1nd0ovm9A4i4ze8jQTqHJVINmHVUVhZlkLvoMpaN4uMDK9JlwCJNzOw9VScus9a3USmQb5Mv/tcTQx66vZQ9we/RWJJxEWtm0VGhkG6BFgXGY3MbB38PJ1Qw8NJ6+YQUQEC9ejn/fBqc1/1ePTyI/gh7DzfPzJgkC5Btbo5N5rINAP1uA61MKDF7UA9buVRzN9xTutmkZFgkDZx0UlpCD93Td3vWM9L6+YQ0SMGaplDLZnf4r9rjmH2X5yeRQzSJm/N4auQmv2NK5dBhTL3rvNNRKYTqEe1rYlhrWuox5PXn8CMTSx4Yu7Yky4htbpfrO+tdVOIqBAC9Yhnn8B7bWuqx59vOIWpf7DWtzljkDZh5+JTcfhykqou9kIAh7qJSoqhraobSojO3HIGk9ad4KIcZopB2oStulMGtHl1d7iXttW6OURUiKQK2cQX6xhWz5q4+hgDtRlikDZRsnj8ykO3s7pZBpSoZJIqZP/rEgBZfjp013mMWX4E2Tk6rZtFxYhB2kQdvZqMs3GpsLGyQNs6Hlo3h4iKyMtBlfBZt7qQNXOW7LmEoYv2Iy0zm++3mTCKID1r1ixUqVIFdnZ2CAoKQkRExAOPX7ZsGfz8/NTxAQEBWLt27T29zPHjx8PLywv29vZo06YNTp/+J0vy/PnzGDBgAHx9fdXz1apVw4QJE5CRkQFTKwPa2q88nOystW4OERWhlxpXxNe9G8LG0gLrj0YjZMEepKRl8j03A5oH6aVLl2LkyJEqSO7fvx/16tVD27ZtERsbm+/xu3btQq9evVSQPXDgADp37qy2yMhIwzGfffYZZsyYgTlz5iA8PByOjo7qNdPS0tTzJ06cQE5ODubOnYujR4/iiy++UMe+//77MAU5OTpDVjfLgBKZh3b+XggNaQJHG0uEnU3Ay9+EI/5GutbNoiJWSifdTg1Jz7lJkyaYOXOmeizBs2LFinjrrbcwevToe47v0aMHUlNTsWbNGsO+pk2bon79+irQyul4e3vjnXfewbvvvqueT0pKgoeHB0JDQ9GzZ8982zFlyhTMnj0bZ8+efah2Jycnq5W/5LWdnZ1RnCLOXUP3uWFwsrXCnrFtYGdtWay/n4i0c+RyEvotiMC11AxUdXfE9wMCWSPBSBRFXNC0Jy3Dy/v27VPD0YYGWViox2FhYfn+jOzPfbyQXrL++HPnziE6OjrPMfKmyZeB+72mkDe1bNmy930+PT1dfQC5N63LgLb192SAJjIzARVc8MvgYPi42uNsfCr+b3YYTsWkaN0sKiKaBun4+HhkZ2erXm5u8lgCbX5k/4OO198W5DXPnDmDr776CoMGDbpvWydNmqSCvX6T3r4WMrNzsPZIlLrPrG4i81S1XGn8MiQYNcqXRnRyGl6aE4b9F69r3SwqidektXblyhW0a9cOL730El5//fX7HjdmzBjV29Zvly5dghZ2nI7H9ZuZcC9tg2bV3DRpAxFpz8vFHj8PCkaDSq5IupWJ3t+EY8vJ/HN5yHRpGqTd3d1haWmJmJiYPPvlsaenZ74/I/sfdLz+9mFe8+rVq2jVqhWaNWuGefPmPbCttra26hpD7k0L+oSx9gFesLI0++9YRGatjKMNFr0WhKeeKIdbmdl4beFeLA7nmtQliaZ/5W1sbNCoUSNs2rTJsE8Sx+RxcHBwvj8j+3MfLzZs2GA4XqZVSTDOfYxcP5Ys79yvKT3op59+Wv3+BQsWqGvhxu5WRjb+OHp7yP7F+j5aN4eIjICDjRW+7dsY3RpWUIVO3v/tiFqcQ2aBkOmz0roBMv2qX79+aNy4MQIDAzF9+nSVvR0SEqKe79u3L3x8fNQ1YTFs2DC0bNkS06ZNQ/v27bFkyRLs3bvX0BOWAvXDhw/Hxx9/jBo1aqigPW7cOJXxLVO1cgfoypUrY+rUqYiLizO05349eGOw6UQMbmZko0IZezSs5Kp1c4jISEhRo6kv1UWlsg74YuMptczlpWs3MfWlekwuNXGaB2mZUiVBUoqPSGKXTKVav369IfHr4sWLeXq5MjS9ePFijB07Vs1rlkC8YsUK+Pv7G44ZNWqUCvQDBw5EYmIiWrRooV5Tip/oe96SLCZbhQoV8rRH4xlpD7TyTq1uSRiTLyNERHryN2FYmxrqS/zo5Yex5nCUWm/+m76N1bA4mSbN50mbquKeJ510MxNNPtmIjOwcrB/+JPw8tbkmTkTGb9eZeAz6cR9S0rJQxc0B3/ZrjOrlnbRuVomXXNLmSdPDW380SgXomh5ODNBE9EDNqrtj+ZBmqld9PuEmuszaxcxvE8UgbSL0Wd0v1vfWuilEZAJqeDhh5dDmCKxSFinpWRgQugffbDtr1Jf06F4M0iYgNjkNu/5OUPdZwISIHpZbaVv8+FoQejapCEn2/mTtcby77DBX0TIhDNImQBJA5MuvFC2oWNZB6+YQkYllfk/qGoAPO9aGpUUp/Lr/Mnp9sxsxybcXHCLjxiBtAlbqV7yqx6FuInq0zO/+zX3VKlrOdlY4cDER7WdsR9idEToyXgzSRu5CQioOXUpUC763r8sgTUSP7ska5bDqzRbw83RC/I0MvDI/HHO3/s3r1EaMQdrIrbozN7p5dXeUc7LVujlEZOKquDvitzeao2sDH1WhbNK6Exjy436kpGVq3TTKB4O0EZMsTP1Qd0cOdRNRIbG3scS07vXw387+sLYshfVHo9Fp5k6ciNZuCV7KH4O0ETselYIzsTdU4kc7f+MtV0pEpnmduk/TymolLS8XO7U29Yszd+KHsPMc/jYiDNImMDe6Vc1ycLaz1ro5RFQCNahUBmveaqH+zmRk5WDcyqMY9MM+JN7M0LppxCBtvGQFm9X6rG6ueEVERTyf+rv+TTCuQ201/P3nsRg8/+V2hJ9l9rfW2JM2UvsvXseVxFsobWuFZ/zKa90cIjKD4e8BLXxVUpmvuyOiktLUfOppf55UPWzSBoO0ka949VwdDy41R0TFxt/HRQ1/y/rUUqXsq81n0HnWThyPYlKZFhikjVBmdg7WHolS91kGlIiKm6Otlcr+nvlyA5RxsMaxqGS8OHMHZm05g6xs9qqLE4O0Edp5Jh4JqRlwc7RR86OJiLTQoa43/hzREs/W9kBmtg5T/jiJbnPCcCY2hR9IMWGQNuKs7hcCvGBtyY+IiLQjRZTm9WmEz7vXg5OdlaqA+MKXOzB94ymkZ2XzoylijABGJi0zG39ERqv7nbgsJREZSVJZ14YVsGFESzwtU7WyczB942mVAc7630WLQdrIbD4Ri9SMbPi42qNhpTJaN4eIyMDTxQ4L+jdR16qlh302LlVlgL/z8yFcS+W86qLAIG1kVh68YigDaiGrahARGVmvWq5VbxzZUlUsK1UKavnLZ6b9he/DzjOxrJAxSBuR5LRMbDkZp+4zq5uIjJmLvbWq/b18SDO1qlbizUyMX3kU7b7cjr9OxmrdvBKDQdqIyLVoKRpQvXxp1PJy0ro5REQPXVZUArZM15L1Bvov2IN+30XgdAyzwB8Xg7QRZnV3quethpSIiEyBlaWFGvr+671WeP1JX1VadOupONWrHv3rYVy+flPrJposBmkjEZeSruZHCy5LSUSmOgT+QfvaKgv8udoear3qJXsuodXUv/DBb0cQlXRL6yaaHAZpI/H74auqBF+9iq5qUXYiIlMlf8Pm9W2MX4cEo0V1d1UIZVH4RbT87C98uOooYpLTtG6iyWCQNrKhbiaMEVFJ0ahyWfz4WhCWDmyKIN+yan516K7zaDF5s5q2xXrg/45B2ghcunYT+y8mqqkMHet6ad0cIqJCFVTVDUsGNsXi14IQWKWs6lnLtC0phtJnfri6fq3T6fiu58Mqv52kTS86uKobyjvb8e0nohJHkmGbVXdX24GL1/Ht9nNYFxmF7afj1VajfGn0DKyELg18UNbRRuvmGo1SOn59eSTJyclwcXFBUlISnJ2dH+tDaPvFNpyMScHkbgHo0aTSY70WEZEpjSJ+t/Mclu65hJsZt+uA21ha4Nk6HujRuKK6nm1KRZ0KMy7oMUhr/GGciE5Gu+nb1ZSFvR88CxcH60d+LSIiU5R0KxOrDl5RmeBHr/6zbrWPqz061PVCO39P1K/oavRTU4siSHO4W2OrDt4e6n66ZnkGaCIy26lbfYKrqC3yShJ+3nsJKw5cwZXEW5i77azavF3s0NbfE8/7e6FR5TKwNKEetsknjs2aNQtVqlSBnZ0dgoKCEBER8cDjly1bBj8/P3V8QEAA1q5dm+d5GcEfP348vLy8YG9vjzZt2uD06dN5jrl27Rp69+6tvu24urpiwIABuHHjBoqTtJNZ3URE//D3ccFHnfwR8UEbzHq5oepJO9pY4mpSGhbsPI/uc8PQ8L8bMOiHvapWuFQ4K8lXbTUf7l66dCn69u2LOXPmqAA9ffp0FYRPnjyJ8uXL33P8rl278NRTT2HSpEno0KEDFi9ejMmTJ2P//v3w9/dXx8hjeX7hwoXw9fXFuHHjcOTIERw7dkwFdvH8888jKioKc+fORWZmJkJCQtCkSRP1esU1rLHvwnV0m70LDjaW2Df2WdjbWD7S6xARlfQlfLedisP6yGhsOB6DlLSsPM97ONsiyNcNAT4uqOPjjDreLqp3XtxK5DVpCcwSHGfOnKke5+TkoGLFinjrrbcwevToe47v0aMHUlNTsWbNGsO+pk2bon79+irQy+l4e3vjnXfewbvvvquelzfMw8MDoaGh6NmzJ44fP47atWtjz549aNy4sTpm/fr1eOGFF3D58mX188XxYcikfpkz2Lm+N6b3bPBIr0FEZE6ysnNw+EqSWsdaqjTuvXBdrXlwt8puDqjt5YzKbo6oVNbBsHm52sHasmgGkUvcNemMjAzs27cPY8aMMeyzsLBQw9NhYWH5/ozsHzlyZJ59bdu2xYoVK9T9c+fOITo6Wr2Gnrxp8mVAflaCtNzKELc+QAs5Xn53eHg4unTpguIQXM0NFxJS0bmBT7H8PiKiklAnvGGlMmob2qq66mXLqKRM64q8kowjV5LUtewLCTfVdje5lC29bP3mfGdTq3p18je6a92aBun4+HhkZ2erXm5u8vjEiRP5/owE4PyOl/365/X7HnTM3UPpVlZWKFu2rOGYu6Wnp6st9zemx9W2jqfaiIjo0dhZW6J5dXe16V1PzVBZ4jJ7RqZ5Xbx2E5eu31L307NycP1mptpys7GywP+6BBjdx8Ds7ock17gnTpxYtJ8GERE9tjKONmhRw11tueXk6BCfmq7WvpZpX0l3bpPTMpGZfe+QOcw9SLu7u8PS0hIxMTF59stjT8/8e5iy/0HH629ln2R35z5Grlvrj4mNzbsoeVZWlsr4vt/vlSH53MPs0pOWa+dERGQaLCxKobyTndpMhaZTsGxsbNCoUSNs2rTJsE8Sx+RxcHBwvj8j+3MfLzZs2GA4XrK5JdDmPkYCqlxr1h8jt4mJiep6uN7mzZvV75Zr1/mxtbVViQC5NyIioiKl09iSJUt0tra2utDQUN2xY8d0AwcO1Lm6uuqio6PV83369NGNHj3acPzOnTt1VlZWuqlTp+qOHz+umzBhgs7a2lp35MgRwzGffvqpeo2VK1fqDh8+rOvUqZPO19dXd+vWLcMx7dq10zVo0EAXHh6u27Fjh65GjRq6Xr16PXS7k5KSJCte3RIRESUVQVzQ/Jq0TKmKi4tTxUckaUuGpGU6lD7x6+LFiyrrWq9Zs2ZqLvPYsWPx/vvvo0aNGiqzWz9HWowaNUpN0xo4cKDqMbdo0UK9pn6OtFi0aBHefPNNtG7dWr1+t27dMGPGjGI+eyIiIiOeJ22qimI+HBERma7kIogLRlEWlIiIiO7FIE1ERGSkGKSJiIiMlOaJY6ZKfym/MCqPERGR6Uu+Ew8KM9WLQfoRpaSkqFsWNCEiorvjgySQFQZmdz8iKXxy9epVODk5oVSpRyvIrq9adunSpRKTIc5zMg38nEwDPyfT+pxkyrDEA1lJMffU4cfBnvQjkg+gQoUKhfIhlMQKZjwn08DPyTTwczIN0nsu7L/lTBwjIiIyUgzSRERERopBWkOyaMeECRPUbUnBczIN/JxMAz8n01CUnxMTx4iIiIwUe9JERERGikGaiIjISDFIExERGSkGaQ3NmjULVapUUetcBwUFISIiAqZi0qRJaNKkiSrmUr58eXTu3BknT57Mc0xaWhqGDh0KNzc3lC5dWq3ZHRMTA1Pw6aefqqIEw4cPN+nzuXLlCl555RXVZnt7ewQEBGDv3r2G56V8oazl7uXlpZ5v06YNTp8+DWOVnZ2NcePGwdfXV7W3WrVq+O9//5unDKOxn9O2bdvQsWNHVfBC/o2tWLEiz/MP0/5r166hd+/eak6uq6srBgwYgBs3bsAYzykzMxP/+c9/1L89R0dHdUzfvn1VMShTPae7DR48WB0zffr0Qj8nBmmNLF26FCNHjlQZgfv370e9evXQtm1bxMbGwhRs3bpVBazdu3djw4YN6j/ic889h9TUVMMxI0aMwOrVq7Fs2TJ1vPyn7Nq1K4zdnj17MHfuXNStWzfPflM7n+vXr6N58+awtrbGunXrcOzYMUybNg1lypQxHPPZZ59hxowZmDNnDsLDw9UfUfl3KF9IjNHkyZMxe/ZszJw5E8ePH1eP5Ry++uorkzkn+T8i/9/lS3p+Hqb98of/6NGj6v/emjVrVEAZOHAgjPGcbt68qf7GyZcruV2+fLn6Qv/iiy/mOc6Uzim33377Tf0dlGB+t0I5Jx1pIjAwUDd06FDD4+zsbJ23t7du0qRJJvmJxMbGSldGt3XrVvU4MTFRZ21trVu2bJnhmOPHj6tjwsLCdMYqJSVFV6NGDd2GDRt0LVu21A0bNsxkz+c///mPrkWLFvd9PicnR+fp6ambMmWKYZ+cp62tre6nn37SGaP27dvrXn311Tz7unbtquvdu7dJnpP8+/ntt98Mjx+m/ceOHVM/t2fPHsMx69at05UqVUp35coVnbGdU34iIiLUcRcuXDDpc7p8+bLOx8dHFxkZqatcubLuiy++MDxXWOfEnrQGMjIysG/fPjWMlbvMqDwOCwuDKUpKSlK3ZcuWVbdyftK7zn2Ofn5+qFSpklGfo4wOtG/fPk+7TfV8Vq1ahcaNG+Oll15SlyQaNGiAb775xvD8uXPnEB0dneecpKyhXHox1nNq1qwZNm3ahFOnTqnHhw4dwo4dO/D888+b7Dnl9jDtl1sZOpXPVk+Ol78h0vM2lb8XMjws52Gq55STk4M+ffrgvffeQ506de55vrDOibW7NRAfH6+urXl4eOTZL49PnDgBUyP/WOXarQyt+vv7q33yh8bGxsbwnzD3OcpzxmjJkiVqOE6Gu+9miudz9uxZNTQsl1Xef/99dV5vv/22Oo9+/foZ2p3fv0NjPafRo0erxQzkC5KlpaX6f/TJJ5+oYUVhiueU28O0X27lS1duVlZW6guyKZyjDNvLNepevXoZ6lyb4jlNnjxZtVH+T+WnsM6JQZoKpfcZGRmpejSmSlYiGzZsmLp2JIl8JYF8eZJv8f/73//UY+lJy+ck1zolSJuin3/+GYsWLcLixYtV7+XgwYPqC6JcDzTVczInMhrVvXt3lRwnXyBN1b59+/Dll1+qL/WPugriw+Jwtwbc3d1VL+DuzGB57OnpCVPy5ptvqoSILVu25FkVTM5DhvUTExNN4hzlP50k7TVs2FB925VNksMkgUfuS0/GlM5HSHZw7dq18+yrVauWWk5P6NttSv8OZWhRetM9e/ZU2cIy3CgJfTLbwFTPKbeHab/c3p1gmpWVpTKJjfkc9QH6woUL6stw7tWiTO2ctm/frtorl7v0fy/kvN555x01Y6cwz4lBWgMy3NioUSN1bS13r0ceBwcHwxTIN2EJ0JLZuHnzZjUlJjc5P8kqzn2OktEpAcIYz7F169Y4cuSI6pnpN+mFyjCq/r4pnY+Qyw93T4uTa7mVK1dW9+Uzkz8Wuc9JhpLlepmxnpNkCt+9Tq984ZX/P6Z6Trk9TPvlVr4syhdLPfk/KO+BXLs25gAtU8k2btyopgTmZmrn1KdPHxw+fDjP3wsZzZEvkX/88UfhnlMhJL7RI1iyZInK2AwNDVVZgAMHDtS5urrqoqOjTeL9HDJkiM7FxUX3119/6aKiogzbzZs3DccMHjxYV6lSJd3mzZt1e/fu1QUHB6vNVOTO7jbF85EMWisrK90nn3yiO336tG7RokU6BwcH3Y8//mg45tNPP1X/7lauXKk7fPiwrlOnTjpfX1/drVu3dMaoX79+Kpt2zZo1unPnzumWL1+uc3d3140aNcpkzklmEBw4cEBt8if4888/V/f1mc4P0/527drpGjRooAsPD9ft2LFDzUjo1auXUZ5TRkaG7sUXX9RVqFBBd/DgwTx/L9LT003ynPJzd3Z3YZ0Tg7SGvvrqK/VH38bGRk3J2r17t85UyD/a/LYFCxYYjpE/Km+88YauTJkyKjh06dJF/cc01SBtiuezevVqnb+/v/pC6Ofnp5s3b16e52XKz7hx43QeHh7qmNatW+tOnjypM1bJycnqM5H/N3Z2drqqVavqPvjggzx/7I39nLZs2ZLv/x35AvKw7U9ISFB/7EuXLq1zdnbWhYSEqKBijOckX6bu9/dCfs4Uz+lhg3RhnBNXwSIiIjJSvCZNRERkpBikiYiIjBSDNBERkZFikCYiIjJSDNJERERGikGaiIjISDFIExERGSkGaSIiIiPFIE1ERGSkGKSJyCAuLg5DhgxRq/vY2tqqxR7atm2LnTt3qudlWb4VK1bwHSMqJlxPmogMunXrppbkXLhwIapWraqWSJQVmRISEvguEWmAtbuJSJFl9cqUKYO//voLLVu2vOddkXVyZc1cPVny8vz58+r+ypUrMXHiRBw7dkwt2devXz988MEHap1d9YemVCl8/fXXWLVqlXp9Wev6s88+w//93//x3Sd6AA53E5FSunRptclwdnp6+j3vyp49e9TtggULEBUVZXi8fft29O3bF8OGDVNBeu7cuQgNDcUnn3yS5+fHjRuneuqHDh1S63T37NkTx48f57tP9ADsSRORwa+//orXX38dt27dQsOGDVWPWoJp3bp1b//BKFUKv/32Gzp37mz4mTZt2qB169YYM2aMYd+PP/6IUaNG4erVq4afGzx4MGbPnm04pmnTpup3SA+biPLHnjQRGUhPVwKrDEu3a9dODU1LIJWe8f1Iz/ijjz4y9MRlk0Avve2bN28ajgsODs7zc/KYPWmiB2PiGBHlYWdnh2effVZtMkT92muvYcKECejfv3++79SNGzfU9eiuXbvm+1pE9OjYkyaiB6pduzZSU1PVfWtra2RnZ+d5XnraJ0+eRPXq1e/ZLCz++ROze/fuPD8nj2vVqsV3n+gB2JMmIkWmWb300kt49dVX1TVoJycn7N27V2Vhd+rUyZDhLVOymjdvruZRSzb4+PHj0aFDBzW3WrK1JTDLEHhkZCQ+/vhjw7u7bNkyNG7cGC1atMCiRYsQERGB+fPn890negAmjhGRIhndH374If7880/8/fffyMzMRMWKFVXgfv/992Fvb4/Vq1dj5MiRauqVj4+PYQrWH3/8oa5LHzhwQPW2/fz81DC5XJtWf2hKlcKsWbNU5vi2bdvUFKzJkyeje/fufPeJHoBBmoiKXH5Z4UT073hNmoiIyEgxSBMRERkpJo4RUZHT6XR8l4keAXvSRERERopBmoiIyEgxSBMRERkpBmkiIiIjxSBNRERkpBikiYiIjBSDNBERkZFikCYiIjJSDNJEREQwTv8P1+nZhWjE2FcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "plt.tight_layout(); plt.savefig(\"2.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7512808-b48d-4146-86a1-5931b1e3aec1",
   "metadata": {},
   "source": [
    "## D.3 Gradient clipping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a74f76-8d2b-4974-a03c-d645445cdc21",
   "metadata": {},
   "source": [
    "- Gradient clipping is yet another technique used to stabilize the training when training LLMs\n",
    "- By setting a threshold, gradients exceeding this limit are scaled down to a maximum magnitude to ensure that the updates to the model's parameters during backpropagation remain within a manageable range\n",
    "- For instance, using the `max_norm=1.0` setting in PyTorch's `clip_grad_norm_` method means that the norm of the gradients is clipped such that their maximum norm does not exceed 1.0\n",
    "- the \"norm\" refers to a measure of the gradient vector's length (or magnitude) in the parameter space of the model\n",
    "- Specifically, it's the L2 norm, also known as the Euclidean norm\n",
    "- Mathematically, for a vector $\\mathbf{v}$ with components $\\mathbf{v} = [v_1, v_2, \\ldots, v_n]$, the L2 norm is defined as:\n",
    "$$\n",
    "\\| \\mathbf{v} \\|_2 = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44838a6-4322-47b2-a935-c00d3a88355f",
   "metadata": {},
   "source": [
    "- The L2 norm is calculated similarly for matrices.\n",
    "- Let's assume our gradient matrix is:\n",
    "$$\n",
    "G = \\begin{bmatrix}\n",
    "1 & 2 \\\\\n",
    "2 & 4\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "- And we want to clip these gradients with a `max_norm` of 1.\n",
    "\n",
    "- First, we calculate the L2 norm of these gradients:\n",
    "$$\n",
    "\\|G\\|_2 = \\sqrt{1^2 + 2^2 + 2^2 + 4^2} = \\sqrt{25} = 5\n",
    "$$\n",
    "\n",
    "- Since $\\|G\\|_2 = 5$ is greater than our `max_norm` of 1, we need to scale down the gradients so that their norm is exactly 1. The scaling factor is calculated as $\\frac{max\\_norm}{\\|G\\|_2} = \\frac{1}{5}$.\n",
    "\n",
    "- Therefore, the scaled gradient matrix $G'$ will be as follows:\n",
    "$$\n",
    "G' = \\frac{1}{5} \\times G = \\begin{bmatrix}\n",
    "\\frac{1}{5} & \\frac{2}{5} \\\\\n",
    "\\frac{2}{5} & \\frac{4}{5}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeb0c3c1-2cff-46f5-8127-24412184428c",
   "metadata": {},
   "source": [
    "- Let's see this in action\n",
    "- First, we initialize a new model and calculate the loss for a training batch like we would do in the regular training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e199e1ff-58c4-413a-855e-5edbe9292649",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import calc_loss_batch\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import calc_loss_batch\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "\n",
    "loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b60f3a-15ec-4846-838d-fdef3df99899",
   "metadata": {},
   "source": [
    "- If we call `.backward()`, PyTorch will calculate the gradients and store them in a `.grad` attribute for each weight (parameter) matrix\n",
    "- Let's define a utility function to calculate the highest gradient based on all model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e70729a3-24d1-411d-a002-2529cd3a8a9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0446, device='mps:0')\n"
     ]
    }
   ],
   "source": [
    "def find_highest_gradient(model):\n",
    "    max_grad = None\n",
    "    for param in model.parameters():\n",
    "        if param.grad is not None:\n",
    "            grad_values = param.grad.data.flatten()\n",
    "            max_grad_param = grad_values.max()\n",
    "            if max_grad is None or max_grad_param > max_grad:\n",
    "                max_grad = max_grad_param\n",
    "    return max_grad\n",
    "\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "734f30e6-6b24-4d4b-ae91-e9a4b871113f",
   "metadata": {},
   "source": [
    "- Applying gradient clipping, we can see that the largest gradient is now substantially smaller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fa81ef8b-4280-400f-a93e-5210f3e62ff0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0201, device='mps:0')\n"
     ]
    }
   ],
   "source": [
    "torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b62c2af0-dac3-4742-be4b-4292c6753099",
   "metadata": {},
   "source": [
    "## D.4 The modified training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76715332-94ec-4185-922a-75cb420819d5",
   "metadata": {},
   "source": [
    "- Now let's add the three concepts covered above (learning rate warmup, cosine decay, and gradient clipping) to the `train_model_simple` function covered in chapter 5 to create the more sophisticated `train_model` function below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "46eb9c84-a293-4016-a523-7ad726e171e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import evaluate_model, generate_and_print_sample\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import evaluate_model, generate_and_print_samplee\n",
    "\n",
    "\n",
    "ORIG_BOOK_VERSION = False\n",
    "\n",
    "\n",
    "def train_model(model, train_loader, val_loader, optimizer, device,\n",
    "                n_epochs, eval_freq, eval_iter, start_context, tokenizer,\n",
    "                warmup_steps, initial_lr=3e-05, min_lr=1e-6):\n",
    "\n",
    "    train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []\n",
    "    tokens_seen, global_step = 0, -1\n",
    "\n",
    "    # Retrieve the maximum learning rate from the optimizer\n",
    "    peak_lr = optimizer.param_groups[0][\"lr\"]\n",
    "\n",
    "    # Calculate the total number of iterations in the training process\n",
    "    total_training_steps = len(train_loader) * n_epochs\n",
    "\n",
    "    # Calculate the learning rate increment during the warmup phase\n",
    "    lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        model.train()\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            global_step += 1\n",
    "\n",
    "            # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "            if global_step < warmup_steps:\n",
    "                # Linear warmup\n",
    "                lr = initial_lr + global_step * lr_increment  \n",
    "            else:\n",
    "                # Cosine annealing after warmup\n",
    "                progress = ((global_step - warmup_steps) / \n",
    "                            (total_training_steps - warmup_steps))\n",
    "                lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n",
    "\n",
    "            # Apply the calculated learning rate to the optimizer\n",
    "            for param_group in optimizer.param_groups:\n",
    "                param_group[\"lr\"] = lr\n",
    "            track_lrs.append(lr)  # Store the current learning rate\n",
    "\n",
    "            # Calculate and backpropagate the loss\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward()\n",
    "\n",
    "            # Apply gradient clipping after the warmup phase to avoid exploding gradients\n",
    "            if ORIG_BOOK_VERSION:\n",
    "                if global_step > warmup_steps:\n",
    "                    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)  \n",
    "            else:\n",
    "                if global_step >= warmup_steps:  # the book originally used global_step > warmup_steps, which led to a skipped clipping step after warmup\n",
    "                    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "                \n",
    "            optimizer.step()\n",
    "            tokens_seen += input_batch.numel()\n",
    "\n",
    "            # Periodically evaluate the model on the training and validation sets\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader,\n",
    "                    device, eval_iter\n",
    "                )\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                track_tokens_seen.append(tokens_seen)\n",
    "                # Print the current losses\n",
    "                print(f\"Ep {epoch+1} (Iter {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, \"\n",
    "                      f\"Val loss {val_loss:.3f}\"\n",
    "                )\n",
    "\n",
    "        # Generate and print a sample from the model to monitor progress\n",
    "        generate_and_print_sample(\n",
    "            model, tokenizer, device, start_context\n",
    "        )\n",
    "\n",
    "    return train_losses, val_losses, track_tokens_seen, track_lrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "55fcd247-ba9d-4b93-a757-0f7ce04fee41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Iter 000000): Train loss 10.969, Val loss 10.938\n",
      "Ep 1 (Iter 000005): Train loss 9.337, Val loss 9.460\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 2 (Iter 000010): Train loss 7.823, Val loss 8.186\n",
      "Ep 2 (Iter 000015): Train loss 6.343, Val loss 6.890\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 3 (Iter 000020): Train loss 6.083, Val loss 6.594\n",
      "Ep 3 (Iter 000025): Train loss 5.677, Val loss 6.786\n",
      "Every effort moves you, the of the of the of the of the, the of the of the, the, the, the, the, the of the, the, the, the, the, the, the, the, the, the, the, the\n",
      "Ep 4 (Iter 000030): Train loss 5.837, Val loss 6.822\n",
      "Ep 4 (Iter 000035): Train loss 6.136, Val loss 7.203\n",
      "Every effort moves youisisis a                                              \n",
      "Ep 5 (Iter 000040): Train loss 5.767, Val loss 7.003\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 6 (Iter 000045): Train loss 5.957, Val loss 6.874\n",
      "Ep 6 (Iter 000050): Train loss 5.633, Val loss 6.839\n",
      "Every effort moves you, the, the, the,, the, the, the, I, the, the,, I, I, the, the, the, the, the, the, the, the, the, the, the, the, the\n",
      "Ep 7 (Iter 000055): Train loss 5.699, Val loss 6.878\n",
      "Ep 7 (Iter 000060): Train loss 5.066, Val loss 6.622\n",
      "Every effort moves you, I had, I had a, I had a a a a of the a, I had, I had been, I I had a of the of the of the of the of the of the of the of the of the of the of\n",
      "Ep 8 (Iter 000065): Train loss 4.593, Val loss 6.548\n",
      "Ep 8 (Iter 000070): Train loss 4.278, Val loss 6.424\n",
      "Every effort moves you.                                                 \n",
      "Ep 9 (Iter 000075): Train loss 4.107, Val loss 6.279\n",
      "Ep 9 (Iter 000080): Train loss 3.146, Val loss 6.215\n",
      "Every effort moves you, in the picture\"I felt--I had been \" it.                                    \n",
      "Ep 10 (Iter 000085): Train loss 2.827, Val loss 6.177\n",
      "Every effort moves you know, and in a little of the Riv--I had a little of the Riv, and I was, and, and in the Riv of the of the fact of the fact--as of the, and, and--because.   \n",
      "Ep 11 (Iter 000090): Train loss 2.579, Val loss 6.135\n",
      "Ep 11 (Iter 000095): Train loss 2.277, Val loss 6.122\n",
      "Every effort moves you know the inevitable of the picture to have been--I turned. Gisburn's an--and it--I  \" to see a smile behind his pictures that he had married her. I had the fact, and it. Gisburn\n",
      "Ep 12 (Iter 000100): Train loss 2.231, Val loss 6.141\n",
      "Ep 12 (Iter 000105): Train loss 1.552, Val loss 6.148\n",
      "Every effort moves you know the inevitable garlanded frame. The mere, and Mrs.            \" to the picture; and I had been the donkey.  \"I had been the bull--and I had\n",
      "Ep 13 (Iter 000110): Train loss 1.800, Val loss 6.157\n",
      "Ep 13 (Iter 000115): Train loss 1.448, Val loss 6.145\n",
      "Every effort moves you know.\" \"I looked up his pictures--I had a little of a flash that he was, and to me to have to see a smile behind his pictures--I had married her--the, I had been the bull--that I had\n",
      "Ep 14 (Iter 000120): Train loss 1.557, Val loss 6.165\n",
      "Ep 14 (Iter 000125): Train loss 1.190, Val loss 6.164\n",
      "Every effort moves you know.\" \"I looked up his pictures--I had a single one in the house.\"        \"I must, and I looked up at my elbow and as I had been the man of the hour. \n",
      "Ep 15 (Iter 000130): Train loss 1.312, Val loss 6.158\n",
      "Every effort moves you know.\" \"I looked up his pictures--I had a single one in the house.\"        \"--it was his close grayish beard--as if he had the donkey. \"There were days when I\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "# Note:\n",
    "# Uncomment the following code to calculate the execution time\n",
    "# import time\n",
    "# start_time = time.time()\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "\n",
    "peak_lr = 0.001  # this was originally set to 5e-4 in the book by mistake\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=peak_lr, weight_decay=0.1)  # the book accidentally omitted the lr assignment\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "n_epochs = 15\n",
    "train_losses, val_losses, tokens_seen, lrs = train_model(\n",
    "    model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,\n",
    "    eval_freq=5, eval_iter=1, start_context=\"Every effort moves you\",\n",
    "    tokenizer=tokenizer, warmup_steps=warmup_steps, \n",
    "    initial_lr=1e-5, min_lr=1e-5\n",
    ")\n",
    "\n",
    "# Note:\n",
    "# Uncomment the following code to show the execution time\n",
    "# end_time = time.time()\n",
    "# execution_time_minutes = (end_time - start_time) / 60\n",
    "# print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "827e8d5e-0872-4b90-98ac-200c80ee2d53",
   "metadata": {},
   "source": [
    "- Looking at the results above, we can see that the model starts out generating incomprehensible strings of words, whereas, towards the end, it's able to produce grammatically more or less correct sentences\n",
    "- If we were to check a few passages it writes towards the end, we would find that they are contained in the training set verbatim -- it simply memorizes the training data\n",
    "- Note that the overfitting here occurs because we have a very, very small training set, and we iterate over it so many times\n",
    "  - The LLM training here primarily serves educational purposes; we mainly want to see that the model can learn to produce coherent text\n",
    "  - Instead of spending weeks or months on training this model on vast amounts of expensive hardware, we load the pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9decec45-4fdf-4ff6-85a7-1806613f8af7",
   "metadata": {},
   "source": [
    "- A quick check that the learning rate behaves as intended"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d8ebb8d2-8308-4a83-a2a6-730c3bf84452",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEmCAYAAACdy8LUAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAATmBJREFUeJzt3QlYVFX/B/AfMOw7IpsKYu64IQiiphluiXtpkSmiiZVvWdqilfI338KtTTOXFvEtU7FFDbcMzQ1kEQUUXErcUEBA9p05/+ccnJFRNFBg5g7fz/NcZ+6dMzPnzuD87tl1GGOMAAAAQFJ01Z0BAAAAqD8EcAAAAAlCAAcAAJAgBHAAAAAJQgAHAACQIARwAAAACUIABwAAkCAEcAAAAAmSqTsD2kwul9ONGzfI3NycdHR01J0dAABQAz5fWkFBATk5OZGubsOVmxHAGxEP3m3atGnMtwAAAIm4du0atW7dusFeDwG8EfGSt+JLs7CwaMy3AgAADZWfny8Kc4qY0FAQwBuRotqcB28EcACA5k2ngZtS0YkNAABAghDAAQAAJAgBHAAAQILUHsDXrFlDbdu2JSMjI/L29qaYmJiHpt++fTt17txZpO/evTvt2bPnvu76ixYtIkdHRzI2NqYhQ4bQxYsXVdJ8/PHH1K9fPzIxMSErK6ta3+fq1avk5+cn0tjZ2dE777xDlZWVDXDGAAAAEg/g27Zto7lz51JwcDDFx8dTz549afjw4ZSZmVlr+sjISPL396cZM2bQqVOnaNy4cWI7c+aMMs3y5ctp1apVtG7dOoqOjiZTU1PxmqWlpco05eXlNHHiRHr11VdrfZ+qqioRvHk6/p6bNm2i0NBQcWEAAACgEZgaeXl5sdmzZyv3q6qqmJOTEwsJCak1/aRJk5ifn5/KMW9vbzZr1ixxXy6XMwcHB7ZixQrl47m5uczQ0JBt2bLlvtfbuHEjs7S0vO/4nj17mK6uLktPT1ceW7t2LbOwsGBlZWV1Pr+8vDzGP2J+CwAAzVNeI8UCtQ0j46XbkydP0oIFC5TH+Aw1vMo7Kiqq1ufw47zEXhMvXe/YsUPcT01NpfT0dPEaCpaWlqJqnj/3hRdeqFPeeFpePW9vb6/yPrzEfvbsWXJ3d6/1eWVlZWKrOfYPGs9XBy/SnqR0MtTXJSOZnvLWxECPWloYkr25EdnxWwsjcd/JyohkempvNQIAaBBqC+BZWVmiqrpmkOT4/rlz52p9Dg/OtaXnxxWPK449KE1dPOh9ar5HbUJCQmjx4sV1fh94dFmFZfTZgQsk59e1dWQg06UnWppRR3u+mVMHOzPq3tqSHC2N8VUAgORgIpcGxGsTatYQKGbfgYa390y6CN6dHcxp7tCOVFopp9KKKiqrqKLCsiq6VVBGGQWllJlfShn5ZZSeX0rllXJKuZkvtppaWRlTn7bW5NnWhvq0tRGBXVcXc9cDgGZTWwC3tbUlPT09ysjIUDnO9x0cHGp9Dj/+sPSKW36M90KvmaZXr151zht/nXt7wyve90F54wwNDcUGjS884Ya4fbZ3axrm9uDvREEuZ3T9dgldyCigC5kFdCG9gM5nFIr9tNwSSjtdQjtOV7+mtYk+De5kR75d7GlgR1syN9Jv9PMBAJBMADcwMCAPDw+KiIgQPckVq3fx/f/85z+1PsfHx0c8/uabbyqPHThwQBznXF1dRYDlaRQBm5eCeW/0B/U4f9D78KFmvDc8H0KmeB8+HWrXrl0f67zh8fFSdczlHHF/ZI+7F2oPw0vUzi1MxDak693mkaKySjp9LZdiUnMo7koOnbqaS7eLK+jXU2li09fTIW/XFjSki514LztzI3yFAKAR1FqFzqubAwICyNPTk7y8vOiLL76goqIiCgwMFI9PnTqVWrVqJdqWuTlz5tCgQYPo008/FcO8tm7dSnFxcbRhwwblPLM8uP/3v/+lDh06iIC+cOFCsYSb4iJBMcY7JydH3PJ2+NOnT4vj7du3JzMzMxo2bJgI1FOmTBHD0ni794cffkizZ89GCVsD7Em6SYwR9Xa2EtXfj8PUUEb929uKjauoktPJK7cpIiWDIlIy6VJWER37O0tsH4Un05MdWtKE3q1oWFcHMjbQa6AzAgB4BEzNVq9ezZydnZmBgYEYVnbixAnlY4MGDWIBAQEq6cPCwljHjh1Fejc3N7Z7926Vx/lQsoULFzJ7e3sxfMzX15edP39eJQ1/TX7q926HDh1Sprl8+TJ75plnmLGxMbO1tWXz5s1jFRUV9To3DCNrHM+tPc5c3gtn3x29xBrbP5kF7Jsj/7Bxa46J91Rsbov2sXlhp1nc5WzxNwcA0NSxQIf/8yiBH/4dr77nw9jy8vKwGlkDuZlXQj4hB4kv6hM135ccLJuuSjs1q4h+O5VGv526TtdySpTHu7eypGn92tKono5kKEOpHACaJhYggDciBPCG992xVFoSnkxebW0o7JXqvg9NjV/zxl25Tdtir9GuhBuidztna2ZAL3o500t9XcjOAm3lANC4sQCzWoCkhCdW9xT3q2PntcbA+1rw4WYrJ/akEwt86Z3hncjBwoiyCstp1cG/acDyQ7RwxxnRux0AoLGgBN6IUAJvWNdvF9OAZYdE9Xn0+74a1SOcd37742wGfX88VXSC43gP9uc82tBrTz1BbWxM1J1FAFATlMCh2eO9zzlvVxuNCt6cvp6uqBX4+RUf2jKzL/m0a0EVVYy2xFylp1b+Re/+nEA3UCIHgAaEKnSQjN2J1QF8VA8n0lS8et3niRa0Jagvhc3yoSc72FKVnFFY3HUavPIvWr7vHOWXVqg7mwCgBRDAQRKuZhdTwvU84jOcjuj27zOvaQIvVxv6YYY3/fJqP3G/rFJOX//1Dz214i/aFHlZVLsDADwqBHCQhN13qs/7PWFLtmbSmq7Ww8WatgX1pQ1TPKhdS1PKKSqn4F1nadjnR+jQuUx1Zw8AJAoBHCRBE3qfP27VOp+zff+bA2nJuG5iyBkfVx4YGkuzfohDj3UAqDcEcNB4PNCdvZFPero6NKIOC5doMt7ZbUpfFzr09lM080lXcU77z2bQkE8P0/rD/6BaHQDqDAEcNN7uO6VvPl+5takBaQO+wtkHfl1p9xsDyNPFmkoqqihk7znyW3WU4u4s1AIA8DAI4KDxwpW9z6VZff4wnR0sRG/1Fc/1IBtTA7qQUUgT10fRR78nU0l5lbqzBwAaDAEcNNrfmQV0Lr1ATIoyvKu0q88fttTpRM82dHDeIJro0VqstMYnhBm56ijFojQOAA+AAA6SKH3zZTwtTfRJm1mZGNCKiT1pY2AfMTUrb/ufhNI4ADwAAjhIZPIW7as+f5DBnexo/1sDaZLn3dI4bxtPup6n7qwBgAZBAAeNdT69gC5mFpKBni4N6WpPzYmlsT4tf+5uafxSVhFNWHucNhz5h+RyrAAMAAjgIIHe54M6tSQLI+2uPn9YaXzfm0+K4XN8bvVP9pyjgI0xlJlfqu6sAYCaoQQOGomvua3Nvc/r2za+9qXeFDKhOxnp69LRi1k04sujFJGSoe6sAYAaIYCDRkq5WSCqjQ1luuTbpXlVnz9oJjd/L2cKf/1J6upoIaZjnbEpjkL2pFAl5lQHaJYQwEGjp07lVchmhjJ1Z0djtLczo99m96Pp/V3F/vojl+jFb6MpswBV6gDNDQI4aGT1uWLxklE9m3f1eW0MZXq0aHRX+npyb3FxE5OaQ36rjolbAGg+EMBB45xJy6cr2cVkrK9HT3e2U3d2NNbI7o608z/9qaO9Gd0qKCP/b07QN0cuiQsgANB+COCgccKTqqvPn+5iRyYGqD5/mCdamtGO2f1pXC8nqpIz+nhPCr2+5RSmYQVoBhDAQfOqzxW9z7uj+rwu+EXO58/3oiVj3UimqyN6709cH0k3cksa+dsCAHVCAAeNknA9j67fLiETAz0ajOrzevVSn+LTlja/7C0WReHNEGO+Ok4nr9xuzK8LANQIARw0SnhCdfX5kC72ZKSvp+7sSI53uxa0c3Z/6uxgTlmFZeS/4QT9fPK6urMFAI0AARw0Bp8iVNn7vJlP3vI42tiY0C+v9qPhbvZUXiWnt7cn0Cd7UjAFK4CWQQAHjXHq2m26mVdK5oYyGtixpbqzI2mmhjJaO9mD5vh2EPsbjlyi2T/FU2kF1hgH0BYI4KAxfk+oLn0P7Yrq84ZaZ/ytoR3pyxd6iQVh9p5JF0PNsgvLGuT1AUC9EMBBY6rP99ypPvdD9XmDGturFf0ww0uscHbqai6N/zqSLt0qbNg3AYAmhwAOGiH2cg5lFpSRhZGMnuyA6vPG6Nz262v9qI2NMV3NKaYJayMxcxuAxCGAg0ZQdF4b7uZABjL8WTbWpC+/vdaferWxotziCnrpu2jadya9Ud4LABqf2n8p16xZQ23btiUjIyPy9vammJiYh6bfvn07de7cWaTv3r077dmz576JQBYtWkSOjo5kbGxMQ4YMoYsXL6qkycnJocmTJ5OFhQVZWVnRjBkzqLBQtUpx//791LdvXzI3N6eWLVvSs88+S5cvX27AMwcFPoPYnqTqQILq88Zla2ZIW2b2pWFd7am8Uk6vbT5JW2Ou4o8RQILUGsC3bdtGc+fOpeDgYIqPj6eePXvS8OHDKTMzs9b0kZGR5O/vLwLuqVOnaNy4cWI7c+aMMs3y5ctp1apVtG7dOoqOjiZTU1PxmqWld1dr4sH77NmzdODAAQoPD6cjR45QUFCQ8vHU1FQaO3YsPf3003T69GkRzLOysmjChAmN/Ik0T9Gp2WLMspWJPvVvb6vu7Gg9YwM9sRDKC33akJwRzf81ib46eBFzqANIDVMjLy8vNnv2bOV+VVUVc3JyYiEhIbWmnzRpEvPz81M55u3tzWbNmiXuy+Vy5uDgwFasWKF8PDc3lxkaGrItW7aI/eTkZL7SA4uNjVWm2bt3L9PR0WFpaWlif/v27Uwmk4n8KOzatUukKS8vr/P55eXliffit/BgC35NZC7vhbP3fk7Ax9SE+P+XFfvOic+eb8E7z7CqKjm+A4AG1lixQG0l8PLycjp58qSo4lbQ1dUV+1FRUbU+hx+vmZ7jpWtFel5yTk9PV0ljaWkpquYVafgtrzb39PRUpuHp+XvzEjvn4eEh9jdu3EhVVVWUl5dHP/zwg0inr6/fwJ9E81ZZJVe2w47q4aTu7DS76VffHt6J/m90V7EfGnmZ3tx2WlStA4DmU1sA51XSPDja29urHOf7PAjXhh9/WHrF7b+lsbNTXaJSJpORjY2NMo2rqyv98ccf9P7775OhoaEI+NevX6ewsLCHnlNZWRnl5+erbPBwUZeyKaeonFqYGlDfdjb4uNRgWn9XMVacL4SyK+GGaBfHhC8Amk/tndg0EQ/kM2fOpICAAIqNjaXDhw+TgYEBPffccw9tJwwJCRElfsXWpk2bJs23FClWHhvRzYFkevhzVOdY8W8DPMlQpkt/pmTSzP/FYUlSAA2ntl9MW1tb0tPTo4yMDJXjfN/BwaHW5/DjD0uvuP23NPd2kqusrBQ90xVpeM94HoB5hzh3d3caOHAg/fjjjxQREaGsZq/NggULRHW7Yrt27Vo9PpHmp4JXn59F73NN8VQnOwoN9BIrwR29mEUB38dQYVmlurMFAJoWwHmJlrc186CoIJfLxb6Pj0+tz+HHa6bneE9yRXpe9c2DcM00vBqbB11FGn6bm5sr2t8VDh48KN6bt5VzxcXFog28Jn6xocjjg/Dqdj40reYGD3b87ywxHpkPbfJ2bYGPSgP4PNGCfpjhTeZGMoq5nEOTv42mvOIKdWcLAGrD1Gjr1q2ih3hoaKjoHR4UFMSsrKxYenq6eHzKlCls/vz5yvTHjx8XvcNXrlzJUlJSWHBwMNPX12dJSUnKNEuXLhWvsXPnTpaYmMjGjh3LXF1dWUlJiTLNiBEjmLu7O4uOjmbHjh1jHTp0YP7+/srHIyIiRI/zxYsXswsXLrCTJ0+y4cOHMxcXF1ZcXFzn80Mv9IebF3Za9H5euOPu9weaIel6Luu1eL/4fp754gjLKihVd5YAJCuvkXqhqzWAc6tXr2bOzs7MwMBADCs7ceKE8rFBgwaxgIAAlfRhYWGsY8eOIr2bmxvbvXv3fUNjFi5cyOzt7cXFga+vLzt//rxKmuzsbBGwzczMmIWFBQsMDGQFBQUqafiwMx7kTU1NWcuWLdmYMWPERUN9IIA/WGlFJesWvE8EiOhL2fX6XKFpnLuZzzyWHBDf0ZBP/2IZeXcvggFA/bFAh/+DuonGwavveVs6bw9HdbqqiJQMmrEpjuwtDClqvq9YOQs0D1/0hFej82Ve27Ywoc0z+1IrK2N1ZwtAUhorFqDbL6hF+J3e5yO7OyJ4a7B2Lc0obJaPWATlcnYxTVoXRddyitWdLQBAAAd14GOMDyRXjxQYhaVDNV4bGxMRxNvZmlJabolYU/z6bQRxAHVDCRya3OELt8TwJCdLI3JvY41vQAIcLY1pS1BfcrU1peu3S+jFb6LpRm6JurMF0KwhgIPaJm9B9bm02FsYiZXMXFqYiDXFeUk8Pe/uIkEA0LQQwKFJlZRX0Z8pd6rPe2Luc6lxsKwO4rxN/Ep2dRDPzEcQB1AHBHBoUn+dz6Ti8ipqbW1MPVtb4tOXICcrYxHEeW/01KwieoEH8QIEcYCmhgAOaul97tfDUayGBdLU2tqEtgZVB/FLt4pEmzhf0x0Amg4CODSZ4vJKijhXXX0+GkuHakXv9J9mepOjpRH9nVlIL35zgrIRxAGaDAI4NJmIlEwqrZCLTlBuTpgnXhu4tDCln2b2FRPyXMionvQlt7hc3dkCaBYQwKHJe5/zsd+oPtcefGgZD+ItzQ3pXHoBBWyMxSpmAE0AARyaBB/3feh89TKuft3R+1zbPNHSjDa/7E1WJvqUcC2XZm6KExP2AICGBXC+fvaff/5J69evp4KCAnHsxo0bVFhY2ND5Ay2a+7ysUk7tWppSF0dzdWcHGkFHe3P633QvMjOUUdSlbHptc7xY8x0ANCSAX7lyhbp3705jx46l2bNn061bt8TxZcuW0dtvv90YeQQt8HvCnerz7qg+12Y9WlvRdwGeZCjTpYPnMumtbaepSo71kgA0IoDPmTOHPD096fbt22RsfHdVovHjx1NERERD5w+0QF5JBR25UH2hh8lbtJ93uxa0booH6evpiGGDH/yWxJctVne2ALROvQP40aNH6cMPPyQDAwOV423btqW0tLSGzBtoiT+TM6i8Sk4d7MxENStov8Gd7OiL592JrxK7NfYa/Xd3CoI4gLoDuFwup6qq+zunXL9+nczN8eMM9wtPvCFuR2Hsd7PCJ+tZ+mwPcf+7Y6n0ZcRFdWcJoHkH8GHDhtEXX3yh3OfDgXjnteDgYBo5cmRD5w8kLq+4go5ezFL+oEPzMsmzDQWP7iruf/HnRfr26CV1ZwlAa8jq+4RPP/2Uhg8fTl27dqXS0lJ68cUX6eLFi2Rra0tbtmxpnFyCZO0/m06VckadHcypvZ2ZurMDahDY35UKSyvp0wMXRFW6tYkBPevRGt8FQFMH8NatW1NCQgJt27ZN3PLS94wZM2jy5MkqndoAuPCku5O3QPP1n6fbi86M3x5LpXd/SSRrU316urO9urMFIGk6rJ7dQ48cOUL9+vUjmUx239jwyMhIGjhwYEPnUbLy8/PJ0tKS8vLyyMKi+U0dmlNUTn0+/lMMIzr09lNixi5ovuRyRvO2J9Bvp9LISF9XTPzi4WKj7mwBSDYW1LsNfPDgwZSTk3PfcZ4x/hhAzepzHrz5vOcI3qCrq0PLn+tBT3VqKebEnx4aRxcyqieCAoAmCOC8wF7bPNbZ2dlkaooSFtyF3udwL309Xfp6cm9yd7YSVepTv4uhtNwSfFAAjdkGPmHCBHHLg/e0adPI0NBQ+RgfVpaYmCiq1gE4vjZ01D/Z4r5fd7R/w10mBjL6PqAPTVwfJZYhnfJdNP38Sj+yMVWdWwIAGqgEzuvv+cZL4Hy8t2Kfbw4ODhQUFEQ//vhjXV8OtNzeM+nEZ9Ds2dqSnFuYqDs7oGGsTQ3EvOl8LfFLt4ooMDSWisoq1Z0tAO0sgW/cuFE54xqf8xzV5fAwuzF5C/wLJytj+mGGFz23LkqsYPbq5nj6dqonGciwSCJAXdT7fwqfsAXBGx4mM7+UolOrOzqOxPAxeIj2dub0/bQ+ZKyvJ+bLf+fnBNFbHQAaYRw49/PPP1NYWBhdvXqVysvLVR6Lj49/lJcELas+54MTeztbUSsrzA0AD9fb2ZrWvtSbXt4URztP3yAHSyNa8EwXfGwADV0CX7VqFQUGBpK9vT2dOnWKvLy8qEWLFnTp0iV65pln6vtyoMW9z/0w9znU0VOd7JTzpq8/fIk2RV7GZwfQ0AH866+/pg0bNtDq1avFimTvvvsuHThwgN544w0xFhyat/S8Uoq9fFvcR+9zqI/nPFrTvKEdxf3/+/0s7TuTjg8QoCEDOK82VwwX41OnFhRUT8QwZcoUzIUOtPvO1Kl92lqLqlCA+k656u/lLJpg5mw9RSev3D9pFAA8YgDnQ8YUM7E5OzvTiRMnxP3U1FSs9wuYvAUeC59nYslYN/LtbEdllXKasSmO/rlViE8VoCEC+NNPP027du0S93lb+FtvvUVDhw6l559/nsaPH1/flwMtcv12MZ26mkt8or5nujmoOzsgUTI9XVr9oruYQyC3uIICvo+hzIJSdWcLQPoBnLd/f/DBB+L+7Nmz6fvvv6cuXbrQRx99RGvXrq13BtasWSPGlhsZGZG3tzfFxMQ8NP327dupc+fOIn337t1pz549Ko/ziWYWLVpEjo6Ooop/yJAhYrnTmngNAl89jU8qb2VlJVZT46uq3fs6K1eupI4dO4pZ51q1akUff/xxvc+vOdlzp/rc29WG7CxQfQ6PN1vbd9P6kEsLE7p+u4RmhMZhoheAe7F6qKioYIsXL2bXrl1jDWHr1q3MwMCAff/99+zs2bNs5syZzMrKimVkZNSa/vjx40xPT48tX76cJScnsw8//JDp6+uzpKQkZZqlS5cyS0tLtmPHDpaQkMDGjBnDXF1dWUlJiTLNiBEjWM+ePdmJEyfY0aNHWfv27Zm/v7/Ke73++uusU6dObOfOnezSpUssLi6O/fHHH/U6v7y8PD6gVdw2B2NWH2Uu74Wz/0VdVndWQEuk3ipk7h/9If6upn4Xzcorq9SdJYB6a6xYUK8AzpmamrLU1NQGeXMvLy82e/Zs5X5VVRVzcnJiISEhtaafNGkS8/PzUznm7e3NZs2aJe7L5XLm4ODAVqxYoXw8NzeXGRoasi1btoh9Hvj5BxkbG6tMs3fvXqajo8PS0tKUaWQyGTt37txjnV9zCuBXsorEj6zr/HB2q6BU3dkBLRJ/JYd1+nCP+Pt6Z/tp8f8cQEoaKxbUuwrd19eXDh8+TI+LTwBz8uRJUcWtoKurK/ajoqJqfQ4/XjM9N3z4cGV63pEuPT1dJQ2fq51XzSvS8Ftebe7p6alMw9Pz946Ojhb7v//+O7Vr147Cw8PJ1dVVVPG//PLLtS6jWlNZWZlY97Xm1tx6n/s80YJsze4udAPwuNydrWm1f2/S1SEKi7tOX0aoNokBNFf1nomNT9Yyf/58SkpKIg8Pj/umVR0zZkydXicrK0usYsYnhKmJ7587d67W5/DgXFt6flzxuOLYw9LY2dmpPC6TycjGxkaZhk9Kc+XKFdHe/r///U/kk3fWe+655+jgwYMPPKeQkBBavHgxNUdYOhQa09Cu9vTR2G704Y4z9MWfF8nJ0pgm9WmDDx2atXoH8Ndee03cfvbZZ7UOAeHBTurkcrkoTfPgzTuxcd999524YDl//jx16tSp1uctWLCA5s6dq9znJfA2bbT/RyY1q4jO3sgnPV0dGu6G3ufQOF7q60I380pozaF/aMFvSWRnYShmcANornQfJbg9aKtP8La1tSU9PT3KyMhQOc73+Vjz2vDjD0uvuP23NJmZmSqPV1ZWiupxRRreg52XyhXBm+M97RUT2TwI763Oe7bX3JrTymP929tiTWdoVG8P60QT3FtRlZzR7M3xlHyj+TRTAdxLbev28WlYeYk2IiJCeYxfBPB9Hx+fWp/Dj9dMz/FpXBXpeXs1D8I10/BSMG/bVqTht7m5uaL9XYFXi/P35m3lXP/+/UVQ/+eff5RpLly4IG5dXFwa6BPQHuGJ1e3fo7o7qjsroOV4LR+fM92nXQsqKq+i6aGxYvpegGaJqREfRsZ7iIeGhoqe30FBQWIYWXp6unh8ypQpbP78+SrDyHjv8JUrV7KUlBQWHBxc6zAy/hp8+FdiYiIbO3ZsrcPI3N3dWXR0NDt27Bjr0KGDyjAy3hu+d+/ebODAgSw+Pl4MIeO93YcOHVqv82sOvdAvZhSI3sHt39/NcovK1Z0daCb439rTKw+Jv70RXxxhBaUV6s4SgOYPI2toq1evZs7OzmI8OB9WxsdmKwwaNIgFBASopA8LC2MdO3YU6d3c3Nju3btVHudDTBYuXMjs7e3FxYGvry87f/68Sprs7GwRsM3MzJiFhQULDAxkBQUFKmn4kLIJEyaINPy1pk2bJp5XH80hgH9x4IL4EQ3cGKPurEAzczW7iHksqR4jHvB9NKvAGHHQUI0VC3T4P+quBdBWvPqeD2Pjq7Rpa3v40M8O08XMQvp0Yk961qO1urMDzczpa7n0woYoKq2Q02RvZ/rvuG6imh2gOcQCtbWBg/RdyCgQwdtAT5eGuqkO3QNoCr3aWNEXz7uL+fc3R1+lb45ewgcPzUa9A/i9E5UoNr6sKJ+cBZqP8ITq3ucDO7YkCyN9dWcHmqkR3Rzog5HVo0Q+2XNOOSc/gLardwDns5hZW1vft/HjfPEQ3ks7ODhY9OoG7cVbXhS9z0f3RO9zUK8ZA1wpwKd6hMhb207TySu38ZWA1qt3AA8NDSUnJyd6//33aceOHWLj9/lqXXw1sqCgIFq1ahUtXbq0cXIMGiHlZgFdyioiQ5ku+XZB9TmoF2/3XjT67jriM/8XR1eyi/C1gFar90xsmzZtok8//ZQmTZqkPDZ69GixtOf69evFGGxnZ2ex9CYP7KDdU6cO7mRHZob1/jMCaHB8JsBV/u70/IYoOpOWT4EbY+nX1/qRlYkBPm3QSvUugUdGRpK7u/t9x/kxxYIhAwYMeOiMZSD96nPF4iV+PVB9DprD1FBG3wf0ISdLI1FDFPS/k1RWKf3pnQEaJIDzub35vOD34scU835nZ2eLdnHQTrx0cyW7mIz0efU55qIGzWJnYUTfB/Yhc0MZxVzOoXd/ThQXnQDapt51nytXrqSJEyfS3r17qU+fPuJYXFycWEHs559/FvuxsbH0/PPPN3xuQSOEJ1VXn/t2ticTA1Sfg+bp7GBBX7/UW1Sj7zx9g5xtTGjesNoXIQKQqkeayIWvu83buxXzg/PVuWbNmiXWzQbtnsiF/7k8ufwQXb9dQmsn96ZnMP85aLBtsVfpvV+SxP3lz/WgSZ7avzogNJ9Y8EjFJ75oCHqZN08J1/NE8DYx0MNSjqDxnu/jTFdzisUSpO//miTWER/QwVbd2QJQXwDnq3nFxMSIZTnvHe89derUhskZaPTkLUO62JOxgZ66swPwr+YN7UTXckpoV8INevXHk/Tzq/2ok4M5PjlofgH8999/p8mTJ1NhYaGoCqg57zC/jwCuveRyppzlCr3PQSp0dXVoxcQedDOvhGIv3xZLkP72Wj/R2Q2gWfVCnzdvHk2fPl0EcF4Sv337tnLLyclpnFyCRjh17TbdyCsV474HdWyp7uwA1JmhTI82TPEkV1tTSsstoRmb4qi4vBKfIDSvAJ6WlkZvvPEGmZiYNE6OQGMppk4d2tWejPRRfQ7SYm1qQBun9SEbUwNKSsujN7acoio5hpdBMwrgw4cPF8PGoPlWn4/C5C0gUW1tTembqR5kINOlP1MyaUl4srqzBNB0beB+fn70zjvvUHJyspg+VV9fdRWqMWPGPHpuQGPFXblNGfllZG4koyc7oPocpMvDxYY+n9SLZv8UT6GRl8UY8ekDXNWdLYDGD+AzZ84Utx999NF9j/FObFVVmLZQm+c+H+7mIEovAFLGO2Fev92ZQvaeoyW7k6mVtbH42waQknr/EvNhYw/aELy1E28n3JOULu6j+hy0RdDAdjTZ25n4VFZztp6i09dy1Z0lgHpBUQr+VXRqNmUVlpGViT71b49JMEA78BrDxWPc6KlOLam0Qk4vb4qlaznF6s4WQMNWofP1vfk630ZGRuL+w/Ae6qCdvc9HuDmQvh6u+UB7yPR06asXe9OkdVGUfDOfpm2MoV9f7U+WJqp9ewA0UZ3mQudTp/Ke5y1atBD3H/hiOjp06dKlhs6jZGnDXOiVVXLy+iSCcorK6YcZXujABlopPa+Uxq05Tun5pdS3nQ1tmu4lxo4DSH4udL54SW33QftFXcoWwZuPnfVp10Ld2QFoFA6WRrQxsA9NXBdFJy7l0IJfkujTST1VZpoE0DSoD4WH2q2oPu/mIKobAbRVF0cLWjO5N+np6tCvp9Loiz8vqjtLAA07jIz3NA8NDaWIiIhaFzM5ePBgfV8SNFRFlZz2nUXvc2g++BTB/x3XjRb8mkRfRlykNjYm9JxHa3VnC6BhAvicOXNEAOcTunTr1g1VTFrs+N9ZlFtcQbZmhuTtiupzaB78vZxFb/Sv//qH5v+SSI6WRhh9AdoRwLdu3UphYWE0cuTIxskRaFzv85HdHUS1IkBz8fawTnTtdgn9nnCDXvnxJP3yaj/qaI8lSEGz1LtR08DAgNq3b984uQGNUV4pp/13qs/9ujuqOzsATb8E6XM9yNPFmgpKKylwYyxlFpTiWwDpLyf65ZdfUh1Gn4GEHb14S/xw2ZkbUp+2NurODkCT4yvubZhaYwnSUCxBChKvQj927BgdOnSI9u7dS25ubvctZvLrr782ZP5Azb3PR3Z3FKURgObI5s4SpOO/Pn5nCdLTtH6KB5qUQJolcCsrKxo/fjwNGjSIbG1txeD0mhtIX2lFFf2RnCHuj+6J6nNo3vgSpN8GeN5ZgjQDS5CCNEvglZWVNHjwYBo2bBg5OGDlHm115MItKiyrFL1v3dtYqzs7AGqHJUhB8iVwmUxGr7zyCpWVlTVoJtasWUNt27YVc617e3tTTEzMQ9Nv376dOnfuLNLzNcn37Nmj8jhvn1+0aBE5OjqSsbExDRkyhC5eVJ2UIScnhyZPniymteO1CjNmzKDCwsJa3+/vv/8mc3Nzka459T7nnddQfQ5wdwnSBc90Fvf5EqSKTp4AkqlC9/LyolOnTjVYBrZt20Zz586l4OBgio+Pp549e9Lw4cPFJDG1iYyMJH9/fxFweT7GjRsntjNnzijTLF++XCy6sm7dOoqOjiZTU1PxmqWld3uR8uB99uxZOnDgAIWHh9ORI0fEgi33qqioEO/35JNPUnNQUl4lqgm5UT2d1J0dAI2CJUhBo7B62rZtG2vXrh1bvXo1i4yMZAkJCSpbfXl5ebHZs2cr96uqqpiTkxMLCQmpNf2kSZOYn5+fyjFvb282a9YscV8ulzMHBwe2YsUK5eO5ubnM0NCQbdmyRewnJyfzLvQsNjZWmWbv3r1MR0eHpaWlqbz2u+++y1566SW2ceNGZmlpWa9zy8vLE+/Db6ViT+IN5vJeOOu/NEJ8lgCgqqKyigV8Hy3+n3gs+YNdzS7CRwRqiQX1LoG/8MILYkETvmxo//79qVevXuTu7q68rY/y8nI6efKkqOJW0NXVFftRUVG1Pocfr5me46VrRXqet/T0dJU0vHMdr5pXpOG3vDrc09NTmYan5+/NS+w1p4Xl1fW8ir8ueNMCX3Wm5ibZ6vMejphlD+AhS5B2dbSgrMJyCvg+Riz4A6Dxw8gacjWyrKwsMbe6vb29ynG+f+7cuVqfw4Nzben5ccXjimMPS2NnZ3df+76NjY0yTXZ2Nk2bNo1+/PHHOi//FhISQosXLyapKi6vpIhzd6rPu6P6HOBBzAxlYvWyCV9H0qWsIpqxKZZ+erkvGRtgCVJoOvUugbu4uDx00xYzZ86kF198kQYOHFjn5yxYsECs96rYrl27RlISkZJJpRVycmlhQt1aSXP9coCmYm9hRJum9yFLY306dTWXXt9yiiqrVBd3AtCoErhCcnIyXb16VVSD1zRmzJg6vwYfR66np0cZGdWlPgW+/6Bhavz4w9Irbvkx3gu9Zhpeza9Ic28nOT5EjvdMVzyfV5/v2rWLVq5cqezZzlde4yX1DRs20PTp0+/Lm6GhodikPnkL732OdZAB/l17O3MxRnzyt9Gi8+fCnWfpk/FY5Ak0tAR+6dIl0VOcr0TGVyRT9ALnk7vwrb7zqnt4eIilSRV4kOT7Pj4+tT6HH6+ZnuM9yRXpXV1dRRCumYa3RfO2bUUafpubmyva3xV4wObvzdvKFe3kp0+fVm4fffSRGErG79f3PKWAj/s+dL76omZUD1SfA9QVn2p41Qu9SEeHaEvMVfrq4N/48KBp1LfX26hRo9jYsWPZrVu3mJmZmejRffToUdGb/MiRI/XuRbd161bRQzw0NFS8VlBQELOysmLp6eni8SlTprD58+cr0x8/fpzJZDK2cuVKlpKSwoKDg5m+vj5LSkpSplm6dKl4jZ07d7LExESRX1dXV1ZSUqJMM2LECObu7s6io6PZsWPHWIcOHZi/v/8D86ntvdB3nLouetUOXnEIvc8BHsGmyFTxf4hv22Ku4jOERo8F9Q7gLVq0UA4Xs7CwYOfOnRP3IyIiWK9evR4pE3xImrOzMzMwMBAXAidOnFA+NmjQIBYQEKCSPiwsjHXs2FGkd3NzY7t371Z5nA9/WrhwIbO3txcXB76+vuz8+fMqabKzs0XA5hch/DwCAwNZQUFBsw3gM0JjxQ/Pp/urv08AqL9le1PE/6N2C3azgykZ+AihUWOBDv+nPiV2a2trMeEKr6p+4okn6NtvvxXTq/7zzz9iVrTi4uLGqy6QGF51z4ew8Q5tde3Jrg75pRXkueRPKq+S0/43B1InB6x7DPAo+M/pvO0J9Gt8Ghnr69HWoL7Us03zmMERmj4W1LsNnLd9JyQkiPu8vZjPenb8+HHRRtyuXbsGyxg0nQNnM0Tw7mBnhuAN8Bh4589lz/agJzvYUklFFU0PjaXLWUX4TKFR1DuAf/jhh6KzF8eDNh8XzqcZ5fOR8+lLQXp2J92dvAUAHo++ni6tfcmD3JwsKLuonAI2xlBWYcOuHwHA1bsKvTZ8+BWvWsfQI+lVoecVV5DnxweooorRn3MHimExAPD4MgtK6dm1kXQtp4R6tLakLTP7kqnhI4/cBQnL15Qq9JordO3fv59KSkrEDGYgTfuT00Xw7uxgjuAN0IDszI1oU6AXWZvoU+L1PHptczxVYKIXaED1DuB8ilFfX1/q2LEjjRw5km7erK5+5auDzZs3ryHzBk049/koVJ8DNLh2Lc3ou2l9yEhflw5fuEXvbE8gufyxKz0BHi2Av/XWW6Svry9mYTMxMVEef/7552nfvn31fTlQI74Aw/G/s8R9P0zeAtAoejtb09rJHiTT1aEdp2/QR+HJorc6QJMH8D/++IOWLVtGrVu3VjneoUMHunLlymNnCJrO/rPpVCVnorONq60pPnqARjK4sx2tnNhT3A+NvEyrMVsbqCOAFxUVqZS8a3Zkk/I84M1ReOINcYupUwEa3zj3VhQ8uqu4/9mBC/RD1GV87NC0AZwPGfvf//6n3Oc9z/mwMj4enE/oAtLAh7VE/ZOtXLwEABpfYH9XesO3g7i/aNdZ2pVQfREN8CjqPaaBB2reiS0uLk6sRPbuu+/S2bNnRQmcT+gC0rD3TDrxvjQ9W1uSc4v7a1QAoHG8NaQD3S4qpx9OXKG5206ThZGMnupkh48bmmYmtgsXLtCAAQNo7Nixokp9woQJdOrUKTG1KkjD7jvV55i8BaBp8VrLxWPcaHRPJ6qUM3r1x3g6eeU2vgaot0eaVYAPSP/ggw9Ujl2/fp2CgoLEWtmg2TLzSyk6NUfcH4nqc4Amp6urQ59O7El5JRV05MItMeXq9ld8qKM9JlKCJpjIpbbx4d99911DvRw0cvU5H8Xi7mxFra1RfQ6gDgYyXVr3Um/x/5AH8pe+jaYr2Zg3HdQQwEE60PscQDOYGMho47Q+1MnenDILyujFb6IpLbdE3dkCiUAAb2bS80op9nJ1e9vI7g7qzg5As2dlYkA/vOxF7WxNRfDmJXE+jzrAv0EAb6Yrj/Vpa02Olsbqzg4A3Jk3/ceXvamVlTGlZhWJIM5nSgRokE5svKf5w+Tm5tb1pUATep+j8xqARnGyMhYrlk1cH0kXMgpp6vfRtPnlvmRprK/urIHUS+C85/nDNhcXF5o6dWrj5hYeC6+ei7+aSzo66H0OoIn4nAw8aLcwNaAzafkUuDGGisoq1Z0tkHoJfOPGjY2bE2h0e+6sPObV1obsLIzwiQNooPZ2ZvTDDG/y/+aEuOB+eVMcbQzkK5rpqTtroGHQBt4ce5/3dFJ3VgDgIbo6WdCm6V5kaqBHUZey6dUfT1J5pRyfGahAAG8mruUUU8L1PNLVIRrhht7nAJquVxsr+v7OWuKHzt+iN7acoooqBHG4CwG8mQi/U33u80QLammOVeMApMC7XQvaMMWTDPR0ad/ZdHr9JwRxuAsBvJlVn/t1R/U5gJQM7NiS1k/1UAbx//wUj5I4CAjgzQAfV3r2Rj7p6erQiG6oPgeQmsGd7KqDuEyX9p/NEEEcbeKAAN6Mxn73e6IF2ZgaqDs7APCIQXzDlLtB/PUtCOLNHQJ4M2r/Ht0D1ecAUvbUPUEcJfHmDQFcy/2dWUjn0gtIX0+HhqP3OYBWBPFvpnqKIP5HcgbNRnV6s4UAruV23yl9D2hvS5YmmJIRQBsM6thSGcQPIIg3WwjgWg5LhwI0jyDOJ3sprahSd7agCSGAa7ELGQV0MbNQDD8Z6mav7uwAQCME8W+nepKhTJcizmXSjE2xmDu9GUEA12LhCTeU40gtjFB9DqCN+P9vxbSrx//OpinfRVNeSYW6swVNAAFcSzHGKPzO2t+jejiqOzsA0Ij6tmsh1hO3MJKJBVBe/OYEZReW4TPXchoRwNesWUNt27YlIyMj8vb2ppiYmIem3759O3Xu3Fmk7969O+3Zs+e+4LVo0SJydHQkY2NjGjJkCF28eFElTU5ODk2ePJksLCzIysqKZsyYQYWFhcrH//rrLxo7dqx4DVNTU+rVqxdt3ryZpCLlZgFdulUk2seGdEX1OYC2c3e2pq1BPmIpUj5x0/MbTlBGfqm6swXaHMC3bdtGc+fOpeDgYIqPj6eePXvS8OHDKTMzs9b0kZGR5O/vLwLuqVOnaNy4cWI7c+aMMs3y5ctp1apVtG7dOoqOjhYBmL9maendP2YevM+ePUsHDhyg8PBwOnLkCAUFBam8T48ePeiXX36hxMRECgwMFOud87RSsDupuvp8cKeWZGZY51VjAUDiq5iFveJDDhZGYgjpc+si6XJWkbqzBY2FqZmXlxebPXu2cr+qqoo5OTmxkJCQWtNPmjSJ+fn5qRzz9vZms2bNEvflcjlzcHBgK1asUD6em5vLDA0N2ZYtW8R+cnIy46ceGxurTLN3716mo6PD0tLSHpjXkSNHssDAwDqfW15enngfftuU+GcwcPlB5vJeONt1+sHnAwDa6Wp2EXtyWfVvgMeSP1jS9Vx1Z6lZy2ukWKDWEnh5eTmdPHlSVHEr6Orqiv2oqKhan8OP10zP8dK1In1qaiqlp6erpLG0tBRV84o0/JZXm3t6eirT8PT8vXmJ/UHy8vLIxsbmgY+XlZVRfn6+yqYOvPrsSnaxWIbw6c52askDAKhPGxsT+vlVH+riaEFZheX0woYTFPlPFr4SLaPWAJ6VlUVVVVVkb6/aRsv3eRCuDT/+sPSK239LY2enGthkMpkIzg9637CwMIqNjRVV6Q8SEhIiLhYUW5s2bUgdfr8z97lvZ3syRfU5QLNkZ25E22b1pb7tbKiwrJKmfR9L+85Ud2wF7aD2NnApOHTokAjc33zzDbm5uT0w3YIFC0QpXbFdu3aNmhrvwKeYfc0Pvc8BmjU+fDQ00ItGuDlQeZWcXtscT5ujr6g7W6ANAdzW1pb09PQoIyND5Tjfd3CofdlLfvxh6RW3/5bm3k5ylZWVomf6ve97+PBhGj16NH3++eeiE9vDGBoail7tNbemlnA9j67fLiETAz2xehEANG9G+nq0ZnJv8vdyJjkj+uC3M7Ry/3lxsQ/SptYAbmBgQB4eHhQREaE8JpfLxb6Pj0+tz+HHa6bneE9yRXpXV1cRhGum4W3RvG1bkYbf5ubmivZ3hYMHD4r35m3lNYeS+fn50bJly1R6qEth8hbfLvZkbKCn7uwAgAbQ09WhT8Z3ozd8O4j9rw79TW9uO01llZh6VdKYmm3dulX0EA8NDRW9w4OCgpiVlRVLT08Xj0+ZMoXNnz9fmf748eNMJpOxlStXspSUFBYcHMz09fVZUlKSMs3SpUvFa+zcuZMlJiaysWPHMldXV1ZSUqJMM2LECObu7s6io6PZsWPHWIcOHZi/v7/y8YMHDzITExO2YMECdvPmTeWWnZ2tsb3Qq6rkzOeTP0XP031nbjbJewKAtGyLvcqeWLBb/E5MXBfJbheVqTtLWi+vkWKB2gM4t3r1aubs7MwMDAzEsLITJ04oHxs0aBALCAhQSR8WFsY6duwo0ru5ubHdu3ffN4xq4cKFzN7eXlwc+Pr6svPnz6uk4YGYB2wzMzNmYWEhhocVFBQoH+fvyT/wezeeH00N4HGXs8V/SrdF+1hJeWWTvCcASM/RC7dYt0X7xO/F4JWH2JWsInVnSavlNVIs0OH/qLsWQFvxqnveG513aGuK9vDFv5+ljccv03j3VvT5870a/f0AQLrOpxdQ4MYYupFXKmZv2zDVgzxcHjxMFjQvFqAXupaQyxntwdznAFBHnRzM6bfZ/albKwvKLion/w3RtD2u6UfOwKNDANcScVduU0Z+GZkbyWhAB1t1ZwcAJMDewoi2BfnQcDd7MczsnZ8TaUl4MlVWydWdNagDBHAtEX5n8pbhbg5kKEPvcwCoGz7Z09rJHjTnTg/1746l0vRNcViSVAIQwLVAlag+r55BDpO3AEB96erq0FtDO9LXk3uTsb4eHblwi8avOS4WRAHNhQCuBaJTsymrsIwsjfVpQHtUnwPAoxnZ3VHMod7KypguZRXRuDXHlX1rQPMggGsBxdSpfLpEfT18pQDw6NycLGnnf/qTt2v1HOp8+lXeLl6BdnGNg197ieOdTfadqa4+H9XTUd3ZAQAtYGtmSJtf9qZZg9op28X9N5ygjPxSdWcNakAAl7gTl3LEEBAbUwPyaddC3dkBAC0h09OlBc90oXUveZC5oUyMdPFbdRTLkmoQBHAt6X0+opuD+A8HANCQ+G/LrtcHUGcHc7G2+ORvo+nTP86jSl0D4Bdfwnib1L6zd6rPu6P6HAAah6utKf32Wn+a5Nma+Nydqw/+TZPWR9G1nGJ85GqEAC5hx//OotziCrI1MyBvVJ8DQCPiqxsuf64nrfZ3FxNGnbqaSyO/PEo7T6fhc1cTBHAJC7/T+/yZbo5iuUAAgMY2uqcT7XnjSfJwsaaCskqas/U0zQ07TfmlFfjwmxgCuESVV8ppv6L6vAeqzwGg6bSxMaFtQX3F7G287PBrfBoN//wI/XU+E19DE0IAl6ijF29RQWkl2ZkbkmdbrCAEAE2Ld5rls7eFzfKhti1M6GZeKU3bGEvv/pyA0ngTQQCX+OQtfOYkVJ8DgLrwAsTeOQNpen9X0tEhCou7jtJ4E0EAl6DSiir6IzlD3Ef1OQBoQge3RaO7ipXNapbG39x6ijILMPlLY0EAlyC+0ACf4tDR0oh6O1urOzsAAIKXq2ppfMfpG+S78jBtirwsFl2ChoUALuHe537dHcUqQgAAmlYa3zm7P/VobSl6qgfvOktj1xyj09dy1Z09rYIALsHq8z9TqqvPsXQoAGiqHq2txOQv/x3XjSyMZHQmLZ/Gf32c3vs5EXOqNxAEcIk5dC6TisurxHJ/vdpYqTs7AAAPxDvYvtTXhQ6+/RQ927t6FrdtcdfoqRV/0WcHLlBRWSU+vceAAC4x4XfW5uWd13R4IxMAgARWN/t0Uk/65VUf6u1sRSUVVbQq4iINWvEXbY6+IlZVhPpDAJeQ4vJKOphSPVHCqB5O6s4OAEC9eLjY0C+v9qO1k3uL3upZhWX0wW9naNjnR+jX+OsI5PWEAC4hB89liitXZxsT6tbKQt3ZAQCoN15z+Ex3R/rjrUH0f6O7krWJPl3KKqK5YQnk+9lhCou7hpXO6ggBXELCE1B9DgDawUCmS9P6u9LR956m90Z0JhtTA7qSXUzv/pxIg1dWV63zTrvwYDqM8W4F0Bjy8/PJ0tKS8vLyyMLi8UrMfNy3x5IDVFYpp91vDCA3J8sGyycAgCY0EW4+cZXWH7kkqtY5KxN9eqGPM03xcREdd6UqvwFjQU0ogUtEREqGCN7tbE2pqyOqzwFAu5gYyGjmwHZ09N3BtGhUV2ptbSyWS153+B96ctlBevXHkxR9KZtQ5rxLVuM+aLDf71Sf87Hf6H0OANo8Ecz0Aa4U0K+tKLiERl6myH+yae+ZdLG52prSePdWYuOrojVnqEKXQLUJX2fXc8mfVF4lp/1vDqRODuYNmk8AAE12Pr2ANkVdpt/i00RHXgWvtjY0oXcrGtHNgaxMDKi5VaEjgEvgS/vl5HWatz2B2tuZ0YG3BqIEDgDNEp/4Zd+ZdPrtVBod/ydLTAyjmDDGw8WahnSxI98u9vRESzNqDgEcVegSsBuTtwAAkKmhjJ71aC22m3kltPP0DdpxKo3OpRdQTGqO2D7Zc05Usw/q2FIsruLpYk12FkZa+emhBK7hV115xRXk+fEBqqhi9OfcgdTeDtXnAAA1XcspFu3lEecy6cSlbPF7WZNLCxPydLGh3i5W1NnBnDrYm5OFkT41Fa3uhb5mzRpq27YtGRkZkbe3N8XExDw0/fbt26lz584ifffu3WnPnj0qj/NeiosWLSJHR0cyNjamIUOG0MWLF1XS5OTk0OTJk8WHaWVlRTNmzKDCwkKVNImJifTkk0+K92nTpg0tX76cmtr+5HTxx8j/6BC8AQDu18bGRIwp/2GGN8UvHCpmegvwcREjdviM03x8+S/x18Wsb8+ujaIe//cH+YRE0NTvY2hJeDJ9dyyVwhNviBL8lewiKimXxvhztVehb9u2jebOnUvr1q0TwfuLL76g4cOH0/nz58nOzu6+9JGRkeTv708hISE0atQo+umnn2jcuHEUHx9P3bp1E2l4oF21ahVt2rSJXF1daeHCheI1k5OTRTDmePC+efMmHThwgCoqKigwMJCCgoLE6ymumIYNGyaCP89bUlISTZ8+XQR7nk4dS4cCAMDDmRvpi5ne+KboBBx/5TbFXb5NCddz6WJGIaXnl9LNvOrtyIVbD5xoxtxQJqrtZw1qR5O9XTTuo1d7FToP2n369KGvvvpK7MvlclHaff3112n+/Pn3pX/++eepqKiIwsPDlcf69u1LvXr1EoGWn46TkxPNmzeP3n77bfE4r7awt7en0NBQeuGFFyglJYW6du1KsbGx5OnpKdLs27ePRo4cSdevXxfPX7t2LX3wwQeUnp5OBgbVvRt5fnbs2EHnzp1rkmqTgtIK6r2kuvr80NtPiXYdAAB4PHklFfR3ZgGdTy+kf24ViuVNM/PLKKOgVNwvrVBdXOVDvy708pPtHvn9tLITW3l5OZ08eZIWLFigPKarqytKvVFRUbU+hx/nJfaaeOmaB1YuNTVVBF3+Ggr8g+MXCvy5PIDzW16SVgRvjqfn7x0dHU3jx48XaQYOHKgM3or3WbZsGd2+fZusra3vy1tZWZnYan5pj3slefidwXT87ywEbwCABmJprC8WVuHbvXghsKCskvJLKqiorErMgqmps8CptQ08KyuLqqqqROm4Jr7Pg3Bt+PGHpVfc/luae6vnZTIZ2djYqKSp7TVqvse9eLU+v1hQbLwm4XE5WRnTRM/Hfx0AAPh3fKIs3sGttbWJmHODD09zsNTMXuwa0YlNW/CaBF5FotiuXbum7iwBAICWUmsAt7W1JT09PcrIyFA5zvcdHBxqfQ4//rD0itt/S5OZWb2utkJlZaXomV4zTW2vUfM97mVoaCjaN2puAAAAWhfAefuyh4cHRUREKI/xTmx838fHp9bn8OM103O8J7kiPe91zgNszTS8LZq3bSvS8Nvc3FzR/q5w8OBB8d68rVyR5siRI6KHes336dSpU63t3wAAAE2KqdnWrVuZoaEhCw0NZcnJySwoKIhZWVmx9PR08fiUKVPY/PnzlemPHz/OZDIZW7lyJUtJSWHBwcFMX1+fJSUlKdMsXbpUvMbOnTtZYmIiGzt2LHN1dWUlJSXKNCNGjGDu7u4sOjqaHTt2jHXo0IH5+/srH8/NzWX29vbi/c+cOSPyaWJiwtavX1/nc8vLy+M9/MUtAAA0T3mNFAvUHsC51atXM2dnZ2ZgYMC8vLzYiRMnlI8NGjSIBQQEqKQPCwtjHTt2FOnd3NzY7t27VR6Xy+Vs4cKFIgDziwNfX192/vx5lTTZ2dkiYJuZmTELCwsWGBjICgoKVNIkJCSwAQMGiNdo1aqVuDCoDwRwAADIa6QArvZx4Nqsscb+AQCAdGj1VKoAAAAgsalUtZmicuNxJ3QBAADpyr8TAxq6whsBvBEVFBSI24aY0AUAAKQfEywtLRvs9dAG3oj4sLQbN26Qubm5mN3nUa/c+AUAnxRGW9rRte2ctO18tPGctO18OJyTdL6jq1evihjA19ngU3Y3FJTAGxH/olq3bt0gr6WNE8No2zlp2/lo4zlp2/lwOCfNx0vdjfF3h05sAAAAEoQADgAAIEEI4BqOz68eHBwsbrWFtp2Ttp2PNp6Ttp0Ph3PSfI39HaETGwAAgAShBA4AACBBCOAAAAAShAAOAAAgQQjgAAAAEoQAruHWrFlDbdu2JSMjI/L29qaYmBiSgpCQEOrTp4+Yhc7Ozo7GjRtH58+fV0lTWlpKs2fPphYtWpCZmRk9++yzlJGRQVKwdOlSMbPSm2++KenzSUtLo5deeknk2djYmLp3705xcXHKx/nczYsWLSJHR0fx+JAhQ+jixYukqaqqqmjhwoXk6uoq8vvEE0/QkiVLVOag1uRzOnLkCI0ePVrM2MX/vnbs2KHyeF3ynpOTQ5MnTxYTh1hZWdGMGTOosLCQNPGcKioq6L333hN/d6ampiLN1KlTxQyWUj2ne73yyisizRdffNHg54QArsG2bdtGc+fOFcMQ4uPjqWfPnjR8+HDKzMwkTXf48GERzE6cOEEHDhwQ/1GHDRtGRUVFyjRvvfUW/f7777R9+3aRnv+nnTBhAmm62NhYWr9+PfXo0UPluNTO5/bt29S/f3/S19envXv3UnJyMn366adkbW2tTLN8+XJatWoVrVu3jqKjo8WPLP8b5BcrmmjZsmW0du1a+uqrryglJUXs83NYvXq1JM6J///g/8/5hXtt6pJ3HhTOnj0r/t+Fh4eLYBMUFESaeE7FxcXit41fdPHbX3/9VVzojxkzRiWdlM6ppt9++038BvJAf68GOScsta65vLy82OzZs5X7VVVVzMnJiYWEhDCpyczMFAvaHz58WOzn5uYyfX19tn37dmWalJQUkSYqKoppqoKCAtahQwd24MABNmjQIDZnzhzJns97773HBgwY8MDH5XI5c3BwYCtWrFAe4+dpaGjItmzZwjSRn58fmz59usqxCRMmsMmTJ0vunPjfzm+//abcr0vek5OTxfNiY2OVafbu3ct0dHRYWloa07Rzqk1MTIxId+XKFUmf0/Xr11mrVq3YmTNnmIuLC/v888+VjzXUOaEErqHKy8vp5MmTooqs5tzqfD8qKoqkhi9kz9nY2Ihbfm68VF7z/Dp37kzOzs4afX68VsHPz08l31I9n127dpGnpydNnDhRNHO4u7vTN998o3w8NTWV0tPTVc6Jz+nMm3I09Zz69etHERERdOHCBbGfkJBAx44do2eeeUay56RQl7zzW14dy79XBZ6e/3bwErtUfit4lTM/D6mek1wupylTptA777xDbm5u9z3eUOeExUw0VFZWlmjPs7e3VznO98+dO0dSwv+YeVsxr67t1q2bOMZ/iAwMDJT/SWueH39ME23dulVU8/Eq9HtJ8XwuXbokqpt5M837778vzuuNN94Q5xEQEKDMd21/g5p6TvPnzxcrQPGLJz09PfF/6OOPPxbVlZwUz0mhLnnnt/xirCaZTCYunDX9/DjeFMDbxP39/ZWLf0jxnJYtWybyyP8/1aahzgkBHJqk1HrmzBlREpIqvgzlnDlzRHsV71CoDfiFFS8BfPLJJ2Kfl8D598TbV3kAl6KwsDDavHkz/fTTT6Lkc/r0aXHxyNsgpXpOzQWvwZo0aZLoqMcvLKXq5MmT9OWXX4qL/UddRrquUIWuoWxtbUUJ4t5ezHzfwcGBpOI///mP6KBx6NAhlaVV+TnwZoLc3FxJnB//T8k7D/bu3VtcKfONd1TjHYr4fV4KktL5cLwnc9euXVWOdenSRaxdzCnyLaW/QV5lyUvhL7zwgujZzKsxeedCPipCquekUJe889t7O7lWVlaKHs+afH6K4H3lyhVxkVxz6U2pndPRo0dFfnnzmeK3gp/XvHnzxIiihjwnBHANxasxPTw8RHtezRIT3/fx8SFNx6+iefDmvTAPHjwohvXUxM+N936ueX689ykPHpp4fr6+vpSUlCRKdIqNl1551azivpTOh+NNGvcO7eNtxy4uLuI+/874j0nNc+LV07yNTlPPifdq5u2INfELYf5/R6rnpFCXvPNbfhHJLzgV+P8/fv68rVyTgzcfDvfnn3+KIY01Se2cpkyZQomJiSq/FbwGiF9c7t+/v2HPqQE64UEj2bp1q+hhGhoaKnotBgUFMSsrK5aenq7xn/mrr77KLC0t2V9//cVu3ryp3IqLi5VpXnnlFebs7MwOHjzI4uLimI+Pj9ikomYvdCmeD+/tK5PJ2Mcff8wuXrzINm/ezExMTNiPP/6oTLN06VLxN7dz506WmJjIxo4dy1xdXVlJSQnTRAEBAaLnb3h4OEtNTWW//vors7W1Ze+++64kzomPcjh16pTY+M/zZ599Ju4remTXJe8jRoxg7u7uLDo6mh07dkyMmvD399fIcyovL2djxoxhrVu3ZqdPn1b5rSgrK5PkOdXm3l7oDXVOCOAabvXq1SIoGBgYiGFlJ06cYFLA/6hr2zZu3KhMw390XnvtNWZtbS0Cx/jx48V/XKkGcCmez++//866desmLhQ7d+7MNmzYoPI4H7q0cOFCZm9vL9L4+vqy8+fPM02Vn58vvhP+f8bIyIi1a9eOffDBByrBQJPP6dChQ7X+v+EXJnXNe3Z2tggEZmZmzMLCggUGBoqAo4nnxC+yHvRbwZ8nxXOqawBviHPCcqIAAAAShDZwAAAACUIABwAAkCAEcAAAAAlCAAcAAJAgBHAAAAAJQgAHAACQIARwAAAACUIABwAAkCAEcAB4oFu3btGrr74qFmYwNDQUc3EPHz6cjh8/Lh7nqy3t2LEDnyCAGmA5UQB4oGeffVassrZp0yZq166dWPmKL6aRnZ2NTw1AzVACB4Ba8dWS+NKIy5Yto8GDB4tVyry8vGjBggU0ZswY5dKI48ePFyVxxT63c+dOsfQqXzudB/7FixeL5RIVeHq+5vMzzzxDxsbGIs3PP/+sfJxfNPDV7PiSp/w1+HsrlgQFgGoI4ABQKzMzM7HxKvKysrL7Ho+NjRW3GzdupJs3byr3edCfOnUqzZkzh5KTk2n9+vUUGhpKH3/8scrzFy5cKEr4CQkJYllWvoZ3SkqKeIyvs75r1y4KCwsTS55u3rxZ5QIBAIiwmAkAPNAvv/xCM2fOpJKSElGiHjRokAi0PXr0UJak+Zrv48aNUz5nyJAhYv10XlJX+PHHH+ndd9+lGzduKJ/3yiuviFK4Qt++fcV7fP311/TGG2/Q2bNnxfrQPC0A3A8lcAB4IF5C5kGXl4ZHjBhBf/31lwiyvET9ILxE/dFHHylL8HzjFwG8lF5cXKxM5+Pjo/I8vq8ogU+bNo1Onz5NnTp1EsH8jz/+wLcEcA8EcAB4KN4GPXToUFHlHRkZKYJrcHDwA9MXFhaKNm8egBVbUlISXbx4UbxWXfCLhNTUVFqyZIko/U+aNImee+45fFMANSCAA0C9dO3alYqKisR9fX19qqqqui/48nbr9u3b37fp6t79yTlx4oTK8/h+ly5dlPsWFhb0/PPP0zfffEPbtm0T1fk5OTn4tgDuwDAyAKgVHyo2ceJEmj59umjzNjc3p7i4OFq+fDmNHTtWpOEdy/iwsv79+4tx4tbW1rRo0SIaNWqUGDvOS808aPNq9TNnztB///tf5etv376dPD09acCAAaKTWkxMDH333Xfisc8++0z0QHd3dxfP52n5GHQrKyt8WwAKDACgFqWlpWz+/Pmsd+/ezNLSkpmYmLBOnTqxDz/8kBUXF4s0u3btYu3bt2cymYy5uLgon7tv3z7Wr18/ZmxszCwsLJiXlxfbsGGD8nH+07NmzRo2dOhQZmhoyNq2bcu2bdumfJyn7dWrFzM1NRXP9/X1ZfHx8fieAGpAL3QAaHK19V4HgPpBGzgAAIAEIYADAABIEDqxAUCTq24GB4DHgRI4AACABCGAAwAASBACOAAAgAQhgAMAAEgQAjgAAIAEIYADAABIEAI4AACABCGAAwAASBACOAAAAEnP/wM4bWd0kwSQ/gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.plot(range(len(lrs)), lrs)\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Steps\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f85b01-859b-4454-a3a3-c7ef593735a6",
   "metadata": {},
   "source": [
    "- And a quick look at the loss curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "445d8155-6eae-4b50-a381-d0820ebc27cc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAUdRJREFUeJztnQdYU+cXxl82iIDgABERNyruVdHWtlpHrbXaam2tdfw73Nphrd1Dq61WrR222tYurVXr3nsP3HsjCCLiYk8h/+d8MSEgWlQgueH9Pc8lyb03yXdzQ957zneGjU6n04EQQgghFoetuQdACCGEkLyhSBNCCCEWCkWaEEIIsVAo0oQQQoiFQpEmhBBCLBSKNCGEEGKhUKQJIYQQC4UiTQghhFgoFGlCCCHEQqFIE2LBhIWFwcbGBgcPHjT3UAghZoAiTUghIyJ7t+WTTz7hOSCE5Il93qsJIQXFpUuXjPf/+ecffPTRRzh16pRxXcmSJflhE0LyhJY0IYWMj4+PcfHw8FDWs+FxuXLlMGnSJPj5+cHJyQkNGjTAqlWr7vhamZmZ6N+/PwIDA3HhwgW1bvHixWjUqBGcnZ1RpUoVfPrpp7h586bxOfJ+P//8M7p27YoSJUqgevXqWLJkiXH7jRs30KtXL5QtWxYuLi5q+8yZM+84hvnz56Nu3bpq39KlS6Nt27ZISkoybpf3qlWrlhqPjPOHH37I8fyIiAj06NEDpUqVgpeXF7p06aLc+gb69u2LZ555BhMnTkT58uXVewwePBgZGRn38ekTonGkCxYhpGiYOXOmzsPDw/h40qRJOnd3d93ff/+tO3nypO6dd97ROTg46E6fPq22nz9/XrrU6Q4cOKBLTU3Vde3aVdewYUNdTEyM2r5lyxb1/N9++0137tw53Zo1a3QBAQG6Tz75xPge8nw/Pz/d7NmzdWfOnNENGzZMV7JkSd21a9fU9sGDB+saNGig27Nnj3q/tWvX6pYsWZLn+KOionT29vZq3LLv4cOHdd9//70uISFBbf/rr7905cuX1/3777+60NBQdevl5aXGJ6Snp+tq1aql69+/v3ru8ePHdS+++KKuZs2aurS0NLVPnz591DENGDBAd+LECd3SpUt1JUqU0E2fPr3QzgshlgpFmhAzirSvr69u7NixOfZp2rSpbtCgQTlEeuvWrbo2bdroWrVqpYuNjTXuK+u++OKLHM//888/lVAakOd/8MEHxseJiYlq3cqVK9Xjzp076/r165ev8e/bt089NywsLM/tVatWVRcDpnz++ee6Fi1aGMcmgpyVlWXcLuLs4uKiW716tVGkK1WqpLt586Zxn+7du+uef/75fI2REGuCc9KEmIn4+HhERUWhZcuWOdbL40OHDuVY98ILLyiX+IYNG5Sb2YDst337dowdOzaHSzw1NRXJycnKvS3Uq1fPuN3V1RXu7u6IiYlRjwcOHIhnn30W+/fvR7t27ZSrOTg4OM8x169fH23atFHu7vbt26v9n3vuOXh6eiqX97lz5/C///0Pr776qvE54noXN79hvGfPnoWbm1uO15XxynMN1KlTB3Z2dsbH4vY+cuRIvj9bQqwFijQhGuDJJ5/EX3/9hZ07d+Lxxx83rk9MTFRz0N26dbvtOTInbMDBwSHHNpmnzsrKUvc7duyI8PBwrFixAmvXrlUiLHPAMiecGxFO2WfHjh1Ys2YNvv32W7z//vvYvXu38YJgxowZaN68+W3PM4y3cePGmDVr1m2vLXPi+RkvIcUJijQhZkKsWV9fX2UJt27d2rheHjdr1izHvmLtBgUF4emnn8by5cuN+0vAmESKV6tW7YHGIgLZp08ftTz88MMYOXJkniJtEEyx9mWRSPVKlSph4cKFePPNN9XxhIaGqkC0vJDxSoS7BMzJ8RNC7g5FmhAzImL48ccfo2rVqiqyW6KqpXBJXpbm0KFDlSv7qaeewsqVK9GqVSslkvLY399fuZ1tbW2VS/no0aMYM2ZMvsYgryHWrbiY09LSsGzZMhWdnRdiMa9fv165uUVo5fGVK1eM+4tVP2zYMOXe7tChg3q9vXv3qghyEXER7wkTJqiI7s8++0y58MWKX7BgAd555x31mBCSDUWaEDMighYXF4e33npLzRHXrl1bpUdJGlRejBgxQrl9xf0tqVoyLyyiKoL35ZdfKjexpD298sor+R6Do6MjRo8erdKgZL5bLOk5c+bkua9Yv1u2bMGUKVPUnLpY0V9//bVymQvyvuL2FiGWCxCZ/5b5axm3INvk+aNGjVIu+oSEBFSoUEG52GlZE3I7NhI9lsd6QgghhJgZFjMhhBBCLBSKNCGEEGKhUKQJIYQQC4UiTQghhFgoFGlCCCHEQqFIE0IIIRYKRfoufP/99wgICFDlFaXMYUhICMyJ5Jd27txZVXWSqk+LFi3KsV2y6aQwhdQ5lnxXaSF45syZHPtcv35dFZSQnFRpFSh1lqVUoymHDx9WubJy3BUrVsRXX31121jmzZun8nFlH8mDlZKSD8K4cePQtGlTVdNZimRI/WjTnsuG+s5SrlJaF0oPZqk3ffny5Rz7SPvGTp06qXxceR3J1TVt2yhs2rRJVb6S1pBSqeu3334r1HM/bdo0VTtbPnNZWrRooYqRaP248mL8+PHqu2nIi9by8X3yySfqWEwX+c5r/bgMXLx4ES+99JIav/xeyP+xFJ7R+u9JQEDAbedNFjlXmjxv5u7wYanMmTNH5+joqPv11191x44d07366qu6UqVK6S5fvmy2Ma1YsUL3/vvv6xYsWKA6ES1cuDDH9vHjx6sOS4sWLdIdOnRI9/TTT+sqV66sS0lJMe7ToUMHXf369XW7du1SnZWqVaume+GFF4zb4+LidN7e3rpevXrpjh49qlooSoein376ybjP9u3bdXZ2drqvvvpKtRqUDkvSXvHIkSP3fWzt27dXHaLkPQ8ePKh78skndf7+/qpjkwFpXVixYkXd+vXrdXv37tU99NBDuuDgYON26ZoUFBSka9u2rWrtKJ9XmTJldKNHjzbuI+0Tpe3hm2++qcb+7bffqmNZtWpVoZ17afu4fPly1X7y1KlTuvfee099XnKsWj6u3ISEhKg2mfXq1dMNHz7cuF6rx/fxxx/r6tSpo7t06ZJxuXLliuaPS7h+/brqNNa3b1/d7t271TikC9nZs2c1/3sSExOT45xJ61X5vdy4caMmzxtF+g40a9ZM9dk1kJmZqdoKjhs3TmcJ5BZpaf3n4+OjmzBhgnGdtDR0cnJS/xiCfJnkedI32IC0K7SxsdFdvHhRPf7hhx90np6ext6+wqhRo1R7QQM9evTQderUKcd4mjdvrnv99dcL7PjkH03GunnzZuOxyD/uvHnzjPtIr2HZZ+fOneqx/DPZ2trqoqOjjftMmzZN9SY2HI/0a5YfXlOkBaJcJBTluZfP+Oeff7aa45J+0tWrV1c/iK1btzaKtJaPT0RaBCgvtHxchv9paXt6J6zp92T48OGqhaockxbPG93deZCeno59+/Yp944BqYksj6ULkSVy/vx5REdH5xiz1E8WF4thzHIrLqkmTZoY95H95dikBrNhn0ceeUSVijQgpSfF9Sz1lw37mL6PYZ+C/GykVKbg5eWlbuV8ZGRk5HhfcY9JzWrT4xNXmbe3d45xSfnKY8eO5WvshX3upfa2lNyUto7i9raW4xL3obgHc49B68cn7l2ZXqpSpYpy64ob1BqOS0rPyu9A9+7dlTu3YcOGqnuZtf2epKenq+5x/fv3Vy5vLZ43inQeXL16Vf2Ymp4kQR7LF9cSMYzrbmOWW/mHNMXe3l4Joek+eb2G6XvcaZ+C+mykNrXMaUqXJen8ZHhP+UeXH4W7Hd/9jl3+AVNSUgrt3EsvZJn/kvmrAQMGqK5RUqdb68clyEWH9KKWuILcaPn4RJBknlFqpEtcgQiXzK1KvXEtH5cgncrkmKRG/OrVq1WXNakj//vvv1vV78miRYsQGxuLvn37Gt9La+eNDTaIxSFWmXRx2rZtm7mHUmDUrFlTdbcSD8H8+fNVS8jNmzdD60RERGD48OGqx7Rp/2prwNA0RJDAPxFtaSgyd+5cFUilZeRCWCzgL774Qj0WS1r+53788Uf13bQWfvnlF3UexRuiVWhJ50GZMmVUk/rcEX/y2MfHB5aIYVx3G7PcSqclUyRiUSI0TffJ6zVM3+NO+xTEZzNkyBDV1Wnjxo052hbKa4sLSa6K73Z89zt2iU6VH97COvdy9S4RoNISUizO+vXr45tvvtH8cYlLT75TEuUqVpQscvExdepUdV8sBy0fnylifdWoUQNnz57V/HmTiG3x5Jgi7UYN7nxr+D0JDw/HunXrcnSE0+J5o0jf4QdVfkylb67plac8lnlES6Ry5crq5JuOWVwvMjdkGLPcypdTflgNbNiwQR2bWAmGfSTVS+ZtDIiVJJagp6encR/T9zHs8yCfjcTCiUCLG1jGJMdjipwPacNo+r4yryU/KqbHJ25l0x8OGZf84xh+kP5r7EV17uU1pdey1o9LWkzK2MRLYFjEQpP5W8N9LR+fKZJadO7cOSVwWj9vMpWUO8Xx9OnTylNgDb8ngvRmF3e8xEoY0OR5u49guWKBhM9LJONvv/2mohhfe+01FT5vGvFX1EgEraQEyCKnbtKkSep+eHi4MWVCxrh48WLd4cOHdV26dMkzZaJhw4Yq7WLbtm0qItc0ZUKiHyVlonfv3iplQj4HSTXInTJhb2+vmzhxooqMlCjYB03BGjhwoEr32LRpU470ieTkZOM+kjohaVkbNmxQqRMtWrRQS+7UiXbt2qk0LkmHKFu2bJ6pEyNHjlRj//777/NMnSjIc//uu++qKPXz58+r8yKPJQJ2zZo1mj6uO2Ea3a3l43vrrbfU91HOm3znJSVHUnEk80DLx2VIl5P/4bFjx+rOnDmjmzVrlhrHX3/9ZdxHy78nmZmZ6txIJHlutHbeKNJ3QXLf5GRKrpuE00suoDmRPD8R59xLnz591HZJMfjwww/VP4V8Odq0aaPyck25du2a+icqWbKkSino16+fEn9TJCdS0jPkNSpUqKD+WXMzd+5cXY0aNdRnI6kIkgf8IOR1XLJI7rQB+XEYNGiQSumQf5CuXbsqITclLCxM17FjR5WLKT+o8kObkZFx2+fYoEEDNfYqVarkeI/COPf9+/dXOanyWvLPLufFINBaPq78irRWj09SasqXL69eS/4P5LFpHrFWj8vA0qVLlRjJ/3lgYKBu+vTpObZr+fdk9erV6vcj93i1eN5s5M+92d6EEEIIKQo4J00IIYRYKBRpQgghxEKhSBNCCCEWCkWaEEIIsVAo0oQQQoiFQpEmhBBCLBSK9F2QilDS+F1urRFrPj4emzbhedMmPG+FB/Ok74KUwZP2bNIUQUrCWRvWfHw8Nm3C86ZNeN4KD1rShBBCiIVCkSaEEEIsFKvvJy2t0w4cOKBa5tna3ts1iTR3Fy5evKjcOdaGNR8fj02b8Lxpk+Jw3iIiIpCcnKx6b0sb1qLC6uek9+zZg2bNmpl7GIQQQqyAkJAQNG3atMjez+otabGgDR+s9IElhBBC7pVLly4pg8+gKUWF1Yu0wcUtAu3n52fu4RBCCNEwtvc4bfrA71ek70YIIYSQfEORJoQQQiwUijQhhBBioVj9nDQhxLrJyspCenq6uYdBrABHR8cin3P+LyjS+eRqYhoWH4xC/5YBsLGxKdyzQgjJFyLO58+fV0JNyIMiAl25cmUl1pYCRTofpGZkot3kLYhLSkENLzs8XNu/8M8MIeSuSIkHSYuxs7NDxYoVLc4CItoiKysLUVFR6jvl7+9vMcYYRTofODvYYUTVaASf/AJHl7WArtYvFnMCCSmuSDVBqQDl6+uLEiVKmHs4xAooW7asEmr5bjk4OMAS4KVnPnk6yAvVbKPQMWkxQg4cLNyzQgj5TzIzM9WtJbkmibZxvPVdMny3LAGKdD4pVfdJhLo1gZPNTaSt/kS52ggh5odeLWLN3yWKdH6xsUGpLuPV3UfSNuHg7o2FeFoIIYQQivQ94VWtKQ57dVD3HdZ/DB0jSgkhFkBAQACmTJmS7/03bdqkrMbY2NhCHddvv/2GUqVKFep7WDu0pO8R325jkKZzQFDGYRzfPL9wzgohxCoRYbzb8sknn9x3t7/XXnst3/sHBwerKGYPD4/7ej9SdDC6+x4p41cd2316oOXlWXDf9hl0j3SFjZ1lRAESQiwbEUYD//zzDz766COcOnXKuK5kyZLG+xL3IgFM+eldLFHJ9xog5ePjc0/PIeaBlvR9UOPZj3FD54aKmRE4t+bHgj8rhBCrRITRsIgVK9az4fHJkyfh5uaGlStXonHjxnBycsK2bdtw7tw5dOnSRbVIFBGXXsbr1q27q7tbXvfnn39G165dVXpa9erVsWTJkju6uw1u6dWrV6NWrVrqfTp06JDjokLSkoYNG6b2K126NEaNGoU+ffrgmWeeuafPYNq0aahataq6UKhZsyb+/PPPHBcm4k2QPGU5fl9fX/WeBn744Qd1LM7OzurzeO6552DtUKTvg7LlvLGrYn91v8yer6FLSyjo80IIuUfkBz45/aZZloLM9nj33Xcxfvx4nDhxAvXq1UNiYiKefPJJrF+/HgcOHFDi2blzZ1y4cOGur/Ppp5+iR48eOHz4sHp+r169cP369TvuLznnEydOVKK5ZcsW9fpvv/22cfuXX36JWbNmYebMmdi+fTvi4+OxaNGiezq2hQsXYvjw4Xjrrbdw9OhRvP766+jXrx82btQH4v7777+YPHkyfvrpJ5w5c0a9ft26ddW2vXv3KsH+7LPPlPdh1apVeOSRR2Dt0N19nzR69m2ET5mDSlmXcWHZV/B/9vOCPTOEkHsiJSMTtT9abZZP7fhn7VHCsWB+TkWEnnjiCeNjLy8v1K9f3/j4888/V2InlvGQIUPu+Dp9+/bFCy+8oO5/8cUXmDp1KkJCQpTI50VGRgZ+/PFHZeUK8toyFgPffvstRo8eraxz4bvvvsOKFSvu6djkIkDGNWjQIPX4zTffxK5du9T6xx57TF0YiFehbdu2qpiIv78/mjVrpvaVba6urnjqqaeUx6FSpUpo2LAhrB1a0veJt6c7dlcZqu6XO/oTdPHZbiFCCLlfmjRpkuOxWNJi0YobWlzN4ooWK/u/LGmxwg2IuLm7uyMmJuaO+4tb3CDQQvny5Y37x8XF4fLly0bBFKQcq7jl7wUZd8uWLXOsk8eyXujevTtSUlJQpUoVvPrqq+piRNzsgly4iDDLtt69eyurXqx/a4eW9APwSJdXcGDy7wjUhePkvk2o9Zj+qpUQUvS4ONgpi9Zc711QiKCaIgK9du1aZW1Wq1YNLi4uai72vzp/5S5rKXPQd2tEktf+RV20SWqwiytb5tzlmAcNGoQJEyZg8+bNynrev3+/mk9fs2aNCrqT+WuJbLfmNC+zWtIy7yFzKxIcIF+I3PMb8gWREyFXdPLFFBeIzFNYCj6lXLC99qd4NG0SPj4dwCpkhJgR+Q0Rl7M5lsKsVCXzv+IiFjezzM+KOzgsLAxFiQS5SaCWCKIBiTwX0bwXxBsgx2OKPK5du7bxsfzWiy6Ie37Tpk3YuXMnjhw5orZJpLvowFdffaXm2uVz2LBhA6wZs1rSSUlJaq6lf//+6Nat223b5UTIifr9999V+7APP/wQ7du3x/Hjx1V0nyXwXIc2mHp4Iy6fv46dodcQXLWMuYdECLEiJJp5wYIFSrjkYkB+B83RmnPo0KEYN26csuYDAwPVHPWNGzfu6QJl5MiRKphN5pJFbJcuXaqOzRCtLlHmIv7NmzdX7ve//vpLiba4uZctW4bQ0FAVLObp6anmw+VzkAhxa8aslnTHjh0xZswYYyBCbitaUgo++OADlX4g8yt//PGH6lByrxGFhYmPhzNeaFZR3V+5YiEQuc/cQyKEWBGTJk1SoiQFSESoxVBp1KhRkY9DUq4kEO3ll19GixYt1Ny4jOVeDCZJ1/rmm2+U675OnToqiluixR999FG1XdzWM2bMUPPU8pu/bt06JeSS8iXbRNAff/xxZZFLkNvff/+tXseasdFZSKcIuRqTIAFDzp1cMUkQg6QcNGjQwLhf69at1WM50fkhMjJSzXNERETAz8+vUMZ+KS4F30/4EGPsZyDRKwglh2yV7uGF8l6EED2pqak4f/688rJZimetOCFWrIilWMYScW7t36nIItASTQWORUdHq1uZBzFFHhu25UVaWppaDCQkFH4Oc3kPF5Rs0AWxR2Zjb2pFtL2ZAjjmDP4ghBAtEx4ergK2xFCS31hJwRJBe/HFF809NKvG6sw9mTORIAfDYhqQUJj0eaIJHr/5LV653hs7I1KL5D0JIaSosLW1VXPGUvFM3NESzCXuaLGmSTEUaUNdWcnNM0Ue363mrCTbS06fYZEgs6JArOlOTfUBDN+sP10k70kIIUWFuHolElt+V6Xa2I4dO4pFxS9zY7EiLXMCIsZSCs+AfDF2796tghbuhNR7laR9wyK5dUXFwEerwtHOFlfPH8bV33oByXcuwUcIIYRY9Jy0VNI5e/as8bHMbxw8eFCVwZNycCNGjFDR35KCYEjBkpzqey3oXlT4lnJBjyYV8PyBd1FG8hi3+AMdxpl7WIQQQjSKWS1pKZgu+XKG+qtSx1XuSwET4Z133lG5edInVeZBRNSlqLolR3IOeqw6JmXpK49lhcwAroeae0iEEEI0illFWnLjJAMs9yLBCYa0LCnwLtHcEhovQQo1atSAJSPWdIUmnbAlsy5sszKA9dkF6gkhhBCrmJPWMgMfrYYJWb2QpbMBjkmBk73mHhIhhBANQpEuBCqUckG9Jq2wIOth/Yo1H0gJtcJ4K0IIIVYMRbqQGPRYNXyT1QOpOgfgwk7g+OLCeitCSDFDpgolsNZAQECAKqN8N/JqYnQ/FNTr3A3pbmVaabI4Q5EuRGv64SYN8FPmU/oVK98BUmIL6+0IIRpAam936NAhz21bt25VAijdne4V6U4lAbZFIZSXLl1SfRdI0UCRLkQGPVoV03XP4FxWeSDxMrDu48J8O0KIhfO///1P9UmWOtC5kUYTTZo0UY0l7pWyZcuqrlFFgdSvkHoUpGigSBcifp4l0KVJVYzOeEW/Yt9vQNi2wnxLQogF89RTTylBNWSwGJD00nnz5ikRv3btmuo2VaFCBSW80kNauj3djdzu7jNnzqhqYJKuKqWR5cIgr65Wki0j71GlShVVhyIjI0Ntk/F9+umnOHTokLLuZTHNujF1d0t5UOlMJS0lpVuVWPRyPAakF7bUtpDOV+XLl1f7DB482Phe+W3m8dlnn6nGFnKBIBa+pOMaSE9Px5AhQ9TryzFLa0spES1IxpB4BaT2hjxXam0MGzYMWsFiG2xYC8PbVEfr/ZGYffNxvGi/AVg6HBiwHXCw3FxvQjRNetK9P8fOCbC79XOYeRPITANsbAEHl/9+3XtopmNvb69aPYrgvf/++8ZezCLQ0kdZxFkErnHjxkpEpWri8uXL0bt3b9UVsFmzZvkStG7duqlmRFKhUcp4ms5fG5BqjDIOES0R2ldffVWtk/oUzz//PI4ePaqE0NDrWXoh5CYpKUm1q5QqkOJyj4mJwSuvvKIE0/RCZOPGjUpA5VYKWMnri9DKe+aHb775Bl9//bVqbSm1NH799Vc8/fTTOHbsmCp2NXXqVCxZsgRz585VYiydqmQR/v33X0yePBlz5sxRbS0lpVcuPrQCRbqQ8XZ3Rv+WlTF+0wto73AQXtfOwSZ0I1CTczqEFApf+N77c7r/BtS51df+5FJgXl+gUiug3/LsfabUBZKv3f7cT+Lu6a369++PCRMmYPPmzcY+yuLqfvbZZ42Ngd5++23j/lLQafXq1UqA8iPSIqonT55UzxEBFr744ovb5pE/+OCDHJa4vKcImYi0WMXSL1ouKu7WK2H27NmqhsUff/wBV1f9xYp0x5K59y+//NLYxVD6Yct6Ozs7BAYGolOnTqrkc35FeuLEieqipWfPnuqxvLYIvngPvv/+e1y4cEGJdatWrdSFj1jSBmSbHEPbtm3h4OCgRDw/n6OlQHd3EfB666qwLVEKI9Jew4bg3ynQhBRjRKSCg4OVNSiIZSlBY+LqFsSilv7M4uaWEskiliK4Ijb54cSJE6oZhkGghbz6Hfzzzz+qm5UImLyHiHZ+38P0verXr28UaEFeU6z5U6dOGdeJBSsCbUCsarG680N8fDyioqLU65oij+X9DS51KSlds2ZN5cqWlpoGunfvjpSUFOXSl4uChQsX4ubNm9AKtKSLAA8XBwx+tBrGrsjA2X3OaPlYJpwdsr+whJAC5L2o+3N3GwjsrH8NcXebMuIICgoRZLGQxQoUK1pc2dKnWRArW9y7YiWKUIsAirta5l0Lip07d6JXr15q3lnc1WK9ixUtLuXCQCxYU8TaFSEvKBo1aqR6P6xcuVJ5Enr06KEs5/nz56sLFrlgkPUyNz9o0CCjJyP3uCwRWtJFRO8WleDr4YxLcan4Y2cYcO0ccGJpUb09IcUHmSO+18UwHy3IfVlnOh99t9e9D0REpD+zuIvFVSwucMP8tLSD7NKlC1566SVlpYoFePp0/tvfSn9nmY+VVCkDu3btyrGPtJkUl7DMi0tEubiKw8PDcx6uo6Oy6v/rvWR+V+amDcj45djEqi0I3N3dlVdAXtcUeSxBcab7yVz3jBkzlJdA5qKvX9d3IhT3vbjgZe5606ZN6iJF5uG1AEW6iBDL+c12+i/tug3roJsWDCx4HYi9N/cSIUT7iHtZBGX06NFKTMVda0AEUyw+EVJx577++uu4fPlyvl9bLEiJ2u7Tp48SUHGlixibIu8hrm2xns+dO6fES9zApsg8taEz4dWrV5GWlnbbe4k1LtHU8l4SaCbzxOIhkEA3w3x0QTBy5Eg1Dy3iK1bxu+++q8Y1fPhwtX3SpEkqAl7m4uWCRgLxxI1fqlQpFcD2yy+/qPGFhobir7/+UqJtOm9tyVCki5CuDSugprcb9qT6ItIlEKgowQv6q2dCSPFCXN43btxQ7mbT+WOZGxb3rayXwDIRm3tpzytWrAiuzMNKgJREW48dOzbHPhIZ/cYbb6gobImylgsCScEyRQLZpPDKY489ptLG8koDk/QtmS8Xi1U6FT733HNo06aNChIrSIYNG6a6JL711ltqCkCiziWaWy42BIlK/+qrr5RXQMYRFhaGFStWqM9ChFqsa5nDlhx0cXsvXbpUpYJpARudJJFZMVI0QOYkxP0jOXbmZv2Jy/jf73tRxj4Fy97uBJ9SuVxqhJB8IVHFYulJr3lLbl9LrOM7FWkmLaElXcQ8HlgOzQK8cPWmC6asP5O9oQCDKAghhFgHFOkiRoJDRnUMVPfn7o1AaESkvsDJkiFFPRRCCCEWDkXaDDSu5Il2tb2RpQP+WbUR2Pc7cHAWcG6DOYZDCCHEQqFIm4l3OtSErQ3w0zkvxNTqo1+5dMT9lTQkhBBilVCkzUS1cm7o3riiuv/2taehc/cDYsOBTfqi8IQQQghF2oyMeKI6nOxtseVCKg43uJX+sPN7IOoAv5mE5BMrT1Ahxfy7xLKgZqS8hwv6tgzAT5tD8c4hX6ys8yxsj/0LLBkKvLoRsLP8knWEmAsp6SiBmFeuXFF5vIaKXYTcr0DLd0m+R5ZULpQibWYGta6Gv3dfwKnLCVjefBg6n1sPRB/RW9Stbm8vRwjRIw0bJF9V8leleAUhD4oItHynTJuBmBuKtJnxKOGAwY9Vw7iVJzFu8zV0eGIMHJYN0c9N1+oMlK5q7iESYtHlNaXqVEZGhrmHQqwABwcHixJogSJtAfQJDsBvO8IQFZeK35OD8Url1sD5zcCyEcDLS+TyztxDJMRikR9VS/thJaRYBI5JBxapJysl2qQgurRzkz6rlji5/6DNN95oW0Pd/27TOSQ8MRGwdwbOb9HnTxNCCCmWWLRIS9eTadOmqWLt0g1GHksR9W+//RbWRrdGFVCtXEnEJmdg2uEs4NHR+g1rPgCSrpp7eIQQQsyARYu0dGaRvqqdOnVSbdOkw0q7du0QEhICa8PezhbvtNe3svx1+3lcrvM/wDsIKFebBU4IIaSYYtEiHRwcjPXr1xsbnktv1G3btqFjx453fI70PI2PjzcuCQkJ0ApP1PZWJUNTM7IwZWMY0HsR0Hc54KmNvqeEEEKKkUhLY++ePXsiMDBQRd01bNgQI0aMUI3G78S4cePg4eFhXGrXrg0thf+/a9J841yKS86gMSubiyeEEKJhkZ47dy5mzZqF2bNnY//+/fj9998xceJEdXsnRo8ejbi4OONy/PhxaImmAV5oW6scMrN0mLj6lH5lWgKw4h2WDCWEkGKGRadgjRw50mhNC3Xr1kV4eLiylvv0udWUIhdOTk5qMSAub60xsn0g1p+Mwcqj0dh/4QYaJe8EQn4CbB2ARn0AjwrmHiIhhJDibkknJyfD1jbnECUfMisrC9ZMTR83PNvIT90fv+IkdDU6AM1eB3rNpUATQkgxwqIt6c6dO2Ps2LHw9/dHnTp1cODAAUyaNAn9+/eHtfNWuxpYeigKIWHXsfZEDNo9+ZW5h0QIIaSIsWhLWvKhJe1q0KBBqFWrFt5++228/vrrqqBJcWi+8b9WldX98atOIiPTxHsQdxFIvGK+wRFCCCkSLFqk3dzcMGXKFDUPnZKSgnPnzmHMmDFwdHREcWDAo1Xh5eqI0CtJ+GdPhH7lsYXA982BVaPMPTxCCCHFWaSLO+7ODhj2eDV1f8q600hMuwl4VgYykoCj/wJn1pl7iIQQQgoRirSF82LzSggoXQJXE9MxffM5wLcB8NAg/cblbwLpyeYeIiGEkEKCIm3hONrbYlQHfYGTGVvP43J8qr6ut0dFIDYc2Dze3EMkhBBSSFCkNUCHIB9VLjQlIxOT1pwGnEoCT07Ub9zxHRB9xNxDJIQQUghQpDVSLvS9J/XW9Lx9ETgVnQDU7ADUehrQZQJLRwBZmeYeJiGEkAKGIq0RGlfyQscgH2TpgPErT+hXdvwKcHIHLu4F9v5q7iESQggpYCjSGuKdDoGwt7XBxlNXsOPsVcC9PNDmI/3GdZ8C8VGwaiRIbvdPwNSGwMQawNyXgd3TgcvHACuvQkcIKZ5QpDVE5TKu6NXcX90fu+IEssSsbtIfqNAESE8AVr4DqyT5OrDpS2BKkP4Yr4cCiZeB44uBlSOBacHA4lsR74ZuYXT/E0KsAIq0xhjWpjrcnOxxLCoeiw9dBGztgM7fADZ2wImlwMkVsBriIoFV7wGTg4BNXwDJ1wDPAKDTJKDfSuDxD4AqjwEOJQDfRtnPu3YW+LKy3tJme09CiIax6Nrd5HZKl3RSlcgmrD6FiatPo2NQeTj7BAHBQ4Dt3wArRgJVHwccnLX78UnJ03UfA4f/AbJu6tf51AVajgBqPwPY3fraVgoGHhkJZGboFwPhO4C0OCDpas5+3MveBEp6A2VrAGVqAqWrAvbZHdMIIcTSoEhrEKnp/deucFyMTcFvO8IwoHVVoPW7QOQ+IHiotgVacHABTi7XC3TAw0CrEUDVNjkF1xQ7B/1ioEEvoHw94GZaTpe5Cq7TZa+zsdVb5mVqZC9lawJlqgMunoV4gIQQkj9sdDrr9gdGRkaiYsWKiIiIgJ+fvv2jNTB/XyTenncIbs722DLyMXi65qpnnp6k7z9tb+F1zuXrd2693lX/1JRsIZayp6UqAX5NCuZ9UmKBQ3OA6MPAlVPA1dNA2l16jbuW04u2dB/zrqNfF3tBb+VLwJ67b/b4hTtdQOQHsfjjLwLOHvqLBgNpCYCT2/2/LiFE81pCS1qjdG1YAT9vDcXJ6AR8u+EsPupcO+cOITP07u9H3wWav170AxTxykjRXyxIrXGJzDbcl/U1O+r3S40F/nlZvz6wM1C9rX590LMFOx6XUsBDA3KOT4LPRKyVaJ8Brt66FcFMitEv9iZeCbHEt00Gmg8EOt6q9CYR9ZPr6OfFHUvcunXNfmzneOvYE/THL0taIvDqBqBcYPa5kspxEgT41GT9OvmMvg7UXyAEPgUEdtK75wkhxQqKtEaxs5UCJ7Xw8q8h+HNXGPoEV0Kl0q7ZO5zbAKRczznnKtZkyg3AS98Cs0AI3QRsnaS3BnOIsdQUv4uT5sOrehe1uJVbDNY/xzvXhUZhIpavm49+qfxIzm1iwSrRPq235g04lgQ8/IGSZbPXGY5TXXwk5f/95XgNuJYB3Mrrhd1A5F4gPRGI2K1f1n4IlKutF2tZyjd4MOudEKIJ6O7WOL1/2Y2tZ66iU73y+P5FkwjnzJvA+U369CyxIgXJMZYUJv8WQL3ngTrP3Nvc69n1wOnVQO2ngYBW2RcDf3a9+/NyWJeu+uXlxfpbrSOpXhJ1brgwMVjL6n4ykJmuP04p5Soibzh+9wr/HbQmfcNPrQBOLgPCtmUH0QnufnqxrvUU4B+cHUxHCLEqd/d9ibQMUkpVGgYaEhKC2bNno3bt2njttddgSVjrnLSBE5fi8eTUrcp7u3BQMBr630V0V40Gdv8I6G4V/hBXbI0OQP2eQLUnsuev5cUkjenCTqBez+z1S4cD+37TB6e1G6NfJ67bo/P1FqdBgEzF2N4FsGWm3wMjHpDTa/SCfXbdLQv+FnKhJedRIt+lXCwhpHiL9MMPP6zEuHfv3oiOjkbNmjVRp04dnDlzBkOHDsVHH92qgmUBWLtICxJAJoFkzQK88M/rD6kLqDsSfwk4Mk+f3nT5aPZ6Fy+9VSZR0CLOYh0Kr6zPDt46s1a/BD4JVHm0kI+K3BGZr5ZphhPL9Ja2TGsIck7EQyHcTAfm99OnnLX/Ijvi/0a43j0vgXEyZ04Isb7AsaNHj6JZs2bq/ty5cxEUFITt27djzZo1GDBggEWJdHHgrXY1sOxwFELCrmPt8ctoV8fnzjtLZHLLYfpFumdJxPOR+UBiNLD/j+z9JGBKXOWm+cfVn9AvxPwpahJ4J4tMa8ictVjYpgVdkq7o19naZ3dME2RuWyq1CeJ+L1lOL9hyK7EKpavrU9Dk1rV00R8bIeTBRTojIwNOTvr5tHXr1uHpp59W9wMDA3Hp0qX7eUnyAJT3cFG5099vPIfxq07iscBycLDLh4tZCoTI8sRnestMrGQJpJIiIeXrs9CHFpC56ICW+sUUmWoQcVapeKbfBRv9BdjNVH1g2nVZQvN+bXGji1jX6wE0e1W/ThxvMs/OIjCEWK5Ii2v7xx9/RKdOnbB27Vp8/vnnan1UVBRKl+bVtzl4vXVV/B0SgdArSZizJwK9HzKJSv4vpLRotTb6hVgHEixoEFZTevyuF1qJYBdrW9LQEmP0t9fO6SPaJR4hLkI/Dx4Zkh0kaEg5kxrqXlWBwSHZFwCSQ16itHUEAxKidZH+8ssv0bVrV0yYMAF9+vRB/fr11folS5YY3eCkaHF3dsDwNtXx8ZJj+GbdaZVHXdKJEb8kDyRmwdldv9wp91oi06+LaJ/Ru78NiIBL4KEsphb6/P8BkXv0LnNJFfMO0qfUlaujXycXgoSQokvByszMRHx8PDw9s6OJw8LCUKJECZQrVw6WQnEIHDOQkZmFdpO34PzVJPh5uuDJuuXRvo4PGlYsBVtb5tSSAkAVgYkBkq9mV2KTdd821ot6XkiEvxRukf1FtL1vLZIfTohGiNRSdHdKSgrkaSLIQnh4OBYuXIhatWqhffv2sCSKk0gL0mf61T/2Iik907iunJuTEusOQT5oVtkrf/PVhNwrUjI15hhw+bi+x7fcjzmhn//Oi24/A/W66+9H7AFCpgO+DfTFbQzIeskxL1FGP0fOfHBiJjQV3d2lSxd069ZNRXLHxsaiefPmcHBwwNWrVzFp0iQMHDiwwAZ48eJFjBo1CitXrkRycjKqVauGmTNnokmTAqrpbGUEVyuDPR+0xZbTV7DqaDTWn4hBTEIa/twVrpZSJRzQtpY3OtTxQavqZeDsQDckKSCkElvJR3Om50mxFwlME9FWwi0CfhS4EQaUMMnpv3ISODJXPw9uKtJSKEdKqhpwLqWvcW5aGEYtJo+lpKyIvZAQDVzcr09F82uc/ToSUMcc/qJH9Xq/mbOFrDFl1CbnRVjWrXoOsr0YV9e7L5Hev38/Jk/W1xieP38+vL29ceDAAfz7778q/aqgRPrGjRto2bIlHnvsMSXSZcuWVbnYpi52cjslHO3RIai8WtJvZmHHuatYfSwaa45dxrWkdJVTLYurox0eDSynBFsiwrUwhx2bnK6Ow8nBVnkHeJFh4chctMxpyyIV7gyoqHOT71uFRsATnwMeFbLXSXqZmzeQbK8Xb0Otd1nuhmQmGEQ6IgSY2xuo+BDwv9XZ+4h7PuHSLXEvqbfWpZmJY65bWS/FeWSs1dpmv64E0J1epbfwpQKfaVU+8RwYjk21UU3Pvs3KMFl3a72UpZWMCkMe+5YJ+guR9mOzX3fDGH18gCEeQEROl2nyOEt/QWS8zQTqdAUeuvVbnHBZf8Ej52PA1uzXXTwEOLdRH18gYzYudvoe9bnXyW1l6Uz3hv75GanAnBf1x9JrfnY+/tqPgGML9ccn3ehMj/lO5YKl013vBdmPx/vrL9CGHQC8qujXrfsU2DZJ38FOFhF2w32bXPcN22Rqpe8yaJX7+lUWi9bNTd+dR3Kjxaq2tbXFQw89pFzfBYUEqIl7QSxnA5UrF2Dd6WKAo70tHq1ZTi1jntFhb9h1rDoWjdVHoxEVl4rlhy+pRfZ7uFoZvNDMH21qlbt7QZQiRi40Np2KwYL9F7H+5GVkZOr/ycuUdELf4Ero1bzS7V3AiGWTOwrcME9tilhVQ/dlC7aIs9SIl8h0SR8zlGDNfb9sYM73kXx/QzMTA/IagnpuIpCYjzGX8MoWaWnKsuwNfYCcqUhLP/c7zc3fCRE+g0inxgEH/tTXcjcV6dDN+kj7e8G3YfZ9EW2ZfjC9MBKkeFF85L29rkTxGxARlC52glycGERaLqok4r+g0d2yrg0XJvnhbt3uNMB9zUnXq1cPr7zyiorwlkImq1atQosWLbBv3z6VliVVyAoCKTMqc9wyF7B582ZUqFABgwYNwquv5pFacgeK25x0fpHTfuRinHKJyxJ6NbvhQ1AFdwxvUwNtzSjWMr7DkXFYsD8SSw5F4UZydlGVWuXdEZecri4yBBcHO/Ro4of+rSrnbDJCyN2qtklJW/kBF5EW0ZbH6n68yf0EfQlWsU7rv5Cdj37pMLD5S8CjYnZHNGFePyAu8pZLN0ufTy7ld0Uc5Vb1PnfMeV+K0hiKBMm8/v7fASd3oLlJiWWxSmWbWLw2potdzse2JuvE+vQJ0j9frNnwHXpr2LShjKTdycWPwQ1tXDJvLSbrlJV+Eyjln31RIc+TgkhSOli6tRny5+V15QJA1huP1/HW5+GgH59Rem7dymdk2ppVni+IV8GQHSDnRc6HqQdBeRUMoq3L+Vjuy/sVQAc5TQWOiYv7xRdfVBHejz/+uMqVFsaNG4ctW7Yo13RB4Oysvyp788030b17d+zZswfDhw9XOdqS+pUXaWlpajGd0xaxp0jfGfkKnI1JxPz9kfhzZziSbwWd1fEVsa6OJ2p7F5lYR8WmYNHBi8pqljEZKOvmpNLKZBGRlkh28QBM3xKK45f0V8oSwC7Bca8+XOXuNczvgbiUDOwPv4HUjEzljXBx5Bw+IcWRSC2JtCDWslQXkxxpcXUbGm24u7urymMFgaOjowoQ27Fjh3HdsGHDlFjv3Lkzz+d88skn+PTTT29bT5HOH9eT0lWf6t93hBkjxGuXd8fwttXRrpDEOintprLmFxyIxI5z14wX2M635p27NfJDy6qlYZ9HVLp8feU5ItabT18xrm9SyROvPlJFBclJW8/8IK91MTYFe8NuYE/YdewLv4FTlxOM43FztlcXCT2b+qO2r3sBHT0hRAtEak2kTQcuFMagK1WqhCeeeAI///yzcd20adMwZswYZSHnBS3pwhNrsWDFshaxftC866uJaTgUEaus4ZVHo5GSkZ0y1ryyF55t5IeOdX3g5uyQ79c8FZ2AGVtDsfjgReO8deUyrqpk6nON/W4LMsvM0qkuYiLGIsoiztHxt6cLBZQugZtZOkTeSDGua1CxFF5s5o+n6pdXgXqEEOsmUksinZWVpYTy66+/RmKi3iUpgWRvvfUW3n//faNl/aCIS10+kK1bs6MR33jjDezevTuHdX03OCf9YNwQsd4Wit+2Z4t1oI8bRijL2uc/xVq+Xpfj03D0YhyORsXh6MV4dT+3GIqYdmtYAc80rICKXg/WnelyfCp+2xGGWbvCEZ+q78Hs5eqoSqU2CfDE/vBY7A2/jgMXYpGYdjNnJKWtDepU8FCWeNMATzSu5KVc7VlZeov975ALKlJeRFuQiPhnGvqqgLs6vh4PNG5CiOUSqSWRHj16NH755RflVpYUKWHbtm3K1SxBXWPHmkQlPgDi1g4ODlbv06NHD+VOl9efPn06evXqla/XoEgXnFj/su28Ej+DsIlYi2UtLmkRa/kqibV5LCpOBaWJIMv9q4mSdpET8ZqLMLeoUhrPNvZTVdEK2pUu45y7J0KNW9zYeSEi20gEuZInGgd4Kgv5vyzjKwlp+Hd/pBLs8GvZfZ3r+3kose5c3xeuGkhnI4RYqUj7+vqq4C1D9ysDixcvVtHXd3JF3w/Lli1TFwWSHy3pVxJExuhu8+Ypi+jN3J4t1jW93VDGzVGJsgRa5UaM7erl3FCngjuCfD1Q189Duc6LKi/7ZmaWSjv7ddt5XElMQ4OKeiu5SSUv1PRxy/ecdW7Eut4Veg2zb1nXBhe75J93aVhBucODKuita/k3E6tephGuJ6XhWqLcpqu8dbkAMtzXb09XFzEj2tZQbnpCiPnRlEhL1PXhw4dRo0aNHOtPnTqFBg0aqLKhlgIt6cIT619viXWCicvYwc5GCZ+IsbiNg3zdEejjbvVR0dcSDdZ1hKqdbqCilwtSM7KUEBtc5PeCWOafPF0bTvbW/fkRYulEakmkpQyoLFOnTs2xfujQocolLXPGlgJFunCJS87A4kMX4Whnq6zG6t4li7WgyL/TrtDryhUuEevpmTkLLoj3QObHpfhKaVdHdd9wa7puy+mrmLL+tIosr+fngR96NYKf54PN1RNCiolIS2ERKVri7++vipgIkhIlg1+xYgUefvhhWAoUaWIuxG198lI83F0cULqkIzxLON5TGVNJKRs+5wBikzNUzfWpPRvikRplC3XMhBDL0pL7CsNu3bo1Tp8+rSqOSYMNWaQ06LFjx/Dnn38W/CgJ0SBiEUvDE/EwlPdwuec6461rlMWyoa2UJS1C3WdmCKauP6PmwgkhxYMHzpM25dChQ2jUqJGqRGYp0JImWkeqnX269LhyoQuPB5bD5B4N4FEi/znkhJBiZEkTQooOscDHdauLCc/Vg5O9LTacjMFT321V+eaEEOuGIk2IRujepCIWDApWEeMR11PQbdoOzN0bYe5hEUIKEYo0IRpCqpotG/KwcnlLC8935h/G6AWHlUucEGJ93FM1CQkOuxsSQEYIKVxkLvrnl5vg+41nMWndaZWbLYVkJE3rQUuqEkI0LNIeHh7/uf3ll19+0DERQv4DKcM6tE111K9YSqVpSRnWzt9tw5TnG6iWmoQQ66BAo7stEUZ3E2sn8kYyBs/aj0ORcaqcqDQSefOJGihVwtHcQyPEaohkdDch5H6QSmRzB7RAr+b+qkLZHzvD8ejETfhzZ5iqW04I0S4MHCPECpBSrGO71sXsV5qrhidS/OTDxcfw1LfbsPPcNXMPjxByn1CkCbEipMLZ8mGt8FmXOvBwccDJ6AS8MGMXBs3ap9zihBBtQZEmxMqwt7PFyy0CsOntR/Fyi0qqVeiKI9Fo8/VmTFpzCsnp2V3LCCGWDUWaECtFump91iUIK4Y/jBZVSiPtZhambjirxHrJoSjVsYsQYtlQpAmxcqSf9+xXm2Nar0aoUMoFl+JSMezvA+jx006WFiXEwqFIE1IMsLGxQce65bH+rdYqPcvZwRZ7wm6o3GqpWHYtMc3cQySE5AFFmpBi1qxjWJvq2PDWo3i6vq9K2ZKKZZKy9dWqkwi7mmTuIRJCTKBIE1IM8S3lgqkvNMS8AS1Qx9cdCak38cOmc0qse07fiYUHIlkPnBALgBXHCCnmZGbpsOZYNObsicCWM1eUdS24OdvjmQYV8HzTigiqcPeSwIRYO5Fmqjh2T7W7CSHWh52tfr5alouxKZi/N1K1wJT7f+4KV0vt8u5KrEW0pcEHIaRooCVNCLmNrCwddpy7hjl7LmDNsctIv1Ve1NHeFh2DfPB8k4p4qEpp1eiDkOJAJC1pQoilIOLbqnoZtdxISseigxfxz54IVcFs8cEotfh7lUCPJn5oU8sb1cuVVEVUCCEFCy1pQki+kOInhyPj8M/eCCw9GIWEtOzKZS4OdqhbwQP1K3qgnl8pNKhYCn6eLir1ixBrIJKW9H8zfvx4jB49GsOHD8eUKVOK4LQQQgyI4Er/alk+7FQbK45cUhb2gQuxSEy7iZCw62ox4OXqiPp+ItylUN+vFOr5eaB0SSd+oIRYY+DYnj178NNPP6FevXrmHgohxR4XRzs829hPLTJ/HXo1EQcj4nA4MhaHImJx/FI8rielY+OpK2oxUNHLRW9p+5XCU/XLo7yHS7H/LAnRvEgnJiaiV69emDFjBsaMGWPu4RBCcs1fVyvnppbnGutTU9JuZuLEpQQl2IduCfe5K0mIuJ6iluWHL+GnLeewcFBLVPQqwc+TEC2L9ODBg9GpUye0bduWIk2IRvpby7y0LAbiUzNwJDIOByNisWB/pBLtV37fi/kDW8DNmWldhGhSpOfMmYP9+/crd3d+SEtLU4uBhISEQhwdISS/uDs7oGW1Mmrp1qgCuny3HacuJ2DI7AP4pU8TRocTkgcWnTMhlV0kSGzWrFlwdnbO13PGjRsHDw8P41K7du1CHych5N6Quehf+jRVjT42n76CMctP8CMkRGspWIsWLULXrl1hZ2dnXJeZmamiTG1tbZXFbLotL0v64sWLSqiLupQbIeS/WXX0Egb8tV/d/6xLHbzcIoAfG7FIIpmCdTtt2rTBkSNHcqzr168fAgMDMWrUqNsEWnByclKLgfj4+EI5YYSQB6dDUHm806Emvlp1Cp8sOaYKpDxasxw/WkK0MCft5uaGoKCgHOtcXV1RunTp29YTQrTJwNZVcf5KEubti1Tz0/8ODEZNHzdzD4sQi8Ci56QJIdaPTF+N7VoXzSt7qaIo/X/bgysJ2VNWhBRnNCfSmzZtYrUxQqwMadzx40uNEVC6hOq+9dqfe9nPmhAtijQhxDrxdHXEL32bwt3ZXpUafWf+YVUvnJDiDEWaEGIxVC1bEj/2bgx7WxssORSFKevOmHtIhJgVijQhxKIIrloGY7vqA0O/WX8Giw9eNPeQCDEbFGlCiMXxfFN/vP5IFXV/5LzD2Bee3V2LkOIERZoQYpG80yEQT9T2RnpmFl77Yx8iriebe0iEFDkUaUKIRWJna4NvejZAHV93XEtKV6lZ0qSDkOIERZoQYrGUcLRXNb693Z1wJiYRg2ftx83MLHMPi5AigyJNCLFofDyclVC7ONhh65mrGP7PQZy5zO52pHhAkSaEWDxBFTwwpWcD2NgAyw9fwhOTt+DZaTswb28EktNvmnt4hBQaFGlCiCZoX8cHs15proLJZL56X/gNjJx/GM3Hrsf7C4/gSGScuYdISPFqsEEIIblzqGWJiU9VDTn+2ROBC9eTMWv3BbVIkFnPphXxdIMK8HBx4IdHNI9F95PWcg9QQkjhk5Wlw67Qa5izJwKrjkardC3B2cEWT9Ytj55N/dE0wFM18SDkQWA/aUIIuUdsbW0QXK2MWm4kpWPhgYuYs+cCTl9OxIL9F9VSpayrsq67NvRDWbfsXvOEaAFa0oQQq0KcgwciYvFPSASWHo5CcnqmWi/GdCN/T7St5Y0napdTdcJpYRNLt6Qp0oQQq0X6Uy89FKXc4YciYnNsk7aYbWp5K9EWl7i9HeNoyZ2hSBcSnJMmhAhRsSlYfzIG645fxs5z14zz14IEmT1Ws6wS7dY1y8LdmUFnJCcU6UKCIk0IycvC3nr6CtadiMGGk5dxIzm73Ki0yXyoSmm0rVVOiXZFrxL8AAko0hRpQogZyMzSYf+FG1h34rKyss9dScqxvXZ5d3RtWAFdGvqinJszz1ExJZJz0tb1wRJCtEnolUSsPxGDtScuY2/YdWTdSlKVAiqPVC+DZxv7qXlsZwc7cw+VFCFMwSKEEAugStmSann1kSoqrWv5kUv4d38kDlyIxcZTV9Ti7myPp+r74tlGfmjkX4pR4qTQYHQ3IYTkg3NXEvHvvkiVi30pLtW4vkoZV2Vdi0vct5QLP0srJZLubuv6YAkh1juHLdHhC/ZHYuXRaKRkZOdhB1ctrazrDkE+qs0msR4iKdLW9cESQopHlPjKW+7wXaHXjetLONrh0ZplUaakE9yc7VHSyQElne2Vm7ykkz3cnB1u3eoXuc88bcsm0kxawks9Qgi5T0RcuzepqJaI68nKFS6CHX4tGSuORN/Ta0m/bBFycZl/0KkWmgZ48bwQy56THjduHBYsWICTJ0/CxcUFwcHB+PLLL1GzZs18vwYtaUJIUSI/qdJGU5aE1JvK2pbbhNQMdT/7sX5d2s3soioGHOxs8EXXukr8iWVASzoPNm/ejMGDB6Np06a4efMm3nvvPbRr1w7Hjx+Hq6tr0Z8lQgj5D6QeeJMAL7Xkh/SbWUi6JdzxqRn4YdNZZYVLr+yzMYl4p0OgSv8ixROLtqRzc+XKFZQrV06J9yOPPJKv59CSJoRorf3mlHWnMXXDWfVYKp9N6dlQudaJ+TCXlmiqonxcXJy69fK68xVqWloa4uPjjUtCQkIRjpAQQh68/eab7Wrim54N4Ghvq0qXPjdth5rzJsUPzYh0VlYWRowYgZYtWyIoKOiu89geHh7GpXbt2kU6TkIIKQi6NKiAua+3UD2wT0Yn4Jnvt6sKaKR4oRmRlrnpo0ePYs6cOXfdb/To0criNiwyf00IIVqkQcVSWDKkJer4uuNaUjpenLEb8/dFmntYpAjRhEgPGTIEy5Ytw8aNG/9zLsDJyQnu7u7Gxc3NrcjGSQghBU15DxfMG9ACHer4qPaab887hPErT6q56wclOi4Va45FIyYhu4IasSwsOhJBYtqGDh2KhQsXYtOmTahcubK5h0QIIUWOVC/7oVcjTFp7Gt9tPIsfN59TZUqnPN8ArvcQUCa/qccvxWPd8RjV9evIxThjytdT9XzRr2UA6vmVKsQjIVYV3T1o0CDMnj0bixcvzpEbLXPNkjedHxjdTQixJhYfvKjSsyR1K9DHDT/3aQI/zzv3vJb9doVeM7bijDKpOy6lTP08XRBxPcW4rnElTyXW7ev4wMFOE87WIoFlQe+Qb5gXM2fORN++fS36gyWEkMJC+l+/9sc+XE1MQ5mSjvipdxMlrgZik9Ox8VSMspg3n76iCqgYcHawxcPVy+KJWt54LLCcCkw7FBGL33aEYdnhKGRk6u228h7OeOmhSnihmT+8XB2L/cmMZO3uwoEiTQixRi7GpuCV3/fixKV4ONrZ4sOnaqnqZWuPX8be8BuqEYgBEWLJt5Y+2C2rlbljL+yY+FTM2n0Bs3aH42piulrnZG+LZxpUQL9WAQj0cX+gMYtVfzk+FeXcneBkr61+3JEUaev6YAkhpLCRSmVv/HMQa45fvm2buMJFlNvW9ka9Ch4q/zq/pN3MxLJDlzBzx3kcvRhvXN+iSmn0bRmgXjevKmgyexqXkoEL15NV/XK5jTC5fykuBXLtUNrVEb1bVELvhyqhdEknaIFIirR1fbCEEFIUSJS3BJT9uv08GvqX0gtzLW9U9LrzPPW91iGfuT0Mq45FG63zil4uSmClu5eI74XrSUZhlvKmd0O03WDki5UuvbhfaVUZVcqWhCUTSZG2rg+WEEKsiajYFPy5Kxx/h1xAbHLGXfct5+YEf68S8C9dQn/rVQKVSpdQFw6eJRxVH+4ZW0KN0eUSfiQXFq89UgVNKnneMR7JnFCkreyDJYQQayQlPROLDl7EkoNRcHG0M4qwQZQrepZQ6/Njpe8+f12J9fqTMcb19SuWwmsPV0H7Ot4F1mNbPAAP2qSEIl1IUKQJIcSykW5fv2wLxb/7L6rgMoNLvX/LyujRpGK+csFF9GMS0lT++PmrSTh/JQmhcns1CdeT0nHwoyceyEKnSBcSFGlCCNEGklL2x85w/LkzDDduudTdne1VKljf4ACUc3dWPbjDriYj9GoiQo1CnKhEOSk9846vHfJeG/X8+4UiXUhQpAkhRHsu9X/3R+KXbeeVJWyoilaqhCOuJKTd8Xni0q7o6aKC0CqXcVVLlbKuqFKmJLzdnTRpSVt0WVBCCCHFD5nTNhRSWXfiMn7eGoo9YTeMAl2mpBOq3BJgvRDrRVnmxaW9pzVBkSaEEGKR2NnaqPKkspyKTkBqRiYql3WFu7MDigsUaUIIIRZPTZ/i2dHQuvwChBBCiBVBkSaEEEIsFIo0IYQQYqFQpAkhhBALhSJNCCGEWChWH92dlaUvMXfp0iVzD4UQQohGuXRLQwyaUlRYvUhfvqzvs9qsWTNzD4UQQogVaIq/v3+RvZ+NTqqSWzE3b97EgQMH4O3tDVtby/fuJyQkoHbt2jh+/Djc3KwjL9Dajsnajscaj4nHY/lo7RxlZWUpgW7YsCHs7YvOvrV6kdYa8fHx8PDwQFxcHNzd3WENWNsxWdvxWOMx8XgsH2s7R4WF5ZuWhBBCSDGFIk0IIYRYKBRpC8PJyQkff/yxurUWrO2YrO14rPGYeDyWj7Wdo8KCc9KEEEKIhUJLmhBCCLFQKNKEEEKIhUKRJoQQQiwUirSFMG7cODRt2lQl9ZcrVw7PPPMMTp06BWth/PjxsLGxwYgRI6BlLl68iJdeegmlS5eGi4sL6tati71790KLZGZm4sMPP0TlypXVsVStWhWff/45tFQ6YcuWLejcuTN8fX3V92vRokU5tsuxfPTRRyhfvrw6xrZt2+LMmTPQ4vFkZGRg1KhR6jvn6uqq9nn55ZcRFRUFLZ8jUwYMGKD2mTJlSpGO0ZKhSFsImzdvxuDBg7Fr1y6sXbtW/UO2a9cOSUlJ0Dp79uzBTz/9hHr16kHL3LhxAy1btoSDgwNWrlypKiV9/fXX8PT0hBb58ssvMW3aNHz33Xc4ceKEevzVV1/h22+/hVaQ/4/69evj+++/z3O7HM/UqVPx448/Yvfu3Urc2rdvj9TUVGjteJKTk7F//351YSW3CxYsUBfyTz/9NLR8jgwsXLhQ/f6JmBMTpOIYsTxiYmLEnNFt3rxZp2USEhJ01atX161du1bXunVr3fDhw3VaZdSoUbpWrVrprIVOnTrp+vfvn2Ndt27ddL169dJpEfl/WbhwofFxVlaWzsfHRzdhwgTjutjYWJ2Tk5Pu77//1mntePIiJCRE7RceHq7TAnc6psjISF2FChV0R48e1VWqVEk3efJks4zPEqElbaFIqTzBy8sLWka8A506dVJuRq2zZMkSNGnSBN27d1dTElLDd8aMGdAqwcHBWL9+PU6fPq0eHzp0CNu2bUPHjh1hDZw/fx7R0dE5vntShrJ58+bYuXMnrOV3QtzDpUqVglaRmti9e/fGyJEjUadOHXMPx+Kw+i5YWv3SytytuFaDgoKgVebMmaPccuLutgZCQ0OVe/jNN9/Ee++9p45r2LBhcHR0RJ8+faA13n33XVU/OTAwEHZ2dmqOeuzYsejVqxesARFoQZrrmCKPDdu0jLjsZY76hRde0HTta5lmkYYV8r9EbocibaHW59GjR5VVo1UiIiIwfPhwNb/u7OwMa7l4Ekv6iy++UI/FkpbzJPOdWhTpuXPnYtasWZg9e7ayYA4ePKguDmVOUIvHU5yQmJUePXqowDi5cNQq+/btwzfffKMu5sUjQG6H7m4LY8iQIVi2bBk2btwIPz8/aPmfLyYmBo0aNVJXybJIcJwE8ch9sdq0hkQIS2s9U2rVqoULFy5Ai4h7Uazpnj17qohhcTm+8cYbKtPAGvDx8cnRU96APDZs07JAh4eHq4tgLVvRW7duVb8T0p/Z8Dshx/XWW28hICDA3MOzCGhJWwhyRTx06FAV4bhp0yaVFqNl2rRpgyNHjuRY169fP+VaFReduFe1hkw/5E6Lk/ncSpUqQYtItHDuHutyXsRjYA3I/5CIscy7N2jQQK0T975EeQ8cOBBaFmhJI5MLeUkF1DJyYZg7XkWi72W9/F4QirRFubjF7bh48WKVK22YM5NAF8nv1BpyDLnn0yX9RX5UtDrPLlamBFuJu1t+KENCQjB9+nS1aBHJXZU5aLFixN194MABTJo0Cf3794dWSExMxNmzZ3MEi4nbXgIu5bjEfT9mzBhUr15dibakL4k7X+oQaO14xJPz3HPPKdeweNvEG2X4nZDtEhuhxXOU+0JDUhzl4qpmzZpmGK0FYu7wcqJHTkVey8yZM63mI9J6CpawdOlSXVBQkErjCQwM1E2fPl2nVeLj49X58Pf31zk7O+uqVKmie//993VpaWk6rbBx48Y8/2/69OljTMP68MMPdd7e3uqctWnTRnfq1CmdFo/n/Pnzd/ydkOdp9RzlhilYOWEXLEIIIcRCYeAYIYQQYqFQpAkhhBALhSJNCCGEWCgUaUIIIcRCoUgTQgghFgpFmhBCCLFQKNKEEEKIhUKRJoQQQiwUijQhJN9Ip6JFixbxEyOkiKBIE6IR+vbtq0Qy99KhQwdzD40QUkiwCxYhGkIEeebMmTnWOTk5mW08hJDChZY0IRpCBFk6BJkunp6eaptY1dOmTUPHjh1V57QqVapg/vz5OZ4v7UMff/xxtV26D7322muqS5Epv/76q+qKJe8lnZekx7kpV69eRdeuXVGiRAnVXWrJkiXGbTdu3ECvXr1QtmxZ9R6yPfdFBSEk/1CkCbEipBXjs88+i0OHDimx7NmzJ06cOKG2JSUlqV69Iup79uzBvHnzsG7duhwiLCIvbVNFvEXQRYCrVauW4z0+/fRT1arz8OHDePLJJ9X7XL9+3fj+x48fx8qVK9X7yuuVKVOmiD8FQqyIXF2xCCEWirT2s7Oz07m6uuZYxo4dq7bLv/OAAQNyPKd58+a6gQMHqvvSVtPT01OXmJho3L58+XKdra2tLjo6Wj329fVV7SrvhLzHBx98YHwsryXrVq5cqR537txZ169fvwI+ckKKL5yTJkRDPPbYY8o6NcXLy8t4v0WLFjm2yeODBw+q+2LZ1q9fH66ursbtLVu2RFZWFk6dOqXc5VFRUWjTps1dx1CvXj3jfXktd3d3xMTEqMcDBw5Ulvz+/fvRrl07PPPMMwgODn7Aoyak+EKRJkRDiCjmdj8XFDKHnB8cHBxyPBZxF6EXZD48PDwcK1aswNq1a5Xgi/t84sSJhTJmQqwdzkkTYkXs2rXrtse1atVS9+VW5qplbtrA9u3bYWtri5o1a8LNzQ0BAQFYv379A41Bgsb69OmDv/76C1OmTMH06dMf6PUIKc7QkiZEQ6SlpSE6OjrHOnt7e2NwlgSDNWnSBK1atcKsWbMQEhKCX375RW2TAK+PP/5YCegnn3yCK1euYOjQoejduze8vb3VPrJ+wIABKFeunLKKExISlJDLfvnho48+QuPGjVV0uIx12bJlxosEQsi9Q5EmREOsWrVKpUWZIlbwyZMnjZHXc+bMwaBBg9R+f//9N2rXrq22ScrU6tWrMXz4cDRt2lQ9lvnjSZMmGV9LBDw1NRWTJ0/G22+/rcT/ueeey/f4HB0dMXr0aISFhSn3+cMPP6zGQwi5P2wkeuw+n0sIsSBkbnjhwoUqWIsQYh1wTpoQQgixUCjShBBCiIXCOWlCrATOXBFifdCSJoQQQiwUijQhhBBioVCkCSGEEAuFIk0IIYRYKBRpQgghxEKhSBNCCCEWCkWaEEIIsVAo0oQQQoiFQpEmhBBCYJn8HzgCkqcDRYCRAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from previous_chapters import plot_losses\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import plot_losses\n",
    "\n",
    "\n",
    "epochs_tensor = torch.linspace(1, n_epochs, len(train_losses))\n",
    "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)\n",
    "plt.tight_layout(); plt.savefig(\"3.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c16fa614-67e1-4254-8b7e-c3e2f690c29c",
   "metadata": {},
   "source": [
    "- Note that the model is overfitting here because the dataset is kept very small for educational purposes (so that the code can be executed on a laptop computer)\n",
    "- For a longer pretraining run on a much larger dataset, see [../../ch05/03_bonus_pretraining_on_gutenberg](../../ch05/03_bonus_pretraining_on_gutenberg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}