LLMs-from-scratch/appendix-D/01_main-chapter-code/appendix-D.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9a5936bd-af17-4a7e-a4d2-e910411708ea",
   "metadata": {},
   "source": [
    "<font size=\"1\">\n",
    "Supplementary code for \"Build a Large Language Model From Scratch\": <a href=\"https://www.manning.com/books/build-a-large-language-model-from-scratch\">https://www.manning.com/books/build-a-large-language-model-from-scratch</a> by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af53bcb1-ff9d-49c7-a0bc-5b8d32ff975b",
   "metadata": {},
   "source": [
    "## Appendix D: Adding Bells and Whistles to the Training Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f58c142-9434-49af-b33a-356b80a45b86",
   "metadata": {},
   "source": [
    "- In this appendix, we add a few more advanced features to the training function, which are used in typical pretraining and finetuning; finetuning is covered in chapters 6 and 7\n",
    "- The next three sections below discuss learning rate warmup, cosine decay, and gradient clipping\n",
    "- The final section adds these techniques to the training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "744def4f-c03f-42ee-97bb-5d7d5b89b723",
   "metadata": {},
   "source": [
    "- We start by initializing a model reusing the code from chapter 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8755bd5e-bc06-4e6e-9e63-c7c82b816cbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch version: 2.2.1\n",
      "tiktoken version: 0.5.1\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "import torch\n",
    "import tiktoken\n",
    "\n",
    "print(\"torch version:\", version(\"torch\"))\n",
    "print(\"tiktoken version:\", version(\"tiktoken\"))\n",
    "\n",
    "\n",
    "from previous_chapters import GPTModel\n",
    "\n",
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,  # Vocabulary size\n",
    "    \"ctx_len\": 256,       # Shortened context length (orig: 1024)\n",
    "    \"emb_dim\": 768,       # Embedding dimension\n",
    "    \"n_heads\": 12,        # Number of attention heads\n",
    "    \"n_layers\": 12,       # Number of layers\n",
    "    \"drop_rate\": 0.1,     # Dropout rate\n",
    "    \"qkv_bias\": False     # Query-key-value bias\n",
    "}\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.eval();  # Disable dropout during inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51574e57-a098-412c-83e8-66dafa5a0b99",
   "metadata": {},
   "source": [
    "- Next, using the same code we used in chapter 5, we initialize the data loaders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "386ca110-2bb4-42f1-bd54-8836df80acaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib.request\n",
    "\n",
    "file_path = \"the-verdict.txt\"\n",
    "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n",
    "\n",
    "if not os.path.exists(file_path):\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        text_data = response.read().decode('utf-8')\n",
    "    with open(file_path, \"w\", encoding=\"utf-8\") as file:\n",
    "        file.write(text_data)\n",
    "else:\n",
    "    with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "        text_data = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae96992b-536a-4684-a924-658b9ffb7e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import create_dataloader_v1\n",
    "\n",
    "# Train/validation ratio\n",
    "train_ratio = 0.90\n",
    "split_idx = int(train_ratio * len(text_data))\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = create_dataloader_v1(\n",
    "    text_data[:split_idx],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"ctx_len\"],\n",
    "    stride=GPT_CONFIG_124M[\"ctx_len\"],\n",
    "    drop_last=True,\n",
    "    shuffle=True\n",
    ")\n",
    "\n",
    "val_loader = create_dataloader_v1(\n",
    "    text_data[split_idx:],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"ctx_len\"],\n",
    "    stride=GPT_CONFIG_124M[\"ctx_len\"],\n",
    "    drop_last=False,\n",
    "    shuffle=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939c08d8-257a-41c6-b842-019f7897ac74",
   "metadata": {},
   "source": [
    "## D.1 Learning rate warmup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fafcd30-ddf7-4a9f-bcf4-b13c052b3133",
   "metadata": {},
   "source": [
    "- When training complex models like LLMs, implementing learning rate warmup can help stabilize the training\n",
    "- In learning rate warmup, we gradually increase the learning rate from a very low value (`initial_lr`) to a user-specified maximum (`peak_lr`)\n",
    "- This way, the model will start the training with small weight updates, which helps decrease the risk of large destabilizing updates during the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2bb4790b-b8b6-4e9e-adf4-704a04b31ddf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "135\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 15\n",
    "peak_lr = 0.01\n",
    "initial_lr = 0.0001\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=peak_lr, weight_decay=0.1)\n",
    "total_training_steps = len(train_loader) * n_epochs\n",
    "\n",
    "print(total_training_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf3a8da-abc4-4b80-a5d8-f1cc1c7cc5f3",
   "metadata": {},
   "source": [
    "- Typically, the number of warmup steps is between 10% and 20% of the total number of steps\n",
    "- We can compute the increment as the difference between the `peak_lr` and `initial_lr` divided by the number of warmup steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e075f80e-a398-4809-be1d-8019e1d31c90",
   "metadata": {},
   "outputs": [],
   "source": [
    "warmup_steps = 20\n",
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "global_step = -1\n",
    "track_lrs = []\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        if global_step < warmup_steps:\n",
    "            lr = initial_lr + global_step * lr_increment\n",
    "        else:\n",
    "            lr = peak_lr\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.param_groups[0][\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights\n",
    "        # ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cb6da121-eeed-4023-bdd8-3666c594b4ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x15624fd10>]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAEmCAYAAABYuVhFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA1FElEQVR4nO3de1xUdf4/8NcgMpDKoLIyoCijsUsFeYGYxtiv6zo1FWWkeVtWzVgvLaVG3yxvUK19ScxycS11t2+2v81L7haVX6MlLK1EkEsWmpcSxcTBW8woym3m8/vD5ejEqAzOcM7A6/l4zAPnc97nzPujNW/OeZ+LSgghQERERIrhI3cCRERE5IjFmYiISGFYnImIiBSGxZmIiEhhWJyJiIgUhsWZiIhIYViciYiIFIbFmYiISGF85U7AW9ntdlRVVaFHjx5QqVRyp0NERDIQQuDcuXMICwuDj4/79ndZnNuoqqoK4eHhcqdBREQKcOzYMfTr189t22NxbqMePXoAuPQPEhgYKHM2REQkB6vVivDwcKkmuAuLcxs1H8oODAxkcSYi6uTc3d7kCWFEREQKw+JMRESkMCzORERECiN7cV61ahUiIiLg7+8PvV6PoqKia8Zv3rwZUVFR8Pf3R0xMDLZu3eqw/L333sM999yD3r17Q6VS4euvv26xjbq6OqSmpqJ3797o3r07xo4di+rqandOi4iIqM1kLc6bNm1CWloaMjIyUFpaisGDB8NkMuHkyZNO43fu3IlJkyYhJSUFZWVlSEpKQlJSEsrLy6WY2tpaJCQkYOnSpVf93KeeegofffQRNm/ejO3bt6Oqqgpjxoxx+/yIiIjaQiWEEHJ9uF6vxx133IG//OUvAC7d2CM8PBxPPvkknnvuuRbxEyZMQG1tLbZs2SKN3XnnnRgyZAhWr17tEHvkyBHodDqUlZVhyJAh0rjFYsEvfvELrF+/Ho888ggAYP/+/bjllltQUFCAO++8s1W5W61WaDQaWCwWnq1NRNRJeaoWyHYpVUNDA0pKSjB//nxpzMfHB0ajEQUFBU7XKSgoQFpamsOYyWRCTk5Oqz+3pKQEjY2NMBqN0lhUVBT69+9/zeJcX1+P+vp66b3Vam31Z3YWQggseL8cZZU/yZ0KEVGr3BHRC39KipY7jRZkK86nT5+GzWZDSEiIw3hISAj279/vdB2z2ew03mw2t/pzzWYz/Pz8EBQU5NJ2MjMz8cILL7T6czqj/eZz2FBUKXcaREStFqrxlzsFp3gTklaaP3++w157811h6LJdh88AAIb1D0La3b+SORsiousLuqmr3Ck4JVtxDg4ORpcuXVqcJV1dXQ2tVut0Ha1W61L81bbR0NCAmpoah73n621HrVZDrVa3+nM6o+bibLw1BAmRwTJnQ0TkvWQ7W9vPzw+xsbHIz8+Xxux2O/Lz82EwGJyuYzAYHOIBIC8v76rxzsTGxqJr164O2zlw4AAqKytd2g45stsFCivOAgDuHNhb5myIiLybrIe109LSMHXqVMTFxSE+Ph4rVqxAbW0tpk2bBgCYMmUK+vbti8zMTADAnDlzMGLECCxfvhyJiYnYuHEjiouLsXbtWmmbZ8+eRWVlJaqqqgBcKrzApT1mrVYLjUaDlJQUpKWloVevXggMDMSTTz4Jg8HQ6jO1qaUD1edQc6ERN/l1QUxfjdzpEBF5NVmL84QJE3Dq1Cmkp6fDbDZjyJAhyM3NlU76qqysdHg+5vDhw7F+/XosWrQICxYsQGRkJHJychAdfflMuw8//FAq7gAwceJEAEBGRgaef/55AMBrr70GHx8fjB07FvX19TCZTHj99dfbYcYdV/Mh7biIXujaRfZ72xAReTVZr3P2ZrzO2dHM/1eMT/ZWY969v8Iff3Oz3OkQEbULT9UC7uLQDWO/mYjIvVic6Yax30xE5F4sznTD2G8mInIvfpPSDSv44VJxvnNgL5kzISLqGFic6Yaw30xE5H4sznRD9pvPwXKxEd3YbyYichsWZ7oh7DcTEbkfv03phjQXZx7SJiJyHxZnajPHfjNPBiMichcWZ2qzK/vN0ew3ExG5DYsztRn7zUREnsFvVGoz9puJiDyDxZnahP1mIiLPYXGmNmG/mYjIc1icqU0K2G8mIvIYfqtSm7DfTETkOSzO5DK7XaDoP/1mwyAWZyIid2NxJpd9Z7Ze7jeHBcqdDhFRh8PiTC7bdfjSXvMdul7wZb+ZiMjt+M1KLmO/mYjIs1icySVX9ptZnImIPIPFmVzCfjMRkeexOJNL2G8mIvI8fruSS9hvJiLyPBZnajX2m4mI2geLM7XavhOX+s3d1b7sNxMReRCLM7Va8yHtOyJ6st9MRORB/IalVms+GYyHtImIPIvFmVrFZhcoquDJYERE7YHFmVrluxNWWOua0F3ti9vYbyYi8igWZ2oV9puJiNqP7N+yq1atQkREBPz9/aHX61FUVHTN+M2bNyMqKgr+/v6IiYnB1q1bHZYLIZCeno7Q0FAEBATAaDTi0KFDDjEHDx7EQw89hODgYAQGBiIhIQGfffaZ2+fWkbDfTETUfmQtzps2bUJaWhoyMjJQWlqKwYMHw2Qy4eTJk07jd+7ciUmTJiElJQVlZWVISkpCUlISysvLpZisrCxkZ2dj9erVKCwsRLdu3WAymVBXVyfFPPDAA2hqasK2bdtQUlKCwYMH44EHHoDZbPb4nL0R+81ERO1MyCg+Pl6kpqZK7202mwgLCxOZmZlO48ePHy8SExMdxvR6vZg5c6YQQgi73S60Wq1YtmyZtLympkao1WqxYcMGIYQQp06dEgDEjh07pBir1SoAiLy8vFbnbrFYBABhsVhavY63+vbHGjHg2S3itvRc0dhkkzsdIiLF8FQtkG3PuaGhASUlJTAajdKYj48PjEYjCgoKnK5TUFDgEA8AJpNJiq+oqIDZbHaI0Wg00Ov1Ukzv3r3xq1/9Cn//+99RW1uLpqYmrFmzBn369EFsbKy7p9khsN9MRNS+fOX64NOnT8NmsyEkJMRhPCQkBPv373e6jtlsdhrffDi6+ee1YlQqFT799FMkJSWhR48e8PHxQZ8+fZCbm4uePXteNd/6+nrU19dL761Waytn6v3YbyYial+dbjdICIHU1FT06dMHX3zxBYqKipCUlIQHH3wQJ06cuOp6mZmZ0Gg00is8PLwds5aPzS5QyH4zEVG7kq04BwcHo0uXLqiurnYYr66uhlardbqOVqu9Znzzz2vFbNu2DVu2bMHGjRtx1113YdiwYXj99dcREBCAt99++6r5zp8/HxaLRXodO3bMtQl7qe9OWHGurgk9eH0zEVG7ka04+/n5ITY2Fvn5+dKY3W5Hfn4+DAaD03UMBoNDPADk5eVJ8TqdDlqt1iHGarWisLBQirlw4QKAS/3tK/n4+MBut181X7VajcDAQIdXZyD1m/n8ZiKidiNbzxkA0tLSMHXqVMTFxSE+Ph4rVqxAbW0tpk2bBgCYMmUK+vbti8zMTADAnDlzMGLECCxfvhyJiYnYuHEjiouLsXbtWgCX+slz587FkiVLEBkZCZ1Oh8WLFyMsLAxJSUkALhX4nj17YurUqUhPT0dAQAD++te/oqKiAomJibL8PSjZ5ec395I5EyKizkPW4jxhwgScOnUK6enpMJvNGDJkCHJzc6UTuiorKx32cIcPH47169dj0aJFWLBgASIjI5GTk4Po6GgpZt68eaitrcWMGTNQU1ODhIQE5Obmwt/fH8Clw+m5ublYuHAhfvvb36KxsRG33XYbPvjgAwwePLh9/wIU7lK/mSeDERG1N5UQQsidhDeyWq3QaDSwWCwd9hB3+XELHlj5JXqofVGWfjcPaxMR/YynagG/bemq2G8mIpIHv3HpqthvJiKSB4szOcV+MxGRfFicyakrr2++NbRj9tSJiJSKxZmcYr+ZiEg+/NYlpwp+uFScDTykTUTU7licqYVLz29mv5mISC4sztTCviorztX/p9/M+2kTEbU7FmdqobnfHK/rhS4+KpmzISLqfFicqYXL1zfzkDYRkRxYnMkB+81ERPJrU3FuamrCp59+ijVr1uDcuXMAgKqqKpw/f96tyVH7Y7+ZiEh+Lj+V6ujRo7j33ntRWVmJ+vp63H333ejRoweWLl2K+vp6rF692hN5Ujthv5mISH4u7znPmTMHcXFx+OmnnxAQECCNP/zww8jPz3drctT+2G8mIpKfy3vOX3zxBXbu3Ak/Pz+H8YiICBw/ftxtiVH7Y7+ZiEgZXN5zttvtsNlsLcZ//PFH9OjRwy1JkTykfrM/+81ERHJyuTjfc889WLFihfRepVLh/PnzyMjIwP333+/O3KidFRw+DQDQs99MRCQrlw9rL1++HCaTCbfeeivq6urwu9/9DocOHUJwcDA2bNjgiRypnew6zEPaRERK4HJx7tevH/bs2YNNmzZhz549OH/+PFJSUpCcnOxwghh5lyabHbvZbyYiUgSXi/OOHTswfPhwJCcnIzk5WRpvamrCjh078F//9V9uTZDax74Tl/vNt/D5zUREsnK55zxy5EicPXu2xbjFYsHIkSPdkhS1v+ZLqNhvJiKSn8vFWQgBlarll/eZM2fQrVs3tyRF7Y/9ZiIi5Wj1Ye0xY8YAuHR29qOPPgq1Wi0ts9ls+OabbzB8+HD3Z0gex34zEZGytLo4azQaAJf2nHv06OFw8pefnx/uvPNOTJ8+3f0Zksex30xEpCytLs5vvfUWgEt3Avvv//5vHsLuQNhvJiJSFpfP1s7IyPBEHiQj9puJiJTF5eIMAP/85z/x7rvvorKyEg0NDQ7LSktL3ZIYtQ/2m4mIlMfls7Wzs7Mxbdo0hISEoKysDPHx8ejduzcOHz6M++67zxM5kgft/c/9tAPZbyYiUgyXi/Prr7+OtWvXYuXKlfDz88O8efOQl5eH2bNnw2KxeCJH8qDLz2/uzX4zEZFCuFycKysrpUumAgICcO7cOQDA5MmTeW9tL3T5+c29ZM6EiIiauVyctVqtdIew/v37Y9euXQCAiooKCCHcmx15VJPNjt1HfgLAfjMRkZK4XJx/+9vf4sMPPwQATJs2DU899RTuvvtuTJgwAQ8//LDbEyTP2VtlxXn2m4mIFMfl4rx27VosXLgQAJCamor//d//xS233IIXX3wRb7zxhssJrFq1ChEREfD394der0dRUdE14zdv3oyoqCj4+/sjJiYGW7dudVguhEB6ejpCQ0MREBAAo9GIQ4cOtdjO//3f/0Gv1yMgIAA9e/ZEUlKSy7l7O/abiYiUyaXi3NTUhCVLlsBsNktjEydORHZ2Np588kn4+fm59OGbNm1CWloaMjIyUFpaisGDB8NkMuHkyZNO43fu3IlJkyYhJSUFZWVlSEpKQlJSEsrLy6WYrKwsZGdnY/Xq1SgsLES3bt1gMplQV1cnxfzrX//C5MmTMW3aNOzZswdfffUVfve737mUe0fAfjMRkUIJF3Xr1k1UVFS4uppT8fHxIjU1VXpvs9lEWFiYyMzMdBo/fvx4kZiY6DCm1+vFzJkzhRBC2O12odVqxbJly6TlNTU1Qq1Wiw0bNgghhGhsbBR9+/YVf/vb324od4vFIgAIi8VyQ9uRS2OTTdyWnisGPLtFfPtjjdzpEBF5JU/VApcPa48aNQrbt2+/4V8KGhoaUFJSAqPRKI35+PjAaDSioKDA6ToFBQUO8QBgMpmk+IqKCpjNZocYjUYDvV4vxZSWluL48ePw8fHB0KFDERoaivvuu89h79uZ+vp6WK1Wh5c3Y7+ZiEi5XL5D2H333YfnnnsO3377LWJjY1vcY3v06NGt2s7p06dhs9kQEhLiMB4SEoL9+/c7XcdsNjuNbz7M3vzzWjGHDx8GADz//PN49dVXERERgeXLl+M3v/kNDh48iF69nB/izczMxAsvvNCquXkD9puJiJTL5eL8xz/+EQDw6quvtlimUqlgs9luPCsPstvtAICFCxdi7NixAC491KNfv37YvHkzZs6c6XS9+fPnIy0tTXpvtVoRHh7u+YQ9pLk4GwbxEioiIqVxuTg3F7cbFRwcjC5duqC6utphvLq6Glqt1uk6Wq32mvHNP6urqxEaGuoQM2TIEACQxm+99VZpuVqtxsCBA1FZWXnVfNVqtcMzrL2Z4/XNPBmMiEhpXO45u4ufnx9iY2ORn58vjdntduTn58NgMDhdx2AwOMQDQF5enhSv0+mg1WodYqxWKwoLC6WY2NhYqNVqHDhwQIppbGzEkSNHMGDAALfNT8nK/9Nv1gR0xS1a9puJiJSmTU+lcpe0tDRMnToVcXFxiI+Px4oVK1BbW4tp06YBAKZMmYK+ffsiMzMTADBnzhyMGDECy5cvR2JiIjZu3Iji4mKsXbsWwKXD6nPnzsWSJUsQGRkJnU6HxYsXIywsTLqOOTAwELNmzUJGRgbCw8MxYMAALFu2DAAwbty49v9LkMHlfnMv+LDfTESkOLIW5wkTJuDUqVNIT0+H2WzGkCFDkJubK53QVVlZCR+fyzv3w4cPx/r167Fo0SIsWLAAkZGRyMnJQXR0tBQzb9481NbWYsaMGaipqUFCQgJyc3Ph7+8vxSxbtgy+vr6YPHkyLl68CL1ej23btqFnz57tN3kZXb6+mf1mIiIlUgnBG2K3hdVqhUajgcViQWCg9xwabrLZMfiFf6O2wYb/m52A28I0cqdEROS1PFULZOs5kzzKq6yobbCx30xEpGAuH9a+2s03VCoV1Gq1y7fwpPbFfjMRkfK5XJyDgoKgUl39S71fv3549NFHkZGR4dAvJmVgv5mISPlcLs7r1q3DwoUL8eijjyI+Ph4AUFRUhLfffhuLFi3CqVOn8Morr0CtVmPBggVuT5jarslmx+6KS8/i5vXNRETK5XJxfvvtt7F8+XKMHz9eGnvwwQcRExODNWvWID8/H/3798dLL73E4qww7DcTEXkHl48779y5E0OHDm0xPnToUOnhEgkJCde82xbJo/mQtp79ZiIiRXO5OIeHh+PNN99sMf7mm29K95o+c+ZMp7lm2JsU/MB+MxGRN3D5sPYrr7yCcePG4eOPP8Ydd9wBACguLsb+/fvxz3/+EwCwe/duTJgwwb2Z0g1ptNlRfKS538ziTESkZC4X59GjR2P//v1Ys2YNDh48CODSYyRzcnIQEREBAHj88cfdmiTduPLjFqnfHKXtIXc6RER0DW26fadOp8PLL7/s7lzIg3YdvrTXzH4zEZHytak419TUoKioCCdPnmzxCMkpU6a4JTFyL17fTETkPVwuzh999BGSk5Nx/vx5BAYGOtyQRKVSsTgrEPvNRETexeWztZ9++mk89thjOH/+PGpqavDTTz9Jr7Nnz3oiR7pB7DcTEXkXl4vz8ePHMXv2bNx0002eyIc8gP1mIiLv4nJxNplMKC4u9kQu5CHsNxMReReXe86JiYl45plnsG/fPsTExKBr164Oy0ePHu225OjGXdlvNgxicSYi8gYuF+fp06cDAF588cUWy1QqFWw2241nRW7T3G8OuqkrfhXCfjMRkTdwuTj//NIpUrYC3k+biMjr8IHLHVzzyWDsNxMReY9W7TlnZ2djxowZ8Pf3R3Z29jVjZ8+e7ZbE6Mbx+mYiIu+kEkKI6wXpdDoUFxejd+/e0Ol0V9+YSoXDhw+7NUGlslqt0Gg0sFgsCAxU5rORSyt/wpjXdyLopq4oXXQ3D2sTEbmZp2pBq/acKyoqnP6ZlI3PbyYi8k7sOXdg7DcTEXknl8/WttlsWLduHfLz850++GLbtm1uS47ajv1mIiLv5XJxnjNnDtatW4fExERER0c7PPiClOPb4xZcaLChJ69vJiLyOi4X540bN+Ldd9/F/fff74l8yE0u95t7s99MRORlXO45+/n54eabb/ZELuRGl/vNvWTOhIiIXNWmR0b++c9/RiuuwCKZOPSbeT9tIiKv4/Jh7S+//BKfffYZPv74Y9x2220tHnzx3nvvuS05aptvfrzcb/5lH/abiYi8jcvFOSgoCA8//LAnciE3Yb+ZiMi7uVScm5qaMHLkSNxzzz3QarWeyolu0OXnN7PfTETkjVzqOfv6+mLWrFmor693axKrVq1CREQE/P39odfrUVRUdM34zZs3IyoqCv7+/oiJicHWrVsdlgshkJ6ejtDQUAQEBMBoNOLQoUNOt1VfX48hQ4ZApVLh66+/dteUZHOp3/wTAPabiYi8lcsnhMXHx6OsrMxtCWzatAlpaWnIyMhAaWkpBg8eDJPJhJMnTzqN37lzJyZNmoSUlBSUlZUhKSkJSUlJKC8vl2KysrKQnZ2N1atXo7CwEN26dYPJZEJdXV2L7c2bNw9hYWFum4/cvvnRgouN7DcTEXk14aJNmzaJgQMHipUrV4qdO3eKPXv2OLxcFR8fL1JTU6X3NptNhIWFiczMTKfx48ePF4mJiQ5jer1ezJw5UwghhN1uF1qtVixbtkxaXlNTI9RqtdiwYYPDelu3bhVRUVFi7969AoAoKytrdd4Wi0UAEBaLpdXrtIe/bDskBjy7Rcz8e7HcqRARdXieqgUunxA2ceJEAI6PhlSpVBBCQKVSwWaztXpbDQ0NKCkpwfz586UxHx8fGI1GFBQUOF2noKAAaWlpDmMmkwk5OTkALj2Yw2w2w2g0Sss1Gg30ej0KCgqk/KurqzF9+nTk5OTgpptuum6u9fX1DofzrVZrq+fZnthvJiLyfi4XZ3c+ler06dOw2WwICQlxGA8JCcH+/fudrmM2m53Gm81maXnz2NVihBB49NFHMWvWLMTFxeHIkSPXzTUzMxMvvPBCq+YlF/abiYg6BpeL84ABAzyRR7tauXIlzp0757DHfj3z58932GO3Wq0IDw/3RHpt1txv7tXNj/1mIiIv5nJxbrZv3z5UVlaioaHBYXz06NGt3kZwcDC6dOmC6upqh/Hq6uqrXqql1WqvGd/8s7q6GqGhoQ4xQ4YMAXDpyVkFBQVQq9UO24mLi0NycjLefvvtFp+rVqtbxCsNn99MRNQxuFycDx8+jIcffhjffvut1GsGID2dypWes5+fH2JjY5Gfn4+kpCQAgN1uR35+Pp544gmn6xgMBuTn52Pu3LnSWF5eHgwGAwBAp9NBq9UiPz9fKsZWqxWFhYV4/PHHAQDZ2dlYsmSJtH5VVRVMJhM2bdoEvV7f6vyV5nK/mYe0iYi8WZseGanT6ZCfnw+dToeioiKcOXMGTz/9NF555RWXE0hLS8PUqVMRFxeH+Ph4rFixArW1tZg2bRoAYMqUKejbty8yMzOlzx8xYgSWL1+OxMREbNy4EcXFxVi7di2AS78kzJ07F0uWLEFkZCR0Oh0WL16MsLAw6ReA/v37O+TQvXt3AMCgQYPQr18/l+egBA1NV/SbWZyJiLyay8W5oKAA27ZtQ3BwMHx8fODj44OEhARkZmZi9uzZLl8DPWHCBJw6dQrp6ekwm80YMmQIcnNzpRO6Kisr4eNz+XLs4cOHY/369Vi0aBEWLFiAyMhI5OTkIDo6WoqZN28eamtrMWPGDNTU1CAhIQG5ubnw9/d3dbpe49vjNVK/ObJPd7nTISKiG6ASwrXHS/Xs2ROlpaXQ6XQYNGgQ/va3v2HkyJH44YcfEBMTgwsXLngqV0WxWq3QaDSwWCwIDAyUOx2s+ux7LPvkAO6L1uKN38fKnQ4RUafgqVrg8p5zdHQ09uzZA51OB71ej6ysLPj5+WHt2rUYOHCg2xIj17DfTETUcbhcnBctWoTa2loAwIsvvogHHngAv/71r9G7d29s2rTJ7QnS9bHfTETUsbhcnE0mk/Tnm2++Gfv378fZs2fRs2dP6Yxtal/sNxMRdSwuP/ii2ffff49PPvkEFy9eRK9evFWknHYdPguA1zcTEXUULhfnM2fOYNSoUfjlL3+J+++/HydOnAAApKSk4Omnn3Z7gnR9zf1mA2/ZSUTUIbhcnJ966il07doVlZWVDg+MmDBhAnJzc92aHF0f+81ERB2Pyz3nf//73/jkk09a3KwjMjISR48edVti1DrsNxMRdTwu7znX1tY6fcTi2bNnFX/v6Y6oud9858BePCGPiKiDcLk4//rXv8bf//536b1KpYLdbkdWVhZGjhzp1uTo+gp+4PXNREQdjcuHtbOysjBq1CgUFxejoaEB8+bNw969e3H27Fl89dVXnsiRrqKhyY7io817zizOREQdhct7ztHR0Th48CASEhLw0EMPoba2FmPGjEFZWRkGDRrkiRzpKr75sQZ1jXb2m4mIOpg2Pc9Zo9Fg4cKFDmM//vgjZsyYIT0dijzv8i072W8mIupI2nwTkp87c+YM3nzzTXdtjlrh8slgPKRNRNSRuK04U/tiv5mIqONicfZSzf3m3uw3ExF1OCzOXurKR0Sy30xE1LG0+oSwMWPGXHN5TU3NjeZCLrjy5iNERNSxtLo4azSa6y6fMmXKDSdE18d+MxFRx9bq4vzWW295Mg9ywZX95pvZbyYi6nDYc/ZCV96yk/1mIqKOh8XZC+2quHzzESIi6nhYnL1MfZMNJUf5/GYioo6MxdnLfPOjhf1mIqIOjsXZy+xiv5mIqMNjcfYy7DcTEXV8LM5e5Mp+s2EQ+81ERB0Vi7MXae43B3f3w6BfsN9MRNRRsTh7keZ+s579ZiKiDo3F2Ytc7jfzkDYRUUfG4uwlHPrNPBmMiKhDY3H2EnuOsd9MRNRZsDh7iebnN7PfTETU8SmiOK9atQoRERHw9/eHXq9HUVHRNeM3b96MqKgo+Pv7IyYmBlu3bnVYLoRAeno6QkNDERAQAKPRiEOHDknLjxw5gpSUFOh0OgQEBGDQoEHIyMhAQ0ODR+bnDs3Fmf1mIqKOT/bivGnTJqSlpSEjIwOlpaUYPHgwTCYTTp486TR+586dmDRpElJSUlBWVoakpCQkJSWhvLxcisnKykJ2djZWr16NwsJCdOvWDSaTCXV1dQCA/fv3w263Y82aNdi7dy9ee+01rF69GgsWLGiXObuK/WYiok5GyCw+Pl6kpqZK7202mwgLCxOZmZlO48ePHy8SExMdxvR6vZg5c6YQQgi73S60Wq1YtmyZtLympkao1WqxYcOGq+aRlZUldDpdq/O2WCwCgLBYLK1ep60KD58RA57dImL/9G9ht9s9/nlERNQ6nqoFsu45NzQ0oKSkBEajURrz8fGB0WhEQUGB03UKCgoc4gHAZDJJ8RUVFTCbzQ4xGo0Ger3+qtsEAIvFgl69rr5XWl9fD6vV6vBqL+w3ExF1LrIW59OnT8NmsyEkJMRhPCQkBGaz2ek6ZrP5mvHNP13Z5vfff4+VK1di5syZV801MzMTGo1GeoWHh197cm7UXJwN7DcTEXUKsvec5Xb8+HHce++9GDduHKZPn37VuPnz58NisUivY8eOtUt+fH4zEVHnI2txDg4ORpcuXVBdXe0wXl1dDa1W63QdrVZ7zfjmn63ZZlVVFUaOHInhw4dj7dq118xVrVYjMDDQ4dUe9hyzoL7JjuDuagz6Rbd2+UwiIpKXrMXZz88PsbGxyM/Pl8bsdjvy8/NhMBicrmMwGBziASAvL0+K1+l00Gq1DjFWqxWFhYUO2zx+/Dh+85vfIDY2Fm+99RZ8fJR5EOHyJVS92G8mIuokfOVOIC0tDVOnTkVcXBzi4+OxYsUK1NbWYtq0aQCAKVOmoG/fvsjMzAQAzJkzByNGjMDy5cuRmJiIjRs3ori4WNrzValUmDt3LpYsWYLIyEjodDosXrwYYWFhSEpKAnC5MA8YMACvvPIKTp06JeVztT12ufD6ZiKizkf24jxhwgScOnUK6enpMJvNGDJkCHJzc6UTuiorKx32aocPH47169dj0aJFWLBgASIjI5GTk4Po6GgpZt68eaitrcWMGTNQU1ODhIQE5Obmwt/fH8ClPe3vv/8e33//Pfr16+eQjxCiHWbdOuw3ExF1TiqhpGrkRaxWKzQaDSwWi8f6z4WHz2DC2l0I7q7G7oWjeFibiEhhPFULlNloJQDArsNnAbDfTETU2bA4Kxj7zUREnROLs0LVNdpQWsl+MxFRZ8TirFB7jtWgvsmOX/Tg9c1ERJ0Ni7NCXe43837aRESdDYuzQl158xEiIupcWJwViP1mIqLOjcVZga7sNw8MZr+ZiKizYXFWIPabiYg6NxZnBSo4fBoA+81ERJ0Vi7PCXOo31wBgv5mIqLNicVaYr4/VoIH9ZiKiTo3FWWGuvGUn+81ERJ0Ti7PC8PpmIiJicVaQK/vNBvabiYg6LRZnBWnuN/fpoYaO/WYiok6LxVlB2G8mIiKAxVlR+PxmIiICWJwVw/H6Zp4MRkTUmbE4KwT7zURE1IzFWSEKfmC/mYiILmFxVgj2m4mIqBmLswLUNdpQdqwGAPvNRETE4qwIZZXsNxMR0WUszgrQfEjbMIj9ZiIiYnFWBPabiYjoSizOMnPsN7M4ExERi7PsmvvNIYFqRPS+Se50iIhIAVicZcb7aRMR0c+xOMuM/WYiIvo5FmcZsd9MRETOKKI4r1q1ChEREfD394der0dRUdE14zdv3oyoqCj4+/sjJiYGW7dudVguhEB6ejpCQ0MREBAAo9GIQ4cOOcScPXsWycnJCAwMRFBQEFJSUnD+/Hm3z+1aSit/Yr+ZiIhakL04b9q0CWlpacjIyEBpaSkGDx4Mk8mEkydPOo3fuXMnJk2ahJSUFJSVlSEpKQlJSUkoLy+XYrKyspCdnY3Vq1ejsLAQ3bp1g8lkQl1dnRSTnJyMvXv3Ii8vD1u2bMGOHTswY8YMj8/3SrsOnwXAfjMREf2MkFl8fLxITU2V3ttsNhEWFiYyMzOdxo8fP14kJiY6jOn1ejFz5kwhhBB2u11otVqxbNkyaXlNTY1Qq9Viw4YNQggh9u3bJwCI3bt3SzEff/yxUKlU4vjx463K22KxCADCYrG0bqJOjFu9Uwx4dotYX3i0zdsgIiL5uKMWOCPrnnNDQwNKSkpgNBqlMR8fHxiNRhQUFDhdp6CgwCEeAEwmkxRfUVEBs9nsEKPRaKDX66WYgoICBAUFIS4uTooxGo3w8fFBYWGh08+tr6+H1Wp1eN2o1b+PxerfD8Nvo/rc8LaIiKjjkLU4nz59GjabDSEhIQ7jISEhMJvNTtcxm83XjG/+eb2YPn0cC6Kvry969ep11c/NzMyERqORXuHh4a2c5dX16uaHe6NDERLof8PbIiKijkP2nrO3mD9/PiwWi/Q6duyY3CkREVEHJWtxDg4ORpcuXVBdXe0wXl1dDa1W63QdrVZ7zfjmn9eL+fkJZ01NTTh79uxVP1etViMwMNDhRURE5AmyFmc/Pz/ExsYiPz9fGrPb7cjPz4fBYHC6jsFgcIgHgLy8PClep9NBq9U6xFitVhQWFkoxBoMBNTU1KCkpkWK2bdsGu90OvV7vtvkRERG1iVtPL2uDjRs3CrVaLdatWyf27dsnZsyYIYKCgoTZbBZCCDF58mTx3HPPSfFfffWV8PX1Fa+88or47rvvREZGhujatav49ttvpZiXX35ZBAUFiQ8++EB888034qGHHhI6nU5cvHhRirn33nvF0KFDRWFhofjyyy9FZGSkmDRpUqvz9tQZekRE5D08VQt85f7lYMKECTh16hTS09NhNpsxZMgQ5ObmSid0VVZWwsfn8g7+8OHDsX79eixatAgLFixAZGQkcnJyEB0dLcXMmzcPtbW1mDFjBmpqapCQkIDc3Fz4+18+8eqdd97BE088gVGjRsHHxwdjx45FdnZ2+02ciIjoKlRCCCF3Et7IarVCo9HAYrGw/0xE1El5qhbwbG0iIiKFkf2wtrdqPuDgjpuREBGRd2quAe4+CM3i3Ebnzp0DALfcjISIiLzbuXPnoNFo3LY99pzbyG63o6qqCj169GjzQyusVivCw8Nx7NixDtO35pyUr6PNB+h4c+po8wE63pya51NZWQmVSoWwsDCHk5dvFPec28jHxwf9+vVzy7Y64k1NOCfl62jzATrenDrafICONyeNRuOR+fCEMCIiIoVhcSYiIlIYFmcZqdVqZGRkQK1Wy52K23BOytfR5gN0vDl1tPkAHW9Onp4PTwgjIiJSGO45ExERKQyLMxERkcKwOBMRESkMizMREZHCsDjLaNWqVYiIiIC/vz/0ej2KiorkTqlVMjMzcccdd6BHjx7o06cPkpKScODAAYeYuro6pKamonfv3ujevTvGjh2L6upqmTJ2zcsvvwyVSoW5c+dKY944n+PHj+P3v/89evfujYCAAMTExKC4uFhaLoRAeno6QkNDERAQAKPRiEOHDsmY8bXZbDYsXrwYOp0OAQEBGDRoEP70pz853NNY6XPasWMHHnzwQYSFhUGlUiEnJ8dheWvyP3v2LJKTkxEYGIigoCCkpKTg/Pnz7TiLy641n8bGRjz77LOIiYlBt27dEBYWhilTpqCqqsphG0qaD3D9f6MrzZo1CyqVCitWrHAYd8ecWJxlsmnTJqSlpSEjIwOlpaUYPHgwTCYTTp48KXdq17V9+3akpqZi165dyMvLQ2NjI+655x7U1tZKMU899RQ++ugjbN68Gdu3b0dVVRXGjBkjY9ats3v3bqxZswa33367w7i3zeenn37CXXfdha5du+Ljjz/Gvn37sHz5cvTs2VOKycrKQnZ2NlavXo3CwkJ069YNJpMJdXV1MmZ+dUuXLsUbb7yBv/zlL/juu++wdOlSZGVlYeXKlVKM0udUW1uLwYMHY9WqVU6Xtyb/5ORk7N27F3l5ediyZQt27NiBGTNmtNcUHFxrPhcuXEBpaSkWL16M0tJSvPfeezhw4ABGjx7tEKek+QDX/zdq9v7772PXrl0ICwtrscwtcxIki/j4eJGamiq9t9lsIiwsTGRmZsqYVducPHlSABDbt28XQghRU1MjunbtKjZv3izFfPfddwKAKCgokCvN6zp37pyIjIwUeXl5YsSIEWLOnDlCCO+cz7PPPisSEhKuutxutwutViuWLVsmjdXU1Ai1Wi02bNjQHim6LDExUTz22GMOY2PGjBHJyclCCO+bEwDx/vvvS+9bk/++ffsEALF7924p5uOPPxYqlUocP3683XJ35ufzcaaoqEgAEEePHhVCKHs+Qlx9Tj/++KPo27evKC8vFwMGDBCvvfaatMxdc+KeswwaGhpQUlICo9Eojfn4+MBoNKKgoEDGzNrGYrEAAHr16gUAKCkpQWNjo8P8oqKi0L9/f0XPLzU1FYmJiQ55A945nw8//BBxcXEYN24c+vTpg6FDh+Kvf/2rtLyiogJms9lhThqNBnq9XrFzGj58OPLz83Hw4EEAwJ49e/Dll1/ivvvuA+Cdc7pSa/IvKChAUFAQ4uLipBij0QgfHx8UFha2e86uslgsUKlUCAoKAuCd87Hb7Zg8eTKeeeYZ3HbbbS2Wu2tOfPCFDE6fPg2bzYaQkBCH8ZCQEOzfv1+mrNrGbrdj7ty5uOuuuxAdHQ0AMJvN8PPzk/4HbBYSEgKz2SxDlte3ceNGlJaWYvfu3S2WeeN8Dh8+jDfeeANpaWlYsGABdu/ejdmzZ8PPzw9Tp06V8nb236BS5/Tcc8/BarUiKioKXbp0gc1mw0svvYTk5GQA8Mo5Xak1+ZvNZvTp08dhua+vL3r16qX4OdbV1eHZZ5/FpEmTpAdFeON8li5dCl9fX8yePdvpcnfNicWZbkhqairKy8vx5Zdfyp1Kmx07dgxz5sxBXl4e/P395U7HLex2O+Li4vA///M/AIChQ4eivLwcq1evxtSpU2XOrm3effddvPPOO1i/fj1uu+02fP3115g7dy7CwsK8dk6dRWNjI8aPHw8hBN544w2502mzkpIS/PnPf0ZpaWmbHxXcWjysLYPg4GB06dKlxdm+1dXV0Gq1MmXluieeeAJbtmzBZ5995vD4TK1Wi4aGBtTU1DjEK3V+JSUlOHnyJIYNGwZfX1/4+vpi+/btyM7Ohq+vL0JCQrxqPgAQGhqKW2+91WHslltuQWVlJQBIeXvTf4PPPPMMnnvuOUycOBExMTGYPHkynnrqKWRmZgLwzjldqTX5a7XaFieNNjU14ezZs4qdY3NhPnr0KPLy8hwer+ht8/niiy9w8uRJ9O/fX/quOHr0KJ5++mlEREQAcN+cWJxl4Ofnh9jYWOTn50tjdrsd+fn5MBgMMmbWOkIIPPHEE3j//fexbds26HQ6h+WxsbHo2rWrw/wOHDiAyspKRc5v1KhR+Pbbb/H1119Lr7i4OCQnJ0t/9qb5AMBdd93V4vK2gwcPYsCAAQAAnU4HrVbrMCer1YrCwkLFzunChQstHmbfpUsX2O12AN45pyu1Jn+DwYCamhqUlJRIMdu2bYPdboder2/3nK+nuTAfOnQIn376KXr37u2w3NvmM3nyZHzzzTcO3xVhYWF45pln8MknnwBw45zafh4b3YiNGzcKtVot1q1bJ/bt2ydmzJghgoKChNlslju163r88ceFRqMRn3/+uThx4oT0unDhghQza9Ys0b9/f7Ft2zZRXFwsDAaDMBgMMmbtmivP1hbC++ZTVFQkfH19xUsvvSQOHTok3nnnHXHTTTeJf/zjH1LMyy+/LIKCgsQHH3wgvvnmG/HQQw8JnU4nLl68KGPmVzd16lTRt29fsWXLFlFRUSHee+89ERwcLObNmyfFKH1O586dE2VlZaKsrEwAEK+++qooKyuTzl5uTf733nuvGDp0qCgsLBRffvmliIyMFJMmTVLcfBoaGsTo0aNFv379xNdff+3wXVFfX6/I+VxvTs78/GxtIdwzJxZnGa1cuVL0799f+Pn5ifj4eLFr1y65U2oVAE5fb731lhRz8eJF8cc//lH07NlT3HTTTeLhhx8WJ06ckC9pF/28OHvjfD766CMRHR0t1Gq1iIqKEmvXrnVYbrfbxeLFi0VISIhQq9Vi1KhR4sCBAzJle31Wq1XMmTNH9O/fX/j7+4uBAweKhQsXOnzRK31On332mdP/d6ZOnSqEaF3+Z86cEZMmTRLdu3cXgYGBYtq0aeLcuXMyzOba86moqLjqd8Vnn32myPlcb07OOCvO7pgTHxlJRESkMOw5ExERKQyLMxERkcKwOBMRESkMizMREZHCsDgTEREpDIszERGRwrA4ExERKQyLMxERkcKwOBMRAODUqVN4/PHH0b9/f6jVami1WphMJnz11VcAAJVKhZycHHmTJOok+MhIIgIAjB07Fg0NDXj77bcxcOBAVFdXIz8/H2fOnJE7NaJOh7fvJCLU1NSgZ8+e+PzzzzFixIgWyyMiInD06FHp/YABA3DkyBEAwAcffIAXXngB+/btk56tvHDhQvj6XvrdX6VS4fXXX8eHH36Izz//HKGhocjKysIjjzzSLnMj8kY8rE1E6N69O7p3746cnBzU19e3WL57924AwFtvvYUTJ05I77/44gtMmTIFc+bMwb59+7BmzRqsW7cOL730ksP6ixcvxtixY7Fnzx4kJydj4sSJ+O677zw/MSIvxT1nIgIA/Otf/8L06dNx8eJFDBs2DCNGjMDEiRNx++23A7i0B/z+++8jKSlJWsdoNGLUqFGYP3++NPaPf/wD8+bNQ1VVlbTerFmz8MYbb0gxd955J4YNG4bXX3+9fSZH5GW450xEAC71nKuqqvDhhx/i3nvvxeeff45hw4Zh3bp1V11nz549ePHFF6U97+7du2P69Ok4ceIELly4IMUZDAaH9QwGA/ecia6BJ4QRkcTf3x9333037r77bixevBh/+MMfkJGRgUcffdRp/Pnz5/HCCy9gzJgxTrdFRG3DPWciuqpbb70VtbW1AICuXbvCZrM5LB82bBgOHDiAm2++ucXLx+fy18uuXbsc1tu1axduueUWz0+AyEtxz5mIcObMGYwbNw6PPfYYbr/9dvTo0QPFxcXIysrCQw89BODSGdv5+fm46667oFar0bNnT6Snp+OBBx5A//798cgjj8DHxwd79uxBeXk5lixZIm1/8+bNiIuLQ0JCAt555x0UFRXhzTfflGu6RIrHE8KICPX19Xj++efx73//Gz/88AMaGxsRHh6OcePGYcGCBQgICMBHH32EtLQ0HDlyBH379pUupfrkk0/w4osvoqysDF27dkVUVBT+8Ic/YPr06QAunRC2atUq5OTkYMeOHQgNDcXSpUsxfvx4GWdMpGwszkTkUc7O8iaia2PPmYiISGFYnImIiBSGJ4QRkUexc0bkOu45ExERKQyLMxERkcKwOBMRESkMizMREZHCsDgTEREpDIszERGRwrA4ExERKQyLMxERkcKwOBMRESnM/wesvY0wHgcfQgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "#plt.tight_layout(); plt.savefig(\"1.pdf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3996b6-3f7a-420a-8584-c5760249f3d8",
   "metadata": {},
   "source": [
    "## D.2 Cosine decay"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5216214-de79-40cf-a733-b1049a73023c",
   "metadata": {},
   "source": [
    "- Another popular technique for training complex deep neural networks is cosine decay, which also adjusts the learning rate across training epochs\n",
    "- In cosine decay, the learning rate follows a cosine curve, decreasing from its initial value to near zero following a half-cosine cycle\n",
    "- This gradual reduction is designed to slow the pace of learning as the model begins to improve its weights; it reduces the risk of overshooting minima as the training progresses,  which is crucial for stabilizing the training in its later stages\n",
    "- Cosine decay is often preferred over linear decay for its smoother transition in learning rate adjustments, but linear decay is also used in practice (for example, [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4e8d2068-a057-4abf-b478-f02cc37191f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "min_lr = 0.1 * initial_lr\n",
    "track_lrs = []\n",
    "\n",
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "global_step = -1\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "        if global_step < warmup_steps:\n",
    "            # Linear warmup\n",
    "            lr = initial_lr + global_step * lr_increment  \n",
    "        else:\n",
    "            # Cosine annealing after warmup\n",
    "            progress = ((global_step - warmup_steps) / \n",
    "                        (total_training_steps - warmup_steps))\n",
    "            lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.param_groups[0][\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0e779e33-8a44-4984-bb23-be0603dc4158",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1563f7390>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAEmCAYAAABYuVhFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABOqElEQVR4nO3deVyU5fr48c8MywzroKAsKopLuYuCImrHForKFsrUPKZmlktaGpallVanfpZli2UuZWqLaXaKzK9RhpmlCIq4pbgkioqAiDAsss08vz+IqTmiAgLPDFzv12texPPcz3Ddalxz7xpFURSEEEIIYTO0agcghBBCCGuSnIUQQggbI8lZCCGEsDGSnIUQQggbI8lZCCGEsDGSnIUQQggbI8lZCCGEsDGSnIUQQggb46h2APbKbDaTnp6Oh4cHGo1G7XCEEEKoQFEU8vPzCQgIQKutu/auJOdaSk9Pp02bNmqHIYQQwgacOnWK1q1b19n7SXKuJQ8PD6DiL8TT01PlaIQQQqjBaDTSpk0bS06oK5Kca6myK9vT01OSsxBCNHF1PbwpE8KEEEIIGyPJWQghhLAxkpyFEEIIG6N6cl60aBHt2rVDr9cTFhZGYmLiFcuvW7eOzp07o9fr6dGjBxs3brS6/80333Dbbbfh7e2NRqNhz549l7xHcXExU6ZMwdvbG3d3d4YOHUpmZmZdVksIIYSoNVWT89q1a4mOjmbu3Lns3r2bXr16ERkZSVZWVpXlt2/fzsiRIxk/fjzJyclERUURFRXFgQMHLGUKCwsZNGgQb7zxxmV/7lNPPcX333/PunXr+PXXX0lPT+f++++v8/oJIYQQtaFRFEVR64eHhYXRt29fPvjgA6BiY482bdrwxBNP8Nxzz11SfsSIERQWFrJhwwbLtf79+xMcHMySJUusyp44cYKgoCCSk5MJDg62XM/Ly6NFixasXr2aBx54AICUlBS6dOlCfHw8/fv3r1bsRqMRg8FAXl6ezNYWQogmqr5ygWpLqUpLS0lKSmLWrFmWa1qtloiICOLj46t8Jj4+nujoaKtrkZGRxMTEVPvnJiUlUVZWRkREhOVa586dCQwMvGJyLikpoaSkxPK90Wis9s9sKhRF4fmYA+xJy0XnpEXnqEXn6ICzoxY3Zwe83XV4uzvj466jhbuOFh46Ar1d8dQ7qR26EELYFNWSc3Z2NiaTCV9fX6vrvr6+pKSkVPlMRkZGleUzMjKq/XMzMjJwdnbGy8urRu8zb948Xn755Wr/nKbocGY+qxPSavycj7uO9j5uBPm4EdTCjS7+nvRsZaCZm3M9RCmEELZPNiGpplmzZlm12it3hRF/i//zPAC9A72YPLgDJeVmSsvNlJSbKSwpJ7ughOyC0r++lpBpLLb6PvFEjtX7BTZ3pWdrA71ae9GnrRc9W3vh5KD6HEYhhKh3qiVnHx8fHBwcLpklnZmZiZ+fX5XP+Pn51aj85d6jtLSU3Nxcq9bz1d5Hp9Oh0+mq/XOaosrkfGtXX27rVr2/k/ziMk5kF3E8u4DU7EL+PFfIgTN5pGYXkpZTRFpOERv2nQXAXedI//bNGdjRh0EdfejY0l0OHRFCNEqqJWdnZ2dCQkKIi4sjKioKqJgQFhcXx9SpU6t8Jjw8nLi4OKZPn265tmnTJsLDw6v9c0NCQnByciIuLo6hQ4cCcPjwYdLS0mr0PsKa2ayQkFrR8g1v713t5zz0TvRobaBHa4PV9byiMvafyWPv6Vz2nsol8UQOuUVl/Hwoi58PVczm9/PUc1s3X27v5ke/oOY4SqtaCNFIqNqtHR0dzdixYwkNDaVfv368++67FBYWMm7cOADGjBlDq1atmDdvHgDTpk1j8ODBLFiwgCFDhrBmzRp27drFsmXLLO+Zk5NDWloa6enpQEXihYoWs5+fHwaDgfHjxxMdHU3z5s3x9PTkiSeeIDw8vNoztcWlDmUYybtYhpuzAz1aGa7+wFUYXJ0Y1MmHQZ18gIrkf/Cskd+PZbPtWDaJqTlkGIv5NP4kn8afpLmbM7d28eX2Hn4M6ugj3d9CCLumanIeMWIE586dY86cOWRkZBAcHExsbKxl0ldaWprV+ZgDBgxg9erVvPDCC8yePZtOnToRExND9+7dLWXWr19vSe4ADz74IABz587lpZdeAuCdd95Bq9UydOhQSkpKiIyM5MMPP2yAGjdelV3afeupBavVaujeykD3VgYmDe5AcZmJ+D/P88OBs2w6mElOYSlrd51i7a5T+Lg7ExXcimGhbbjer25PihFCiIag6jpneybrnK09umoXPx/KZNYdnZk4uEOD/uxyk5nE1Bxi/8hg4/4Msgv+XvLWs7WBYSGtuSe4FQYXWbIlhKhb9ZULJDnXkiTnv5nMCsGv/ER+cTnrpw6kZ2sv1WIpM5n59fA51iWdIu5QFuXmin/ers4OPBDSmocHtKN9C3fV4hNCNC6NbhMS0XgcTDeSX1yOh86Rrv7qflBxctAS0dWXiK6+nC8oIWZPOmt3pnEks8AyPn3j9S14ZGAQN3TykdneQgibJMlZXLMdxyvGm21txrS3u47xg4J4ZGA74v88zyfbUolLyWLL4XNsOXyO63zdmXJTR+7qGYCDVpK0EMJ2SHIW1yz+r+Qc3qH6S6gakkajYUBHHwZ09OFEdiErt59g3a5THMksYNqaPbz381Eev6kj9wYHyCxvIYRNkN9E4ppUTsYC6F+D9c1qaefjxkv3dGP7rFuYcet1eLk6cTy7kKfX7eXmBVtYk5hGmcmsdphCiCZOkrO4Jn+kGykoKcdT70gXlceba8Lg4sQTt3Ti92dv5rk7OuPt5sypnIs8981+It/Zyg/7zyJzJYUQapHkLK5JZZd2WHtvuxy3ddc5MmlwB35/9mZevKsr3m7OHM8uZPIXu7nvw+0k/FU/IYRoSJKcxTWp3HzEHrq0r8TF2YHxg4LY8syNPHlLJ1ydHdhzKpcRy3YwfuVOjmXlqx2iEKIJkeQsaq3MZGbXiZrvp23LPPRORN96HVueuZGH+gfioNUQl5LF7e/+xv/beIj84jK1QxRCNAGSnEWt7T+TR2GpCS9XJzo3sm0yW3roeTWqB5ue+hcRXVpSblZYtvU4Ny/4lW+TT8t4tBCiXklyFrVW2aUdFtQcrR2ON1dH+xbufDy2Lyse7ks7b1fO5Zfw1Nq9DFsSz8F0o9rhCSEaKUnOotYqNx9pLF3aV3JT55b8+NS/eCbyelycHNh18gJ3f/A7b8SmUFxmUjs8IUQjI8lZ1EppuZldJy4AEN7BR+VoGobO0YEpN3UkbsZg7ujuh8mssHjLn9zx3m+WXgQhhKgLkpxFrew7ncvFMhPN3Zzp1LJpHSQR4OXC4odCWDo6BF9PHanZhYz8aAfP/XcfeRdlwpgQ4tpJcha1Utml3b994x1vvprIbn5sih7MqLBAANbsPMWtb//KL4ezVI5MCGHvJDmLWolvQuPNV+Kpd+K1+3rw1cRw2vu4kZVfwrgVO3n+2/0UlpSrHZ4Qwk5JchY1VlJusow32/vmI3WlX1BzNk67gXED2wHwRUIady78jaSTOeoGJoSwS5KcRY3tPZVHSbkZH3cdHZvYePOV6J0cmHt3N1Y/GkaAQc/J80UMWxLPG7EplJbLYRpCiOqT5Cxq7O8tO5uj0TTN8eYrGdDRh9in/sXQPq0xK7B4y58MW7KdtPNFaocmhLATkpxFjcUfzwakS/tKPPVOLBjeiyUP9cHg4sTe03ncufA31u9NVzs0IYQdkOQsaqS4zMTutFwAwjtIcr6a27v788O0G+jbrhkFJeU8+WUyz369j6JSmSwmhLg8Sc6iRpLTciktN9PSQ0d7Hze1w7ELAV4ufPlYf568uSMaDazddYp7PtjG4Qw56UoIUTVJzqJG4o//fUSkjDdXn6ODlujbrueLR8No6aHjWFYBUYu2EZN8Ru3QhBA2SJKzqBHLftrSpV0rAzr48MO0G7ihkw8Xy0xMX7uHud8dkNncQggrkpxFtRWXmdhTOd4sk8Fqzdtdx8px/Xji5o4ArIo/yYPL4jmbd1HlyIQQtkKSs6i2pJMXKDWZ8fPU09bbVe1w7JqDVsOM265n+dhQPPWO7E7L5a6Fv7P9WLbaoQkhbIAkZ1Ft/+zSlvHmunFLF182PHEDXf09OV9YyuhPElm5LRVFUdQOTQihIknOotoqNx+RLu26FejtyjePD+D+3q0wmRVe+v4gs77ZL+PQQjRhkpxFtRSVlrP3dC4gm4/UB72TAwuG9+L5O7ug1VSccPXvj3ZwLr9E7dCEECqQ5CyqJenkBcpMCq28XGjT3EXtcBoljUbDY/9qzycP98VD78iukxe454PfOXAmT+3QhBANTJKzqJa/99OW8eb6duP1LYmZMpD2Ldw4m1fMA0u2E3sgQ+2whBANSPXkvGjRItq1a4derycsLIzExMQrll+3bh2dO3dGr9fTo0cPNm7caHVfURTmzJmDv78/Li4uREREcPToUasyR44c4d5778XHxwdPT08GDRrEL7/8Uud1a0z+3nykucqRNA0dWrjz7eMDGXxdC4rLzEz+Ionlv8tEMSGaClWT89q1a4mOjmbu3Lns3r2bXr16ERkZSVZWVpXlt2/fzsiRIxk/fjzJyclERUURFRXFgQMHLGXmz5/PwoULWbJkCQkJCbi5uREZGUlxcbGlzF133UV5eTmbN28mKSmJXr16cdddd5GRIa2TqhSWlLPvdEXXqmw+0nAMLk4sHxvKqLBAFAX+s+EgL63/A5NZErQQjZ6ion79+ilTpkyxfG8ymZSAgABl3rx5VZYfPny4MmTIEKtrYWFhysSJExVFURSz2az4+fkpb775puV+bm6uotPplC+//FJRFEU5d+6cAihbt261lDEajQqgbNq0qdqx5+XlKYCSl5dX7Wfs1S8pmUrbZzcog96IUzuUJslsNitLfz2mtH12g9L22Q3KIysSlYLiMrXDEkIo9ZcLVGs5l5aWkpSUREREhOWaVqslIiKC+Pj4Kp+Jj4+3Kg8QGRlpKZ+amkpGRoZVGYPBQFhYmKWMt7c3119/PZ9++imFhYWUl5ezdOlSWrZsSUhIyGXjLSkpwWg0Wr2aCkuXdpC0mtWg0WiY8K8OfDiqDzpHLXEpWYxYFk+WsfjqDwsh7JJqyTk7OxuTyYSvr6/VdV9f38t2L2dkZFyxfOXXK5XRaDT8/PPPJCcn4+HhgV6v5+233yY2NpZmzZpdNt558+ZhMBgsrzZt2tSswnZsx/EcQLq01XZnD3++nNAfbzdnDpwxErVITrYSorFSfUJYQ1MUhSlTptCyZUt+++03EhMTiYqK4u677+bs2bOXfW7WrFnk5eVZXqdOnWrAqNWTX1xmWcoj65vV1yewGd8+XjGTOz2vmAcWb+e3o+fUDksIUcdUS84+Pj44ODiQmZlpdT0zMxM/P78qn/Hz87ti+cqvVyqzefNmNmzYwJo1axg4cCB9+vThww8/xMXFhVWrVl02Xp1Oh6enp9WrKdh5IgeTWaGttysBXrK+2RYEervyzeQB9AtqTn5JOeNW7OS7PXL0pBCNiWrJ2dnZmZCQEOLi4izXzGYzcXFxhIeHV/lMeHi4VXmATZs2WcoHBQXh5+dnVcZoNJKQkGApU1RUBFSMb/+TVqvFbJbtEv+XpUtbWs02xcvVmc/G9+OeXgGUmxWmr93Dym2paoclhKgjqnZrR0dH89FHH7Fq1SoOHTrE5MmTKSwsZNy4cQCMGTOGWbNmWcpPmzaN2NhYFixYQEpKCi+99BK7du1i6tSpQMV48vTp03n11VdZv349+/fvZ8yYMQQEBBAVFQVUJPhmzZoxduxY9u7dy5EjR3jmmWdITU1lyJAhDf5nYOss+2nLeLPN0Tk68O6IYB4e0A5FgZe+P8jbPx2WtdBCNAKOav7wESNGcO7cOebMmUNGRgbBwcHExsZaJnSlpaVZtXAHDBjA6tWreeGFF5g9ezadOnUiJiaG7t27W8rMnDmTwsJCJkyYQG5uLoMGDSI2Nha9Xg9UdKfHxsby/PPPc/PNN1NWVka3bt347rvv6NWrV8P+Adi4vItl/JEu4822TKvVMPfurni7ObNg0xEWbj7G+cJSXrm3Ow5a2clNCHulUeRjdq0YjUYMBgN5eXmNdvz554OZPPrpLtr7uLH56RvVDkdcxRcJJ3kh5gCKAnf28OOdEcHoHB3UDkuIRq2+ckGTm60tqs+yvlm6tO3CqLC2LPp3H5wdtGzcn8EjK3dSUFKudlhCiFqQ5Cwu65+HXQj7cGcPf1aM64ubswPbjp1n5LIdnC+QYyeFsDeSnEWVcotKOZRRsQuaHHZhXwZ29OHLCf1p7ubM/jN5DFsaT0ae7CYmhD2R5CyqlJCag6JAx5butPTQqx2OqKGerb34elI4rbxcOH6ukOFL4zmVU6R2WEKIapLkLKr0d5e2tJrtVfsW7qyd2J+23q6k5RQxfGk8qdmFaoclhKgGSc6iSjv+mgwW3t5H5UjEtWjdzJWvJobToYUbZ/OKGb40niOZsh+3ELZOkrO4RE5hKSl/HaggLWf75+upZ+3EcDr7eXAuv4QHl+2w7JcuhLBNkpzFJRL+ajVf7+uBt7tO5WhEXfBx17FmQn96tjaQU1jKvz/aQXLaBbXDEkJchiRncYnKLm1pNTcuXq7OfP5oGKFtm2EsLuehjxMsH8SEELZFkrO4ROXmI7KfduPjqXdi1SP9GNDBm8JSE2NXJMqRk0LYIEnOwkp2QQlHMgsACAuS5NwYuekc+eThvtx4fQuKy8w8umoXW49IghbCltQqOZeXl/Pzzz+zdOlS8vMrJg6lp6dTUFBQp8GJhpfw1xGRnf08aObmrHI0or7onRxYOjqEiC6+lJSbeexTSdBC2JIaJ+eTJ0/So0cP7r33XqZMmcK5cxX/Q7/xxhs8/fTTdR6gaFjxx7MB6dJuCnSODnw4qg8RXVpKghbCxtQ4OU+bNo3Q0FAuXLiAi4uL5fp9991HXFxcnQYnGp7l/GbZT7tJcHbU8uGoEKsELWPQQqivxsn5t99+44UXXsDZ2brLs127dpw5c6bOAhMNLyu/mD/PFaLRyHhzU/K/CfrRVZKghVBbjZOz2WzGZDJdcv306dN4eHjUSVBCHTv+Gm/u6u+JwdVJ5WhEQ5IELYRtqXFyvu2223j33Xct32s0GgoKCpg7dy533nlnXcYmGph0aTdtkqCFsB01Ts4LFixg27ZtdO3aleLiYv79739burTfeOON+ohRNJCE43J+c1NXVYL+/Wi22mEJ0eRoFEVRavpQeXk5a9euZe/evRQUFNCnTx9GjRplNUGssTMajRgMBvLy8vD09FQ7nGuWaSwm7P/FodXAnrm34amXbu2mrLTczONfJPHzoSz0TlpWjetHmHxoE+IS9ZULapyct27dyoABA3B0dLS6Xl5ezvbt2/nXv/5VZ8HZssaWnGOSzzB97R56tjawfuogtcMRNqC03MzEz3bxy+FzuDk78NmjYfQJbKZ2WELYlPrKBTXu1r7pppvIycm55HpeXh433XRTnQQlGt4O6dIW/8PZUcvih0L+3urzk0Q5zUqIBlLj5KwoChqN5pLr58+fx83NrU6CEg3Psp+2JGfxD3onBz4eG0rfds3ILy5n9PIEDmfIedBC1DfHqxepcP/99wMVs7MffvhhdLq/jxI0mUzs27ePAQMG1H2Eot6l517k5PkiHLQaQttJt6Ww5upcsRf3Q8sT2Xsql1Ef72DtxHA6tHBXOzQhGq1qt5wNBgMGgwFFUfDw8LB8bzAY8PPzY8KECXz++ef1GauoJ5Vd2t1bGfCQiWCiCh56Jz4d14+u/p5kF5Qy6qME0s4XqR2WEI1WtVvOK1asACp2Anv66aelC7sRkfXNojoMrk58Nr4fDy7bwdGsAv798Q6+mhhOgFfTWaUhREOp8Zjz3LlzJTE3MnJ+s6gub3cdXzwaRjtvV05fuMi/P9pBlrFY7bCEaHSq3XL+p6+//pqvvvqKtLQ0SktLre7t3r27TgITDeNUThGnL1zEUashtK2MN4ura+mpZ/Vj/Rm+NJ4T54sY9XECayb0x9tdd/WHhRDVUuOW88KFCxk3bhy+vr4kJyfTr18/vL29OX78OHfccUd9xCjqUeV4c8/WBtx0tfqsJpqgAC8XvnysP36eeo5mFfDQ8kRyi0qv/qAQolpqnJw//PBDli1bxvvvv4+zszMzZ85k06ZNPPnkk+TlyRpIeyNd2qK22jR3ZfVjYfi46zh01sjYTxLJLy5TOywhGoUaJ+e0tDTLkikXFxfy8yvWPI4ePZovv/yybqMT9UpRFBL+OolKNh8RtdG+hTtfPBpGM1cn9p7O47FPd1FcdumpdUKImqlxcvbz87PsEBYYGMiOHTsASE1NpRbbdAsVncq5yJncizg5aAht21ztcISdut7Pg8/Gh+Guc2TH8Rymrt5NmcmsdlhC2LUaJ+ebb76Z9evXAzBu3Dieeuopbr31VkaMGMF9991X4wAWLVpEu3bt0Ov1hIWFkZiYeMXy69ato3Pnzuj1enr06MHGjRut7iuKwpw5c/D398fFxYWIiAiOHj16yfv83//9H2FhYbi4uNCsWTOioqJqHLu9iz9ecdpQcBsvXJwdVI5G2LPurQwsHxuKzlHLz4eymPn1Psxm+bAuRG3VODkvW7aM559/HoApU6bwySef0KVLF1555RUWL15co/dau3Yt0dHRzJ07l927d9OrVy8iIyPJysqqsvz27dsZOXIk48ePJzk5maioKKKiojhw4IClzPz581m4cCFLliwhISEBNzc3IiMjKS7+e7nHf//7X0aPHs24cePYu3cv27Zt49///ndN/yjs3g7p0hZ1KKy9N4sf6oOjVsO3yWd46fs/pDdNiNpSaqCsrEx5+eWXlVOnTtXkscvq16+fMmXKFMv3JpNJCQgIUObNm1dl+eHDhytDhgyxuhYWFqZMnDhRURRFMZvNip+fn/Lmm29a7ufm5io6nU758ssvLXVo1aqV8vHHH19T7Hl5eQqg5OXlXdP7qMVsNithr/2stH12g7Lt6Dm1wxGNSEzyaaXdcxuUts9uUN76MUXtcISoV/WVC2rUcnZ0dGT+/PmUl5df84eC0tJSkpKSiIiIsFzTarVEREQQHx9f5TPx8fFW5QEiIyMt5VNTU8nIyLAqYzAYCAsLs5TZvXs3Z86cQavV0rt3b/z9/bnjjjusWt9VKSkpwWg0Wr3s2YnzRWQYi3F20NJH1jeLOnRvcCv+c293AN7ffIyPfzuuckRC2J8ad2vfcsst/Prrr9f8g7OzszGZTPj6+lpd9/X1JSMjo8pnMjIyrli+8uuVyhw/XvGL4qWXXuKFF15gw4YNNGvWjBtvvLHKozArzZs3z2o/8TZt2tSgtrancn1zcKAXeicZbxZ166H+bXkm8noAXv2/Q3y185TKEQlhX2q868Qdd9zBc889x/79+wkJCblkK8977rmnzoKrD2ZzxSzS559/nqFDhwIV+4a3bt2adevWMXHixCqfmzVrFtHR0ZbvjUajXSdo2U9b1LfHb+yA8WIZS7ce57lv9uGud+TOHv5qhyWEXahxcn788ccBePvtty+5p9FoMJmqt8bRx8cHBwcHMjMzra5nZmbi5+dX5TN+fn5XLF/5NTMzE39/f6sywcHBAJbrXbt2tdzX6XS0b9+etLS0y8ar0+msjsm0Z4qiyOYjot5pNBqeu6MzxuIyvkw8xbQ1ybjpHBl8XQu1QxPC5tW4W9tsNl/2Vd3EDODs7ExISAhxcXFW7x0XF0d4eHiVz4SHh1uVB9i0aZOlfFBQEH5+flZljEYjCQkJljIhISHodDoOHz5sKVNWVsaJEydo27ZtteO3Z8ezCzmXX4Kzo5bgNl5qhyMaMY1Gw6tRPRjS058yk8Kkz5JIOnn54SMhRIUaJ+e6FB0dzUcffcSqVas4dOgQkydPprCwkHHjxgEwZswYZs2aZSk/bdo0YmNjWbBgASkpKbz00kvs2rWLqVOnAhW/CKZPn86rr77K+vXr2b9/P2PGjCEgIMCyjtnT05NJkyYxd+5cfvrpJw4fPszkyZMBGDZsWMP+Aaiksks7JLCZjDeLeueg1fDO8GAGX9eCi2UmHl6xk4Pp9j2hUoj6pupJByNGjODcuXPMmTOHjIwMgoODiY2NtUzoSktLQ6v9+/PDgAEDWL16NS+88AKzZ8+mU6dOxMTE0L17d0uZmTNnUlhYyIQJE8jNzWXQoEHExsai1+stZd58800cHR0ZPXo0Fy9eJCwsjM2bN9OsWdOYtSxd2qKhOTtqWfJQCGM+SWDniQuM+SSBdZMGEOQjx88KURWNosguAbVhNBoxGAzk5eXh6empdjjVpigKfV/7meyCUr6aGE6/INm2UzScvItljFy2g4NnjbTycuHryeH4G1zUDkuIWquvXKBqt7ZoeMeyCsguKEXvpKVXG4Pa4YgmxuDixKfj+9Hex40zuRd56OMEcgrlqEkh/pck5yamsks7tG1zdI4y3iwano+7js8eDSPAoOfPc4U8vCKRgpJr39hIiMakxsn5f3fJqnzl5+dTWiqfgG1d5eYj/dtLd7ZQTysvFz57NIzmbs7sO53HBDlqUggrNU7OXl5eNGvW7JKXl5cXLi4utG3blrlz51o2+xC2w2xWLIddyGQwobYOLdxZNa4fbs4ObP/zPNPWJFMuR00KAdQiOa9cuZKAgABmz55NTEwMMTExzJ49m1atWrF48WImTJjAwoULef311+sjXnENjmTlk1NYiouTAz1be6kdjhD0aG3go7GhODtq+fGPTGZ/u19OshKCWiylWrVqFQsWLGD48OGWa3fffTc9evRg6dKlxMXFERgYyGuvvcbs2bPrNFhxbXb8tb45tF0znBxkuoGwDQM6+PD+yN5M/jyJr3adppmrM7Pu7KJ2WEKoqsa/obdv307v3r0vud67d2/LyU+DBg264laYQh2yvlnYqshufrw+tCcAS7ceZ8mvf6ockRDqqnFybtOmDcuXL7/k+vLlyy0HQZw/f77JbOhhL8xmhYTUv8ab5bALYYOGh7bh+b9azK//kMKaRPmAL5quGndrv/XWWwwbNowffviBvn37ArBr1y5SUlL4+uuvAdi5cycjRoyo20jFNUnJyCe3qAw3Zwe6t5L1zcI2Pfav9uQUlbJ4y5/M/nY/Bhcn7pCTrEQTVOPkfM8995CSksLSpUs5cuQIUHGMZExMDO3atQOw7FUtbEdll3bfoOYy3ixs2szI68ktKv3rJKs9eOidGNTJR+2whGhQtdpbOygoSGZj2xk5v1nYi8qTrPIulrFxfwYTPtvF6sf6ywlqokmpVXLOzc0lMTGRrKysS9Yzjxkzpk4CE3XHZFZITK3cfESSs7B9DloN74wIJr94F78dzebhFYl8PSmcji091A5NiAZR4+T8/fffM2rUKAoKCvD09ESj0VjuaTQaSc426NBZI8bicjx0jnQLsJ9DOkTTpnN0YMlDIfz74wT2nsrloY8T+XpyOK2buaodmhD1rsaDjzNmzOCRRx6hoKCA3NxcLly4YHnl5Mgh6raosku7X1BzHGW8WdgRN50jKx/uS6eW7mQYixmzPJHsghK1wxKi3tX4N/WZM2d48skncXWVT6/24u/9tKVLW9ifZm7OfDY+jFZeLhzPrjgoI7+4TO2whKhXNU7OkZGR7Nq1qz5iEfWg3GQmMVX20xb2zc+g57Px/fB2c+bAGSOPrpKDMkTjVuMx5yFDhvDMM89w8OBBevTogZOTk9X9e+65p86CE9fuj3Qj+SXleOod6eIv483CfrVv4c6qR/oxctkOElJzmLo6mSUP9ZGhGtEoaZQa7jKv1V7+fwSNRoPJ1DQ+zRqNRgwGA3l5eXh62m7SW/rrn8z7IYWILr58PDZU7XCEuGY7jp9n7CeJlJSbGdqnNW8+0BOtVnP1B4WoB/WVC2r8kdNsNl/21VQSsz2R/bRFY9O/vTeL/t0HB62G/+4+zWsbD8lJVqLRkf6gRqzMZGan7KctGqGIrr7M/+ugjOW/p/LhFjkoQzQu1RpzXrhwIRMmTECv17Nw4cIrln3yySfrJDBx7Q6cyaOw1ISXqxOd/WTzBtG4DA1pTe7FMv6z4SBv/ngYg4sTD/Vvq3ZYQtSJaiXnd955h1GjRqHX63nnnXcuW06j0UhytiGVXdphQc1lTE40SuMHBZFbVMr7m4/x4ncH8HJ14q6eAWqHJcQ1q1ZyTk1NrfK/hW2T/bRFUxB963XkFJbyRUIaT62tOChj8HUt1A5LiGsiY86NVJnJzK4TFwDoL5PBRCOm0Wh45d7u3NXTnzKTwqTPktiddkHtsIS4JjVe52wymVi5ciVxcXFVHnyxefPmOgtO1N6+07lcLDPR3M2Z6+SwANHIOWg1vD08GGNxOVuPnGPcip18NTGc62WuhbBTNW45T5s2jWnTpmEymejevTu9evWyegnbUNml3b+9jDeLpsHZUcuSh/rQJ9CLvItljF6ewKmcIrXDEqJWatxyXrNmDV999RV33nlnfcQj6ki87KctmiBXZ0c+ebgvI5bu4HBmPg8tT+DrSQNo4aFTOzQhaqTGLWdnZ2c6duxYH7GIOlJSbiLpZMWYm0wGE02Nl6szn47vR5vmLpw8X8SYTxLJuygHZQj7UqsjI9977z3ZkceG7T2VR3GZGR93Zzq2dFc7HCEanK+nns8eCcPHXcehs0YeXbWTi6Wyg6GwHzXu1v7999/55Zdf+OGHH+jWrdslB1988803dRacqJ3K8eaw9t5oNDLeLJqmdj5ufPpIP0Ysi2fniQtMWb2bpaNDcJKDMoQdqPG/Ui8vL+677z4GDx6Mj48PBoPB6iXUV3l+s3Rpi6aua4AnnzzcF52jls0pWcz8eh9ms/T6CdtXo+RcXl7OTTfdxLx581ixYkWVr9pYtGgR7dq1Q6/XExYWRmJi4hXLr1u3js6dO6PX6+nRowcbN260uq8oCnPmzMHf3x8XFxciIiI4evRole9VUlJCcHAwGo2GPXv21Cp+W1JcZiLprzWectiFENC3XXMWP9QHR62Gb5PP8MqGgzIsJ2xejZKzo6MjkyZNoqSkpM4CWLt2LdHR0cydO5fdu3fTq1cvIiMjycrKqrL89u3bGTlyJOPHjyc5OZmoqCiioqI4cOCApcz8+fNZuHAhS5YsISEhATc3NyIjIykuLr7k/WbOnElAQOPZ7i85LZfScjMtPHS093FTOxwhbMLNnX15a1jFUs+V20+wMO6YyhEJcRVKDQ0ePFj59ttva/rYZfXr10+ZMmWK5XuTyaQEBAQo8+bNq7L88OHDlSFDhlhdCwsLUyZOnKgoiqKYzWbFz89PefPNNy33c3NzFZ1Op3z55ZdWz23cuFHp3Lmz8scffyiAkpycXO248/LyFEDJy8ur9jMN4e2fDittn92gPLF6t9qhCGFzVvx+XGn77Aal7bMblFXbU9UORzQC9ZULajwh7PHHH2fGjBmcPn2akJAQ3NysW2c9e/as9nuVlpaSlJTErFmzLNe0Wi0RERHEx8dX+Ux8fDzR0dFW1yIjI4mJiQEq9v7OyMggIiLCct9gMBAWFkZ8fDwPPvggAJmZmTz22GPExMTg6up61VhLSkqsegyMRmO169mQ5PxmIS7v4YFBXCgq4724o8xd/wcGFyfuDW6ldlhCXKLGybkyuf3z9CmNRoOiKGg0Gkym6i9XyM7OxmQy4evra3Xd19eXlJSUKp/JyMiosnxGRoblfuW1y5VRFIWHH36YSZMmERoayokTJ64a67x583j55ZerVS+1FJeZ2JOWC8jmI0JczvSITuQWlbIq/iQzvtqLp4sTN13fUu2whLBS4+TcGE6lev/998nPz7dqsV/NrFmzrFrsRqORNm3a1Ed4tbb75AVKTWb8PPW08756b4AQTZFGo2Hu3d3IvVjGd3vSmfx5Ep+PDyO0XXO1QxPCosbJuW3bujvM3MfHBwcHBzIzM62uZ2Zm4ufnV+Uzfn5+Vyxf+TUzMxN/f3+rMsHBwUDF4Rzx8fHodNZb+oWGhjJq1ChWrVp1yc/V6XSXlLc1/+zSlvXNQlyeVqvhrWG9yLtYxpbD53hk5U7WTgyni7+n2qEJAVzDkZEHDx4kNjaW9evXW71qwtnZmZCQEOLi4izXzGYzcXFxhIeHV/lMeHi4VXmATZs2WcoHBQXh5+dnVcZoNJKQkGAps3DhQvbu3cuePXvYs2ePZSnW2rVree2112pUB1vyz8MuhBBX5uSgZfGoEELbNsNYXM6YTxI5eb5Q7bCEAGrRcj5+/Dj33Xcf+/fvt4w1A5aWWk3GnAGio6MZO3YsoaGh9OvXj3fffZfCwkLGjRsHwJgxY2jVqhXz5s0DKk7FGjx4MAsWLGDIkCGsWbOGXbt2sWzZMksc06dP59VXX6VTp04EBQXx4osvEhAQQFRUFACBgYFWMbi7V2xx2aFDB1q3bl3TPxKbUFRazt7TuQCEt/dRNxgh7ISLswPLH+7LiKXxpGTkM3p5Il9PCqelp17t0EQTV6sjI4OCgsjKysLV1ZU//viDrVu3EhoaypYtW2ocwIgRI3jrrbeYM2cOwcHB7Nmzh9jYWMuErrS0NM6ePWspP2DAAFavXs2yZcvo1asXX3/9NTExMXTv3t1SZubMmTzxxBNMmDCBvn37UlBQQGxsLHp94/0fLunkBcpMCq28XGjT3EXtcISwGwYXJz4d34+23q6k5VQclJFbVKp2WKKJ0yhKzbbK8fHxYfPmzfTs2RODwUBiYiLXX389mzdvZsaMGSQnJ9dXrDbFaDRiMBjIy8vD01P9car5sSl8uOVP7u/TireHB6sdjhB2J+18EQ8s2U5Wfgk9Wxv4/NEwPPVOV39QNGn1lQtq3HI2mUx4eHgAFYk6PT0dqJgodvjw4ToLTNSM7KctxLUJ9Hbli0fDaO7mzL7TeYxbsZPCknK1wxJNVI2Tc/fu3dm7dy8AYWFhzJ8/n23btvHKK6/Qvn37Og9QXF1hSTn7TucBsr5ZiGvRydeDz8b3w1PvSNLJCzy6ahfFZXLUpGh4NU7OL7zwAmazGYBXXnmF1NRUbrjhBjZu3MjChQvrPEBxdTtP5FBuVmjdzIU2zWV9sxDXoluAgU/Hh+GucyT++HkmfJZESbkkaNGwapycIyMjuf/++wHo2LEjKSkpZGdnk5WVxc0331znAYqr23E8B5AubSHqSnAbL1aM64uLkwNbj5xj6upkykxmtcMSTUit1zkfO3aMH3/8kYsXL9K8uayrVZPspy1E3evbrjkfjw3F2VHLpoOZPLV2DyY5C1o0kBon5/Pnz3PLLbdw3XXXceedd1qWOY0fP54ZM2bUeYDiyvKLyzhwRsabhagPAzv6sPShEJwcNGzYd5Znvt6LWRK0aAA1Ts5PPfUUTk5OpKWlWZ3mNGLECGJjY+s0OHF1u05cwGRWaOvtSoCXrG8Woq7d1Lkl74/sg4NWwze7z/DCdweo4QpUIWqsxsn5p59+4o033rhkJ61OnTpx8uTJOgtMVE+8LKESot7d3t2Pt4f3QqOB1Qlp/GfDIUnQol7VODkXFhZWef5xTk6OzR8M0Rj9vZ+2JGch6tO9wa14Y2jFefWfbEvlzR8PS4IW9abGyfmGG27g008/tXyv0Wgwm83Mnz+fm266qU6DE1eWd7GMP9IrxptlMpgQ9W94aBv+c283AD7c8ifvxR1VOSLRWNX44Iv58+dzyy23sGvXLkpLS5k5cyZ//PEHOTk5bNu2rT5iFJexMzUHswLtfdzwlY36hWgQo8PbUVJu5tX/O8S7Px9Fg4ZpEZ3UDks0MrXaIezIkSMMGjSIe++9l8LCQu6//36Sk5Pp0KFDfcQoLqNyvDlMurSFaFCP3tCe5+7oDMA7Px/hvZ+lBS3qVo1bzgAGg4Hnn3/e6trp06eZMGGC5ehGUf92yPpmIVQzaXBFY+T1H1J45+cjANKCFnWm1puQ/K/z58+zfPnyuno7cRW5RaUcPGsEoH972QRGCDVMGtxBWtCiXtRZchYNKyE1B0WBDi3caOkh481CqEUStKgPkpztlHRpC2E7JEGLuibJ2U5Vrm8Ob++jciRCCJAELepWtSeEVZ5EdTm5ubnXGouoppzCUlIy8gEIk/FmIWzG/04SMysK0yM6odFoVI5M2JtqJ2eDwXDV+2PGjLnmgMTVJaZWtJqv83XHx112ZRPClvwzQb8Xd5TiMhPP3dFZErSokWon5xUrVtRnHKIG/u7SlvFmIWzRpMEdcHbQ8sqGgyzdepyLZSZeursbWq0kaFE9MuZshyo3H5H9tIWwXY8MCuL/3dcDjQY+jT/JrG/2y3nQotokOduZ7IISjmQWALIzmBC27t9hgSwY1gutBtbuOkX0V3soN5nVDkvYAUnOdibheA4Anf08aO7mrHI0Qoirub9Pa94f2QdHrYbv9qQzdXUypeWSoMWVSXK2M/HHswHp0hbCngzp6c+Sh0JwdtAS+0cGEz/bRXGZSe2whA2T5GxndvzVcpbNR4SwLxFdfVn+cCh6Jy2/HD7HIyt3UlhSrnZYwkZJcrYjWfnFHMsqQKOBsCBZ3yyEvbmhUwtWjeuHm7MD2/88z6iPE7hQWKp2WMIGSXK2I5Wt5i5+nni5ynizEPYorL03nz8ahperE3tO5TJsaTzpuRfVDkvYGEnOdkT20xaicegd2Ix1E8PxN+g5llXAA4u3cyyrQO2whA2R5GxHdsjmI0I0Gp18Pfh68gDat3AjPa+YYUu2s/dUrtphCRshydlOZBqLOZ5diFYDfWW8WYhGoZWXC19PGkCv1gYuFJUx8qMd/Hb0nNphCRsgydlOVHZpdwswYHBxUjkaIURdae7mzBeP9WdQRx+KSk08snInG/alqx2WUJlNJOdFixbRrl079Ho9YWFhJCYmXrH8unXr6Ny5M3q9nh49erBx40ar+4qiMGfOHPz9/XFxcSEiIoKjR/8+vu3EiROMHz+eoKAgXFxc6NChA3PnzqW01HZnTVr205bxZiEaHXedI8sfDmVIT3/KTApPfJnMZztOqh2WUJHqyXnt2rVER0czd+5cdu/eTa9evYiMjCQrK6vK8tu3b2fkyJGMHz+e5ORkoqKiiIqK4sCBA5Yy8+fPZ+HChSxZsoSEhATc3NyIjIykuLgYgJSUFMxmM0uXLuWPP/7gnXfeYcmSJcyePbtB6lwbf++nLV3aQjRGOkcHFj7Ym4f6B6Io8GLMAd78MQWz7MfdJGkURVH1bz4sLIy+ffvywQcfAGA2m2nTpg1PPPEEzz333CXlR4wYQWFhIRs2bLBc69+/P8HBwSxZsgRFUQgICGDGjBk8/fTTAOTl5eHr68vKlSt58MEHq4zjzTffZPHixRw/frxacRuNRgwGA3l5eXh6eta02jWSnnuRAa9vxkGrYc+cW/HQS7e2EI2Voii8F3eUd3+u6O27u1cAbz7QE72Tg8qRiarUVy5QteVcWlpKUlISERERlmtarZaIiAji4+OrfCY+Pt6qPEBkZKSlfGpqKhkZGVZlDAYDYWFhl31PqEjgzZtfvlVaUlKC0Wi0ejWUyvHm7q0MkpiFaOQ0Gg3TI67jzQd64qjV8P3edEYvl81KmhpVk3N2djYmkwlfX1+r676+vmRkZFT5TEZGxhXLV36tyXseO3aM999/n4kTJ1421nnz5mEwGCyvNm3aXLlydahyvFm6tIVoOoaFtmHVI/3w0Duy88QF7l+8nRPZhWqHJRqI6mPOajtz5gy33347w4YN47HHHrtsuVmzZpGXl2d5nTp1qsFi3JEq65uFaIoGdvThm8kDaOXlQmp2Ifd9uI2kkzlqhyUagKrJ2cfHBwcHBzIzM62uZ2Zm4ufnV+Uzfn5+Vyxf+bU675mens5NN93EgAEDWLZs2RVj1el0eHp6Wr0awukLRZzKuYiDVkPfdtJyFqKp6eTrwbdTBtDTshY6QZZaNQGqJmdnZ2dCQkKIi4uzXDObzcTFxREeHl7lM+Hh4VblATZt2mQpHxQUhJ+fn1UZo9FIQkKC1XueOXOGG2+8kZCQEFasWIFWa5udCJVd2j1bG3DTOaocjRBCDS099KyZ0J9bu/pSWm5m6upkPtxyDJXn84p6pHpGio6O5qOPPmLVqlUcOnSIyZMnU1hYyLhx4wAYM2YMs2bNspSfNm0asbGxLFiwgJSUFF566SV27drF1KlTgb8mU0yfzquvvsr69evZv38/Y8aMISAggKioKODvxBwYGMhbb73FuXPnyMjIuOyYtJosR0RKl7YQTZqrsyNLHgrhkYFBAMyPPcyMr/bKudCNlOpNsREjRnDu3DnmzJlDRkYGwcHBxMbGWiZ0paWlWbVqBwwYwOrVq3nhhReYPXs2nTp1IiYmhu7du1vKzJw5k8LCQiZMmEBubi6DBg0iNjYWvV4PVLS0jx07xrFjx2jdurVVPLb0SVRRFDnsQghh4aDVMOfurrT1duWVDQf5JvkMf2YXsmx0CL6eerXDE3VI9XXO9qoh1jmnnS/iX2/+gpODhr1zb8PVWfXPUkIIG7HtWDZTVu8mt6iMlh46lo0JJbiNl9phNTmNcp2zuLLKVnOv1l6SmIUQVgZ29OG7KQO5ztedrPwShi+N55vdp9UOS9QRSc42LF66tIUQV9DW241vHh9IRJeKiWLRX+3l5e//oMxkVjs0cY0kOdsoRVH+sfmIJGchRNXcdY4sGx3CEzd3BGDFthOMXLaDTGOxypGJayHJ2UadPF9EhrEYZwctIW2bqR2OEMKGabUaZtx2PctGh+Chc2TXyQsMWfib5QO+sD+SnG1UZZd2cKCXbHgvhKiW27r58f0Tg+js50F2QSkPLU9g6a9/2tQqFFE9kpxtlHRpCyFqo52PG98+PpD7e7fCZFaY90MKEz5LIrdIDs6wJ5KcbZDV+mZJzkKIGnJxdmDB8F78J6o7Tg4aNh3M5M73fmPnCdmX215IcrZBx7MLycovwdlRS+9AL7XDEULYIY1Gw+j+bfn28YEE+biRnlfMiKXxLIw7isks3dy2TpKzDars0u4j481CiGvUvZWB758YxP29W2FW4O1NRxj1sczmtnWSnG3Q313aPipHIoRoDNx1jrw9IpgFw3rh6uzAjuM5RL67lY37z6odmrgMSc42pmK8+a/DLmTzESFEHRoa0poNTwyiW4AnuUVlPP7FbqavSSavqEzt0MT/kORsY45lFZBdUILOUUuvNga1wxFCNDLtW7jz7eMDmXpTR7QaiNmTTuS7W/nt6Dm1QxP/IMnZxlR2aYe2a4bOUcabhRB1z9lRy9OR1/P15AG083Ylw1jM6OWJzPnuAIUl5WqHJ5DkbHPiZQmVEKKB9AlsxsZpN/BQ/0AAPo0/yW3vbGXL4SyVIxOSnG2I2fz3eLNsPiKEaAiuzo68GtWDTx/pRysvF87kXuThFTt5au0ecgpl4xK1SHK2IUezCsgpLMXFyYGerb3UDkcI0YT867oW/PTUv3hkYBAaDXybfIaIt3/luz1nZPtPFUhytiHxf2YDFePNzo7yVyOEaFhuOkfm3N2VbyYP4HpfD3IKS5m2Zg8PLU/gaGa+2uE1KZIBbEjleLN0aQsh1NQ7sBnfPzGIGbdeh7Ojlm3HznPHe7/x6oaD5BfLsquGIMnZRpjNCgmpsr5ZCGEbnB21PHFLJ35+ajC3dvWl3Kzw8e+p3PTWr/w36TRm2QK0XklythEpGfnkFpXh5uxAj1ayvlkIYRsCvV35aEwoK8f1JcjHjeyCEmas28t9H26T86LrkSRnGxFvWd/cHCcH+WsRQtiWG69vSez0G3j29s64Ojuw93QeIz/awbgViaRkGNUOr9GRLGAjLPtpS5e2EMJG6RwdmHxjB3595iZG92+Lo1bDL4fPccd7vzHjq72cyb2odoiNhiRnG2AyKyTI5iNCCDvRwkPHf6K6syl6MEN6+KMo8N/dp7nxzV+Y9c1+TuUUqR2i3ZPkbAMOnTViLC7HXedItwBPtcMRQohqCfJxY9GoPsRMGUh4e2/KTApfJqZx41tbeHrdXlKzC9UO0W5JcrYBlV3a/YKa4yjjzUIIOxPcxosvJ/Tnq4nh3NDJB5NZ4euk09yyYAtPfpnM/tN5aododxzVDkBgmfEoXdpCCHvWL6g5n40PY3faBT7YfIzNKVms35vO+r3phLZtxriBQUR285VGSDVIclZZuclMYqrspy2EaDz6BDbjk4f7cuBMHh/9dpz/23eWXScvsOvkBQIMekaHt2N4aGu83XVqh2qzNIpsmlorRqMRg8FAXl4enp61HyfedzqXez7YhqfekeQ5t+Gg1dRhlEIIob5MYzGf7zjJFwlplsM0HLUabunSkmEhbbjx+hZ225quq1zwv6TlrLLKLu1+Qd6SmIUQjZKvp54Zt13PlJs6sn5POp8nnGTf6Tx+/COTH//IxMddx9A+rbg3uBVd/D3QaOR3oSRnlf29n3ZzlSMRQoj6pXdyYHjfNgzv24aUDCNf7zrNt8lnyC4oYenW4yzdepy23q7c3s2P27v70au1F9om2miRbu1aqouujHKTmV4v/0RhqYn/e3IQ3QJk204hRNNSZjLzS0oW/919mi2Hz1FSbrbc8zfouaVLSwZ1bEF4B28MLk4qRlq1+urWtolO/kWLFtGuXTv0ej1hYWEkJiZesfy6devo3Lkzer2eHj16sHHjRqv7iqIwZ84c/P39cXFxISIigqNHj1qVycnJYdSoUXh6euLl5cX48eMpKCio87pdyf4zeRSWmjC4ONHFT9Y3CyGaHicHLbd182Pp6FB2v3grH47qw929AnBzduBsXjGf70hj0udJ9H7lJ6IWbeOtHw+z/Vh2oz8dS/Vu7bVr1xIdHc2SJUsICwvj3XffJTIyksOHD9OyZctLym/fvp2RI0cyb9487rrrLlavXk1UVBS7d++me/fuAMyfP5+FCxeyatUqgoKCePHFF4mMjOTgwYPo9XoARo0axdmzZ9m0aRNlZWWMGzeOCRMmsHr16gare2WXdlhQ8ybbdSOEEJXcdI7c2cOfO3v4U1xmYtuxbLYeOcfvx7L581whe07lsudULh/8cgyNBtr7uNGrtRc9Wxvo2caLji3d8dTbXuu6NlTv1g4LC6Nv37588MEHAJjNZtq0acMTTzzBc889d0n5ESNGUFhYyIYNGyzX+vfvT3BwMEuWLEFRFAICApgxYwZPP/00AHl5efj6+rJy5UoefPBBDh06RNeuXdm5cyehoaEAxMbGcuedd3L69GkCAgKuGndddGWM+SSRrUfOMffurowbGFSr9xBCiKbgbN5Fth07z7Zj2SSm5lx2H28fdx3tfdwI8nEjqIUb/gY9Ldx1+Hjo8HZzppmrc502hhrlbO3S0lKSkpKYNWuW5ZpWqyUiIoL4+Pgqn4mPjyc6OtrqWmRkJDExMQCkpqaSkZFBRESE5b7BYCAsLIz4+HgefPBB4uPj8fLysiRmgIiICLRaLQkJCdx3332X/NySkhJKSkos3xuN134Ky9O3XUf/9s256fpLewiEEEL8zd/gwgMhrXkgpDUA2QUl7Dudy95Teew7ncuBdCPn8kvILqh4JZ7IqfJ9HLQaDC5OuDg54OrsQP/23vwnqntDVqVaVE3O2dnZmEwmfH19ra77+vqSkpJS5TMZGRlVls/IyLDcr7x2pTL/22Xu6OhI8+bNLWX+17x583j55ZerWbPq6dnai56tver0PYUQoinwcddxc2dfbu789+/6/OIyTmQXcTy7gNTsQk5kF5Jp/DthXygqw2RWLGutAdp6u6oR/lWpPuZsL2bNmmXVYjcajbRp00bFiIQQQvyTh96JHq0N9Ghd9cqXMpOZnMJS8i6WUVRqoqikHE8bnAEOKidnHx8fHBwcyMzMtLqemZmJn59flc/4+fldsXzl18zMTPz9/a3KBAcHW8pkZWVZvUd5eTk5OTmX/bk6nQ6dTraaE0IIe+XkoMXXU4+vp17tUK5K1aVUzs7OhISEEBcXZ7lmNpuJi4sjPDy8ymfCw8OtygNs2rTJUj4oKAg/Pz+rMkajkYSEBEuZ8PBwcnNzSUpKspTZvHkzZrOZsLCwOqufEEIIUSuKytasWaPodDpl5cqVysGDB5UJEyYoXl5eSkZGhqIoijJ69Gjlueees5Tftm2b4ujoqLz11lvKoUOHlLlz5ypOTk7K/v37LWVef/11xcvLS/nuu++Uffv2Kffee68SFBSkXLx40VLm9ttvV3r37q0kJCQov//+u9KpUydl5MiR1Y47Ly9PAZS8vLw6+FMQQghhj+orF6g+5jxixAjOnTvHnDlzyMjIIDg4mNjYWMuErrS0NLTavxv4AwYMYPXq1bzwwgvMnj2bTp06ERMTY1njDDBz5kwKCwuZMGECubm5DBo0iNjYWMsaZ4AvvviCqVOncsstt6DVahk6dCgLFy5suIoLIYQQl6H6Omd7VV9r24QQQtiPRr19pxBCCCH+JslZCCGEsDGSnIUQQggbo/qEMHtVOVRfF9t4CiGEsE+VOaCup29Jcq6l/Px8ANklTAghBPn5+RgMVe9MVhsyW7uWzGYz6enpeHh4oNHU7oSTyi1AT5061WhmfEudbF9jqw80vjo1tvpA46tTZX3S0tLQaDQEBARYLfu9VtJyriWtVkvr1q3r5L08PT0bxT/Wf5I62b7GVh9ofHVqbPWBxlcng8FQL/WRCWFCCCGEjZHkLIQQQtgYSc4q0ul0zJ07t1GddiV1sn2NrT7Q+OrU2OoDja9O9V0fmRAmhBBC2BhpOQshhBA2RpKzEEIIYWMkOQshhBA2RpKzEEIIYWMkOato0aJFtGvXDr1eT1hYGImJiWqHVC3z5s2jb9++eHh40LJlS6Kiojh8+LBVmeLiYqZMmYK3tzfu7u4MHTqUzMxMlSKumddffx2NRsP06dMt1+yxPmfOnOGhhx7C29sbFxcXevTowa5duyz3FUVhzpw5+Pv74+LiQkREBEePHlUx4iszmUy8+OKLBAUF4eLiQocOHfjPf/5jtaexrddp69at3H333QQEBKDRaIiJibG6X534c3JyGDVqFJ6ennh5eTF+/HgKCgoasBZ/u1J9ysrKePbZZ+nRowdubm4EBAQwZswY0tPTrd7DluoDV/87+qdJkyah0Wh49913ra7XRZ0kOatk7dq1REdHM3fuXHbv3k2vXr2IjIwkKytL7dCu6tdff2XKlCns2LGDTZs2UVZWxm233UZhYaGlzFNPPcX333/PunXr+PXXX0lPT+f+++9XMerq2blzJ0uXLqVnz55W1+2tPhcuXGDgwIE4OTnxww8/cPDgQRYsWECzZs0sZebPn8/ChQtZsmQJCQkJuLm5ERkZSXFxsYqRX94bb7zB4sWL+eCDDzh06BBvvPEG8+fP5/3337eUsfU6FRYW0qtXLxYtWlTl/erEP2rUKP744w82bdrEhg0b2Lp1KxMmTGioKli5Un2KiorYvXs3L774Irt37+abb77h8OHD3HPPPVblbKk+cPW/o0rffvstO3bsICAg4JJ7dVInRaiiX79+ypQpUyzfm0wmJSAgQJk3b56KUdVOVlaWAii//vqroiiKkpubqzg5OSnr1q2zlDl06JACKPHx8WqFeVX5+flKp06dlE2bNimDBw9Wpk2bpiiKfdbn2WefVQYNGnTZ+2azWfHz81PefPNNy7Xc3FxFp9MpX375ZUOEWGNDhgxRHnnkEatr999/vzJq1ChFUeyvToDy7bffWr6vTvwHDx5UAGXnzp2WMj/88IOi0WiUM2fONFjsVfnf+lQlMTFRAZSTJ08qimLb9VGUy9fp9OnTSqtWrZQDBw4obdu2Vd555x3Lvbqqk7ScVVBaWkpSUhIRERGWa1qtloiICOLj41WMrHby8vIAaN68OQBJSUmUlZVZ1a9z584EBgbadP2mTJnCkCFDrOIG+6zP+vXrCQ0NZdiwYbRs2ZLevXvz0UcfWe6npqaSkZFhVSeDwUBYWJjN1mnAgAHExcVx5MgRAPbu3cvvv//OHXfcAdhnnf6pOvHHx8fj5eVFaGiopUxERARarZaEhIQGj7mm8vLy0Gg0eHl5AfZZH7PZzOjRo3nmmWfo1q3bJffrqk5y8IUKsrOzMZlM+Pr6Wl339fUlJSVFpahqx2w2M336dAYOHEj37t0ByMjIwNnZ2fI/YCVfX18yMjJUiPLq1qxZw+7du9m5c+cl9+yxPsePH2fx4sVER0cze/Zsdu7cyZNPPomzszNjx461xF3Vv0FbrdNzzz2H0Wikc+fOODg4YDKZeO211xg1ahSAXdbpn6oTf0ZGBi1btrS67+joSPPmzW2+jsXFxTz77LOMHDnSclCEPdbnjTfewNHRkSeffLLK+3VVJ0nO4ppMmTKFAwcO8Pvvv6sdSq2dOnWKadOmsWnTJvR6vdrh1Amz2UxoaCj/7//9PwB69+7NgQMHWLJkCWPHjlU5utr56quv+OKLL1i9ejXdunVjz549TJ8+nYCAALutU1NRVlbG8OHDURSFxYsXqx1OrSUlJfHee++xe/fuWh8VXF3Sra0CHx8fHBwcLpntm5mZiZ+fn0pR1dzUqVPZsGEDv/zyi9XxmX5+fpSWlpKbm2tV3lbrl5SURFZWFn369MHR0RFHR0d+/fVXFi5ciKOjI76+vnZVHwB/f3+6du1qda1Lly6kpaUBWOK2p3+DzzzzDM899xwPPvggPXr0YPTo0Tz11FPMmzcPsM86/VN14vfz87tk0mh5eTk5OTk2W8fKxHzy5Ek2bdpkdbyivdXnt99+Iysri8DAQMvvipMnTzJjxgzatWsH1F2dJDmrwNnZmZCQEOLi4izXzGYzcXFxhIeHqxhZ9SiKwtSpU/n222/ZvHkzQUFBVvdDQkJwcnKyqt/hw4dJS0uzyfrdcsst7N+/nz179lheoaGhjBo1yvLf9lQfgIEDB16yvO3IkSO0bdsWgKCgIPz8/KzqZDQaSUhIsNk6FRUVXXKYvYODA2azGbDPOv1TdeIPDw8nNzeXpKQkS5nNmzdjNpsJCwtr8JivpjIxHz16lJ9//hlvb2+r+/ZWn9GjR7Nv3z6r3xUBAQE888wz/Pjjj0Ad1qn289jEtVizZo2i0+mUlStXKgcPHlQmTJigeHl5KRkZGWqHdlWTJ09WDAaDsmXLFuXs2bOWV1FRkaXMpEmTlMDAQGXz5s3Krl27lPDwcCU8PFzFqGvmn7O1FcX+6pOYmKg4Ojoqr732mnL06FHliy++UFxdXZXPP//cUub1119XvLy8lO+++07Zt2+fcu+99ypBQUHKxYsXVYz88saOHau0atVK2bBhg5Kamqp88803io+PjzJz5kxLGVuvU35+vpKcnKwkJycrgPL2228rycnJltnL1Yn/9ttvV3r37q0kJCQov//+u9KpUydl5MiRNlef0tJS5Z577lFat26t7Nmzx+p3RUlJiU3W52p1qsr/ztZWlLqpkyRnFb3//vtKYGCg4uzsrPTr10/ZsWOH2iFVC1Dla8WKFZYyFy9eVB5//HGlWbNmiqurq3LfffcpZ8+eVS/oGvrf5GyP9fn++++V7t27KzqdTuncubOybNkyq/tms1l58cUXFV9fX0Wn0ym33HKLcvjwYZWivTqj0ahMmzZNCQwMVPR6vdK+fXvl+eeft/pFb+t1+uWXX6r8f2fs2LGKolQv/vPnzysjR45U3N3dFU9PT2XcuHFKfn6+CrW5cn1SU1Mv+7vil19+scn6XK1OVakqOddFneTISCGEEMLGyJizEEIIYWMkOQshhBA2RpKzEEIIYWMkOQshhBA2RpKzEEIIYWMkOQshhBA2RpKzEEIIYWMkOQshhBA2RpKzEAKAc+fOMXnyZAIDA9HpdPj5+REZGcm2bdsA0Gg0xMTEqBukEE2EHBkphABg6NChlJaWsmrVKtq3b09mZiZxcXGcP39e7dCEaHJk+04hBLm5uTRr1owtW7YwePDgS+63a9eOkydPWr5v27YtJ06cAOC7777j5Zdf5uDBg5azlZ9//nkcHSs++2s0Gj788EPWr1/Pli1b8Pf3Z/78+TzwwAMNUjch7JF0awshcHd3x93dnZiYGEpKSi65v3PnTgBWrFjB2bNnLd//9ttvjBkzhmnTpnHw4EGWLl3KypUree2116yef/HFFxk6dCh79+5l1KhRPPjggxw6dKj+KyaEnZKWsxACgP/+97889thjXLx4kT59+jB48GAefPBBevbsCVS0gL/99luioqIsz0RERHDLLbcwa9Ysy7XPP/+cmTNnkp6ebnlu0qRJLF682FKmf//+9OnThw8//LBhKieEnZGWsxACqBhzTk9PZ/369dx+++1s2bKFPn36sHLlyss+s3fvXl555RVLy9vd3Z3HHnuMs2fPUlRUZCkXHh5u9Vx4eLi0nIW4ApkQJoSw0Ov13Hrrrdx66628+OKLPProo8ydO5eHH364yvIFBQW8/PLL3H///VW+lxCidqTlLIS4rK5du1JYWAiAk5MTJpPJ6n6fPn04fPgwHTt2vOSl1f7962XHjh1Wz+3YsYMuXbrUfwWEsFPSchZCcP78eYYNG8YjjzxCz5498fDwYNeuXcyfP597770XqJixHRcXx8CBA9HpdDRr1ow5c+Zw1113ERgYyAMPPIBWq2Xv3r0cOHCAV1991fL+69atIzQ0lEGDBvHFF1+QmJjI8uXL1aquEDZPJoQJISgpKeGll17ip59+4s8//6SsrIw2bdowbNgwZs+ejYuLC99//z3R0dGcOHGCVq1aWZZS/fjjj7zyyiskJyfj5ORE586defTRR3nssceAiglhixYtIiYmhq1bt+Lv788bb7zB8OHDVayxELZNkrMQol5VNctbCHFlMuYshBBC2BhJzkIIIYSNkQlhQoh6JSNnQtSctJyFEEIIGyPJWQghhLAxkpyFEEIIGyPJWQghhLAxkpyFEEIIGyPJWQghhLAxkpyFEEIIGyPJWQghhLAxkpyFEEIIG/P/Ac1/dd9vNGhpAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "#plt.tight_layout(); plt.savefig(\"2.pdf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7512808-b48d-4146-86a1-5931b1e3aec1",
   "metadata": {},
   "source": [
    "## D.3 Gradient clipping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a74f76-8d2b-4974-a03c-d645445cdc21",
   "metadata": {},
   "source": [
    "- Gradient clipping is yet another technique used to stabilize the training when training LLMs\n",
    "- By setting a threshold, gradients exceeding this limit are scaled down to a maximum magnitude to ensure that the updates to the model's parameters during backpropagation remain within a manageable range\n",
    "- For instance, using the `max_norm=1.0` setting in PyTorch's `clip_grad_norm_` method means that the norm of the gradients is clipped such that their maximum norm does not exceed 1.0\n",
    "- the \"norm\" refers to a measure of the gradient vector's length (or magnitude) in the parameter space of the model\n",
    "- Specifically, it's the L2 norm, also known as the Euclidean norm\n",
    "- Mathematically, for a vector $\\mathbf{v}$ with components $\\mathbf{v} = [v_1, v_2, \\ldots, v_n]$, the L2 norm is defined as:\n",
    "$$\n",
    "\\| \\mathbf{v} \\|_2 = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44838a6-4322-47b2-a935-c00d3a88355f",
   "metadata": {},
   "source": [
    "- The L2 norm is calculated similarly for matrices.\n",
    "- Let's assume our gradient matrix is:\n",
    "$$\n",
    "G = \\begin{bmatrix}\n",
    "1 & 2 \\\\\n",
    "2 & 4\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "- And we want to clip these gradients with a `max_norm` of 1.\n",
    "\n",
    "- First, we calculate the L2 norm of these gradients:\n",
    "$$\n",
    "\\|G\\|_2 = \\sqrt{1^2 + 2^2 + 2^2 + 4^2} = \\sqrt{25} = 5\n",
    "$$\n",
    "\n",
    "- Since $\\|G\\|_2 = 5$ is greater than our `max_norm` of 1, we need to scale down the gradients so that their norm is exactly 1. The scaling factor is calculated as $\\frac{max\\_norm}{\\|G\\|_2} = \\frac{1}{5}$.\n",
    "\n",
    "- Therefore, the scaled gradient matrix $G'$ will be as follows:\n",
    "$$\n",
    "G' = \\frac{1}{5} \\times G = \\begin{bmatrix}\n",
    "\\frac{1}{5} & \\frac{2}{5} \\\\\n",
    "\\frac{2}{5} & \\frac{4}{5}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeb0c3c1-2cff-46f5-8127-24412184428c",
   "metadata": {},
   "source": [
    "- Let's see this in action\n",
    "- First, we initialize a new model and calculate the loss for a training batch like we would do in the regular training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e199e1ff-58c4-413a-855e-5edbe9292649",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import calc_loss_batch\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "\n",
    "loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b60f3a-15ec-4846-838d-fdef3df99899",
   "metadata": {},
   "source": [
    "- If we call `.backward()`, PyTorch will calculate the gradients and store them in a `.grad` attribute for each weight (parameter) matrix\n",
    "- Let's define a utility function to calculate the highest gradient based on all model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e70729a3-24d1-411d-a002-2529cd3a8a9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0373)\n"
     ]
    }
   ],
   "source": [
    "def find_highest_gradient(model):\n",
    "    max_grad = None\n",
    "    for param in model.parameters():\n",
    "        if param.grad is not None:\n",
    "            grad_values = param.grad.data.flatten()\n",
    "            max_grad_param = grad_values.max()\n",
    "            if max_grad is None or max_grad_param > max_grad:\n",
    "                max_grad = max_grad_param\n",
    "    return max_grad\n",
    "\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "734f30e6-6b24-4d4b-ae91-e9a4b871113f",
   "metadata": {},
   "source": [
    "- Applying gradient clipping, we can see that the largest gradient is now substantially smaller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fa81ef8b-4280-400f-a93e-5210f3e62ff0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0166)\n"
     ]
    }
   ],
   "source": [
    "torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b62c2af0-dac3-4742-be4b-4292c6753099",
   "metadata": {},
   "source": [
    "## D.4 The modified training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76715332-94ec-4185-922a-75cb420819d5",
   "metadata": {},
   "source": [
    "- Now let's add the three concepts covered above (learning rate warmup, cosine decay, and gradient clipping) to the `train_model_simple` function covered in chapter 5 to create the more sophisticated `train_model` function below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "46eb9c84-a293-4016-a523-7ad726e171e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import evaluate_model, generate_and_print_sample\n",
    "\n",
    "\n",
    "def train_model(model, train_loader, val_loader, optimizer, device, n_epochs,\n",
    "                eval_freq, eval_iter, start_context, warmup_steps=10,\n",
    "                initial_lr=3e-05, min_lr=1e-6):\n",
    "\n",
    "    train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []\n",
    "    tokens_seen, global_step = 0, -1\n",
    "\n",
    "    # Retrieve the maximum learning rate from the optimizer\n",
    "    peak_lr = optimizer.param_groups[0][\"lr\"]\n",
    "\n",
    "    # Calculate the total number of iterations in the training process\n",
    "    total_training_steps = len(train_loader) * n_epochs\n",
    "\n",
    "    # Calculate the learning rate increment during the warmup phase\n",
    "    lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        model.train()\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            global_step += 1\n",
    "\n",
    "            # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "            if global_step < warmup_steps:\n",
    "                # Linear warmup\n",
    "                lr = initial_lr + global_step * lr_increment  \n",
    "            else:\n",
    "                # Cosine annealing after warmup\n",
    "                progress = ((global_step - warmup_steps) / \n",
    "                            (total_training_steps - warmup_steps))\n",
    "                lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n",
    "\n",
    "            # Apply the calculated learning rate to the optimizer\n",
    "            for param_group in optimizer.param_groups:\n",
    "                param_group[\"lr\"] = lr\n",
    "            track_lrs.append(lr)  # Store the current learning rate\n",
    "\n",
    "            # Calculate and backpropagate the loss\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward()\n",
    "\n",
    "            # Apply gradient clipping after the warmup phase to avoid exploding gradients\n",
    "            if global_step > warmup_steps:\n",
    "                torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "            \n",
    "            optimizer.step()\n",
    "            tokens_seen += input_batch.numel()\n",
    "\n",
    "            # Periodically evaluate the model on the training and validation sets\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader,\n",
    "                    device, eval_iter\n",
    "                )\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                track_tokens_seen.append(tokens_seen)\n",
    "                # Print the current losses\n",
    "                print(f\"Ep {epoch+1} (Iter {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
    "\n",
    "        # Generate and print a sample from the model to monitor progress\n",
    "        generate_and_print_sample(\n",
    "            model, train_loader.dataset.tokenizer,\n",
    "            device, start_context\n",
    "        )\n",
    "\n",
    "    return train_losses, val_losses, track_tokens_seen, track_lrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "55fcd247-ba9d-4b93-a757-0f7ce04fee41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Iter 000000): Train loss 10.914, Val loss 10.940\n",
      "Ep 1 (Iter 000005): Train loss 8.903, Val loss 9.313\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 2 (Iter 000010): Train loss 7.362, Val loss 7.789\n",
      "Ep 2 (Iter 000015): Train loss 6.273, Val loss 6.814\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 3 (Iter 000020): Train loss 5.958, Val loss 6.609\n",
      "Ep 3 (Iter 000025): Train loss 5.675, Val loss 6.592\n",
      "Every effort moves you.                                                 \n",
      "Ep 4 (Iter 000030): Train loss 5.607, Val loss 6.565\n",
      "Ep 4 (Iter 000035): Train loss 5.063, Val loss 6.483\n",
      "Every effort moves you, and, and the to the to the to the to the to the, and, and the, and the, and, and, and the, and the, and, and the, and, and, and the, and, and the\n",
      "Ep 5 (Iter 000040): Train loss 4.384, Val loss 6.379\n",
      "Every effort moves you, I was, and I had been.                   \"I, I had the picture, as a little's his pictures, I had been, I was his\n",
      "Ep 6 (Iter 000045): Train loss 4.638, Val loss 6.306\n",
      "Ep 6 (Iter 000050): Train loss 3.690, Val loss 6.196\n",
      "Every effort moves you know the to me a little of his pictures--I had been.  \"I was the's--and, I felt to see a little of his pictures--I had been. \"I of Jack's \"strong. \"I\n",
      "Ep 7 (Iter 000055): Train loss 3.157, Val loss 6.148\n",
      "Ep 7 (Iter 000060): Train loss 2.498, Val loss 6.157\n",
      "Every effort moves you know it was not that, and he was to the fact of the of a and he was--his, the fact of the donkey, in the of the his head to have.   \"I had been his pictures--and by his\n",
      "Ep 8 (Iter 000065): Train loss 2.182, Val loss 6.178\n",
      "Ep 8 (Iter 000070): Train loss 1.998, Val loss 6.193\n",
      "Every effort moves you know,\" was not that my dear, his pictures--so handsome, in a so that he was a year after Jack's resolve had been his painting.     \"Oh, I had the donkey. \"There were, with his\n",
      "Ep 9 (Iter 000075): Train loss 1.824, Val loss 6.211\n",
      "Ep 9 (Iter 000080): Train loss 1.742, Val loss 6.201\n",
      "Every effort moves you know,\" was not that my hostess was \"interesting\": on that point I could have given Miss Croft the fact, and.         \"Oh, as I turned, and down the room, in his\n",
      "Ep 10 (Iter 000085): Train loss 1.285, Val loss 6.234\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the fact with a little: \"Yes--and by me to me to have to see a smile behind his close grayish beard--as if he had the donkey. \"There were days when I\n",
      "Ep 11 (Iter 000090): Train loss 1.256, Val loss 6.236\n",
      "Ep 11 (Iter 000095): Train loss 0.803, Val loss 6.255\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 12 (Iter 000100): Train loss 0.731, Val loss 6.284\n",
      "Ep 12 (Iter 000105): Train loss 0.889, Val loss 6.299\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 13 (Iter 000110): Train loss 0.703, Val loss 6.316\n",
      "Ep 13 (Iter 000115): Train loss 0.517, Val loss 6.315\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 14 (Iter 000120): Train loss 0.594, Val loss 6.324\n",
      "Ep 14 (Iter 000125): Train loss 0.481, Val loss 6.325\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 15 (Iter 000130): Train loss 0.529, Val loss 6.324\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "\n",
    "peak_lr = 5e-4\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=peak_lr, weight_decay=0.1)\n",
    "\n",
    "n_epochs = 15\n",
    "train_losses, val_losses, tokens_seen, lrs = train_model(\n",
    "    model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,\n",
    "    eval_freq=5, eval_iter=1, start_context=\"Every effort moves you\",\n",
    "    warmup_steps=10, initial_lr=1e-5, min_lr=1e-5\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "827e8d5e-0872-4b90-98ac-200c80ee2d53",
   "metadata": {},
   "source": [
    "- Looking at the results above, we can see that the model starts out generating incomprehensible strings of words, whereas, towards the end, it's able to produce grammatically more or less correct sentences\n",
    "- If we were to check a few passages it writes towards the end, we would find that they are contained in the training set verbatim -- it simply memorizes the training data\n",
    "- Note that the overfitting here occurs because we have a very, very small training set, and we iterate over it so many times\n",
    "  - The LLM training here primarily serves educational purposes; we mainly want to see that the model can learn to produce coherent text\n",
    "  - Instead of spending weeks or months on training this model on vast amounts of expensive hardware, we load the pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9decec45-4fdf-4ff6-85a7-1806613f8af7",
   "metadata": {},
   "source": [
    "- A quick check that the learning rate behaves as intended"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d8ebb8d2-8308-4a83-a2a6-730c3bf84452",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEmCAYAAACdy8LUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABP+UlEQVR4nO3deVxU1fsH8M8dhpkBZGZEggEFxNwVN5AJ10wKi3LJUomvuJBYaW6VS7m0WLhkmWa5tGi/zK3UjNQicJdNwBVBTBIVBwScYZF15vz+IG5NojI4MDPM83695sWXe5478xy+xsM595x7OcYYAyGEEEIsisDUCRBCCCHEcFTACSGEEAtEBZwQQgixQFTACSGEEAtEBZwQQgixQFTACSGEEAtEBZwQQgixQFTACSGEEAskNHUCzZlOp0NOTg4cHR3BcZyp0yGEEGICjDEUFxfD3d0dAoHxxs1UwBtRTk4OPDw8TJ0GIYQQM3Dt2jW0adPGaO9HBbwROTo6Aqj5P00qlZo4G0IIIaZQVFQEDw8PviYYCxXwRlQ7bS6VSqmAE0KIlTP2pVRaxEYIIYRYICrghBBCiAWiAk4IIYRYILMo4OvWrUPbtm0hkUigVCqRmJh43/hdu3ahc+fOkEgk8PHxwf79+/XaGWNYvHgx3NzcYGdnh8DAQGRmZurFFBYWIjQ0FFKpFHK5HOHh4SgpKeHb//rrL3Acd9crPj7eeB0nhBBCGsjkBXzHjh2YM2cOlixZgpSUFPTs2RNBQUHIy8urM/7kyZMICQlBeHg4UlNTMXLkSIwcORLnz5/nY1asWIE1a9Zg/fr1SEhIgIODA4KCglBeXs7HhIaG4sKFC4iOjkZUVBSOHj2KiIiIuz7vjz/+wM2bN/mXr6+v8X8IhBBCiKGYifn7+7Np06bx32u1Wubu7s4iIyPrjB8zZgwLDg7WO6ZUKtnUqVMZY4zpdDqmUCjYypUr+Xa1Ws3EYjHbtm0bY4yxtLQ0BoAlJSXxMQcOHGAcx7EbN24wxhjLyspiAFhqamqD+6bRaBgAptFoGvwehBBCLFtj1QKTbiOrrKxEcnIyFixYwB8TCAQIDAxEXFxcnefExcVhzpw5eseCgoKwd+9eAEBWVhZUKhUCAwP5dplMBqVSibi4OIwbNw5xcXGQy+Xw8/PjYwIDAyEQCJCQkIBRo0bxx4cPH47y8nJ07NgRc+fOxfDhw+/Zn4qKClRUVPDfFxUV1e8HYQYSrhQg8kA6tDoGWxsOIqEAtjYCiIUCSCW2kNuL4ORQ+1UEF0cx2rS0h4ujGAIB3WWOEEKamkkLeH5+PrRaLVxdXfWOu7q6Ij09vc5zVCpVnfEqlYpvrz12vxgXFxe9dqFQCCcnJz6mRYsWWLVqFfr37w+BQICffvoJI0eOxN69e+9ZxCMjI/Hee+/Vp+tm5//ir+L0NbXB54lsBGjd0g5tWtrBw8kenVwd0UnhiM4KR8jtRcZPlBBCCAC6kcs9OTs76430+/bti5ycHKxcufKeBXzBggV659TefccSXL9dBgB4/Yn26OYuQ5VWhyqtDhXVOmjKqnD7TiXUpVUovFOJ26WVUBWV46amHJVaHbLyS5GVX3rXe7pKxeiskKKXhxy+Xi3Ry1MOqcS2qbtGCCHNkkkLuLOzM2xsbJCbm6t3PDc3FwqFos5zFArFfeNrv+bm5sLNzU0vplevXnzMfxfJVVdXo7Cw8J6fCwBKpRLR0dH3bBeLxRCLxfdsN2e1BTyomwLdW8vqdU61VgdVUTmuFZbh+u07yMovRYaqGOmqYtxQlyG3qAK5Rbdw5NItAADHAR1dHNHHqyUea+eEAe2d0aqFZf68CCHE1ExawEUiEXx9fRETE4ORI0cCqHmCV0xMDKZPn17nOQEBAYiJicGsWbP4Y9HR0QgICAAAeHt7Q6FQICYmhi/YRUVFSEhIwKuvvsq/h1qtRnJyMr+qPDY2FjqdDkql8p75nj59Wu+PguaivEqL/JKaa/dtWtrV+zyhjQBtWtqjTUt7AK302orLq3AptxgXcoqQmq1G8tXbyC68g4zcYmTkFmNbYjYAoHtrKQZ2eAQDOzjD16slxEIbo/WLEEKaM5NPoc+ZMwcTJkyAn58f/P39sXr1apSWlmLSpEkAgLCwMLRu3RqRkZEAgJkzZ2Lw4MFYtWoVgoODsX37dpw6dQobN24EUHOv2VmzZmHp0qXo0KEDvL29sWjRIri7u/N/JHTp0gXDhg3DlClTsH79elRVVWH69OkYN24c3N3dAQBbtmyBSCRC7969AQC7d+/GN998g6+++qqJf0KNr3b03UIshMzOOFPcjhJb+Ho5wdfLCWE1f1vhVnEFUrJv49RfhTh+uQAXbxbh/I2a15eH/4SjWIgnurhgWDcFBnd6BPYik//zJIQQs2Xy35Bjx47FrVu3sHjxYqhUKvTq1QsHDx7kF6FlZ2frPT+1X79++OGHH7Bw4UK8/fbb6NChA/bu3Yvu3bvzMXPnzkVpaSkiIiKgVqsxYMAAHDx4EBKJhI/ZunUrpk+fjqFDh0IgEGD06NFYs2aNXm4ffPABrl69CqFQiM6dO2PHjh144YUXGvkn0vSu374DoGb03ZjPLX/EUYygbgoEdau5TJFXXI7jmfk4lpmPY5m3kF9SiZ9P5+Dn0zmQ2AowuOMjeMbHDU92daViTggh/8Exxpipk2iuioqKIJPJoNFozPppZN/HX8XCvecR2MUFX03oa5IcdDqG1GtqHDx/EwfOq/hZAQBwENngaR83jO7TBkpvJ9q2RgixKI1VC2hYQ/hiWXMt2zQEAg6+Xi3h69USbz/TBRdyinDwvAr7zuQgu/AOfky+jh+Tr6O13A7P92mNMX4e8HAyXb6EEGJqVMCJ3hS6OeA4Dt1by9C9tQxvPNURyVdv46eU64g6cxM31GVYG3sZnx+6jCc6uWB8gBcGdXiERuWEEKtDBZyYxQj8XjiOg19bJ/i1dcKS57ohOi0XO5Ku4fjlfMSk5yEmPQ9erezxP6UXxvh5QGZP+8wJIdaBCjj5VwE3jxH4vUhsbfBcT3c819Mdf94qwdb4bOxKvoarBXfw4f6L+PSPSwjx98TkAd5oLTfvvhBCyMOiRWyNyBIWsZVXadF50UEAwJnFT1ncCPZOZTX2nc7B5pN/IV1VDAAQCjgM7+mOiMHt0Flhnj93Qoj1oEVspFHUjr4dxUJI7Szvn4O9SIhx/p4Y29cDRzPzseHInzj5ZwF2p97A7tQbGNLpEcwK7IieHnJTp0oIIUZleb+xiVHVLmBr3ch7wBsbx3EY3PERDO74CM5eV2PD0Ss4cO4mDmXcwqGMWxja2QWzn+xY79vEEkKIuRM8OIQ0Z+a8gK2herSRY91LfRD7xuMY3acNBBwQk56HZ9cex5TvTuFCjsbUKRJCyEOjAm7lLGUBW0O0dXbAqjE98cecwRjVuzUEHBCdlovgNccxc3sqrhXeMXWKhBDSYFTArZy57QFvDO0eaYFPx/bC77MHY3jPmnvd/3w6B0NXHcGHv6ZBfafSxBkSQojhqIBbueY4hX4v7V1aYE1Ib0S9PgD927dCpVaHTceyMHjlYWw6egUV1VpTp0gIIfVGBdzKNecp9Hvp3lqG78OV2DypLzorHKEpq8KH+y8i6NOjiE3PffAbEEKIGaACbsXKKv95DriHFYzA/43jODzeyQW/zhiIlS/0gIujGH8V3MHkzacweXMS/sovNXWKhBByX1TArdgNdc31b0vdA24MNgIOL/p5IPbNxzF1cDvY2nCITc/DU58exYqD6SitqDZ1ioQQUicq4Fbs2t/T55a+B9wYWoiFWPB0FxycNQiDOj6CSq0OXxz+E099ehQxF2lanRBifqiAWzFrWsBWX48+0gJbJvXFxvG+aNPSDjfUZQjfcgrTtqYgr6jc1OkRQgiPCrgVs4YtZA3BcRye6qbA77MHYeqgdrARcPj13E0M/eQIvo+/Cp2OHh9ACDE9KuBWzBpXoBvCXiTEgme6YN/0/ujZRobi8mos3HseYzbE4cqtElOnRwixclTArRhNoddPN3cZdr/WH+8+1xUOIhucunobT392DF8duwItjcYJISZCBdyK3aAp9HqzEXCY2N8bv88ZjIEdnFFRrcPSXy9iLI3GCSEmQgXcStXsAa+5hai17QF/GK3ldvhusj8+GuVz12icro0TQpoSFXArxe8Blwghs7c1cTaWheM4vKT0xG+zB2FA+39G46FfJSBHXWbq9AghVoIKuJW6Rte/H1qblvb4v3B/fDiqO+xsbRB3pQDDVh9F1NkcU6dGCLECVMCtFK1ANw6O4xCq9ML+mQPRs40MReXVmP5DKubsOI3i8ipTp0cIacaogFsp2gNuXN7ODvjx1X6Y8UR7CDhgd+oNPP3ZMSRfvW3q1AghzRQVcCtFW8iMz9ZGgDlPdcLOqQHwcLLD9dtlGLshDhuO/EkL3AghRkcF3ErRFHrj8WvrhP0zBmJ4T3dU6xgiD6Tj5e9O4XZppalTI4Q0I1TArRTtAW9cjhJbfDauFyKf94FIKEBseh6eWXMMyVcLTZ0aIaSZoAJuhf69B5ym0BsPx3EI8ffE3tf6o52zA25qyjFmQzzW05Q6IcQIqIBbIb094Ha0B7yxdXWXYt/rAzCilzu0OoZlB9IxeUsSCmlKnRDyEKiAWyHaA970WoiFWD22F5Y97wOxUIDDGbfwDK1SJ4Q8BCrgVuh6IV3/NgWO4zDO3xN7p9VMqauKyjFuYxx+SMg2dWqEEAtkFgV83bp1aNu2LSQSCZRKJRITE+8bv2vXLnTu3BkSiQQ+Pj7Yv3+/XjtjDIsXL4abmxvs7OwQGBiIzMxMvZjCwkKEhoZCKpVCLpcjPDwcJSV1P5Ti8uXLcHR0hFwuf6h+mgtagW5aXdxqptSf8VGgSsvw9p5zWLD7LCqqtaZOjRBiQUxewHfs2IE5c+ZgyZIlSElJQc+ePREUFIS8vLw640+ePImQkBCEh4cjNTUVI0eOxMiRI3H+/Hk+ZsWKFVizZg3Wr1+PhIQEODg4ICgoCOXl5XxMaGgoLly4gOjoaERFReHo0aOIiIi46/OqqqoQEhKCgQMHGr/zJkJ7wE2vhViIdS/1wdxhncBxwLbEaxi7IR4qTfmDTyaEEABgJubv78+mTZvGf6/Vapm7uzuLjIysM37MmDEsODhY75hSqWRTp05ljDGm0+mYQqFgK1eu5NvVajUTi8Vs27ZtjDHG0tLSGACWlJTExxw4cIBxHMdu3Lih995z585l//vf/9i3337LZDKZQX3TaDQMANNoNAad19iGrz3GvOZFsYPnb5o6FcIYO5Sey3q8+xvzmhfFfD+IZolZBaZOiRBiRI1VC0w6Aq+srERycjICAwP5YwKBAIGBgYiLi6vznLi4OL14AAgKCuLjs7KyoFKp9GJkMhmUSiUfExcXB7lcDj8/Pz4mMDAQAoEACQkJ/LHY2Fjs2rUL69atq1d/KioqUFRUpPcyRzSFbl4e7+SCfdP7o7PCEfklFQjZGI/v4v4CY7TVjBBybyYt4Pn5+dBqtXB1ddU77urqCpVKVec5KpXqvvG1Xx8U4+LiotcuFArh5OTExxQUFGDixInYvHkzpFJpvfoTGRkJmUzGvzw8POp1XlO6U1mNglLaA25uvFo5YPdr/fBsDzdU6xgW/3wBb/14FuVVdF2cEFI3k18DN1dTpkzBSy+9hEGDBtX7nAULFkCj0fCva9euNWKGDXPj79E37QE3P/YiIdaG9Mbbz3SGgAN+TL6OlzbFI7+kwtSpEULMkEkLuLOzM2xsbJCbm6t3PDc3FwqFos5zFArFfeNrvz4o5r+L5Kqrq1FYWMjHxMbG4uOPP4ZQKIRQKER4eDg0Gg2EQiG++eabOnMTi8WQSqV6L3NDC9jMG8dxiBj0KL6brIRUIkRKthojPj+BdJV5Xo4hhJiOSQu4SCSCr68vYmJi+GM6nQ4xMTEICAio85yAgAC9eACIjo7m4729vaFQKPRiioqKkJCQwMcEBARArVYjOTmZj4mNjYVOp4NSqQRQc5389OnT/Ov999+Ho6MjTp8+jVGjRhnnB2AC9BhRyzCggzP2TOuPtq3scUNdhtFfnERseu6DTySEWA+jLolrgO3btzOxWMw2b97M0tLSWEREBJPL5UylUjHGGBs/fjybP38+H3/ixAkmFArZxx9/zC5evMiWLFnCbG1t2blz5/iYZcuWMblczn7++Wd29uxZNmLECObt7c3Kysr4mGHDhrHevXuzhIQEdvz4cdahQwcWEhJyzzybyyr0j35NY17zoti7+86bOhVSD4UlFWzshpPMa14U854fxb46doXpdDpTp0UIMUBj1QKhqf+AGDt2LG7duoXFixdDpVKhV69eOHjwIL8ILTs7GwLBPxMF/fr1ww8//ICFCxfi7bffRocOHbB37150796dj5k7dy5KS0sREREBtVqNAQMG4ODBg5BIJHzM1q1bMX36dAwdOhQCgQCjR4/GmjVrmq7jJkJT6JalpYMI301WYvHP57E96Ro+iErD5bwSvD+iG2xtaAkLIdaMY4z2qjSWoqIiyGQyaDQas7kePuLz4zhzXYMN430R1K3udQbE/DDG8PXxLHy4/yIYA/o92gpfhvpCZk8LEQkxd41VC+hPeCtTOwL3oBG4ReE4Di8PbIdN4/3gILLByT8LMOqLE7haUGrq1AghJkIF3Ir8ew94a1rEZpECu7rix1f7obXcDlfyS/H8Fydx+pra1GkRQkyACrgVqd0DLqU94Bati5sUe6b1Q/fWUhSUVmLcxjhEp9EKdUKsDRVwK0IL2JoPF0cJdkQE4PFOj6C8Soep/3cK/xf3l6nTIoQ0ISrgVoT2gDcvDmIhvgrzw7i+HtAxYNHPF7DsQDp0OlqXSog1oAJuRWgE3vwIbQSIfN4HbzzZEQCw/sifmLXjND1bnBArQAXcitBTyJonjuPw+tAOWPViTwgFHPadyUHY14nQ3KkydWqEkEZEBdyKXKMp9GZttG8bfDupL1qIhUjIKsQL608iR11m6rQIIY2ECrgVoSn05m9gh0ew65UAKKQSZOaV4IUvT+JyXomp0yKENAIq4FaitKIahbQH3Cp0cZPip9f6od0jDsjRlOPF9SeRmn3b1GkRQoysQQW8uroaf/zxBzZs2IDi4mIAQE5ODkpK6C99c3VDTXvArUlruR1+fKUferaR4fadKry0KQFHLt0ydVqEECMyuIBfvXoVPj4+GDFiBKZNm4Zbt2p+KSxfvhxvvvmm0RMkxvHPFjKaPrcWTg4i/DDlMQzs4IyyKi3CNyfh59M3TJ0WIcRIDC7gM2fOhJ+fH27fvg07u3+mYkeNGnXXc7qJ+aAV6NbJQSzE1xP64rme7qjWMczcfhrfnsgydVqEECMw+HGix44dw8mTJyESifSOt23bFjdu0F/35ooWsFkvkVCAz8b2gpO9LbbEXcV7v6ShsLQSc57sCI7jTJ0eIaSBDB6B63Q6aLV33yTi+vXrcHR0NEpSxPjoLmzWTSDg8O7wbvwNX9bGXsbbe85DS3dtI8RiGVzAn3rqKaxevZr/nuM4lJSUYMmSJXjmmWeMmRsxIppCJ7U3fPlolA8EHLAtMRuzdpxGlVZn6tQIIQ1g8BT6qlWrEBQUhK5du6K8vBwvvfQSMjMz4ezsjG3btjVGjsQIaAqd1HpJ6QmZnS1mbk/FL2dyUFZZjc9f6gOJrY2pUyOEGMDgAt6mTRucOXMGO3bswJkzZ1BSUoLw8HCEhobqLWoj5oP2gJP/Cu7hBnuRDV75Phl/XMzD5M1J2BTmBwexwb8SCCEmwjHGDLoIdvToUfTr1w9Cof5/6NXV1Th58iQGDRpk1AQtWVFREWQyGTQaDaRSqcnyuJRbjKc+PQqpRIiz7waZLA9ifuL+LMDLW5JQWqlFb085Nk/0h8ye7hNAiDE1Vi0w+Br4kCFDUFhYeNdxjUaDIUOGGCUpYly0B5zcS8CjrbB1ymOQ2dkiNVuNcZvikV9SYeq0CCH1YHABZ4zVufWkoKAADg4ORkmKGBctYCP308tDjh1TH4NzCzEu3izCmA1x9BAUQixAvS94Pf/88wBqVrJOnDgRYrGYb9NqtTh79iz69etn/AzJQ6st4B5ONAIndeuskGLn1Mfwv68ScOVWKV5cH4cfpijh1Yr+KCfEXNV7BC6TySCTycAYg6OjI/+9TCaDQqFAREQEvv/++8bMlTQQ7QEn9dHukRbY9Wo/tG1ljxvqMry4Pg6XcotNnRYh5B7qPQL/9ttvAdTcce3NN9+k6XILQlvISH21ltth5ysBCPs6EemqYozZEIfvJvujRxu5qVMjhPyHwdfAlyxZQsXbwtA1cGIIF0cJtkc8hp4ecqj/fpJZ0l93L1wlhJhWgzZ9/vjjj9i5cyeys7NRWVmp15aSkmKUxIhx0B5w0hByexG2vqxE+OYkJGQVIuzrRHw90Q/9HnU2dWqEkL8ZPAJfs2YNJk2aBFdXV6SmpsLf3x+tWrXClStX8PTTTzdGjuQh1I6+ZXa2kEpofy+pvxZiIbZM9ucfRzrp2yQcpWeKE2I2DC7gX3zxBTZu3Ii1a9dCJBJh7ty5iI6OxowZM6DRaBojR/IQaAEbeRgSWxtsCvPDE51dUFGtw8tbTiE2PdfUaRFC0IACnp2dzW8Xs7OzQ3FxzSrV8ePH073QzRBd/yYPS2Jrg/X/88VTXV1RqdVh6v8l47cLKlOnRYjVM7iAKxQK/k5snp6eiI+PBwBkZWXBwLuykiZAd2EjxiASCrAutA+Ce7ihSsswbWsKfj1709RpEWLVDC7gTzzxBPbt2wcAmDRpEmbPno0nn3wSY8eOxahRo4yeIHk4NAInxmJrI8BnY3thZC93VOsYXt+Wgp9P3zB1WoRYLYNXoW/cuBE6Xc3zg6dNm4ZWrVrh5MmTGD58OKZOnWr0BMnDoT3gxJiENgKsGtMLQhsBfky+/vfzxBle8G1j6tQIsToGjcCrq6uxdOlSqFT/XP8aN24c1qxZg9dffx0ikahBSaxbtw5t27aFRCKBUqlEYmLifeN37dqFzp07QyKRwMfHB/v379drZ4xh8eLFcHNzg52dHQIDA5GZmakXU1hYiNDQUEilUsjlcoSHh6OkpIRvz8jIwJAhQ+Dq6gqJRIJ27dph4cKFqKqqalAfTYUWsRFjsxFwWDG6B15SeoIx4K0fz2BbYrap0yLE6hhUwIVCIVasWIHq6mqjJbBjxw7MmTMHS5YsQUpKCnr27ImgoCDk5eXVGX/y5EmEhIQgPDwcqampGDlyJEaOHInz58/zMStWrMCaNWuwfv16JCQkwMHBAUFBQSgvL+djQkNDceHCBURHRyMqKgpHjx5FREQE325ra4uwsDD8/vvvyMjIwOrVq7Fp0yYsWbLEaH1vbCUV1bh9p+YPDtoDToxJIODw4cjumNivLRgDFuw+h+/i/jJ1WoRYF2ag4cOHs82bNxt62j35+/uzadOm8d9rtVrm7u7OIiMj64wfM2YMCw4O1jumVCrZ1KlTGWOM6XQ6plAo2MqVK/l2tVrNxGIx27ZtG2OMsbS0NAaAJSUl8TEHDhxgHMexGzdu3DPX2bNnswEDBtS7bxqNhgFgGo2m3ucYU/rNIuY1L4r1ePc3k3w+af50Oh1bGnWBec2LYl7zotimo3+aOiVCzE5j1QKDr4E//fTTmD9/Ps6dOwdfX9+7bqs6fPjwer9XZWUlkpOTsWDBAv6YQCBAYGAg4uLi6jwnLi4Oc+bM0TsWFBSEvXv3AqhZDa9SqRAYGMi3y2QyKJVKxMXFYdy4cYiLi4NcLoefnx8fExgYCIFAgISEhDoX412+fBkHDx7kn8pWl4qKClRU/PMs5aKiovv/ABoZTZ+TxsZxHN5+pkvNKvVDf2LprxehYwwRgx41dWqENHsGF/DXXnsNAPDJJ5/c1cZxHLRabb3fKz8/H1qtFq6urnrHXV1dkZ6eXuc5KpWqzvja6/K1Xx8U4+LiotcuFArh5OSkd30fAPr164eUlBRUVFQgIiIC77///j37ExkZiffee++e7U2NVqCTpsBxHN58qhNsBAKsicnER/tr/tulIk5I4zJ4G5lOp7vny5DibSl27NiBlJQU/PDDD/j111/x8ccf3zN2wYIF0Gg0/OvatWtNmOndaA84aSocx2HOkx0xc2gHAMBH+9Ox4cifJs6KkOatQQ8zMRZnZ2fY2NggN1f/1oy5ublQKBR1nqNQKO4bX/s1NzcXbm5uejG9evXiY/67SK66uhqFhYV3fa6HhwcAoGvXrtBqtYiIiMAbb7wBGxubu3ITi8UQi8UP6naToRE4aWqzn+wIjgNW/5GJyAPpYABeGUwjcUIag8EjcGMSiUTw9fVFTEwMf0yn0yEmJgYBAQF1nhMQEKAXDwDR0dF8vLe3NxQKhV5MUVEREhIS+JiAgACo1WokJyfzMbGxsdDpdFAqlffMV6fToaqqit8Hb+5oDzgxhVmBHTE7sCMAYNmBdHx5mEbihDQGk47AAWDOnDmYMGEC/Pz84O/vj9WrV6O0tBSTJk0CAISFhaF169aIjIwEAMycORODBw/GqlWrEBwcjO3bt+PUqVPYuHEjgJqpvFmzZmHp0qXo0KEDvL29sWjRIri7u2PkyJEAgC5dumDYsGGYMmUK1q9fj6qqKkyfPh3jxo2Du7s7AGDr1q2wtbWFj48PxGIxTp06hQULFmDs2LGwtbWMp3rRIjZiKjMDO4DjgE+iL2H5wXQwMLz2eHtTp0VIs2LyAj527FjcunULixcvhkqlQq9evXDw4EF+EVp2djYEgn8mCvr164cffvgBCxcuxNtvv40OHTpg79696N69Ox8zd+5clJaWIiIiAmq1GgMGDMDBgwchkUj4mK1bt2L69OkYOnQoBAIBRo8ejTVr1vDtQqEQy5cvx6VLl8AYg5eXF6ZPn47Zs2c3wU/l4f17DzgVcGIKM4Z2AAdgVfQlrDiYAcaAaUOoiBNiLBxj9ASSxlJUVASZTAaNRgOpVNqkn52hKkbQ6qOQ29vi9OKnmvSzCfm3tTGZWBV9CQDwVlAnKuLE6jRWLTB4BH6vvc0cx0EsFjf4dqrEuGj6nJiL14fWTKd//PslrPwtA4wxTH+ig6nTIsTiGVzA5XI5OI67Z3ubNm0wceJELFmyRG/qmzQtfgGbnBawEdOb/kQHcByHlb9l4OPfL4GxmsJOCGk4gwv45s2b8c4772DixInw9/cHACQmJmLLli1YuHAhbt26hY8//hhisRhvv/220RMm9UMjcGJuaqfOV/6WwU+pUxEnpOEMLuBbtmzBqlWrMGbMGP7Yc889Bx8fH2zYsAExMTHw9PTEhx9+SAXchK4V0h5wYn6mDWkPjgNWHKwp4gw1i90IIYYzeI775MmT6N27913He/fuzd+/fMCAAcjOpscLmtJ1Nd2FjZin1x5vj3nDOgOo2Wb22R+ZDziDEFIXgwu4h4cHvv7667uOf/311/xdywoKCtCyZcuHz440GH8N3IlG4MT8vPr4o5j/dE0R//SPS1j9xyUTZ0SI5TF4Cv3jjz/Giy++iAMHDqBv374AgFOnTiE9PR0//vgjACApKQljx441bqak3orLq6CufQ64nAo4MU+vDH4UHIDIA+lY/UcmGKu5FSshpH4MLuDDhw9Heno6NmzYgEuXav5qfvrpp7F37160bdsWAPDqq68aNUlimBvqmtG33N4WjhLLuGscsU5TBz8Kjqt5+MlnMTVT6VTECamfBt2JzdvbG8uWLTN2LsRIrtMCNmJBah87SkWcEMM0qICr1WokJiYiLy/vrgd7hIWFGSUx0nD8FjLaA04sRMSgR8GBw4f7L1IRJ6SeDC7gv/zyC0JDQ1FSUgKpVKp3UxeO46iAmwF6jCixRFMGtQMAvogzALMDO9z3xlGEWDODV6G/8cYbmDx5MkpKSqBWq3H79m3+VVhY2Bg5EgNRASeWasqgdnjnmS4AgDUxmfj0j0zQ4xoIqZvBBfzGjRuYMWMG7O1petZc0R5wYsmmDGqHhcFUxAl5EIMLeFBQEE6dOtUYuRAjoT3gxNK9PJCKOCEPYvA18ODgYLz11ltIS0uDj48PbG31tykNHz7caMkRw9EecNJcvDyw5pr40l8vYk1MJsAYZj/Zka6JE/I3gwv4lClTAADvv//+XW0cx0Gr1T58VqTBaA84aU70injsZQCgIk7I3wwu4P/dNkbMC+0BJ80NFXFC6kYP7G5maA84aY70ronHXsYn0ZfomjixevUaga9ZswYRERGQSCRYs2bNfWNnzJhhlMRIw9AWMtJc/XskvvbvkfgcGokTK1avAv7pp58iNDQUEokEn3766T3jOI6jAm5iVMBJc0ZFnJB/1KuAZ2Vl1fm/ifmp3QPu4URT6KR5enlgO3Achw+i0qiIE6tG18CbmX9G4FTASfMVPsAbi57tCgBYG3sZq36na+LE+hi8Cl2r1WLz5s2IiYmp82EmsbGxRkuOGEZvDzhNoZNmLnyANwDgg6g0fH6oZiT+xlM0EifWw+ACPnPmTGzevBnBwcHo3r07/cdiRmpH3y3tbdFC3KAHzRFiUcIHeIMD8D4VcWKFDP4tv337duzcuRPPPPNMY+RDHgJNnxNrNPnvkXhtEWdgePOpTlTESbNn8DVwkUiE9u3bN0Yu5CHxe8Bp+pxYmckDvLH472vi6w79iY9/z6Br4qTZa9DjRD/77DP6j8MM0RYyYs0mD/DGkueoiBPrYfAU+vHjx3Ho0CEcOHAA3bp1u+thJrt37zZacsQw/4zAaQqdWKdJ/Wum09/7JQ3rDv0JxoC3gmg6nTRPBhdwuVyOUaNGNUYu5CHRCJwQ/SL+xeE/AVARJ82TQQW8uroaQ4YMwVNPPQWFQtFYOZEGokVshNSgIk6sgUHXwIVCIV555RVUVFQ0Vj6kgYrKq6Apoz3ghNSa1N8b7/59TfyLw39i5W90TZw0LwYvYvP390dqampj5EIewg3aA07IXSb+p4gvO5hORZw0GwYX8Ndeew1vvPEGPv/8c8TFxeHs2bN6r4ZYt24d2rZtC4lEAqVSicTExPvG79q1C507d4ZEIoGPjw/279+v184Yw+LFi+Hm5gY7OzsEBgYiMzNTL6awsBChoaGQSqWQy+UIDw9HSUkJ33748GGMGDECbm5ucHBwQK9evbB169YG9a8p0PQ5IXWb2N8b7w3vBgDYcOQK3o9KoyJOmgdmII7j7noJBAL+q6G2b9/ORCIR++abb9iFCxfYlClTmFwuZ7m5uXXGnzhxgtnY2LAVK1awtLQ0tnDhQmZra8vOnTvHxyxbtozJZDK2d+9edubMGTZ8+HDm7e3NysrK+Jhhw4axnj17svj4eHbs2DHWvn17FhISwrd/+OGHbOHChezEiRPs8uXLbPXq1UwgELBffvml3n3TaDQMANNoNAb/XAz1zfErzGteFHvl/041+mcRYon+L+4v5jUvinnNi2Lv7DnLtFqdqVMiVqKxagHHmGF/il69evW+7V5eXgb9AaFUKtG3b198/vnnAACdTgcPDw+8/vrrmD9//l3xY8eORWlpKaKiovhjjz32GHr16oX169eDMQZ3d3e88cYbePPNNwEAGo0Grq6u2Lx5M8aNG4eLFy+ia9euSEpKgp+fHwDg4MGDeOaZZ3D9+nW4u7vXmWtwcDBcXV3xzTff1KtvRUVFkMlk0Gg0kEqlBv1cDPVBVBq+Pp6FKQO98U5w10b9LEIs1c6ka5i3+ywYA8b19cBHo3wgENDCNtK4GqsWGDyF7uXldd+XISorK5GcnIzAwMB/EhIIEBgYiLi4uDrPiYuL04sHgKCgID4+KysLKpVKL0Ymk0GpVPIxcXFxkMvlfPEGgMDAQAgEAiQkJNwzX41GAycnp3u2V1RUoKioSO/VVGgPOCEPNqavBz4Z0xMCDtiedA1v/ngGWh1NpxPL1ODVTmlpacjOzkZlZaXe8eHDh9f7PfLz86HVauHq6qp33NXVFenp6XWeo1Kp6oxXqVR8e+2x+8W4uLjotQuFQjg5OfEx/7Vz504kJSVhw4YN9+xPZGQk3nvvvXu2NybaA05I/Yzq3QZCgQCzdpzG7pQbqNIyfDqmJ4Q29HRlYlkMLuBXrlzBqFGjcO7cOXAcxy8Gqd1fqdVqjZuhGTh06BAmTZqETZs2oVu3bveMW7BgAebMmcN/X1RUBA8Pj6ZIkRaxEWKA53q6w9aGw/QfUvHLmRxUa3X4bFxviIRUxInlMPhf68yZM+Ht7Y28vDzY29vjwoULOHr0KPz8/HD48GGD3svZ2Rk2NjbIzc3VO56bm3vPG8UoFIr7xtd+fVBMXl6eXnt1dTUKCwvv+twjR47gueeew6effoqwsLD79kcsFkMqleq9mgLtASfEcMO6u2H9/3whshHgwHkVXtuajIrq5jcAIc2XwQU8Li4O77//PpydnSEQCCAQCDBgwABERkZixowZBr2XSCSCr68vYmJi+GM6nQ4xMTEICAio85yAgAC9eACIjo7m4729vaFQKPRiioqKkJCQwMcEBARArVYjOTmZj4mNjYVOp4NSqeSPHT58GMHBwVi+fDkiIiIM6ltToj3ghDRMYFdXbAzzhVgowB8X8xDxXTLKq6iIE8tgcAHXarVwdHQEUDOCzsnJAVCzuC0jI8PgBObMmYNNmzZhy5YtuHjxIl599VWUlpZi0qRJAICwsDAsWLCAj585cyYOHjyIVatWIT09He+++y5OnTqF6dOnA6iZyp81axaWLl2Kffv24dy5cwgLC4O7uztGjhwJAOjSpQuGDRuGKVOmIDExESdOnMD06dMxbtw4fgX6oUOHEBwcjBkzZmD06NFQqVRQqVQoLCw0uI+NjabPCWm4xzu54JuJfSGxFeDIpVt4ecsplFVSEScWwNB9ZwMGDGB79uxhjDEWEhLChg0bxo4fP87CwsJYt27dGrSXbe3atczT05OJRCLm7+/P4uPj+bbBgwezCRMm6MXv3LmTdezYkYlEItatWzf266+/6rXrdDq2aNEi5urqysRiMRs6dCjLyMjQiykoKGAhISGsRYsWTCqVskmTJrHi4mK+fcKECQzAXa/BgwfXu19NtQ/862O0B5yQhxX3Zz7rsugA85oXxV5cf5IVl1eZOiXSTJjNPvDffvsNpaWleP7553H58mU8++yzuHTpElq1aoUdO3bgiSeeMPofGZaqqfaBv/9LGr45kYWIQe3w9jNdGu1zCGnukq8WYuI3SSiuqEZPDzm2TOoLub3I1GkRC2c2+8CDgoLw/PPPAwDat2+P9PR05OfnIy8vj4q3ifyzB5wWsBHyMHy9nLB1ihIt7W1x5poaYzfEI6+43NRpEVKnBu+ZuHz5Mn777TeUlZXd9+YmpPHRHnBCjKdHGzl2TA3AI45iZOQWY+yGeNxQl5k6LULuYnABLygowNChQ9GxY0c888wzuHnzJgAgPDwcb7zxhtETJA9Gd2EjxLg6ujpi19QAtJbbISu/FC9+eRJXbpU8+ERCmpDBBXz27NmwtbVFdnY27O3/KRhjx47FwYMHjZoceTBNWRWKyqsBAK3lNAInxFjaOjvgx1cD0O4RB+RoyjFmQzwu3my62yMT8iAGF/Dff/8dy5cvR5s2bfSOd+jQ4YEPOiHGV7sH3MlBBAfaA06IUbnJ7LBzagC6uEmRX1KBcRvjkZp929RpEQKgAQW8tLRUb+Rdq7CwEGKx2ChJkfqjBWyENC7nFmJsn/IY+njKoSmrwv++SkDcnwWmTosQwwv4wIED8d133/HfcxwHnU6HFStWYMiQIUZNjjwYLWAjpPHJ7G3xf+FK9G/fCqWVWkz8NhGH0vMefCIhjcjgAr5ixQps3LgRTz/9NCorKzF37lx0794dR48exfLlyxsjR3IfdBc2QpqGg1iIryf0RWAXF1RU6zDlu1OIOptj6rSIFTO4gHfv3h2XLl3CgAEDMGLECP6mLqmpqXj00UcbI0dyHzSFTkjTkdja4Mv/+WJ4T3dU6xhmbEvF9sRsU6dFrFSDVj3JZDK88847eseuX7+OiIgIbNy40SiJkfqhKXRCmpatjQCfju0FB7ENtiVew/zd51BQWonXHn+Uf6wyIU3BaA+/LSgowNdff22styP1RHvACWl6NgIOH43ywWuP18w6rvwtAx9EXYROZ9CdqQl5KPT0egtGe8AJMR2O4zB3WGcsDK55/sA3J7IwZ+dpVGl1Js6MWAsq4BaM9oATYnovD2yHT8f2hFDAYe/pHEz57hTuVFabOi1iBaiAWzBawEaIeRjVuw02TfCDxFaAwxm3EPpVAm6XVpo6LdLM1XvYVvsEsntRq9UPmwsxEC1gI8R8DOnkgq0vP4bJm5OQmq3Gixvi8N1kf7jT5S3SSOo9ApfJZPd9eXl5ISwsrDFzJf9Be8AJMS++Xi2x65UAKKQSXM4rwQtfnsTlvGJTp0WaqXqPwL/99tvGzIM0AE2hE2J+Oro64qfX+mH81wm4cqsUz39xEpvC/KBs18rUqZFmhq6BW7BrNIVOiFlqLbfDj6/0Qx9POYrKqzH+60T8cobu2kaMiwq4BaM94ISYLycHEX6Y8hiGdVOgUqvD69tSsfHon2CM9ooT46ACbqE0ZVUopj3ghJg1ia0N1oX2waT+bQEAH+1Px7v7LkBLN3whRkAF3ELVjr5b0R5wQsyajYDDkue68Td82RJ3Fa98n4yySq2JMyOWjgq4haItZIRYlpcHtsO6l/pAJBQgOi0XIZviUVBSYeq0iAWjAm6haAsZIZYnuIcbtr6shMzOFqevqTH6y5O4cqvE1GkRC0UF3ELRFjJCLFPftk746dV+aNPSDn8V3MGoL07i5OV8U6dFLBAVcAtFU+iEWK72Li2w57X+6OMph6asCmHfJOKHBHquODEMFXALRVPohFi2RxzF+GHKYxjRyx3VOoa395zD+7+k0Qp1Um9UwC0UTaETYvkktjZYPbYX3niyI4CaR5JO+e4UisurTJwZsQRUwC2Q3h5wKuCEWDSO4/D60A5Y91IfSGwFiE3PwwtfxuFa4R1Tp0bMHBVwC/TvPeD2ItoDTkhzENzDDTunBsDFUYyM3GKMXHcCyVcLTZ0WMWNUwC0QLWAjpHnq0UaOn6f3Rzd3KQpKKxGyMQE7kmhxG6kbFXALRAvYCGm+3GR22PVKAH8P9Xk/ncM7e86hslpn6tSImTF5AV+3bh3atm0LiUQCpVKJxMTE+8bv2rULnTt3hkQigY+PD/bv36/XzhjD4sWL4ebmBjs7OwQGBiIzM1MvprCwEKGhoZBKpZDL5QgPD0dJyT83UygvL8fEiRPh4+MDoVCIkSNHGq2/xkAL2Ahp3uxFQnwR2gdvPtURHAdsTchGyKZ45BWVmzo1YkZMWsB37NiBOXPmYMmSJUhJSUHPnj0RFBSEvLy8OuNPnjyJkJAQhIeHIzU1FSNHjsTIkSNx/vx5PmbFihVYs2YN1q9fj4SEBDg4OCAoKAjl5f/8ww8NDcWFCxcQHR2NqKgoHD16FBEREXy7VquFnZ0dZsyYgcDAwMb7ATQQTaET0vwJBBymP9EB30zoC0eJEMlXb+PZtceRfPW2qVMj5oKZkL+/P5s2bRr/vVarZe7u7iwyMrLO+DFjxrDg4GC9Y0qlkk2dOpUxxphOp2MKhYKtXLmSb1er1UwsFrNt27YxxhhLS0tjAFhSUhIfc+DAAcZxHLtx48ZdnzlhwgQ2YsSIBvVPo9EwAEyj0TTo/HsZtvoo85oXxWIv5hr1fQkh5unKrRIWuOow85oXxdq//Sv7Pv4vU6dEDNBYtcBkI/DKykokJyfrjXAFAgECAwMRFxdX5zlxcXF3jYiDgoL4+KysLKhUKr0YmUwGpVLJx8TFxUEul8PPz4+PCQwMhEAgQEJCgtH615hoCp0Q6+Lt7IC90/rjGR8FqrQM7+w5jwW7z6Kimp5oZs1MVsDz8/Oh1Wrh6uqqd9zV1RUqlarOc1Qq1X3ja78+KMbFxUWvXSgUwsnJ6Z6fW18VFRUoKirSexkb7QEnxDo5iIVY91IfzBvWGRwHbEu8hjEb4vk/6In1MfkituYkMjISMpmMf3l4eBj9M2pv7kB7wAmxPhzH4dXHH8XmSf6Q2dnizDU1gtccR3RarqlTIyZgsgLu7OwMGxsb5Obq/8PLzc2FQqGo8xyFQnHf+NqvD4r57yK56upqFBYW3vNz62vBggXQaDT869q1aw/1fnWhBWyEkMEdH0HU6wPQ06PmYShTvjuFpVFptNXMypisgItEIvj6+iImJoY/ptPpEBMTg4CAgDrPCQgI0IsHgOjoaD7e29sbCoVCL6aoqAgJCQl8TEBAANRqNZKTk/mY2NhY6HQ6KJXKh+qTWCyGVCrVexnbP9e/aQ84IdbMw8keu6YGIHyANwDgq+NZeHED3YLVmph0Cn3OnDnYtGkTtmzZgosXL+LVV19FaWkpJk2aBAAICwvDggUL+PiZM2fi4MGDWLVqFdLT0/Huu+/i1KlTmD59OoCa6aVZs2Zh6dKl2LdvH86dO4ewsDC4u7vze7m7dOmCYcOGYcqUKUhMTMSJEycwffp0jBs3Du7u7vxnpaWl4fTp0ygsLIRGo8Hp06dx+vTpJvvZ3AuNwAkhtURCARY92xUbx/tCKhH+PaV+DL9feLj1PMRCGHVNewOsXbuWeXp6MpFIxPz9/Vl8fDzfNnjwYDZhwgS9+J07d7KOHTsykUjEunXrxn799Ve9dp1OxxYtWsRcXV2ZWCxmQ4cOZRkZGXoxBQUFLCQkhLVo0YJJpVI2adIkVlxcrBfj5eXFANz1MkRjbB0I35zEvOZFse9OZhntPQkhli+7oJSN+Pw485oXxbzmRbH39l1gFVVaU6dFWONtI+MYY/Tw2UZSVFQEmUwGjUZjtOn0YauPIl1VjG8n9cWQTi4PPoEQYjUqq3VY+Vs6Nh3LAgD0aCPD6rG90O6RFibOzLo1Ri0AaBW6RWGM4cbfU+geNIVOCPkPkVCAd4K74qswP8jsbHH2ugbBa47jh4Rs0Fit+aECbkGKyqpRXPH3HnA5LWIjhNQtsKsrDs4aiP7tW6GsSou395zDlO+SUVBSYerUiBFRAbcg1/5ege7cQgQ7kY2JsyGEmDM3mR3+b7ISC4O7QGQjwB8XcxG0+hgOpdf9rAlieaiAW5DaFeitaQsZIaQeBAIOLw9sh73T+qOjawvkl1Rg0uYkzP/pLIrLq0ydHnlIVMAtCN0DnRDSEF3dpdg3fQAm9/cGxwHbk64h6NOjOJZ5y9SpkYdABdyC0B5wQkhDSWxtsPi5rtg+5TF4OtkjR1OO8V8nYsHuczQat1BUwC3IPwWcptAJIQ2jbNcKB2cNxMR+bQEA2xKzMWz1MRzOoGvjloYKuAWhKXRCiDHYi4R4d3g3bJvyGDyc7HBDXYaJ3ybh9W2puFVMK9UtBRVwC0F7wAkhxhbwaCscnDkILw/whoADfjmTg6GrDmN7YjZ0Oto3bu6ogFsI2gNOCGkMDmIhFj7bFT9PG4DuraUoKq/G/N3nMG5jPDJzi02dHrkPKuAWgvaAE0Iak08bGfa+1h8Lg7vAztYGiX8V4unPjmFpVBotcjNTVMAtBO0BJ4Q0NqGNAC8PbIfoOYPwZFdXVOsYvjqehSEfH8GPyddpWt3MUAG3ELSAjRDSVNq0tMemMD9sntQX7ZwdkF9SgTd3ncHo9Sdx9rra1OmRv1EBtxC0B5wQ0tQe7+SCg7MGYcHTneEgskFqthrDPz+BmdtTca3wjqnTs3pUwC3EPyNwmkInhDQdkVCAqYMfReybj+P53q0BAD+fzsHQVUfw4a9pUN+pNHGG1osKuIWgETghxJRcpRJ8MrYXol4fgP7tW6FSq8OmY1kYvPIwNh79E2WVWlOnaHWogFsAxhhfwGkPOCHElLq3luH7cCU2T+qLzgpHaMqq8NH+dAxaeQjfHM9CeRUV8qZCBdwCaMqqUEJ7wAkhZoLjODzeyQW/zhiIlS/0QJuWdrhVXIH3o9IweOUhbDn5FxXyJkAF3ALUjr5pDzghxJzYCDi86OeB2Dcex0ejfNBabofcogos2XcBQz4+jK+PZ6H078EHMT4q4BagdgEb7QEnhJgjkVCAl5SeiH1zMJaO7A43mQQ3NeX4ICoN/ZbFYtXvGcgvoXusGxsVcAtAC9gIIZZALLTB/x7zwuG3akbk3s4O0JRVYW3sZfRfFot39pzDn7dKTJ1msyE0dQLkwf5ZwEYjcEKI+RMLbfCS0hNj+3ogOk2FL49cwZlramxNyMbWhGwMaO+M8QFeGNrZBUIbGkc2FBVwC0B3YSOEWCIbAYdh3d0Q1E2BxKxCbDp2BTHpeTh+OR/HL+fDTSbBS/6eGOvvARdHianTtThUwC0ATaETQiwZx3FQtmsFZbtWuFZ4Bz8kZmNH0jXc1JRjVfQlrI7JxKAOzhjt2waBXVwhsaXFuvVBBdzM/XsPON2FjRBi6Tyc7DFvWGfMCuyA/edu4v/iriIlW41DGbdwKOMWHCVCPNvDHaP7tEYfz5YQCDhTp2y2qICbuX/vAacROCGkuRALbTCqdxuM6t0Gf94qwZ6UG9idch05mnJsS8zGtsRsuErFCOqmwLBuCvh7O9H18v/gGGP0fLhGUlRUBJlMBo1GA6lU2qD3OH9Dg2fXHodzCzFOLQw0coaEEGI+dDqG+CsF+DHlOqIv5KL4X3vIW9rb4smurhjSyQX9HnWGzN7WhJkaxhi1oC40AjdztICNEGItBAIO/do7o197Z1RUa3HycgEOnlfh9zQVbt+pws5T17Hz1HUIOKCXhxwDOzyCQR2d0aONHLZWODqnAm7maAEbIcQaiYU2GNLZBUM6u+BDbXck/lWI6LRcHMvMx+W8EqRkq5GSrcZnMZmQ2ArQo40cvl4t4evZEn28WsLJQWTqLjQ6KuBmjhawEUKsndBGgH6POqPfo84AgBx1GY5n5uNo5i0cv5wP9Z0qJGYVIjGrkD/Hw8kOnVyl6KxwRKe/X97ODs1qpE4F3MzRFDohhOhzl9thTF8PjOnrAZ2O4Up+KVKu3kby1dtIyb6NzLwSXCssw7XCMvxxMZc/Tyjg4CaXoI3cHh5OdmjT0h6t5XZo1UKElvYiODmIILe3RQuxEBxn/qvfqYCbOZpCJ4SQexMIOLR3aYH2Li0wpq8HAEBzpwppN4uQoSpCRm4x0lXFuKQqRmmlli/scVfu/Z62NhwcxEJIhDawE9kgYlA7hPh7NlGP6s8s5hLWrVuHtm3bQiKRQKlUIjEx8b7xu3btQufOnSGRSODj44P9+/frtTPGsHjxYri5ucHOzg6BgYHIzMzUiyksLERoaCikUinkcjnCw8NRUqJ/j96zZ89i4MCBkEgk8PDwwIoVK4zT4XqiPeCEEGI4mb0tAh5thYn9vRH5fA/sea0/zr0bhJPzn8CuVwLw6diemPNkR4zxa4P+7Vuhi5sUbjIJJLY1JbFKy6C+UwVVUTmy8kvN94lqzMS2b9/ORCIR++abb9iFCxfYlClTmFwuZ7m5uXXGnzhxgtnY2LAVK1awtLQ0tnDhQmZra8vOnTvHxyxbtozJZDK2d+9edubMGTZ8+HDm7e3NysrK+Jhhw4axnj17svj4eHbs2DHWvn17FhISwrdrNBrm6urKQkND2fnz59m2bduYnZ0d27BhQ737ptFoGACm0Wga8JNhTKfTsXPX1ezAuZusvKq6Qe9BCCGk/u5UVLMbt++wzNxidu66miVmFbAc9Z2Hes+HrQX3YvIC7u/vz6ZNm8Z/r9Vqmbu7O4uMjKwzfsyYMSw4OFjvmFKpZFOnTmWM1RQ9hULBVq5cyber1WomFovZtm3bGGOMpaWlMQAsKSmJjzlw4ADjOI7duHGDMcbYF198wVq2bMkqKir4mHnz5rFOnTrVu2+N9X8aIYQQy9FYtcCkU+iVlZVITk5GYOA/NygRCAQIDAxEXFxcnefExcXpxQNAUFAQH5+VlQWVSqUXI5PJoFQq+Zi4uDjI5XL4+fnxMYGBgRAIBEhISOBjBg0aBJFIpPc5GRkZuH379kP2nBBCCHk4Ji3g+fn50Gq1cHV11Tvu6uoKlUpV5zkqleq+8bVfHxTj4uKi1y4UCuHk5KQXU9d7/Psz/quiogJFRUV6L0IIIaQxmMUituYiMjISMpmMf3l4eJg6JUIIIc2USQu4s7MzbGxskJubq3c8NzcXCoWiznMUCsV942u/PigmLy9Pr726uhqFhYV6MXW9x78/478WLFgAjUbDv65du1Z3xwkhhJCHZNICLhKJ4Ovri5iYGP6YTqdDTEwMAgIC6jwnICBALx4AoqOj+Xhvb28oFAq9mKKiIiQkJPAxAQEBUKvVSE5O5mNiY2Oh0+mgVCr5mKNHj6Kqqkrvczp16oSWLVvWmZtYLIZUKtV7EUIIIY3CqEviGmD79u1MLBazzZs3s7S0NBYREcHkcjlTqVSMMcbGjx/P5s+fz8efOHGCCYVC9vHHH7OLFy+yJUuW1LmNTC6Xs59//pmdPXuWjRgxos5tZL1792YJCQns+PHjrEOHDnrbyNRqNXN1dWXjx49n58+fZ9u3b2f29vZNuo2MEEKI5Wu228gYY2zt2rXM09OTiUQi5u/vz+Lj4/m2wYMHswkTJujF79y5k3Xs2JGJRCLWrVs39uuvv+q163Q6tmjRIubq6srEYjEbOnQoy8jI0IspKChgISEhrEWLFkwqlbJJkyax4uJivZgzZ86wAQMGMLFYzFq3bs2WLVtmUL+ogBNCCGmsWkDPA29EGo0Gcrkc165do+l0QgixUkVFRfDw8IBarYZMJjPa+9K90BtRcXExANBqdEIIISguLjZqAacReCPS6XTIycmBo6Njg59sU/uXW3MaxTe3PjW3/gDNr0/NrT8A9ckS1PYnOzsbHMfB3d0dAoHx1o7TCLwRCQQCtGnTxijv1RxXtTe3PjW3/gDNr0/NrT8A9ckSyGSyRukP3ciFEEIIsUBUwAkhhBALRAXczInFYixZsgRisdjUqRhNc+tTc+sP0Pz61Nz6A1CfLEFj94cWsRFCCCEWiEbghBBCiAWiAk4IIYRYICrghBBCiAWiAk4IIYRYICrgZm7dunVo27YtJBIJlEolEhMTTZ1SvURGRqJv375wdHSEi4sLRo4ciYyMDL2Y8vJyTJs2Da1atUKLFi0wevTou57Bbq6WLVsGjuMwa9Ys/pgl9ufGjRv43//+h1atWsHOzg4+Pj44deoU384Yw+LFi+Hm5gY7OzsEBgYiMzPThBnfn1arxaJFi+Dt7Q07Ozs8+uij+OCDD/Dvtbrm3KejR4/iueeeg7u7OziOw969e/Xa65N7YWEhQkNDIZVKIZfLER4ejpKSkibshb779amqqgrz5s2Dj48PHBwc4O7ujrCwMOTk5Oi9hyX16b9eeeUVcByH1atX6x03Rp+ogJuxHTt2YM6cOViyZAlSUlLQs2dPBAUFIS8vz9SpPdCRI0cwbdo0xMfHIzo6GlVVVXjqqadQWlrKx8yePRu//PILdu3ahSNHjiAnJwfPP/+8CbOun6SkJGzYsAE9evTQO25p/bl9+zb69+8PW1tbHDhwAGlpaVi1apXe8+5XrFiBNWvWYP369UhISICDgwOCgoJQXl5uwszvbfny5fjyyy/x+eef4+LFi1i+fDlWrFiBtWvX8jHm3KfS0lL07NkT69atq7O9PrmHhobiwoULiI6ORlRUFI4ePYqIiIim6sJd7tenO3fuICUlBYsWLUJKSgp2796NjIwMDB8+XC/Okvr0b3v27EF8fDzc3d3vajNKn4z6bDNiVP7+/mzatGn891qtlrm7u7PIyEgTZtUweXl5DAA7cuQIY6zmeeu2trZs165dfMzFixcZABYXF2eqNB+ouLiYdejQgUVHR7PBgwezmTNnMsYssz/z5s1jAwYMuGe7TqdjCoWCrVy5kj+mVquZWCxm27Zta4oUDRYcHMwmT56sd+z5559noaGhjDHL6hMAtmfPHv77+uSelpbGALCkpCQ+5sCBA4zjOHbjxo0my/1e/tunuiQmJjIA7OrVq4wxy+3T9evXWevWrdn58+eZl5cX+/TTT/k2Y/WJRuBmqrKyEsnJyQgMDOSPCQQCBAYGIi4uzoSZNYxGowEAODk5AQCSk5NRVVWl17/OnTvD09PTrPs3bdo0BAcH6+UNWGZ/9u3bBz8/P7z44otwcXFB7969sWnTJr49KysLKpVKr08ymQxKpdJs+9SvXz/ExMTg0qVLAIAzZ87g+PHjePrppwFYZp9q1Sf3uLg4yOVy+Pn58TGBgYEQCARISEho8pwbQqPRgOM4yOVyAJbZJ51Oh/Hjx+Ott95Ct27d7mo3Vp/oYSZmKj8/H1qtFq6urnrHXV1dkZ6ebqKsGkan02HWrFno378/unfvDgBQqVQQiUT8f6S1XF1doVKpTJDlg23fvh0pKSlISkq6q80S+3PlyhV8+eWXmDNnDt5++20kJSVhxowZEIlEmDBhAp93Xf8GzbVP8+fPR1FRETp37gwbGxtotVp8+OGHCA0NBQCL7FOt+uSuUqng4uKi1y4UCuHk5GT2/QNq1pHMmzcPISEh/MM/LLFPy5cvh1AoxIwZM+psN1afqICTRjdt2jScP38ex48fN3UqDXbt2jXMnDkT0dHRkEgkpk7HKHQ6Hfz8/PDRRx8BAHr37o3z589j/fr1mDBhgomza5idO3di69at+OGHH9CtWzecPn0as2bNgru7u8X2yVpUVVVhzJgxYIzhyy+/NHU6DZacnIzPPvsMKSkpDX6MdH3RFLqZcnZ2ho2NzV2rmHNzc6FQKEyUleGmT5+OqKgoHDp0SO/RqgqFApWVlVCr1Xrx5tq/5ORk5OXloU+fPhAKhRAKhThy5AjWrFkDoVAIV1dXi+oPALi5uaFr1656x7p06YLs7GwA4PO2pH+Db731FubPn49x48bBx8cH48ePx+zZsxEZGQnAMvtUqz65KxSKuxa5VldXo7Cw0Kz7V1u8r169iujoaL1Hb1pan44dO4a8vDx4enryvyuuXr2KN954A23btgVgvD5RATdTIpEIvr6+iImJ4Y/pdDrExMQgICDAhJnVD2MM06dPx549exAbGwtvb2+9dl9fX9ja2ur1LyMjA9nZ2WbZv6FDh+LcuXM4ffo0//Lz80NoaCj/vy2pPwDQv3//u7b2Xbp0CV5eXgAAb29vKBQKvT4VFRUhISHBbPt0584dCAT6v9ZsbGyg0+kAWGafatUn94CAAKjVaiQnJ/MxsbGx0Ol0UCqVTZ5zfdQW78zMTPzxxx9o1aqVXrul9Wn8+PE4e/as3u8Kd3d3vPXWW/jtt98AGLFPDV97Rxrb9u3bmVgsZps3b2ZpaWksIiKCyeVyplKpTJ3aA7366qtMJpOxw4cPs5s3b/KvO3fu8DGvvPIK8/T0ZLGxsezUqVMsICCABQQEmDBrw/x7FTpjltefxMREJhQK2YcffsgyMzPZ1q1bmb29Pfv+++/5mGXLljG5XM5+/vlndvbsWTZixAjm7e3NysrKTJj5vU2YMIG1bt2aRUVFsaysLLZ7927m7OzM5s6dy8eYc5+Ki4tZamoqS01NZQDYJ598wlJTU/kV2fXJfdiwYax3794sISGBHT9+nHXo0IGFhISYqkv37VNlZSUbPnw4a9OmDTt9+rTe74qKigqL7FNd/rsKnTHj9IkKuJlbu3Yt8/T0ZCKRiPn7+7P4+HhTp1QvAOp8ffvtt3xMWVkZe+2111jLli2Zvb09GzVqFLt586bpkjbQfwu4Jfbnl19+Yd27d2disZh17tyZbdy4Ua9dp9OxRYsWMVdXVyYWi9nQoUNZRkaGibJ9sKKiIjZz5kzm6enJJBIJa9euHXvnnXf0ioE59+nQoUN1/nczYcIExlj9ci8oKGAhISGsRYsWTCqVskmTJrHi4mIT9KbG/fqUlZV1z98Vhw4dssg+1aWuAm6MPtHjRAkhhBALRNfACSGEEAtEBZwQQgixQFTACSGEEAtEBZwQQgixQFTACSGEEAtEBZwQQgixQFTACSGEEAtEBZwQQgixQFTACSH3dOvWLbz66qvw9PSEWCyGQqFAUFAQTpw4AQDgOA579+41bZKEWCl6nCgh5J5Gjx6NyspKbNmyBe3atUNubi5iYmJQUFBg6tQIsXo0AieE1EmtVuPYsWNYvnw5hgwZAi8vL/j7+2PBggUYPnw4/2jEUaNGgeM4/nsA+Pnnn9GnTx9IJBK0a9cO7733Hqqrq/l2juPw5Zdf4umnn4adnR3atWuHH3/8kW+vrKzE9OnT4ebmBolEAi8vL/6RoISQGlTACSF1atGiBVq0aIG9e/eioqLirvakpCQAwLfffoubN2/y3x87dgxhYWGYOXMm0tLSsGHDBmzevBkffvih3vmLFi3C6NGjcebMGYSGhmLcuHG4ePEiAGDNmjXYt28fdu7ciYyMDGzdulXvDwRCCEAPMyGE3NNPP/2EKVOmoKysDH369MHgwYMxbtw49OjRA0DNSHrPnj0YOXIkf05gYCCGDh2KBQsW8Me+//57zJ07Fzk5Ofx5r7zyCr788ks+5rHHHkOfPn3wxRdfYMaMGbhw4QL++OMPcBzXNJ0lxMLQCJwQck+jR49GTk4O9u3bh2HDhuHw4cPo06cPNm/efM9zzpw5g/fff58fwbdo0QJTpkzBzZs3cefOHT4uICBA77yAgAB+BD5x4kScPn0anTp1wowZM/D77783Sv8IsWRUwAkh9yWRSPDkk09i0aJFOHnyJCZOnIglS5bcM76kpATvvfceTp8+zb/OnTuHzMxMSCSSen1mnz59kJWVhQ8++ABlZWUYM2YMXnjhBWN1iZBmgQo4IcQgXbt2RWlpKQDA1tYWWq1Wr71Pnz7IyMhA+/bt73oJBP/8yomPj9c7Lz4+Hl26dOG/l0qlGDt2LDZt2oQdO3bgp59+QmFhYSP2jBDLQtvICCF1KigowIsvvojJkyejR48ecHR0xKlTp7BixQqMGDECANC2bVvExMSgf//+EIvFaNmyJRYvXoxnn30Wnp6eeOGFFyAQCHDmzBmcP38eS5cu5d9/165d8PPzw4ABA7B161YkJibi66+/BgB88skncHNzQ+/evSEQCLBr1y4oFArI5XJT/CgIMU+MEELqUF5ezubPn8/69OnDZDIZs7e3Z506dWILFy5kd+7cYYwxtm/fPta+fXsmFAqZl5cXf+7BgwdZv379mJ2dHZNKpczf359t3LiRbwfA1q1bx5588kkmFotZ27Zt2Y4dO/j2jRs3sl69ejEHBwcmlUrZ0KFDWUpKSpP1nRBLQKvQCSFNrq7V64QQw9A1cEIIIcQCUQEnhBBCLBAtYiOENDm6ckfIw6MROCGEEGKBqIATQgghFogKOCGEEGKBqIATQgghFogKOCGEEGKBqIATQgghFogKOCGEEGKBqIATQgghFogKOCGEEGKB/h84pTpWAJRRGAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.plot(range(len(lrs)), lrs)\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Steps\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f85b01-859b-4454-a3a3-c7ef593735a6",
   "metadata": {},
   "source": [
    "- And a quick look at the loss curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "445d8155-6eae-4b50-a381-d0820ebc27cc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/ipykernel_41293/1615996262.py:6: UserWarning: The figure layout has changed to tight\n",
      "  plt.tight_layout(); plt.savefig(\"3.pdf\")\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 500x300 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnUAAAHWCAYAAAARl3+JAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB0j0lEQVR4nO3dd1QU198G8Gd2F5al9yZFRJRir1HsEtEYjS0aQ4wl1W5MjMmbaEwxRmMSY4om5hdNsaWoMXbsig27KGIXRAEV6X133j9GFlZRwQCzLM/nnDnsTv3uzQYeZ+beEURRFEFERERENZpC7gKIiIiI6L9jqCMiIiIyAQx1RERERCaAoY6IiIjIBDDUEREREZkAhjoiIiIiE8BQR0RERGQCGOqIiIiITABDHREREZEJYKgjohrnypUrEAQBx48fl7sUIiKjwVBHRLIQBOGh04wZM+QukYioRlHJXQAR1U43btzQv165ciWmT5+OuLg4/Txra2s5yiIiqrF4po6IZOHu7q6f7OzsIAiC/r2rqyu+/PJLeHl5Qa1Wo1mzZti0adMD96XVajFq1CgEBgYiPj4eAPDPP/+gRYsWsLCwQL169fDhhx+iqKhIv40gCPjpp5/Qv39/WFpaIiAgAGvXrtUvv3PnDiIiIuDi4gKNRoOAgAAsXrz4gTX89ddfaNy4MTQaDZycnBAWFobs7Gz98p9++glBQUGwsLBAYGAgvv/+e4PtExISMHjwYNjb28PR0RHPPPMMrly5ol8+YsQI9OvXD3PnzoWHhwecnJwwduxYFBYWlrvNici0MdQRkdH5+uuv8cUXX2Du3Lk4efIkwsPD0bdvX5w/f/6+dfPz8/Hss8/i+PHj2LNnD3x8fLBnzx68+OKLmDhxIs6cOYMffvgBS5YswcyZMw22/fDDDzF48GCcPHkSTz31FCIiIpCamgoAmDZtGs6cOYONGzciNjYWCxYsgLOzc5n13rhxA0OHDsWoUaMQGxuLnTt3YsCAARBFEQCwdOlSTJ8+HTNnzkRsbCw+/fRTTJs2Db/88gsAoLCwEOHh4bCxscGePXsQFRUFa2tr9OzZEwUFBfrj7NixAxcvXsSOHTvwyy+/YMmSJViyZEllNDkRmQKRiEhmixcvFu3s7PTvPT09xZkzZxqs07p1a3HMmDGiKIri5cuXRQDinj17xO7du4sdOnQQ09LS9Ot2795d/PTTTw22/+2330QPDw/9ewDi+++/r3+flZUlAhA3btwoiqIo9unTRxw5cmS56j9y5IgIQLxy5UqZy/39/cVly5YZzPv444/Fdu3a6Wtr2LChqNPp9Mvz8/NFjUYjbt68WRRFURw+fLjo6+srFhUV6dd59tlnxSFDhpSrRiIyfbynjoiMSkZGBq5fv47Q0FCD+aGhoThx4oTBvKFDh8LLywvbt2+HRqPRzz9x4gSioqIMzsxptVrk5eUhJycHlpaWAIAmTZrol1tZWcHW1hYpKSkAgNGjR2PgwIE4evQoevTogX79+qF9+/Zl1ty0aVN0794djRs3Rnh4OHr06IFBgwbBwcEB2dnZuHjxIl566SW88sor+m2KiopgZ2enr/fChQuwsbEx2G9eXh4uXryofx8SEgKlUql/7+HhgVOnTj2kNYmoNmGoI6Ia66mnnsLvv/+O/fv3o1u3bvr5WVlZ+PDDDzFgwID7trGwsNC/NjMzM1gmCAJ0Oh0AoFevXrh69So2bNiAyMhIdO/eHWPHjsXcuXPv26dSqURkZCT27duHLVu24JtvvsF7772HgwcP6gPkokWL0LZt2/u2K663ZcuWWLp06X37dnFxKVe9REQMdURkVGxtbeHp6YmoqCh07txZPz8qKgpt2rQxWHf06NFo1KgR+vbti/Xr1+vXb9GiBeLi4lC/fv3/VIuLiwuGDx+O4cOHo2PHjpgyZUqZoQ6QAlZoaChCQ0Mxffp0+Pr6YvXq1Zg8eTI8PT1x6dIlRERElLltixYtsHLlSri6usLW1vY/1UxEtRdDHREZnSlTpuCDDz6Av78/mjVrhsWLF+P48eNlnskaP348tFotnn76aWzcuBEdOnTA9OnT8fTTT8PHxweDBg2CQqHAiRMnEBMTg08++aRcNUyfPh0tW7ZESEgI8vPzsW7dOgQFBZW57sGDB7Ft2zb06NEDrq6uOHjwIG7evKlf/8MPP8SECRNgZ2eHnj17Ij8/H4cPH8adO3cwefJkRERE4PPPP8czzzyDjz76CF5eXrh69SpWrVqFt99+G15eXo/fmERUazDUEZHRmTBhAtLT0/Hmm28iJSUFwcHBWLt2LQICAspcf9KkSdDpdHjqqaewadMmhIeHY926dfjoo48we/ZsmJmZITAwEC+//HK5azA3N8e7776LK1euQKPRoGPHjlixYkWZ69ra2mL37t2YN28eMjIy4Ovriy+++AK9evUCALz88suwtLTE559/jilTpsDKygqNGzfGpEmTAACWlpbYvXs3pk6digEDBiAzMxN16tRB9+7deeaOiMpNEMW7fe6JiIiIqMbiOHVEREREJoChjoiIiMgEMNQRERERmQCGOiIiIiITwFBHREREZAIY6oiIiIhMAENdBX333XeoW7cuLCws0LZtWxw6dEjukirV7t270adPH3h6ekIQBKxZs8ZguSiKmD59Ojw8PKDRaBAWFobz588brJOamoqIiAjY2trC3t4eL730ErKysgzWOXnyJDp27AgLCwt4e3tjzpw599Xy559/IjAwEBYWFmjcuDE2bNhQ6Z/3cc2aNQutW7eGjY0NXF1d0a9fP8TFxRmsk5eXh7Fjx8LJyQnW1tYYOHAgkpOTDdaJj49H7969YWlpCVdXV0yZMgVFRUUG6+zcuRMtWrSAWq1G/fr1sWTJkvvqMdbv5YIFC9CkSRPY2trC1tYW7dq1w8aNG/XL2UZl++yzzyAIgn4cO4BtVWzGjBkQBMFgCgwM1C9nO5VITEzECy+8ACcnJ2g0GjRu3BiHDx/WL+fvc6Bu3br3fZ8EQcDYsWMB1MDvk0jltmLFCtHc3Fz8+eefxdOnT4uvvPKKaG9vLyYnJ8tdWqXZsGGD+N5774mrVq0SAYirV682WP7ZZ5+JdnZ24po1a8QTJ06Iffv2Ff38/MTc3Fz9Oj179hSbNm0qHjhwQNyzZ49Yv359cejQofrl6enpopubmxgRESHGxMSIy5cvFzUajfjDDz/o14mKihKVSqU4Z84c8cyZM+L7778vmpmZiadOnaryNiiP8PBwcfHixWJMTIx4/Phx8amnnhJ9fHzErKws/Tqvv/666O3tLW7btk08fPiw+MQTT4jt27fXLy8qKhIbNWokhoWFiceOHRM3bNggOjs7i++++65+nUuXLomWlpbi5MmTxTNnzojffPONqFQqxU2bNunXMebv5dq1a8X169eL586dE+Pi4sT/+7//E83MzMSYmBhRFNlGZTl06JBYt25dsUmTJuLEiRP189lWkg8++EAMCQkRb9y4oZ9u3rypX852kqSmpoq+vr7iiBEjxIMHD4qXLl0SN2/eLF64cEG/Dn+fi2JKSorBdykyMlIEIO7YsUMUxZr3fWKoq4A2bdqIY8eO1b/XarWip6enOGvWLBmrqjr3hjqdTie6u7uLn3/+uX5eWlqaqFarxeXLl4uiKIpnzpwRAYjR0dH6dTZu3CgKgiAmJiaKoiiK33//vejg4CDm5+fr15k6darYsGFD/fvBgweLvXv3Nqinbdu24muvvVapn7GypKSkiADEXbt2iaIotYuZmZn4559/6teJjY0VAYj79+8XRVEK0AqFQkxKStKvs2DBAtHW1lbfNm+//bYYEhJicKwhQ4aI4eHh+vc17Xvp4OAg/vTTT2yjMmRmZooBAQFiZGSk2LlzZ32oY1uV+OCDD8SmTZuWuYztVGLq1Klihw4dHricv8/LNnHiRNHf31/U6XQ18vvEy6/lVFBQgCNHjiAsLEw/T6FQICwsDPv375exsupz+fJlJCUlGbSBnZ0d2rZtq2+D/fv3w97eHq1atdKvExYWBoVCgYMHD+rX6dSpE8zNzfXrhIeHIy4uDnfu3NGvU/o4xesYa1unp6cDABwdHQEAR44cQWFhocFnCAwMhI+Pj0FbNW7cGG5ubvp1wsPDkZGRgdOnT+vXeVg71KTvpVarxYoVK5CdnY127dqxjcowduxY9O7d+77Pw7YydP78eXh6eqJevXqIiIhAfHw8ALZTaWvXrkWrVq3w7LPPwtXVFc2bN8eiRYv0y/n7/H4FBQX4/fffMWrUKAiCUCO/Twx15XTr1i1otVqD/3AA4ObmhqSkJJmqql7Fn/NhbZCUlARXV1eD5SqVCo6OjgbrlLWP0sd40DrG2NY6nQ6TJk1CaGgoGjVqBECq39zcHPb29gbr3ttWj9sOGRkZyM3NrRHfy1OnTsHa2hpqtRqvv/46Vq9ejeDgYLbRPVasWIGjR49i1qxZ9y1jW5Vo27YtlixZgk2bNmHBggW4fPkyOnbsiMzMTLZTKZcuXcKCBQsQEBCAzZs3Y/To0ZgwYQJ++eUXAPx9XpY1a9YgLS0NI0aMAFAz/79TVWhtIrrP2LFjERMTg71798pdilFq2LAhjh8/jvT0dPz1118YPnw4du3aJXdZRiUhIQETJ05EZGQkLCws5C7HqPXq1Uv/ukmTJmjbti18fX3xxx9/QKPRyFiZcdHpdGjVqhU+/fRTAEDz5s0RExODhQsXYvjw4TJXZ5z+97//oVevXvD09JS7lMfGM3Xl5OzsDKVSeV+vl+TkZLi7u8tUVfUq/pwPawN3d3ekpKQYLC8qKkJqaqrBOmXto/QxHrSOsbX1uHHjsG7dOuzYsQNeXl76+e7u7igoKEBaWprB+ve21eO2g62tLTQaTY34Xpqbm6N+/fpo2bIlZs2ahaZNm+Lrr79mG5Vy5MgRpKSkoEWLFlCpVFCpVNi1axfmz58PlUoFNzc3ttUD2Nvbo0GDBrhw4QK/U6V4eHggODjYYF5QUJD+UjV/nxu6evUqtm7dipdfflk/ryZ+nxjqysnc3BwtW7bEtm3b9PN0Oh22bduGdu3ayVhZ9fHz84O7u7tBG2RkZODgwYP6NmjXrh3S0tJw5MgR/Trbt2+HTqdD27Zt9evs3r0bhYWF+nUiIyPRsGFDODg46NcpfZzidYylrUVRxLhx47B69Wps374dfn5+BstbtmwJMzMzg88QFxeH+Ph4g7Y6deqUwS/NyMhI2Nra6n8ZP6odauL3UqfTIT8/n21USvfu3XHq1CkcP35cP7Vq1QoRERH612yrsmVlZeHixYvw8PDgd6qU0NDQ+4ZZOnfuHHx9fQHw9/m9Fi9eDFdXV/Tu3Vs/r0Z+nyrUraKWW7FihahWq8UlS5aIZ86cEV999VXR3t7eoNdLTZeZmSkeO3ZMPHbsmAhA/PLLL8Vjx46JV69eFUVR6gJvb28v/vPPP+LJkyfFZ555pswu8M2bNxcPHjwo7t27VwwICDDoAp+Wlia6ubmJw4YNE2NiYsQVK1aIlpaW93WBV6lU4ty5c8XY2Fjxgw8+MJou8KIoiqNHjxbt7OzEnTt3GnSHz8nJ0a/z+uuviz4+PuL27dvFw4cPi+3atRPbtWunX17cFb5Hjx7i8ePHxU2bNokuLi5ldoWfMmWKGBsbK3733XdldoU31u/lO++8I+7atUu8fPmyePLkSfGdd94RBUEQt2zZIooi2+hhSvd+FUW2VbE333xT3Llzp3j58mUxKipKDAsLE52dncWUlBRRFNlOxQ4dOiSqVCpx5syZ4vnz58WlS5eKlpaW4u+//65fh7/PJVqtVvTx8RGnTp1637Ka9n1iqKugb775RvTx8RHNzc3FNm3aiAcOHJC7pEq1Y8cOEcB90/Dhw0VRlLrBT5s2TXRzcxPVarXYvXt3MS4uzmAft2/fFocOHSpaW1uLtra24siRI8XMzEyDdU6cOCF26NBBVKvVYp06dcTPPvvsvlr++OMPsUGDBqK5ubkYEhIirl+/vso+d0WV1UYAxMWLF+vXyc3NFceMGSM6ODiIlpaWYv/+/cUbN24Y7OfKlStir169RI1GIzo7O4tvvvmmWFhYaLDOjh07xGbNmonm5uZivXr1DI5RzFi/l6NGjRJ9fX1Fc3Nz0cXFRezevbs+0Iki2+hh7g11bCvJkCFDRA8PD9Hc3FysU6eOOGTIEIOx19hOJf7991+xUaNGolqtFgMDA8Uff/zRYDl/n0s2b94sArjvs4tizfs+CaIoihU7t0dERERExob31BERERGZAIY6IiIiIhPAUEdERERkAhjqiIiIiEwAQx0RERGRCWCoIyIiIjIBDHWPIT8/HzNmzEB+fr7cpRg1tlP5sJ3Kj21VPmyn8mE7lR/bqnzkbieOU/cYMjIyYGdnh/T0dNja2spdjtFiO5UP26n82Fblw3YqH7ZT+bGtykfuduKZOiIiIiITwFBHREREZAJUchdQ1YqKinDs2DG4ublBoaicDJuZmQkASExMREZGRqXs0xSxncqH7VR+bKvyYTuVD9up/NhW5VO6ndLS0pCcnIzmzZtDpaqeuGXy99RFR0ejTZs2cpdBREREtdChQ4fQunXrajmWyZ+pc3NzAyA1qoeHh8zVEBERUW1w48YNtGnTRp9DqoPJh7riS64eHh7w8vKSuRoiIiKqTSrr1q9yHavajkREREREVYahjoiIiMgEMNQRERERmQCTv6eOiIioNK1Wi8LCQrnLoBrOzMwMSqVS7jIMMNQREVGtIIoikpKSkJaWJncpZCLs7e3h7u4OQRDkLgUAQx0REdUSxYHO1dUVlpaWRvOHmGoeURSRk5ODlJQUADCaIdMY6oiIyORptVp9oHNycpK7HDIBGo0GAJCSkgJXV1ejuBTLjhJERGTyiu+hs7S0lLkSMiXF3ydjuUeToY6IiGoNXnKlymRs3yeGOiIiIiITwFBHRERUy9StWxfz5s0r9/o7d+6EIAhV3nN4yZIlsLe3r9JjmDKGOiIiIiMlCMJDpxkzZjzWfqOjo/Hqq6+We/327dvjxo0bsLOze6zjUfVg71ciIiIjdePGDf3rlStXYvr06YiLi9PPs7a21r8WRRFarRYq1aP/tLu4uFSoDnNzc7i7u1doG6p+PFP3H+UVarHr5AWcPxwpdylERGRi3N3d9ZOdnR0EQdC/P3v2LGxsbLBx40a0bNkSarUae/fuxcWLF/HMM8/Azc0N1tbWaN26NbZu3Wqw33svvwqCgJ9++gn9+/eHpaUlAgICsHbtWv3yey+/Fl8m3bx5M4KCgmBtbY2ePXsahNCioiJMmDAB9vb2cHJywtSpUzF8+HD069evQm2wYMEC+Pv7w9zcHA0bNsRvv/2mXyaKImbMmAEfHx+o1Wp4enpiwoQJ+uXff/89AgICYGFhATc3NwwaNKhCx65pGOr+o2X/bkLbv59AnQ0jgKJ8ucshIqJyEkUROQVFskyiKFba53jnnXfw2WefITY2Fk2aNEFWVhaeeuopbNu2DceOHUPPnj3Rp08fxMfHP3Q/H374IQYPHoyTJ0/iqaeeQkREBFJTUx+4fk5ODubOnYvffvsNu3fvRnx8PN566y398tmzZ2Pp0qVYvHgxoqKikJGRgTVr1lTos61evRoTJ07Em2++iZiYGLz22msYOXIkduzYAQD4+++/8dVXX+GHH37A+fPnsWbNGjRu3BgAcPjwYUyYMAEfffQR4uLisGnTJnTq1KlCx69pePn1P2rSrA3unLCBhy4V2rMboWzUT+6SiIioHHILtQievlmWY5/5KByW5pXzJ/ijjz7Ck08+qX/v6OiIpk2b6t9//PHHWL16NdauXYtx48Y9cD8jRozA0KFDAQCffvop5s+fj0OHDqFnz55lrl9YWIiFCxfC398fADBu3Dh89NFH+uXffPMN3n33XfTv3x8A8O2332LDhg0V+mxz587FiBEjMGbMGADA5MmTceDAAcydOxddu3ZFfHw83N3dERYWBjMzM/j4+KBNmzYAgPj4eFhZWeHpp5+GjY0NfH190bx58wodv6bhmbr/qHldZ2wSOgIAMg789oi1iYiIKlerVq0M3mdlZeGtt95CUFAQ7O3tYW1tjdjY2EeeqWvSpIn+tZWVFWxtbfWPwSqLpaWlPtAB0qOyitdPT09HcnKyPmABgFKpRMuWLSv02WJjYxEaGmowLzQ0FLGxsQCAZ599Frm5uahXrx5eeeUVrF69GkVFRQCAJ598Er6+vqhXrx6GDRuGpUuXIicnp0LHr2l4pu4/UioEpNTrD1z6B7aJO4DsW4CVs9xlERHRI2jMlDjzUbhsx64sVlZWBu/feustREZGYu7cuahfvz40Gg0GDRqEgoKCh+7HzMzM4L0gCNDpdBVavzIvK5eHt7c34uLisHXrVkRGRmLMmDH4/PPPsWvXLtjY2ODo0aPYuXMntmzZgunTp2PGjBmIjo422WFTeKauEoQ0a4uTOj8oRS0Q87fc5RARUTkIggBLc5UsU1U+iSAqKgojRoxA//790bhxY7i7u+PKlStVdryy2NnZwc3NDdHR0fp5Wq0WR48erdB+goKCEBUVZTAvKioKwcHB+vcajQZ9+vTB/PnzsXPnTuzfvx+nTp0CAKhUKoSFhWHOnDk4efIkrly5gu3bt/+HT2bceKauEnQMcMHXuk5ooriM/CNLoW77mtwlERFRLRUQEIBVq1ahT58+EAQB06ZNe+gZt6oyfvx4zJo1C/Xr10dgYCC++eYb3Llzp0KBdsqUKRg8eDCaN2+OsLAw/Pvvv1i1apW+N++SJUug1WrRtm1bWFpa4vfff4dGo4Gvry/WrVuHS5cuoVOnTnBwcMCGDRug0+nQsGHDqvrIsuOZukpgpzFDYp2nUCgqoU45AaSclbskIiKqpb788ks4ODigffv26NOnD8LDw9GiRYtqr2Pq1KkYOnQoXnzxRbRr1w7W1tYIDw+HhYVFuffRr18/fP3115g7dy5CQkLwww8/YPHixejSpQsAwN7eHosWLUJoaCiaNGmCrVu34t9//4WTkxPs7e2xatUqdOvWDUFBQVi4cCGWL1+OkJCQKvrE8hPE6r4AXs2uXbsGb29vJCQkwMvLq8qO89OeS/Dd8hKeVB4FOrwBhM2osmMREVHF5OXl4fLly/Dz86tQqKDKo9PpEBQUhMGDB+Pjjz+Wu5xK8bDvVXXlj9J4pq6SdAt0xSqt1AtWd2IloNPKXBEREZF8rl69ikWLFuHcuXM4deoURo8ejcuXL+P555+XuzSTxVBXSeq5WOOiQweki5ZQZF4HruyRuyQiIiLZKBQKLFmyBK1bt0ZoaChOnTqFrVu3IigoSO7STBY7SlSijkFeWHegHSJU24ATK4B6XeQuiYiISBbe3t739VylqsUzdZWoe6Ar/r57CVY8sxbIz5K5IiIiIqotGOoqUau6jjhvHoQ/izrhauingNLs0RsRERERVQKGukpkrlKgUwNXTCl6HasK2gEqtdwlERERUS3BUFfJugW6AgC2xj74eXlERERElY2hrpJ1aegCQQBu3biKjK1zgOPL5C6JiIiIagGGukrmZK1Gc297dFUeh+3emUDU14Bpj+9MRERERoChrgp0D3LDBm1bnLFoDrQbC4jV/8w9IiKiYl26dMGkSZP07+vWrYt58+Y9dBtBELBmzZr/fOzK2s/DzJgxA82aNavSY9QEDHVVoHuQKzJhif5ZU5HbKAJQKOUuiYiIaqA+ffqgZ8+eZS7bs2cPBEHAyZMnK7zf6OhovPrqq/+1PAMPClY3btxAr169KvVYVDaGuirQ0M0Gdew1yC/SYf+lW3KXQ0RENdRLL72EyMhIXLt27b5lixcvRqtWrdCkSZMK79fFxQWWlpaVUeIjubu7Q63maBDVgaGuCgiCoO8Fu/9UHHBgIXCFo2oTEVHFPP3003BxccGSJUsM5mdlZeHPP//ESy+9hNu3b2Po0KGoU6cOLC0t0bhxYyxfvvyh+7338uv58+fRqVMnWFhYIDg4GJGRkfdtM3XqVDRo0ACWlpaoV68epk2bhsLCQgDAkiVL8OGHH+LEiRMQBAGCIOhrvvfy66lTp9CtWzdoNBo4OTnh1VdfRVZWyWD9I0aMQL9+/TB37lx4eHjAyckJY8eO1R+rPHQ6HT766CN4eXlBrVajWbNm2LRpk355QUEBxo0bBw8PD1hYWMDX1xezZs0CAIiiiBkzZsDHxwdqtRqenp6YMGFCuY8tJz4mrIp0C3LFbweuot7ZH4HT/wLBzwB1Q+Uui4iI7lWQXfFtlGpAefdPqLYI0OYDggIw0zx6v+ZW5T6MSqXCiy++iCVLluC9996DIAgAgD///BNarRZDhw5FVlYWWrZsialTp8LW1hbr16/HsGHD4O/vjzZt2jzyGDqdDgMGDICbmxsOHjyI9PR0g/vvitnY2GDJkiXw9PTEqVOn8Morr8DGxgZvv/02hgwZgpiYGGzatAlbt24FANjZ2d23j+zsbISHh6Ndu3aIjo5GSkoKXn75ZYwbN84guO7YsQMeHh7YsWMHLly4gCFDhqBZs2Z45ZVXytVuX3/9Nb744gv88MMPaN68OX7++Wf07dsXp0+fRkBAAObPn4+1a9fijz/+gI+PDxISEpCQkAAA+Pvvv/HVV19hxYoVCAkJQVJSEk6cOFGu48qNoa6KtKvnBI2ZEr/ltMNQ9b9A3EYg9w6gcZC7NCIiKu1Tz4pv8+wSIKS/9Prsv8CfIwDfDsDI9SXrzGsM5Ny+f9sZ6RU61KhRo/D5559j165d6NKlCwDp0uvAgQNhZ2cHOzs7vPXWW/r1x48fj82bN+OPP/4oV6jbunUrzp49i82bN8PTU2qLTz/99L774N5//33967p16+Ktt97CihUr8Pbbb0Oj0cDa2hoqlQru7u4PPNayZcuQl5eHX3/9FVZWUrj99ttv0adPH8yePRtubm4AAAcHB3z77bdQKpUIDAxE7969sW3btnKHurlz52Lq1Kl47rnnAACzZ8/Gjh07MG/ePHz33XeIj49HQEAAOnToAEEQ4Ovrq982Pj4e7u7uCAsLg5mZGXx8fMrVjsaAl1+riIWZEqH1nXFG9MUtq/qAtgA4vVrusoiIqIYJDAxE+/bt8fPPPwMALly4gD179uCll14CAGi1Wnz88cdo3LgxHB0dYW1tjc2bNyM+Pr5c+4+NjYW3t7c+0AFAu3bt7ltv5cqVCA0Nhbu7O6ytrfH++++X+xilj9W0aVN9oAOA0NBQ6HQ6xMXF6eeFhIRAqSzpZOjh4YGUlPIN6p+RkYHr168jNNTw6lhoaChiY2MBSJd4jx8/joYNG2LChAnYsmWLfr1nn30Wubm5qFevHl555RWsXr0aRUVFFfqccuGZuirUPcgVW2OTsU7ojBG4AJxYAbQaJXdZRERU2v9dr/g2ylI3/gf2kfYh3HOeZNKp/1ZXKS+99BLGjx+P7777DosXL4a/vz86d+4MAPj888/x9ddfY968eWjcuDGsrKwwadIkFBQUVNrx9+/fj4iICHz44YcIDw+HnZ0dVqxYgS+++KLSjlGamZnhs9MFQYBOV3nDg7Vo0QKXL1/Gxo0bsXXrVgwePBhhYWH466+/4O3tjbi4OGzduhWRkZEYM2aM/kzpvXUZG1nP1O3evRt9+vSBp6dnmePYiKKI6dOnw8PDAxqNBmFhYTh//rw8xT6Grg2lzhLf324BUVAACQeB2xdlroqIiAyYW1V8UpY6J6JUSfNK30/3sP0+hsGDB0OhUGDZsmX49ddfMWrUKP39dVFRUXjmmWfwwgsvoGnTpqhXrx7OnTtX7n0HBQUhISEBN27c0M87cOCAwTr79u2Dr68v3nvvPbRq1QoBAQG4evWq4cc1N4dWq33ksU6cOIHs7JL7DaOioqBQKNCwYcNy1/wwtra28PT0RFSUYQfFqKgoBAcHG6w3ZMgQLFq0CCtXrsTff/+N1NRUAIBGo0GfPn0wf/587Ny5E/v378epU5UX0quKrKEuOzsbTZs2xXfffVfm8jlz5mD+/PlYuHAhDh48CCsrK4SHhyMvL6+aK3087nYWaFTHFimiA5Kd20szT66UtygiIqpxrK2tMWTIELz77ru4ceMGRowYoV8WEBCAyMhI7Nu3D7GxsXjttdeQnJxc7n2HhYWhQYMGGD58OE6cOIE9e/bgvffeM1gnICAA8fHxWLFiBS5evIj58+dj9WrDW4rq1q2Ly5cv4/jx47h16xby8/PvO1ZERAQsLCwwfPhwxMTEYMeOHRg/fjyGDRumv5+uMkyZMgWzZ8/GypUrERcXh3feeQfHjx/HxIkTAQBffvklli9fjrNnz+LcuXP4888/4e7uDnt7eyxZsgT/+9//EBMTg0uXLuH333+HRqMxuO/OWMka6nr16oVPPvkE/fv3v2+ZKIqYN28e3n//fTzzzDNo0qQJfv31V1y/fr3KR6auTN0CpS/pBqV0mhwnlgOVeAqZiIhqh5deegl37txBeHi4wf1v77//Plq0aIHw8HB06dIF7u7u6NevX7n3q1AosHr1auTm5qJNmzZ4+eWXMXPmTIN1+vbtizfeeAPjxo1Ds2bNsG/fPkybNs1gnYEDB6Jnz57o2rUrXFxcyhxWxdLSEps3b0Zqaipat26NQYMGoXv37vj2228r1hiPMGHCBEyePBlvvvkmGjdujE2bNmHt2rUICAgAIPXknTNnDlq1aoXWrVvjypUr2LBhAxQKBezt7bFo0SKEhoaiSZMm2Lp1K/799184OTlVao1VQRBF43gwqSAIWL16tf6LeOnSJfj7++PYsWMGI1R37twZzZo1w9dff12u/V67dg3e3t5ISEiAl5dXFVT+cCcS0vDMd1FwVmsRrR4DoSATGLkR8G1f7bUQEdVWeXl5uHz5Mvz8/GBhYSF3OWQiHva9kiN/GG3v16SkJAC473Ssm5ubfllZ8vPzkZGRoZ8yMzOrtM5HaVzHDs7WatzKVyLZ++6jXk48fFBIIiIioooy2lD3uGbNmqUft8fOzs7gpkg5KBQCugW6AAA2KbtIM0+vAQpzZauJiIiITI/RhrriwQvvvdkzOTn5oQMbvvvuu0hPT9dPZ86cqdI6y6P4vrrFCR4Q7byB/AwgboPMVREREZEpMdpQ5+fnB3d3d2zbtk0/LyMjAwcPHixzUMRiarUatra2+snGxqY6yn2ojgHOMFcqcPVOHu7UHyDNjP1X3qKIiIjIpMg6+HBWVhYuXLigf1/cFdrR0RE+Pj6YNGkSPvnkEwQEBMDPzw/Tpk2Dp6dnhXr1GAMrtQpP+Dth97mb2GT+JJ5/7gmgfpjcZREREZEJkTXUHT58GF27dtW/nzx5MgBg+PDhWLJkCd5++21kZ2fj1VdfRVpaGjp06IBNmzbVyJ5L3QNdsfvcTfxzRYXnw5+SuxwiolqpMp9KQGRs3yejGdKkqsg9pEmxhNQcdJyzA0qFgKPvPwk7SzNpvDqF0V4BJyIyGTqdDufPn4dSqYSLiwvMzc31T2QgqihRFFFQUICbN29Cq9UiICAAinv+nsuRP/js12ri7WiJBm7WOJechV3nb6Jv2lLg6K/A0OWAeyO5yyMiMmkKhQJ+fn64ceMGrl9/jGe9EpXB0tISPj4+9wU6uTDUVaNugW44l5yF7bHJ6CucANLjgVN/MtQREVUDc3Nz+Pj4oKio6JHPKCV6FKVSCZVKZVRnfBnqqlH3IFcs3HURO8/dRNHwcVCF9Aca8v46IqLqIggCzMzMYGZmJncpRJXOOM4X1hLNve1hb2mGtJxCHBMbAI0GAmYaucsiIiIiE8BQV41USgW6NJCeLrEtNkXmaoiIiMiUMNRVs25B0tMltp9NBrSFwO65wMKOQF6GzJURERFRTcZQV806B7hAqRBwLjkLCWkFwMk/gKSTQOxauUsjIiKiGoyhrprZWZqhla8DAGDb2RSg6XPSghMrZKyKiIiIajqGOhl0D3IFcDfUNRkMQACu7AHS4uUtjIiIiGoshjoZdAuU7qs7eCkVWRbugF8nacGJlTJWRURERDUZQ50M/F2s4OtkiQKtDnvP3wKaDpUWnFgOmPZT24iIiKiKMNTJQBAEdAuULsFuP5sMBPUBzCyB1IvAtcMyV0dEREQ1EUOdTML0Q5vchM7MCgjqKy04sVzGqoiIiKimYqiTSeu6jrBWq3ArKx+nEtNLesHG/A0U5ctbHBEREdU4DHUyMVcp0KmBM4C7vWD9OgE2nkBeGnBus7zFERERUY3DUCej4l6w288mAwrl3eFNABxfJmNVREREVBMx1MmoS0MXCAIQk5iBpPQ8oNnz0oLzW4CMG/IWR0RERDUKQ52MnK3VaOZtDwDYEZcCuDQEvJ8AbDyAO5flLY6IiIhqFIY6mXW/O7TJttgUacbgX4BJJwHf9jJWRURERDUNQ53Miu+ri7pwC3mFWsDGXbq/joiIiKgCGOpkFuRhAw87C+QWarH/4u2SBdpCIPGIfIURERFRjcJQJ7PST5fYdjZZmpmVAnwVAvzcE8hJlbE6IiIiqikY6oxA96C7jwyLTYEoioC1K2DtBljYATfjZK6OiIiIagKV3AUQ0N7fGRZmClxPz8PZpEwEedgCQ34HbD0BpZnc5REREVENwDN1RsDCTIlQf+npEtvP3u0F6+DLQEdERETlxlBnJLoFFQ9tkmy4QKcFUmJlqIiIiIhqEoY6I1HcWeJYQhpuZ+VLM9MTga+bAou6A/mZMlZHRERExo6hzkh42GkQ4mkLUQR2xt2UZtp6Aio1UJgNxKySt0AiIiIyagx1RqT46RJbziRJMwQBaPGi9ProLzJVRURERDUBQ50R6dXYA4DUWUJ/Cbbp84BCJQ1EnBQjY3VERERkzBjqjEiQhy2aeNmhUCti9bFEaaa1C9DwKen1sd/kK46IiIiMGkOdkRncyhsAsDI6QRqIGABaDJd+nlgBFObJVBkREREZM4Y6I9O3mScszBQ4n5KFYwlp0kz/roCdN5CXBsT+K2d5REREZKQY6oyMrYUZnrp7b93KQwnSTIUSaP6C9JodJoiIiKgMDHVGaMjdS7DrTl5Hdn6RNLP5CwAE4Moe4PZF+YojIiIio8RQZ4Ta+DnCz9kK2QVarD95Q5pp5wXUD5Nes8MEERER3YOhzggJglDSYeJwQsmC4jHrji8DtIUyVEZERETGiqHOSA1sWQdKhYAjV+/gQsrdR4Q17AVYuQBZycD5LfIWSEREREaFoc5IudpYoGtD6QkTK6Pvnq1TmgEd3gC6TQPqtJSxOiIiIjI2DHVGbEhr6RLsqqOJKCjSSTPbjQU6vQXYuMtYGRERERkbhjoj1rWhC1xs1LidXYDtZ5PlLoeIiIiMGEOdEVMpFRjU0gsAsCK6VIcJbSEQswpY9Sqg08lUHRERERkThjojV9wLdve5m7iRnivN1BUB/04CTq4ELu+UrTYiIiIyHgx1Rs7P2Qpt/ByhE4G/Dl+TZppp7t5bNwVwbihvgURERGQUGOpqgOInTPxxJAE6nSjN7DIV6PY+YFdHxsqIiIjIWDDU1QBPNfaAjVqFhNRc7L90W+5yiIiIyAgx1NUAGnMl+jbzBFBqzDoA0BYBcRuBdZMBUZSpOiIiIjIGDHU1RPGYdZtOJyE95+4jwgpzgL9GAYf/ByQckrE6IiIikhtDXQ3RuI4dAt1tUFCkw5rjidJMC1sgZID0+uiv8hVHREREsmOoqyEEQcBzd8/WrYhOgFh8ubXFi9LP06uAvAyZqiMiIiK5MdTVIP2a14G5SoHYGxmISbwb4LzbSMOaFOYAMX/JWyARERHJhqGuBrG3NEd4iPTM15WH46WZggC0HC695iVYIiKiWouhroYpHrPun+PXkVeolWY2eQ5QmAHXjwE3TspYHREREcnFqEOdVqvFtGnT4OfnB41GA39/f3z88ccl95PVQu39neDloEFmXhE2xtyQZlo5AUFPS695to6IiKhWMupQN3v2bCxYsADffvstYmNjMXv2bMyZMwfffPON3KXJRqEQ9M+DXXGo1Jh1xR0mTv4BFObKUBkRERHJyahD3b59+/DMM8+gd+/eqFu3LgYNGoQePXrg0KHaPSbboJZeEATg4OVUXLmVLc306wLY+wD56cCZtXKWR0RERDIw6lDXvn17bNu2DefOnQMAnDhxAnv37kWvXr1krkxenvYadApwAQD8cfju2TqFAmh+92zd0V9kqoyIiIjkYtSh7p133sFzzz2HwMBAmJmZoXnz5pg0aRIiIiIeuE1+fj4yMjL0U2ZmZjVWXH2KnzDx15FrKNLqpJnNngcEBXA1Crh1QcbqiIiIqLoZdaj7448/sHTpUixbtgxHjx7FL7/8grlz5+KXXx58JmrWrFmws7PTT8HBwdVYcfUJC3KDo5U5UjLzsTPupjTTrg4Q0EPqCZt4WN4CiYiIqFoJohF3JfX29sY777yDsWPH6ud98skn+P3333H27Nkyt8nPz0d+fr7+fWJiIoKDg5GQkAAvL68qr7k6fbLuDH7aexlPBrth0YutpJm3LwJqW8DaRd7iiIiIarFr167B29u7WvOHUZ+py8nJgUJhWKJSqYROp3vgNmq1Gra2tvrJxsamqsuUTfEl2O1nU5CSmSfNdPJnoCMiIqqFjDrU9enTBzNnzsT69etx5coVrF69Gl9++SX69+8vd2lGIcDNBs197KHViVh1NPH+FTJuVH9RREREJAujDnXffPMNBg0ahDFjxiAoKAhvvfUWXnvtNXz88cdyl2Y0ip8w8Ud0QsmgzEX5wOLewFfBQFrCQ7YmIiIiU2HUoc7Gxgbz5s3D1atXkZubi4sXL+KTTz6Bubm53KUZjaebesLSXIlLt7IRfeWONFOllp4JCwDx++UrjoiIiKqNUYc6ejRrtQpPN/EAAKyMLnVWrtdsYFIM0GSwTJURERFRdWKoMwHFHSY2nLqBzLxCaaZbiDTECREREdUKDHUmoIWPA/xdrJBbqMW/J8roHHHtMBB/sPoLIyIiomrDUGcCBEHAc619AAAro+MNF8b8DfwUBqwZLXWgICIiIpPEUGci+reoA5VCwIlr6TiblFGyoH4YYO0KpF4Eor6Wr0AiIiKqUgx1JsLZWo2wIDcA93SYsLADwj+VXu+eC6RekqE6IiIiqmoMdSakuMPE6mOJyC/SlixoNBDw6wxo84ENUwDjfTIcERERPSaGOhPSqYEL3G0tkJZTiC2nk0sWCALQ+0tAaQ5c2Aqc+Ue+IomIiKhKMNSZEKVCwLOtpIcG/3H4nidJONcHQidJrze9C+RnVm9xREREVKUY6kzMsy2lS7B7L9zCtTs5hgs7TgYc6gKZ14Eds6q/OCIiIqoyDHUmxsfJEu39nSCKwJ+HrxkuNNMAT30hvT64EEg6Vf0FEhERUZVgqDNBxR0m/jycAK3unk4RAWFA8DOAqAXWTQZ0OhkqJCIiosrGUGeCwkPcYacxw/X0POy9cOv+FXp+BphbA9cOAcd+q/4CiYiIqNIx1JkgCzMl+jXzBAD8tv/q/SvYegJd/096vf0TPmmCiIjIBDDUmahh7XyhEICtsck4cjX1/hXavAa0HAEMXwuo1NVeHxEREVUuhjoTVd/VBoNbSffWfbrhLMR7BxxWqoA+XwOuQTJUR0RERJWNoc6EvfFkA2jMlDhy9Q42n056+MrJpwFtYfUURkRERJWOoc6Eudla4JWOfgCA2ZviUKh9QE/XnbOBhR2AAwuqsToiIiKqTAx1Ju7Vzv5wsjLH5VvZWH4ovuyV7OoAog64FVe9xREREVGlYagzcdZqFSaFBQAAvt56Hpl5ZVxibfo8MGoL8Mx31VwdERERVRaGulrguTY+qOdshdvZBfhh16X7V1AoAJ+21V8YERERVRqGulrATKnA2z0DAQA/7b2EpPS8B6+ccR1Y/xZQkPPgdYiIiMjoMNTVEuEhbmjl64C8Qh2+ijxX9kqiCPw2AIheBOyZW70FEhER0X/CUFdLCIKAd5+SxqT780gC4pIyy1oJ6Pa+9DpqPnCTHSeIiIhqCoa6WqSlrwOeauwOnQh8tjG27JUCewMNegK6QmD9m9LZOyIiIjJ6DHW1zJTwQKgUAnbE3cS+C7fuX0EQgF5zAJUGuLIHOPlH9RdJREREFcZQV8v4OVshoq0PAODTjbHQ6co4E+fgC3SeIr3e8h6Qe6caKyQiIqLHwVBXC03oHgBrtQoxiRlYe+J62Su1Gw84NwSybwLbPq7eAomIiKjCGOpqISdrNUZ38QcAfL45DnmF2vtXUpkDvb+QXh/+Gbh2pBorJCIioopiqKulRoX6wd3WAolpufht/9WyV/LrCDR5DoAIrH8D0JUR/oiIiMgoMNTVUhpzJSb3aAAA+Gb7eaTlFJS9Yo9PAAs74MYJYMMU4NaFaqySiIiIyouhrhYb2MILge42yMgrwnc7HhDWrF2AsBnS68P/A5YO5DAnRERERoihrhZTKgS800t6fNgv+64iIfUBjwZrORIY9DNQP0y6HCsI0vyifOC3/sCBhdJrIiIikg1DXS3XuYELQus7oUCrw9wtD3iChCAAjQYCL/wNdHmnZP7F7dIUNQ9QmJXMZ8AjIiKqdgx1tZwgCHi3l/T4sH+OX8fJa2mP2qDkdZ1WQM/ZQIfJgOLuV0lbBHzdDPh9IHB8GZCXXiV1ExERkSGV3AWQ/BrVsUP/5nWw+lgiPt0Qi+WvPAGhdHh7EGsX4InXDeclHgEyr0vTha2A0hwI6AE0GiA9fszcqmo+BBERUS3HM3UEAHizRwOYqxQ4cCkVO+NuPv6OfNoC4w4DXf4PcG4AaAuAs+uAv0YBn9cH/hwJxK6TetGmJQBZKUB+ZuV9ECIiolpKEEXT7sp47do1eHt7IyEhAV5eXnKXY9RmbYjFD7svoYGbNTZM6AiV8j9mflEEkk8Dp1cBMX8Dd66UvZ5HU+C13SXvF3QAspKAiD8Bz+bSvJN/AtE/SYMiqyykM4AqtfTaTAOYWUpnAc2tSl5rHIEGPUr2m3EdEBTSfJX5f/tsREREDyFH/uDlV9Ib06U+VkQn4FxyFv4+eg1DWvv8tx0KAuDeSJq6TQOuHwViVgFn1wM5t6UOFdp8QKk23C7nlvR4MqFUqEy7CiQcqNjx7X2ABqdK3i8fCtw4Djz/B9AgXJoX8zcQ+cHdIGgJ2HgADn6Ao1/JTztvhkAiIjJ6DHWkZ2dphvHd6uOT9bH4Yss59GnqCUvzSvqKCAJQp6U0hc8smS+KgK7IcN2RG4HCXClQFQvuJ13OLQ6CRcVTnjQVZEtTYU7Ja2vX+2sQFFKAK5aTCqQnlFrpWBm1KwA7r5KQ59wAaDf2cVuCiIioSvDyKxnIL9Ii7MtdSEjNxZtPNsD47gFyl1S5ir/uxR1Bsm8DaVdKgmD6Nekycepl4M5l6WdRruE+HOsBE0qFv2VDpHDY6zMptAJA7h3pp8ahKj8NEREZKV5+JdmpVUpMCQ/EhOXHsHDXRTzXxgcuNupHb1hT3Nur18pJmh5EFIGsZCD1UknQK32mDwCuRUuXk0uP1Xf0NyByGmDjCbgGAW7BgGuI9NO5IWBmUXmfiYiICAx1VIanG3vgpz2XcPJaOuZvO4+P+zWSuyT5CAJg4y5Nvu3LXueFVVLYc6pfMi/ntvSzeHiXi9tK7VMBOPqXBD3XIMAtBHCoCyiUVfZRiIjItPHyK5Vp/8XbGLroAJQKAVve6AR/F2u5S6p58jKAlFgg5Yw0JZ8BUk6XXJq9V72uwItrSt5fiZKCorXr/WcYiYjIqPHyKxmNdv5O6B7oim1nUzBn01n8MKyV3CXVPBa20rh9Pm1L5hVfzk0+XSronQFungWcS92/mJ8JLHlKev32ZcDSUXod+680tp+9L2DvLfXMNb/ncjAREdVKDHX0QO/0CsSOuBRsPp2Mw1dS0aquo9wl1XylL+fW714yX6eVeu4Wy0qRztIV5pYEOgA4/LP0vN3SLJ2lgGfvI4U8e59Sr70BC7uq/UxERMV0OmlEA1ErjSNafJUhP1MascBMU/JkIW3R3VtV7l4wNLhweO+8Usus3QDl3XuY8zKAgizpXmeN/d0atED2Lem1/iqHcP97cyuTu7+ZoY4eKMDNBkNae2P5oQR8uiEWf49uX77Hh1HFKZSA2qbkvZM/MP4IoC00XM+3vTTwcloCkBYPFGRK4/rl3AKulzEcCwC0HS31zAWkHr7HlwHujQGvNiXP7CWiqpF9S/rHmZVLSYDITAJuX7g7RFNBqZ95D5hXIA22HvZByX4jpwNJp4BOU0ru972wDdjy/t0VSv2uvi/YwPD9a7tL1tk6A7i0E+jwBhD8jDQvIRpY9YoU1HRaKbTpJ53h+9Lha8qlko5okR8Ah/8HdH4H6PquNO/2eeD7JyrepmMOAq6B0uv93wG7PgNavwz0/kKal5MKfNHg0fvp/yPQdEjFj2/EGOrood4Ia4A1x67jaHwa/jl+Hf2a15G7pNpFaWb4vtOUkteiCOSlSeEuLUEaby8tXpqKX+feAWzcSra5cwXY8BagtgWmXi2Zf3q1FBbdGkln+RjeyVTpdEBhthRASg85dHG79P9L/bCSs9vntwLnNt0d8iir5GdhjhS0tAWlAlihNIamYz1gzP6S/S5+CrgVBwxfB/h1lObF/iv9f1gRGgfDUJd4FLiyB2gWUTIvP0O6naOiSv//nnpZ+gdiZnLJPG2B1Bmsou4dgxSAQegDSg0y/7AQ+qD3kP5BrFAZDlZfvN+yzvKZOIY6eihXWwu80qke5m87j0krj2NTTBLeCm+A+q42j96YqpYgSL/oNQ7So9bKkp9peElDoQICwqUzBqXP0m37GEi9KL1W20m9cd0bSSHPvbHUQ9dMU3WfhWquonwgP0v6PhbfKqDTSkP9iCLg3aakV/fNOOksFcS738vSP3H/fF0RYOUM+JQ6mxP1tXTm64kx0n2rAHDyDyl86QckLzUwefHrwruDlBdmS9v4tANGbSrZ7+rXpftdX98rfecB4MYxIHpRxdqj8J5xLc0spKfmlA44lk7SIOZK81KPPFRL66nM7/60KPVabXgmHwBCJwHNXwC8WpfM8+0AvLgWBiHm3mBz3/t76u/whhQUXUqd6XJvBIzaLP3+KA5R+qnUe0FpOK/4MisgnUV7+kvDY7kGAR88oONYeXV+W5pKs3Z58H5FEfeNV2pCGOrokcZ08UdSei7+OnINm04nYcuZJAxs4YVJTzZAHXv+oTdq9/4hcGkIRPxhOE+nk/7AmVsCKWeB/HQgfp80FRMUgFOAYdDzasXBlR+XTnf3ySh3L62VDh/Fl9zsvKRhbgDpclLsWukPZfMXSvZz8g/pmcbFf1SVpf/Ymkl/XJVmhn+E7bxL/mBri6Qe2YDhPwxi/5U67+RnljFlGL7XFkjbNB8GPPOt9LooH/j57qP43k0E1Hd7z0d9DRxfWrG2Cgg3/M7umCUNCN50aEmoSzolPfKvIgqyDd97tQby0g3Hm/RpD3R6u+S50ubWd39aSmFLaV4SvIpfq+75nVj6udbFGg2Qpv8iIOz+edYugHXn/7Zfz2b3z7OwMwzWj8NYApQgGE8tVYChjh7JwkyJOYOa4uWO9TB3cxy2nEnGn0eu4Z/j1xHxhA/Gdq0PZ2sTGqC4tlEogH7fSa+LCoBb54DkGOkPZXIMkBQj3bN3K06aiv94DvoZaDRQen1uC3BwAeDXSfqXfrGz6wELe+nMhJWzFAKNYSy+wjzpBm1BAdh6lMy/uOPuDd150hmXwty74av4Utvdy2zF9ztpC6Rw4d9V2v76cWDTO1IgG/hTyX4XPyWdpSoObMVB6GG6/B/QZar0OvMG8O9E6b6s0qHu8GLD8F0erV8Bes+VXufeAX7oJAWSaTdL1jm+DIjbULH9FuWXvC4eixEw/ANq4wG4BAIo/sNa+ifuny8oDXuFA0DzCEDUGZ45btgLsPW8e8bLooyfd1+bW0v/0DG3kuaV9lwZYbNuqDQR1RAMdVRuDdxs8OOLrXA0/g4+3xSH/ZduY3HUFfwRnYCXOtbDKx39YGNh9ugdkfFSmUtn49wbAU2fk+aJonTJ7N6g596kZLtbcdI9SZpSPXW1hcCK5+85gCD1ULN0Lgl6lo7Se5VaumzXbKh0XxIAxB8ATv0JuAYDrV8q2c1fo6T933fT9t332nwpuBXllvzs+y0Q3FfaPm4D8NdI6XLVyPUl+/37pZKBo8vLs0VJqCvIBuL3Gw5EDUjhKefWA3YglBFE1CU9+QApDDd86v4zowFPSs8j1hbe0w53J4M2uruOXen7YkUpaAnKuzfjm0uz63WV/vuobaUQZDCVMc/cWjpLWMzMAphw9P6P2n2aNP0XxTfDl+bb/sGDgxPVIgx1VGEtfByw7JW22HvhFuZsisOpROnJE7/tv4IxXepjWDtfWJgZwdkYqhyCIJ3NsvWQQkRZAsKlEGBbKjAU5gDebaXefzm3pU4dEKWAk3tH6vlWFt92JaHuZhwQ/ZMUaEqHujNrpZBSEaUvt5lp7t4DdM9lGM8W0pk6MwvpMprZ3bM8pe99UpqVvFeaG16WcgkEBv96/zAyz/4i9RxUmhsGN5VF2XXcy9YTGLr8/vkdJ1esDe5l7Qq8efb++W1f/W/7JSJZGP0TJRITEzF16lRs3LgROTk5qF+/PhYvXoxWrco3GC6fKFG1RFHExpgkzN0Sh0s3pT+aHnYWmNg9AINaekGl5JAZdJe2CMhNlQJecdDLuSXdL5Z9SwppglIamqB4uILrx4G4jdIluMaDSvZ1aJEUhAxu2L57g7aglMKTQTDTSKG0+B5DUTTp+2qISH5y5A+jDnV37txB8+bN0bVrV4wePRouLi44f/48/P394e/vX659MNRVjyKtDquOJuKrredwIz0PAFDPxQpvPtkQvRq5Q6HgH1AiIqo9akyoS0hIgCAI+iIPHTqEZcuWITg4GK++Wnmn7d955x1ERUVhz549j70PhrrqlVeoxe8HruL7nReRmi3dDN64jh2mhDdExwBnDl5MRES1ghz547GujT3//PPYsWMHACApKQlPPvkkDh06hPfeew8fffRRpRW3du1atGrVCs8++yxcXV3RvHlzLFr08DGD8vPzkZGRoZ8yMzMrrR56NAszJV7uWA+7pnTBxO4BsDJX4lRiOl78+RCGLjqAo/H/cUwiIiIiKtNjhbqYmBi0adMGAPDHH3+gUaNG2LdvH5YuXYolS5ZUWnGXLl3CggULEBAQgM2bN2P06NGYMGECfvnllwduM2vWLNjZ2emn4ODgSquHys/GwgxvPNkAu9/uilGhfjBXKnDgUioGfL8PL/9yGLE3MuQukYiIyKQ81uVXa2trxMTEoG7duujbty9CQ0MxdepUxMfHo2HDhsjNzX30TsrB3NwcrVq1wr59JeMwTZgwAdHR0di/f3+Z2+Tn5yM/v2S8pMTERAQHB/Pyq8wS03Lx9dZz+OvINejufuO6BbpidBd/tK7r+PCNiYiIapgac/k1JCQECxcuxJ49exAZGYmePXsCAK5fvw4nJ6dKK87Dw+O+M21BQUGIj49/4DZqtRq2trb6ycaGj7MyBnXsNZgzqCm2vNEJvZt4QBCA7WdT8OzC/Xh24T7sOJsCI+6zQ0REZPQeK9TNnj0bP/zwA7p06YKhQ4eiaVPp8TJr167VX5atDKGhoYiLizOYd+7cOfj6+lbaMah61Xe1wXfPt8D2N7vgudbeMFMKiL5yByOXROOp+Xux9sR1aHUMd0RERBX12EOaaLVaZGRkwMGhZITzK1euwNLSEq6urpVSXHR0NNq3b48PP/wQgwcPxqFDh/DKK6/gxx9/RERERLn2wd6vxi0pPQ8/7bmEZYfikVOgBQD4OlnitU7+GNiyDtQqDmJMREQ1T40Z0iQ3NxeiKMLS0hIAcPXqVaxevRpBQUEIDw+v1ALXrVuHd999F+fPn4efnx8mT56MV155pdzbM9TVDHeyC/Dr/qtYsu8y7uRITwpwtVHj5Y5+eL6tL6zVfPgJERHVHDUm1PXo0QMDBgzA66+/jrS0NAQGBsLMzAy3bt3Cl19+idGjR1dFrY+Foa5mySkowvJDCfhpzyX9IMZ2GjMMb+eLEaF+cLQyl7lCIiKiR6sxHSWOHj2Kjh07AgD++usvuLm54erVq/j1118xf/78Si2QahdLcxVe6uCHXVO6Ys7AJqjnbIX03ELM334B7T/bhhlrTyMxrXJ6VxMREZmSxwp1OTk5+l6lW7ZswYABA6BQKPDEE0/g6tWrlVog1U7mKgUGt/ZG5OTO+D6iBRrVsUVeoQ5L9l1B5zk78NafJ3AhJUvuMomIiIzGY4W6+vXrY82aNUhISMDmzZvRo0cPAEBKSgpsbW0rtUCq3ZQKAU819sC/4zrgt5faoF09JxTpRPx15Bqe/GoXXv/tCFYfu4YDl24j/nYOCop0cpdMREQki8e6+3z69Ol4/vnn8cYbb6Bbt25o164dAOmsXfPmzSu1QCIAEAQBHQNc0DHABcfi7+D7nRcReSYZm04nYdPppFLrAc7WanjaWcDTXgMPOw087S30Pz3tNXCxVkOh4DNoiYjItDz2kCZJSUm4ceMGmjZtCoVCOuF36NAh2NraIjAwsFKL/C/YUcJ0nU/OxG8HruJCShaup+Xienpeuc7UqRQC3GwtUMdeA4+7ga/O3Z9P+Duxpy0REf1nNab3a2nXrl0DAKMNTAx1tYcoikjNLsD1tDxcT8/FjbRc3EjPQ+LdnzfScpGcmf/QwY097Szw60ttUN+VTyIhIqLHJ0f+eKxTEjqdDp988gm++OILZGVJN6vb2NjgzTffxHvvvac/c0dUnQRBgJO1Gk7WajT2sitznSKtDimZ+biRnovraXn6n9fTcnHyWjqup+dh0ML9WDyiNZr7OJS5DyIiImP0WKHuvffew//+9z989tlnCA0NBQDs3bsXM2bMQF5eHmbOnFmpRRJVFpVSAU97DTztNWh5z9PmUrMLMHJJNE4kpOH5RQex4IUW6NKwcp6OQkREVNUe6/Krp6cnFi5ciL59+xrM/+effzBmzBgkJiZWWoH/FS+/UkVk5xdh9NKj2H3uJlQKAXOfbYp+zevIXRYREdUwNWbw4dTU1DI7QwQGBiI1NfU/F0UkFyu1Cj+92Ap9m3qiSCdi0srj+HnvZbnLIiIieqTHCnVNmzbFt99+e9/8b7/9Fk2aNPnPRRHJyVylwLwhzTCifV0AwEfrzmDOprP4j32KiIiIqtRj3VM3Z84c9O7dG1u3btWPUbd//34kJCRgw4YNlVogkRwUCgEf9AmGi40an2+Ow/c7L+J2VgFm9m8ElZIdgYiIyPg81l+nzp0749y5c+jfvz/S0tKQlpaGAQMG4PTp0/jtt98qu0YiWQiCgLFd6+OzAY2hEICVhxMweulR5BVq5S6NiIjoPv95nLrSTpw4gRYtWkCrNZ4/euwoQZVh8+kkjF9+DAVFOrSp64hFw1vBTmMmd1lERGSkakxHCaLaJjzEHb+OagMbtQqHrqRiyA/7kZKRJ3dZREREegx1ROX0RD0nrHytHVxs1DiblIkBC/bh8q1sucsiIiICwFBHVCHBnrb4+/X2qOtkiWt3cjFowT7EJKbLXRYREVHFer8OGDDgocvT0tL+Sy1ENYKPkyX+fL09Riw+hNPXM/Dcjwfw47CWaF/fWe7SiIioFqvQmTo7O7uHTr6+vnjxxRerqlYio+Fio8aKV59Ae38nZOUXYcTiaGw4dUPusoiIqBar1N6vxoi9X6kq5Rdp8cbK49hwKgmCAHz0TCMMe8L30RsSEZFJY+9XohpGrVLim6EtENHWB6IITFsTg68iz/HpE0REVO0Y6oj+I6VCwCf9GmFi9wAAwNfbzmPaPzHQ6hjsiIio+jDUEVUCQRDwxpMN8PEzIRAE4PcD8Zj8x3HoGOyIiKiaMNQRVaJh7erim6HNYaYU8M/x65i1MVbukoiIqJZgqCOqZE838cTcZ5sCABbtuYzFUZdlroiIiGoDhjqiKvBMszp4u2dDAMBH685gUwyHOyEioqrFUEdURUZ39scLT0i9YieuOI4jV1PlLomIiEwYQx1RFREEATP6hCAsyBX5RTq89MthXLyZJXdZRERkohjqiKqQSqnA/KHN0dTLDmk5hRix+BBuZubLXRYREZkghjqiKmZprsL/RrSGj6MlElJz8dIv0cgpKJK7LCIiMjEMdUTVwNlajV9GtYGDpRlOXkvH+GXHUKTVyV0WERGZEIY6omri52yFn4a3hlqlwLazKZi+9jQfJ0ZERJWGoY6oGrX0dcDXzzWHIADLDsbj+50X5S6JiIhMBEMdUTXr2cgdM/qEAAA+3xyH1ceuyVwRERGZAoY6IhkMb18Xr3aqBwB4+6+TiLpwS+aKiIiopmOoI5LJOz0D8XQTDxRqRbz+2xGcTcqo0uMVaXXYc/4mLt/KrtLjEBGRPFRyF0BUWykUAuY+2xQpmfk4dDkVI36Oxuqx7eFhp6nU46RmF2D5oXgsPXAV19PzYGuhwtY3O8PVxqJSj0NERPLimToiGVmYKbFoWCvUd7VGUkYeRvwcjYy8wkrZ96lr6XjrzxN4YtY2fL45DtfT8wAAGXlF+OjfM5VyDCIiMh4MdUQys7M0w5KRreFio0ZcciZG/34EBUWPN4ZdQZEO/xxPxIDvo9Dn273468g1FBTp0MTLDl882xR/j24PhQCsO3kDO+JSKvmTEBGRnBjqiIyAl4MlFo9oDStzJaIu3MbUv09WaAy7lIw8zNt6DqGzt2PiiuM4Gp8GM6WAZ5p5YtWY9vhnbCgGtvRCS18HvNTBDwDw/uoYPtmCiMiE8J46IiPRqI4dvn+hJUYticbqY4nwtLfAlPDAB64viiKOxqfhl31XsDHmBgq1Ugh0sVEjoq0Pnm/rU+Z9c2882QAbTiUhMS0XX0Wew3u9g6vsMxERUfVhqCMyIp0buGDWgMZ4+6+T+G7HRXjaaxDR1tdgnbxCLf49cR2/7L+CmMSSHrMtfR0wvH1d9Axxh7nqwSfhLc1V+KRfI4xcEo2fo67gmWZ10KiOXZV9JiIiqh4MdURGZnArb1xPy8W8recxbU0M3G0t0D3IDdfTcvH7gatYEZ2A1OwCAIC5SoG+TT0xon3dCgWzroGueLqJB9advIF3V53CmrGhUCqEqvpIRERUDRjqiIzQxO4BuJ6Wiz8OX8O4ZccQWt8Z288mQ3f3NjtPOwu80M4Xz7X2gaOV+WMdY3qfYOw+dxOnEtOxZN8V/b12RERUM7GjBJEREgQBM/s3RqcGLsgt1GJrrBTonqjniIUvtMDut7tiTJf6jx3oAMDVxgLv9AoCAHyxJQ6JabmVVT4REcmAZ+qIjJSZUoHvI1rgg39OQ2OuwAtP+CLQ3bZSj/Fca2+sPnYN0VfuYPqaGPw0vBUEgZdhiYhqIp6pIzJi1moVvhjcFJ/0a1zpgQ6Qnmoxa0BjmCkFbDubgo0xSZV+DCIiqh4MdUS1XH1XG4zu7A8AmLH2dKU90YKIiKoXQx0RYUzX+qjnbIWUzHzM2XRW7nKIiOgxMNQRESzMlJjZvzEA4PcD8ThyNVXmioiIqKIY6ogIANDO3wnPtvQCALy76tRjP3+WiIjkwVBHRHr/91QQHK3McS45C4v2XJK7HCIiqoAaFeo+++wzCIKASZMmyV0KkUlysDLH9KelZ8F+ve08Lt/KlrkiIiIqrxoT6qKjo/HDDz+gSZMmcpdCZNKeaeaJjgHOKCjS4b3VpyCKotwlERFROdSIUJeVlYWIiAgsWrQIDg4OcpdDZNIEQcAn/RpBrVJg38XbWHU0Ue6SiIioHGpEqBs7dix69+6NsLAwuUshqhV8nawwKawBAOCT9WeQml0gc0VERPQoRh/qVqxYgaNHj2LWrFnlWj8/Px8ZGRn6KTMzs4orJDJNL3f0Q6C7De7kFOKT9WfkLoeIiB7BqENdQkICJk6ciKVLl8LCwqJc28yaNQt2dnb6KTg4uIqrJDJNZkoFZg1oDEEAVh1NRNSFW3KXREREDyGIRnwX9Jo1a9C/f38olUr9PK1WC0EQoFAokJ+fb7AMkM7U5efn698nJiYiODgYCQkJ8PLyqrbaiUzFB//E4Jf9V1HXyRKbJnWChZny0RsREdVy165dg7e3d7XmD1W1HOUxde/eHadOnTKYN3LkSAQGBmLq1Kn3BToAUKvVUKvV+vcZGRlVXieRKXsrvCE2n07Glds5+Gb7eUwJD5S7JCIiKoNRX361sbFBo0aNDCYrKys4OTmhUaNGcpdHVCvYWJhhRt8QAMAPuy4hLon3qRIRGSOjDnVEZBx6NnJHj2A3FOlEvLvqJHQ6o71rg4io1jLqy69l2blzp9wlENVKHz4Tgn0Xb+NofBqWHorHsCd85S6JiIhK4Zk6IioXDzsN3uohjV03Z+NZJGfkyVwRERGVxlBHROU2rF1dNPW2R2Z+ET7897Tc5RARUSkMdURUbkqFgM8GNIZSIWDDqSRsPZMsd0lERHQXQx0RVUiQhy1e7ugHAHhn1UnEJKbLXBEREQEMdUT0GCZ1b4AQT1vcyirAkB/2Y8/5m3KXRERU6zHUEVGFacyVWP7qE2jv74TsAi1GLo7GmmOJcpdFRFSrMdQR0WOxtTDDkpFt0LepJ4p0IiatPI4fdl2EET95kIjIpDHUEdFjM1cpMG9IM7xy9x67WRvP4qN1Zzg4MRGRDBjqiOg/USgEvNc7GO/3DgIALI66gvHLjyGvUCtzZUREtQtDHRFVipc71sP8oc1hrlRg/akbGP7zIaTnFspdFhFRrcFQR0SVpm9TTywZ1Ro2ahUOXk7F4IX7kZTOJ08QEVUHhjoiqlTt/Z2x8rV2cLVRIy45EwO+j8K55Ey5yyIiMnkMdURU6YI9bbFqTHv4u1jhenoeBi3Yh+grqXKXRURk0hjqiKhKeDlY4q/X26OlrwMy8ooQ8dNBbIq5IXdZREQmi6GOiKqMg5U5lr7cFk8Gu6GgSIfRS4/i1/1X5C6LiMgkMdQRUZWyMFNi4QstEdHWB6IITP/nNOZsOstBiomIKhlDHRFVOaVCwCf9GuHNJxsAAL7feRFv/XkShVqdzJUREZkOhjoiqhaCIGB89wDMGdgESoWAv49ew8u/HEZ2fpHcpRERmQSGOiKqVoNbe+OnF1tBY6bErnM3MXTRAdzKype7LCKiGo+hjoiqXddAVyx/9Qk4Wpnj5LV0DFywD1duZctdFhFRjcZQR0SyaOZtj79Ht4ePoyWu3s7BoIX7EX87R+6yiIhqLIY6IpKNn7MV/h7dHoHuNriVlY9hPx/EzUxeiiUiehwMdUQkKxcbNX4d1QbejhpcvZ2DEYsPITOvUO6yiIhqHIY6IpKdq60Ffh3VFk5W5jh9PQOv/34E+UVaucsiIqpRGOqIyCj4OVthycg2sDJXIurCbUz+4wS0Og5QTERUXgx1RGQ0GnvZ4YdhrWCmFLD+5A18+O9pPnmCiKicGOqIyKh0CHDGl4ObQRCAX/dfxXc7LshdEhFRjcBQR0RGp09TT3zwdDAAYO6Wc1h+KF7mioiIjB9DHREZpRGhfhjb1R8A8N7qU9h8OknmioiIjBtDHREZrbd6NMSQVt7QicD45cdw8NJtuUsiIjJaDHVEZLQEQcDM/o0QFuSGgiIdXv71MGJvZMhdFhGRUWKoIyKjplIq8O3zzdG6rgMy84ow/OdDSEjl48SIiO7FUEdERs/CTImfXmyNhm42SMnMx/CfD+F2Fh8nRkRUGkMdEdUIdpZm+GVUG9Sx1+DSrWyMWhKN7PwiucsiIjIaDHVEVGO421ngl1Ft4GBphhPX0vH670dQUKSTuywiIqPAUEdENUp9V2ssHtkGGjMl9py/hSl/nYCOjxMjImKoI6Kap5m3PRa80AIqhYB/jl/HJ+tj+TgxIqr1GOqIqEbq0tAVnz/bBADwc9RlLNx1SeaKiIjkxVBHRDVW/+ZeeL93EABg9qaz+ONwgswVERHJh6GOiGq0lzvWw2ud6wEA3l11ClvPJMtcERGRPBjqiKjGe6dnIAa28IJWJ2LssqM4fCVV7pKIiKodQx0R1XiCIOCzgY3RLdAV+UU6PL/oID789zRucYBiIqpFGOqIyCSYKRX47vkW6NrQBQVaHRZHXUHnOTvwVeQ5ZOYVyl0eEVGVY6gjIpOhMVfi5xGt8dtLbdC4jh2yC7T4ett5dJqzAz/tuYS8Qq3cJRIRVRmGOiIyKYIgoGOAC9aOC8WCiBao52KFOzmF+GR9LLrN3YmV0fEo0vIpFERkehjqiMgkCYKAXo09sGVSJ8we2Bgedha4np6HqX+fQo95u7Hx1A0OWExEJoWhjohMmkqpwJDWPtjxVhe83zsIDpZmuHQzG6OXHsUz30Vh7/lbcpdIRFQpGOqIqFawMFPi5Y71sPvtrpjYPQBW5kqcvJaOF/53EM8vOoDjCWlyl0hE9J8w1BFRrWJjYYY3nmyAXW93xcjQujBXKrDv4m30+y4Kr/12GOeTM+UukYjosTDUEVGt5Gytxgd9QrD9rc54tqUXFAKw+XQywuftxlt/nsC1Ozlyl0hEVCEMdURUq3k5WOLzZ5ti86ROCA9xg04E/jpyDd3m7sKH/57GzUwOYExENYNK7gKIiIxBgJsNfhjWCsfi7+DzzXHYd/E2Fkddwa/7r6KVrwOeDHbDk8Fu8HWykrtUIqIyCaKJ9+m/du0avL29kZCQAC8vL7nLIaIaYu/5W/giMg7H4tMM5jdws74b8NzRpI4dFApBngKJyKjJkT+M+kzdrFmzsGrVKpw9exYajQbt27fH7Nmz0bBhQ7lLIyIT1yHAGR0CnJGQmoOtscmIPJOMg5dTcS45C+eSs/DdjotwtVGje5AbegS7oZ2/EyzMlHKXTUS1mFGfqevZsyeee+45tG7dGkVFRfi///s/xMTE4MyZM7CyKt8lEJ6pI6LKkp5TiJ3nUrDlTDJ2xd1EVn6RfpmluRKdAlzwZLAbugW6wsHKXMZKiUhucuQPow5197p58yZcXV2xa9cudOrUqVzbMNQRUVXIL9LiwKVUbD0jncVLysjTL1MIQKu6jujB+/CIai1efn2E9PR0AICjo6PMlRBRbadWKdG5gQs6N3DBR8+EICYxA5FnkrDlTDLOJmXi0OVUHLqcik/WxyLAVboPLyzYDU297KHkfXhEVAVqzJk6nU6Hvn37Ii0tDXv37n3gevn5+cjPLxmCIDExEcHBwTxTR0TV5t778LS6kl+zjlbm6NLABV0DXdEpwAV2lmYyVkpEVYWXXx9i9OjR2LhxI/bu3fvQxpkxYwY+/PDD++Yz1BGRHErfh7c77iYyS92Hp1QIaOnjgK6BrugW6IoGbtYQBJ7FIzIFDHUPMG7cOPzzzz/YvXs3/Pz8Hrouz9QRkbEq1Opw+Mod7IxLwfazKTifkmWwvI69Bl0auqBboCva+ztDY87etEQ1FUPdPURRxPjx47F69Wrs3LkTAQEBFd4HO0oQkbFKSM3BjrgU7Dibgn0XbyO/SKdfplYp0M7fCV0bSmfxvB0tZayUiCqKoe4eY8aMwbJly/DPP/8YjE1nZ2cHjUZTrn0w1BFRTZBboMX+S7ew/WwKdpy9icS0XIPl9V2t0S3QFV0buqJVXQeYKfmURyJjxlB3jwfdW7J48WKMGDGiXPtgqCOimkYURZxPycL2s9Jl2iNX7xh0trBRq9CqrgN8nazg62R5d7KCl4MGahUv2RIZAw5pcg8jzptERFVGEAQ0cLNBAzcbvN7ZH+k5hdhz4Sa2n03BrribuJ1dgB1xNwHcvGc7wNNOAx9HKej5OFnC17Ek+NlYsKctkSkz6lBHRESAnaUZnm7iiaebeEKnE3EyMR2nr6cj/nYOrt7OwdXUHMTfzkZ2gRaJablITMvF/ku379uPo5W5PvD5OlrCx8kKdZ0s0dDdhoGPyAQw1BER1SAKhYBm3vZo5m1vMF8URdzKKkB8arYU9G7nID41B1duZyP+dg5uZxcg9e50PCHNYFtrtQqzBzZB7yYe1fdBiKjSMdQREZkAQRDgYqOGi40aLX3vf+pOZl4h4lNzpLN7qTm4elsKfxdvZiE5Ix9jlx3F4at18W6vIJir2AmDqCZiqCMiqgVsLMwQ4mmHEE87g/lFWh3mbjmHhbsuYnHUFRxPSMO3z7dAHfvyjTBARMaD/xwjIqrFVEoF3ukViEUvtoKthQrH4tPw9Pw92BmXIndpRFRBDHVERIQng92wfkJHNKpjizs5hRi5JBpfbokzGEqFiIwbQx0REQEAvB0t8dfr7RHR1geiCMzffgEv/nwQt7LyH70xEcmOoY6IiPQszJSY2b8x5g1pBo2ZElEXbqP3/D2IvpIqd2lE9AgMdUREdJ9+zetg7bhQ+LtYITkjH8/9eACLdl/ioPBERoyhjoiIyhTgZoO14zqgb1NPaHUiZm6IxWu/HUF6bqHcpRFRGRjqiIjogazUKnz9XDN83K8RzJUKbDmTjD7f7EVMYrrcpRHRPRjqiIjooQRBwLAnfPHX6HaoY69BfGoOBizYh+WH4nk5lsiIMNQREVG5NPGyx/oJHdA90BUFRTq8u+oU3vzzBHIKiuQujYjAUEdERBVgb2mORS+2wts9G0IhAKuOJqL/d/tw8WaW3KUR1XoMdUREVCEKhYAxXepj6ctPwNlajbjkTPT9Zi/Wnbwud2lEtRqf/UpERI+lnb8TNkzogPHLj+Hg5VSMW3YMG07dgLejJazNVbBUq2CtVsLSXAVrtQqW5kpYqVV3JyWs1SpozJQQBEHuj0JkEhjqiIjosbnaWmDpy23xZeQ5fL/zIjacSqrQ9oIAWJlLgc9arYKlWgmruyGwvqs1xncPgLWaf6qIyoP/pxAR0X+iUirwds9AdG7ggqgLt5BdoEVOQRGy8rXIyS9CVn4Rcgq0yM4vQnZBEbLztcguKIIoAqIIZN1dJyXT8HFk286mIPJMMr6LaIEgD1uZPh1RzcFQR0RElaJtPSe0redUrnVFUURuoVYKePeEvez8IqTlFOL7HRdw6VY2+n0XhY/7NcLgVt5V/AmIajaGOiIiqnaCIMDSXAVLcxVcbNRlrvNUYw9M/uM4dsbdxNt/ncShy6n4+JlG0Jgrq7laopqBvV+JiMgoOVqZ4+fhrTElXBo+5a8j19DvuyhcSOHwKURlYagjIiKjpVAIGNtVGj7FxUYaPuWZb/di7QkOn0J0L4Y6IiIyeu38nbB+Qge0q+eE7AItJiw/hvfXnEJ+kVbu0oiMBkMdERHVCK42Fvj95bYY360+AOD3A/EYtGA/4m/nyFwZkXFgqCMiohpDqRDwZo+GWDKyNRwszXAqMR29v9mDLacrNj4ekSliqCMiohqnS0NXrJ/QES187JGZV4RXfzuCmevPoFCrk7s0Itkw1BERUY3kaa/Bytfa4eUOfgCARXsu47kfD+BGeq7MlRHJg6GOiIhqLDOlAu8/HYyFL7SEjYUKR67eQe/5e7H73E25SyOqdhx8mIiIaryejdwR5GGDMUuP4vT1DAxffAjju9bHxLAGUCoEucvTyykoQvSVO4i6cAt7z99CWk4BOgQ4o0ewOzoEOMPCjAMr0+MTRFEU5S6iKl27dg3e3t5ISEiAl5eX3OUQEVEVyivU4uN1Z7D0YDwAoF09J3w9tBlcbSxkqadIq8PJxHREnb+FvRdu4Wj8HRRqy/6zqzFTolMDZzwZ7I7uga5wsDKv5mqpMsmRPxjqiIjI5PxzPBHvrjqFnAItXGzUmDOwCVr4OMBWo4IgVN2ZO1EUcfFmFvaev4W9F27j4KXbyMwvMljH084CofWd0SHAGQ6W5th+NgVbTifhenqefh2lQkDrug54MtgdPYLd4O1oWWU1U9VgqKsCDHVERLXThZQsjFl6BOeSSx4rZq1WwdPeAnXsNajjoEEde8u7P6XJ1UYNRQUv1yal5yHqwi3pkuqFW0jJzDdYbqcxQ3t/J7Sv74wO9Z1R18nyvmApiiJOX8/AljPJiDyTjNgbGQbLA91t0CNECnghnrZVGkypcjDUVQGGOiKi2iunoAgz18diU0wSbmcXPHJ9M6UADztNqdBn+NrD3gL5RTocuHhbH+Iu3sw22IdapUDruo7S2bj6zgj2tK3wfX0JqTmIPJOMLWeSEH3lDrS6kj/VnnYWeDLYDT1C3NHGzxFmSvZ5NEYMdVWAoY6IiAAgt0CLxLRcXE/LRWJaLhLvGP5MysgzCE9lEQRAAFB6NYUANK5jpw9xLXwdKrXDw53sAmw/m4LIM8nYde4mcgtLHo1ma6FCt0BXPBnsjs4NXWCtZv9HY8FQVwUY6oiIqDyKtDokZ+bfDXk5uJ6Wh2v64JeDxLRc5BVKgxvXc7FCqL8zQus7o109J9hZmlVLjXmFWkRduIUtp5Ox7WwybmWVnH00VyoQWt8JTwa7IyzYVbbOISRhqKsCDHVERFQZRFHEnZxCFOl0RhGYtDoRx+Lv3L1Mm4zLt0ouAwsC0NzbXn8fXj0XaxkrrZ0Y6qoAQx0REZm64l63m09LAe9EQprBcn8XK33Aa+plX+HOIFRxDHVVgKGOiIhqm6T0PETGSj1p91+8ZTA2nquNGmHBbugR7IZ2/k5Qqyrv/j9RFHEzMx8Jd3IQn5qDhNRcZOUXIcDVGsGetghwtYG5qnZ07GCoqwIMdUREVJtl5BViZ9xNbDmdhJ1xN5FVatw8a7UKXRq6oEeIO7o0dIGtxaPvDUzPLURCag6ulQpuCXdy7s7LRX6R7oHbmikFBLjaIMTTFsGetgjxtEOQhw1synHcmoahrgow1BEREUnyi7TYf/E2Iu+Oh1d6TD0zpYAn6jmhR4g7WvjYIyUzH9dS7w9uGXlFDzmC1BvYw04Db0cNvB0sYaVW4WxSBk5fz0DmA7b1dbJEsIetQdhztVFXynh8eYVapGYX6Kc7OQW4nSX9DA9xR6M6dv/5GGVhqKsCDHVERET30+lEnLiWhi1nkrHldNJ94+09jLO1ObwcLOHtaAlvBw28HS3h42gJbwdLeNhblDl2niiKuHYnF6evZ+DMjQycuZ6OM9czDJ6kUZqTlTmCS4W8YA9b+DpZIiuvCLfvCWf6wJZdcN+ynAJtmfsHgE/7N8bzbX3K/bkrgqGuCjDUERERPdrFm1mIPJOMzaeTcDElC572Gng53A1rd8+6eTtawstBA6tKHA8vNbsAsTcycPpuyDt9PQMXb2bhEUMGVoiZUoCDpTkcrQynp5t4oo2fY+UdqBQ58gdHKSQiIiL4u1jDv7M1Xu/sX63HdbQyR2h9acy/YnmFWpxNyrwb8tJx5kYGYm9k6McJtLFQlYSze8Kag5U5nO75aaOu2mf+GguGOiIiIjIqFmZKNPO2RzNve/08rU7EnZwC2FqY1ZoetBXFUEdERERGT6kQ4GytlrsMo8aoS0RERGQCGOqIiIiITABDHREREZEJYKgjIiIiMgEMdUREREQmgKGOiIiIyAQw1BERERGZAIY6IiIiIhNQI0Ldd999h7p168LCwgJt27bFoUOH5C6JiIiIyKgYfahbuXIlJk+ejA8++ABHjx5F06ZNER4ejpSUFLlLIyIiIjIaRh/qvvzyS7zyyisYOXIkgoODsXDhQlhaWuLnn3+WuzQiIiIio2HUoa6goABHjhxBWFiYfp5CoUBYWBj2799f5jb5+fnIyMjQT5mZmdVVLhEREZFsjDrU3bp1C1qtFm5ubgbz3dzckJSUVOY2s2bNgp2dnX4KDg6ujlKJiIiIZKWSu4DK9u6772Ly5Mn69wkJCWjUqBFu3LghY1VERERUmxTnDp1OV23HNOpQ5+zsDKVSieTkZIP5ycnJcHd3L3MbtVoNtVqtf5+TkwMAaNOmTdUVSkRERFSG5ORk+Pj4VMuxjDrUmZubo2XLlti2bRv69esHQEq827Ztw7hx48q1j+bNm+PQoUNwc3ODQmHUV5srRWZmJoKDg3HmzBnY2NjIXY5RYduUje3yYGybB2PblI3t8mC1rW10Oh2Sk5PRvHnzajumUYc6AJg8eTKGDx+OVq1aoU2bNpg3bx6ys7MxcuTIcm2vUqnQunXrKq7SeGRkZAAA6tSpA1tbW5mrMS5sm7KxXR6MbfNgbJuysV0erDa2TXWdoStm9KFuyJAhuHnzJqZPn46kpCQ0a9YMmzZtuq/zBBEREVFtZvShDgDGjRtX7sutRERERLWR6d9kVsuo1Wp88MEHBp1FSMK2KRvb5cHYNg/Gtikb2+XB2DZVTxBFUZS7CCIiIiL6b3imjoiIiMgEMNQRERERmQCGOiIiIiITwFBnImbNmoXWrVvDxsYGrq6u6NevH+Li4uQuy+h89tlnEAQBkyZNkrsUo5CYmIgXXngBTk5O0Gg0aNy4MQ4fPix3WbLTarWYNm0a/Pz8oNFo4O/vj48//hi17Rbk3bt3o0+fPvD09IQgCFizZo3BclEUMX36dHh4eECj0SAsLAznz5+Xp9hq9rC2KSwsxNSpU9G4cWNYWVnB09MTL774Iq5fvy5fwdXoUd+b0l5//XUIgoB58+ZVW32mjKHOROzatQtjx47FgQMHEBkZicLCQvTo0QPZ2dlyl2Y0oqOj8cMPP6BJkyZyl2IU7ty5g9DQUJiZmWHjxo04c+YMvvjiCzg4OMhdmuxmz56NBQsW4Ntvv0VsbCxmz56NOXPm4JtvvpG7tGqVnZ2Npk2b4rvvvitz+Zw5czB//nwsXLgQBw8ehJWVFcLDw5GXl1fNlVa/h7VNTk4Ojh49imnTpuHo0aNYtWoV4uLi0LdvXxkqrX6P+t4UW716NQ4cOABPT89qqqwWEMkkpaSkiADEXbt2yV2KUcjMzBQDAgLEyMhIsXPnzuLEiRPlLkl2U6dOFTt06CB3GUapd+/e4qhRowzmDRgwQIyIiJCpIvkBEFevXq1/r9PpRHd3d/Hzzz/Xz0tLSxPVarW4fPlyGSqUz71tU5ZDhw6JAMSrV69WT1FG4kFtc+3aNbFOnTpiTEyM6OvrK3711VfVXpsp4pk6E5Weng4AcHR0lLkS4zB27Fj07t0bYWFhcpdiNNauXYtWrVrh2WefhaurK5o3b45FixbJXZZRaN++PbZt24Zz584BAE6cOIG9e/eiV69eMldmPC5fvoykpCSD/6fs7OzQtm1b7N+/X8bKjFN6ejoEQYC9vb3cpchOp9Nh2LBhmDJlCkJCQuQux6TUiCdKUMXodDpMmjQJoaGhaNSokdzlyG7FihU4evQooqOj5S7FqFy6dAkLFizA5MmT8X//93+Ijo7GhAkTYG5ujuHDh8tdnqzeeecdZGRkIDAwEEqlElqtFjNnzkRERITcpRmNpKQkALjvkY1ubm76ZSTJy8vD1KlTMXTo0FrzzNOHmT17NlQqFSZMmCB3KSaHoc4EjR07FjExMdi7d6/cpcguISEBEydORGRkJCwsLOQux6jodDq0atUKn376KQCgefPmiImJwcKFC2t9qPvjjz+wdOlSLFu2DCEhITh+/DgmTZoET0/PWt82VDGFhYUYPHgwRFHEggUL5C5HdkeOHMHXX3+No0ePQhAEucsxObz8amLGjRuHdevWYceOHfDy8pK7HNkdOXIEKSkpaNGiBVQqFVQqFXbt2oX58+dDpVJBq9XKXaJsPDw8EBwcbDAvKCgI8fHxMlVkPKZMmYJ33nkHzz33HBo3boxhw4bhjTfewKxZs+QuzWi4u7sDAJKTkw3mJycn65fVdsWB7urVq4iMjORZOgB79uxBSkoKfHx89L+Tr169ijfffBN169aVu7waj2fqTIQoihg/fjxWr16NnTt3ws/PT+6SjEL37t1x6tQpg3kjR45EYGAgpk6dCqVSKVNl8gsNDb1v2Jtz587B19dXpoqMR05ODhQKw3/zKpVK6HQ6mSoyPn5+fnB3d8e2bdvQrFkzAEBGRgYOHjyI0aNHy1ucESgOdOfPn8eOHTvg5OQkd0lGYdiwYffd2xweHo5hw4Zh5MiRMlVlOhjqTMTYsWOxbNky/PPPP7CxsdHf02JnZweNRiNzdfKxsbG5775CKysrODk51fr7Dd944w20b98en376KQYPHoxDhw7hxx9/xI8//ih3abLr06cPZs6cCR8fH4SEhODYsWP48ssvMWrUKLlLq1ZZWVm4cOGC/v3ly5dx/PhxODo6wsfHB5MmTcInn3yCgIAA+Pn5Ydq0afD09ES/fv3kK7qaPKxtPDw8MGjQIBw9ehTr1q2DVqvV/052dHSEubm5XGVXi0d9b+4NuGZmZnB3d0fDhg2ru1TTI3f3W6ocAMqcFi9eLHdpRodDmpT4999/xUaNGolqtVoMDAwUf/zxR7lLMgoZGRnixIkTRR8fH9HCwkKsV6+e+N5774n5+flyl1atduzYUebvleHDh4uiKA1rMm3aNNHNzU1Uq9Vi9+7dxbi4OHmLriYPa5vLly8/8Hfyjh075C69yj3qe3MvDmlSeQRRrGVDpBMRERGZIHaUICIiIjIBDHVEREREJoChjoiIiMgEMNQRERERmQCGOiIiIiITwFBHREREZAIY6oiIiIhMAEMdERERkQlgqCMiqgSCIGDNmjVyl0FEtRhDHRHVeCNGjIAgCPdNPXv2lLs0IqJqo5K7ACKiytCzZ08sXrzYYJ5arZapGiKi6sczdURkEtRqNdzd3Q0mBwcHANKl0QULFqBXr17QaDSoV68e/vrrL4PtT506hW7dukGj0cDJyQmvvvoqsrKyDNb5+eefERISArVaDQ8PD4wbN85g+a1bt9C/f39YWloiICAAa9eu1S+7c+cOIiIi4OLiAo1Gg4CAgPtCKBHRf8FQR0S1wrRp0zBw4ECcOHECEREReO655xAbGwsAyM7ORnh4OBwcHBAdHY0///wTW7duNQhtCxYswNixY/Hqq6/i1KlTWLt2LerXr29wjA8//BCDBw/GyZMn8dRTTyEiIgKpqan64585cwYbN25EbGwsFixYAGdn5+prACIyfSIRUQ03fPhwUalUilZWVgbTzJkzRVEURQDi66+/brBN27ZtxdGjR4uiKIo//vij6ODgIGZlZemXr1+/XlQoFGJSUpIoiqLo6ekpvvfeew+sAYD4/vvv699nZWWJAMSNGzeKoiiKffr0EUeOHFk5H5iIqAy8p46ITELXrl2xYMECg3mOjo761+3atTNY1q5dOxw/fhwAEBsbi6ZNm8LKykq/PDQ0FDqdDnFxcRAEAdevX0f37t0fWkOTJk30r62srGBra4uUlBQAwOjRozFw4EAcPXoUPXr0QL9+/dC+ffvH+qxERGVhqCMik2BlZXXf5dDKotFoyrWemZmZwXtBEKDT6QAAvXr1wtWrV7FhwwZERkaie/fuGDt2LObOnVvp9RJR7cR76oioVjhw4MB974OCggAAQUFBOHHiBLKzs/XLo6KioFAo0LBhQ9jY2KBu3brYtm3bf6rBxcUFw4cPx++//4558+bhxx9//E/7IyIqjWfqiMgk5OfnIykpyWCeSqXSd0b4888/0apVK3To0AFLly7FoUOH8L///Q8AEBERgQ8++ADDhw/HjBkzcPPmTYwfPx7Dhg2Dm5sbAGDGjBl4/fXX4erqil69eiEzMxNRUVEYP358ueqbPn06WrZsiZCQEOTn52PdunX6UElEVBkY6ojIJGzatAkeHh4G8xo2bIizZ88CkHqmrlixAmPGjIGHhweWL1+O4OBgAIClpSU2b96MiRMnonXr1rC0tMTAgQPx5Zdf6vc1fPhw5OXl4auvvsJbb70FZ2dnDBo0qNz1mZub491338WVK1eg0WjQsWNHrFixohI+ORGRRBBFUZS7CCKiqiQIAlavXo1+/frJXQoRUZXhPXVEREREJoChjoiIiMgE8J46IjJ5vMuEiGoDnqkjIiIiMgEMdUREREQmgKGOiIiIyAQw1BERERGZAIY6IiIiIhPAUEdERERkAhjqiIiIiEwAQx0RERGRCWCoIyIiIjIB/w/6zM1gPqjqWwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from previous_chapters import plot_losses\n",
    "\n",
    "plt.figure(figsize=(5, 3))\n",
    "epochs_tensor = torch.linspace(1, n_epochs, len(train_losses))\n",
    "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)\n",
    "# plt.tight_layout(); plt.savefig(\"3.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c16fa614-67e1-4254-8b7e-c3e2f690c29c",
   "metadata": {},
   "source": [
    "- Note that the model is overfitting here because the dataset is kept very small for educational purposes (so that the code can be executed on a laptop computer)\n",
    "- For a longer pretraining run on a much larger dataset, see [../../ch05/03_bonus_pretraining_on_gutenberg](../../ch05/03_bonus_pretraining_on_gutenberg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}