LLMs-from-scratch/appendix-D/01_main-chapter-code/appendix-D.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9a5936bd-af17-4a7e-a4d2-e910411708ea",
   "metadata": {},
   "source": [
    "<font size=\"1\">\n",
    "Supplementary code for \"Build a Large Language Model From Scratch\": <a href=\"https://www.manning.com/books/build-a-large-language-model-from-scratch\">https://www.manning.com/books/build-a-large-language-model-from-scratch</a> by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af53bcb1-ff9d-49c7-a0bc-5b8d32ff975b",
   "metadata": {},
   "source": [
    "## Appendix D: Adding Bells and Whistles to the Training Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f58c142-9434-49af-b33a-356b80a45b86",
   "metadata": {},
   "source": [
    "- In this appendix, we add a few more advanced features to the training function, which are used in typical pretraining and finetuning; finetuning is covered in chapters 6 and 7\n",
    "- The next three sections below discuss learning rate warmup, cosine decay, and gradient clipping\n",
    "- The final section adds these techniques to the training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "744def4f-c03f-42ee-97bb-5d7d5b89b723",
   "metadata": {},
   "source": [
    "- We start by initializing a model reusing the code from chapter 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8755bd5e-bc06-4e6e-9e63-c7c82b816cbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch version: 2.2.2\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "import torch\n",
    "\n",
    "print(\"torch version:\", version(\"torch\"))\n",
    "\n",
    "\n",
    "from previous_chapters import GPTModel\n",
    "\n",
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,  # Vocabulary size\n",
    "    \"ctx_len\": 256,       # Shortened context length (orig: 1024)\n",
    "    \"emb_dim\": 768,       # Embedding dimension\n",
    "    \"n_heads\": 12,        # Number of attention heads\n",
    "    \"n_layers\": 12,       # Number of layers\n",
    "    \"drop_rate\": 0.1,     # Dropout rate\n",
    "    \"qkv_bias\": False     # Query-key-value bias\n",
    "}\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.eval();  # Disable dropout during inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51574e57-a098-412c-83e8-66dafa5a0b99",
   "metadata": {},
   "source": [
    "- Next, using the same code we used in chapter 5, we initialize the data loaders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "386ca110-2bb4-42f1-bd54-8836df80acaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib.request\n",
    "\n",
    "file_path = \"the-verdict.txt\"\n",
    "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n",
    "\n",
    "if not os.path.exists(file_path):\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        text_data = response.read().decode('utf-8')\n",
    "    with open(file_path, \"w\", encoding=\"utf-8\") as file:\n",
    "        file.write(text_data)\n",
    "else:\n",
    "    with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "        text_data = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae96992b-536a-4684-a924-658b9ffb7e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import create_dataloader_v1\n",
    "\n",
    "# Train/validation ratio\n",
    "train_ratio = 0.90\n",
    "split_idx = int(train_ratio * len(text_data))\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = create_dataloader_v1(\n",
    "    text_data[:split_idx],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"ctx_len\"],\n",
    "    stride=GPT_CONFIG_124M[\"ctx_len\"],\n",
    "    drop_last=True,\n",
    "    shuffle=True\n",
    ")\n",
    "\n",
    "val_loader = create_dataloader_v1(\n",
    "    text_data[split_idx:],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"ctx_len\"],\n",
    "    stride=GPT_CONFIG_124M[\"ctx_len\"],\n",
    "    drop_last=False,\n",
    "    shuffle=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939c08d8-257a-41c6-b842-019f7897ac74",
   "metadata": {},
   "source": [
    "## D.1 Learning rate warmup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fafcd30-ddf7-4a9f-bcf4-b13c052b3133",
   "metadata": {},
   "source": [
    "- When training complex models like LLMs, implementing learning rate warmup can help stabilize the training\n",
    "- In learning rate warmup, we gradually increase the learning rate from a very low value (`initial_lr`) to a user-specified maximum (`peak_lr`)\n",
    "- This way, the model will start the training with small weight updates, which helps decrease the risk of large destabilizing updates during the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2bb4790b-b8b6-4e9e-adf4-704a04b31ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 15\n",
    "initial_lr = 0.0001\n",
    "peak_lr = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf3a8da-abc4-4b80-a5d8-f1cc1c7cc5f3",
   "metadata": {},
   "source": [
    "- Typically, the number of warmup steps is between 10% and 20% of the total number of steps\n",
    "- We can compute the increment as the difference between the `peak_lr` and `initial_lr` divided by the number of warmup steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e075f80e-a398-4809-be1d-8019e1d31c90",
   "metadata": {},
   "outputs": [],
   "source": [
    "warmup_steps = 20\n",
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "global_step = -1\n",
    "track_lrs = []\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        if global_step < warmup_steps:\n",
    "            lr = initial_lr + global_step * lr_increment\n",
    "        else:\n",
    "            lr = peak_lr\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.param_groups[0][\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights\n",
    "        # ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cb6da121-eeed-4023-bdd8-3666c594b4ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAEmCAYAAABYuVhFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA1FElEQVR4nO3de1xUdf4/8NcgMpDKoLIyoCijsUsFeYGYxtiv6zo1FWWkeVtWzVgvLaVG3yxvUK19ScxycS11t2+2v81L7haVX6MlLK1EkEsWmpcSxcTBW8woym3m8/vD5ejEqAzOcM7A6/l4zAPnc97nzPujNW/OeZ+LSgghQERERIrhI3cCRERE5IjFmYiISGFYnImIiBSGxZmIiEhhWJyJiIgUhsWZiIhIYViciYiIFIbFmYiISGF85U7AW9ntdlRVVaFHjx5QqVRyp0NERDIQQuDcuXMICwuDj4/79ndZnNuoqqoK4eHhcqdBREQKcOzYMfTr189t22NxbqMePXoAuPQPEhgYKHM2REQkB6vVivDwcKkmuAuLcxs1H8oODAxkcSYi6uTc3d7kCWFEREQKw+JMRESkMCzORERECiN7cV61ahUiIiLg7+8PvV6PoqKia8Zv3rwZUVFR8Pf3R0xMDLZu3eqw/L333sM999yD3r17Q6VS4euvv26xjbq6OqSmpqJ3797o3r07xo4di+rqandOi4iIqM1kLc6bNm1CWloaMjIyUFpaisGDB8NkMuHkyZNO43fu3IlJkyYhJSUFZWVlSEpKQlJSEsrLy6WY2tpaJCQkYOnSpVf93KeeegofffQRNm/ejO3bt6Oqqgpjxoxx+/yIiIjaQiWEEHJ9uF6vxx133IG//OUvAC7d2CM8PBxPPvkknnvuuRbxEyZMQG1tLbZs2SKN3XnnnRgyZAhWr17tEHvkyBHodDqUlZVhyJAh0rjFYsEvfvELrF+/Ho888ggAYP/+/bjllltQUFCAO++8s1W5W61WaDQaWCwWnq1NRNRJeaoWyHYpVUNDA0pKSjB//nxpzMfHB0ajEQUFBU7XKSgoQFpamsOYyWRCTk5Oqz+3pKQEjY2NMBqN0lhUVBT69+9/zeJcX1+P+vp66b3Vam31Z3YWQggseL8cZZU/yZ0KEVGr3BHRC39KipY7jRZkK86nT5+GzWZDSEiIw3hISAj279/vdB2z2ew03mw2t/pzzWYz/Pz8EBQU5NJ2MjMz8cILL7T6czqj/eZz2FBUKXcaREStFqrxlzsFp3gTklaaP3++w157811h6LJdh88AAIb1D0La3b+SORsiousLuqmr3Ck4JVtxDg4ORpcuXVqcJV1dXQ2tVut0Ha1W61L81bbR0NCAmpoah73n621HrVZDrVa3+nM6o+bibLw1BAmRwTJnQ0TkvWQ7W9vPzw+xsbHIz8+Xxux2O/Lz82EwGJyuYzAYHOIBIC8v76rxzsTGxqJr164O2zlw4AAqKytd2g45stsFCivOAgDuHNhb5myIiLybrIe109LSMHXqVMTFxSE+Ph4rVqxAbW0tpk2bBgCYMmUK+vbti8zMTADAnDlzMGLECCxfvhyJiYnYuHEjiouLsXbtWmmbZ8+eRWVlJaqqqgBcKrzApT1mrVYLjUaDlJQUpKWloVevXggMDMSTTz4Jg8HQ6jO1qaUD1edQc6ERN/l1QUxfjdzpEBF5NVmL84QJE3Dq1Cmkp6fDbDZjyJAhyM3NlU76qqysdHg+5vDhw7F+/XosWrQICxYsQGRkJHJychAdfflMuw8//FAq7gAwceJEAEBGRgaef/55AMBrr70GHx8fjB07FvX19TCZTHj99dfbYcYdV/Mh7biIXujaRfZ72xAReTVZr3P2ZrzO2dHM/1eMT/ZWY969v8Iff3Oz3OkQEbULT9UC7uLQDWO/mYjIvVic6Yax30xE5F4sznTD2G8mInIvfpPSDSv44VJxvnNgL5kzISLqGFic6Yaw30xE5H4sznRD9pvPwXKxEd3YbyYichsWZ7oh7DcTEbkfv03phjQXZx7SJiJyHxZnajPHfjNPBiMichcWZ2qzK/vN0ew3ExG5DYsztRn7zUREnsFvVGoz9puJiDyDxZnahP1mIiLPYXGmNmG/mYjIc1icqU0K2G8mIvIYfqtSm7DfTETkOSzO5DK7XaDoP/1mwyAWZyIid2NxJpd9Z7Ze7jeHBcqdDhFRh8PiTC7bdfjSXvMdul7wZb+ZiMjt+M1KLmO/mYjIs1icySVX9ptZnImIPIPFmVzCfjMRkeexOJNL2G8mIvI8fruSS9hvJiLyPBZnajX2m4mI2geLM7XavhOX+s3d1b7sNxMReRCLM7Va8yHtOyJ6st9MRORB/IalVms+GYyHtImIPIvFmVrFZhcoquDJYERE7YHFmVrluxNWWOua0F3ti9vYbyYi8igWZ2oV9puJiNqP7N+yq1atQkREBPz9/aHX61FUVHTN+M2bNyMqKgr+/v6IiYnB1q1bHZYLIZCeno7Q0FAEBATAaDTi0KFDDjEHDx7EQw89hODgYAQGBiIhIQGfffaZ2+fWkbDfTETUfmQtzps2bUJaWhoyMjJQWlqKwYMHw2Qy4eTJk07jd+7ciUmTJiElJQVlZWVISkpCUlISysvLpZisrCxkZ2dj9erVKCwsRLdu3WAymVBXVyfFPPDAA2hqasK2bdtQUlKCwYMH44EHHoDZbPb4nL0R+81ERO1MyCg+Pl6kpqZK7202mwgLCxOZmZlO48ePHy8SExMdxvR6vZg5c6YQQgi73S60Wq1YtmyZtLympkao1WqxYcMGIYQQp06dEgDEjh07pBir1SoAiLy8vFbnbrFYBABhsVhavY63+vbHGjHg2S3itvRc0dhkkzsdIiLF8FQtkG3PuaGhASUlJTAajdKYj48PjEYjCgoKnK5TUFDgEA8AJpNJiq+oqIDZbHaI0Wg00Ov1Ukzv3r3xq1/9Cn//+99RW1uLpqYmrFmzBn369EFsbKy7p9khsN9MRNS+fOX64NOnT8NmsyEkJMRhPCQkBPv373e6jtlsdhrffDi6+ee1YlQqFT799FMkJSWhR48e8PHxQZ8+fZCbm4uePXteNd/6+nrU19dL761Waytn6v3YbyYial+dbjdICIHU1FT06dMHX3zxBYqKipCUlIQHH3wQJ06cuOp6mZmZ0Gg00is8PLwds5aPzS5QyH4zEVG7kq04BwcHo0uXLqiurnYYr66uhlardbqOVqu9Znzzz2vFbNu2DVu2bMHGjRtx1113YdiwYXj99dcREBCAt99++6r5zp8/HxaLRXodO3bMtQl7qe9OWHGurgk9eH0zEVG7ka04+/n5ITY2Fvn5+dKY3W5Hfn4+DAaD03UMBoNDPADk5eVJ8TqdDlqt1iHGarWisLBQirlw4QKAS/3tK/n4+MBut181X7VajcDAQIdXZyD1m/n8ZiKidiNbzxkA0tLSMHXqVMTFxSE+Ph4rVqxAbW0tpk2bBgCYMmUK+vbti8zMTADAnDlzMGLECCxfvhyJiYnYuHEjiouLsXbtWgCX+slz587FkiVLEBkZCZ1Oh8WLFyMsLAxJSUkALhX4nj17YurUqUhPT0dAQAD++te/oqKiAomJibL8PSjZ5ec395I5EyKizkPW4jxhwgScOnUK6enpMJvNGDJkCHJzc6UTuiorKx32cIcPH47169dj0aJFWLBgASIjI5GTk4Po6GgpZt68eaitrcWMGTNQU1ODhIQE5Obmwt/fH8Clw+m5ublYuHAhfvvb36KxsRG33XYbPvjgAwwePLh9/wIU7lK/mSeDERG1N5UQQsidhDeyWq3QaDSwWCwd9hB3+XELHlj5JXqofVGWfjcPaxMR/YynagG/bemq2G8mIpIHv3HpqthvJiKSB4szOcV+MxGRfFicyakrr2++NbRj9tSJiJSKxZmcYr+ZiEg+/NYlpwp+uFScDTykTUTU7licqYVLz29mv5mISC4sztTCviorztX/p9/M+2kTEbU7FmdqobnfHK/rhS4+KpmzISLqfFicqYXL1zfzkDYRkRxYnMkB+81ERPJrU3FuamrCp59+ijVr1uDcuXMAgKqqKpw/f96tyVH7Y7+ZiEh+Lj+V6ujRo7j33ntRWVmJ+vp63H333ejRoweWLl2K+vp6rF692hN5Ujthv5mISH4u7znPmTMHcXFx+OmnnxAQECCNP/zww8jPz3drctT+2G8mIpKfy3vOX3zxBXbu3Ak/Pz+H8YiICBw/ftxtiVH7Y7+ZiEgZXN5zttvtsNlsLcZ//PFH9OjRwy1JkTykfrM/+81ERHJyuTjfc889WLFihfRepVLh/PnzyMjIwP333+/O3KidFRw+DQDQs99MRCQrlw9rL1++HCaTCbfeeivq6urwu9/9DocOHUJwcDA2bNjgiRypnew6zEPaRERK4HJx7tevH/bs2YNNmzZhz549OH/+PFJSUpCcnOxwghh5lyabHbvZbyYiUgSXi/OOHTswfPhwJCcnIzk5WRpvamrCjh078F//9V9uTZDax74Tl/vNt/D5zUREsnK55zxy5EicPXu2xbjFYsHIkSPdkhS1v+ZLqNhvJiKSn8vFWQgBlarll/eZM2fQrVs3tyRF7Y/9ZiIi5Wj1Ye0xY8YAuHR29qOPPgq1Wi0ts9ls+OabbzB8+HD3Z0gex34zEZGytLo4azQaAJf2nHv06OFw8pefnx/uvPNOTJ8+3f0Zksex30xEpCytLs5vvfUWgEt3Avvv//5vHsLuQNhvJiJSFpfP1s7IyPBEHiQj9puJiJTF5eIMAP/85z/x7rvvorKyEg0NDQ7LSktL3ZIYtQ/2m4mIlMfls7Wzs7Mxbdo0hISEoKysDPHx8ejduzcOHz6M++67zxM5kgft/c/9tAPZbyYiUgyXi/Prr7+OtWvXYuXKlfDz88O8efOQl5eH2bNnw2KxeCJH8qDLz2/uzX4zEZFCuFycKysrpUumAgICcO7cOQDA5MmTeW9tL3T5+c29ZM6EiIiauVyctVqtdIew/v37Y9euXQCAiooKCCHcmx15VJPNjt1HfgLAfjMRkZK4XJx/+9vf4sMPPwQATJs2DU899RTuvvtuTJgwAQ8//LDbEyTP2VtlxXn2m4mIFMfl4rx27VosXLgQAJCamor//d//xS233IIXX3wRb7zxhssJrFq1ChEREfD394der0dRUdE14zdv3oyoqCj4+/sjJiYGW7dudVguhEB6ejpCQ0MREBAAo9GIQ4cOtdjO//3f/0Gv1yMgIAA9e/ZEUlKSy7l7O/abiYiUyaXi3NTUhCVLlsBsNktjEydORHZ2Np588kn4+fm59OGbNm1CWloaMjIyUFpaisGDB8NkMuHkyZNO43fu3IlJkyYhJSUFZWVlSEpKQlJSEsrLy6WYrKwsZGdnY/Xq1SgsLES3bt1gMplQV1cnxfzrX//C5MmTMW3aNOzZswdfffUVfve737mUe0fAfjMRkUIJF3Xr1k1UVFS4uppT8fHxIjU1VXpvs9lEWFiYyMzMdBo/fvx4kZiY6DCm1+vFzJkzhRBC2O12odVqxbJly6TlNTU1Qq1Wiw0bNgghhGhsbBR9+/YVf/vb324od4vFIgAIi8VyQ9uRS2OTTdyWnisGPLtFfPtjjdzpEBF5JU/VApcPa48aNQrbt2+/4V8KGhoaUFJSAqPRKI35+PjAaDSioKDA6ToFBQUO8QBgMpmk+IqKCpjNZocYjUYDvV4vxZSWluL48ePw8fHB0KFDERoaivvuu89h79uZ+vp6WK1Wh5c3Y7+ZiEi5XL5D2H333YfnnnsO3377LWJjY1vcY3v06NGt2s7p06dhs9kQEhLiMB4SEoL9+/c7XcdsNjuNbz7M3vzzWjGHDx8GADz//PN49dVXERERgeXLl+M3v/kNDh48iF69nB/izczMxAsvvNCquXkD9puJiJTL5eL8xz/+EQDw6quvtlimUqlgs9luPCsPstvtAICFCxdi7NixAC491KNfv37YvHkzZs6c6XS9+fPnIy0tTXpvtVoRHh7u+YQ9pLk4GwbxEioiIqVxuTg3F7cbFRwcjC5duqC6utphvLq6Glqt1uk6Wq32mvHNP6urqxEaGuoQM2TIEACQxm+99VZpuVqtxsCBA1FZWXnVfNVqtcMzrL2Z4/XNPBmMiEhpXO45u4ufnx9iY2ORn58vjdntduTn58NgMDhdx2AwOMQDQF5enhSv0+mg1WodYqxWKwoLC6WY2NhYqNVqHDhwQIppbGzEkSNHMGDAALfNT8nK/9Nv1gR0xS1a9puJiJSmTU+lcpe0tDRMnToVcXFxiI+Px4oVK1BbW4tp06YBAKZMmYK+ffsiMzMTADBnzhyMGDECy5cvR2JiIjZu3Iji4mKsXbsWwKXD6nPnzsWSJUsQGRkJnU6HxYsXIywsTLqOOTAwELNmzUJGRgbCw8MxYMAALFu2DAAwbty49v9LkMHlfnMv+LDfTESkOLIW5wkTJuDUqVNIT0+H2WzGkCFDkJubK53QVVlZCR+fyzv3w4cPx/r167Fo0SIsWLAAkZGRyMnJQXR0tBQzb9481NbWYsaMGaipqUFCQgJyc3Ph7+8vxSxbtgy+vr6YPHkyLl68CL1ej23btqFnz57tN3kZXb6+mf1mIiIlUgnBG2K3hdVqhUajgcViQWCg9xwabrLZMfiFf6O2wYb/m52A28I0cqdEROS1PFULZOs5kzzKq6yobbCx30xEpGAuH9a+2s03VCoV1Gq1y7fwpPbFfjMRkfK5XJyDgoKgUl39S71fv3549NFHkZGR4dAvJmVgv5mISPlcLs7r1q3DwoUL8eijjyI+Ph4AUFRUhLfffhuLFi3CqVOn8Morr0CtVmPBggVuT5jarslmx+6KS8/i5vXNRETK5XJxfvvtt7F8+XKMHz9eGnvwwQcRExODNWvWID8/H/3798dLL73E4qww7DcTEXkHl48779y5E0OHDm0xPnToUOnhEgkJCde82xbJo/mQtp79ZiIiRXO5OIeHh+PNN99sMf7mm29K95o+c+ZMp7lm2JsU/MB+MxGRN3D5sPYrr7yCcePG4eOPP8Ydd9wBACguLsb+/fvxz3/+EwCwe/duTJgwwb2Z0g1ptNlRfKS538ziTESkZC4X59GjR2P//v1Ys2YNDh48CODSYyRzcnIQEREBAHj88cfdmiTduPLjFqnfHKXtIXc6RER0DW26fadOp8PLL7/s7lzIg3YdvrTXzH4zEZHytak419TUoKioCCdPnmzxCMkpU6a4JTFyL17fTETkPVwuzh999BGSk5Nx/vx5BAYGOtyQRKVSsTgrEPvNRETexeWztZ9++mk89thjOH/+PGpqavDTTz9Jr7Nnz3oiR7pB7DcTEXkXl4vz8ePHMXv2bNx0002eyIc8gP1mIiLv4nJxNplMKC4u9kQu5CHsNxMReReXe86JiYl45plnsG/fPsTExKBr164Oy0ePHu225OjGXdlvNgxicSYi8gYuF+fp06cDAF588cUWy1QqFWw2241nRW7T3G8OuqkrfhXCfjMRkTdwuTj//NIpUrYC3k+biMjr8IHLHVzzyWDsNxMReY9W7TlnZ2djxowZ8Pf3R3Z29jVjZ8+e7ZbE6Mbx+mYiIu+kEkKI6wXpdDoUFxejd+/e0Ol0V9+YSoXDhw+7NUGlslqt0Gg0sFgsCAxU5rORSyt/wpjXdyLopq4oXXQ3D2sTEbmZp2pBq/acKyoqnP6ZlI3PbyYi8k7sOXdg7DcTEXknl8/WttlsWLduHfLz850++GLbtm1uS47ajv1mIiLv5XJxnjNnDtatW4fExERER0c7PPiClOPb4xZcaLChJ69vJiLyOi4X540bN+Ldd9/F/fff74l8yE0u95t7s99MRORlXO45+/n54eabb/ZELuRGl/vNvWTOhIiIXNWmR0b++c9/RiuuwCKZOPSbeT9tIiKv4/Jh7S+//BKfffYZPv74Y9x2220tHnzx3nvvuS05aptvfrzcb/5lH/abiYi8jcvFOSgoCA8//LAnciE3Yb+ZiMi7uVScm5qaMHLkSNxzzz3QarWeyolu0OXnN7PfTETkjVzqOfv6+mLWrFmor693axKrVq1CREQE/P39odfrUVRUdM34zZs3IyoqCv7+/oiJicHWrVsdlgshkJ6ejtDQUAQEBMBoNOLQoUNOt1VfX48hQ4ZApVLh66+/dteUZHOp3/wTAPabiYi8lcsnhMXHx6OsrMxtCWzatAlpaWnIyMhAaWkpBg8eDJPJhJMnTzqN37lzJyZNmoSUlBSUlZUhKSkJSUlJKC8vl2KysrKQnZ2N1atXo7CwEN26dYPJZEJdXV2L7c2bNw9hYWFum4/cvvnRgouN7DcTEXk14aJNmzaJgQMHipUrV4qdO3eKPXv2OLxcFR8fL1JTU6X3NptNhIWFiczMTKfx48ePF4mJiQ5jer1ezJw5UwghhN1uF1qtVixbtkxaXlNTI9RqtdiwYYPDelu3bhVRUVFi7969AoAoKytrdd4Wi0UAEBaLpdXrtIe/bDskBjy7Rcz8e7HcqRARdXieqgUunxA2ceJEAI6PhlSpVBBCQKVSwWaztXpbDQ0NKCkpwfz586UxHx8fGI1GFBQUOF2noKAAaWlpDmMmkwk5OTkALj2Yw2w2w2g0Sss1Gg30ej0KCgqk/KurqzF9+nTk5OTgpptuum6u9fX1DofzrVZrq+fZnthvJiLyfi4XZ3c+ler06dOw2WwICQlxGA8JCcH+/fudrmM2m53Gm81maXnz2NVihBB49NFHMWvWLMTFxeHIkSPXzTUzMxMvvPBCq+YlF/abiYg6BpeL84ABAzyRR7tauXIlzp0757DHfj3z58932GO3Wq0IDw/3RHpt1txv7tXNj/1mIiIv5nJxbrZv3z5UVlaioaHBYXz06NGt3kZwcDC6dOmC6upqh/Hq6uqrXqql1WqvGd/8s7q6GqGhoQ4xQ4YMAXDpyVkFBQVQq9UO24mLi0NycjLefvvtFp+rVqtbxCsNn99MRNQxuFycDx8+jIcffhjffvut1GsGID2dypWes5+fH2JjY5Gfn4+kpCQAgN1uR35+Pp544gmn6xgMBuTn52Pu3LnSWF5eHgwGAwBAp9NBq9UiPz9fKsZWqxWFhYV4/PHHAQDZ2dlYsmSJtH5VVRVMJhM2bdoEvV7f6vyV5nK/mYe0iYi8WZseGanT6ZCfnw+dToeioiKcOXMGTz/9NF555RWXE0hLS8PUqVMRFxeH+Ph4rFixArW1tZg2bRoAYMqUKejbty8yMzOlzx8xYgSWL1+OxMREbNy4EcXFxVi7di2AS78kzJ07F0uWLEFkZCR0Oh0WL16MsLAw6ReA/v37O+TQvXt3AMCgQYPQr18/l+egBA1NV/SbWZyJiLyay8W5oKAA27ZtQ3BwMHx8fODj44OEhARkZmZi9uzZLl8DPWHCBJw6dQrp6ekwm80YMmQIcnNzpRO6Kisr4eNz+XLs4cOHY/369Vi0aBEWLFiAyMhI5OTkIDo6WoqZN28eamtrMWPGDNTU1CAhIQG5ubnw9/d3dbpe49vjNVK/ObJPd7nTISKiG6ASwrXHS/Xs2ROlpaXQ6XQYNGgQ/va3v2HkyJH44YcfEBMTgwsXLngqV0WxWq3QaDSwWCwIDAyUOx2s+ux7LPvkAO6L1uKN38fKnQ4RUafgqVrg8p5zdHQ09uzZA51OB71ej6ysLPj5+WHt2rUYOHCg2xIj17DfTETUcbhcnBctWoTa2loAwIsvvogHHngAv/71r9G7d29s2rTJ7QnS9bHfTETUsbhcnE0mk/Tnm2++Gfv378fZs2fRs2dP6Yxtal/sNxMRdSwuP/ii2ffff49PPvkEFy9eRK9evFWknHYdPguA1zcTEXUULhfnM2fOYNSoUfjlL3+J+++/HydOnAAApKSk4Omnn3Z7gnR9zf1mA2/ZSUTUIbhcnJ966il07doVlZWVDg+MmDBhAnJzc92aHF0f+81ERB2Pyz3nf//73/jkk09a3KwjMjISR48edVti1DrsNxMRdTwu7znX1tY6fcTi2bNnFX/v6Y6oud9858BePCGPiKiDcLk4//rXv8bf//536b1KpYLdbkdWVhZGjhzp1uTo+gp+4PXNREQdjcuHtbOysjBq1CgUFxejoaEB8+bNw969e3H27Fl89dVXnsiRrqKhyY7io817zizOREQdhct7ztHR0Th48CASEhLw0EMPoba2FmPGjEFZWRkGDRrkiRzpKr75sQZ1jXb2m4mIOpg2Pc9Zo9Fg4cKFDmM//vgjZsyYIT0dijzv8i072W8mIupI2nwTkp87c+YM3nzzTXdtjlrh8slgPKRNRNSRuK04U/tiv5mIqONicfZSzf3m3uw3ExF1OCzOXurKR0Sy30xE1LG0+oSwMWPGXHN5TU3NjeZCLrjy5iNERNSxtLo4azSa6y6fMmXKDSdE18d+MxFRx9bq4vzWW295Mg9ywZX95pvZbyYi6nDYc/ZCV96yk/1mIqKOh8XZC+2quHzzESIi6nhYnL1MfZMNJUf5/GYioo6MxdnLfPOjhf1mIqIOjsXZy+xiv5mIqMNjcfYy7DcTEXV8LM5e5Mp+s2EQ+81ERB0Vi7MXae43B3f3w6BfsN9MRNRRsTh7keZ+s579ZiKiDo3F2Ytc7jfzkDYRUUfG4uwlHPrNPBmMiKhDY3H2EnuOsd9MRNRZsDh7iebnN7PfTETU8SmiOK9atQoRERHw9/eHXq9HUVHRNeM3b96MqKgo+Pv7IyYmBlu3bnVYLoRAeno6QkNDERAQAKPRiEOHDknLjxw5gpSUFOh0OgQEBGDQoEHIyMhAQ0ODR+bnDs3Fmf1mIqKOT/bivGnTJqSlpSEjIwOlpaUYPHgwTCYTTp486TR+586dmDRpElJSUlBWVoakpCQkJSWhvLxcisnKykJ2djZWr16NwsJCdOvWDSaTCXV1dQCA/fv3w263Y82aNdi7dy9ee+01rF69GgsWLGiXObuK/WYiok5GyCw+Pl6kpqZK7202mwgLCxOZmZlO48ePHy8SExMdxvR6vZg5c6YQQgi73S60Wq1YtmyZtLympkao1WqxYcOGq+aRlZUldDpdq/O2WCwCgLBYLK1ep60KD58RA57dImL/9G9ht9s9/nlERNQ6nqoFsu45NzQ0oKSkBEajURrz8fGB0WhEQUGB03UKCgoc4gHAZDJJ8RUVFTCbzQ4xGo0Ger3+qtsEAIvFgl69rr5XWl9fD6vV6vBqL+w3ExF1LrIW59OnT8NmsyEkJMRhPCQkBGaz2ek6ZrP5mvHNP13Z5vfff4+VK1di5syZV801MzMTGo1GeoWHh197cm7UXJwN7DcTEXUKsvec5Xb8+HHce++9GDduHKZPn37VuPnz58NisUivY8eOtUt+fH4zEVHnI2txDg4ORpcuXVBdXe0wXl1dDa1W63QdrVZ7zfjmn63ZZlVVFUaOHInhw4dj7dq118xVrVYjMDDQ4dUe9hyzoL7JjuDuagz6Rbd2+UwiIpKXrMXZz88PsbGxyM/Pl8bsdjvy8/NhMBicrmMwGBziASAvL0+K1+l00Gq1DjFWqxWFhYUO2zx+/Dh+85vfIDY2Fm+99RZ8fJR5EOHyJVS92G8mIuokfOVOIC0tDVOnTkVcXBzi4+OxYsUK1NbWYtq0aQCAKVOmoG/fvsjMzAQAzJkzByNGjMDy5cuRmJiIjRs3ori4WNrzValUmDt3LpYsWYLIyEjodDosXrwYYWFhSEpKAnC5MA8YMACvvPIKTp06JeVztT12ufD6ZiKizkf24jxhwgScOnUK6enpMJvNGDJkCHJzc6UTuiorKx32aocPH47169dj0aJFWLBgASIjI5GTk4Po6GgpZt68eaitrcWMGTNQU1ODhIQE5Obmwt/fH8ClPe3vv/8e33//Pfr16+eQjxCiHWbdOuw3ExF1TiqhpGrkRaxWKzQaDSwWi8f6z4WHz2DC2l0I7q7G7oWjeFibiEhhPFULlNloJQDArsNnAbDfTETU2bA4Kxj7zUREnROLs0LVNdpQWsl+MxFRZ8TirFB7jtWgvsmOX/Tg9c1ERJ0Ni7NCXe43837aRESdDYuzQl158xEiIupcWJwViP1mIqLOjcVZga7sNw8MZr+ZiKizYXFWIPabiYg6NxZnBSo4fBoA+81ERJ0Vi7PCXOo31wBgv5mIqLNicVaYr4/VoIH9ZiKiTo3FWWGuvGUn+81ERJ0Ti7PC8PpmIiJicVaQK/vNBvabiYg6LRZnBWnuN/fpoYaO/WYiok6LxVlB2G8mIiKAxVlR+PxmIiICWJwVw/H6Zp4MRkTUmbE4KwT7zURE1IzFWSEKfmC/mYiILmFxVgj2m4mIqBmLswLUNdpQdqwGAPvNRETE4qwIZZXsNxMR0WUszgrQfEjbMIj9ZiIiYnFWBPabiYjoSizOMnPsN7M4ExERi7PsmvvNIYFqRPS+Se50iIhIAVicZcb7aRMR0c+xOMuM/WYiIvo5FmcZsd9MRETOKKI4r1q1ChEREfD394der0dRUdE14zdv3oyoqCj4+/sjJiYGW7dudVguhEB6ejpCQ0MREBAAo9GIQ4cOOcScPXsWycnJCAwMRFBQEFJSUnD+/Hm3z+1aSit/Yr+ZiIhakL04b9q0CWlpacjIyEBpaSkGDx4Mk8mEkydPOo3fuXMnJk2ahJSUFJSVlSEpKQlJSUkoLy+XYrKyspCdnY3Vq1ejsLAQ3bp1g8lkQl1dnRSTnJyMvXv3Ii8vD1u2bMGOHTswY8YMj8/3SrsOnwXAfjMREf2MkFl8fLxITU2V3ttsNhEWFiYyMzOdxo8fP14kJiY6jOn1ejFz5kwhhBB2u11otVqxbNkyaXlNTY1Qq9Viw4YNQggh9u3bJwCI3bt3SzEff/yxUKlU4vjx463K22KxCADCYrG0bqJOjFu9Uwx4dotYX3i0zdsgIiL5uKMWOCPrnnNDQwNKSkpgNBqlMR8fHxiNRhQUFDhdp6CgwCEeAEwmkxRfUVEBs9nsEKPRaKDX66WYgoICBAUFIS4uTooxGo3w8fFBYWGh08+tr6+H1Wp1eN2o1b+PxerfD8Nvo/rc8LaIiKjjkLU4nz59GjabDSEhIQ7jISEhMJvNTtcxm83XjG/+eb2YPn0cC6Kvry969ep11c/NzMyERqORXuHh4a2c5dX16uaHe6NDERLof8PbIiKijkP2nrO3mD9/PiwWi/Q6duyY3CkREVEHJWtxDg4ORpcuXVBdXe0wXl1dDa1W63QdrVZ7zfjmn9eL+fkJZ01NTTh79uxVP1etViMwMNDhRURE5AmyFmc/Pz/ExsYiPz9fGrPb7cjPz4fBYHC6jsFgcIgHgLy8PClep9NBq9U6xFitVhQWFkoxBoMBNTU1KCkpkWK2bdsGu90OvV7vtvkRERG1iVtPL2uDjRs3CrVaLdatWyf27dsnZsyYIYKCgoTZbBZCCDF58mTx3HPPSfFfffWV8PX1Fa+88or47rvvREZGhujatav49ttvpZiXX35ZBAUFiQ8++EB888034qGHHhI6nU5cvHhRirn33nvF0KFDRWFhofjyyy9FZGSkmDRpUqvz9tQZekRE5D08VQt85f7lYMKECTh16hTS09NhNpsxZMgQ5ObmSid0VVZWwsfn8g7+8OHDsX79eixatAgLFixAZGQkcnJyEB0dLcXMmzcPtbW1mDFjBmpqapCQkIDc3Fz4+18+8eqdd97BE088gVGjRsHHxwdjx45FdnZ2+02ciIjoKlRCCCF3Et7IarVCo9HAYrGw/0xE1El5qhbwbG0iIiKFkf2wtrdqPuDgjpuREBGRd2quAe4+CM3i3Ebnzp0DALfcjISIiLzbuXPnoNFo3LY99pzbyG63o6qqCj169GjzQyusVivCw8Nx7NixDtO35pyUr6PNB+h4c+po8wE63pya51NZWQmVSoWwsDCHk5dvFPec28jHxwf9+vVzy7Y64k1NOCfl62jzATrenDrafICONyeNRuOR+fCEMCIiIoVhcSYiIlIYFmcZqdVqZGRkQK1Wy52K23BOytfR5gN0vDl1tPkAHW9Onp4PTwgjIiJSGO45ExERKQyLMxERkcKwOBMRESkMizMREZHCsDjLaNWqVYiIiIC/vz/0ej2KiorkTqlVMjMzcccdd6BHjx7o06cPkpKScODAAYeYuro6pKamonfv3ujevTvGjh2L6upqmTJ2zcsvvwyVSoW5c+dKY944n+PHj+P3v/89evfujYCAAMTExKC4uFhaLoRAeno6QkNDERAQAKPRiEOHDsmY8bXZbDYsXrwYOp0OAQEBGDRoEP70pz853NNY6XPasWMHHnzwQYSFhUGlUiEnJ8dheWvyP3v2LJKTkxEYGIigoCCkpKTg/Pnz7TiLy641n8bGRjz77LOIiYlBt27dEBYWhilTpqCqqsphG0qaD3D9f6MrzZo1CyqVCitWrHAYd8ecWJxlsmnTJqSlpSEjIwOlpaUYPHgwTCYTTp48KXdq17V9+3akpqZi165dyMvLQ2NjI+655x7U1tZKMU899RQ++ugjbN68Gdu3b0dVVRXGjBkjY9ats3v3bqxZswa33367w7i3zeenn37CXXfdha5du+Ljjz/Gvn37sHz5cvTs2VOKycrKQnZ2NlavXo3CwkJ069YNJpMJdXV1MmZ+dUuXLsUbb7yBv/zlL/juu++wdOlSZGVlYeXKlVKM0udUW1uLwYMHY9WqVU6Xtyb/5ORk7N27F3l5ediyZQt27NiBGTNmtNcUHFxrPhcuXEBpaSkWL16M0tJSvPfeezhw4ABGjx7tEKek+QDX/zdq9v7772PXrl0ICwtrscwtcxIki/j4eJGamiq9t9lsIiwsTGRmZsqYVducPHlSABDbt28XQghRU1MjunbtKjZv3izFfPfddwKAKCgokCvN6zp37pyIjIwUeXl5YsSIEWLOnDlCCO+cz7PPPisSEhKuutxutwutViuWLVsmjdXU1Ai1Wi02bNjQHim6LDExUTz22GMOY2PGjBHJyclCCO+bEwDx/vvvS+9bk/++ffsEALF7924p5uOPPxYqlUocP3683XJ35ufzcaaoqEgAEEePHhVCKHs+Qlx9Tj/++KPo27evKC8vFwMGDBCvvfaatMxdc+KeswwaGhpQUlICo9Eojfn4+MBoNKKgoEDGzNrGYrEAAHr16gUAKCkpQWNjo8P8oqKi0L9/f0XPLzU1FYmJiQ55A945nw8//BBxcXEYN24c+vTpg6FDh+Kvf/2rtLyiogJms9lhThqNBnq9XrFzGj58OPLz83Hw4EEAwJ49e/Dll1/ivvvuA+Cdc7pSa/IvKChAUFAQ4uLipBij0QgfHx8UFha2e86uslgsUKlUCAoKAuCd87Hb7Zg8eTKeeeYZ3HbbbS2Wu2tOfPCFDE6fPg2bzYaQkBCH8ZCQEOzfv1+mrNrGbrdj7ty5uOuuuxAdHQ0AMJvN8PPzk/4HbBYSEgKz2SxDlte3ceNGlJaWYvfu3S2WeeN8Dh8+jDfeeANpaWlYsGABdu/ejdmzZ8PPzw9Tp06V8nb236BS5/Tcc8/BarUiKioKXbp0gc1mw0svvYTk5GQA8Mo5Xak1+ZvNZvTp08dhua+vL3r16qX4OdbV1eHZZ5/FpEmTpAdFeON8li5dCl9fX8yePdvpcnfNicWZbkhqairKy8vx5Zdfyp1Kmx07dgxz5sxBXl4e/P395U7HLex2O+Li4vA///M/AIChQ4eivLwcq1evxtSpU2XOrm3effddvPPOO1i/fj1uu+02fP3115g7dy7CwsK8dk6dRWNjI8aPHw8hBN544w2502mzkpIS/PnPf0ZpaWmbHxXcWjysLYPg4GB06dKlxdm+1dXV0Gq1MmXluieeeAJbtmzBZ5995vD4TK1Wi4aGBtTU1DjEK3V+JSUlOHnyJIYNGwZfX1/4+vpi+/btyM7Ohq+vL0JCQrxqPgAQGhqKW2+91WHslltuQWVlJQBIeXvTf4PPPPMMnnvuOUycOBExMTGYPHkynnrqKWRmZgLwzjldqTX5a7XaFieNNjU14ezZs4qdY3NhPnr0KPLy8hwer+ht8/niiy9w8uRJ9O/fX/quOHr0KJ5++mlEREQAcN+cWJxl4Ofnh9jYWOTn50tjdrsd+fn5MBgMMmbWOkIIPPHEE3j//fexbds26HQ6h+WxsbHo2rWrw/wOHDiAyspKRc5v1KhR+Pbbb/H1119Lr7i4OCQnJ0t/9qb5AMBdd93V4vK2gwcPYsCAAQAAnU4HrVbrMCer1YrCwkLFzunChQstHmbfpUsX2O12AN45pyu1Jn+DwYCamhqUlJRIMdu2bYPdboder2/3nK+nuTAfOnQIn376KXr37u2w3NvmM3nyZHzzzTcO3xVhYWF45pln8MknnwBw45zafh4b3YiNGzcKtVot1q1bJ/bt2ydmzJghgoKChNlslju163r88ceFRqMRn3/+uThx4oT0unDhghQza9Ys0b9/f7Ft2zZRXFwsDAaDMBgMMmbtmivP1hbC++ZTVFQkfH19xUsvvSQOHTok3nnnHXHTTTeJf/zjH1LMyy+/LIKCgsQHH3wgvvnmG/HQQw8JnU4nLl68KGPmVzd16lTRt29fsWXLFlFRUSHee+89ERwcLObNmyfFKH1O586dE2VlZaKsrEwAEK+++qooKyuTzl5uTf733nuvGDp0qCgsLBRffvmliIyMFJMmTVLcfBoaGsTo0aNFv379xNdff+3wXVFfX6/I+VxvTs78/GxtIdwzJxZnGa1cuVL0799f+Pn5ifj4eLFr1y65U2oVAE5fb731lhRz8eJF8cc//lH07NlT3HTTTeLhhx8WJ06ckC9pF/28OHvjfD766CMRHR0t1Gq1iIqKEmvXrnVYbrfbxeLFi0VISIhQq9Vi1KhR4sCBAzJle31Wq1XMmTNH9O/fX/j7+4uBAweKhQsXOnzRK31On332mdP/d6ZOnSqEaF3+Z86cEZMmTRLdu3cXgYGBYtq0aeLcuXMyzOba86moqLjqd8Vnn32myPlcb07OOCvO7pgTHxlJRESkMOw5ExERKQyLMxERkcKwOBMRESkMizMREZHCsDgTEREpDIszERGRwrA4ExERKQyLMxERkcKwOBMRAODUqVN4/PHH0b9/f6jVami1WphMJnz11VcAAJVKhZycHHmTJOok+MhIIgIAjB07Fg0NDXj77bcxcOBAVFdXIz8/H2fOnJE7NaJOh7fvJCLU1NSgZ8+e+PzzzzFixIgWyyMiInD06FHp/YABA3DkyBEAwAcffIAXXngB+/btk56tvHDhQvj6XvrdX6VS4fXXX8eHH36Izz//HKGhocjKysIjjzzSLnMj8kY8rE1E6N69O7p3746cnBzU19e3WL57924AwFtvvYUTJ05I77/44gtMmTIFc+bMwb59+7BmzRqsW7cOL730ksP6ixcvxtixY7Fnzx4kJydj4sSJ+O677zw/MSIvxT1nIgIA/Otf/8L06dNx8eJFDBs2DCNGjMDEiRNx++23A7i0B/z+++8jKSlJWsdoNGLUqFGYP3++NPaPf/wD8+bNQ1VVlbTerFmz8MYbb0gxd955J4YNG4bXX3+9fSZH5GW450xEAC71nKuqqvDhhx/i3nvvxeeff45hw4Zh3bp1V11nz549ePHFF6U97+7du2P69Ok4ceIELly4IMUZDAaH9QwGA/ecia6BJ4QRkcTf3x9333037r77bixevBh/+MMfkJGRgUcffdRp/Pnz5/HCCy9gzJgxTrdFRG3DPWciuqpbb70VtbW1AICuXbvCZrM5LB82bBgOHDiAm2++ucXLx+fy18uuXbsc1tu1axduueUWz0+AyEtxz5mIcObMGYwbNw6PPfYYbr/9dvTo0QPFxcXIysrCQw89BODSGdv5+fm46667oFar0bNnT6Snp+OBBx5A//798cgjj8DHxwd79uxBeXk5lixZIm1/8+bNiIuLQ0JCAt555x0UFRXhzTfflGu6RIrHE8KICPX19Xj++efx73//Gz/88AMaGxsRHh6OcePGYcGCBQgICMBHH32EtLQ0HDlyBH379pUupfrkk0/w4osvoqysDF27dkVUVBT+8Ic/YPr06QAunRC2atUq5OTkYMeOHQgNDcXSpUsxfvx4GWdMpGwszkTkUc7O8iaia2PPmYiISGFYnImIiBSGJ4QRkUexc0bkOu45ExERKQyLMxERkcKwOBMRESkMizMREZHCsDgTEREpDIszERGRwrA4ExERKQyLMxERkcKwOBMRESnM/wesvY0wHgcfQgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "total_training_steps = len(train_loader) * n_epochs\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "#plt.tight_layout(); plt.savefig(\"1.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3996b6-3f7a-420a-8584-c5760249f3d8",
   "metadata": {},
   "source": [
    "## D.2 Cosine decay"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5216214-de79-40cf-a733-b1049a73023c",
   "metadata": {},
   "source": [
    "- Another popular technique for training complex deep neural networks is cosine decay, which also adjusts the learning rate across training epochs\n",
    "- In cosine decay, the learning rate follows a cosine curve, decreasing from its initial value to near zero following a half-cosine cycle\n",
    "- This gradual reduction is designed to slow the pace of learning as the model begins to improve its weights; it reduces the risk of overshooting minima as the training progresses,  which is crucial for stabilizing the training in its later stages\n",
    "- Cosine decay is often preferred over linear decay for its smoother transition in learning rate adjustments, but linear decay is also used in practice (for example, [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4e8d2068-a057-4abf-b478-f02cc37191f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "min_lr = 0.1 * initial_lr\n",
    "track_lrs = []\n",
    "\n",
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "global_step = -1\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "        if global_step < warmup_steps:\n",
    "            # Linear warmup\n",
    "            lr = initial_lr + global_step * lr_increment  \n",
    "        else:\n",
    "            # Cosine annealing after warmup\n",
    "            progress = ((global_step - warmup_steps) / \n",
    "                        (total_training_steps - warmup_steps))\n",
    "            lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.param_groups[0][\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0e779e33-8a44-4984-bb23-be0603dc4158",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAEmCAYAAABYuVhFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOqElEQVR4nO3deVyU5fr48c8MywzroKAsKopLuYuCImrHForKFsrUPKZmlktaGpallVanfpZli2UuZWqLaXaKzK9RhpmlCIq4pbgkioqAiDAsss08vz+IqTmiAgLPDFzv12texPPcz3Ddalxz7xpFURSEEEIIYTO0agcghBBCCGuSnIUQQggbI8lZCCGEsDGSnIUQQggbI8lZCCGEsDGSnIUQQggbI8lZCCGEsDGSnIUQQggb46h2APbKbDaTnp6Oh4cHGo1G7XCEEEKoQFEU8vPzCQgIQKutu/auJOdaSk9Pp02bNmqHIYQQwgacOnWK1q1b19n7SXKuJQ8PD6DiL8TT01PlaIQQQqjBaDTSpk0bS06oK5Kca6myK9vT01OSsxBCNHF1PbwpE8KEEEIIGyPJWQghhLAxkpyFEEIIG6N6cl60aBHt2rVDr9cTFhZGYmLiFcuvW7eOzp07o9fr6dGjBxs3brS6/80333Dbbbfh7e2NRqNhz549l7xHcXExU6ZMwdvbG3d3d4YOHUpmZmZdVksIIYSoNVWT89q1a4mOjmbu3Lns3r2bXr16ERkZSVZWVpXlt2/fzsiRIxk/fjzJyclERUURFRXFgQMHLGUKCwsZNGgQb7zxxmV/7lNPPcX333/PunXr+PXXX0lPT+f++++v8/oJIYQQtaFRFEVR64eHhYXRt29fPvjgA6BiY482bdrwxBNP8Nxzz11SfsSIERQWFrJhwwbLtf79+xMcHMySJUusyp44cYKgoCCSk5MJDg62XM/Ly6NFixasXr2aBx54AICUlBS6dOlCfHw8/fv3r1bsRqMRg8FAXl6ezNYWQogmqr5ygWpLqUpLS0lKSmLWrFmWa1qtloiICOLj46t8Jj4+nujoaKtrkZGRxMTEVPvnJiUlUVZWRkREhOVa586dCQwMvGJyLikpoaSkxPK90Wis9s9sKhRF4fmYA+xJy0XnpEXnqEXn6ICzoxY3Zwe83XV4uzvj466jhbuOFh46Ar1d8dQ7qR26EELYFNWSc3Z2NiaTCV9fX6vrvr6+pKSkVPlMRkZGleUzMjKq/XMzMjJwdnbGy8urRu8zb948Xn755Wr/nKbocGY+qxPSavycj7uO9j5uBPm4EdTCjS7+nvRsZaCZm3M9RCmEELZPNiGpplmzZlm12it3hRF/i//zPAC9A72YPLgDJeVmSsvNlJSbKSwpJ7ughOyC0r++lpBpLLb6PvFEjtX7BTZ3pWdrA71ae9GnrRc9W3vh5KD6HEYhhKh3qiVnHx8fHBwcLpklnZmZiZ+fX5XP+Pn51aj85d6jtLSU3Nxcq9bz1d5Hp9Oh0+mq/XOaosrkfGtXX27rVr2/k/ziMk5kF3E8u4DU7EL+PFfIgTN5pGYXkpZTRFpOERv2nQXAXedI//bNGdjRh0EdfejY0l0OHRFCNEqqJWdnZ2dCQkKIi4sjKioKqJgQFhcXx9SpU6t8Jjw8nLi4OKZPn265tmnTJsLDw6v9c0NCQnByciIuLo6hQ4cCcPjwYdLS0mr0PsKa2ayQkFrR8g1v713t5zz0TvRobaBHa4PV9byiMvafyWPv6Vz2nsol8UQOuUVl/Hwoi58PVczm9/PUc1s3X27v5ke/oOY4SqtaCNFIqNqtHR0dzdixYwkNDaVfv368++67FBYWMm7cOADGjBlDq1atmDdvHgDTpk1j8ODBLFiwgCFDhrBmzRp27drFsmXLLO+Zk5NDWloa6enpQEXihYoWs5+fHwaDgfHjxxMdHU3z5s3x9PTkiSeeIDw8vNoztcWlDmUYybtYhpuzAz1aGa7+wFUYXJ0Y1MmHQZ18gIrkf/Cskd+PZbPtWDaJqTlkGIv5NP4kn8afpLmbM7d28eX2Hn4M6ugj3d9CCLumanIeMWIE586dY86cOWRkZBAcHExsbKxl0ldaWprV+ZgDBgxg9erVvPDCC8yePZtOnToRExND9+7dLWXWr19vSe4ADz74IABz587lpZdeAuCdd95Bq9UydOhQSkpKiIyM5MMPP2yAGjdelV3afeupBavVaujeykD3VgYmDe5AcZmJ+D/P88OBs2w6mElOYSlrd51i7a5T+Lg7ExXcimGhbbjer25PihFCiIag6jpneybrnK09umoXPx/KZNYdnZk4uEOD/uxyk5nE1Bxi/8hg4/4Msgv+XvLWs7WBYSGtuSe4FQYXWbIlhKhb9ZULJDnXkiTnv5nMCsGv/ER+cTnrpw6kZ2sv1WIpM5n59fA51iWdIu5QFuXmin/ers4OPBDSmocHtKN9C3fV4hNCNC6NbhMS0XgcTDeSX1yOh86Rrv7qflBxctAS0dWXiK6+nC8oIWZPOmt3pnEks8AyPn3j9S14ZGAQN3TykdneQgibJMlZXLMdxyvGm21txrS3u47xg4J4ZGA74v88zyfbUolLyWLL4XNsOXyO63zdmXJTR+7qGYCDVpK0EMJ2SHIW1yz+r+Qc3qH6S6gakkajYUBHHwZ09OFEdiErt59g3a5THMksYNqaPbz381Eev6kj9wYHyCxvIYRNkN9E4ppUTsYC6F+D9c1qaefjxkv3dGP7rFuYcet1eLk6cTy7kKfX7eXmBVtYk5hGmcmsdphCiCZOkrO4Jn+kGykoKcdT70gXlceba8Lg4sQTt3Ti92dv5rk7OuPt5sypnIs8981+It/Zyg/7zyJzJYUQapHkLK5JZZd2WHtvuxy3ddc5MmlwB35/9mZevKsr3m7OHM8uZPIXu7nvw+0k/FU/IYRoSJKcxTWp3HzEHrq0r8TF2YHxg4LY8syNPHlLJ1ydHdhzKpcRy3YwfuVOjmXlqx2iEKIJkeQsaq3MZGbXiZrvp23LPPRORN96HVueuZGH+gfioNUQl5LF7e/+xv/beIj84jK1QxRCNAGSnEWt7T+TR2GpCS9XJzo3sm0yW3roeTWqB5ue+hcRXVpSblZYtvU4Ny/4lW+TT8t4tBCiXklyFrVW2aUdFtQcrR2ON1dH+xbufDy2Lyse7ks7b1fO5Zfw1Nq9DFsSz8F0o9rhCSEaKUnOotYqNx9pLF3aV3JT55b8+NS/eCbyelycHNh18gJ3f/A7b8SmUFxmUjs8IUQjI8lZ1EppuZldJy4AEN7BR+VoGobO0YEpN3UkbsZg7ujuh8mssHjLn9zx3m+WXgQhhKgLkpxFrew7ncvFMhPN3Zzp1LJpHSQR4OXC4odCWDo6BF9PHanZhYz8aAfP/XcfeRdlwpgQ4tpJcha1Utml3b994x1vvprIbn5sih7MqLBAANbsPMWtb//KL4ezVI5MCGHvJDmLWolvQuPNV+Kpd+K1+3rw1cRw2vu4kZVfwrgVO3n+2/0UlpSrHZ4Qwk5JchY1VlJusow32/vmI3WlX1BzNk67gXED2wHwRUIady78jaSTOeoGJoSwS5KcRY3tPZVHSbkZH3cdHZvYePOV6J0cmHt3N1Y/GkaAQc/J80UMWxLPG7EplJbLYRpCiOqT5Cxq7O8tO5uj0TTN8eYrGdDRh9in/sXQPq0xK7B4y58MW7KdtPNFaocmhLATkpxFjcUfzwakS/tKPPVOLBjeiyUP9cHg4sTe03ncufA31u9NVzs0IYQdkOQsaqS4zMTutFwAwjtIcr6a27v788O0G+jbrhkFJeU8+WUyz369j6JSmSwmhLg8Sc6iRpLTciktN9PSQ0d7Hze1w7ELAV4ufPlYf568uSMaDazddYp7PtjG4Qw56UoIUTVJzqJG4o//fUSkjDdXn6ODlujbrueLR8No6aHjWFYBUYu2EZN8Ru3QhBA2SJKzqBHLftrSpV0rAzr48MO0G7ihkw8Xy0xMX7uHud8dkNncQggrkpxFtRWXmdhTOd4sk8Fqzdtdx8px/Xji5o4ArIo/yYPL4jmbd1HlyIQQtkKSs6i2pJMXKDWZ8fPU09bbVe1w7JqDVsOM265n+dhQPPWO7E7L5a6Fv7P9WLbaoQkhbIAkZ1Ft/+zSlvHmunFLF182PHEDXf09OV9YyuhPElm5LRVFUdQOTQihIknOotoqNx+RLu26FejtyjePD+D+3q0wmRVe+v4gs77ZL+PQQjRhkpxFtRSVlrP3dC4gm4/UB72TAwuG9+L5O7ug1VSccPXvj3ZwLr9E7dCEECqQ5CyqJenkBcpMCq28XGjT3EXtcBoljUbDY/9qzycP98VD78iukxe454PfOXAmT+3QhBANTJKzqJa/99OW8eb6duP1LYmZMpD2Ldw4m1fMA0u2E3sgQ+2whBANSPXkvGjRItq1a4derycsLIzExMQrll+3bh2dO3dGr9fTo0cPNm7caHVfURTmzJmDv78/Li4uREREcPToUasyR44c4d5778XHxwdPT08GDRrEL7/8Uud1a0z+3nykucqRNA0dWrjz7eMDGXxdC4rLzEz+Ionlv8tEMSGaClWT89q1a4mOjmbu3Lns3r2bXr16ERkZSVZWVpXlt2/fzsiRIxk/fjzJyclERUURFRXFgQMHLGXmz5/PwoULWbJkCQkJCbi5uREZGUlxcbGlzF133UV5eTmbN28mKSmJXr16cdddd5GRIa2TqhSWlLPvdEXXqmw+0nAMLk4sHxvKqLBAFAX+s+EgL63/A5NZErQQjZ6ion79+ilTpkyxfG8ymZSAgABl3rx5VZYfPny4MmTIEKtrYWFhysSJExVFURSz2az4+fkpb775puV+bm6uotPplC+//FJRFEU5d+6cAihbt261lDEajQqgbNq0qdqx5+XlKYCSl5dX7Wfs1S8pmUrbZzcog96IUzuUJslsNitLfz2mtH12g9L22Q3KIysSlYLiMrXDEkIo9ZcLVGs5l5aWkpSUREREhOWaVqslIiKC+Pj4Kp+Jj4+3Kg8QGRlpKZ+amkpGRoZVGYPBQFhYmKWMt7c3119/PZ9++imFhYWUl5ezdOlSWrZsSUhIyGXjLSkpwWg0Wr2aCkuXdpC0mtWg0WiY8K8OfDiqDzpHLXEpWYxYFk+WsfjqDwsh7JJqyTk7OxuTyYSvr6/VdV9f38t2L2dkZFyxfOXXK5XRaDT8/PPPJCcn4+HhgV6v5+233yY2NpZmzZpdNt558+ZhMBgsrzZt2tSswnZsx/EcQLq01XZnD3++nNAfbzdnDpwxErVITrYSorFSfUJYQ1MUhSlTptCyZUt+++03EhMTiYqK4u677+bs2bOXfW7WrFnk5eVZXqdOnWrAqNWTX1xmWcoj65vV1yewGd8+XjGTOz2vmAcWb+e3o+fUDksIUcdUS84+Pj44ODiQmZlpdT0zMxM/P78qn/Hz87ti+cqvVyqzefNmNmzYwJo1axg4cCB9+vThww8/xMXFhVWrVl02Xp1Oh6enp9WrKdh5IgeTWaGttysBXrK+2RYEervyzeQB9AtqTn5JOeNW7OS7PXL0pBCNiWrJ2dnZmZCQEOLi4izXzGYzcXFxhIeHV/lMeHi4VXmATZs2WcoHBQXh5+dnVcZoNJKQkGApU1RUBFSMb/+TVqvFbJbtEv+XpUtbWs02xcvVmc/G9+OeXgGUmxWmr93Dym2paoclhKgjqnZrR0dH89FHH7Fq1SoOHTrE5MmTKSwsZNy4cQCMGTOGWbNmWcpPmzaN2NhYFixYQEpKCi+99BK7du1i6tSpQMV48vTp03n11VdZv349+/fvZ8yYMQQEBBAVFQVUJPhmzZoxduxY9u7dy5EjR3jmmWdITU1lyJAhDf5nYOss+2nLeLPN0Tk68O6IYB4e0A5FgZe+P8jbPx2WtdBCNAKOav7wESNGcO7cOebMmUNGRgbBwcHExsZaJnSlpaVZtXAHDBjA6tWreeGFF5g9ezadOnUiJiaG7t27W8rMnDmTwsJCJkyYQG5uLoMGDSI2Nha9Xg9UdKfHxsby/PPPc/PNN1NWVka3bt347rvv6NWrV8P+Adi4vItl/JEu4822TKvVMPfurni7ObNg0xEWbj7G+cJSXrm3Ow5a2clNCHulUeRjdq0YjUYMBgN5eXmNdvz554OZPPrpLtr7uLH56RvVDkdcxRcJJ3kh5gCKAnf28OOdEcHoHB3UDkuIRq2+ckGTm60tqs+yvlm6tO3CqLC2LPp3H5wdtGzcn8EjK3dSUFKudlhCiFqQ5Cwu65+HXQj7cGcPf1aM64ubswPbjp1n5LIdnC+QYyeFsDeSnEWVcotKOZRRsQuaHHZhXwZ29OHLCf1p7ubM/jN5DFsaT0ae7CYmhD2R5CyqlJCag6JAx5butPTQqx2OqKGerb34elI4rbxcOH6ukOFL4zmVU6R2WEKIapLkLKr0d5e2tJrtVfsW7qyd2J+23q6k5RQxfGk8qdmFaoclhKgGSc6iSjv+mgwW3t5H5UjEtWjdzJWvJobToYUbZ/OKGb40niOZsh+3ELZOkrO4RE5hKSl/HaggLWf75+upZ+3EcDr7eXAuv4QHl+2w7JcuhLBNkpzFJRL+ajVf7+uBt7tO5WhEXfBx17FmQn96tjaQU1jKvz/aQXLaBbXDEkJchiRncYnKLm1pNTcuXq7OfP5oGKFtm2EsLuehjxMsH8SEELZFkrO4ROXmI7KfduPjqXdi1SP9GNDBm8JSE2NXJMqRk0LYIEnOwkp2QQlHMgsACAuS5NwYuekc+eThvtx4fQuKy8w8umoXW49IghbCltQqOZeXl/Pzzz+zdOlS8vMrJg6lp6dTUFBQp8GJhpfw1xGRnf08aObmrHI0or7onRxYOjqEiC6+lJSbeexTSdBC2JIaJ+eTJ0/So0cP7r33XqZMmcK5cxX/Q7/xxhs8/fTTdR6gaFjxx7MB6dJuCnSODnw4qg8RXVpKghbCxtQ4OU+bNo3Q0FAuXLiAi4uL5fp9991HXFxcnQYnGp7l/GbZT7tJcHbU8uGoEKsELWPQQqivxsn5t99+44UXXsDZ2brLs127dpw5c6bOAhMNLyu/mD/PFaLRyHhzU/K/CfrRVZKghVBbjZOz2WzGZDJdcv306dN4eHjUSVBCHTv+Gm/u6u+JwdVJ5WhEQ5IELYRtqXFyvu2223j33Xct32s0GgoKCpg7dy533nlnXcYmGph0aTdtkqCFsB01Ts4LFixg27ZtdO3aleLiYv79739burTfeOON+ohRNJCE43J+c1NXVYL+/Wi22mEJ0eRoFEVRavpQeXk5a9euZe/evRQUFNCnTx9GjRplNUGssTMajRgMBvLy8vD09FQ7nGuWaSwm7P/FodXAnrm34amXbu2mrLTczONfJPHzoSz0TlpWjetHmHxoE+IS9ZULapyct27dyoABA3B0dLS6Xl5ezvbt2/nXv/5VZ8HZssaWnGOSzzB97R56tjawfuogtcMRNqC03MzEz3bxy+FzuDk78NmjYfQJbKZ2WELYlPrKBTXu1r7pppvIycm55HpeXh433XRTnQQlGt4O6dIW/8PZUcvih0L+3urzk0Q5zUqIBlLj5KwoChqN5pLr58+fx83NrU6CEg3Psp+2JGfxD3onBz4eG0rfds3ILy5n9PIEDmfIedBC1DfHqxepcP/99wMVs7MffvhhdLq/jxI0mUzs27ePAQMG1H2Eot6l517k5PkiHLQaQttJt6Ww5upcsRf3Q8sT2Xsql1Ef72DtxHA6tHBXOzQhGq1qt5wNBgMGgwFFUfDw8LB8bzAY8PPzY8KECXz++ef1GauoJ5Vd2t1bGfCQiWCiCh56Jz4d14+u/p5kF5Qy6qME0s4XqR2WEI1WtVvOK1asACp2Anv66aelC7sRkfXNojoMrk58Nr4fDy7bwdGsAv798Q6+mhhOgFfTWaUhREOp8Zjz3LlzJTE3MnJ+s6gub3cdXzwaRjtvV05fuMi/P9pBlrFY7bCEaHSq3XL+p6+//pqvvvqKtLQ0SktLre7t3r27TgITDeNUThGnL1zEUashtK2MN4ura+mpZ/Vj/Rm+NJ4T54sY9XECayb0x9tdd/WHhRDVUuOW88KFCxk3bhy+vr4kJyfTr18/vL29OX78OHfccUd9xCjqUeV4c8/WBtx0tfqsJpqgAC8XvnysP36eeo5mFfDQ8kRyi0qv/qAQolpqnJw//PBDli1bxvvvv4+zszMzZ85k06ZNPPnkk+TlyRpIeyNd2qK22jR3ZfVjYfi46zh01sjYTxLJLy5TOywhGoUaJ+e0tDTLkikXFxfy8yvWPI4ePZovv/yybqMT9UpRFBL+OolKNh8RtdG+hTtfPBpGM1cn9p7O47FPd1FcdumpdUKImqlxcvbz87PsEBYYGMiOHTsASE1NpRbbdAsVncq5yJncizg5aAht21ztcISdut7Pg8/Gh+Guc2TH8Rymrt5NmcmsdlhC2LUaJ+ebb76Z9evXAzBu3Dieeuopbr31VkaMGMF9991X4wAWLVpEu3bt0Ov1hIWFkZiYeMXy69ato3Pnzuj1enr06MHGjRut7iuKwpw5c/D398fFxYWIiAiOHj16yfv83//9H2FhYbi4uNCsWTOioqJqHLu9iz9ecdpQcBsvXJwdVI5G2LPurQwsHxuKzlHLz4eymPn1Psxm+bAuRG3VODkvW7aM559/HoApU6bwySef0KVLF1555RUWL15co/dau3Yt0dHRzJ07l927d9OrVy8iIyPJysqqsvz27dsZOXIk48ePJzk5maioKKKiojhw4IClzPz581m4cCFLliwhISEBNzc3IiMjKS7+e7nHf//7X0aPHs24cePYu3cv27Zt49///ndN/yjs3g7p0hZ1KKy9N4sf6oOjVsO3yWd46fs/pDdNiNpSaqCsrEx5+eWXlVOnTtXkscvq16+fMmXKFMv3JpNJCQgIUObNm1dl+eHDhytDhgyxuhYWFqZMnDhRURRFMZvNip+fn/Lmm29a7ufm5io6nU758ssvLXVo1aqV8vHHH19T7Hl5eQqg5OXlXdP7qMVsNithr/2stH12g7Lt6Dm1wxGNSEzyaaXdcxuUts9uUN76MUXtcISoV/WVC2rUcnZ0dGT+/PmUl5df84eC0tJSkpKSiIiIsFzTarVEREQQHx9f5TPx8fFW5QEiIyMt5VNTU8nIyLAqYzAYCAsLs5TZvXs3Z86cQavV0rt3b/z9/bnjjjusWt9VKSkpwWg0Wr3s2YnzRWQYi3F20NJH1jeLOnRvcCv+c293AN7ffIyPfzuuckRC2J8ad2vfcsst/Prrr9f8g7OzszGZTPj6+lpd9/X1JSMjo8pnMjIyrli+8uuVyhw/XvGL4qWXXuKFF15gw4YNNGvWjBtvvLHKozArzZs3z2o/8TZt2tSgtrancn1zcKAXeicZbxZ166H+bXkm8noAXv2/Q3y185TKEQlhX2q868Qdd9zBc889x/79+wkJCblkK8977rmnzoKrD2ZzxSzS559/nqFDhwIV+4a3bt2adevWMXHixCqfmzVrFtHR0ZbvjUajXSdo2U9b1LfHb+yA8WIZS7ce57lv9uGud+TOHv5qhyWEXahxcn788ccBePvtty+5p9FoMJmqt8bRx8cHBwcHMjMzra5nZmbi5+dX5TN+fn5XLF/5NTMzE39/f6sywcHBAJbrXbt2tdzX6XS0b9+etLS0y8ar0+msjsm0Z4qiyOYjot5pNBqeu6MzxuIyvkw8xbQ1ybjpHBl8XQu1QxPC5tW4W9tsNl/2Vd3EDODs7ExISAhxcXFW7x0XF0d4eHiVz4SHh1uVB9i0aZOlfFBQEH5+flZljEYjCQkJljIhISHodDoOHz5sKVNWVsaJEydo27ZtteO3Z8ezCzmXX4Kzo5bgNl5qhyMaMY1Gw6tRPRjS058yk8Kkz5JIOnn54SMhRIUaJ+e6FB0dzUcffcSqVas4dOgQkydPprCwkHHjxgEwZswYZs2aZSk/bdo0YmNjWbBgASkpKbz00kvs2rWLqVOnAhW/CKZPn86rr77K+vXr2b9/P2PGjCEgIMCyjtnT05NJkyYxd+5cfvrpJw4fPszkyZMBGDZsWMP+Aaiksks7JLCZjDeLeueg1fDO8GAGX9eCi2UmHl6xk4Pp9j2hUoj6pupJByNGjODcuXPMmTOHjIwMgoODiY2NtUzoSktLQ6v9+/PDgAEDWL16NS+88AKzZ8+mU6dOxMTE0L17d0uZmTNnUlhYyIQJE8jNzWXQoEHExsai1+stZd58800cHR0ZPXo0Fy9eJCwsjM2bN9OsWdOYtSxd2qKhOTtqWfJQCGM+SWDniQuM+SSBdZMGEOQjx88KURWNosguAbVhNBoxGAzk5eXh6empdjjVpigKfV/7meyCUr6aGE6/INm2UzScvItljFy2g4NnjbTycuHryeH4G1zUDkuIWquvXKBqt7ZoeMeyCsguKEXvpKVXG4Pa4YgmxuDixKfj+9Hex40zuRd56OMEcgrlqEkh/pck5yamsks7tG1zdI4y3iwano+7js8eDSPAoOfPc4U8vCKRgpJr39hIiMakxsn5f3fJqnzl5+dTWiqfgG1d5eYj/dtLd7ZQTysvFz57NIzmbs7sO53HBDlqUggrNU7OXl5eNGvW7JKXl5cXLi4utG3blrlz51o2+xC2w2xWLIddyGQwobYOLdxZNa4fbs4ObP/zPNPWJFMuR00KAdQiOa9cuZKAgABmz55NTEwMMTExzJ49m1atWrF48WImTJjAwoULef311+sjXnENjmTlk1NYiouTAz1be6kdjhD0aG3go7GhODtq+fGPTGZ/u19OshKCWiylWrVqFQsWLGD48OGWa3fffTc9evRg6dKlxMXFERgYyGuvvcbs2bPrNFhxbXb8tb45tF0znBxkuoGwDQM6+PD+yN5M/jyJr3adppmrM7Pu7KJ2WEKoqsa/obdv307v3r0vud67d2/LyU+DBg264laYQh2yvlnYqshufrw+tCcAS7ceZ8mvf6ockRDqqnFybtOmDcuXL7/k+vLlyy0HQZw/f77JbOhhL8xmhYTUv8ab5bALYYOGh7bh+b9azK//kMKaRPmAL5quGndrv/XWWwwbNowffviBvn37ArBr1y5SUlL4+uuvAdi5cycjRoyo20jFNUnJyCe3qAw3Zwe6t5L1zcI2Pfav9uQUlbJ4y5/M/nY/Bhcn7pCTrEQTVOPkfM8995CSksLSpUs5cuQIUHGMZExMDO3atQOw7FUtbEdll3bfoOYy3ixs2szI68ktKv3rJKs9eOidGNTJR+2whGhQtdpbOygoSGZj2xk5v1nYi8qTrPIulrFxfwYTPtvF6sf6ywlqokmpVXLOzc0lMTGRrKysS9Yzjxkzpk4CE3XHZFZITK3cfESSs7B9DloN74wIJr94F78dzebhFYl8PSmcji091A5NiAZR4+T8/fffM2rUKAoKCvD09ESj0VjuaTQaSc426NBZI8bicjx0jnQLsJ9DOkTTpnN0YMlDIfz74wT2nsrloY8T+XpyOK2buaodmhD1rsaDjzNmzOCRRx6hoKCA3NxcLly4YHnl5Mgh6raosku7X1BzHGW8WdgRN50jKx/uS6eW7mQYixmzPJHsghK1wxKi3tX4N/WZM2d48skncXWVT6/24u/9tKVLW9ifZm7OfDY+jFZeLhzPrjgoI7+4TO2whKhXNU7OkZGR7Nq1qz5iEfWg3GQmMVX20xb2zc+g57Px/fB2c+bAGSOPrpKDMkTjVuMx5yFDhvDMM89w8OBBevTogZOTk9X9e+65p86CE9fuj3Qj+SXleOod6eIv483CfrVv4c6qR/oxctkOElJzmLo6mSUP9ZGhGtEoaZQa7jKv1V7+fwSNRoPJ1DQ+zRqNRgwGA3l5eXh62m7SW/rrn8z7IYWILr58PDZU7XCEuGY7jp9n7CeJlJSbGdqnNW8+0BOtVnP1B4WoB/WVC2r8kdNsNl/21VQSsz2R/bRFY9O/vTeL/t0HB62G/+4+zWsbD8lJVqLRkf6gRqzMZGan7KctGqGIrr7M/+ugjOW/p/LhFjkoQzQu1RpzXrhwIRMmTECv17Nw4cIrln3yySfrJDBx7Q6cyaOw1ISXqxOd/WTzBtG4DA1pTe7FMv6z4SBv/ngYg4sTD/Vvq3ZYQtSJaiXnd955h1GjRqHX63nnnXcuW06j0UhytiGVXdphQc1lTE40SuMHBZFbVMr7m4/x4ncH8HJ14q6eAWqHJcQ1q1ZyTk1NrfK/hW2T/bRFUxB963XkFJbyRUIaT62tOChj8HUt1A5LiGsiY86NVJnJzK4TFwDoL5PBRCOm0Wh45d7u3NXTnzKTwqTPktiddkHtsIS4JjVe52wymVi5ciVxcXFVHnyxefPmOgtO1N6+07lcLDPR3M2Z6+SwANHIOWg1vD08GGNxOVuPnGPcip18NTGc62WuhbBTNW45T5s2jWnTpmEymejevTu9evWyegnbUNml3b+9jDeLpsHZUcuSh/rQJ9CLvItljF6ewKmcIrXDEqJWatxyXrNmDV999RV33nlnfcQj6ki87KctmiBXZ0c+ebgvI5bu4HBmPg8tT+DrSQNo4aFTOzQhaqTGLWdnZ2c6duxYH7GIOlJSbiLpZMWYm0wGE02Nl6szn47vR5vmLpw8X8SYTxLJuygHZQj7UqsjI9977z3ZkceG7T2VR3GZGR93Zzq2dFc7HCEanK+nns8eCcPHXcehs0YeXbWTi6Wyg6GwHzXu1v7999/55Zdf+OGHH+jWrdslB1988803dRacqJ3K8eaw9t5oNDLeLJqmdj5ufPpIP0Ysi2fniQtMWb2bpaNDcJKDMoQdqPG/Ui8vL+677z4GDx6Mj48PBoPB6iXUV3l+s3Rpi6aua4AnnzzcF52jls0pWcz8eh9ms/T6CdtXo+RcXl7OTTfdxLx581ixYkWVr9pYtGgR7dq1Q6/XExYWRmJi4hXLr1u3js6dO6PX6+nRowcbN260uq8oCnPmzMHf3x8XFxciIiI4evRole9VUlJCcHAwGo2GPXv21Cp+W1JcZiLprzWectiFENC3XXMWP9QHR62Gb5PP8MqGgzIsJ2xejZKzo6MjkyZNoqSkpM4CWLt2LdHR0cydO5fdu3fTq1cvIiMjycrKqrL89u3bGTlyJOPHjyc5OZmoqCiioqI4cOCApcz8+fNZuHAhS5YsISEhATc3NyIjIykuLr7k/WbOnElAQOPZ7i85LZfScjMtPHS093FTOxwhbMLNnX15a1jFUs+V20+wMO6YyhEJcRVKDQ0ePFj59ttva/rYZfXr10+ZMmWK5XuTyaQEBAQo8+bNq7L88OHDlSFDhlhdCwsLUyZOnKgoiqKYzWbFz89PefPNNy33c3NzFZ1Op3z55ZdWz23cuFHp3Lmz8scffyiAkpycXO248/LyFEDJy8ur9jMN4e2fDittn92gPLF6t9qhCGFzVvx+XGn77Aal7bMblFXbU9UORzQC9ZULajwh7PHHH2fGjBmcPn2akJAQ3NysW2c9e/as9nuVlpaSlJTErFmzLNe0Wi0RERHEx8dX+Ux8fDzR0dFW1yIjI4mJiQEq9v7OyMggIiLCct9gMBAWFkZ8fDwPPvggAJmZmTz22GPExMTg6up61VhLSkqsegyMRmO169mQ5PxmIS7v4YFBXCgq4724o8xd/wcGFyfuDW6ldlhCXKLGybkyuf3z9CmNRoOiKGg0Gkym6i9XyM7OxmQy4evra3Xd19eXlJSUKp/JyMiosnxGRoblfuW1y5VRFIWHH36YSZMmERoayokTJ64a67x583j55ZerVS+1FJeZ2JOWC8jmI0JczvSITuQWlbIq/iQzvtqLp4sTN13fUu2whLBS4+TcGE6lev/998nPz7dqsV/NrFmzrFrsRqORNm3a1Ed4tbb75AVKTWb8PPW08756b4AQTZFGo2Hu3d3IvVjGd3vSmfx5Ep+PDyO0XXO1QxPCosbJuW3bujvM3MfHBwcHBzIzM62uZ2Zm4ufnV+Uzfn5+Vyxf+TUzMxN/f3+rMsHBwUDF4Rzx8fHodNZb+oWGhjJq1ChWrVp1yc/V6XSXlLc1/+zSlvXNQlyeVqvhrWG9yLtYxpbD53hk5U7WTgyni7+n2qEJAVzDkZEHDx4kNjaW9evXW71qwtnZmZCQEOLi4izXzGYzcXFxhIeHV/lMeHi4VXmATZs2WcoHBQXh5+dnVcZoNJKQkGAps3DhQvbu3cuePXvYs2ePZSnW2rVree2112pUB1vyz8MuhBBX5uSgZfGoEELbNsNYXM6YTxI5eb5Q7bCEAGrRcj5+/Dj33Xcf+/fvt4w1A5aWWk3GnAGio6MZO3YsoaGh9OvXj3fffZfCwkLGjRsHwJgxY2jVqhXz5s0DKk7FGjx4MAsWLGDIkCGsWbOGXbt2sWzZMksc06dP59VXX6VTp04EBQXx4osvEhAQQFRUFACBgYFWMbi7V2xx2aFDB1q3bl3TPxKbUFRazt7TuQCEt/dRNxgh7ISLswPLH+7LiKXxpGTkM3p5Il9PCqelp17t0EQTV6sjI4OCgsjKysLV1ZU//viDrVu3EhoaypYtW2ocwIgRI3jrrbeYM2cOwcHB7Nmzh9jYWMuErrS0NM6ePWspP2DAAFavXs2yZcvo1asXX3/9NTExMXTv3t1SZubMmTzxxBNMmDCBvn37UlBQQGxsLHp94/0fLunkBcpMCq28XGjT3EXtcISwGwYXJz4d34+23q6k5VQclJFbVKp2WKKJ0yhKzbbK8fHxYfPmzfTs2RODwUBiYiLXX389mzdvZsaMGSQnJ9dXrDbFaDRiMBjIy8vD01P9car5sSl8uOVP7u/TireHB6sdjhB2J+18EQ8s2U5Wfgk9Wxv4/NEwPPVOV39QNGn1lQtq3HI2mUx4eHgAFYk6PT0dqJgodvjw4ToLTNSM7KctxLUJ9Hbli0fDaO7mzL7TeYxbsZPCknK1wxJNVI2Tc/fu3dm7dy8AYWFhzJ8/n23btvHKK6/Qvn37Og9QXF1hSTn7TucBsr5ZiGvRydeDz8b3w1PvSNLJCzy6ahfFZXLUpGh4NU7OL7zwAmazGYBXXnmF1NRUbrjhBjZu3MjChQvrPEBxdTtP5FBuVmjdzIU2zWV9sxDXoluAgU/Hh+GucyT++HkmfJZESbkkaNGwapycIyMjuf/++wHo2LEjKSkpZGdnk5WVxc0331znAYqr23E8B5AubSHqSnAbL1aM64uLkwNbj5xj6upkykxmtcMSTUit1zkfO3aMH3/8kYsXL9K8uayrVZPspy1E3evbrjkfjw3F2VHLpoOZPLV2DyY5C1o0kBon5/Pnz3PLLbdw3XXXceedd1qWOY0fP54ZM2bUeYDiyvKLyzhwRsabhagPAzv6sPShEJwcNGzYd5Znvt6LWRK0aAA1Ts5PPfUUTk5OpKWlWZ3mNGLECGJjY+s0OHF1u05cwGRWaOvtSoCXrG8Woq7d1Lkl74/sg4NWwze7z/DCdweo4QpUIWqsxsn5p59+4o033rhkJ61OnTpx8uTJOgtMVE+8LKESot7d3t2Pt4f3QqOB1Qlp/GfDIUnQol7VODkXFhZWef5xTk6OzR8M0Rj9vZ+2JGch6tO9wa14Y2jFefWfbEvlzR8PS4IW9abGyfmGG27g008/tXyv0Wgwm83Mnz+fm266qU6DE1eWd7GMP9IrxptlMpgQ9W94aBv+c283AD7c8ifvxR1VOSLRWNX44Iv58+dzyy23sGvXLkpLS5k5cyZ//PEHOTk5bNu2rT5iFJexMzUHswLtfdzwlY36hWgQo8PbUVJu5tX/O8S7Px9Fg4ZpEZ3UDks0MrXaIezIkSMMGjSIe++9l8LCQu6//36Sk5Pp0KFDfcQoLqNyvDlMurSFaFCP3tCe5+7oDMA7Px/hvZ+lBS3qVo1bzgAGg4Hnn3/e6trp06eZMGGC5ehGUf92yPpmIVQzaXBFY+T1H1J45+cjANKCFnWm1puQ/K/z58+zfPnyuno7cRW5RaUcPGsEoH972QRGCDVMGtxBWtCiXtRZchYNKyE1B0WBDi3caOkh481CqEUStKgPkpztlHRpC2E7JEGLuibJ2U5Vrm8Ob++jciRCCJAELepWtSeEVZ5EdTm5ubnXGouoppzCUlIy8gEIk/FmIWzG/04SMysK0yM6odFoVI5M2JtqJ2eDwXDV+2PGjLnmgMTVJaZWtJqv83XHx112ZRPClvwzQb8Xd5TiMhPP3dFZErSokWon5xUrVtRnHKIG/u7SlvFmIWzRpMEdcHbQ8sqGgyzdepyLZSZeursbWq0kaFE9MuZshyo3H5H9tIWwXY8MCuL/3dcDjQY+jT/JrG/2y3nQotokOduZ7IISjmQWALIzmBC27t9hgSwY1gutBtbuOkX0V3soN5nVDkvYAUnOdibheA4Anf08aO7mrHI0Qoirub9Pa94f2QdHrYbv9qQzdXUypeWSoMWVSXK2M/HHswHp0hbCngzp6c+Sh0JwdtAS+0cGEz/bRXGZSe2whA2T5GxndvzVcpbNR4SwLxFdfVn+cCh6Jy2/HD7HIyt3UlhSrnZYwkZJcrYjWfnFHMsqQKOBsCBZ3yyEvbmhUwtWjeuHm7MD2/88z6iPE7hQWKp2WMIGSXK2I5Wt5i5+nni5ynizEPYorL03nz8ahperE3tO5TJsaTzpuRfVDkvYGEnOdkT20xaicegd2Ix1E8PxN+g5llXAA4u3cyyrQO2whA2R5GxHdsjmI0I0Gp18Pfh68gDat3AjPa+YYUu2s/dUrtphCRshydlOZBqLOZ5diFYDfWW8WYhGoZWXC19PGkCv1gYuFJUx8qMd/Hb0nNphCRsgydlOVHZpdwswYHBxUjkaIURdae7mzBeP9WdQRx+KSk08snInG/alqx2WUJlNJOdFixbRrl079Ho9YWFhJCYmXrH8unXr6Ny5M3q9nh49erBx40ar+4qiMGfOHPz9/XFxcSEiIoKjR/8+vu3EiROMHz+eoKAgXFxc6NChA3PnzqW01HZnTVr205bxZiEaHXedI8sfDmVIT3/KTApPfJnMZztOqh2WUJHqyXnt2rVER0czd+5cdu/eTa9evYiMjCQrK6vK8tu3b2fkyJGMHz+e5ORkoqKiiIqK4sCBA5Yy8+fPZ+HChSxZsoSEhATc3NyIjIykuLgYgJSUFMxmM0uXLuWPP/7gnXfeYcmSJcyePbtB6lwbf++nLV3aQjRGOkcHFj7Ym4f6B6Io8GLMAd78MQWz7MfdJGkURVH1bz4sLIy+ffvywQcfAGA2m2nTpg1PPPEEzz333CXlR4wYQWFhIRs2bLBc69+/P8HBwSxZsgRFUQgICGDGjBk8/fTTAOTl5eHr68vKlSt58MEHq4zjzTffZPHixRw/frxacRuNRgwGA3l5eXh6eta02jWSnnuRAa9vxkGrYc+cW/HQS7e2EI2Voii8F3eUd3+u6O27u1cAbz7QE72Tg8qRiarUVy5QteVcWlpKUlISERERlmtarZaIiAji4+OrfCY+Pt6qPEBkZKSlfGpqKhkZGVZlDAYDYWFhl31PqEjgzZtfvlVaUlKC0Wi0ejWUyvHm7q0MkpiFaOQ0Gg3TI67jzQd64qjV8P3edEYvl81KmhpVk3N2djYmkwlfX1+r676+vmRkZFT5TEZGxhXLV36tyXseO3aM999/n4kTJ1421nnz5mEwGCyvNm3aXLlydahyvFm6tIVoOoaFtmHVI/3w0Duy88QF7l+8nRPZhWqHJRqI6mPOajtz5gy33347w4YN47HHHrtsuVmzZpGXl2d5nTp1qsFi3JEq65uFaIoGdvThm8kDaOXlQmp2Ifd9uI2kkzlqhyUagKrJ2cfHBwcHBzIzM62uZ2Zm4ufnV+Uzfn5+Vyxf+bU675mens5NN93EgAEDWLZs2RVj1el0eHp6Wr0awukLRZzKuYiDVkPfdtJyFqKp6eTrwbdTBtDTshY6QZZaNQGqJmdnZ2dCQkKIi4uzXDObzcTFxREeHl7lM+Hh4VblATZt2mQpHxQUhJ+fn1UZo9FIQkKC1XueOXOGG2+8kZCQEFasWIFWa5udCJVd2j1bG3DTOaocjRBCDS099KyZ0J9bu/pSWm5m6upkPtxyDJXn84p6pHpGio6O5qOPPmLVqlUcOnSIyZMnU1hYyLhx4wAYM2YMs2bNspSfNm0asbGxLFiwgJSUFF566SV27drF1KlTgb8mU0yfzquvvsr69evZv38/Y8aMISAggKioKODvxBwYGMhbb73FuXPnyMjIuOyYtJosR0RKl7YQTZqrsyNLHgrhkYFBAMyPPcyMr/bKudCNlOpNsREjRnDu3DnmzJlDRkYGwcHBxMbGWiZ0paWlWbVqBwwYwOrVq3nhhReYPXs2nTp1IiYmhu7du1vKzJw5k8LCQiZMmEBubi6DBg0iNjYWvV4PVLS0jx07xrFjx2jdurVVPLb0SVRRFDnsQghh4aDVMOfurrT1duWVDQf5JvkMf2YXsmx0CL6eerXDE3VI9XXO9qoh1jmnnS/iX2/+gpODhr1zb8PVWfXPUkIIG7HtWDZTVu8mt6iMlh46lo0JJbiNl9phNTmNcp2zuLLKVnOv1l6SmIUQVgZ29OG7KQO5ztedrPwShi+N55vdp9UOS9QRSc42LF66tIUQV9DW241vHh9IRJeKiWLRX+3l5e//oMxkVjs0cY0kOdsoRVH+sfmIJGchRNXcdY4sGx3CEzd3BGDFthOMXLaDTGOxypGJayHJ2UadPF9EhrEYZwctIW2bqR2OEMKGabUaZtx2PctGh+Chc2TXyQsMWfib5QO+sD+SnG1UZZd2cKCXbHgvhKiW27r58f0Tg+js50F2QSkPLU9g6a9/2tQqFFE9kpxtlHRpCyFqo52PG98+PpD7e7fCZFaY90MKEz5LIrdIDs6wJ5KcbZDV+mZJzkKIGnJxdmDB8F78J6o7Tg4aNh3M5M73fmPnCdmX215IcrZBx7MLycovwdlRS+9AL7XDEULYIY1Gw+j+bfn28YEE+biRnlfMiKXxLIw7isks3dy2TpKzDars0u4j481CiGvUvZWB758YxP29W2FW4O1NRxj1sczmtnWSnG3Q313aPipHIoRoDNx1jrw9IpgFw3rh6uzAjuM5RL67lY37z6odmrgMSc42pmK8+a/DLmTzESFEHRoa0poNTwyiW4AnuUVlPP7FbqavSSavqEzt0MT/kORsY45lFZBdUILOUUuvNga1wxFCNDLtW7jz7eMDmXpTR7QaiNmTTuS7W/nt6Dm1QxP/IMnZxlR2aYe2a4bOUcabhRB1z9lRy9OR1/P15AG083Ylw1jM6OWJzPnuAIUl5WqHJ5DkbHPiZQmVEKKB9AlsxsZpN/BQ/0AAPo0/yW3vbGXL4SyVIxOSnG2I2fz3eLNsPiKEaAiuzo68GtWDTx/pRysvF87kXuThFTt5au0ecgpl4xK1SHK2IUezCsgpLMXFyYGerb3UDkcI0YT867oW/PTUv3hkYBAaDXybfIaIt3/luz1nZPtPFUhytiHxf2YDFePNzo7yVyOEaFhuOkfm3N2VbyYP4HpfD3IKS5m2Zg8PLU/gaGa+2uE1KZIBbEjleLN0aQsh1NQ7sBnfPzGIGbdeh7Ojlm3HznPHe7/x6oaD5BfLsquGIMnZRpjNCgmpsr5ZCGEbnB21PHFLJ35+ajC3dvWl3Kzw8e+p3PTWr/w36TRm2QK0XklythEpGfnkFpXh5uxAj1ayvlkIYRsCvV35aEwoK8f1JcjHjeyCEmas28t9H26T86LrkSRnGxFvWd/cHCcH+WsRQtiWG69vSez0G3j29s64Ojuw93QeIz/awbgViaRkGNUOr9GRLGAjLPtpS5e2EMJG6RwdmHxjB3595iZG92+Lo1bDL4fPccd7vzHjq72cyb2odoiNhiRnG2AyKyTI5iNCCDvRwkPHf6K6syl6MEN6+KMo8N/dp7nxzV+Y9c1+TuUUqR2i3ZPkbAMOnTViLC7HXedItwBPtcMRQohqCfJxY9GoPsRMGUh4e2/KTApfJqZx41tbeHrdXlKzC9UO0W5JcrYBlV3a/YKa4yjjzUIIOxPcxosvJ/Tnq4nh3NDJB5NZ4euk09yyYAtPfpnM/tN5aododxzVDkBgmfEoXdpCCHvWL6g5n40PY3faBT7YfIzNKVms35vO+r3phLZtxriBQUR285VGSDVIclZZuclMYqrspy2EaDz6BDbjk4f7cuBMHh/9dpz/23eWXScvsOvkBQIMekaHt2N4aGu83XVqh2qzNIpsmlorRqMRg8FAXl4enp61HyfedzqXez7YhqfekeQ5t+Gg1dRhlEIIob5MYzGf7zjJFwlplsM0HLUabunSkmEhbbjx+hZ225quq1zwv6TlrLLKLu1+Qd6SmIUQjZKvp54Zt13PlJs6sn5POp8nnGTf6Tx+/COTH//IxMddx9A+rbg3uBVd/D3QaOR3oSRnlf29n3ZzlSMRQoj6pXdyYHjfNgzv24aUDCNf7zrNt8lnyC4oYenW4yzdepy23q7c3s2P27v70au1F9om2miRbu1aqouujHKTmV4v/0RhqYn/e3IQ3QJk204hRNNSZjLzS0oW/919mi2Hz1FSbrbc8zfouaVLSwZ1bEF4B28MLk4qRlq1+urWtolO/kWLFtGuXTv0ej1hYWEkJiZesfy6devo3Lkzer2eHj16sHHjRqv7iqIwZ84c/P39cXFxISIigqNHj1qVycnJYdSoUXh6euLl5cX48eMpKCio87pdyf4zeRSWmjC4ONHFT9Y3CyGaHicHLbd182Pp6FB2v3grH47qw929AnBzduBsXjGf70hj0udJ9H7lJ6IWbeOtHw+z/Vh2oz8dS/Vu7bVr1xIdHc2SJUsICwvj3XffJTIyksOHD9OyZctLym/fvp2RI0cyb9487rrrLlavXk1UVBS7d++me/fuAMyfP5+FCxeyatUqgoKCePHFF4mMjOTgwYPo9XoARo0axdmzZ9m0aRNlZWWMGzeOCRMmsHr16gare2WXdlhQ8ybbdSOEEJXcdI7c2cOfO3v4U1xmYtuxbLYeOcfvx7L581whe07lsudULh/8cgyNBtr7uNGrtRc9Wxvo2caLji3d8dTbXuu6NlTv1g4LC6Nv37588MEHAJjNZtq0acMTTzzBc889d0n5ESNGUFhYyIYNGyzX+vfvT3BwMEuWLEFRFAICApgxYwZPP/00AHl5efj6+rJy5UoefPBBDh06RNeuXdm5cyehoaEAxMbGcuedd3L69GkCAgKuGndddGWM+SSRrUfOMffurowbGFSr9xBCiKbgbN5Fth07z7Zj2SSm5lx2H28fdx3tfdwI8nEjqIUb/gY9Ldx1+Hjo8HZzppmrc502hhrlbO3S0lKSkpKYNWuW5ZpWqyUiIoL4+Pgqn4mPjyc6OtrqWmRkJDExMQCkpqaSkZFBRESE5b7BYCAsLIz4+HgefPBB4uPj8fLysiRmgIiICLRaLQkJCdx3332X/NySkhJKSkos3xuN134Ky9O3XUf/9s256fpLewiEEEL8zd/gwgMhrXkgpDUA2QUl7Dudy95Teew7ncuBdCPn8kvILqh4JZ7IqfJ9HLQaDC5OuDg54OrsQP/23vwnqntDVqVaVE3O2dnZmEwmfH19ra77+vqSkpJS5TMZGRlVls/IyLDcr7x2pTL/22Xu6OhI8+bNLWX+17x583j55ZerWbPq6dnai56tver0PYUQoinwcddxc2dfbu789+/6/OIyTmQXcTy7gNTsQk5kF5Jp/DthXygqw2RWLGutAdp6u6oR/lWpPuZsL2bNmmXVYjcajbRp00bFiIQQQvyTh96JHq0N9Ghd9cqXMpOZnMJS8i6WUVRqoqikHE8bnAEOKidnHx8fHBwcyMzMtLqemZmJn59flc/4+fldsXzl18zMTPz9/a3KBAcHW8pkZWVZvUd5eTk5OTmX/bk6nQ6dTraaE0IIe+XkoMXXU4+vp17tUK5K1aVUzs7OhISEEBcXZ7lmNpuJi4sjPDy8ymfCw8OtygNs2rTJUj4oKAg/Pz+rMkajkYSEBEuZ8PBwcnNzSUpKspTZvHkzZrOZsLCwOqufEEIIUSuKytasWaPodDpl5cqVysGDB5UJEyYoXl5eSkZGhqIoijJ69Gjlueees5Tftm2b4ujoqLz11lvKoUOHlLlz5ypOTk7K/v37LWVef/11xcvLS/nuu++Uffv2Kffee68SFBSkXLx40VLm9ttvV3r37q0kJCQov//+u9KpUydl5MiR1Y47Ly9PAZS8vLw6+FMQQghhj+orF6g+5jxixAjOnTvHnDlzyMjIIDg4mNjYWMuErrS0NLTavxv4AwYMYPXq1bzwwgvMnj2bTp06ERMTY1njDDBz5kwKCwuZMGECubm5DBo0iNjYWMsaZ4AvvviCqVOncsstt6DVahk6dCgLFy5suIoLIYQQl6H6Omd7VV9r24QQQtiPRr19pxBCCCH+JslZCCGEsDGSnIUQQggbo/qEMHtVOVRfF9t4CiGEsE+VOaCup29Jcq6l/Px8ANklTAghBPn5+RgMVe9MVhsyW7uWzGYz6enpeHh4oNHU7oSTyi1AT5061WhmfEudbF9jqw80vjo1tvpA46tTZX3S0tLQaDQEBARYLfu9VtJyriWtVkvr1q3r5L08PT0bxT/Wf5I62b7GVh9ofHVqbPWBxlcng8FQL/WRCWFCCCGEjZHkLIQQQtgYSc4q0ul0zJ07t1GddiV1sn2NrT7Q+OrU2OoDja9O9V0fmRAmhBBC2BhpOQshhBA2RpKzEEIIYWMkOQshhBA2RpKzEEIIYWMkOato0aJFtGvXDr1eT1hYGImJiWqHVC3z5s2jb9++eHh40LJlS6Kiojh8+LBVmeLiYqZMmYK3tzfu7u4MHTqUzMxMlSKumddffx2NRsP06dMt1+yxPmfOnOGhhx7C29sbFxcXevTowa5duyz3FUVhzpw5+Pv74+LiQkREBEePHlUx4iszmUy8+OKLBAUF4eLiQocOHfjPf/5jtaexrddp69at3H333QQEBKDRaIiJibG6X534c3JyGDVqFJ6ennh5eTF+/HgKCgoasBZ/u1J9ysrKePbZZ+nRowdubm4EBAQwZswY0tPTrd7DluoDV/87+qdJkyah0Wh49913ra7XRZ0kOatk7dq1REdHM3fuXHbv3k2vXr2IjIwkKytL7dCu6tdff2XKlCns2LGDTZs2UVZWxm233UZhYaGlzFNPPcX333/PunXr+PXXX0lPT+f+++9XMerq2blzJ0uXLqVnz55W1+2tPhcuXGDgwIE4OTnxww8/cPDgQRYsWECzZs0sZebPn8/ChQtZsmQJCQkJuLm5ERkZSXFxsYqRX94bb7zB4sWL+eCDDzh06BBvvPEG8+fP5/3337eUsfU6FRYW0qtXLxYtWlTl/erEP2rUKP744w82bdrEhg0b2Lp1KxMmTGioKli5Un2KiorYvXs3L774Irt37+abb77h8OHD3HPPPVblbKk+cPW/o0rffvstO3bsICAg4JJ7dVInRaiiX79+ypQpUyzfm0wmJSAgQJk3b56KUdVOVlaWAii//vqroiiKkpubqzg5OSnr1q2zlDl06JACKPHx8WqFeVX5+flKp06dlE2bNimDBw9Wpk2bpiiKfdbn2WefVQYNGnTZ+2azWfHz81PefPNNy7Xc3FxFp9MpX375ZUOEWGNDhgxRHnnkEatr999/vzJq1ChFUeyvToDy7bffWr6vTvwHDx5UAGXnzp2WMj/88IOi0WiUM2fONFjsVfnf+lQlMTFRAZSTJ08qimLb9VGUy9fp9OnTSqtWrZQDBw4obdu2Vd555x3Lvbqqk7ScVVBaWkpSUhIRERGWa1qtloiICOLj41WMrHby8vIAaN68OQBJSUmUlZVZ1a9z584EBgbadP2mTJnCkCFDrOIG+6zP+vXrCQ0NZdiwYbRs2ZLevXvz0UcfWe6npqaSkZFhVSeDwUBYWJjN1mnAgAHExcVx5MgRAPbu3cvvv//OHXfcAdhnnf6pOvHHx8fj5eVFaGiopUxERARarZaEhIQGj7mm8vLy0Gg0eHl5AfZZH7PZzOjRo3nmmWfo1q3bJffrqk5y8IUKsrOzMZlM+Pr6Wl339fUlJSVFpahqx2w2M336dAYOHEj37t0ByMjIwNnZ2fI/YCVfX18yMjJUiPLq1qxZw+7du9m5c+cl9+yxPsePH2fx4sVER0cze/Zsdu7cyZNPPomzszNjx461xF3Vv0FbrdNzzz2H0Wikc+fOODg4YDKZeO211xg1ahSAXdbpn6oTf0ZGBi1btrS67+joSPPmzW2+jsXFxTz77LOMHDnSclCEPdbnjTfewNHRkSeffLLK+3VVJ0nO4ppMmTKFAwcO8Pvvv6sdSq2dOnWKadOmsWnTJvR6vdrh1Amz2UxoaCj/7//9PwB69+7NgQMHWLJkCWPHjlU5utr56quv+OKLL1i9ejXdunVjz549TJ8+nYCAALutU1NRVlbG8OHDURSFxYsXqx1OrSUlJfHee++xe/fuWh8VXF3Sra0CHx8fHBwcLpntm5mZiZ+fn0pR1dzUqVPZsGEDv/zyi9XxmX5+fpSWlpKbm2tV3lbrl5SURFZWFn369MHR0RFHR0d+/fVXFi5ciKOjI76+vnZVHwB/f3+6du1qda1Lly6kpaUBWOK2p3+DzzzzDM899xwPPvggPXr0YPTo0Tz11FPMmzcPsM86/VN14vfz87tk0mh5eTk5OTk2W8fKxHzy5Ek2bdpkdbyivdXnt99+Iysri8DAQMvvipMnTzJjxgzatWsH1F2dJDmrwNnZmZCQEOLi4izXzGYzcXFxhIeHqxhZ9SiKwtSpU/n222/ZvHkzQUFBVvdDQkJwcnKyqt/hw4dJS0uzyfrdcsst7N+/nz179lheoaGhjBo1yvLf9lQfgIEDB16yvO3IkSO0bdsWgKCgIPz8/KzqZDQaSUhIsNk6FRUVXXKYvYODA2azGbDPOv1TdeIPDw8nNzeXpKQkS5nNmzdjNpsJCwtr8JivpjIxHz16lJ9//hlvb2+r+/ZWn9GjR7Nv3z6r3xUBAQE888wz/Pjjj0Ad1qn289jEtVizZo2i0+mUlStXKgcPHlQmTJigeHl5KRkZGWqHdlWTJ09WDAaDsmXLFuXs2bOWV1FRkaXMpEmTlMDAQGXz5s3Krl27lPDwcCU8PFzFqGvmn7O1FcX+6pOYmKg4Ojoqr732mnL06FHliy++UFxdXZXPP//cUub1119XvLy8lO+++07Zt2+fcu+99ypBQUHKxYsXVYz88saOHau0atVK2bBhg5Kamqp88803io+PjzJz5kxLGVuvU35+vpKcnKwkJycrgPL2228rycnJltnL1Yn/9ttvV3r37q0kJCQov//+u9KpUydl5MiRNlef0tJS5Z577lFat26t7Nmzx+p3RUlJiU3W52p1qsr/ztZWlLqpkyRnFb3//vtKYGCg4uzsrPTr10/ZsWOH2iFVC1Dla8WKFZYyFy9eVB5//HGlWbNmiqurq3LfffcpZ8+eVS/oGvrf5GyP9fn++++V7t27KzqdTuncubOybNkyq/tms1l58cUXFV9fX0Wn0ym33HKLcvjwYZWivTqj0ahMmzZNCQwMVPR6vdK+fXvl+eeft/pFb+t1+uWXX6r8f2fs2LGKolQv/vPnzysjR45U3N3dFU9PT2XcuHFKfn6+CrW5cn1SU1Mv+7vil19+scn6XK1OVakqOddFneTISCGEEMLGyJizEEIIYWMkOQshhBA2RpKzEEIIYWMkOQshhBA2RpKzEEIIYWMkOQshhBA2RpKzEEIIYWMkOQshhBA2RpKzEAKAc+fOMXnyZAIDA9HpdPj5+REZGcm2bdsA0Gg0xMTEqBukEE2EHBkphABg6NChlJaWsmrVKtq3b09mZiZxcXGcP39e7dCEaHJk+04hBLm5uTRr1owtW7YwePDgS+63a9eOkydPWr5v27YtJ06cAOC7777j5Zdf5uDBg5azlZ9//nkcHSs++2s0Gj788EPWr1/Pli1b8Pf3Z/78+TzwwAMNUjch7JF0awshcHd3x93dnZiYGEpKSi65v3PnTgBWrFjB2bNnLd//9ttvjBkzhmnTpnHw4EGWLl3KypUree2116yef/HFFxk6dCh79+5l1KhRPPjggxw6dKj+KyaEnZKWsxACgP/+97889thjXLx4kT59+jB48GAefPBBevbsCVS0gL/99luioqIsz0RERHDLLbcwa9Ysy7XPP/+cmTNnkp6ebnlu0qRJLF682FKmf//+9OnThw8//LBhKieEnZGWsxACqBhzTk9PZ/369dx+++1s2bKFPn36sHLlyss+s3fvXl555RVLy9vd3Z3HHnuMs2fPUlRUZCkXHh5u9Vx4eLi0nIW4ApkQJoSw0Ov13Hrrrdx66628+OKLPProo8ydO5eHH364yvIFBQW8/PLL3H///VW+lxCidqTlLIS4rK5du1JYWAiAk5MTJpPJ6n6fPn04fPgwHTt2vOSl1f7962XHjh1Wz+3YsYMuXbrUfwWEsFPSchZCcP78eYYNG8YjjzxCz5498fDwYNeuXcyfP597770XqJixHRcXx8CBA9HpdDRr1ow5c+Zw1113ERgYyAMPPIBWq2Xv3r0cOHCAV1991fL+69atIzQ0lEGDBvHFF1+QmJjI8uXL1aquEDZPJoQJISgpKeGll17ip59+4s8//6SsrIw2bdowbNgwZs+ejYuLC99//z3R0dGcOHGCVq1aWZZS/fjjj7zyyiskJyfj5ORE586defTRR3nssceAiglhixYtIiYmhq1bt+Lv788bb7zB8OHDVayxELZNkrMQol5VNctbCHFlMuYshBBC2BhJzkIIIYSNkQlhQoh6JSNnQtSctJyFEEIIGyPJWQghhLAxkpyFEEIIGyPJWQghhLAxkpyFEEIIGyPJWQghhLAxkpyFEEIIGyPJWQghhLAxkpyFEEIIG/P/Ac1/dd9vNGhpAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "#plt.tight_layout(); plt.savefig(\"2.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7512808-b48d-4146-86a1-5931b1e3aec1",
   "metadata": {},
   "source": [
    "## D.3 Gradient clipping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a74f76-8d2b-4974-a03c-d645445cdc21",
   "metadata": {},
   "source": [
    "- Gradient clipping is yet another technique used to stabilize the training when training LLMs\n",
    "- By setting a threshold, gradients exceeding this limit are scaled down to a maximum magnitude to ensure that the updates to the model's parameters during backpropagation remain within a manageable range\n",
    "- For instance, using the `max_norm=1.0` setting in PyTorch's `clip_grad_norm_` method means that the norm of the gradients is clipped such that their maximum norm does not exceed 1.0\n",
    "- the \"norm\" refers to a measure of the gradient vector's length (or magnitude) in the parameter space of the model\n",
    "- Specifically, it's the L2 norm, also known as the Euclidean norm\n",
    "- Mathematically, for a vector $\\mathbf{v}$ with components $\\mathbf{v} = [v_1, v_2, \\ldots, v_n]$, the L2 norm is defined as:\n",
    "$$\n",
    "\\| \\mathbf{v} \\|_2 = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44838a6-4322-47b2-a935-c00d3a88355f",
   "metadata": {},
   "source": [
    "- The L2 norm is calculated similarly for matrices.\n",
    "- Let's assume our gradient matrix is:\n",
    "$$\n",
    "G = \\begin{bmatrix}\n",
    "1 & 2 \\\\\n",
    "2 & 4\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "- And we want to clip these gradients with a `max_norm` of 1.\n",
    "\n",
    "- First, we calculate the L2 norm of these gradients:\n",
    "$$\n",
    "\\|G\\|_2 = \\sqrt{1^2 + 2^2 + 2^2 + 4^2} = \\sqrt{25} = 5\n",
    "$$\n",
    "\n",
    "- Since $\\|G\\|_2 = 5$ is greater than our `max_norm` of 1, we need to scale down the gradients so that their norm is exactly 1. The scaling factor is calculated as $\\frac{max\\_norm}{\\|G\\|_2} = \\frac{1}{5}$.\n",
    "\n",
    "- Therefore, the scaled gradient matrix $G'$ will be as follows:\n",
    "$$\n",
    "G' = \\frac{1}{5} \\times G = \\begin{bmatrix}\n",
    "\\frac{1}{5} & \\frac{2}{5} \\\\\n",
    "\\frac{2}{5} & \\frac{4}{5}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeb0c3c1-2cff-46f5-8127-24412184428c",
   "metadata": {},
   "source": [
    "- Let's see this in action\n",
    "- First, we initialize a new model and calculate the loss for a training batch like we would do in the regular training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e199e1ff-58c4-413a-855e-5edbe9292649",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import calc_loss_batch\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "\n",
    "loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b60f3a-15ec-4846-838d-fdef3df99899",
   "metadata": {},
   "source": [
    "- If we call `.backward()`, PyTorch will calculate the gradients and store them in a `.grad` attribute for each weight (parameter) matrix\n",
    "- Let's define a utility function to calculate the highest gradient based on all model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e70729a3-24d1-411d-a002-2529cd3a8a9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0373)\n"
     ]
    }
   ],
   "source": [
    "def find_highest_gradient(model):\n",
    "    max_grad = None\n",
    "    for param in model.parameters():\n",
    "        if param.grad is not None:\n",
    "            grad_values = param.grad.data.flatten()\n",
    "            max_grad_param = grad_values.max()\n",
    "            if max_grad is None or max_grad_param > max_grad:\n",
    "                max_grad = max_grad_param\n",
    "    return max_grad\n",
    "\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "734f30e6-6b24-4d4b-ae91-e9a4b871113f",
   "metadata": {},
   "source": [
    "- Applying gradient clipping, we can see that the largest gradient is now substantially smaller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fa81ef8b-4280-400f-a93e-5210f3e62ff0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0166)\n"
     ]
    }
   ],
   "source": [
    "torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b62c2af0-dac3-4742-be4b-4292c6753099",
   "metadata": {},
   "source": [
    "## D.4 The modified training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76715332-94ec-4185-922a-75cb420819d5",
   "metadata": {},
   "source": [
    "- Now let's add the three concepts covered above (learning rate warmup, cosine decay, and gradient clipping) to the `train_model_simple` function covered in chapter 5 to create the more sophisticated `train_model` function below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "46eb9c84-a293-4016-a523-7ad726e171e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import evaluate_model, generate_and_print_sample\n",
    "\n",
    "\n",
    "def train_model(model, train_loader, val_loader, optimizer, device, n_epochs,\n",
    "                eval_freq, eval_iter, start_context, warmup_steps=10,\n",
    "                initial_lr=3e-05, min_lr=1e-6):\n",
    "\n",
    "    train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []\n",
    "    tokens_seen, global_step = 0, -1\n",
    "\n",
    "    # Retrieve the maximum learning rate from the optimizer\n",
    "    peak_lr = optimizer.param_groups[0][\"lr\"]\n",
    "\n",
    "    # Calculate the total number of iterations in the training process\n",
    "    total_training_steps = len(train_loader) * n_epochs\n",
    "\n",
    "    # Calculate the learning rate increment during the warmup phase\n",
    "    lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        model.train()\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            global_step += 1\n",
    "\n",
    "            # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "            if global_step < warmup_steps:\n",
    "                # Linear warmup\n",
    "                lr = initial_lr + global_step * lr_increment  \n",
    "            else:\n",
    "                # Cosine annealing after warmup\n",
    "                progress = ((global_step - warmup_steps) / \n",
    "                            (total_training_steps - warmup_steps))\n",
    "                lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n",
    "\n",
    "            # Apply the calculated learning rate to the optimizer\n",
    "            for param_group in optimizer.param_groups:\n",
    "                param_group[\"lr\"] = lr\n",
    "            track_lrs.append(lr)  # Store the current learning rate\n",
    "\n",
    "            # Calculate and backpropagate the loss\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward()\n",
    "\n",
    "            # Apply gradient clipping after the warmup phase to avoid exploding gradients\n",
    "            if global_step > warmup_steps:\n",
    "                torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "            \n",
    "            optimizer.step()\n",
    "            tokens_seen += input_batch.numel()\n",
    "\n",
    "            # Periodically evaluate the model on the training and validation sets\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader,\n",
    "                    device, eval_iter\n",
    "                )\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                track_tokens_seen.append(tokens_seen)\n",
    "                # Print the current losses\n",
    "                print(f\"Ep {epoch+1} (Iter {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
    "\n",
    "        # Generate and print a sample from the model to monitor progress\n",
    "        generate_and_print_sample(\n",
    "            model, train_loader.dataset.tokenizer,\n",
    "            device, start_context\n",
    "        )\n",
    "\n",
    "    return train_losses, val_losses, track_tokens_seen, track_lrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "55fcd247-ba9d-4b93-a757-0f7ce04fee41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Iter 000000): Train loss 10.914, Val loss 10.940\n",
      "Ep 1 (Iter 000005): Train loss 8.415, Val loss 8.846\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 2 (Iter 000010): Train loss 6.291, Val loss 6.838\n",
      "Ep 2 (Iter 000015): Train loss 6.057, Val loss 6.834\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 3 (Iter 000020): Train loss 6.630, Val loss 7.308\n",
      "Ep 3 (Iter 000025): Train loss 5.847, Val loss 6.838\n",
      "Every effort moves you, I, I, I, I, I, I, I, I, I had, I, I, I, I, I, I, I had, I, I, I, I, I had, I had, I\n",
      "Ep 4 (Iter 000030): Train loss 8.879, Val loss 9.904\n",
      "Ep 4 (Iter 000035): Train loss 5.453, Val loss 6.866\n",
      "Every effort moves you.                                                 \n",
      "Ep 5 (Iter 000040): Train loss 5.454, Val loss 6.880\n",
      "Every effort moves you. \"I the the the the the the the the the the a-- the, and I had been, and I had been, and I had been his the picture, and I had been, and, and I had been, and I\n",
      "Ep 6 (Iter 000045): Train loss 5.279, Val loss 6.642\n",
      "Ep 6 (Iter 000050): Train loss 4.472, Val loss 6.542\n",
      "Every effort moves you, and, and in the picture--I to me--as of the picture. Gisburn, and, and, and to see of the picture--as of the picture--as, and, and, and, and, and, and\n",
      "Ep 7 (Iter 000055): Train loss 3.858, Val loss 6.370\n",
      "Ep 7 (Iter 000060): Train loss 2.623, Val loss 6.249\n",
      "Every effort moves you know the fact that a little of the picture to the fact of the last of the house of the fact of the fact of the fact of Jack's it.    \"Oh, I had the donkey. \"Oh, I was\n",
      "Ep 8 (Iter 000065): Train loss 1.856, Val loss 6.311\n",
      "Ep 8 (Iter 000070): Train loss 2.269, Val loss 6.284\n",
      "Every effort moves you know,\" was not that my dear, the picture, and Mrs.                                    \n",
      "Ep 9 (Iter 000075): Train loss 1.585, Val loss 6.299\n",
      "Ep 9 (Iter 000080): Train loss 1.382, Val loss 6.319\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. Gisburn. Gisburn's past!  \"Oh, and the moment--as Jack himself, and; and as I had been the man of the hour. \n",
      "Ep 10 (Iter 000085): Train loss 1.089, Val loss 6.391\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the honour being _mine_--because he had always his\n",
      "Ep 11 (Iter 000090): Train loss 0.874, Val loss 6.409\n",
      "Ep 11 (Iter 000095): Train loss 0.432, Val loss 6.413\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 12 (Iter 000100): Train loss 0.389, Val loss 6.485\n",
      "Ep 12 (Iter 000105): Train loss 0.405, Val loss 6.521\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 13 (Iter 000110): Train loss 0.293, Val loss 6.543\n",
      "Ep 13 (Iter 000115): Train loss 0.204, Val loss 6.558\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 14 (Iter 000120): Train loss 0.239, Val loss 6.576\n",
      "Ep 14 (Iter 000125): Train loss 0.183, Val loss 6.580\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 15 (Iter 000130): Train loss 0.165, Val loss 6.580\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "\n",
    "peak_lr = 5e-4\n",
    "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)\n",
    "\n",
    "n_epochs = 15\n",
    "train_losses, val_losses, tokens_seen, lrs = train_model(\n",
    "    model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,\n",
    "    eval_freq=5, eval_iter=1, start_context=\"Every effort moves you\",\n",
    "    warmup_steps=10, initial_lr=1e-5, min_lr=1e-5\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "827e8d5e-0872-4b90-98ac-200c80ee2d53",
   "metadata": {},
   "source": [
    "- Looking at the results above, we can see that the model starts out generating incomprehensible strings of words, whereas, towards the end, it's able to produce grammatically more or less correct sentences\n",
    "- If we were to check a few passages it writes towards the end, we would find that they are contained in the training set verbatim -- it simply memorizes the training data\n",
    "- Note that the overfitting here occurs because we have a very, very small training set, and we iterate over it so many times\n",
    "  - The LLM training here primarily serves educational purposes; we mainly want to see that the model can learn to produce coherent text\n",
    "  - Instead of spending weeks or months on training this model on vast amounts of expensive hardware, we load the pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9decec45-4fdf-4ff6-85a7-1806613f8af7",
   "metadata": {},
   "source": [
    "- A quick check that the learning rate behaves as intended"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d8ebb8d2-8308-4a83-a2a6-730c3bf84452",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEmCAYAAACdy8LUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQqklEQVR4nO3deVxUVf8H8M8dhhnWmRERBpTN3BUVRSZcM0k0yq1SiVzQxMxSs0UtlceeDLc2zdzyJ/Zkbi1quBSBubEK7ghikuAyICADyD5zfn8QtyZRGR28M/B9v17z4uGe7535Hp7kyzn3nHs5xhgDIYQQQsyKSOgECCGEEGI4KuCEEEKIGaICTgghhJghKuCEEEKIGaICTgghhJghKuCEEEKIGaICTgghhJghKuCEEEKIGRILnUBTptPpcOPGDdjb24PjOKHTIYQQIgDGGEpKSuDq6gqRyHjjZirgjejGjRtwc3MTOg1CCCEmICcnB23atDHa+1EBb0T29vYAav9Pk8lkAmdDCCFECMXFxXBzc+NrgrFQAW9EddPmMpmMCjghhDRzxr6USovYCCGEEDNEBZwQQggxQ1TACSGEEDMkeAFfu3YtPD09YWVlBZVKhaSkpPvG7969G506dYKVlRW8vb1x4MABvXbGGBYvXgwXFxdYW1sjICAAmZmZejFLly5F3759YWNjA4VCUe/nZGdnIygoCDY2NnBycsK7776LmpqaR+orIYQQYiyCFvCdO3di7ty5CA8PR2pqKnr06IHAwEDk5eXVGx8XF4fg4GBMnToVp06dwqhRozBq1CicP3+ej1mxYgVWr16N9evXIzExEba2tggMDERFRQUfU1VVhZdeegkzZsyo93O0Wi2CgoJQVVWFuLg4bN26FZGRkVi8eLFxfwCEEELIw2IC8vPzYzNnzuS/12q1zNXVlUVERNQbP3bsWBYUFKR3TKVSsenTpzPGGNPpdEypVLKVK1fy7UVFRUwqlbLt27ff9X5btmxhcrn8ruMHDhxgIpGIqdVq/ti6deuYTCZjlZWVDe6fRqNhAJhGo2nwOYQQQpqWxqoFgm0jq6qqQkpKChYsWMAfE4lECAgIQHx8fL3nxMfHY+7cuXrHAgMDsWfPHgBAVlYW1Go1AgIC+Ha5XA6VSoX4+HiMHz++QbnFx8fD29sbzs7Oep8zY8YMXLhwAT4+PvWeV1lZicrKSv774uLiBn2eKUi8UoCIg+nQ6hgsLThIxCJYWoggFYsgs7KEwkYCB9u6rxI42UvRpoUNnOylEInoLnOEEPK4CVbA8/PzodVq9YokADg7OyM9Pb3ec9Rqdb3xarWab687dq+YhrjX5/zzM+oTERGBJUuWNPhzTMn/Eq7idE6RwedJLERo3cIabVpYw83BBh2d7dFRaY9OSnsobCTGT5QQQggAupGLUS1YsEBvhqDu7jvm4NrtcgDAm0+3Q1dXOaq1OlRrdais0UFTXo3bZVUoulONwrIq3L5TBXVxBW5qKlCl1SEr/w6y8u/c9Z7OMik6KWXo6aZAb48W6OmugMzK8nF3jRBCmiTBCrijoyMsLCyQm5urdzw3NxdKpbLec5RK5X3j677m5ubCxcVFL6Znz54Nzk2pVN61Gr7uc++VGwBIpVJIpdIGf44pqSvggV2V6NZa3qBzarQ6qIsrkFNYjmu3y5CVfwcZ6hKkq0twvagcucWVyC2+hSOXbgEAOA7o4GSPXh4t8GRbB/Rv54iWdub58yKEEKEJVsAlEgl69+6NmJgYjBo1CkDt07tiYmLwxhtv1HuOv78/YmJiMGfOHP5YdHQ0/P39AQBeXl5QKpWIiYnhC3ZxcTESExPvueL8Xp+zdOlS5OXlwcnJif8cmUyGLl26GN5ZE1dRrUV+ae21+zYtrBt8nthChDYtbNCmhQ2AlnptJRXVuJRbggs3inEquwgpV28ju7AMGbklyMgtwfakbABAt9YyDGjfCgPaO6K3RwtIxRZG6xchhDRlgk6hz507F5MmTYKvry/8/Pzw+eef486dOwgNDQUATJw4Ea1bt0ZERAQAYPbs2Rg0aBA++eQTBAUFYceOHTh58iQ2btwIoPY+s3PmzMFHH32E9u3bw8vLC4sWLYKrqyv/RwJQu8e7sLAQ2dnZ0Gq1OH36NACgXbt2sLOzw9ChQ9GlSxdMmDABK1asgFqtxsKFCzFz5kyzHWHfT93o204qhtzaOFPc9laW6O3hgN4eDphY+/cVbpVUIjX7Nk7+WYjjlwtw8WYxzl+vfa37/Q/YS8V4urMThnVVYlDHVrCR0BUeQgi5F0F/Q44bNw63bt3C4sWLoVar0bNnTxw6dIhfMJadna337NS+ffviu+++w8KFC/H++++jffv22LNnD7p168bHvPfee7hz5w7CwsJQVFSE/v3749ChQ7CysuJjFi9ejK1bt/Lf160qP3z4MJ566ilYWFggKioKM2bMgL+/P2xtbTFp0iR8+OGHjf0jEcS122UAakffjfnc8lb2UgR2VSKwa+1liLySChzPzMexzHwcy7yF/NIq7D19A3tP34CVpQiDOrTCs94ueKaLMxVzQgj5F44xxoROoqkqLi6GXC6HRqMx6aeRfZtwFQv3nEdAZyd8PamPIDnodAyncopw6PxNHDyv5mcFAMBWYoHh3i54oVcbqLwcaNsaIcSsNFYtoGEN4Ytl7bVsYYhEHHp7tEBvjxZ4/9nOuHCjGIfOq7HvzA1kF5bh+5Rr+D7lGlorrDGmV2uM9XWDm4Nw+RJCiNCogBO9KXRTwHEcurWWo1trOd4e2gEpV2/jh9RriDpzE9eLyrEm9jK+PHwZT3d0wgR/Dwxs34pG5YSQZocKODGJEfi9cBwHX08H+Ho6IPz5rohOy8XO5Bwcv5yPmPQ8xKTnwaOlDV5ReWCsrxvkNrTPnBDSPFABJ/8o4KYxAr8XK0sLPN/DFc/3cMUft0qxLSEbu1NycLWgDEsPXMRnv11CsJ87pvT3QmuFafeFEEIeFS1ia0TmsIitolqLTosOAQDOLB5qdiPYsqoa7Dt9A5FxfyJdXQIAEIs4jOjhirBBbdFJaZo/d0JI80GL2EijqBt920vFkFmb338ONhIxxvu5Y1wfNxzNzMeGI38g7o8C/HjqOn48dR2DO7bCnIAO6OGmEDpVQggxKvP7jU2Mqm4BW+tG3gPe2DiOw6AOrTCoQyucvVaEDUev4OC5mziccQuHM25hSCcnvPVMhwbfJpYQQkyd6MEhpCkz5QVsD6t7GwXWvtwLsW8/hRd6tYGIA2LS8/DcmuOY9s1JXLihETpFQgh5ZFTAmzlzWcD2MDwdbfHJ2B74be4gjPZpDREHRKflImj1cczecQo5hWVCp0gIIQ+NCngzZ2p7wBtD21Z2+GxcT/z61iCM6OEKANh7+gaGfHIES/enoaisSuAMCSHEcFTAm7mmOIV+L+2c7LA62AdRb/ZHv3YtUaXVYdOxLAxa+Ts2Hb2Cyhqt0CkSQkiDUQFv5pryFPq9dGstx7dTVYgM7YNOSntoyqux9MBFBH52FLHpuQ9+A0IIMQFUwJux8qq/nwPu1gxG4P/EcRye6uiE/bMGYOWL3eFkL8WfBWWYEnkSUyKT8Wf+HaFTJISQ+6IC3oxdL6q9/m2ue8CNwULE4SVfN8S+8xSmD2oLSwsOsel5GPrZUaw4lI47lTVCp0gIIfWiAt6M5fw1fW7ue8CNwU4qxoLhnXFozkAM7NAKVVodvvr9Dwz97ChiLtK0OiHE9FABb8aa0wK2hnqilR22hvbBxgm90aaFNa4XlWPq1pOYuS0VecUVQqdHCCE8KuDNWHPYQvYwOI7D0K5K/PrWQEwf2BYWIg77z93EkE+P4NuEq9Dp6PEBhBDhUQFvxprjCnRD2EjEWPBsZ+x7ox96tJGjpKIGC/ecx9gN8bhyq1To9AghzRwV8GaMptAbpqurHD++3g//eb4LbCUWOHn1NoZ/cQxfH7sCLY3GCSECoQLejF2nKfQGsxBxmNzPC7/OHYQB7R1RWaPDR/svYhyNxgkhAqEC3kzV7gGvvYVoc9sD/ihaK6zxzRQ/fDza+67ROF0bJ4Q8TlTAmyl+D7iVGHIbS4GzMS8cx+FllTt+eWsg+rf7ezQe8nUibhSVC50eIaSZoALeTOXQ9e9H1qaFDf431Q9LR3eDtaUF4q8UYNjnRxF19obQqRFCmgEq4M0UrUA3Do7jEKLywIHZA9CjjRzFFTV447tTmLvzNEoqqoVOjxDShFEBb6ZoD7hxeTna4vsZfTHr6XYQccCPp65j+BfHkHL1ttCpEUKaKCrgzRRtITM+SwsR5g7tiF3T/eHmYI1rt8sxbkM8Nhz5gxa4EUKMjgp4M0VT6I3H19MBB2YNwIgerqjRMUQcTMer35zE7TtVQqdGCGlCqIA3U7QHvHHZW1nii/E9ETHGGxKxCLHpeXh29TGkXC0UOjVCSBNBBbwZ+ucecJpCbzwcxyHYzx17Xu+Hto62uKmpwNgNCVhPU+qEECOgAt4M6e0Bt6Y94I2ti6sM+97sj5E9XaHVMSw7mI4pW5NRSFPqhJBHQAW8GaI94I+fnVSMz8f1xLIx3pCKRfg94xaepVXqhJBHIHgBX7t2LTw9PWFlZQWVSoWkpKT7xu/evRudOnWClZUVvL29ceDAAb12xhgWL14MFxcXWFtbIyAgAJmZmXoxhYWFCAkJgUwmg0KhwNSpU1Faqn8/619++QVPPvkk7O3t0apVK7zwwgv4888/jdJnoV0rpOvfQuA4DuP93LFnZu2Uurq4AuM3xuO7xGyhUyOEmCFBC/jOnTsxd+5chIeHIzU1FT169EBgYCDy8vLqjY+Li0NwcDCmTp2KU6dOYdSoURg1ahTOnz/Px6xYsQKrV6/G+vXrkZiYCFtbWwQGBqKiooKPCQkJwYULFxAdHY2oqCgcPXoUYWFhfHtWVhZGjhyJp59+GqdPn8Yvv/yC/Px8jBkzpvF+GI8RrUAXVmeX2in1Z72VqNYyvP/TOSz48Swqa7RCp0YIMSdMQH5+fmzmzJn891qtlrm6urKIiIh648eOHcuCgoL0jqlUKjZ9+nTGGGM6nY4plUq2cuVKvr2oqIhJpVK2fft2xhhjaWlpDABLTk7mYw4ePMg4jmPXr19njDG2e/duJhaLmVar5WP27dvHOI5jVVVVDe6fRqNhAJhGo2nwOY/D69+mMI95UezrY1eETqVZ0+l0bO3hTOY5P4p5zItiI788zm4WlQudFiHEyBqrFgg2Aq+qqkJKSgoCAgL4YyKRCAEBAYiPj6/3nPj4eL14AAgMDOTjs7KyoFar9WLkcjlUKhUfEx8fD4VCAV9fXz4mICAAIpEIiYmJAIDevXtDJBJhy5Yt0Gq10Gg0+N///oeAgABYWpr/oi+6C5tp4DgOrz/VDlsm94Hc2hKnc4rw3JrjSP6TtpoRQh5MsAKen58PrVYLZ2dnvePOzs5Qq9X1nqNWq+8bX/f1QTFOTk567WKxGA4ODnyMl5cXfv31V7z//vuQSqVQKBS4du0adu3add8+VVZWori4WO9limgK3bQ81dEJ+97oh05Ke+SXViJ4YwK+if8TjNFWM0LIvQm+iM0UqdVqTJs2DZMmTUJycjKOHDkCiUSCF1988b6/VCMiIiCXy/mXm5vbY8y6YcqqalBwh/aAmxqPlrb48fW+eK67C2p0DIv3XsC7359FRTVdFyeE1E+wAu7o6AgLCwvk5ubqHc/NzYVSqaz3HKVSed/4uq8Pivn3IrmamhoUFhbyMWvXroVcLseKFSvg4+ODgQMH4ttvv0VMTAw/zV6fBQsWQKPR8K+cnJwH/Rgeu+t/jb5pD7jpsZGIsSbYB+8/2wkiDvg+5Rpe3pSA/NJKoVMjhJggwQq4RCJB7969ERMTwx/T6XSIiYmBv79/vef4+/vrxQNAdHQ0H+/l5QWlUqkXU1xcjMTERD7G398fRUVFSElJ4WNiY2Oh0+mgUqkAAGVlZRCJ9H80FhYWfI73IpVKIZPJ9F6mhh5iYto4jkPYwCfwzRQVZFZipGYXYeSXJ5CuNs3LMYQQARl1SZyBduzYwaRSKYuMjGRpaWksLCyMKRQKplarGWOMTZgwgc2fP5+PP3HiBBOLxWzVqlXs4sWLLDw8nFlaWrJz587xMcuWLWMKhYLt3buXnT17lo0cOZJ5eXmx8vK/V/cOGzaM+fj4sMTERHb8+HHWvn17FhwczLfHxMQwjuPYkiVL2KVLl1hKSgoLDAxkHh4erKysrMH9M8VV6N/EZTGPeVHs1a3JDw4mgrqcV8IGrYhlHvOiWJdFB1nMRbXQKRFCHkJj1QJBCzhjjK1Zs4a5u7sziUTC/Pz8WEJCAt82aNAgNmnSJL34Xbt2sQ4dOjCJRMK6du3K9u/fr9eu0+nYokWLmLOzM5NKpWzIkCEsIyNDL6agoIAFBwczOzs7JpPJWGhoKCspKdGL2b59O/Px8WG2trasVatWbMSIEezixYsG9c0UC/jH+9OYx7wo9p9954VOhTRAYWklG7chjnnMi2Je82u3/ul0OqHTIoQYoLFqAccYLXVtLMXFxZDL5dBoNCYznT5zWyr2n7uJRc91wdT+XkKnQxqgqkaHxXvPY0dy7ZqKYD93fDiyKywtaA0qIeagsWoB/QZoZmgPuPmRiEWIGOONhUGdwXHA9qRsTPq/JGjKqoVOjRAiICrgzUzdIjY3WsRmVjiOw6sD2mLTBF/YSiwQ90cBRn91AlcL7gidGiFEIFTAm5F/7gFvTSNwsxTQxRnfz+iL1gprXMm/gzFfxeF0TpHQaRFCBEAFvBmp2wMuoz3gZq2ziww/zeyLbq1lKLhThfEb4xGdlvvgEwkhTQoV8GaE9oA3HU72VtgZ5o+nOrZCRbUO0/93Ev+L/1PotAghjxEV8GaEFrA1LbZSMb6e6IvxfdygY8CivRew7GA6dDraWEJIc0AFvBmhEXjTI7aoXaH+9jMdAADrj/yBOTtP07PFCWkGqIA3I/QUsqaJ4zi8OaQ9PnmpB8QiDvvO3MDEzbTNjJCmjgp4M5JDU+hN2gu922BLaB/YScVIzCrEi+vjcKOoXOi0CCGNhAp4M0JT6E3fgPatsPs1fyhlVsjMK8WL6+JwOa9U6LQIIY2ACngzcaeyBoW0B7xZ6Owiww+v90XbVra4oanAS+vjcCr7ttBpEUKM7KEKeE1NDX777Tds2LABJSUlAIAbN26gtJT+0jdV14toD3hz0lphje9f64sebeS4XVaNlzcl4silW0KnRQgxIoML+NWrV+Ht7Y2RI0di5syZuHWr9pfC8uXL8c477xg9QWIcf28ho+nz5sLBVoLvpj2JAe0dUV6txdTIZOw9fV3otAghRmJwAZ89ezZ8fX1x+/ZtWFv/PRU7evRoxMTEGDU5Yjy0Ar15spWKsXlSHzzfwxU1OobZO05jy4ksodMihBiB2NATjh07hri4OEgkEr3jnp6euH6d/ro3VbSArfmSiEX4YlxPONhYYmv8VSz5OQ2Fd6ow95kO4DhO6PQIIQ/J4BG4TqeDVnv3TSKuXbsGe3t7oyRFjI/uwta8iUQc/jOiK3/DlzWxl/H+T+ehpbu2EWK2DC7gQ4cOxeeff85/z3EcSktLER4ejmeffdaYuREjoil0UnfDl49He0P013PF5+w8jWqtTujUCCEPweAp9E8++QSBgYHo0qULKioq8PLLLyMzMxOOjo7Yvn17Y+RIjICm0Emdl1XukFtbYvaOU/j5zA2UV9Xgy5d7wcrSQujUCCEGMLiAt2nTBmfOnMHOnTtx5swZlJaWYurUqQgJCdFb1EZMB+0BJ/8W1N0FNhILvPZtCn67mIcpkcnYNNEXtlKDfyUQQgTCMcYMugh29OhR9O3bF2Kx/j/0mpoaxMXFYeDAgUZN0JwVFxdDLpdDo9FAJpMJlsel3BIM/ewoZFZinP1PoGB5ENMT/0cBXt2ajDtVWvi4KxA52Q9yG7pPACHG1Fi1wOBr4IMHD0ZhYeFdxzUaDQYPHmyUpIhx0R5wci/+T7TEtmlPQm5tiVPZRRi/KQH5pZVCp0UIaQCDCzhjrN6tJwUFBbC1tTVKUsS4aAEbuZ+ebgrsnP4kHO2kuHizGGM3xNNDUAgxAw2+4DVmzBgAtStZJ0+eDKlUyrdptVqcPXsWffv2NX6G5JHVFXA3BxqBk/p1Usqwa/qTeOXrRFy5dQcvrY/Hd9NU8GhJf5QTYqoaPAKXy+WQy+VgjMHe3p7/Xi6XQ6lUIiwsDN9++21j5koeEu0BJw3RtpUdds/oC8+WNrheVI6X1sfjUm6J0GkRQu6hwSPwLVu2AKi949o777xD0+VmhLaQkYZqrbDGrtf8MXFzEtLVJRi7IR7fTPFD9zYKoVMjhPyLwdfAw8PDqXibGboGTgzhZG+FHWFPooebAkV/Pcks+c+7F64SQoT1UJs+v//+e+zatQvZ2dmoqqrSa0tNTTVKYsQ4aA84eRgKGwm2varC1MhkJGYVYuLmJGye7Iu+TzgKnRoh5C8Gj8BXr16N0NBQODs749SpU/Dz80PLli1x5coVDB8+vDFyJI+gbvQtt7aEzIr295KGs5OKsXWKH/840tAtyThKzxQnxGQYXMC/+uorbNy4EWvWrIFEIsF7772H6OhozJo1CxqNpjFyJI+AFrCRR2FlaYFNE33xdCcnVNbo8OrWk4hNzxU6LUIIHqKAZ2dn89vFrK2tUVJSu0p1woQJdC90E0TXv8mjsrK0wPpXemNoF2dUaXWY/r8U/HJBLXRahDR7BhdwpVLJ34nN3d0dCQkJAICsrCwYeFdW8hjQXdiIMUjEIqwN6YWg7i6o1jLM3JaK/WdvCp0WIc2awQX86aefxr59+wAAoaGheOutt/DMM89g3LhxGD16tNETJI+GRuDEWCwtRPhiXE+M6umKGh3Dm9tTsff0daHTIqTZMriAb9y4ER988AEAYObMmfi///s/dO7cGR9++CHWrVtncAJr166Fp6cnrKysoFKpkJSUdN/43bt3o1OnTrCysoK3tzcOHDig184Yw+LFi+Hi4gJra2sEBAQgMzNTL6awsBAhISGQyWRQKBSYOnUqSktL73qfVatWoUOHDpBKpWjdujWWLl1qcP+ERnvAiTGJLUT4ZGxPvNi7DXQMmLPzNL5PuSZ0WoQ0T8wA1dXVbMmSJSwnJ8eQ0+5px44dTCKRsP/7v/9jFy5cYNOmTWMKhYLl5ubWG3/ixAlmYWHBVqxYwdLS0tjChQuZpaUlO3fuHB+zbNkyJpfL2Z49e9iZM2fYiBEjmJeXFysvL+djhg0bxnr06MESEhLYsWPHWLt27VhwcLDeZ7355pusY8eObO/evezKlSvs5MmT7NdffzWofxqNhgFgGo3GoPOMqeeSX5jHvCiWdkO4HEjTo9Xq2IIfzzKPeVHMc34U+y7xqtApEWKyGqsWGFTAGWPM1taWZWVlGeXD/fz82MyZM/nvtVotc3V1ZREREfXGjx07lgUFBekdU6lUbPr06YwxxnQ6HVMqlWzlypV8e1FREZNKpWz79u2MMcbS0tIYAJacnMzHHDx4kHEcx65fv87HiMVilp6e/kj9E7qAl1RUM495UcxjXhTTlFcJkgNpunQ6HQvfe57/b2xrXJbQKRFikhqrFhg8hT5kyBAcOXLkkUf+VVVVSElJQUBAAH9MJBIhICAA8fHx9Z4THx+vFw8AgYGBfHxWVhbUarVejFwuh0ql4mPi4+OhUCjg6+vLxwQEBEAkEiExMREA8PPPP6Nt27aIioqCl5cXPD098eqrr9b7GNV/qqysRHFxsd5LSNdpDzhpRBzHIfz5Lpg2wAsAsHjvBXx97IrAWRHSfBh8J7bhw4dj/vz5OHfuHHr37n3XbVVHjBjRoPfJz8+HVquFs7Oz3nFnZ2ekp6fXe45ara43Xq1W8+11x+4X4+TkpNcuFovh4ODAx1y5cgVXr17F7t278c0330Cr1eKtt97Ciy++iNjY2Hv2KSIiAkuWLHlQ1x8b2gNOGhvHcXj/2c61q9QP/4GP9l+EjjGEDXxC6NQIafIMLuCvv/46AODTTz+9q43jOGi12kfPSmA6nQ6VlZX45ptv0KFDBwDA5s2b0bt3b2RkZKBjx471nrdgwQLMnTuX/764uBhubm6PJef60Ap08jhwHId3hnaEhUiE1TGZ+PhA7R/gVMQJaVwGT6HrdLp7vgwp3o6OjrCwsEBurv5dnXJzc6FUKus9R6lU3je+7uuDYvLy8vTaa2pqUFhYyMe4uLhALBbzxRsAOnfuDKD2Rjb3IpVKIZPJ9F5Coj3g5HHhOA5zn+mA2UPaAwA+PpCODUf+EDgrQpo2gwu4sUgkEvTu3RsxMTH8MZ1Oh5iYGPj7+9d7jr+/v148AERHR/PxXl5eUCqVejHFxcVITEzkY/z9/VFUVISUlBQ+JjY2FjqdDiqVCgDQr18/1NTU4I8//v4FdOnSJQCAh4fHo3T7saIROHnc3nqmA+YE1BbxiIPpWE9FnJDGY9QlcQbasWMHk0qlLDIykqWlpbGwsDCmUCiYWq1mjDE2YcIENn/+fD7+xIkTTCwWs1WrVrGLFy+y8PDwereRKRQKtnfvXnb27Fk2cuTIereR+fj4sMTERHb8+HHWvn17vW1kWq2W9erViw0cOJClpqaykydPMpVKxZ555hmD+if0KvTnVh9jHvOi2K8X1IJ8Pmm+Po++xK9O/+rwZaHTIURQJrONzNjWrFnD3N3dmUQiYX5+fiwhIYFvGzRoEJs0aZJe/K5du1iHDh2YRCJhXbt2Zfv379dr1+l0bNGiRczZ2ZlJpVI2ZMgQlpGRoRdTUFDAgoODmZ2dHZPJZCw0NJSVlJToxVy/fp2NGTOG2dnZMWdnZzZ58mRWUFBgUN+ELuC0B5wI6Yvf/i7iaw9nCp0OIYJprFrAMUY3MG8sxcXFkMvl0Gg0j/16eGllDbqF/wIAOPefobCnbWREAGtiMvFJdO3lp3cDO2Lm4HYCZ0TI49dYtUCwa+CkcdXtAVfYWFLxJoJ5c0h7vP1M7WLQlb9kYO3hywJnREjTYfA2snvdnITjOEilUkgkkkdOijw62gNOTMWbQ9qD44BVv17Cyl8ywBjDG0+3FzotQsyewQVcoVCA47h7trdp0waTJ09GeHg4RCIa4AuFX4GuoC1kRHhvPN0eHMdh5S8ZWPXrJTBWW9gJIQ/P4AIeGRmJDz74AJMnT4afnx8AICkpCVu3bsXChQtx69YtrFq1ClKpFO+//77REyYNQyNwYmrqrn+v/CWDvy5ORZyQh2dwAd+6dSs++eQTjB07lj/2/PPPw9vbGxs2bEBMTAzc3d2xdOlSKuACyimkPeDE9Mwc3A4cB6w4VFvEGYBZVMQJeSgGz3HHxcXBx8fnruM+Pj78A0P69+9/3zuWkcZ3rYjuwkZM0+tPtcO8YZ0AAJ9GX8IXv2UKnBEh5sngAu7m5obNmzffdXzz5s38fb8LCgrQokWLR8+OPDT+GrgDjcCJ6Znx1BOYP7y2iH/22yV8/tslgTMixPwYPIW+atUqvPTSSzh48CD69OkDADh58iTS09Px/fffAwCSk5Mxbtw442ZKGqykohpFZdUAgNYKKuDENL026AlwqL3l6ue/ZYKx2luxEkIaxuACPmLECKSnp2PDhg38/cGHDx+OPXv2wNPTEwAwY8YMoyZJDHO9iPaAE/MwfdAT4Ljah598EVM7lU5FnJCGMbiAA7UPDVm2bJmxcyFGco0WsBEzUvfYUSrihBjmoQp4UVERkpKSkJeXB51Op9c2ceJEoyRGHh6/hYz2gBMzETbwCXDgsPTARSrihDSQwQX8559/RkhICEpLSyGTyfRu6sJxHBVwE0CPESXmaNrAtgDAF3EG4K2A9ve9cRQhzZnBq9DffvttTJkyBaWlpSgqKsLt27f5V2FhYWPkSAxEBZyYq2kD2+KDZzsDAFbHZOKz3zJBz1sipH4GF/Dr169j1qxZsLGh6VlTRXvAiTmbNrAtFgZRESfkQQwu4IGBgTh58mRj5EKMhPaAE3P36gAq4oQ8iMHXwIOCgvDuu+8iLS0N3t7esLTU36Y0YsQIoyVHDEd7wElT8eqA2mviH+2/iNUxmQBjeOuZDnRNnJC/GFzAp02bBgD48MMP72rjOA5arfbRsyIPjfaAk6ZEr4jH1j5LnIo4IbUMLuD/3jZGTAvtASdNDRVxQupHD+xuYmgPOGmK9K6Jx17Gp9GX6Jo4afYaNAJfvXo1wsLCYGVlhdWrV983dtasWUZJjDwc2kJGmqp/jsTX/DUSn0sjcdKMNaiAf/bZZwgJCYGVlRU+++yze8ZxHEcFXGBUwElTRkWckL81qIBnZWXV+7+J6anbA+7mQFPopGl6dUBbcByH/0alUREnzRpdA29i/h6BUwEnTdfU/l5Y9FwXAMCa2Mv45Fe6Jk6aH4NXoWu1WkRGRiImJqbeh5nExsYaLTliGL094DSFTpq4qf29AAD/jUrDl4drR+JvD6WROGk+DC7gs2fPRmRkJIKCgtCtWzf6x2JC6kbfLWwsYSd9qAfNEWJWpvb3AgfgQyripBky+Lf8jh07sGvXLjz77LONkQ95BDR9TpqjKX+NxOuKOAPDO0M7UhEnTZ7B18AlEgnatWvXGLmQR8TvAafpc9LMTOnvhcV/XRNfe/gPrPo1g66JkybvoR4n+sUXX9A/DhNEW8hIczalvxfCn6ciTpoPg6fQjx8/jsOHD+PgwYPo2rXrXQ8z+fHHH42WHDHM3yNwmkInzVNov9rp9CU/p2Ht4T/AGPBuIE2nk6bJ4AKuUCgwevToxsiFPCIagROiX8S/+v0PAFTESdNkUAGvqanB4MGDMXToUCiVysbKiTwkWsRGSC0q4qQ5MOgauFgsxmuvvYbKykqjJrF27Vp4enrCysoKKpUKSUlJ943fvXs3OnXqBCsrK3h7e+PAgQN67YwxLF68GC4uLrC2tkZAQAAyMzP1YgoLCxESEgKZTAaFQoGpU6eitLS03s+7fPky7O3toVAoHqmfjam4ohqactoDTkid0H5e+M9f18S/+v0PrPyFromTpsXgRWx+fn44deqU0RLYuXMn5s6di/DwcKSmpqJHjx4IDAxEXl5evfFxcXEIDg7G1KlTcerUKYwaNQqjRo3C+fPn+ZgVK1Zg9erVWL9+PRITE2Fra4vAwEBUVFTwMSEhIbhw4QKio6MRFRWFo0ePIiws7K7Pq66uRnBwMAYMGGC0PjeG67QHnJC7TP5XEV92KJ2KOGk6mIF27tzJ2rZty9asWcPi4uLYmTNn9F6G8vPzYzNnzuS/12q1zNXVlUVERNQbP3bsWBYUFKR3TKVSsenTpzPGGNPpdEypVLKVK1fy7UVFRUwqlbLt27czxhhLS0tjAFhycjIfc/DgQcZxHLt+/bree7/33nvslVdeYVu2bGFyudygvmk0GgaAaTQag857GL9eUDOPeVHsudXHGv2zCDE3kSeymMe8KOYxL4r9Z995ptPphE6JNCONVQsMHoGPHz8eWVlZmDVrFvr164eePXvCx8eH/2qIqqoqpKSkICAggD8mEokQEBCA+Pj4es+Jj4/XiweAwMBAPj4rKwtqtVovRi6XQ6VS8THx8fFQKBTw9fXlYwICAiASiZCYmMgfi42Nxe7du7F27doG9aeyshLFxcV6r8eF9oATcm+T+nrio1HdAABbTvyJRXvPQ6ejkTgxbwbPtRrzaWT5+fnQarVwdnbWO+7s7Iz09PR6z1Gr1fXGq9Vqvr3u2P1inJyc9NrFYjEcHBz4mIKCAkyePBnffvstZDJZg/oTERGBJUuWNCjW2GgFOiH398qTHpBYiDDvx7P4NiEbNVqGj0d7QySihW3EPBlcwD08PBojD5Mzbdo0vPzyyxg4cGCDz1mwYAHmzp3Lf19cXAw3N7fGSO8utAeckAcb28cNlmIOb+86gx3JOajS6rDyxR6woCJOzNBDr3ZKS0tDdnY2qqqq9I6PGDGiwe/h6OgICwsL5Obm6h3Pzc295zY1pVJ53/i6r7m5uXBxcdGL6dmzJx/z70VyNTU1KCws5M+PjY3Fvn37sGrVKgC1K9t1Oh3EYjE2btyIKVOm3JWbVCqFVCptaPeNikbghDTMaJ82EItEmLPzNH5MvY5qLcNnY3tAbEFPVybmxeACfuXKFYwePRrnzp0Dx3H8is66/ZVarbbB7yWRSNC7d2/ExMRg1KhRAACdToeYmBi88cYb9Z7j7++PmJgYzJkzhz8WHR0Nf39/AICXlxeUSiViYmL4gl1cXIzExETMmDGDf4+ioiKkpKSgd+/eAGoLtk6ng0qlAlB7nfyffdm7dy+WL1+OuLg4tG7dusF9fFxoDzghDfd8D1dYWnB447tT+PnMDdRodfhivA8kYirixIwYuurtueeeYyNHjmS3bt1idnZ2LC0tjR07doz5+fmxo0ePGryKbseOHUwqlbLIyEiWlpbGwsLCmEKhYGq1mjHG2IQJE9j8+fP5+BMnTjCxWMxWrVrFLl68yMLDw5mlpSU7d+4cH7Ns2TKmUCjY3r172dmzZ9nIkSOZl5cXKy8v52OGDRvGfHx8WGJiIjt+/Dhr3749Cw4OvmeeprwKXVNexa+wLamobtTPIqQpib6gZu3fP8A85kWxqZFJrKK6RuiUSBPUWLXA4ALesmVLfruYTCZj6enpjDHGYmJiWM+ePR8qiTVr1jB3d3cmkUiYn58fS0hI4NsGDRrEJk2apBe/a9cu1qFDByaRSFjXrl3Z/v379dp1Oh1btGgRc3Z2ZlKplA0ZMoRlZGToxRQUFLDg4GBmZ2fHZDIZCw0NZSUlJffM0ZQLeNoNDfOYF8V6LvmlUT+HkKbocHou6/BBbRGfuDmRlVdRESfG1Vi1gGPMsLsatGjRAqmpqfDy8sITTzyBr7/+GoMHD8Yff/wBb29vlJWVNcZEgVkqLi6GXC6HRqNp8Er2hxGdlotp35yEd2s5fn6zf6N9DiFN1YnL+Zi6NRkV1Tr0b+eITRN9YS2xEDot0kQ0Vi0w+IJPt27dcObMGQCASqXCihUrcOLECXz44Ydo27at0RIjDZdTSHvACXkU/do5IjLUDzYSCxy/nI9JW5JQWlkjdFqE3JfBBXzhwoXQ6XQAgA8//BBZWVkYMGAADhw4gNWrVxs9QfJgdQvY3BxoARshD+vJti3xv6l+sJeKkZRViJCvE1FUVvXgEwkRiMEFPDAwEGPGjAEAtGvXDunp6cjPz0deXh6efvppoydIHozuwkaIcfT2cMC2aSq0sLHEmZwijNuQgLySigefSIgAHnrPxOXLl/HLL7+gvLwcDg4OxsyJGIj2gBNiPN3bKLBzuj9a2UuRkVuCcRsScL2oXOi0CLmLwQW8oKAAQ4YMQYcOHfDss8/i5s2bAICpU6fi7bffNnqC5MHoLmyEGFcHZ3vsnu6P1gprZOXfwUvr4nDlVv2PGyZEKAYX8LfeeguWlpbIzs6Gjc3fBWPcuHE4dOiQUZMjD6Ypr0ZxRe1im9YKGoETYiyejrb4foY/2rayxQ1NBcZuSMDFm4/vAUWEPIjBBfzXX3/F8uXL0aZNG73j7du3x9WrV42WGGmYuueAO9hKYEvPASfEqFzk1tg13R+dXWTIL63E+I0JOJV9W+i0CAHwEAX8zp07eiPvOoWFhYLdB7w5owVshDQuRzspdkx7Er3cFdCUV+OVrxMR/0eB0GkRYngBHzBgAL755hv+e47joNPpsGLFCgwePNioyZEHowVshDQ+uY0l/jdVhX7tWuJOlRaTtyThcHreg08kpBEZXMBXrFiBjRs3Yvjw4aiqqsJ7772Hbt264ejRo1i+fHlj5Ejugx5iQsjjYSsVY/OkPgjo7ITKGh2mfXMSUWdvCJ0WacYe6k5sly5dQv/+/TFy5EjcuXMHY8aMwalTp/DEE080Ro7kPmgKnZDHx8rSAute6Y0RPVxRo2OYtf0UdiRlC50WaaYeatWTXC7HBx98oHfs2rVrCAsLw8aNG42SGGkYmkIn5PGytBDhs3E9YSu1wPakHMz/8RwK7lTh9aee4B+rTMjjYLSH3xYUFGDz5s3GejvSQLQHnJDHz0LE4ePR3nj9qdpZx5W/ZOC/UReh0xn0bChCHgk9vd6M0R5wQoTDcRzeG9YJC4M6AwD+70QW5u46jWqtTuDMSHNBBdyM0R5wQoT36oC2+GxcD4hFHPacvoFp35xEWRU9yYw0PirgZowWsBFiGkb7tMGmSb6wshTh94xbCPk6Ebfv0JPMSONq8LCt7glk91JUVPSouRAD0QI2QkzH4I5O2Pbqk5gSmYxT2UV4aUM8vpniB1e6vEUaSYNH4HK5/L4vDw8PTJw4sTFzJf9Ce8AJMS29PVpg92v+UMqscDmvFC+ui8PlvBKh0yJNVINH4Fu2bGnMPMhDoCl0QkxPB2d7/PB6X0zYnIgrt+5gzFdx2DTRF6q2LYVOjTQxdA3cjOXQFDohJqm1whrfv9YXvdwVKK6owYTNSfj5DN21jRgXFXAzRnvACTFdDrYSfDftSQzrqkSVVoc3t5/CxqN/gDHaK06Mgwq4mdKUV6OE9oATYtKsLC2wNqQXQvt5AgA+PpCO/+y7AC3d8IUYARVwM1U3+m5Je8AJMWkWIg7hz3flb/iyNf4qXvs2BeVVWoEzI+aOCriZoi1khJiXVwe0xdqXe0EiFiE6LRfBmxJQUFopdFrEjFEBN1O0hYwQ8xPU3QXbXlVBbm2J0zlFeGFdHK7cKhU6LWKmqICbKdpCRoh56uPpgB9m9EWbFtb4s6AMo7+KQ9zlfKHTImaICriZoil0QsxXOyc7/PR6P/RyV0BTXo2J/5eE7xLpueLEMFTAzRRNoRNi3lrZS/HdtCcxsqcranQM7/90Dh/+nEYr1EmDUQE3UzSFToj5s7K0wOfjeuLtZzoAqH0k6bRvTqKkolrgzIg5oAJuhvT2gFMBJ8SscRyHN4e0x9qXe8HKUoTY9Dy8uC4eOYVlQqdGTBwVcDP0zz3gNhLaA05IUxDU3QW7pvvDyV6KjNwSjFp7AilXC4VOi5gwKuBmiBawEdI0dW+jwN43+qGrqwwFd6oQvDERO5NpcRupn0kU8LVr18LT0xNWVlZQqVRISkq6b/zu3bvRqVMnWFlZwdvbGwcOHNBrZ4xh8eLFcHFxgbW1NQICApCZmakXU1hYiJCQEMhkMigUCkydOhWlpX/vx/z9998xcuRIuLi4wNbWFj179sS2bduM1+lHQAvYCGm6XOTW2P2aP38P9Xk/nMMHP51DVY1O6NSIiRG8gO/cuRNz585FeHg4UlNT0aNHDwQGBiIvL6/e+Li4OAQHB2Pq1Kk4deoURo0ahVGjRuH8+fN8zIoVK7B69WqsX78eiYmJsLW1RWBgICoqKviYkJAQXLhwAdHR0YiKisLRo0cRFham9zndu3fHDz/8gLNnzyI0NBQTJ05EVFRU4/0wGogWsBHStNlIxPgqpBfeGdoBHAdsS8xG8KYE5BVXPPhk0nwwgfn5+bGZM2fy32u1Wubq6soiIiLqjR87diwLCgrSO6ZSqdj06dMZY4zpdDqmVCrZypUr+faioiImlUrZ9u3bGWOMpaWlMQAsOTmZjzl48CDjOI5dv379nrk+++yzLDQ0tMF902g0DADTaDQNPqchXt2azDzmRbFv4rKM+r6EENMTezGXdQs/xDzmRbE+H0Wzk38WCp0SMVBj1QJBR+BVVVVISUlBQEAAf0wkEiEgIADx8fH1nhMfH68XDwCBgYF8fFZWFtRqtV6MXC6HSqXiY+Lj46FQKODr68vHBAQEQCQSITEx8Z75ajQaODg43LO9srISxcXFeq/GQFPohDQfgzs5Yd8b/dHeyQ55JZUYvzEe2xKvCp0WMQGCFvD8/HxotVo4OzvrHXd2doZara73HLVafd/4uq8PinFyctJrF4vFcHBwuOfn7tq1C8nJyQgNDb1nfyIiIiCXy/mXm5vbPWMfBU2hE9K8eDnaYs/MfnjWW4lqLcMHP53Hgh/PorKGnmjWnAl+DdwcHD58GKGhodi0aRO6du16z7gFCxZAo9Hwr5ycHKPnQnvACWmebKVirH25F+YN6wSOA7Yn5WDshgT+D3rS/AhawB0dHWFhYYHc3Fy947m5uVAqlfWeo1Qq7xtf9/VBMf9eJFdTU4PCwsK7PvfIkSN4/vnn8dlnn2HixIn37Y9UKoVMJtN7GVvdzR1oDzghzQ/HcZjx1BOIDPWD3NoSZ3KKELT6OKLTch98MmlyBC3gEokEvXv3RkxMDH9Mp9MhJiYG/v7+9Z7j7++vFw8A0dHRfLyXlxeUSqVeTHFxMRITE/kYf39/FBUVISUlhY+JjY2FTqeDSqXij/3+++8ICgrC8uXL9VaoC4n2gBNCBnVohag3+6OHW+3DUKZ9cxIfRaXRVrPmxqhL4h7Cjh07mFQqZZGRkSwtLY2FhYUxhULB1Go1Y4yxCRMmsPnz5/PxJ06cYGKxmK1atYpdvHiRhYeHM0tLS3bu3Dk+ZtmyZUyhULC9e/eys2fPspEjRzIvLy9WXl7OxwwbNoz5+PiwxMREdvz4cda+fXsWHBzMt8fGxjIbGxu2YMECdvPmTf5VUFDQ4L41xsrDTUf/YB7zotjr36YY7T0JIeapslrLPvz5AvOYF8U85kWxEV8eZ9kFd4ROi/xLY61CF7yAM8bYmjVrmLu7O5NIJMzPz48lJCTwbYMGDWKTJk3Si9+1axfr0KEDk0gkrGvXrmz//v167Tqdji1atIg5OzszqVTKhgwZwjIyMvRiCgoKWHBwMLOzs2MymYyFhoaykpISvn3SpEkMwF2vQYMGNbhfjfF/Wvje88xjXhT7eH+a0d6TEGLefjl/k3n/tdXMO/wQ++X8TaFTIv/QWAWcY4zRs+saSXFxMeRyOTQajdGuh7+69SR+u5iL/47sign+nkZ5T0KI+cspLMOb20/hdE4RAGBKPy/MH94JEjGtVRZaY9QCgFahmx1+C5kD7QEnhPzNzcEGu6b7Y9oALwC1jyZ9cX0crtwqfcCZxFxRATcjjDFc/2sRmxstYiOE/ItELMIHQV3w9URfyK0tcfaaBkGrj+O7xGzQZGvTQwXcjBSX16Ck8q894AoagRNC6hfQxRmH5gxAv3YtUV6txfs/ncO0b1JQUFopdGrEiKiAm5Gcv6bPHe0ksJZYCJwNIcSUucit8b8pKiwM6gyJhQi/XcxF4OfHcDi9/gdFEfNDBdyM1O0Bb033QCeENIBIxOHVAW2xZ2Y/dHC2Q35pJUIjkzH/h7MoqagWOj3yiKiAmxG6Bzoh5GF0cZVh3xv9MaWfFzgO2JGcg8DPjuJY5i2hUyOPgAq4GaG7sBFCHpaVpQUWP98FO6Y9CXcHG9zQVGDC5iQs+PEcjcbNFBVwM0KPESWEPCpV25Y4NGcAJvf1BABsT8rGsM+P4fcMujZubqiAmxGaQieEGIONRIz/jOiK7dOehJuDNa4XlWPylmS8uf0UbpXQSnVzQQXcTNAecEKIsfk/0RKHZg/Eq/29IOKAn8/cwJBPfseOpGzodLRv3NRRATcTtAecENIYbKViLHyuC/bO7I9urWUorqjB/B/PYfzGBGTmlgidHrkPKuBmgvaAE0Iak3cbOfa83g8LgzrD2tICSX8WYvgXx/BRVBotcjNRVMDNBO0BJ4Q0NrGFCK8OaIvouQPxTBdn1OgYvj6ehcGrjuD7lGs0rW5iqICbCVrARgh5XNq0sMGmib6IDO2Dto62yC+txDu7z+CF9XE4e61I6PTIX6iAmwnaA04Iedye6uiEQ3MGYsHwTrCVWOBUdhFGfHkCs3ecQk5hmdDpNXtUwM3E3yNwmkInhDw+ErEI0wc9gdh3nsIYn9YAgL2nb2DIJ0ewdH8aisqqBM6w+aICbiZoBE4IEZKzzAqfjuuJqDf7o1+7lqjS6rDpWBYGrfwdG4/+gfIqrdApNjtUwM0AY4wv4LQHnBAipG6t5fh2qgqRoX3QSWkPTXk1Pj6QjoErD+P/jmehopoK+eNCBdwMaMqrUUp7wAkhJoLjODzV0Qn7Zw3Ayhe7o00La9wqqcSHUWkYtPIwtsb9SYX8MaACbgbqRt+0B5wQYkosRBxe8nVD7NtP4ePR3mitsEZucSXC913A4FW/Y/PxLNz5a/BBjI8KuBmoW8BGe8AJIaZIIhbhZZU7Yt8ZhI9GdYOL3Ao3NRX4b1Qa+i6LxSe/ZiC/lO6xbmxUwM0ALWAjhJgDqdgCrzzpgd/frR2ReznaQlNejTWxl9FvWSw++Okc/rhVKnSaTYZY6ATIg/29gI1G4IQQ0ycVW+BllTvG9XFDdJoa645cwZmcImxLzMa2xGz0b+eICf4eGNLJCWILGkc+LCrgZoDuwkYIMUcWIg7DurkgsKsSSVmF2HTsCmLS83D8cj6OX86Hi9wKL/u5Y5yfG5zsrYRO1+xQATcDNIVOCDFnHMdB1bYlVG1bIqewDN8lZWNncg5uairwSfQlfB6TiYHtHfFC7zYI6OwMK0tarNsQVMBN3D/3gNNd2Agh5s7NwQbzhnXCnID2OHDuJv4XfxWp2UU4nHELhzNuwd5KjOe6u+KFXq3Ry70FRCJO6JRNFhVwE/fPPeA0AieENBVSsQVG+7TBaJ82+ONWKX5KvY4fU6/hhqYC25OysT0pG84yKQK7KjGsqxJ+Xg50vfxfOMYYPR+ukRQXF0Mul0Oj0UAmkz3Ue5y/rsFza47D0U6KkwsDjJwhIYSYDp2OIeFKAb5PvYboC7ko+cce8hY2lnimizMGd3RC3yccIbexFDBTwxijFtSHRuAmjhawEUKaC5GIQ992jujbzhGVNVrEXS7AofNq/Jqmxu2yauw6eQ27Tl6DiAN6uikwoH0rDOzgiO5tFLBshqNzKuAmjhawEUKaI6nYAoM7OWFwJycs1XZD0p+FiE7LxbHMfFzOK0VqdhFSs4vwRUwmrCxF6N5Ggd4eLdDbvQV6ebSAg61E6C40OirgJo4WsBFCmjuxhQh9n3BE3yccAQA3ispxPDMfRzNv4fjlfBSVVSMpqxBJWYX8OW4O1ujoLEMnpT06/vXycrRtUiN1k+jJ2rVr4enpCSsrK6hUKiQlJd03fvfu3ejUqROsrKzg7e2NAwcO6LUzxrB48WK4uLjA2toaAQEByMzM1IspLCxESEgIZDIZFAoFpk6ditJS/TsEnT17FgMGDICVlRXc3NywYsUK43TYADSFTggh+lwV1hjbxw1fvtwLqQufwW9zB2HFC90xztcN7Z3sAAA5heX47WIuvjx8GW9uP4Whnx1F50WHMGBFLII3JuC9789gdUwmfki5ht8z8nAmpwg5hWUoqaiGuSwNE3wEvnPnTsydOxfr16+HSqXC559/jsDAQGRkZMDJyemu+Li4OAQHByMiIgLPPfccvvvuO4waNQqpqano1q0bAGDFihVYvXo1tm7dCi8vLyxatAiBgYFIS0uDlVXtzQJCQkJw8+ZNREdHo7q6GqGhoQgLC8N3330HoHbRwdChQxEQEID169fj3LlzmDJlChQKBcLCwh7bz4em0Akh5N5EIg7tnOzQzskOY/u4AQA0ZdVIu1mMDHUxMnJLkK4uwSV1Ce5UaZFTWI6cwnLEX7n3e1pacLCVimEltoC1xAJhA9si2M/9MfWo4QRfha5SqdCnTx98+eWXAACdTgc3Nze8+eabmD9//l3x48aNw507dxAVFcUfe/LJJ9GzZ0+sX78ejDG4urri7bffxjvvvAMA0Gg0cHZ2RmRkJMaPH4+LFy+iS5cuSE5Ohq+vLwDg0KFDePbZZ3Ht2jW4urpi3bp1+OCDD6BWqyGR1F5LmT9/Pvbs2YP09PQG9e1RVx4yxuD9n19RWlmD3+YOQru//rIkhBBiGJ2OQV1cgetF5bh2uww5hbVfrxeVo/BONYrKqnC7rAoV1bq7zl0Y1BmvDmj70J/dJFehV1VVISUlBQsWLOCPiUQiBAQEID4+vt5z4uPjMXfuXL1jgYGB2LNnDwAgKysLarUaAQF/b7mSy+VQqVSIj4/H+PHjER8fD4VCwRdvAAgICIBIJEJiYiJGjx6N+Ph4DBw4kC/edZ+zfPly3L59Gy1atLgrt8rKSlRW/v3EneLiYsN+IPXYEfYkrt0uh5sDjcAJIeRhiUQcXBXWcFVYo4+nwz3jyqu0uF1WhbIqLSqqtSiv1prsDKigBTw/Px9arRbOzs56x52dne85ylWr1fXGq9Vqvr3u2P1i/j09LxaL4eDgoBfj5eV113vUtdVXwCMiIrBkyZJ7d9hAHMehW2s5urWWG+09CSGE3Ju1xALWEtMs2P9mEovYmooFCxZAo9Hwr5ycHKFTIoQQ0kQJWsAdHR1hYWGB3NxcveO5ublQKpX1nqNUKu8bX/f1QTF5eXl67TU1NSgsLNSLqe89/vkZ/yaVSiGTyfRehBBCSGMQtIBLJBL07t0bMTEx/DGdToeYmBj4+/vXe46/v79ePABER0fz8V5eXlAqlXoxxcXFSExM5GP8/f1RVFSElJQUPiY2NhY6nQ4qlYqPOXr0KKqrq/U+p2PHjvVOnxNCCCGPFRPYjh07mFQqZZGRkSwtLY2FhYUxhULB1Go1Y4yxCRMmsPnz5/PxJ06cYGKxmK1atYpdvHiRhYeHM0tLS3bu3Dk+ZtmyZUyhULC9e/eys2fPspEjRzIvLy9WXl7OxwwbNoz5+PiwxMREdvz4cda+fXsWHBzMtxcVFTFnZ2c2YcIEdv78ebZjxw5mY2PDNmzY0OC+aTQaBoBpNJpH+RERQggxY41VCwQv4IwxtmbNGubu7s4kEgnz8/NjCQkJfNugQYPYpEmT9OJ37drFOnTowCQSCevatSvbv3+/XrtOp2OLFi1izs7OTCqVsiFDhrCMjAy9mIKCAhYcHMzs7OyYTCZjoaGhrKSkRC/mzJkzrH///kwqlbLWrVuzZcuWGdQvKuCEEEIaqxYIvg+8KWusvX+EEELMR2PVAlqFTgghhJghwW+l2pTVTW4Y44YuhBBCzFNdDTD2hDcV8EZUUlICAHBzcxM4E0IIIUIrKSmBXG68G3PRNfBGpNPpcOPGDdjb24PjuId6j+LiYri5uSEnJ6fJXEdvan1qav0Bml6fmlp/AOqTOajrT3Z2NjiOg6urK0Qi4125phF4IxKJRGjTpo1R3qsp3himqfWpqfUHaHp9amr9AahP5kAulzdKf2gRGyGEEGKGqIATQgghZogKuImTSqUIDw+HVCoVOhWjaWp9amr9AZpen5pafwDqkzlo7P7QIjZCCCHEDNEInBBCCDFDVMAJIYQQM0QFnBBCCDFDVMAJIYQQM0QF3MStXbsWnp6esLKygkqlQlJSktApNUhERAT69OkDe3t7ODk5YdSoUcjIyNCLqaiowMyZM9GyZUvY2dnhhRdeQG5urkAZG2bZsmXgOA5z5szhj5ljf65fv45XXnkFLVu2hLW1Nby9vXHy5Em+nTGGxYsXw8XFBdbW1ggICEBmZqaAGd+fVqvFokWL4OXlBWtrazzxxBP473//q3cPalPu09GjR/H888/D1dUVHMdhz549eu0Nyb2wsBAhISGQyWRQKBSYOnUqSktLH2Mv9N2vT9XV1Zg3bx68vb1ha2sLV1dXTJw4ETdu3NB7D3Pq07+99tpr4DgOn3/+ud5xY/SJCrgJ27lzJ+bOnYvw8HCkpqaiR48eCAwMRF5entCpPdCRI0cwc+ZMJCQkIDo6GtXV1Rg6dCju3LnDx7z11lv4+eefsXv3bhw5cgQ3btzAmDFjBMy6YZKTk7FhwwZ0795d77i59ef27dvo168fLC0tcfDgQaSlpeGTTz5BixYt+JgVK1Zg9erVWL9+PRITE2Fra4vAwEBUVFQImPm9LV++HOvWrcOXX36JixcvYvny5VixYgXWrFnDx5hyn+7cuYMePXpg7dq19bY3JPeQkBBcuHAB0dHRiIqKwtGjRxEWFva4unCX+/WprKwMqampWLRoEVJTU/Hjjz8iIyMDI0aM0Iszpz79008//YSEhAS4urre1WaUPhn16eLEqPz8/NjMmTP577VaLXN1dWURERECZvVw8vLyGAB25MgRxhhjRUVFzNLSku3evZuPuXjxIgPA4uPjhUrzgUpKSlj79u1ZdHQ0GzRoEJs9ezZjzDz7M2/ePNa/f/97tut0OqZUKtnKlSv5Y0VFRUwqlbLt27c/jhQNFhQUxKZMmaJ3bMyYMSwkJIQxZl59AsB++ukn/vuG5J6WlsYAsOTkZD7m4MGDjOM4dv369ceW+738u0/1SUpKYgDY1atXGWPm26dr166x1q1bs/PnzzMPDw/22Wef8W3G6hONwE1UVVUVUlJSEBAQwB8TiUQICAhAfHy8gJk9HI1GAwBwcHAAAKSkpKC6ulqvf506dYK7u7tJ92/mzJkICgrSyxswz/7s27cPvr6+eOmll+Dk5AQfHx9s2rSJb8/KyoJardbrk1wuh0qlMtk+9e3bFzExMbh06RIA4MyZMzh+/DiGDx8OwDz7VKchucfHx0OhUMDX15ePCQgIgEgkQmJi4mPP+WFoNBpwHAeFQgHAPPuk0+kwYcIEvPvuu+jatetd7cbqEz3MxETl5+dDq9XC2dlZ77izszPS09MFyurh6HQ6zJkzB/369UO3bt0AAGq1GhKJhP9HWsfZ2RlqtVqALB9sx44dSE1NRXJy8l1t5tifK1euYN26dZg7dy7ef/99JCcnY9asWZBIJJg0aRKfd33/DZpqn+bPn4/i4mJ06tQJFhYW0Gq1WLp0KUJCQgDALPtUpyG5q9VqODk56bWLxWI4ODiYfP+A2nUk8+bNQ3BwMP/wD3Ps0/LlyyEWizFr1qx6243VJyrgpNHNnDkT58+fx/Hjx4VO5aHl5ORg9uzZiI6OhpWVldDpGIVOp4Ovry8+/vhjAICPjw/Onz+P9evXY9KkSQJn93B27dqFbdu24bvvvkPXrl1x+vRpzJkzB66urmbbp+aiuroaY8eOBWMM69atEzqdh5aSkoIvvvgCqampD/0Y6YaiKXQT5ejoCAsLi7tWMefm5kKpVAqUleHeeOMNREVF4fDhw3qPVlUqlaiqqkJRUZFevKn2LyUlBXl5eejVqxfEYjHEYjGOHDmC1atXQywWw9nZ2az6AwAuLi7o0qWL3rHOnTsjOzsbAPi8zem/wXfffRfz58/H+PHj4e3tjQkTJuCtt95CREQEAPPsU52G5K5UKu9a5FpTU4PCwkKT7l9d8b569Sqio6P1Hr1pbn06duwY8vLy4O7uzv+uuHr1Kt5++214enoCMF6fqICbKIlEgt69eyMmJoY/ptPpEBMTA39/fwEzaxjGGN544w389NNPiI2NhZeXl1577969YWlpqde/jIwMZGdnm2T/hgwZgnPnzuH06dP8y9fXFyEhIfz/Nqf+AEC/fv3u2tp36dIleHh4AAC8vLygVCr1+lRcXIzExEST7VNZWRlEIv1faxYWFtDpdADMs091GpK7v78/ioqKkJKSwsfExsZCp9NBpVI99pwboq54Z2Zm4rfffkPLli312s2tTxMmTMDZs2f1fle4urri3XffxS+//ALAiH16+LV3pLHt2LGDSaVSFhkZydLS0lhYWBhTKBRMrVYLndoDzZgxg8nlcvb777+zmzdv8q+ysjI+5rXXXmPu7u4sNjaWnTx5kvn7+zN/f38BszbMP1ehM2Z+/UlKSmJisZgtXbqUZWZmsm3btjEbGxv27bff8jHLli1jCoWC7d27l509e5aNHDmSeXl5sfLycgEzv7dJkyax1q1bs6ioKJaVlcV+/PFH5ujoyN577z0+xpT7VFJSwk6dOsVOnTrFALBPP/2UnTp1il+R3ZDchw0bxnx8fFhiYiI7fvw4a9++PQsODhaqS/ftU1VVFRsxYgRr06YNO336tN7visrKSrPsU33+vQqdMeP0iQq4iVuzZg1zd3dnEomE+fn5sYSEBKFTahAA9b62bNnCx5SXl7PXX3+dtWjRgtnY2LDRo0ezmzdvCpe0gf5dwM2xPz///DPr1q0bk0qlrFOnTmzjxo167Tqdji1atIg5OzszqVTKhgwZwjIyMgTK9sGKi4vZ7Nmzmbu7O7OysmJt27ZlH3zwgV4xMOU+HT58uN5/N5MmTWKMNSz3goICFhwczOzs7JhMJmOhoaGspKREgN7Uul+fsrKy7vm74vDhw2bZp/rUV8CN0Sd6nCghhBBihugaOCGEEGKGqIATQgghZogKOCGEEGKGqIATQgghZogKOCGEEGKGqIATQgghZogKOCGEEGKGqIATQgghZogKOCHknm7duoUZM2bA3d0dUqkUSqUSgYGBOHHiBACA4zjs2bNH2CQJaabocaKEkHt64YUXUFVVha1bt6Jt27bIzc1FTEwMCgoKhE6NkGaPRuCEkHoVFRXh2LFjWL58OQYPHgwPDw/4+flhwYIFGDFiBP9oxNGjR4PjOP57ANi7dy969eoFKysrtG3bFkuWLEFNTQ3fznEc1q1bh+HDh8Pa2hpt27bF999/z7dXVVXhjTfegIuLC6ysrODh4cE/EpQQUosKOCGkXnZ2drCzs8OePXtQWVl5V3tycjIAYMuWLbh58yb//bFjxzBx4kTMnj0baWlp2LBhAyIjI7F06VK98xctWoQXXngBZ86cQUhICMaPH4+LFy8CAFavXo19+/Zh165dyMjIwLZt2/T+QCCEAPQwE0LIPf3www+YNm0aysvL0atXLwwaNAjjx49H9+7dAdSOpH/66SeMGjWKPycgIABDhgzBggUL+GPffvst3nvvPdy4cYM/77XXXsO6dev4mCeffBK9evXCV199hVmzZuHChQv47bffwHHc4+ksIWaGRuCEkHt64YUXcOPGDezbtw/Dhg3D77//jl69eiEyMvKe55w5cwYffvghP4K3s7PDtGnTcPPmTZSVlfFx/v7+euf5+/vzI/DJkyfj9OnT6NixI2bNmoVff/21UfpHiDmjAk4IuS8rKys888wzWLRoEeLi4jB58mSEh4ffM760tBRLlizB6dOn+de5c+eQmZkJKyurBn1mr169kJWVhf/+978oLy/H2LFj8eKLLxqrS4Q0CVTACSEG6dKlC+7cuQMAsLS0hFar1Wvv1asXMjIy0K5du7teItHfv3ISEhL0zktISEDnzp3572UyGcaNG4dNmzZh586d+OGHH1BYWNiIPSPEvNA2MkJIvQoKCvDSSy9hypQp6N69O+zt7XHy5EmsWLECI0eOBAB4enoiJiYG/fr1g1QqRYsWLbB48WI899xzcHd3x4svvgiRSIQzZ87g/Pnz+Oijj/j33717N3x9fdG/f39s27YNSUlJ2Lx5MwDg008/hYuLC3x8fCASibB7924olUooFAohfhSEmCZGCCH1qKioYPPnz2e9evVicrmc2djYsI4dO7KFCxeysrIyxhhj+/btY+3atWNisZh5eHjw5x46dIj17duXWVtbM5lMxvz8/NjGjRv5dgBs7dq17JlnnmFSqZR5enqynTt38u0bN25kPXv2ZLa2tkwmk7EhQ4aw1NTUx9Z3QswBrUInhDx29a1eJ4QYhq6BE0IIIWaICjghhBBihmgRGyHksaMrd4Q8OhqBE0IIIWaICjghhBBihqiAE0IIIWaICjghhBBihqiAE0IIIWaICjghhBBihqiAE0IIIWaICjghhBBihqiAE0IIIWbo/wFVGCrhrpUbVgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.plot(range(len(lrs)), lrs)\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Steps\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f85b01-859b-4454-a3a3-c7ef593735a6",
   "metadata": {},
   "source": [
    "- And a quick look at the loss curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "445d8155-6eae-4b50-a381-d0820ebc27cc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABhwklEQVR4nO3dd3xT9frA8U+SrnSXQhelpUChUMoeQsEFCogIKOLgIohXlCHiQOQqCC4EERFUEO8Vfg6GC0RZFmRI2RsEyiq7pczumXx/f5w2baVgW1qSlOf9eh2SnPmcpOTJ95zv0CmlFEIIIYSwOXprByCEEEKIkkmSFkIIIWyUJGkhhBDCRkmSFkIIIWyUJGkhhBDCRkmSFkIIIWyUJGkhhBDCRkmSFkIIIWyUJGkhhBDCRkmSFsKGnThxAp1Ox+7du60dihDCCiRJC1HJdDrdDafx48dbO0QhhI1ysHYAQlR1CQkJlucLFy5k3LhxxMXFWea5u7tbIywhhB2QkrQQlSwgIMAyeXl5odPpLK/9/PyYOnUqwcHBODs706xZM1asWHHdfZlMJgYNGkRERASnTp0C4JdffqFFixa4uLhQp04dJkyYQF5enmUbnU7Hf//7X3r37o2rqyvh4eEsWbLEsvzKlSv069ePGjVqYDQaCQ8PZ86cOdeN4ccffyQqKgqj0Yivry+dO3cmPT3dsvy///0vDRs2xMXFhYiICD7//PNi258+fZq+ffvi7e1NtWrV6NmzJydOnLAsHzhwIL169WLKlCkEBgbi6+vLsGHDyM3NLfV7LkSVoYQQt8ycOXOUl5eX5fXUqVOVp6enmj9/vjp06JB67bXXlKOjozp8+LBSSqn4+HgFqF27dqmsrCzVu3dv1bx5c5WUlKSUUmr9+vXK09NTzZ07Vx07dkz9/vvvqnbt2mr8+PGWYwAqODhYzZs3Tx05ckSNGDFCubu7q0uXLimllBo2bJhq1qyZ2rZtm4qPj1cxMTFqyZIlJcZ/7tw55eDgoKZOnari4+PV3r171WeffaZSU1OVUkp9++23KjAwUP3000/q+PHj6qefflLVqlVTc+fOVUoplZOToxo2bKgGDRqk9u7dqw4cOKCefPJJ1aBBA5Wdna2UUmrAgAHK09NTPf/88+rgwYPq119/Va6urmr27NkV+2EIYQckSQtxC/09SQcFBan33nuv2DqtW7dWQ4cOVUoVJuk///xTderUSXXo0EFdvXrVsm6nTp3U+++/X2z7b775RgUGBlpeA+rNN9+0vE5LS1OAWr58uVJKqR49eqinn366VPHv2LFDAerEiRMlLq9bt66aN29esXnvvPOOateunSW2Bg0aKLPZbFmenZ2tjEajWrlypVJKS9KhoaEqLy/Pss6jjz6qHnvssVLFKERVIvekhbCSlJQUzp07R3R0dLH50dHR7Nmzp9i8J554guDgYP744w+MRqNl/p49e4iNjeW9996zzDOZTGRlZZGRkYGrqysATZo0sSx3c3PD09OTpKQkAIYMGcIjjzzCzp07uf/+++nVqxft27cvMeamTZvSqVMnoqKi6NKlC/fffz99+vTBx8eH9PR0jh07xjPPPMOzzz5r2SYvLw8vLy9LvEePHsXDw6PYfrOysjh27JjldWRkJAaDwfI6MDCQffv23eDdFKJqkiQthB144IEH+Pbbb9m0aRP33nuvZX5aWhoTJkzg4YcfvmYbFxcXy3NHR8diy3Q6HWazGYBu3bpx8uRJli1bRkxMDJ06dWLYsGFMmTLlmn0aDAZiYmLYuHEjv//+OzNmzOCNN95gy5Ytlh8EX375JW3btr1mu4J4W7ZsyXfffXfNvmvUqFGqeIW4nUiSFsJKPD09CQoKIjY2lrvuussyPzY2ljZt2hRbd8iQITRu3JiHHnqIpUuXWtZv0aIFcXFx1KtX76ZiqVGjBgMGDGDAgAF07NiRUaNGlZikQUuY0dHRREdHM27cOEJDQ1m0aBEvv/wyQUFBHD9+nH79+pW4bYsWLVi4cCF+fn54enreVMxC3A4kSQthRaNGjeKtt96ibt26NGvWjDlz5rB79+4SS5ovvPACJpOJBx98kOXLl9OhQwfGjRvHgw8+SEhICH369EGv17Nnzx7279/Pu+++W6oYxo0bR8uWLYmMjCQ7O5vffvuNhg0blrjuli1bWL16Nffffz9+fn5s2bKFCxcuWNafMGECI0aMwMvLi65du5Kdnc327du5cuUKL7/8Mv369ePDDz+kZ8+evP322wQHB3Py5El+/vlnXnvtNYKDg8v/ZgpRBUmSFsKKRowYQXJyMq+88gpJSUk0atSIJUuWEB4eXuL6I0eOxGw288ADD7BixQq6dOnCb7/9xttvv82kSZNwdHQkIiKCf//736WOwcnJiTFjxnDixAmMRiMdO3ZkwYIFJa7r6enJ+vXrmTZtGikpKYSGhvLRRx/RrVs3AP7973/j6urKhx9+yKhRo3BzcyMqKoqRI0cC4Orqyvr16xk9ejQPP/wwqamp1KxZk06dOknJWogS6JRSytpBCCGEEOJa0pmJEEIIYaMkSQshhBA2SpK0EEIIYaMkSQshhBA2SpK0EEIIYaMkSQshhBA2SpL0DXz22WfUrl0bFxcX2rZty9atW60az/r16+nRowdBQUHodDoWL15cbLlSinHjxhEYGIjRaKRz584cOXKk2DqXL1+mX79+eHp64u3tzTPPPENaWlqxdfbu3UvHjh1xcXGhVq1aTJ48+ZpYfvjhByIiInBxcSEqKoply5bd1LlNnDiR1q1b4+HhgZ+fH7169So25jJo/TsPGzYMX19f3N3deeSRRzh//nyxdU6dOkX37t1xdXXFz8+PUaNGFRu2EWDt2rW0aNECZ2dn6tWrx9y5c6+JpyI/+5kzZ9KkSRM8PT3x9PSkXbt2LF++3O7PqyQffPABOp3O0i7ans9v/Pjx6HS6YlNERITdn1eBs2fP8q9//QtfX1+MRiNRUVFs377dstxev09q1659zeem0+kYNmwYYIefm3XH97BdCxYsUE5OTuqrr75Sf/31l3r22WeVt7e3On/+vNViWrZsmXrjjTfUzz//rAC1aNGiYss/+OAD5eXlpRYvXqz27NmjHnroIRUWFqYyMzMt63Tt2lU1bdpUbd68Wf3555+qXr166oknnrAsT05OVv7+/qpfv35q//79av78+cpoNKovvvjCsk5sbKwyGAxq8uTJ6sCBA+rNN99Ujo6Oat++feU+ty5duqg5c+ao/fv3q927d6sHHnhAhYSEqLS0NMs6zz//vKpVq5ZavXq12r59u7rjjjtU+/btLcvz8vJU48aNVefOndWuXbvUsmXLVPXq1dWYMWMs6xw/fly5urqql19+WR04cEDNmDFDGQwGtWLFCss6Ff3ZL1myRC1dulQdPnxYxcXFqf/85z/K0dFR7d+/367P6++2bt2qateurZo0aaJefPFFy3x7Pb+33npLRUZGqoSEBMt04cIFuz8vpZS6fPmyCg0NVQMHDlRbtmxRx48fVytXrlRHjx61rGOv3ydJSUnFPrOYmBgFqDVr1iil7O9zkyR9HW3atFHDhg2zvDaZTCooKEhNnDjRilEV+nuSNpvNKiAgQH344YeWeVevXlXOzs5q/vz5SimlDhw4oAC1bds2yzrLly9XOp1OnT17Viml1Oeff658fHwsY/sqpdTo0aNVgwYNLK/79u2runfvXiyetm3bqueee67Czi8pKUkBat26dZZzcXR0VD/88INlnYMHDypAbdq0SSml/YjR6/UqMTHRss7MmTOVp6en5Xxee+01FRkZWexYjz32mOrSpYvl9a347H18fNR///vfKnNeqampKjw8XMXExKi77rrLkqTt+fzeeust1bRp0xKX2fN5KaX9n+7QocN1l1el75MXX3xR1a1bV5nNZrv83ORydwlycnLYsWMHnTt3tszT6/V07tyZTZs2WTGy64uPjycxMbFYzF5eXrRt29YS86ZNm/D29qZVq1aWdTp37oxer2fLli2Wde68806cnJws63Tp0oW4uDiuXLliWafocQrWqcj3Jjk5GYBq1aoBsGPHDnJzc4sdNyIigpCQkGLnFxUVhb+/f7G4UlJS+Ouvv0oVe2V/9iaTiQULFpCenk67du2qzHkNGzaM7t27XxODvZ/fkSNHCAoKok6dOvTr149Tp05VifNasmQJrVq14tFHH8XPz4/mzZvz5ZdfWpZXle+TnJwcvv32WwYNGoROp7PLz02SdAkuXryIyWQq9iEB+Pv7k5iYaKWobqwgrhvFnJiYiJ+fX7HlDg4OVKtWrdg6Je2j6DGut05FvTdms5mRI0cSHR1N48aNLcd0cnLC29v7hudX3thTUlLIzMystM9+3759uLu74+zszPPPP8+iRYto1KiR3Z8XwIIFC9i5cycTJ068Zpk9n1/btm2ZO3cuK1asYObMmcTHx9OxY0dSU1Pt+rwAjh8/zsyZMwkPD2flypUMGTKEESNG8H//93/F4rP375PFixdz9epVBg4caDmWvX1uMsCGsDnDhg1j//79bNiwwdqhVJgGDRqwe/dukpOT+fHHHxkwYADr1q2zdlg37fTp07z44ovExMQUG7+6KigYNASgSZMmtG3bltDQUL7//nuMRqMVI7t5ZrOZVq1a8f777wPQvHlz9u/fz6xZsxgwYICVo6s4//vf/+jWrRtBQUHWDqXcpCRdgurVq2MwGK6p8Xf+/HkCAgKsFNWNFcR1o5gDAgJISkoqtjwvL4/Lly8XW6ekfRQ9xvXWqYj3Zvjw4fz222+sWbOm2LCFAQEB5OTkcPXq1RueX3lj9/T0xGg0Vtpn7+TkRL169WjZsiUTJ06kadOmfPLJJ3Z/Xjt27CApKYkWLVrg4OCAg4MD69atY/r06Tg4OODv72/X51eUt7c39evX5+jRo3b/uQUGBtKoUaNi8xo2bGi5nF8Vvk9OnjzJqlWrio0IZ4+fmyTpEjg5OdGyZUtWr15tmWc2m1m9ejXt2rWzYmTXFxYWRkBAQLGYU1JS2LJliyXmdu3acfXqVXbs2GFZ548//sBsNtO2bVvLOuvXryc3N9eyTkxMDA0aNMDHx8eyTtHjFKxzM++NUorhw4ezaNEi/vjjD8LCwootb9myJY6OjsWOGxcXx6lTp4qd3759+4p9ccTExODp6Wn5Qvqn2G/VZ282m8nOzrb78+rUqRP79u1j9+7dlqlVq1b069fP8tyez6+otLQ0jh07RmBgoN1/btHR0dc0cTx8+DChoaGA/X+fAMyZMwc/Pz+6d+9umWeXn1uZqpndRhYsWKCcnZ3V3Llz1YEDB9TgwYOVt7d3sRp/t1pqaqratWuX2rVrlwLU1KlT1a5du9TJkyeVUlqTCW9vb/XLL7+ovXv3qp49e5bYZKJ58+Zqy5YtasOGDSo8PLxYk4mrV68qf39/1b9/f7V//361YMEC5erqek2TCQcHBzVlyhR18OBB9dZbb910E6whQ4YoLy8vtXbt2mLNJzIyMizrPP/88yokJET98ccfavv27apdu3aqXbt2luUFTSfuv/9+tXv3brVixQpVo0aNEptOjBo1Sh08eFB99tlnJTadqMjP/vXXX1fr1q1T8fHxau/ever1119XOp1O/f7773Z9XtdTtHa3PZ/fK6+8otauXavi4+NVbGys6ty5s6pevbpKSkqy6/NSSmsu5+DgoN577z115MgR9d133ylXV1f17bffWtax5+8Tk8mkQkJC1OjRo69ZZm+fmyTpG5gxY4YKCQlRTk5Oqk2bNmrz5s1WjWfNmjUKuGYaMGCAUkprNjF27Fjl7++vnJ2dVadOnVRcXFyxfVy6dEk98cQTyt3dXXl6eqqnn35apaamFltnz549qkOHDsrZ2VnVrFlTffDBB9fE8v3336v69esrJycnFRkZqZYuXXpT51bSeQFqzpw5lnUyMzPV0KFDlY+Pj3J1dVW9e/dWCQkJxfZz4sQJ1a1bN2U0GlX16tXVK6+8onJzc4uts2bNGtWsWTPl5OSk6tSpU+wYBSrysx80aJAKDQ1VTk5OqkaNGqpTp06WBG3P53U9f0/S9np+jz32mAoMDFROTk6qZs2a6rHHHivWjthez6vAr7/+qho3bqycnZ1VRESEmj17drHl9vx9snLlSgVcE69S9ve56ZRSqmxlbyGEEELcCnJPWgghhLBRkqSFEEIIGyVJWgghhLBRkqSFEEIIGyVJWgghhLBRkqSFEEIIGyVJ+gays7MZP3482dnZ1g6lUlTl85Nzs09ybvZJzq3ySDvpG0hJScHLy4vk5GQ8PT2tHU6Fq8rnJ+dmn+Tc7JOcW+WRkrQQQghhoyRJCyGEEDaqyo8nnZeXx65du/D390evL9tvktTUVADOnj1LSkpKZYRnVVX5/OTc7JOcm326Hc7t9OnTZGRk0Lx5cxwcbl3qrPL3pLdt20abNm2sHYYQQogqYOvWrbRu3fqWHa/Kl6T9/f0B7Y0NDAy0cjRCCCHsUUJCAm3atLHklFulyifpgkvcgYGBBAcHWzkaIYQQ9qyst01v+ni39GhCCCGEKDVJ0kIIIYSNkiQthBBC2Kgqf09aCFG1mc1mcnJyrB2GqAKcnJxu+T3nfyJJugzSsvNwd5a3TAhbkZOTQ3x8PGaz2dqhiCpAr9cTFhaGk5OTtUOxkIxTCnkmMy9+txnjkd947cnu+DWMtnZIQtz2lFIkJCRgMBioVauWzZWAhH0xm82cO3eOhIQEQkJC0Ol01g4JkCRdKg4GPV3O/4+HDD8Sv2o/NFxi7ZCEuO3l5eWRkZFBUFAQrq6u1g5HVAE1atTg3Llz5OXl4ejoaO1wAKk4VmqGlv0BCL20HnU53srRCCFMJhOATV2aFPat4G+p4G/LFkiSLqUO7aJZb26KHsXlNZ9ZOxwhRD5buSwp7J8t/i1Jki4lL6Mj+2o+BoDbgfmQnWbliIQQQlR1kqTLoF50b06Y/XExpWHes9Da4QghBAC1a9dm2rRppV5/7dq16HQ6rl69WmkxAcydOxdvb+9KPUZVJ0m6DO6O8GehvhsAWbGfQ9UeQEwIUcF0Ot0Np/Hjx5drv9u2bWPw4MGlXr99+/YkJCTg5eVVruOJW0dqd5eBs4OBrMaPk75vPm7JR+H4Wqh7j7XDEkLYiYSEBMvzhQsXMm7cOOLi4izz3N3dLc+VUphMplKNXVyjRo0yxeHk5ERAQECZthHWISXpMnqgVQN+NN0JgGnzF1aORghhTwICAiyTl5cXOp3O8vrQoUN4eHiwfPlyWrZsibOzMxs2bODYsWP07NkTf39/3N3dad26NatWrSq2379f7tbpdPz3v/+ld+/euLq6Eh4ezpIlhU1H/365u+Cy9MqVK2nYsCHu7u507dq12I+KvLw8RowYgbe3N76+vowePZoBAwbQq1evMr0HM2fOpG7dujg5OdGgQQO++eYbyzKlFOPHjyckJARnZ2eCgoIYMWKEZfnnn39OeHg4Li4u+Pv706dPnzId2x5Jki6jliE+rHR7CAD9kRUgzbGEsAlKKTJy8qwyqQq89fX666/zwQcfcPDgQZo0aUJaWhoPPPAAq1evZteuXXTt2pUePXpw6tSpG+5nwoQJ9O3bl7179/LAAw/Qr18/Ll++fN31MzIymDJlCt988w3r16/n1KlTvPrqq5blkyZN4rvvvmPOnDnExsaSkpLC4sWLy3RuixYt4sUXX+SVV15h//79PPfcczz99NOsWbMGgJ9++omPP/6YL774giNHjrB48WKioqIA2L59OyNGjODtt98mLi6OFStWcOedd5bp+PZILneXkV6vo3mLNqzfEMWdhn2w7b/Q5T1rhyXEbS8z10SjcSutcuwDb3fB1alivk7ffvtt7rvvPsvratWq0bRpU8vrd955h0WLFrFkyRKGDx9+3f0MHDiQJ554AoD333+f6dOns3XrVrp27Vri+rm5ucyaNYu6desCMHz4cN5++23L8hkzZjBmzBh69+4NwKeffsqyZcvKdG5Tpkxh4MCBDB06FICXX36ZzZs3M2XKFO655x5OnTpFQEAAnTt3xtHRkZCQENq0aQPAqVOncHNz48EHH8TDw4PQ0FCaN29epuPbI6uWpNevX0+PHj0ICgpCp9Nd86tMKcW4ceMIDAzEaDTSuXNnjhw5Yp1gi+jVrCZzTNofunnn19IcSwhRYVq1alXsdVpaGq+++ioNGzbE29sbd3d3Dh48+I8l6SZNmlieu7m54enpSVJS0nXXd3V1tSRogMDAQMv6ycnJnD9/3pIwAQwGAy1btizTuR08eJDo6OLdKkdHR3Pw4EEAHn30UTIzM6lTpw7PPvssixYtIi8vD4D77ruP0NBQ6tSpQ//+/fnuu+/IyMgo0/HtkVVL0unp6TRt2pRBgwbx8MMPX7N88uTJTJ8+nf/7v/8jLCyMsWPH0qVLFw4cOICLi4sVItaE+3twMaAjJy5+Tc28VPSJeyG0vdXiEUKA0dHAgbe7WO3YFcXNza3Y61dffZWYmBimTJlCvXr1MBqN9OnT5x9H/vp7t5Y6ne6GA5GUtH5FXsYvjVq1ahEXF8eqVauIiYlh6NChfPjhh6xbtw4PDw927tzJ2rVr+f333xk3bhzjx49n27ZtVbqZl1VL0t26dePdd9+1XD4pSinFtGnTePPNN+nZsydNmjTh66+/5ty5c2W+D1IZejavxdDcFxno87UkaCFsgE6nw9XJwSpTZfZUFRsby8CBA+nduzdRUVEEBARw4sSJSjteSby8vPD392fbtm2WeSaTiZ07d5ZpPw0bNiQ2NrbYvNjYWBo1amR5bTQa6dGjB9OnT2ft2rVs2rSJffv2AeDg4EDnzp2ZPHkye/fu5cSJE/zxxx83cWa2z2bvScfHx5OYmEjnzp0t87y8vGjbti2bNm3i8ccft2J08FDTIN5fVhvzmRxOXkon1NftnzcSQogyCg8P5+eff6ZHjx7odDrGjh1rlaE5X3jhBSZOnEi9evWIiIhgxowZXLlypUw/UEaNGkXfvn1p3rw5nTt35tdff+Xnn3+21FafO3cuJpOJtm3b4urqyrfffovRaCQ0NJTffvuN48ePc+edd+Lj48OyZcswm800aNCgsk7ZJths7e7ExEQA/P39i8339/e3LCtJdnY2KSkplik1NbVS4vPzdCG6XnUAFu88CxcOV8pxhBC3t6lTp+Lj40P79u3p0aMHXbp0oUWLFrc8jtGjR/PEE0/w1FNP0a5dO9zd3enSpUuZbj326tWLTz75hClTphAZGckXX3zBnDlzuPvuuwHw9vbmyy+/JDo6miZNmrBq1Sp+/fVXfH198fb25ueff+bee++lYcOGzJo1i/nz5xMZGVlJZ2wbdOpW33S4Dp1Ox6JFiyxt7jZu3Eh0dDTnzp0jMDDQsl7fvn3R6XQsXFhyt5zjx49nwoQJ18w/ffo0wcHBFRrzzzvPMOb7bSxxfYf65uPoXtwNPrUr9BhVVnYqzO0O/o2h1+fWjkbYoaysLOLj4wkLC7NqHZXbldlspmHDhvTt25d33nnH2uFUiBv9TZ05c4ZatWpVSi65EZstSRf0hnP+/Pli88+fP3/DnnLGjBlDcnKyZTpw4EClxdglMgC9o5Hzua4ogxOcLdv9mdvagSWQsAd2fwep5/95fSGEVZ08eZIvv/ySw4cPs2/fPoYMGUJ8fDxPPvmktUOr0mw2SYeFhREQEMDq1ast81JSUtiyZQvt2rW77nbOzs54enpaJg8Pj0qL0c3Zgfsa+TMubyBTIhdB42trqIvrOL5We2zUCzz8b7SmEMIG6PV65s6dS+vWrYmOjmbfvn2sWrWKhg0bWju0Ks2qFcfS0tI4evSo5XV8fDy7d++mWrVqhISEMHLkSN59913Cw8MtTbCCgoLK3A1dZerdvCZL9pxj4V8ZvNTTjKPBZn/32A6zCY7md2vY9jnrxiKEKJVatWpdUzNbVD6rJunt27dzzz2FA1S8/PLLAAwYMIC5c+fy2muvkZ6ezuDBg7l69SodOnRgxYoVNnX/qUN4dXzdnLiUnsOGoxe5xysJ/CPBBgcPtxlntkPmZXDxguA2/7y+EELcpqxa7Lv77rtRSl0zzZ07F9Aqk7399tskJiaSlZXFqlWrqF+/vjVDvoajQU+PpkGAImDJkzArGuLXWTss23Ykv+vGup3g4BKY/yQcl/dMCCH+Tq7NVoBezWsCOnam+mgztsy2ajw278jv2mP9Ltq96bilcGipVUMSQghbZLOdmdiTpsFehFV346tL99HP+XeIWwZXTkhzrJKknIPEfYAO6nUG7xDwCoaIB60dmRBC2BwpSVcAnU5Hr2Y1OaZqstelJaBg65fWDss2FZSig1uBW3WtS9W7XgP/RjfeTgghbkOSpCtIr+ZBAExPza8It+sbyEm3YkQ26nB+kg63zkAIQghhTyRJV5BQXzdahHiz2tSMFGMtyEqGvSX3inbbyssubB9d//7C+TnpsP9nWP+hVcISwt7cfffdjBw50vK6du3aTJs27YbblDQccHlU1H5uZPz48TRr1qxSj2EvJElXoF7Na6LQ870uv5S4ZTbYRq+rtuFkLOSmg0cgBBSOdUv6BfjxaVgzETKvWC8+ISpZjx496Nq1a4nL/vzzT3Q6HXv37i3zfrdt28bgwYNvNrxirpcoExIS6NatW4UeS1yfJOkK1D0qEAe9jumX22J2cIULByF+vbXDsh3OXhDZGxo/UrwduU9tqBEBygRHV193cyHs3TPPPENMTAxnzpy5ZtmcOXNo1aoVTZo0KWHLG6tRowaurq4VEeI/CggIwNnZ+ZYcS0iSrlC+7s7cVb8GKbixx/cBbeZWaY5lEdwSHp0LXd67dln9/KsPBRXLhKiCHnzwQWrUqGHpC6JAWloaP/zwA8888wyXLl3iiSeeoGbNmri6uhIVFcX8+fNvuN+/X+4+cuQId955Jy4uLjRq1IiYmJhrthk9ejT169fH1dWVOnXqMHbsWHJzcwFtyMgJEyawZ88edDodOp2uWP8VRS9379u3j3vvvRej0Yivry+DBw8mLS3NsnzgwIH06tWLKVOmEBgYiK+vL8OGDbMcqzTMZjNvv/02wcHBODs706xZM1asWGFZnpOTw/DhwwkMDMTFxYXQ0FAmTpwIgFKK8ePHExISgrOzM0FBQYwYMaLUx7Y2SdIVTGszDVOu3qnNKGiOJW4svEiSNpusG4uwbznpZZ9MeYXbm/K0ebmZpdtvGTg4OPDUU08xd+5cig5A+MMPP2AymXjiiSfIysqiZcuWLF26lP379zN48GD69+/P1q1bS3UMs9nMww8/jJOTE1u2bGHWrFmMHj36mvU8PDyYO3cuBw4c4JNPPuHLL7/k448/BuCxxx7jlVdeITIykoSEBBISEnjssceu2Ud6ejpdunTBx8eHbdu28cMPP7Bq1SqGDx9ebL01a9Zw7Ngx1qxZw//93/8xd+7ca36o3Mgnn3zCRx99xJQpU9i7dy9dunThoYce4siRIwBMnz6dJUuW8P333xMXF8d3331H7dq1Afjpp5/4+OOP+eKLLzhy5AiLFy8mKiqq1Me2NmknXcE6N/TH3dmB2OTqJNfpgNe5DbDtv3D/u9YOzbrO7QJHV6hev+QuU2u11boJzbwCZ7ZByB23PkZRNbwfVPZtHp2r3YoBOPQr/DAQQjvA00U62ZkWBRmXrt12fHKZDjVo0CA+/PBD1q1bZxlHec6cOTzyyCN4eXnh5eXFq6++aln/hRdeYOXKlXz//fe0afPP3eiuWrWKQ4cOsXLlSoKCtPfi/fffv+Y+8ptvvml5Xrt2bV599VUWLFjAa6+9htFoxN3dHQcHhxuOOjhv3jyysrL4+uuvcXNzA+DTTz+lR48eTJo0CX9/bfAcHx8fPv30UwwGAxEREXTv3p3Vq1fz7LPPluo9mzJlCqNHj+bxxx8HYNKkSaxZs4Zp06bx2WefcerUKcLDw+nQoQM6nY7Q0FDLtqdOnSIgIIDOnTvj6OhISEhIqd5HWyEl6QpmdDLQtbH2R73YuYc2c+c3Ws3m21nMW/BZG9gxB4CsXBPPf7ODqTGHteUGB6h3n/b88EorBSlE5YuIiKB9+/Z89dVXABw9epQ///yTZ555BgCTycQ777xDVFQU1apVw93dnZUrV3Lq1KlS7f/gwYPUqlXLkqCBEkcOXLhwIdHR0QQEBODu7s6bb75Z6mMUPVbTpk0tCRogOjoas9lMXFycZV5kZCQGg8HyOjAwkKSkpFIdIyUlhXPnzhEdHV1sfnR0NAcPHgS0S+q7d++mQYMGjBgxgt9/L7xt9uijj5KZmUmdOnV49tlnWbRoEXl5edgLKUlXgt7Na/LjjjN8HF+bf7V/EUPTx8DhNq5ooZRWijY4Q9hdAKyNu8CKvxKJOXieZzuG4eHiqN2X3v+jlqQ7v2XloIXd+s+5sm9jKPL/M6KHtg/d38owI/fdXFxFPPPMM7zwwgt89tlnzJkzh7p163LXXdr/jQ8//JBPPvmEadOmERUVhZubGyNHjiQnJ6fCjr9p0yb69evHhAkT6NKlC15eXixYsICPPvqowo5RlKOjY7HXOp0Os9lcYftv0aIF8fHxLF++nFWrVtG3b186d+7Mjz/+SK1atYiLi2PVqlXExMQwdOhQy5WMv8dli6QkXQnuqOOLv6czV7NMrK45VBsV63am08GTC2D0CfCtC8CaQ9qvaJNZseX4ZW29ep21L8akv+DqaSsFK+yek1vZJ0OR8orBQZvnaCzdfsuhb9++6PV65s2bx9dff82gQYPQ5d8Gio2NpWfPnvzrX/+iadOm1KlTh8OHD5d63w0bNuT06dMkJCRY5m3evLnYOhs3biQ0NJQ33niDVq1aER4ezsmTJ4ufrpMTJtON64c0bNiQPXv2kJ5eeG8+NjYWvV5PgwYNSh3zjXh6ehIUFHTNMJmxsbE0atSo2HqPPfYYX375JQsXLuSnn37i8mXtu8VoNNKjRw+mT5/O2rVr2bRpE/v2VdyPrsokSboSGPQ6HmqqXWpavPts4YIK/OVol5y0JiJms+KPuMJLXbHHLmpPXKsVDl15RC55i6rL3d2dxx57jDFjxpCQkMDAgQMty8LDw4mJiWHjxo0cPHiQ5557jvPnz5d63507d6Z+/foMGDCAPXv28Oeff/LGG28UWyc8PJxTp06xYMECjh07xvTp01m0aFGxdWrXrk18fDy7d+/m4sWLZGdfe8uuX79+uLi4MGDAAPbv38+aNWt44YUX6N+/v+V+dEUYNWoUkyZNYuHChcTFxfH666+ze/duXnzxRQCmTp3K/PnzOXToEIcPH+aHH34gICAAb29v5s6dy//+9z/279/P8ePH+fbbbzEajcXuW9sySdKVpKCW96qDSaQkHoefn4Pv+lg5KitQShtUo4j955K5kFr4H37j0SKVcQqaYh2WpliianvmmWe4cuUKXbp0KXb/+M0336RFixZ06dKFu+++m4CAAHr16lXq/er1ehYtWkRmZiZt2rTh3//+N++9V7zZ40MPPcRLL73E8OHDadasGRs3bmTs2LHF1nnkkUfo2rUr99xzDzVq1CixGZirqysrV67k8uXLtG7dmj59+tCpUyc+/fTTsr0Z/2DEiBG8/PLLvPLKK0RFRbFixQqWLFlCeHg4oNVUnzx5Mq1ataJ169acOHGCZcuWodfr8fb25ssvvyQ6OpomTZqwatUqfv31V3x9fSs0xsqiU6pqd4l15swZatWqxenTpwkODr5lx1VK0WXaeg6fT2P6AzV4aE1XrbOOYdughm2NiV2pzu2G2XdBSDt4ejnodExbdZhpq47QJqwaW+O1y1Hb3uhMDQ9nOH8AZrYDBxd4Ld5S+hbi77KysoiPjycsLAwXFxdrhyOqgBv9TVkrl0hJupLodDpLaXreIRN0/QCe/eP2StBQ2DmJq6+l6dUf+fej+7QMplGgJwAbCy55+zUEr1qQlwUn/rzl4QohhC2RJF2JejbTkvTm45c526A/1Gxp5YisoKA5VbjWvCopJYu9Z7R2pXc3qEF0Pe2Sk+WSt04Hd74KD824Pd8vIYQoQpJ0JarpbaRtWDUAftxepK/evIprSmHT0i/C2R3a83Bt1Ku1cRcAaBrshZ+HC+3rVQeKVB4DaDkQWjyljTcthBC3MUnSlezJtiEAzNt6ktyMZPh1JHzSBLLTbrxhVXAkBlAQEAWeWsWY1Ye0Wqr3Rmg1P9vUroaDXseZK5mcupRhrUiFEMIm2XSSNplMjB07lrCwMIxGI3Xr1uWdd97Bnuq6dWscSHV3Z86nZPP7kTSIXwepCbB7nrVDq3wFzajy++XOzjPx5xGtxHxvhB8Abs4ONA/xBv5Wmr56GjbPhP0/3bJwhRDC1th0kp40aRIzZ87k008/5eDBg0yaNInJkyczY8YMa4dWak4Oektp+v82n4I7hmoLNn9WtQeSMOXB0T+05/nNqrYcv0xGjgk/D2cigzwtq7avm3/J+2iRJH00Bla8Dlu+uGUhC/tkTz/ahW2zxb8lm+4WdOPGjfTs2ZPu3bsDWuP6+fPnl3o0GFvRr20In685ytb4yxzq1oMIl3e1kbHilkHDHtYOr3Kc3gLZyWCsZqkAVlCr+94IP/T6wkE2outV55PVR9h07BJms9KWhXeB2h2hwQNaW+uSBuUQtzVHR0d0Oh0XLlygRo0alh67hCgPpRQXLlxAp9PZVHehNp2k27dvz+zZszl8+DD169dnz549bNiwgalTp1o7tDLx93ShS+MAlu5N4P+2JzGx9TPw50ew6bOqm6QLLnXX6wx6A0opy/3oe/IvdRdoVssbo6OBS+k5xJ1PpWGgJ3jVhIG/3eqohR0xGAwEBwdz5swZTpw4Ye1wRBWg0+kIDg4uNhiItdl0kn799ddJSUkhIiICg8GAyWTivffeo1+/ftfdJjs7u1j3dampqbci1H80oF1tlu5NYNGus4wZPhDP2OlwahOc2QHBVbCpUUGPYfmXuo9dSOP05UycDHo61Ctea9vJQU+bsGqsO3yB2KMXtSQtRCm4u7sTHh5Obm6utUMRVYCjo6NNJWiw8ST9/fff89133zFv3jwiIyPZvXs3I0eOJCgoiAEDBpS4zcSJE5kwYcItjvSfta7tQ0SAB4cSU/n+cB7/juoDe+Zr96b7fGXt8CrW1VNw4aA2WEbde4HCS9131PXFzfnaP7v2dX1Zd/gCG49d4t8d6xQuSEvSaok36QsG27kEJWyHwWCwuS9WISqKTVccGzVqFK+//jqPP/44UVFR9O/fn5deeomJEyded5sxY8aQnJxsmQ4cOHALI74+nU7HgPa1Afh600nMbfMrkP21uOqN+FTQgUmtttqgGcDqg/n3oxvUKHGT6PzS9Zbjl8g15Q9EYjbDzPbwy1DtHrcQQtxmbDpJZ2RkoNcXD9FgMNxwHFJnZ2c8PT0tk4eHR2WHWWo9mwXh6eLAqcsZrEsJ0MZWVibYMsvaoVUscx641bB0YJKckcv2k1eAwvbRf9co0BNvV0fSc0zsPXNVm6nXQ91O2vPDKyo7aiGEsDk2naR79OjBe++9x9KlSzlx4gSLFi1i6tSp9O7d29qhlYurkwN9W9UC4P82nYB2w7UFO7+GrBTrBVbR7hgCrxzWHoH1Ry5gMivC/dwJ8S15wAy9Xke7OloXobEyKpYQQgA2nqRnzJhBnz59GDp0KA0bNuTVV1/lueee45133rF2aOX2rztC0em07jFP+LSD6vUhOwV2fWvt0CqWXg+ORqBI06uGfjfaorCL0KLtpeveCzoDXIyDy/GVE6sQQtgom07SHh4eTJs2jZMnT5KZmcmxY8d49913cXJysnZo5Va7uht319fuy36z5TS0G6Yt2DxT6wDE3qWc0+4l5zOZFWvjCu5H3zhJR9fVStK7Tl0lMye/oxejN4S2154fkdK0EOL2YtNJuqp6Kr8C2ffbT5MR8Yh27/b+t+2/ww6l4H9dYGoEJO4DYPfpK1zJyMXTxYGWoT433DysuhuBXi7kmMxsO3G5cIHlkrfclxZC3F4kSVvBXeE1CPV1JTUrj8X7r0C/HyCyN+jtvBlJyjnIvAyZV6Ga1oyqoFb33Q38cDDc+M9Np9MVdhFatB/v/L6/ObHh9hiYRAgh8kmStgK9Xkf/O0IB+HrTCZvsL7ZcvGrCa8fhmd/ByQ0o3hVoaVwzvjRA9XDwCQNTDhxfW6EhCyGELZMkbSWPtqyF0dHAocRUtsbnlz43fAxLX7VeUKnnb37QDwdnCGoGwNmrmRxKTEWvg7vql9w++u8K2kvvP5fM1Yz8cbd1OrnkLYS4LUmSthIvV0d6Na8JaJ2bkJoAq8bD9v/B5eO3NpjzB2D+k/BRffj8DshJL/s+SrgaUFCKbhnqg49b6Sr7+Xu6ULeGG0rB5uMlNMU6ElOsYpoQQlRlkqStaEB77ZL3ir8SSXQOg7bPQ8/PwLPmrQngyklY9LzWq1fcUm1ejQjLpeoy+etnmNESNn5qmfXHwZIH1Pgn0ZamWEWSdGg0OLpBWiIk7il7fEIIYYckSVtRRIAnbcOqYTIr5m05Cd0mQbMntUvGlSktCZa9piXVPfMBBQ0fgn//AQ98WLjelZPwfz3g5MZ/3ueRGLh0VEuiQGaOiY3HtCTb6Tq9jF1PiZXHHJyh7j3a84JuR4UQooqz6QE2bgcD2tdmS/xl5m09xbB76+HsUIk1vLOSYeMM2PQ55OZf0q5zN3QaZxnzuZj1H0L8ekAHA5Zcf79ms5akwdIV6MZjF8nOM1PT20h9f/cyhdmuji96HRy/kE5ichYBXi7agsYPg6Mr1GpTpv0JIYS9kiRtZfc18ifA04XElCxW7E+kZ6Qv7Pw/OPALPPVLxYz8lJsJW7+EDVMhU+tDm6AW0PktLUlfz91jQO8AzfsXzku/pDWzqh5eOO/cTsi4CM6eENIOgNVFanXrytj+28vVkcY1vdh7JpnYoxd5pGWwtqDxI9okhBC3CbncbWWOBj392oYAMHfjCa0m8/opcDJWGyHrZh1fB9NbQMxYLUFXbwB9v4Fn/7hxggatSVWPacXHu/7zI/isLfwyvHD0roLLz3XvAYMjSinWlLIr0Osp8ZK3EELcZqQkbQMebxPC9D+OsOvUVfYmZtKkzbOw5j3YNAOi+txcT2RewZCeBJ7BcM8YaPI4GMr5sSsFqee0kbt2fQN7F0Lrfxe2Xc7vdORgQioJyVkYHQ2WQTPKKrqeL7PWHWPj0UsopQpL40rB+b/gwGK4983CDc7u1O61m/OKTKZrXxt9IKwjeASU7z0QQohbSJK0Dajh4Uz3qEAW7z7H15tOMuWBZ7QSa8Ie+G9nMDhpiVqXf+FDpwPyX4ffV9j/d04G/DAQ/BrCfRO0eb514V8/Qa07wNHl5gLV6eDRuXDHUFj9Npz4EzZ/Xrg8/D4A/jik1eqOrueLi2P57rG3Cq2Gk0FPYkoWxy+mU7dG/n3thD0w+y7tedEkveZ9OBpT+gP4NYKWA6Htc+WKTwghbgVJ0jbiqfa1Wbz7HEv2nOM/DzSkWounYOtsOLv9xht6BRc+N+XAkZXaQBRRj0JAY23+P13WLqtabWDAr1oJevXb2j3psDvBXbu0XXg/umy1uosyOhloEerN5uOX2Xj0YmGSDmgCUX0hYXfxDaqHa/fK9Q75k6HI8/zXOgNciYeEvZB0QCt5F8hK0QY5qXO3dn723o+6EKJKkCRtI5rX8iaqphf7ziazcNtphtz3DtTrDHlZ2iVeZQZUYachBfOqhRXuxNEID80AvSN4BFZuwDqddg+6zt1a6dZbu69+KS2b3aevAqXvCvR6outWZ/Pxy8QevUT/drW1mXo9PPLltSt3nVj6HadfghPrtdJ0gZOxsPZ97RL+iJ2F85PPgmeQJG0hhFVIkrYROp2Op9qFMurHvXy7+SSD76yDoaCXrdJycIYWT5Xr+KlZuSzZc44O9aoT6luGzkx0Oks3oKCNk60UNAr0LGw6VU7t61Xno5jDbDp+CZNZYdBXUKJ089UGNCnKWE2b51PkR48pV6sk5+QGoe3Aq5bW0YxXTS1xe9YENz/th4MQQlQCSdI2pEfTIN5fdpCzVzNZffA890femspNMQfOM3bxfhJTsvBwcWB2/1a0q1u+Cl8FXYF2Kmet7qKaBnvh7uxAcmYuB86lEBXsddP7vK6QttpU1KWjYM7VOmj5a1HJ2+kdwCMoP2kHaQm8zXPgXUtbbsrLv9QuJXFhA8xm7W86L1v7EWrK1m6TmYpUsPSpDc75t5dSzsHFw+DqCwFR2jylYP9P+RUzc4tsn6vts+DR8jxPW6fFU4U/6E9vg82faa1N7hlTGN+iIVp/Dparhn9/LEGrQRDxgPb8/AFY9Zb2f7HHJxX+9lmDJGkb4uJo4LHWIcxad4yvN52s9CSdlJrF+CV/sWyf1kuYk0FPalYeT321hcl9mtC7efA/7KG4XJOZ9YcvADd/qRvAwaCnbVg1Vh9KIvbYxcpN0iXxawijT8Lpzdr42CnnIOWs9ph8Vkve5jxIPqVNBYq2K98yE2KnQ9vBcOcobZ7K/8KRErgoYDZrTSQzLmmPuemQmwW5GVo/B7mZRZ7nP945Cjzy633s/AZ2fq0lqw4vafPSL8HMdlqCzMvRHs25/xzLgF+1OiYAh5bCsle1Hgkf+6ZwnZ+eKfs5hrYvTNIpZ7UfviHtiyfpI79rfS6URX4HSoD23h35HarXL3t8NkqStI3p1zaE2euPseHoRY4mpVLPz6PCj6GU4vvtp3lv6UFSsvIw6HU827EOQ+6qy38W7WPpvgReWriHs1cyGXZPvVJ3RrLtxGVSs/PwdXOiabB3hcTavl51LUkfvcjzd9WtkH2WiaOLdt+9pMp3pjxIO5+fuPOTd8q54n2vJ+zVmsAVLQRcPQWzOmglk4AmENgEAptqXywV0XmNqBgFJbeCv3+zGfIyry3hWdb9W50RUzakX9R+7BV8roeWwtFVUOceaPSQNi/poNZ/virjwDEtnipM0mnn4cxW8IsoXK43aPNvRO+gtR7ROxZWttQVaZHh6qvV3ShaQVWn0+IH7bwMTvn7cdT2YyiyT4ND/qOT9j4UCIiCbpOvbQp5/7taPRxLC5aSHv+maG+JvvWg5+fg4nnj87YjkqRtTK1qrnRq6E/MgfN8s+kkE3o2rtD9x19MZ8zPe9l8/DIAUTW9+OCRKCKDtFLqjCeaE+xj5Iv1x5ny+2HOXMnknV6NcTT8c6nvj4Pape67G/ihr6D7xwXjS287cZnsPFPldptaVgYH7fK21w0GROnxCbQZbKn5DkDiXshO0SqrnYwtsj9n8G+kJe6AKK35nE+Ydi+8vG3bbwcFlSj1+X8bKQlwbLVWKk2/CBmXtdJZdtrfLsnmFZYuTXkwbDO45F+t+e0l2P6V1syv4ApI0gGYFV32+F7cCz7aYDqc3qrt18GlMEkbfQoTtLMXuPpog8k4GvMn17895j93q154jEY9tcFxfGoXznP2gOc3aAmyYHJwzk+szvmJ9B/+Xzd+WJv+7qnFZX8fivKtq01/1+yJm9uvhz8073dz+7Ax8j/fBg1sX5uYA+f5cccZXu3SAA+Xmy9d5ZrMfPnncaatOkJOnhkXRz2v3NeAp6Nr41AkAev1OsY80JBgHyNvLfmLBdtOcy45i8+ebP6PcVTk/egCDfw9qO7uxMW0HHadusod5ewcxWqcXKFW6+Lz6neF52O1ZJ2wV6sdn7gPclLh3C5tKkpn0L7kh20rTNbndmnt5H3DtWOUhlKQnQpZV7XLgplXtHkFA5cAHPxVm1/nbkuNfdKStPvzBmftS75gMjiDg5O2j793HKPMxb+ELxzW9lutDrjnjy2emghntuXf0yy4t3m95/mPZhM8OLVwv4uGwL4ftIFhWj2tzbsYB78MK917UpSp6KXg/B+Zpryy78eyC4NWEi069Gvde7QEXbT+g5sfvBKnrVveKynVw4t31Qvaj5aC+8jCbkmStkHt6/pSt4Ybxy6kc9eHa2lXx5f29XyJrludUF/XMveFvef0VV7/eR8HE1IA6Bhenfd6RRHie/0v9/7tahPoZeSF+btYf/gCfb/YzJyBra9bYzv+YjrHL6bjoNfRMbx6ieuUh06no13d6vy65xwbj160vyRdEoOj1oY9oLE26hlol1KvxOcn7j1aBZgrJ7TJlK1V9Claml75hlYKf+R/Wq90oCX8/T9q9yszr2pJsWhCzryq9RZXlFcIvLSv8PWGj+HsDnh8fmGSProaFj9ftnN0cof/nC0S7xjtMm+vWYWlpXO7YOG/yrZfgPvfKRxOVa/XSsIZRYY19aqlNV909QXX6uBaTSt1OnuUcBm2yOXaglI0QKexcNfowgpUoF2uHXP2Hy7F/u3x7/9XS7p1otdLD3jiusqVpE+fPo1OpyM4WLtPsXXrVubNm0ejRo0YPHhwhQZ49uxZRo8ezfLly8nIyKBevXrMmTOHVq1aVehxbIlOp+OtHpEM+24nl9NzWLovgaX7EgCo6W2kfV1foutVp31dX/w8r9/MKSMnj49+P8yc2HjMCrxdHRn3YCN6N69ZqkTfuZE/C5+7g0Fzt3MwIYXen8cy5+nWRARce7+noBTdJqxahZT8i4qu68uve84Re+wSL1fonm2IXl94CbBo8zCzGVITtI5ainLx1hJQ0cubpzZDbClqtBqctUusRp/i9xpBq9zjVqN40nBy1Ursedn5PxiytIpIeVkUu9letPMYx7/9APQI1ErRRUv9rtW1nvAKEqXBqRTPHYvfu73nTW0gGNciPwwLetm7GUafa+fpDcWTthC3gE6p69Vrv76OHTsyePBg+vfvT2JiIg0aNCAyMpIjR47wwgsvMG7cuAoJ7sqVKzRv3px77rmHIUOGUKNGDY4cOULdunWpW7d0lYjOnDlDrVq1OH36tOVHhb3INZnZc/oqsUcvEXvsIrtOXSHXVPzjCvdztyTstnV88TJqCXLd4Qu8sWgfZ65kAtCrWRBjH2yEr3vZx6o+fTmDgXO2cuxCOh7ODsz8V0s6/K203O+/m4k9eok3uzfk3x3rlPOMr3/8jpPX4KDXsfut+3F3lgtAFkoVltbi18OhZVoiLEjCLt6Fz435zx2NFXdsc5522V2nl2ZmokqzVi4pV5L28fFh8+bNNGjQgOnTp7Nw4UJiY2P5/fffef755zl+/HiFBPf6668TGxvLn3/+We592HOS/ruMnDy2nbjCxqMX2XjsEvvPJRdrOqjXaRXBqrs7W7rmrOlt5N3ejbmnwc3dJ07OyGXwN9vZEn8ZB72OiQ9H8WgrrS1walYuzd+OIc+sWPPq3YRVL0NnKKXUcfIfnL6cyVcDW91Ud6NCCFEe1sol5SqS5Obm4uyslchWrVrFQw9ptRQjIiJISEiosOCWLFlCly5dePTRR1m3bh01a9Zk6NChPPvss9fdJjs7m+zsbMvr1NTUCovH2lydHLirfg3uqq9VvLmakcPm45csJe3jF9LZcyYZ0Ao1T7cP45X76+NWASVPL1dHvn6mDa/9uJdfdp9j1I97OXMlk5Gdw9lw5CJ5ZkVYdbdKSdCgdRG64PJpYo9ekiQthLhtlOvbOzIyklmzZtG9e3diYmJ45513ADh37hy+vhVXsef48ePMnDmTl19+mf/85z9s27aNESNG4OTkxIABA0rcZuLEiUyYMKHCYrBl3q5OdG0cSNfGWj/dCcmZbDx6iWMX0rg/MoBmtbwr9HjODgY+7tuMYB8jn605xierj3DmSiZ5Zu0eYUV0YHI97etVZ8G208QelfGlhRC3j3Jd7l67di29e/cmJSWFAQMG8NVXXwHwn//8h0OHDvHzzz9XSHBOTk60atWKjRs3WuaNGDGCbdu2sWnTphK3+XtJ+uzZszRq1KhKXO62JfO2nGLsL/sxmQv/fOb9uy3t61Vcze6iLqZl0+rdVQBsf7Mz1ctxb10IIcrLri5333333Vy8eJGUlBR8fAprQQ4ePBhX11K22SyFwMBAGjVqVGxew4YN+emn69fcdHZ2tlyKB0hJSamweEShJ9uGEOjtwvDvdpKeY8Ld2YFWtatV2vGquzsTEeDBocRUNh27RI+mQZV2LCGEsBXl6jw4MzOT7OxsS4I+efIk06ZNIy4uDj+/irvkGR0dTVxcXLF5hw8fJjQ0tMKOIcrvngZ+LHyuHU2DvRh2Tz2cHCq3L+r2dbVS+sZjcslbCHF7KNe3as+ePfn6668BuHr1Km3btuWjjz6iV69ezJw5s8KCe+mll9i8eTPvv/8+R48eZd68ecyePZthw8rRm5CoFI1revHL8A4Mubvy+9Uu6CI09uilf1hTCCGqhnIl6Z07d9KxY0cAfvzxR/z9/Tl58iRff/0106dPr7DgWrduzaJFi5g/fz6NGzfmnXfeYdq0afTrV7X6ZhWl0yasGga9jlOXMzh9OcPa4QghRKUr1z3pjIwMPDy00Zl+//13Hn74YfR6PXfccQcnT56s0AAffPBBHnzwwQrdp7BPHi6ONA32Yuepq2w8dpHHqoVYOyQhhKhU5SpJ16tXj8WLF3P69GlWrlzJ/fdr43kmJSXh6Vl1hggTtic6v/a4XPIWQtwOylWSHjduHE8++SQvvfQS9957L+3atQO0UnXz5s0rNEAhimpftzoz/jjKkj3nWL4/oXD43vzlBS0KC18X397DxYHIIE8aB3nRuKYXjWt6ElbdHUMFDa0phBAVqVxJuk+fPnTo0IGEhASaNm1qmd+pUyd69+59gy2FuDktQr0JqebKqcsZ1/RjXhqpWXlsPn7ZMp42gNHRQKMgTxoHeRJZ04vGQV6E+7uXagxtIYSoTOXqzKSoM2fOANhsRyFVqe9uocnJM3MpXeuwRpc/7m/B2A46yz8lL0tKzWb/2WT+OpdieczM/dvwjYCTg56IAA8ig7TSdtswX+r5yQhIQtyu7KozE7PZzLvvvstHH31EWloaAB4eHrzyyiu88cYb6PVSAhGVx8lBT6BX+UZy8nV3pmGgJ4/mvzaZFfEX09h/Vkva+88l89fZFFKz89h7Jpm9RfpCH/dgI56ODqugsxBCiH9WriT9xhtv8L///Y8PPviA6OhoADZs2MD48ePJysrivffeq9AghagsBr2Oen4e1PPzoFfzmgCYzYrTVzK0xH0umR0nr7A1/jITfj1AUmo2r3VpUKrxuIUQ4maV63J3UFAQs2bNsox+VeCXX35h6NChnD17tsICvFlyuVvcLKUUn689xocrtd7vHm5Rk0mPNJF71kLcRqyVS8r1LXP58mUiIiKumR8REcHly5dL2EII+6XT6Rh2Tz0m92mCQa/j551nefbr7WTk5Fk7NCFEFVeuJN20aVM+/fTTa+Z/+umnNGnS5KaDEsIW9W1Viy+faomLo561cRd4YvZmLqVl//OGQghRTuW6Jz158mS6d+/OqlWrLG2kN23axOnTp1m2bFmFBiiELbk3wp95z97BoLnb2HMmmT6zNvH1oDbUqlZxo78JIUSBcpWk77rrLg4fPkzv3r25evUqV69e5eGHH+avv/7im2++qegYhbApLUJ8+PH59tT0NhJ/MZ2HZ27kr3PJ1g5LCFEF3XQ76aL27NlDixYtMJmubXdqLVJxTFSW8ylZDPhqK4cSU3F3dmB2/5a0z++2VAhRtdhVxTEhBPh7urDwuXa0DatGWnYeA+Zs5dc956wdlhCiCpEkLcRN8DI68n+D2vBAVAC5JsWIBbuYExtv7bCEEFWEJGkhbpKLo4EZT7TgqXahKAUTfj3ApBWHqMA7SUKI21SZanc//PDDN1x+9erVm4lFCLtl0OuY8FAk/p4ufLgyjplrj3E+JUs6PRFC3JQyJWkvL69/XP7UU0/dVEBC2KuCTk9qeDgz5ud9/LzzLJfTc/jk8eZ4GR2tHZ4Qwg6VKUnPmTOnsuIQosro26oW1d2dGPrdTtbGXaDrtPVM7tOEjuE1rB2aEMLOyHU4ISrBvRH+fP9cO8Kqu5GQnEX//21l3C/7pStRIUSZSJIWopI0CfZm6YgOPNUuFICvN53kgU/+ZMfJK1aOTAhhL+wqSX/wwQfodDpGjhxp7VCEKBVXJwfe7tmYb55pQ4CnCycuZfDorI1MXnGInDyztcMTQtg4u0nS27Zt44svvpABPIRd6hheg5Uv3cnDzWtiVvD52mP0/CyWgwkp1g5NCGHD7CJJp6Wl0a9fP7788kt8fHysHY4Q5eJldGTqY82Y9a8WVHNz4mBCCg99uoGZa49hMkubaiHEtewiSQ8bNozu3bvTuXNna4cixE3r2jiQlSPvpHNDf3JNikkrDtH3i02cuJhu7dCEEDbG5pP0ggUL2LlzJxMnTizV+tnZ2aSkpFim1NTUSo5QiLKr4eHMl0+15MM+TXB3dmDHySt0++RPvtl8UnoqE0JY2HSSPn36NC+++CLfffcdLi4updpm4sSJeHl5WaZGjRpVcpRClI9Op+PRVrVYMbIj7er4kplrYuzi/QyYs43E5CxrhyeEsAEVOlRlRVu8eDG9e/fGYDBY5plMJnQ6HXq9nuzs7GLLQCtJZ2dnW16fPXuWRo0ayVCVwqaZzYq5G08wacUhsvPMeLo4MHdQG1qESB0MIWyBDFVZgk6dOrFv3z52795tmVq1akW/fv3YvXv3NQkawNnZGU9PT8vk4eFhhciFKBu9XsegDmEsHdGRpsFepGTl8cK8XSRn5lo7NCGEFdl0kvbw8KBx48bFJjc3N3x9fWncuLG1wxOiwtXzc+fbf7cl1NeVs1cz+c/P++QetRC3MZtO0kLcjjxcHJn+eHMc9DqW7ktg4bbT1g5JCGElZRpgwxasXbvW2iEIUema1vJmVJcGTFx+iPG//kWr2j7U85NbN0LcbqQkLYSNerZjHTqGVycr18wL83eTlWuydkhCiFtMkrQQNkqv1/HRo03xze+d7IPlh6wdkhDiFpMkLYQN8/N0YcqjTQGYu/EEqw6ct3JEQohbSZK0EDbungg/nukQBsCoH/dwPkU6OhHidiFJWgg78FrXBkQGeXIlI5eRC3bLgBxC3CYkSQthB5wdDMx4ojmuTgY2Hb/ErHXHrB2SEOIWkCQthJ2oU8OdCQ9FAjA15jA7T12xckRCiMomSVoIO9KnZTAPNQ3CZFaMmL+LlCzpNlSIqkyStBB2RKfT8W7vxtSqZuTMFek2VIiqTpK0EHbGs0i3ob/tTeCHHWesHZIQopJIkhbCDjUP8eHl++sD8NYvf3HsQpqVIxJCVAZJ0kLYqefvrEt0PV8yc028MG8X2XnSbagQVY0kaSHslF6vY2rfZvi4OnIgIYVJy+OsHZIQooJJkhbCjvkX6Tb0q9h4/jgk3YYKUZVIkhbCznVq6M/A9rUBePWHvdJtqBBViCRpIaqA17tF0DDQk8vpOTw6axNHk6QimRBVgSRpIaoAF0cDs/7VglrVjJy6nMHDn8ey8ehFa4clhLhJkqSFqCJCfd1YPDSaFiHepGTl8dRXW/l++2lrhyWEuAmSpIWoQnzdnZn37B082CSQPLPitR/3MnnFIcwyapYQdkmStBBVjIujgemPN+eFe+sB8PnaY7wwfxdZudKOWgh7I0laiCpIr9fxyv0NmPJoUxwNOpbuS+CJLzdzMS27Qvafnp3H/zbE0+2TP3l/2UFy8swVsl8hRHE2naQnTpxI69at8fDwwM/Pj169ehEXJx02CFFafVoG8/WgtngZHdl16iq9PovlyPnUcu/vQmo2H648RLuJq3nntwMcTEhh9vrjPDZ7E2evZlZg5EIIsPEkvW7dOoYNG8bmzZuJiYkhNzeX+++/n/T0dGuHJoTdaFfXl5+HtifU15UzVzJ5+PONbDhStprfxy+kMebnfURP+oPP1hwjJSuPsOpujOgUjqeLA7tOXaX79D9ZG5dUSWchxO1Jp+xonLsLFy7g5+fHunXruPPOO0u1zZkzZ6hVqxanT58mODi4kiMUwnZdTs/huW+2s+3EFQx6He/2aswTbUJuuM2uU1f4Yt1xVh5IpOCbolktb56/qy73NfLHoNdx+nIGQ7/byb6zyeh0MPyeeozsXB+DXncLzkqIW8NaucThlh2pAiQnJwNQrVq1666TnZ1NdnbhfbfU1PJf2hOiKqnm5sS3/27Laz/u5Zfd5xjz8z5OXExndNcI9EUSqtmsWBOXxBfrj7M1/rJlfqcIP567qy6ta/ug0xWuX6uaKz8Oace7vx3km80nmfHHUXacvMInjzenhofzLT1HIaoauylJm81mHnroIa5evcqGDRuuu9748eOZMGHCNfOlJC2ERinFtFVH+GT1EQC6Rgbw8WPNMOh1/LL7LLPXH+dIfo9ljgYdvZrVZPCddQj39/jHff+y+yxjft5HRo4JPw9nPn2yBW3Crv+jWgh7Ya2StN0k6SFDhrB8+XI2bNhwwzfo7yXps2fP0qhRI0nSQvzNol1nGP3jPnJMZiICPLiakUtifr/f7s4O9GsbwtPRYQR4uZRpv0eTUhny7U6OJKVh0OsY1aUBz91Zp1jpWwh7I0n6BoYPH84vv/zC+vXrCQsLK9O2ck9aiOvbGn+Z577ZzpWMXAD8PJwZ1CGMJ9uG4OniWO79ZuTk8cai/SzadRaAzg39+ejRpni5ln+fQliTJOkSKKV44YUXWLRoEWvXriU8PLzM+5AkLcSNnbiYzqx1x2gR4kPP5kE4OxgqZL9KKeZvPc34JX+RYzIT7GNkZr+WRAV7Vcj+hbiVJEmXYOjQocybN49ffvmFBg0aWOZ7eXlhNBpLtQ9J0kJY1/6zyQz5bgenL2fiZNAztkcj/tU2RC5/C7tirVxi0+2kZ86cSXJyMnfffTeBgYGWaeHChdYOTQhRSo1revHb8I7c18ifHJOZsYv3M3LhbtKy86wdmhA2z6abYNlwIV8IUQZero7M7t+SL/88zqQVcfyy+xzL9yUSWdOTliE+tAzVJj/PslVSE6Kqs+kkLYSoOnQ6HYPvrEvzEB9e+X4Ppy5nsOvUVXadusp/N8QDEOxjpEWRpB0R4IGDwaYv+AlRqSRJCyFuqda1q7Fu1N2cupzBjpNX2HnqCjtOXiUuMYUzVzI5cyWTJXvOAWB0NNCsljctQr1pGepDixAfvF2drHwGQtw6kqSFELecTqcj1NeNUF83Hm6hVcJJzcplz+nk/KStJe/UrDw2Hb/EpuOXLNs2CvTkgagAujYOpJ6fu7VOQYhbQpK0EMImeLg40iG8Oh3CqwNa96RHL6Sx86SWtHecusLxC+kcSEjhQEIKU34/TH1/d7o1DuSBqEDq+7tLjXFR5dh0E6yKIE2whKg6LqVls/pQEsv3JbDh6EVyTYVfX3VquNGtcQDdGgcSGeQpCVtUKGknXUkkSQtRNSVn5rL64HmW7Utk/ZEL5OSZLctCqrnSLSqABxoH0iTYSxK2uGmSpCuJJGkhqr7UrFz+OJTEiv2JrIlLIiu3MGHX9DbStXEA9zXyp1ktb1wcK6ZHNXF7kSRdSSRJC3F7ycjJY23cBZbtS+CPQ0lk5Jgsy5wMeprW8qJ17Wq0DqtGy1Cfm+qjXNw+ZDxpIYSoAK5ODjwQpVUmy8o1sf7wBZbvT2TD0YtcSM1m24krbDtxBdYeQ6+DhoGetK5djbZh1WhVu5qMgS1siiRpIUSV5eJo4P7IAO6PDEApxclLGWw9cZmt8ZfZduIyJy9l8Ne5FP46l8LcjScAqFPdjda1q9EmTJuCfYxyT1tYjSRpIcRtQafTUbu6G7Wru9G3VS0AzqdkWRL21vjLxJ1P5fjFdI5fTGfh9tMA1PZ1ZfCddenTMhgnB+n9TNxack9aCCHyJWfksv2klrC3nrjMvjPJ5Jm1r8ggLxeG3FOPvq2CK2w4T2E/pOJYJZEkLYQor/TsPBZuO82sdcdISs0GIMDThefvqsPjbUKkpvhtRIaqFEIIG+Pm7MCgDmGsf+0e3u4ZSaCXC4kpWYz/9QAdJ6/hv38eJ7NI7XEhKpokaSGE+AcujgaealebtaPu5r3ejanpbeRCajbvLj1Ix8l/8MW6Y6TL+NiiEkiSFkKIUnJ2MNCvbShrXr2bSY9EUauakYtpOUxcfogOk/7gszVHSc3KtXaYogqRJC2EEGXk5KDnsdYh/PHK3XzYpwm1fV25kpHLhyvj6DBpDdNXHyE5U5K1uHlScUwIIW5SnsnMb3sTmPHHEY5dSAdApwN3JwdcnQ24OTvg5uSAq5MBd2cHXJ0dcHc24OrkkL8sfx1nA0ZHA3lmhcmsyDUp8kxm8sxFHi3LzJZ1TGatG1R/TxeCfYwE+7gS7GPEy+gobbwriPQ4JoQQdsrBoKdX85r0aBrEsn1asj58Po3U7DxSs/OAbKvE5e7skJ+0jdT0LkzeBY/erpLEbZ0kaSGEqCAGvY4eTYN4sEkgF9KySc82kZ6dp005eYWvc0xF5uWRkW0iLTuPjBwTWbkmDHodDgYdDno9DkWfG3QY9Doc9XoMBh2Oeh0OBm0ds1IkJGdx5komZ65kcjEtm7TsPA4lpnIoMbXEeN2cDNT0MeLv6YKfhwv+ns74eTjj51nw3IUaHs7S1MyK7CJJf/bZZ3z44YckJibStGlTZsyYQZs2bawdlhBClEin0+Hn4QIe1oshM8fE2auZnLmSkf9YMGVw5komF1KzSc8xcfh8GofPp91wX15GR/w8nPOTuZbEa3g4o5QiK9dEZq6JzBwzmbnaj4zCeUWe55rIyjWTlWPC2VGPp4sjHi4OeBod8XRxxNPokP+YP/9v8zxdHHFx1Ft+lDga9Bj0Vf8qgM0n6YULF/Lyyy8za9Ys2rZty7Rp0+jSpQtxcXH4+flZOzwhhLBJRicD9fzcqefnXuLyrFwT5/KTd1JqNudTsriQ/5hU5DEnz0xyZi7JmbkcSbpxMi+t1Gy4mJZz0/vR67RbDQVXFBzzrzg4OmhXGwquQDgadCx8rp1dXhGw+SQ9depUnn32WZ5++mkAZs2axdKlS/nqq694/fXXrRydEELYJxdHA3VquFOnRslJHEApRUpmHudTs0hKKZ7AL6ZlY9DrMDoacHE0YHQy4OJgwOikLzav4LmLo/bc2VFPTp6ZlMxcUrLySMnMJTWr8HlKVi6pWXmkZOWSkplX+Doz19JFawGzgpw8M1q6v3GnMvZ6692mk3ROTg47duxgzJgxlnl6vZ7OnTuzadMmK0YmhBBVn06nw8vVES9XR+r7W/HaPdoPBlN+7fYck5m8/Jrvufk133NNijyzmdw8Ra752uWOevtscWzTSfrixYuYTCb8/f2Lzff39+fQoUMlbpOdnU12dmFNytTUkitMCCGEsB86XX4FOgN2edm6vOzzp8UNTJw4ES8vL8vUqFEja4ckhBBClItNJ+nq1atjMBg4f/58sfnnz58nICCgxG3GjBlDcnKyZTpw4MCtCFUIIYSocDadpJ2cnGjZsiWrV6+2zDObzaxevZp27dqVuI2zszOenp6WycPDuvdRhBBCiPKy6XvSAC+//DIDBgygVatWtGnThmnTppGenm6p7S2EEEJUVTafpB977DEuXLjAuHHjSExMpFmzZqxYseKaymRCCCFEVWPzSRpg+PDhDB8+vFzbmvM7nk9ISKjIkIQQQtxGCnJIQU65VewiSd+Mgkpn0o2oEEKIm3X+/HlCQkJu2fGq/FCVeXl57Nq1C39/f/R20Jg9NTWVRo0aceDAgSpT6a2qnVNVOx+oeuck52P77O2czGYz58+fp3nz5jg43LrybZVP0vYmJSUFLy8vkpOT8fT0tHY4FaKqnVNVOx+oeuck52P7quI5VQbbL1oKIYQQtylJ0kIIIYSNkiRtY5ydnXnrrbdwdna2digVpqqdU1U7H6h65yTnY/uq4jlVBrknLYQQQtgoKUkLIYQQNkqStBBCCGGjJEkLIYQQNkqStI2YOHEirVu3xsPDAz8/P3r16kVcXJy1w6owH3zwATqdjpEjR1o7lJty9uxZ/vWvf+Hr64vRaCQqKort27dbO6xyMZlMjB07lrCwMIxGI3Xr1uWdd97BnqqprF+/nh49ehAUFIROp2Px4sXFliulGDduHIGBgRiNRjp37syRI0esE2wp3Oh8cnNzGT16NFFRUbi5uREUFMRTTz3FuXPnrBdwKfzTZ1TU888/j06nY9q0abcsPlsnSdpGrFu3jmHDhrF582ZiYmLIzc3l/vvvJz093dqh3bRt27bxxRdf0KRJE2uHclOuXLlCdHQ0jo6OLF++nAMHDvDRRx/h4+Nj7dDKZdKkScycOZNPP/2UgwcPMmnSJCZPnsyMGTOsHVqppaen07RpUz777LMSl0+ePJnp06cza9YstmzZgpubG126dCErK+sWR1o6NzqfjIwMdu7cydixY9m5cyc///wzcXFxPPTQQ1aItPT+6TMqsGjRIjZv3kxQUNAtisxOKGGTkpKSFKDWrVtn7VBuSmpqqgoPD1cxMTHqrrvuUi+++KK1Qyq30aNHqw4dOlg7jArTvXt3NWjQoGLzHn74YdWvXz8rRXRzALVo0SLLa7PZrAICAtSHH35omXf16lXl7Oys5s+fb4UIy+bv51OSrVu3KkCdPHny1gR1k653TmfOnFE1a9ZU+/fvV6Ghoerjjz++5bHZKilJ26jk5GQAqlWrZuVIbs6wYcPo3r07nTt3tnYoN23JkiW0atWKRx99FD8/P5o3b86XX35p7bDKrX379qxevZrDhw8DsGfPHjZs2EC3bt2sHFnFiI+PJzExsdjfnpeXF23btmXTpk1WjKziJCcno9Pp8Pb2tnYo5WY2m+nfvz+jRo0iMjLS2uHYnCo/CpY9MpvNjBw5kujoaBo3bmztcMptwYIF7Ny5k23btlk7lApx/PhxZs6cycsvv8x//vMftm3bxogRI3BycmLAgAHWDq/MXn/9dVJSUoiIiMBgMGAymXjvvffo16+ftUOrEImJiQDXjD3v7+9vWWbPsrKyGD16NE888YRd9309adIkHBwcGDFihLVDsUmSpG3QsGHD2L9/Pxs2bLB2KOV2+vRpXnzxRWJiYnBxcbF2OBXCbDbTqlUr3n//fQCaN2/O/v37mTVrll0m6e+//57vvvuOefPmERkZye7duxk5ciRBQUF2eT63k9zcXPr27YtSipkzZ1o7nHLbsWMHn3zyCTt37kSn01k7HJskl7ttzPDhw/ntt99Ys2YNwcHB1g6n3Hbs2EFSUhItWrTAwcEBBwcH1q1bx/Tp03FwcMBkMlk7xDILDAykUaNGxeY1bNiQU6dOWSmimzNq1Chef/11Hn/8caKioujfvz8vvfQSEydOtHZoFSIgIAAoHFO+wPnz5y3L7FFBgj558iQxMTF2XYr+888/SUpKIiQkxPI9cfLkSV555RVq165t7fBsgpSkbYRSihdeeIFFixaxdu1awsLCrB3STenUqRP79u0rNu/pp58mIiKC0aNHYzAYrBRZ+UVHR1/TLO7w4cOEhoZaKaKbk5GRcc0Y6waDAbPZbKWIKlZYWBgBAQGsXr2aZs2aAdrwiFu2bGHIkCHWDa6cChL0kSNHWLNmDb6+vtYO6ab079//mvoqXbp0oX///jz99NNWisq2SJK2EcOGDWPevHn88ssveHh4WO6ZeXl5YTQarRxd2Xl4eFxzP93NzQ1fX1+7vc/+0ksv0b59e95//3369u3L1q1bmT17NrNnz7Z2aOXSo0cP3nvvPUJCQoiMjGTXrl1MnTqVQYMGWTu0UktLS+Po0aOW1/Hx8ezevZtq1aoREhLCyJEjeffddwkPDycsLIyxY8cSFBREr169rBf0DdzofAIDA+nTpw87d+7kt99+w2QyWb4nqlWrhpOTk7XCvqF/+oz+/kPD0dGRgIAAGjRocKtDtU3Wrl4uNECJ05w5c6wdWoWx9yZYSin166+/qsaNGytnZ2cVERGhZs+ebe2Qyi0lJUW9+OKLKiQkRLm4uKg6deqoN954Q2VnZ1s7tFJbs2ZNif9vBgwYoJTSmmGNHTtW+fv7K2dnZ9WpUycVFxdn3aBv4EbnEx8ff93viTVr1lg79Ov6p8/o76QJVnEyCpYQQghho6TimBBCCGGjJEkLIYQQNkqStBBCCGGjJEkLIYQQNkqStBBCCGGjJEkLIYQQNkqStBBCCGGjJEkLIYQQNkqStBCi1HQ6HYsXL7Z2GELcNiRJC2EnBg4ciE6nu2bq2rWrtUMTQlQSGWBDCDvStWtX5syZU2yes7OzlaIRQlQ2KUkLYUecnZ0JCAgoNvn4+ADapeiZM2fSrVs3jEYjderU4ccffyy2/b59+7j33nsxGo34+voyePBg0tLSiq3z1VdfERkZibOzM4GBgQwfPrzY8osXL9K7d29cXV0JDw9nyZIllmVXrlyhX79+1KhRA6PRSHh4+DU/KoQQpSdJWogqZOzYsTzyyCPs2bOHfv368fjjj3Pw4EEA0tPT6dKlCz4+Pmzbto0ffviBVatWFUvCM2fOZNiwYQwePJh9+/axZMkS6tWrV+wYEyZMoG/fvuzdu5cHHniAfv36cfnyZcvxDxw4wPLlyzl48CAzZ86kevXqt+4NEKKqsfYwXEKI0hkwYIAyGAzKzc2t2PTee+8ppbThTp9//vli27Rt21YNGTJEKaXU7NmzlY+Pj0pLS7MsX7p0qdLr9SoxMVEppVRQUJB64403rhsDoN58803L67S0NAWo5cuXK6WU6tGjh3r66acr5oSFEEruSQthR+655x5mzpxZbF61atUsz9u1a1dsWbt27di9ezcABw8epGnTpri5uVmWR0dHYzabiYuLQ6fTce7cOTp16nTDGJo0aWJ57ubmhqenJ0lJSQAMGTKERx55hJ07d3L//ffTq1cv2rdvX65zFUJIxTEh7Iqbm9s1l58ritFoLNV6jo6OxV7rdDrMZjMA3bp14+TJkyxbtoyYmBg6derEsGHDmDJlSoXHK8TtQO5JC1GFbN68+ZrXDRs2BKBhw4bs2bOH9PR0y/LY2Fj0ej0NGjTAw8OD2rVrs3r16puKoUaNGgwYMIBvv/2WadOmMXv27JvanxC3MylJC2FHsrOzSUxMLDbPwcHBUjnrhx9+oFWrVnTo0IHvvvuOrVu38r///Q+Afv368dZbbzFgwADGjx/PhQsXeOGFF+jfvz/+/v4AjB8/nueffx4/Pz+6detGamoqsbGxvPDCC6WKb9y4cbRs2ZLIyEiys7P57bffLD8ShBBlJ0laCDuyYsUKAgMDi81r0KABhw4dArSa1wsWLGDo0KEEBgYyf/58GjVqBICrqysrV67kxRdfpHXr1ri6uvLII48wdepUy74GDBhAVlYWH3/8Ma+++irVq1enT58+pY7PycmJMWPGcOLECYxGIx07dmTBggUVcOZC3J50Sill7SCEEDdPp9OxaNEievXqZe1QhBAVRO5JCyGEEDZKkrQQQghho+SetBBVhNy5EqLqkZK0EEIIYaMkSQshhBA2SpK0EEIIYaMkSQshhBA2SpK0EEIIYaMkSQshhBA2SpK0EEIIYaMkSQshhBA2SpK0EEIIYaP+H7rihofIxFkmAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from previous_chapters import plot_losses\n",
    "\n",
    "epochs_tensor = torch.linspace(1, n_epochs, len(train_losses))\n",
    "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)\n",
    "# plt.tight_layout(); plt.savefig(\"3.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c16fa614-67e1-4254-8b7e-c3e2f690c29c",
   "metadata": {},
   "source": [
    "- Note that the model is overfitting here because the dataset is kept very small for educational purposes (so that the code can be executed on a laptop computer)\n",
    "- For a longer pretraining run on a much larger dataset, see [../../ch05/03_bonus_pretraining_on_gutenberg](../../ch05/03_bonus_pretraining_on_gutenberg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}