{ "cells": [ { "cell_type": "markdown", "id": "9a5936bd-af17-4a7e-a4d2-e910411708ea", "metadata": {}, "source": [ "\n", "Supplementary code for \"Build a Large Language Model From Scratch\": https://www.manning.com/books/build-a-large-language-model-from-scratch by Sebastian Raschka
\n", "Code repository: https://github.com/rasbt/LLMs-from-scratch\n", "" ] }, { "cell_type": "markdown", "id": "af53bcb1-ff9d-49c7-a0bc-5b8d32ff975b", "metadata": {}, "source": [ "## Appendix D: Adding Bells and Whistles to the Training Loop" ] }, { "cell_type": "markdown", "id": "4f58c142-9434-49af-b33a-356b80a45b86", "metadata": {}, "source": [ "- In this appendix, we add a few more advanced features to the training function, which are used in typical pretraining and finetuning; finetuning is covered in chapters 6 and 7\n", "- The next three sections below discuss learning rate warmup, cosine decay, and gradient clipping\n", "- The final section adds these techniques to the training function" ] }, { "cell_type": "markdown", "id": "744def4f-c03f-42ee-97bb-5d7d5b89b723", "metadata": {}, "source": [ "- We start by initializing a model reusing the code from chapter 5:" ] }, { "cell_type": "code", "execution_count": 1, "id": "8755bd5e-bc06-4e6e-9e63-c7c82b816cbe", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "torch version: 2.2.1\n" ] } ], "source": [ "from importlib.metadata import version\n", "import torch\n", "\n", "print(\"torch version:\", version(\"torch\"))\n", "\n", "\n", "from previous_chapters import GPTModel\n", "\n", "GPT_CONFIG_124M = {\n", " \"vocab_size\": 50257, # Vocabulary size\n", " \"context_length\": 256, # Shortened context length (orig: 1024)\n", " \"emb_dim\": 768, # Embedding dimension\n", " \"n_heads\": 12, # Number of attention heads\n", " \"n_layers\": 12, # Number of layers\n", " \"drop_rate\": 0.1, # Dropout rate\n", " \"qkv_bias\": False # Query-key-value bias\n", "}\n", "\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "torch.manual_seed(123)\n", "model = GPTModel(GPT_CONFIG_124M)\n", "model.eval(); # Disable dropout during inference" ] }, { "cell_type": "markdown", "id": "51574e57-a098-412c-83e8-66dafa5a0b99", "metadata": {}, "source": [ "- Next, using the same code we used in chapter 5, we initialize the data loaders:" ] }, { "cell_type": "code", "execution_count": 2, "id": "386ca110-2bb4-42f1-bd54-8836df80acaa", "metadata": {}, "outputs": [], "source": [ "import os\n", "import urllib.request\n", "\n", "file_path = \"the-verdict.txt\"\n", "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n", "\n", "if not os.path.exists(file_path):\n", " with urllib.request.urlopen(url) as response:\n", " text_data = response.read().decode('utf-8')\n", " with open(file_path, \"w\", encoding=\"utf-8\") as file:\n", " file.write(text_data)\n", "else:\n", " with open(file_path, \"r\", encoding=\"utf-8\") as file:\n", " text_data = file.read()" ] }, { "cell_type": "code", "execution_count": 3, "id": "ae96992b-536a-4684-a924-658b9ffb7e9c", "metadata": {}, "outputs": [], "source": [ "from previous_chapters import create_dataloader_v1\n", "\n", "# Train/validation ratio\n", "train_ratio = 0.90\n", "split_idx = int(train_ratio * len(text_data))\n", "\n", "\n", "torch.manual_seed(123)\n", "\n", "train_loader = create_dataloader_v1(\n", " text_data[:split_idx],\n", " batch_size=2,\n", " max_length=GPT_CONFIG_124M[\"context_length\"],\n", " stride=GPT_CONFIG_124M[\"context_length\"],\n", " drop_last=True,\n", " shuffle=True\n", ")\n", "\n", "val_loader = create_dataloader_v1(\n", " text_data[split_idx:],\n", " batch_size=2,\n", " max_length=GPT_CONFIG_124M[\"context_length\"],\n", " stride=GPT_CONFIG_124M[\"context_length\"],\n", " drop_last=False,\n", " shuffle=False\n", ")" ] }, { "cell_type": "markdown", "id": "939c08d8-257a-41c6-b842-019f7897ac74", "metadata": {}, "source": [ "## D.1 Learning rate warmup" ] }, { "cell_type": "markdown", "id": "7fafcd30-ddf7-4a9f-bcf4-b13c052b3133", "metadata": {}, "source": [ "- When training complex models like LLMs, implementing learning rate warmup can help stabilize the training\n", "- In learning rate warmup, we gradually increase the learning rate from a very low value (`initial_lr`) to a user-specified maximum (`peak_lr`)\n", "- This way, the model will start the training with small weight updates, which helps decrease the risk of large destabilizing updates during the training" ] }, { "cell_type": "code", "execution_count": 4, "id": "2bb4790b-b8b6-4e9e-adf4-704a04b31ddf", "metadata": {}, "outputs": [], "source": [ "n_epochs = 15\n", "initial_lr = 0.0001\n", "peak_lr = 0.01" ] }, { "cell_type": "markdown", "id": "5bf3a8da-abc4-4b80-a5d8-f1cc1c7cc5f3", "metadata": {}, "source": [ "- Typically, the number of warmup steps is between 10% and 20% of the total number of steps\n", "- We can compute the increment as the difference between the `peak_lr` and `initial_lr` divided by the number of warmup steps" ] }, { "cell_type": "code", "execution_count": 5, "id": "e075f80e-a398-4809-be1d-8019e1d31c90", "metadata": {}, "outputs": [], "source": [ "warmup_steps = 20\n", "lr_increment = (peak_lr - initial_lr) / warmup_steps\n", "\n", "global_step = -1\n", "track_lrs = []\n", "\n", "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)\n", "\n", "for epoch in range(n_epochs):\n", " for input_batch, target_batch in train_loader:\n", " optimizer.zero_grad()\n", " global_step += 1\n", " \n", " if global_step < warmup_steps:\n", " lr = initial_lr + global_step * lr_increment\n", " else:\n", " lr = peak_lr\n", " \n", " # Apply the calculated learning rate to the optimizer\n", " for param_group in optimizer.param_groups:\n", " param_group[\"lr\"] = lr\n", " track_lrs.append(optimizer.param_groups[0][\"lr\"])\n", " \n", " # Calculate loss and update weights\n", " # ..." ] }, { "cell_type": "code", "execution_count": 6, "id": "cb6da121-eeed-4023-bdd8-3666c594b4ed", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4PElEQVR4nO3de1zUZb4H8M8Mlxm8MCCsDCjIaOyhgrxAjJh73FZqaikj3TSXo2auVkul0cnyBtXaoTA3F9dSdzvZvjYv625ReYyWsLRiBLloqeElUUwcUHFmEOU285w/iF9NojI48JuBz/v1mpfN8/v+Zr6PGl9/z/P8fo9CCCFAREREbkcpdwJERETUMRZpIiIiN8UiTURE5KZYpImIiNwUizQREZGbYpEmIiJyUyzSREREbopFmoiIyE15y52Ap7Lb7aiursbAgQOhUCjkToeIiGQmhEB9fT3CwsKgVLrmGphFuouqq6sRHh4udxpERORmTp48iaFDh7rks1iku2jgwIEA2v4w/P39Zc6GiIjkZrVaER4eLtUHV2CR7qL2IW5/f38WaSIikrhyCpQLx4iIiNwUizQREZGbYpEmIiJyU7IX6TVr1iAyMhJqtRp6vR7FxcVXjd+6dSuio6OhVqsRGxuL7du3Oxx/9913ceeddyIoKAgKhQJ79+697DMaGxuRlpaGoKAgDBgwAFOmTEFNTY0ru0VERHTdZC3SW7ZsQXp6OjIzM1FWVoaRI0fCYDCgtra2w/jCwkJMnz4dc+bMQXl5OVJSUpCSkoL9+/dLMQ0NDRg/fjxeeeWVK37vU089hQ8//BBbt27Fzp07UV1djcmTJ7u8f0RERNdDIYQQcn25Xq/Hrbfeij//+c8A2h4QEh4ejieeeALPPffcZfHTpk1DQ0MDtm3bJrWNHTsWo0aNwtq1ax1ijx8/Dp1Oh/LycowaNUpqt1gs+NnPfoaNGzfiN7/5DQCgoqICN954I4xGI8aOHdup3K1WKzQaDSwWC1d3ExFRt9QF2W7Bam5uRmlpKRYtWiS1KZVKJCUlwWg0dniO0WhEenq6Q5vBYEBubm6nv7e0tBQtLS1ISkqS2qKjoxEREXHVIt3U1ISmpibpvdVq7fR39iWlJ85j9Y4jaLHZ5U6FiKjTvJVKvP1wgtxpXEa2In327FnYbDaEhIQ4tIeEhKCioqLDc0wmU4fxJpOp099rMpng6+uLgIAApz4nKysLL7zwQqe/p69a9clhfH7krNxpEBE5xddL9iVaHeLDTDpp0aJFDlfx7U+WoR80tdqw53gdACDz3pswqL+vzBkREXWO0k33YJCtSAcHB8PLy+uyVdU1NTXQarUdnqPVap2Kv9JnNDc3w2w2O1xNX+tzVCoVVCpVp7+nL9pbZUZjix3BA3zx0LhIbjxCRHSdZLu+9/X1RVxcHAoKCqQ2u92OgoICJCYmdnhOYmKiQzwA5OfnXzG+I3FxcfDx8XH4nEOHDqGqqsqpz6HLFX57DgCQOCKYBZqIyAVkHe5OT0/HrFmzEB8fj4SEBKxatQoNDQ2YPXs2AGDmzJkYMmQIsrKyAADz58/HhAkTsHLlSiQnJ2Pz5s0oKSnB+vXrpc+sq6tDVVUVqqurAbQVYKDtClqr1UKj0WDOnDlIT0/HoEGD4O/vjyeeeAKJiYmdXtlNHTMe+75IDw+SORMiot5B1iI9bdo0nDlzBhkZGTCZTBg1ahTy8vKkxWFVVVUOe3KOGzcOGzduxNKlS7F48WJERUUhNzcXMTExUswHH3wgFXkAePDBBwEAmZmZeP755wEAr732GpRKJaZMmYKmpiYYDAa8/vrrPdDj3utSsw3lVecBAONGsEgTEbmCrPdJezLeJ+3o8yNnMOPNYoRp1PjyuV9xuJuI+pzuqAvuueacPI6R89FERC7HIk0u8cOiMQ51ExG5Cos0Xbf6xhZ8fcoCgEWaiMiVWKTpuu05XgebXSAyqB+GBPjJnQ4RUa/BIk3XrfAoh7qJiLoDizRdtx8/xISIiFyHRZquy/mGZhw83bYjGB9iQkTkWizSdF2KKtuuon8eMgA/G8hnmxMRuRKLNF0XaaibV9FERC7HIk3XhfPRRETdh0Wauqy2vhFHay9AoQDGDh8kdzpERL0OizR1WfujQG8K9UdAP1+ZsyEi6n1YpKnL2os0d70iIuoeLNLUZYVSkeZ8NBFRd2CRpi757vxFVNVdhJdSgVt1nI8mIuoOLNLUJe1D3bcM1WCAylvmbIiIeicWaeoSzkcTEXU/FmlymhACxmOcjyYi6m4s0uS04+cu4rSlEb5eSsQNC5Q7HSKiXotFmpxW+O1ZAMDoiACofbxkzoaIqPdikSanGXnrFRFRj2CRJqcIIaQinchFY0RE3YpFmpxyuOYCzjU0Q+2jxKjwALnTISLq1VikySnt89G3Rg6Crzf/+hARdSf+lCWncKibiKjnsEhTp9nsArt5fzQRUY9hkaZOO1hthbWxFQNV3ogJ85c7HSKiXo9FmjrNeKxtPlo/fBC8vfhXh4iou/EnLXVa+9aUY4dzPpqIqCewSFOntNjsKK6sA8D5aCKiniJ7kV6zZg0iIyOhVquh1+tRXFx81fitW7ciOjoaarUasbGx2L59u8NxIQQyMjIQGhoKPz8/JCUl4ciRIw4xhw8fxn333Yfg4GD4+/tj/Pjx+PTTT13et97kq+/MuNhsQ2A/H0RrB8qdDhFRnyBrkd6yZQvS09ORmZmJsrIyjBw5EgaDAbW1tR3GFxYWYvr06ZgzZw7Ky8uRkpKClJQU7N+/X4rJzs5GTk4O1q5di6KiIvTv3x8GgwGNjY1SzD333IPW1lbs2LEDpaWlGDlyJO655x6YTKZu77OnMv5oqFupVMicDRFRHyFklJCQINLS0qT3NptNhIWFiaysrA7jp06dKpKTkx3a9Hq9eOSRR4QQQtjtdqHVasWKFSuk42azWahUKrFp0yYhhBBnzpwRAMSuXbukGKvVKgCI/Pz8TudusVgEAGGxWDp9jiebvt4ohj27TfytsFLuVIiI3FJ31AXZrqSbm5tRWlqKpKQkqU2pVCIpKQlGo7HDc4xGo0M8ABgMBim+srISJpPJIUaj0UCv10sxQUFB+I//+A/87W9/Q0NDA1pbW7Fu3ToMHjwYcXFxru5mr9DYYkPJifMAgETORxMR9Rhvub747NmzsNlsCAkJcWgPCQlBRUVFh+eYTKYO49uHqdt/vVqMQqHAJ598gpSUFAwcOBBKpRKDBw9GXl4eAgOvvDdyU1MTmpqapPdWq7WTPfV85VVmNLfaMXigCiN+1l/udIiI+gzZF471NCEE0tLSMHjwYHz++ecoLi5GSkoK7r33Xpw+ffqK52VlZUGj0Uiv8PDwHsxaXsbvn9edOCIICgXno4mIeopsRTo4OBheXl6oqalxaK+pqYFWq+3wHK1We9X49l+vFrNjxw5s27YNmzdvxm233YYxY8bg9ddfh5+fH95+++0r5rto0SJYLBbpdfLkSec67MEKpf2jeX80EVFPkq1I+/r6Ii4uDgUFBVKb3W5HQUEBEhMTOzwnMTHRIR4A8vPzpXidTgetVusQY7VaUVRUJMVcvHgRQNv8948plUrY7fYr5qtSqeDv7+/w6gsuNrdi70kzAN4fTUTU02SbkwaA9PR0zJo1C/Hx8UhISMCqVavQ0NCA2bNnAwBmzpyJIUOGICsrCwAwf/58TJgwAStXrkRycjI2b96MkpISrF+/HkDbfPOCBQuwfPlyREVFQafTYdmyZQgLC0NKSgqAtkIfGBiIWbNmISMjA35+fvjLX/6CyspKJCcny/L74M72HD+PVrvAkAA/hA/qJ3c6RER9iqxFetq0aThz5gwyMjJgMpkwatQo5OXlSQu/qqqqHK54x40bh40bN2Lp0qVYvHgxoqKikJubi5iYGClm4cKFaGhowLx582A2mzF+/Hjk5eVBrVYDaBtmz8vLw5IlS/CrX/0KLS0tuPnmm/H+++9j5MiRPfsb4AHa94/mUDcRUc9TCCGE3El4IqvVCo1GA4vF0quHvif9+Qt89Z0Fr00biftHD5U7HSIit9UddaHPre6mzrNcasH+UxYAQOJwzkcTEfU0Fmm6ouLKOtgFMDy4P7QatdzpEBH1OSzSdEWFP7o/moiIeh6LNF2RUbo/mkPdRERyYJGmDp270IQKUz0AYOzwQTJnQ0TUN7FIU4d2H6sDAERrByJogErmbIiI+iYWaeoQ56OJiOTHIk0dMh5rm49OHM4iTUQkFxZpuozJ0ohjZxqgVAB6FmkiItmwSNNljMfahrpjhmig8fORORsior6rS0W6tbUVn3zyCdatW4f6+rYVwNXV1bhw4YJLkyN5tN96xaFuIiJ5Ob3BxokTJ3DXXXehqqoKTU1NuOOOOzBw4EC88soraGpqwtq1a7sjT+pB7ftHc9EYEZG8nL6Snj9/PuLj43H+/Hn4+flJ7ffff/9lez2T5zlZdxHfnb8Eb6UCt0by/mgiIjk5fSX9+eefo7CwEL6+vg7tkZGROHXqlMsSI3m0D3WPCg9Af5WsO5kSEfV5Tl9J2+122Gy2y9q/++47DBw40CVJkXx4fzQRkftwukjfeeedWLVqlfReoVDgwoULyMzMxK9//WtX5kY9TAjB+WgiIjfi9HjmypUrYTAYcNNNN6GxsRG//e1vceTIEQQHB2PTpk3dkSP1kG/PNKC2vgm+3kqMiQiUOx0ioj7P6SI9dOhQ7Nu3D1u2bMG+fftw4cIFzJkzB6mpqQ4LycjztD9lLC4iEGofL5mzISIip4v0rl27MG7cOKSmpiI1NVVqb21txa5du/Cf//mfLk2Qeo7x+/nocRzqJiJyC07PSd9+++2oq6u7rN1iseD22293SVLU8+x28cP+0TewSBMRuQOni7QQAgqF4rL2c+fOoX///i5JinreoZp6nL/Ygn6+XrhlaIDc6RAREZwY7p48eTKAttXcDz30EFSqH/YYttls+OqrrzBu3DjXZ0g9on1V962Rg+DjxUe6ExG5g04XaY1GA6DtSnrgwIEOi8R8fX0xduxYzJ071/UZUo/gfDQRkfvpdJF+6623ALQ9Wey///u/ObTdi7Ta7Cg61rbOYNyIYJmzISKidk6v7s7MzOyOPEhGB6qtqG9qhb/aGzeF+cudDhERfa9LD2f+5z//iX/84x+oqqpCc3Ozw7GysjKXJEY9p30+Wj88CF7KyxcFEhGRPJxeIZSTk4PZs2cjJCQE5eXlSEhIQFBQEI4dO4a77767O3KkblbI+WgiIrfkdJF+/fXXsX79eqxevRq+vr5YuHAh8vPz8eSTT8JisXRHjtSNmlvtKDl+HgCf101E5G6cLtJVVVXSrVZ+fn6or68HAMyYMYPP7vZA+74z41KLDUH9ffHzwdzFjIjInThdpLVarfTEsYiICOzevRsAUFlZCSGEa7Ojbld4tG0+euyIICg5H01E5FacLtK/+tWv8MEHHwAAZs+ejaeeegp33HEHpk2bhvvvv9/lCVL3Mh7jfDQRkbtyukivX78eS5YsAQCkpaXhf//3f3HjjTfixRdfxBtvvOF0AmvWrEFkZCTUajX0ej2Ki4uvGr9161ZER0dDrVYjNjYW27dvdzguhEBGRgZCQ0Ph5+eHpKQkHDly5LLP+b//+z/o9Xr4+fkhMDAQKSkpTufu6RpbbCg7YQYAJA5nkSYicjdOFenW1lYsX74cJpNJanvwwQeRk5ODJ554Ar6+vk59+ZYtW5Ceno7MzEyUlZVh5MiRMBgMqK2t7TC+sLAQ06dPx5w5c1BeXo6UlBSkpKRg//79Ukx2djZycnKwdu1aFBUVoX///jAYDGhsbJRi/vWvf2HGjBmYPXs29u3bhy+//BK//e1vncq9Nyg9cR7NNju0/mrogvlwGiIityOc1L9/f1FZWensaR1KSEgQaWlp0nubzSbCwsJEVlZWh/FTp04VycnJDm16vV488sgjQggh7Ha70Gq1YsWKFdJxs9ksVCqV2LRpkxBCiJaWFjFkyBDx17/+9bpyt1gsAoCwWCzX9Tlyys77Rgx7dpt4anO53KkQEXm87qgLTg93T5w4ETt37rzufxw0NzejtLQUSUlJUptSqURSUhKMRmOH5xiNRod4ADAYDFJ8ZWUlTCaTQ4xGo4Fer5diysrKcOrUKSiVSowePRqhoaG4++67Ha7G+4r2rSnHcj6aiMgtOf3EsbvvvhvPPfccvv76a8TFxV32DO9JkyZ16nPOnj0Lm82GkJAQh/aQkBBUVFR0eI7JZOowvn34vf3Xq8UcO3YMAPD888/jj3/8IyIjI7Fy5Ur88pe/xOHDhzFo0KAOv7upqQlNTU3Se6vV2ql+uqsLTa3Y913bfe1cNEZE5J6cLtK///3vAQB//OMfLzumUChgs9muP6tuZLfbAQBLlizBlClTALRtHjJ06FBs3boVjzzySIfnZWVl4YUXXuixPLvbnso62OwCEYP6YWhgP7nTISKiDjg93G2326/4cqZABwcHw8vLCzU1NQ7tNTU10Gq1HZ6j1WqvGt/+69ViQkNDAQA33XSTdFylUmH48OGoqqq6Yr6LFi2CxWKRXidPnuxMN92W8VjbUDdXdRMRuS+ni7Sr+Pr6Ii4uDgUFBVKb3W5HQUEBEhMTOzwnMTHRIR4A8vPzpXidTgetVusQY7VaUVRUJMXExcVBpVLh0KFDUkxLSwuOHz+OYcOGXTFflUoFf39/h5cnk57XfQOLNBGRu+rSLliukp6ejlmzZiE+Ph4JCQlYtWoVGhoaMHv2bADAzJkzMWTIEGRlZQEA5s+fjwkTJmDlypVITk7G5s2bUVJSgvXr1wNoG25fsGABli9fjqioKOh0OixbtgxhYWHSfdD+/v549NFHkZmZifDwcAwbNgwrVqwAADzwwAM9/5sgA/PFZhyobptT55U0EZH7krVIT5s2DWfOnEFGRgZMJhNGjRqFvLw8aeFXVVUVlMofLvbHjRuHjRs3YunSpVi8eDGioqKQm5uLmJgYKWbhwoVoaGjAvHnzYDabMX78eOTl5UGtVksxK1asgLe3N2bMmIFLly5Br9djx44dCAwM7LnOy2j3sToIAdwweAAG+6uvfQIREclCIQQfuN0VVqsVGo0GFovF44a+n//gADYUHseMscPwh5SYa59ARETX1B11QbY5aZIP948mIvIMTg93X+n+YIVCAZVK5fSjQalnnalvwuGaCwCAsZyPJiJya04X6YCAACgUV97ScOjQoXjooYeQmZnpMJ9M7mH397de3Rjqj8D+/AcVEZE7c7pIb9iwAUuWLMFDDz2EhIQEAEBxcTHefvttLF26FGfOnMGrr74KlUqFxYsXuzxhuj6F3z8KlEPdRETuz+ki/fbbb2PlypWYOnWq1HbvvfciNjYW69atQ0FBASIiIvDSSy+xSLshI+ejiYg8htPj0YWFhRg9evRl7aNHj5Y2sRg/fvxVn95F8qg2X8LxcxehVAC36jp+RjkREbkPp4t0eHg43nzzzcva33zzTYSHhwMAzp0712fuOfYk7btexQ4NgL/aR+ZsiIjoWpwe7n711VfxwAMP4KOPPsKtt94KACgpKUFFRQX++c9/AgD27NmDadOmuTZTum6cjyYi8ixOF+lJkyahoqIC69atw+HDhwG0bV+Zm5uLyMhIAMBjjz3m0iTp+gkhOB9NRORhuvRYUJ1Oh5dfftnVuVA3qqq7iGpLI3y8FIgfxvloIiJP0KUibTabUVxcjNraWml/5nYzZ850SWLkWu1D3aPDA+Hn6yVzNkRE1BlOF+kPP/wQqampuHDhAvz9/R0ebKJQKFik3VR7kU7kUDcRkcdwenX3008/jYcffhgXLlyA2WzG+fPnpVddXV135EjXqW0+mkWaiMjTOF2kT506hSeffBL9+vXrjnyoGxytvYCzF5qg8lZidESA3OkQEVEnOV2kDQYDSkpKuiMX6ibtQ923Rg6Cypvz0UREnsLpOenk5GQ888wzOHjwIGJjY+Hj4/hQjEmTJrksOXKN9q0pOdRNRORZnC7Sc+fOBQC8+OKLlx1TKBSw2WzXnxW5jN0usPtY21oBFmkiIs/idJH+6S1X5N4OnrbCcqkFA1TeuGWIRu50iIjICdzwuZdrX9WdoBsEby/+cRMReZJOXUnn5ORg3rx5UKvVyMnJuWrsk08+6ZLEyDWMx76/9Wo4h7qJiDyNQgghrhWk0+lQUlKCoKAg6HS6K3+YQoFjx465NEF3ZbVaodFoYLFY4O/vL3c6HWqx2THqhX+jodmGbU+MRwyHu4mIuk131IVOXUlXVlZ2+N/k3r4+ZUFDsw0aPx/cFOqe/5AgIqIr4yRlL9Y+Hz12+CAolYprRBMRkbtxenW3zWbDhg0bUFBQ0OEGGzt27HBZcnR9jNL+0cEyZ0JERF3hdJGeP38+NmzYgOTkZMTExDhssEHuo6nVhj3H2+6P5v7RRESeyekivXnzZvzjH//Ar3/96+7Ih1ykvMqMplY7ggeocMPgAXKnQ0REXeD0nLSvry9uuOGG7siFXOjHu15xtIOIyDN1aavKP/3pT+jEnVskox/moznUTUTkqZwe7v7iiy/w6aef4qOPPsLNN9982QYb7777rsuSo6651GxD+cnzAFikiYg8mdNFOiAgAPfff3935EIuUnKiDi02gTCNGhGDuO83EZGncqpIt7a24vbbb8edd94JrVbbXTnRdSqU5qODOR9NROTBnJqT9vb2xqOPPoqmpiaXJrFmzRpERkZCrVZDr9ejuLj4qvFbt25FdHQ01Go1YmNjsX37dofjQghkZGQgNDQUfn5+SEpKwpEjRzr8rKamJowaNQoKhQJ79+51VZdkVcj5aCKiXsHphWMJCQkoLy93WQJbtmxBeno6MjMzUVZWhpEjR8JgMKC2trbD+MLCQkyfPh1z5sxBeXk5UlJSkJKSgv3790sx2dnZyMnJwdq1a1FUVIT+/fvDYDCgsbHxss9buHAhwsLCXNYfuVkbW/D1d2YA3D+aiMjjCSdt2bJFDB8+XKxevVoUFhaKffv2ObyclZCQINLS0qT3NptNhIWFiaysrA7jp06dKpKTkx3a9Hq9eOSRR4QQQtjtdqHVasWKFSuk42azWahUKrFp0yaH87Zv3y6io6PFgQMHBABRXl7e6bwtFosAICwWS6fP6QmfHDSJYc9uExOyd8idChFRn9IddcHphWMPPvggAMctKRUKBYQQUCgUsNlsnf6s5uZmlJaWYtGiRVKbUqlEUlISjEZjh+cYjUakp6c7tBkMBuTm5gJo2wDEZDIhKSlJOq7RaKDX62E0GqX8a2pqMHfuXOTm5qJfv2svrmpqanIY5rdarZ3uZ0/68Xw0ERF5NqeLtCt3wTp79ixsNhtCQkIc2kNCQlBRUdHhOSaTqcN4k8kkHW9vu1KMEAIPPfQQHn30UcTHx+P48ePXzDUrKwsvvPBCp/olJ85HExH1Hk4X6WHDhnVHHj1q9erVqK+vd7iCv5ZFixY5XMFbrVaEh4d3R3pddr6hGd+cbrvCHzucRZqIyNM5XaTbHTx4EFVVVWhubnZonzRpUqc/Izg4GF5eXqipqXFor6mpueItXlqt9qrx7b/W1NQgNDTUIWbUqFEA2nbqMhqNUKlUDp8THx+P1NRUvP3225d9r0qluize3ew+1nYV/fOQAfjZQPfOlYiIrs3pIn3s2DHcf//9+Prrr6W5aADS/bjOzEn7+voiLi4OBQUFSElJAQDY7XYUFBTg8ccf7/CcxMREFBQUYMGCBVJbfn4+EhMTAQA6nQ5arRYFBQVSUbZarSgqKsJjjz0GAMjJycHy5cul86urq2EwGLBlyxbo9fpO5+9uCrk1JRFRr9KlrSp1Oh0KCgqg0+lQXFyMc+fO4emnn8arr77qdALp6emYNWsW4uPjkZCQgFWrVqGhoQGzZ88GAMycORNDhgxBVlaW9P0TJkzAypUrkZycjM2bN6OkpATr168H0PaPhQULFmD58uWIioqCTqfDsmXLEBYWJv1DICIiwiGHAQPadokaMWIEhg4d6nQf3IXx+ytpDnUTEfUOThdpo9GIHTt2IDg4GEqlEkqlEuPHj0dWVhaefPJJp++hnjZtGs6cOYOMjAyYTCaMGjUKeXl50sKvqqoqKJU/3M49btw4bNy4EUuXLsXixYsRFRWF3NxcxMTESDELFy5EQ0MD5s2bB7PZjPHjxyMvLw9qtdrZ7nqMWmsjjtZegEIBjB0+SO50iIjIBRRCOLedVWBgIMrKyqDT6TBixAj89a9/xe23345vv/0WsbGxuHjxYnfl6lasVis0Gg0sFgv8/f3lTgfv7z2F+Zv3ImaIP7Y98Qu50yEi6nO6oy44fSUdExODffv2QafTQa/XIzs7G76+vli/fj2GDx/ukqTIeYVHOR9NRNTbOF2kly5dioaGBgDAiy++iHvuuQe/+MUvEBQUhC1btrg8Qeqc9vnoRM5HExH1Gk4XaYPBIP33DTfcgIqKCtTV1SEwMJA7LsnkZN1FVNVdhJdSgVt1nI8mIuotnN5go93Ro0fx8ccf49KlSxg0iIVBTu1X0SOHajBA1eVb34mIyM04XaTPnTuHiRMn4uc//zl+/etf4/Tp0wCAOXPm4Omnn3Z5gnRtu6XndXOom4ioN3G6SD/11FPw8fFBVVWVw8YU06ZNQ15enkuTo2sTQvAhJkREvZTTY6P//ve/8fHHH1/20I+oqCicOHHCZYlR51SebYDJ2ghfLyXihgXKnQ4REbmQ01fSDQ0NHW7tWFdX5/bPtu6N2uejR0cEQO3jJXM2RETkSk4X6V/84hf429/+Jr1XKBSw2+3Izs7G7bff7tLk6No41E1E1Hs5PdydnZ2NiRMnoqSkBM3NzVi4cCEOHDiAuro6fPnll92RI12B3S6kRWPjbuCiMSKi3sbpK+mYmBgcPnwY48ePx3333YeGhgZMnjwZ5eXlGDFiRHfkSFdwuLYe5xqa4efjhZFDA+ROh4iIXKxLN9VqNBosWbLEoe27777DvHnzpN2oqPsZv7+Kjo8MhK93l295JyIiN+Wyn+znzp3Dm2++6aqPo07gfDQRUe/Gyy8PZbML7D7WXqQ5H01E1BuxSHuog9VW1De2YqDKGzeHyb9VJhERuR6LtIcq/PYsAEA/fBC8vfjHSETUG3V64djkyZOvetxsNl9vLuSEQul53ZyPJiLqrTpdpDUazTWPz5w587oTomtrsdmx53gdAO4fTUTUm3W6SL/11lvdmQc54avvzLjYbENgPx9EawfKnQ4REXUTTmZ6oMKjP2xNqVQqZM6GiIi6C4u0B+J8NBFR38Ai7WEaW2worToPgPPRRES9HYu0hymrOo/mVjsGD1RhxM/6y50OERF1IxZpD2P89oenjCkUnI8mIurNWKQ9jPHbHxaNERFR78Yi7UEamlqx96QZADfVICLqC1ikPcie43VotQsMDfRD+KB+cqdDRETdjEXag/x4PpqIiHo/FmkPYjzG+Wgior6ERdpDWC62YP8pCwAgcTjno4mI+gK3KNJr1qxBZGQk1Go19Ho9iouLrxq/detWREdHQ61WIzY2Ftu3b3c4LoRARkYGQkND4efnh6SkJBw5ckQ6fvz4ccyZMwc6nQ5+fn4YMWIEMjMz0dzc3C39c4WiynOwC2D4z/pDq1HLnQ4REfUA2Yv0li1bkJ6ejszMTJSVlWHkyJEwGAyora3tML6wsBDTp0/HnDlzUF5ejpSUFKSkpGD//v1STHZ2NnJycrB27VoUFRWhf//+MBgMaGxsBABUVFTAbrdj3bp1OHDgAF577TWsXbsWixcv7pE+d4U01M2njBER9R1CZgkJCSItLU16b7PZRFhYmMjKyuowfurUqSI5OdmhTa/Xi0ceeUQIIYTdbhdarVasWLFCOm42m4VKpRKbNm26Yh7Z2dlCp9N1Om+LxSIACIvF0ulzrofhtZ1i2LPbxLZ91T3yfURE5JzuqAuyXkk3NzejtLQUSUlJUptSqURSUhKMRmOH5xiNRod4ADAYDFJ8ZWUlTCaTQ4xGo4Fer7/iZwKAxWLBoEGDrqc73ebshSZUmOoBAGOHu2eORETkep3eT7o7nD17FjabDSEhIQ7tISEhqKio6PAck8nUYbzJZJKOt7ddKeanjh49itWrV+PVV1+9Yq5NTU1oamqS3lut1ivGutru74e6o7UDETRA1WPfS0RE8pJ9Tlpup06dwl133YUHHngAc+fOvWJcVlYWNBqN9AoPD++xHPkoUCKivknWIh0cHAwvLy/U1NQ4tNfU1ECr1XZ4jlarvWp8+6+d+czq6mrcfvvtGDduHNavX3/VXBctWgSLxSK9Tp48ee0OusgPDzHhrVdERH2JrEXa19cXcXFxKCgokNrsdjsKCgqQmJjY4TmJiYkO8QCQn58vxet0Omi1WocYq9WKoqIih888deoUfvnLXyIuLg5vvfUWlMqr/1aoVCr4+/s7vHqCydKIY2cboFQACTrORxMR9SWyzkkDQHp6OmbNmoX4+HgkJCRg1apVaGhowOzZswEAM2fOxJAhQ5CVlQUAmD9/PiZMmICVK1ciOTkZmzdvRklJiXQlrFAosGDBAixfvhxRUVHQ6XRYtmwZwsLCkJKSAuCHAj1s2DC8+uqrOHPmjJTPla7g5WI8dhYAEDNEA42fj8zZEBFRT5K9SE+bNg1nzpxBRkYGTCYTRo0ahby8PGnhV1VVlcNV7rhx47Bx40YsXboUixcvRlRUFHJzcxETEyPFLFy4EA0NDZg3bx7MZjPGjx+PvLw8qNVtDwHJz8/H0aNHcfToUQwdOtQhHyFED/S68wqPcj6aiKivUgh3q0oewmq1QqPRwGKxdOvQ920v78Ap8yW8/XACJvz8Z932PUREdH26oy70+dXd7uxk3UWcMl+Ct1KB+GGBcqdDREQ9jEXajRV+2zYfPSo8AP1Vss9MEBFRD2ORdmOF3D+aiKhPY5F2U0IIqUiPZZEmIuqTWKTd1LdnGnCmvgm+3kqMieB8NBFRX8Qi7aaM389Hxw8LhNrHS+ZsiIhIDizSborz0URExCLthux2Ie18xYeYEBH1XSzSbqjCVI/zF1vQz9cLtwwNkDsdIiKSCYu0G2q/PzpBNwg+XvwjIiLqq1gB3JC0f/RwDnUTEfVlLNJuptVmR3FlHQDuH01E1NexSLuZ/dVW1De1wl/tjZvCembPaiIick8s0m6mfT567PAgeCkVMmdDRERyYpF2M9J8NG+9IiLq81ik3Uhzqx17jnM+moiI2rBIu5G9J81obLEjqL8vfh4yQO50iIhIZizSbsT4o12vFArORxMR9XUs0m6kfdEYn9dNREQAi7TbuNRsQ3mVGQDno4mIqA2LtJsoPXEezTY7tP5qRAb1kzsdIiJyAyzSbsJ47Iehbs5HExERwCLtNgp5fzQREf0Ei7QbuNDUiq++swBgkSYioh+wSLuBPZV1sNkFIgb1w9BAzkcTEVEbFmk3wFuviIioIyzSboDz0URE1BEWaZmZLzbj4GkrACBxOIs0ERH9gEVaZruP1UEI4IbBAzDYXy13OkRE5EZYpGVm5Hw0ERFdAYu0zKT5aA51ExHRT7hFkV6zZg0iIyOhVquh1+tRXFx81fitW7ciOjoaarUasbGx2L59u8NxIQQyMjIQGhoKPz8/JCUl4ciRIw4xdXV1SE1Nhb+/PwICAjBnzhxcuHDB5X27mjP1TThS2/adY1mkiYjoJ2Qv0lu2bEF6ejoyMzNRVlaGkSNHwmAwoLa2tsP4wsJCTJ8+HXPmzEF5eTlSUlKQkpKC/fv3SzHZ2dnIycnB2rVrUVRUhP79+8NgMKCxsVGKSU1NxYEDB5Cfn49t27Zh165dmDdvXrf398eMx9quom8K9Udgf98e/W4iIvIAQmYJCQkiLS1Nem+z2URYWJjIysrqMH7q1KkiOTnZoU2v14tHHnlECCGE3W4XWq1WrFixQjpuNpuFSqUSmzZtEkIIcfDgQQFA7NmzR4r56KOPhEKhEKdOnepU3haLRQAQFoulcx3twHP/2ieGPbtN/OHDA13+DCIicg+uqAs/JeuVdHNzM0pLS5GUlCS1KZVKJCUlwWg0dniO0Wh0iAcAg8EgxVdWVsJkMjnEaDQa6PV6KcZoNCIgIADx8fFSTFJSEpRKJYqKijr83qamJlitVofX9ZqeEIGnkn6Ou2NDr/uziIio9/GW88vPnj0Lm82GkJAQh/aQkBBUVFR0eI7JZOow3mQyScfb264WM3jwYIfj3t7eGDRokBTzU1lZWXjhhRc62bPOuWVoAG4ZGuDSzyQiot5D9jlpT7Fo0SJYLBbpdfLkSblTIiKiXk7WIh0cHAwvLy/U1NQ4tNfU1ECr1XZ4jlarvWp8+6/XivnpwrTW1lbU1dVd8XtVKhX8/f0dXkRERN1J1iLt6+uLuLg4FBQUSG12ux0FBQVITEzs8JzExESHeADIz8+X4nU6HbRarUOM1WpFUVGRFJOYmAiz2YzS0lIpZseOHbDb7dDr9S7rHxER0XVx2RK0Ltq8ebNQqVRiw4YN4uDBg2LevHkiICBAmEwmIYQQM2bMEM8995wU/+WXXwpvb2/x6quvim+++UZkZmYKHx8f8fXXX0sxL7/8sggICBDvv/+++Oqrr8R9990ndDqduHTpkhRz1113idGjR4uioiLxxRdfiKioKDF9+vRO590dq/iIiMhzdUddkHXhGABMmzYNZ86cQUZGBkwmE0aNGoW8vDxp4VdVVRWUyh8u+MeNG4eNGzdi6dKlWLx4MaKiopCbm4uYmBgpZuHChWhoaMC8efNgNpsxfvx45OXlQa3+4dnY77zzDh5//HFMnDgRSqUSU6ZMQU5OTs91nIiI6BoUQgghdxKeyGq1QqPRwGKxcH6aiIi6pS5wdTcREZGbkn2421O1D0C44qEmRETk+drrgSsHqFmku6i+vh4AEB4eLnMmRETkTurr66HRaFzyWZyT7iK73Y7q6moMHDgQCoWiS59htVoRHh6OkydP9pp5bfbJM7BPnoF98gztfaqqqoJCoUBYWJjDgufrwSvpLlIqlRg6dKhLPqs3PhyFffIM7JNnYJ88g0ajcXmfuHCMiIjITbFIExERuSkWaRmpVCpkZmZCpVLJnYrLsE+egX3yDOyTZ+jOPnHhGBERkZvilTQREZGbYpEmIiJyUyzSREREbopFWkZr1qxBZGQk1Go19Ho9iouL5U6p07KysnDrrbdi4MCBGDx4MFJSUnDo0CGHmMbGRqSlpSEoKAgDBgzAlClTUFNTI1PGznn55ZehUCiwYMECqc0T+3Pq1Cn813/9F4KCguDn54fY2FiUlJRIx4UQyMjIQGhoKPz8/JCUlIQjR47ImPHV2Ww2LFu2DDqdDn5+fhgxYgT+8Ic/ODyG0d37tGvXLtx7770ICwuDQqFAbm6uw/HO5F9XV4fU1FT4+/sjICAAc+bMwYULF3qwF46u1qeWlhY8++yziI2NRf/+/REWFoaZM2eiurra4TM8qU8/9eijj0KhUGDVqlUO7a7oE4u0TLZs2YL09HRkZmairKwMI0eOhMFgQG1trdypdcrOnTuRlpaG3bt3Iz8/Hy0tLbjzzjvR0NAgxTz11FP48MMPsXXrVuzcuRPV1dWYPHmyjFl3zp49e7Bu3TrccsstDu2e1p/z58/jtttug4+PDz766CMcPHgQK1euRGBgoBSTnZ2NnJwcrF27FkVFRejfvz8MBgMaGxtlzPzKXnnlFbzxxhv485//jG+++QavvPIKsrOzsXr1ainG3fvU0NCAkSNHYs2aNR0e70z+qampOHDgAPLz87Ft2zbs2rUL8+bN66kuXOZqfbp48SLKysqwbNkylJWV4d1338WhQ4cwadIkhzhP6tOPvffee9i9ezfCwsIuO+aSPrlsZ2pySkJCgkhLS5Pe22w2ERYWJrKysmTMqutqa2sFALFz504hhBBms1n4+PiIrVu3SjHffPONACCMRqNcaV5TfX29iIqKEvn5+WLChAli/vz5QgjP7M+zzz4rxo8ff8XjdrtdaLVasWLFCqnNbDYLlUolNm3a1BMpOi05OVk8/PDDDm2TJ08WqampQgjP6xMA8d5770nvO5P/wYMHBQCxZ88eKeajjz4SCoVCnDp1qsdyv5Kf9qkjxcXFAoA4ceKEEMJz+/Tdd9+JIUOGiP3794thw4aJ1157TTrmqj7xSloGzc3NKC0tRVJSktSmVCqRlJQEo9EoY2ZdZ7FYAACDBg0CAJSWlqKlpcWhj9HR0YiIiHDrPqalpSE5Odkhb8Az+/PBBx8gPj4eDzzwAAYPHozRo0fjL3/5i3S8srISJpPJoU8ajQZ6vd5t+zRu3DgUFBTg8OHDAIB9+/bhiy++wN133w3AM/v0Y53J32g0IiAgAPHx8VJMUlISlEolioqKejznrrBYLFAoFAgICADgmX2y2+2YMWMGnnnmGdx8882XHXdVn/jsbhmcPXsWNpsNISEhDu0hISGoqKiQKauus9vtWLBgAW677TbExMQAAEwmE3x9faX/CduFhITAZDLJkOW1bd68GWVlZdizZ89lxzyxP8eOHcMbb7yB9PR0LF68GHv27MGTTz4JX19fzJo1S8q7o7+H7tqn5557DlarFdHR0fDy8oLNZsNLL72E1NRUAPDIPv1YZ/I3mUwYPHiww3Fvb28MGjTII/rY2NiIZ599FtOnT5eec+2JfXrllVfg7e2NJ598ssPjruoTizRdt7S0NOzfvx9ffPGF3Kl02cmTJzF//nzk5+dDrVbLnY5L2O12xMfH43/+538AAKNHj8b+/fuxdu1azJo1S+bsuuYf//gH3nnnHWzcuBE333wz9u7diwULFiAsLMxj+9SXtLS0YOrUqRBC4I033pA7nS4rLS3Fn/70J5SVlXV5F8TO4nC3DIKDg+Hl5XXZyuCamhpotVqZsuqaxx9/HNu2bcOnn37qsCuYVqtFc3MzzGazQ7y79rG0tBS1tbUYM2YMvL294e3tjZ07dyInJwfe3t4ICQnxqP4AQGhoKG666SaHthtvvBFVVVUAIOXtSX8Pn3nmGTz33HN48MEHERsbixkzZuCpp55CVlYWAM/s0491Jn+tVnvZAtPW1lbU1dW5dR/bC/SJEyeQn5/vsFuUp/Xp888/R21tLSIiIqSfFydOnMDTTz+NyMhIAK7rE4u0DHx9fREXF4eCggKpzW63o6CgAImJiTJm1nlCCDz++ON47733sGPHDuh0OofjcXFx8PHxcejjoUOHUFVV5ZZ9nDhxIr7++mvs3btXesXHxyM1NVX6b0/qDwDcdtttl90Wd/jwYQwbNgwAoNPpoNVqHfpktVpRVFTktn26ePHiZfv0enl5wW63A/DMPv1YZ/JPTEyE2WxGaWmpFLNjxw7Y7Xbo9foez7kz2gv0kSNH8MknnyAoKMjhuKf1acaMGfjqq68cfl6EhYXhmWeewccffwzAhX3q+no3uh6bN28WKpVKbNiwQRw8eFDMmzdPBAQECJPJJHdqnfLYY48JjUYjPvvsM3H69GnpdfHiRSnm0UcfFREREWLHjh2ipKREJCYmisTERBmzds6PV3cL4Xn9KS4uFt7e3uKll14SR44cEe+8847o16+f+Pvf/y7FvPzyyyIgIEC8//774quvvhL33Xef0Ol04tKlSzJmfmWzZs0SQ4YMEdu2bROVlZXi3XffFcHBwWLhwoVSjLv3qb6+XpSXl4vy8nIBQPzxj38U5eXl0krnzuR/1113idGjR4uioiLxxRdfiKioKDF9+nS5unTVPjU3N4tJkyaJoUOHir179zr8vGhqavLIPnXkp6u7hXBNn1ikZbR69WoREREhfH19RUJCgti9e7fcKXUagA5fb731lhRz6dIl8fvf/14EBgaKfv36ifvvv1+cPn1avqSd9NMi7Yn9+fDDD0VMTIxQqVQiOjparF+/3uG43W4Xy5YtEyEhIUKlUomJEyeKQ4cOyZTttVmtVjF//nwREREh1Gq1GD58uFiyZInDD3t379Onn37a4f87s2bNEkJ0Lv9z586J6dOniwEDBgh/f38xe/ZsUV9fL0Nv2lytT5WVlVf8efHpp596ZJ860lGRdkWfuAsWERGRm+KcNBERkZtikSYiInJTLNJERERuikWaiIjITbFIExERuSkWaSIiIjfFIk1EROSmWKSJiIjcFIs0ERGRm2KRJiLJmTNn8NhjjyEiIgIqlQparRYGgwFffvklAEChUCA3N1feJIn6EO4nTUSSKVOmoLm5GW+//TaGDx+OmpoaFBQU4Ny5c3KnRtQn8dndRAQAMJvNCAwMxGeffYYJEyZcdjwyMhInTpyQ3g8bNgzHjx8HALz//vt44YUXcPDgQYSFhWHWrFlYsmQJvL3brgMUCgVef/11fPDBB/jss88QGhqK7Oxs/OY3v+mRvhF5Kg53ExEAYMCAARgwYAByc3PR1NR02fE9e/YAAN566y2cPn1aev/5559j5syZmD9/Pg4ePIh169Zhw4YNeOmllxzOX7ZsGaZMmYJ9+/YhNTUVDz74IL755pvu7xiRB+OVNBFJ/vWvf2Hu3Lm4dOkSxowZgwkTJuDBBx/ELbfcAqDtivi9995DSkqKdE5SUhImTpyIRYsWSW1///vfsXDhQlRXV0vnPfroo3jjjTekmLFjx2LMmDF4/fXXe6ZzRB6IV9JEJJkyZQqqq6vxwQcf4K677sJnn32GMWPGYMOGDVc8Z9++fXjxxRelK/EBAwZg7ty5OH36NC5evCjFJSYmOpyXmJjIK2mia+DCMSJyoFarcccdd+COO+7AsmXL8Lvf/Q6ZmZl46KGHOoy/cOECXnjhBUyePLnDzyKiruOVNBFd1U033YSGhgYAgI+PD2w2m8PxMWPG4NChQ7jhhhsueymVP/yI2b17t8N5u3fvxo033tj9HSDyYLySJiIAwLlz5/DAAw/g4Ycfxi233IKBAweipKQE2dnZuO+++wC0rfAuKCjAbbfdBpVKhcDAQGRkZOCee+5BREQEfvOb30CpVGLfvn3Yv38/li9fLn3+1q1bER8fj/Hjx+Odd95BcXEx3nzzTbm6S+QRuHCMiAAATU1NeP755/Hvf/8b3377LVpaWhAeHo4HHngAixcvhp+fHz788EOkp6fj+PHjGDJkiHQL1scff4wXX3wR5eXl8PHxQXR0NH73u99h7ty5ANoWjq1Zswa5ubnYtWsXQkND8corr2Dq1Kky9pjI/bFIE1G362hVOBFdG+ekiYiI3BSLNBERkZviwjEi6nacVSPqGl5JExERuSkWaSIiIjfFIk1EROSmWKSJiIjcFIs0ERGRm2KRJiIiclMs0kRERG6KRZqIiMhNsUgTERG5qf8H+YW5GvGWa24AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "plt.figure(figsize=(5, 3))\n", "plt.ylabel(\"Learning rate\")\n", "plt.xlabel(\"Step\")\n", "total_training_steps = len(train_loader) * n_epochs\n", "plt.plot(range(total_training_steps), track_lrs)\n", "plt.tight_layout(); plt.savefig(\"1.pdf\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "7b3996b6-3f7a-420a-8584-c5760249f3d8", "metadata": {}, "source": [ "## D.2 Cosine decay" ] }, { "cell_type": "markdown", "id": "c5216214-de79-40cf-a733-b1049a73023c", "metadata": {}, "source": [ "- Another popular technique for training complex deep neural networks is cosine decay, which also adjusts the learning rate across training epochs\n", "- In cosine decay, the learning rate follows a cosine curve, decreasing from its initial value to near zero following a half-cosine cycle\n", "- This gradual reduction is designed to slow the pace of learning as the model begins to improve its weights; it reduces the risk of overshooting minima as the training progresses, which is crucial for stabilizing the training in its later stages\n", "- Cosine decay is often preferred over linear decay for its smoother transition in learning rate adjustments, but linear decay is also used in practice (for example, [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838))" ] }, { "cell_type": "code", "execution_count": 7, "id": "4e8d2068-a057-4abf-b478-f02cc37191f6", "metadata": {}, "outputs": [], "source": [ "import math\n", "\n", "min_lr = 0.1 * initial_lr\n", "track_lrs = []\n", "\n", "lr_increment = (peak_lr - initial_lr) / warmup_steps\n", "global_step = -1\n", "\n", "for epoch in range(n_epochs):\n", " for input_batch, target_batch in train_loader:\n", " optimizer.zero_grad()\n", " global_step += 1\n", " \n", " # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n", " if global_step < warmup_steps:\n", " # Linear warmup\n", " lr = initial_lr + global_step * lr_increment \n", " else:\n", " # Cosine annealing after warmup\n", " progress = ((global_step - warmup_steps) / \n", " (total_training_steps - warmup_steps))\n", " lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n", " \n", " # Apply the calculated learning rate to the optimizer\n", " for param_group in optimizer.param_groups:\n", " param_group[\"lr\"] = lr\n", " track_lrs.append(optimizer.param_groups[0][\"lr\"])\n", " \n", " # Calculate loss and update weights" ] }, { "cell_type": "code", "execution_count": 8, "id": "0e779e33-8a44-4984-bb23-be0603dc4158", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABPcklEQVR4nO3de1xT9f8H8NcubOM6EJQBgoBaeEFQkAlaZlJYdkHNWyZm3lPTsDTNa9/6kZpdKPPSRa28ZSWZX6MMTRMmyMW74A1FxXER2WDABtvn9wex2ldUhsDZ4P18PPagnfM5Z++P4d6ez5XHGGMghBBCiMXhcx0AIYQQQupHSZoQQgixUJSkCSGEEAtFSZoQQgixUJSkCSGEEAtFSZoQQgixUJSkCSGEEAtFSZoQQgixUEKuA7BWBoMB+fn5cHR0BI/H4zocQgghHGOMoaysDJ6enuDzm+YZmJJ0I+Xn58Pb25vrMAghhFiYa9euoWPHjk1yL0rSjeTo6Aig9n+Gk5MTx9EQQgjhmlqthre3tzE/NAVK0o1U18Tt5ORESZoQQohRU3aB0sAxQgghxEJRkiaEEEIsFCVpQgghxEJxnqTXrl0LX19fSCQSyOVypKWl3bP8rl27EBAQAIlEgsDAQOzbt8/k/E8//YQnn3wSrq6u4PF4OH78+B33qKqqwsyZM+Hq6goHBweMGDECBQUFTVktQggh5IFxmqR37tyJ2NhYLFu2DJmZmQgKCkJUVBQKCwvrLZ+SkoKxY8di0qRJyMrKQnR0NKKjo3H69GljGY1GgwEDBmDlypV3/dzXX38dv/zyC3bt2oVDhw4hPz8fw4cPb/L6EUIIIQ+CxxhjXH24XC5H37598dlnnwGoXSDE29sbs2fPxltvvXVH+dGjR0Oj0WDv3r3GY/369UNwcDDWr19vUvbKlSvw8/NDVlYWgoODjcdVKhXat2+Pbdu24YUXXgAAZGdno1u3blAoFOjXr1+DYler1ZBKpVCpVDS6mxBCSLPkBc6mYOl0OmRkZGDhwoXGY3w+H5GRkVAoFPVeo1AoEBsba3IsKioKCQkJDf7cjIwMVFdXIzIy0ngsICAAPj4+90zSWq0WWq3W+F6tVjf4M9uSjKu3sfbgRTDGIBLyIRYK/v7Jh7OdDdwcxHB1EMPNXgQ3RzFkUgmcJDZch00IIRaJsyRdXFwMvV4Pd3d3k+Pu7u7Izs6u9xqlUllveaVS2eDPVSqVEIlEcHZ2Nus+cXFxWLFiRYM/p636aP95HLlYbNY1rvYi+LrZw9fVHn5udvBv74BALyk6utjSkquEkDaNFjNpoIULF5o8xdetLEP+oa3R49iVEgDAoqcDYCcSQltjgK7GgKpqPUordCgu16G4XItbmtqfpRXVuKXR4ZZGh4yrt03u52xng0AvKQK9pOjVUYq+vu3g6iDmomqEEMIJzpK0m5sbBALBHaOqCwoKIJPJ6r1GJpOZVf5u99DpdCgtLTV5mr7ffcRiMcRiShD3knm1FNoaA9wcxJjyiH+DnoLLqqpx9VYFcos1uFKsQe4tDc4XlCFHWYbSimr8daEYf13458k8QOaIiM5uiOjsijD/dtRUTghp1ThL0iKRCCEhIUhKSkJ0dDSA2oFjSUlJmDVrVr3XhIeHIykpCXPnzjUe279/P8LDwxv8uSEhIbCxsUFSUhJGjBgBAMjJyUFeXp5Z9yF3UlyqTaYRnV0b3EztKLFBTy8penpJTY5ra/TIUZbh1A0VTl1XISuvFDkFZchW1r6+Ts4FnweEdHLBk91liOohg4+rXZPXiRBCuMRpc3dsbCwmTJiA0NBQhIWF4eOPP4ZGo8HEiRMBADExMfDy8kJcXBwAYM6cORg4cCDWrFmDoUOHYseOHUhPT8fGjRuN9ywpKUFeXh7y8/MB1CZgoPYJWiaTQSqVYtKkSYiNjUW7du3g5OSE2bNnIzw8vMEju0n9Ui7dAgD07+L6wPcSCwXo1dEZvTo6A/LaY8XlWhy9fAspl25BcekWcos1OHblNo5duY339p1DgMwRT/aQ4ameMnTzoBH3hBDrx2mSHj16NIqKirB06VIolUoEBwcjMTHRODgsLy/PZE/OiIgIbNu2DYsXL8aiRYvQtWtXJCQkoGfPnsYye/bsMSZ5ABgzZgwAYNmyZVi+fDkA4KOPPgKfz8eIESOg1WoRFRWFzz//vAVq3HpptDU4fq0UABDR2a1ZPsPNQYxnennimV6eAIDrtyvwx9kC/HamAGlXSoxP2fFJFxAgc8QLIR3xfLAX2jtSNwUhxDpxOk/amtE8aVMHcwoxcdMxdHSxxZEFj7f459/W6JCUXYjfzihxKKcIOr0BACDg8zDwofYY0acjnujuDpGQ80X2CCGtVKuaJ01aF0VdU3czPUXfj4u9CC+EdMQLIR1RWqHDLydv4seM6zh+rRQHsgtxILsQ7R3FeDHMBy/KfeDuJOEkTkIIMQcladIkUuoGjTVBf/SDcrYTYXy/ThjfrxMuFZXjp8zr2JV+HYVlWnySdAFrD17EU4EemBDeCSGdXGguNiHEYlFzdyNRc/c/Sit06P2f/WAMSFs0GB0s8Cm1Wm9A4mklvlFcwbEr/8zHDuooxauDuuCJbu7g8ylZE0Iaj5q7iUU6evkWGAO6dnCwyAQNADYCPp4N8sSzQZ44k6/CNylXkXD8Bk5cV2Hatxno2sEBrw7qjGd7eUIooH5rQohloG8j8sDqpl5FdOa+qbshenhKsfKFXkh+63HMHNQZjmIhLhSW4/WdJzBozZ/YlpqH6r8HnhFCCJcoSZMHVpekwzkaNNZYbg5ivBkVgOSFj+PNqIfhai/CtZJKLNp9Ck98eAg/H78Bg4F6gwgh3KEkTR5IgboKFwvLweMB/fzbcR1OozhJbDBzUBccWfA4lj7THW4OIly5VYE5O45j6KdHcCC7ADR0gxDCBUrS5IHUTb3q6SmFs52I42gejK1IgFcG+OHQm4PwxpMPwVEsxLmbaryyOR2jNiiMi7UQQkhLoSRNHkjKv9brbi3sxULMerwrDs8fhGmP+kMs5OPYlduIXpuMN3adQKG6iusQCSFtBCVp0miMMSRfrOuPbj1Juo6LvQgLn+6GQ28OwvA+XgCAHzKuY9AHf+LzPy+iqlrPcYSEkNaOkjRptGsllbhRWgkhn4cwP+vsj24ImVSCD0cFY/erEQj2doZGp8eqxBw8+dFhHMguuP8NCCGkkShJk0ara+ru7eMMO1Hrn3Lf28cFP82IwIejgtDBUYy8kgq8sjkdr27NQAE1gRNCmgEladJo1jr16kHw+TwM79MRB994DFMf9YeAz8O+U0pErjmEbxVXoKcpW4SQJkRJmjQKY8zqFjFpSvZiIRY93Q2/zBqAIG9nlGlrsOTnMxixLgXnbqq5Do8Q0kpQkiaNcqGwHMXlWkhs+Ojt48x1OJzp7umEn2ZE4J3ne8BBLMTxa6V49tMjiE+6QKuWEUIeGCVp0igpF2v7o/v6toNYKOA4Gm4J+DzEhPsiad5ARPVwR42B4cP95zHs82TkKMu4Do8QYsUoSZNGSb7UeqdeNZa7kwTrXwrBJ2OCIbW1wekbajzz6V9Ye/AiauipmhDSCJSkidn0Boajl2uTdP82NGisIXg8Hp4P9sL+1x9FZLcOqNYzrP4tByPWpeByUTnX4RFCrAwlaWK2M/kqlFXVwFEiRA/Ptr2X9t10cJLgi5hQfDgqCE4SIU5cV2Fo/BF8f+warQNOCGkwStLEbHWjuuV+rrT38j3weLXTtX5/fSDC/V1RWa3H/B9PYua2TKgqqrkOjxBiBegblpgt+WLrW6+7OcmkEnw3WY4FQwIg/Hte9ZBPDhu7DAgh5G4oSROz6GoMOHalBADQvwv1RzeUgM/DjMc646dXI+DnZo+bqiqM/eIoPtx/nhZAIYTcFSVpYpbj10pRVW2Aq70ID7k7cB2O1enV0Rl7Zw/A6FBvMAbEJ11AzNepKCrTch0aIcQCUZImZqlr6g7v7Aoej8dxNNbJXizEyhd64ePRwbATCZB88Raejv+Lmr8JIXegJE3Movh70Bg1dT+46N5e2DOrPx5yd0BRmRYvfnEUaw9ehIGavwkhf6MkTRqsQleDrGu3AdCgsabSpYMjEmb2x/A+XjAwYPVvOZj8TTpUlTT6mxBCSZqYIf3KbVTrGbycbeHTzo7rcFoNO5EQa0YGYeWIQIiFfBzILkT02mRcKKAlRQlp6yhJkwZLvkT90c2Fx+NhdF8f/DA9Al7Otsgt1iB6bTIST9/kOjRCCIcoSZMG+6c/mpq6m0tgRyn2zOqPcH9XaHR6TP8uEx/8lkPTtAhpoyhJkwZRVVTj9A0VACDcnwaNNSdXBzG+nRSGSQP8AACfHbyIyVuOQV1F/dSEtDWcJ+m1a9fC19cXEokEcrkcaWlp9yy/a9cuBAQEQCKRIDAwEPv27TM5zxjD0qVL4eHhAVtbW0RGRuLChQsmZc6fP4/nn38ebm5ucHJywoABA3Dw4MEmr1trcjT3FgwM8G9vD5lUwnU4rZ5QwMeSZ7rj49HBkNjwcTCnCC+sS8G1kgquQyOEtCBOk/TOnTsRGxuLZcuWITMzE0FBQYiKikJhYWG95VNSUjB27FhMmjQJWVlZiI6ORnR0NE6fPm0ss2rVKsTHx2P9+vVITU2Fvb09oqKiUFVVZSzzzDPPoKamBgcOHEBGRgaCgoLwzDPPQKlUNnudrZWxqZt2vWpR0b29sGtaBNydxDhfUI7otcnIuFrCdViEkJbCOBQWFsZmzpxpfK/X65mnpyeLi4urt/yoUaPY0KFDTY7J5XI2bdo0xhhjBoOByWQytnr1auP50tJSJhaL2fbt2xljjBUVFTEA7PDhw8YyarWaAWD79+9vcOwqlYoBYCqVqsHXWLMnPvyTdVqwl+07mc91KG3SzdJKNjT+MOu0YC/rumgf2515neuQCCH/oznyAmdP0jqdDhkZGYiMjDQe4/P5iIyMhEKhqPcahUJhUh4AoqKijOVzc3OhVCpNykilUsjlcmMZV1dXPPzww/jmm2+g0WhQU1ODDRs2oEOHDggJCWnqarYKRWVanC+o3Qu5nz8NGuOCTCrB99PCEdXDHTq9AXN3Hsea33No4RNCWjnOknRxcTH0ej3c3d1Njru7u9+12VmpVN6zfN3Pe5Xh8Xj4448/kJWVBUdHR0gkEnz44YdITEyEi4vLXePVarVQq9Umr7Yi5e+pV909nOBiL+I4mrbLTiTEunEhmPFYZwDApwcuYvb2LFTq9BxHRghpLpwPHGtpjDHMnDkTHTp0wF9//YW0tDRER0fj2Wefxc2bd5+TGhcXB6lUanx5e3u3YNTcoqlXloPP52HBkACsfqEXbAQ8/PfUTYzZqEChuur+FxNCrA5nSdrNzQ0CgQAFBQUmxwsKCiCTyeq9RiaT3bN83c97lTlw4AD27t2LHTt2oH///ujTpw8+//xz2NraYsuWLXeNd+HChVCpVMbXtWvXzKuwFUv5O0lH0KAxizEy1BvfTZLD2c4GJ66raIUyQlopzpK0SCRCSEgIkpKSjMcMBgOSkpIQHh5e7zXh4eEm5QFg//79xvJ+fn6QyWQmZdRqNVJTU41lKipqp7Dw+aZV5/P5MBgMd41XLBbDycnJ5NUWXCupQF5JBYR8Hvr6teM6HPIvcn9X/DyzP/zb2yNfVYUR61KMe30TQloHTpu7Y2Nj8cUXX2DLli04d+4cZsyYAY1Gg4kTJwIAYmJisHDhQmP5OXPmIDExEWvWrEF2djaWL1+O9PR0zJo1C0Btf/PcuXPx7rvvYs+ePTh16hRiYmLg6emJ6OhoALWJ3sXFBRMmTMCJEydw/vx5vPnmm8jNzcXQoUNb/M/A0tU1dQd5O8NBLOQ4GvK/Orna48fpEejj4wx1VQ3GfZlKS4kS0opwmqRHjx6NDz74AEuXLkVwcDCOHz+OxMRE48CvvLw8k37iiIgIbNu2DRs3bkRQUBB++OEHJCQkoGfPnsYy8+fPx+zZszF16lT07dsX5eXlSExMhERSuwCHm5sbEhMTUV5ejscffxyhoaE4cuQIfv75ZwQFBbXsH4AVqBs0RrteWS4XexG2Tu6HyG7u0NUYMGNrJr5VXOE6LEJIE+AxxmgORyOo1WpIpVKoVKpW2/TNGIP8/5JQWKbFtily6pO2cDV6A5buOYNtqXkAgJmDOuONJx+mzVAIaSHNkRfa3Ohu0nCXispRWKaFWMhHH5+7T08jlkEo4OO96J6IfeIhAMDag5fw5g8nUa2/+1gLQohloyRN7qpuVHeorwskNgKOoyENwePx8Nrgrlg5IhACPg8/ZFzHlG/SodHWcB0aIaQRKEmTu0q5SFOvrNXovj74IiYEEhs+/swpwotfHMVtjY7rsAghZqIkTeqlNzAoLtcm6XAaNGaVHg9wx/Yp/eDy91zqURsUKKBFTwixKpSkSb3O3VRDVVkNB7EQvbykXIdDGqm3jwt2TQ+HzEmCC4XleGF9CvJu0XaXhFgLStKkXnVTr+R+7SAU0K+JNevSwRG7poejk6sdrpVU4oX1KThPq5MRYhXo25fUq27QGDV1tw7e7eywa1o4HnZ3RGGZFqM2KHDiWinXYRFC7oOSNLmDrsaAtNza5SX7d6FBY61FBycJdk7rhyBvZ5RWVOPFL44aV5QjhFgmStLkDievl6JCp0c7exEednfkOhzShJztRNg6WY6Izq7Q6PSYsCkNSecK7n8hIYQTlKTJHYxN3f6u4PNptarWxkEsxNcv9zUuIzrt2wz8fPwG12ERQurRqCRdU1ODP/74Axs2bEBZWe0AlPz8fJSXlzdpcIQbyRdrB41Rf3TrJbERYN1LfRAd7IkaA8PcncfxQ8Z1rsMihPwPs7c1unr1KoYMGYK8vDxotVo88cQTcHR0xMqVK6HVarF+/frmiJO0kEqdHll5pQCoP7q1sxHw8eGoYNiKhNieloc3fziBar0BY8N8uA6NEPI3s5+k58yZg9DQUNy+fRu2trbG48OGDbtjr2difTKu3oZOb4CHVAJfVzuuwyHNjM/n4f+G9cSE8E5gDFj40yl8QztoEWIxzH6S/uuvv5CSkgKRSGRy3NfXFzduUL+WtUu+9E9TN+2e1DbweDwsf64HbAR8fHkkF0t/PgNdjQGTH/HnOjRC2jyzn6QNBgP0ev0dx69fvw5HRxoJbO3qBo31p/W62xQej4e3h3bDjMc6AwDe/e85rD90ieOoCCFmJ+knn3wSH3/8sfE9j8dDeXk5li1bhqeffropYyMtTF1VjVPXSwHQoLG2iMfjYX7Uw5gzuCsA4P1fs/Fp0gWOoyKkbTO7uXvNmjWIiopC9+7dUVVVhRdffBEXLlyAm5sbtm/f3hwxkhaSdrkEBgb4udnD09n2/heQVofH4+H1Jx6CjYCHD34/jzX7z6Nab8DrTzxE3R+EcMDsJN2xY0ecOHECO3fuxIkTJ1BeXo5JkyZh3LhxJgPJiPX5d380adtmPd4VNgI+4n7NRvyBi6g2MMyPepgSNSEtzOwkffjwYURERGDcuHEYN26c8XhNTQ0OHz6MRx99tEkDJC1HQf3R5F+mDewMGwEf7+w9i3V/XgKfB7zxJCVqQlqS2X3SgwYNQklJyR3HVSoVBg0a1CRBkZZXXK5FtrJ2YZp+/u04joZYilcG+GH5s90BAGsPXsJH+89zHBEhbYvZSZoxVu+/pG/dugV7e/smCYq0vLqn6ACZI1wdxBxHQyzJy/39sOSZ2kQdf+AiPv6DEjUhLaXBzd3Dhw8HUDuw5OWXX4ZY/M8XuV6vx8mTJxEREdH0EZIWYZx6RauMkXpMGuAHg4HhvX3n8PEfFyDg8TD771HghJDm0+AkLZVKAdQ+STs6OpoMEhOJROjXrx+mTJnS9BGSFqH4e9BYBA0aI3cx5VF/6BnD+79mY83+8+DzeZg5qAvXYRHSqjU4SW/atAlA7cpib7zxBjVttyI3Sitx5VYFBHwewvyoP5rc3fSBnaE3MKz+LQerf8uBgM/D9IGduQ6LkFbL7NHdy5Yta444CIdS/t71KtBLCkeJDcfREEs3c1AXGAwMa/afx/u/ZkPI59ESooQ0E7OTNAD88MMP+P7775GXlwedTmdyLjMzs0kCIy3HOPWqCzV1k4aZPbgr9Izh4z8u4N3/ngOPx8OkAX5ch0VIq2P26O74+HhMnDgR7u7uyMrKQlhYGFxdXXH58mU89dRTzREjaUaMMeOgsQiaH03MMDfyIbz2eG2f9H/2nsXm5FyOIyKk9TE7SX/++efYuHEjPv30U4hEIsyfPx/79+/Ha6+9BpVK1RwxkmZ0uVgDpboKIiEfIZ1cuA6HWJnXn3gIMwfV9kkv/+UsvqVtLglpUmYn6by8PONUK1tbW5SV1S6AMX78eFq72wrVPUWH+LhAYiPgOBpibXg8Ht548mHj4LElP5/B9rQ8jqMipPUwO0nLZDLjimM+Pj44evQoACA3NxeMsaaNjjQ7mnpFHhSPx8OCIQ9j6qO1g8cW7T6F3VnXOY6KkNbB7CT9+OOPY8+ePQCAiRMn4vXXX8cTTzyB0aNHY9iwYWYHsHbtWvj6+kIikUAulyMtLe2e5Xft2oWAgABIJBIEBgZi3759JucZY1i6dCk8PDxga2uLyMhIXLhw53Z7//3vfyGXy2FrawsXFxdER0ebHbu1MxiYcdBYBA0aIw+Ax+Nh4VMBiAnvBMaAN3adROLpm1yHRYj1Y2bS6/Wsurra+H779u1s9uzZLD4+nmm1WrPutWPHDiYSidjXX3/Nzpw5w6ZMmcKcnZ1ZQUFBveWTk5OZQCBgq1atYmfPnmWLFy9mNjY27NSpU8Yy77//PpNKpSwhIYGdOHGCPffcc8zPz49VVlYay/zwww/MxcWFrVu3juXk5LAzZ86wnTt3mhW7SqViAJhKpTLrOkty+kYp67RgL+u+5Femq9FzHQ5pBfR6A5v3/XHWacFe1mXRf9mBc/X/XSakNWqOvGBWkq6urmYrVqxg165da5IPDwsLYzNnzjS+1+v1zNPTk8XFxdVbftSoUWzo0KEmx+RyOZs2bRpjjDGDwcBkMhlbvXq18XxpaSkTi8Vs+/btxjp4eXmxL7/88oFibw1J+ovDl1inBXvZy1+nch0KaUVq9AY2c2sG67RgL3vo7X0s+WIR1yER0iKaIy+Y1dwtFAqxatUq1NTUPPATvE6nQ0ZGBiIjI43H+Hw+IiMjoVAo6r1GoVCYlAeAqKgoY/nc3FwolUqTMlKpFHK53FgmMzMTN27cAJ/PR+/eveHh4YGnnnoKp0+ffuA6WRuaekWag4DPw0ejgxHZzR3aGgMmb0lHxtU7d84jhNyf2X3SgwcPxqFDhx74g4uLi6HX6+Hu7m5y3N3dHUqlst5rlErlPcvX/bxXmcuXLwMAli9fjsWLF2Pv3r1wcXHBY489Vu8WnHW0Wi3UarXJy5pV6w1IvVybpMNp0BhpYjYCPj57sTce6eqGCp0eL399DKeu0xRNQsxl9opjTz31FN566y2cOnUKISEhd6zh/dxzzzVZcM3BYDAAAN5++22MGDECQO265B07dsSuXbswbdq0eq+Li4vDihUrWizO5nbyugoanR7Odjbo7uHEdTikFZLYCLBxfCgmbEpDWm4Jxn+dip1Tw/GwzJHr0AixGmYn6VdffRUA8OGHH95xjsfjQa/XN+g+bm5uEAgEKCgoMDleUFAAmUxW7zUymeye5et+FhQUwMPDw6RMcHAwABiPd+/e3XheLBbD398feXl3n9+5cOFCxMbGGt+r1Wp4e3vfr5oWq27qVbi/K/j8O/cHJ6Qp2IoE+Prlvhj3ZSpOXCvFuC9T8f20fvBv78B1aIRYBbObuw0Gw11fDU3QQO32liEhIUhKSjK5d1JSEsLDw+u9Jjw83KQ8AOzfv99Y3s/PDzKZzKSMWq1GamqqsUxISAjEYjFycnKMZaqrq3HlyhV06tTprvGKxWI4OTmZvKzZP/3R1NRNmpeDWIhvJoahm4cTisu1GPdlKq6VVHAdFiFWwewk3ZRiY2PxxRdfYMuWLTh37hxmzJgBjUaDiRMnAgBiYmKwcOFCY/k5c+YgMTERa9asQXZ2NpYvX4709HTMmjULQO2T/Ny5c/Huu+9iz549OHXqFGJiYuDp6WmcB+3k5ITp06dj2bJl+P3335GTk4MZM2YAAEaOHNmyfwAcqarWI/3qbQBARBcaNEaan9TOBt9OCkPn9va4qarCuC9ToVRVcR0WIRavUbtgNZXRo0ejqKgIS5cuhVKpRHBwMBITE40Dv/Ly8sDn//PviIiICGzbtg2LFy/GokWL0LVrVyQkJKBnz57GMvPnz4dGo8HUqVNRWlqKAQMGIDExERKJxFhm9erVEAqFGD9+PCorKyGXy3HgwAG4uLSNtaszr96GrsYAdycx/N1oX3DSMtwcxNg6uR9GbVAgr6QC4748ip3TwuHmIOY6NEIsFo8xWsuzMdRqNaRSKVQqldU1fX/wWw4+O3gRw3p74aPRwVyHQ9qYayUVGL1BgXxVFbp5OGH7FDmc7URch0XIA2uOvMBpczfhRnLdoDHqjyYc8G5nh+8my+HmIMa5m2pM+DoNZVXVXIdFiEWiJN3GlFVV4+Tf81Vp0Bjhin97B2ydLIeLnQ1OXFdh0pZ0VOoaPvCUkLbC7CT9vwt61L3Kysqg0+maI0bShI5dKYHewNDJ1Q4dXey4Doe0YQ/LHPHtJDkcxUKk5ZZg+ncZ0NZQoibk38xO0s7OznBxcbnj5ezsDFtbW3Tq1AnLli0zLhpCLEvKRZp6RSxHTy8pNk3sC1sbAQ6dL8LcHcdRo6fvDkLqmJ2kN2/eDE9PTyxatAgJCQlISEjAokWL4OXlhXXr1mHq1KmIj4/H+++/3xzxkgeUTOt1EwsT6tsOG2NCIBLw8etpJeb/eBIGA41nJQRoxBSsLVu2YM2aNRg1apTx2LPPPovAwEBs2LABSUlJ8PHxwXvvvYdFixY1abDkwZRodDh3s3bN8X7+9CRNLMcjXdvjsxd7Y8bWTPyUeQMOYiFWPNcDPB6thkfaNrOfpFNSUtC7d+87jvfu3du409SAAQPuucQm4cbRvzfUeNjdEe0daW4qsSxP9pBhzcgg8HjAN4qrWPVbzv0vIqSVMztJe3t746uvvrrj+FdffWVcy/rWrVttZmEQa5J8kaZeEcsW3dsL70UHAgDW/XkJaw9e5DgiQrhldnP3Bx98gJEjR+LXX39F3759AQDp6enIzs7GDz/8AAA4duwYRo8e3bSRkgem+Ls/uj8tBUos2ItyH2i0NXhv3zms/i0H9iIBXu7vx3VYhHDC7CT93HPPITs7Gxs2bMD58+cB1G5fmZCQAF9fXwAwroVNLMdNVSUuF2vA5wFhfu24DoeQe5ryqD/KtDWIT7qA5b+chb1YiJGh1rvrHCGN1ai1u/38/Gj0tpWpe4oO9JJCamvDcTSE3N/rkV2h0dbgqyO5WPDjSdiJhBjay+P+FxLSijQqSZeWliItLQ2FhYV3zIeOiYlpksBI00qumx9NTd3ESvB4PCwe2g0abQ12HLuGuTuzYCcSYFBAB65DI6TFmJ2kf/nlF4wbNw7l5eVwcnIymSLB4/EoSVsgxhgUf6/XTYuYEGvC4/Hw3rBAaHR6/HIiH9O/y8DmiWE0+JG0GWaP7p43bx5eeeUVlJeXo7S0FLdv3za+SkpKmiNG8oCu3qpAvqoKNgIeQjtRfzSxLgI+Dx+OCkJktw7Q1hgwecsxZOXd5josQlqE2Un6xo0beO2112BnR+s+W4u6Xa96+7jAViTgOBpCzGcj4OOzF/sgorMrNDo9Xt50zLgwDyGtmdlJOioqCunp6c0RC2kmKXVTr2gpUGLFJDYCfBETij4+zlBVVmP8V6m4XFTOdViENCuz+6SHDh2KN998E2fPnkVgYCBsbExHCj/33HNNFhx5cAYDw9G69bq7UD8esW72YiE2TQzD2I1HcfamGi99mYrvp4fTjm6k1eIxxsxayZ7Pv/vDN4/Hg17fNraaU6vVkEqlUKlUcHJy4jqcu8pWqjHk479gayPAiWVPQiSkLcSJ9btVrsWoDQpcKtLA19UO308LRwcnCddhkTauOfKC2d/YBoPhrq+2kqCtSd3UqzC/dpSgSavh6iDGd5Pl6Ohiiyu3KjD+qzTc1tB+9qT1oW/tVo6mXpHWykNqi22T+6GDoxg5BWWYsCkNZVXVXIdFSJNqUJ90fHw8pk6dColEgvj4+HuWfe2115okMPLgavQGpF6unRZH+0eT1sjH1Q5bJ8sxeuNRnLyuwqQt6dgyMYxmMZBWo0F90n5+fkhPT4erqyv8/O6+0D2Px8Ply5ebNEBLZQ190ll5tzHs8xQ4SYTIWvokBHzam5e0TqdvqDB241GUaWsw8KH22BgTArGQEjVpWc2RFxr0JJ2bm1vvfxPLVjf1KryzKyVo0qr19JJi08S+GP9VGg6dL8LcHcfx6djeEAqoR49YN/oNbsXqNtWgpm7SFoT6tsPGmBCIBHz8elqJ+T+ehMFg1uQVQiyO2fOk9Xo9Nm/ejKSkpHo32Dhw4ECTBUcaT1ujx7Erdf3RNGiMtA2PdG2Pz17sjRlbM/FT5g04iIVY8VwPkz0GCLEmZifpOXPmYPPmzRg6dCh69uxJv/wWKvNqKbQ1BrR3FKNLBweuwyGkxTzZQ4Y1I4Pw+vfH8Y3iKuzFQiwYEsB1WIQ0itlJeseOHfj+++/x9NNPN0c8pIn8e+oV/UOKtDXRvb1QodNj0e5TWPfnJTiIhZg5qAvXYRFiNrP7pEUiEbp0oV92S5di7I+mpm7SNr0o98HbT3cDAKz+LQebk2nQK7E+jdqq8pNPPoGZq4mSFqTR1uD4tVIANGiMtG1THvXHa4O7AgCW/3IWu9KvcRwRIeYxu7n7yJEjOHjwIH799Vf06NHjjg02fvrppyYLjjRO2pUS1BgYvNvZwrsdbTxA2rbXI7tCo63BV0dyseDHk7ATCTG0lwfXYRHSIGYnaWdnZwwbNqw5YiFNxDj1yp+eognh8XhYPLQbNNoa7Dh2DXN3ZsFOJMCggA5ch0bIfZnV3F1TU4NBgwYhLi4OmzZtqvfVGGvXroWvry8kEgnkcjnS0tLuWX7Xrl0ICAiARCJBYGAg9u3bZ3KeMYalS5fCw8MDtra2iIyMxIULF+q9l1arRXBwMHg8Ho4fP96o+C1N8sW/B43R1pSEAKhN1O8NC8SzQZ6o1jNM/y7D+I9ZQiyZWUlaKBRi+vTp0Gq1TRbAzp07ERsbi2XLliEzMxNBQUGIiopCYWFhveVTUlIwduxYTJo0CVlZWYiOjkZ0dDROnz5tLLNq1SrEx8dj/fr1SE1Nhb29PaKiolBVVXXH/ebPnw9PT88mqw/Xbmt0OHtTDaB2pTFCSC0Bn4cPRwUhslsHaGsMmLzlmHHsBiEWi5lp4MCBbPfu3eZedldhYWFs5syZxvd6vZ55enqyuLi4esuPGjWKDR061OSYXC5n06ZNY4wxZjAYmEwmY6tXrzaeLy0tZWKxmG3fvt3kun379rGAgAB25swZBoBlZWU1OG6VSsUAMJVK1eBrWsKvp/JZpwV7WeSaP7kOhRCLVKmrYWM3KlinBXtZr+W/sbP5lvV3mFiv5sgLZo/ufvXVVzFv3jx89tlnUCgUOHnypMnLHDqdDhkZGYiMjDQe4/P5iIyMhEKhqPcahUJhUh4AoqKijOVzc3OhVCpNykilUsjlcpN7FhQUYMqUKfj2229hZ3f/wVVarRZqtdrkZYlo6hUh9yaxEeCLmFD08XGGqrIa479Kw+Wicq7DIqReZg8cGzNmDADTLSl5PB4YY+DxeNDr9Q2+V3FxMfR6Pdzd3U2Ou7u7Izs7u95rlEplveWVSqXxfN2xu5VhjOHll1/G9OnTERoaiitXrtw31ri4OKxYsaJB9eLSP/3RNGiMkLuxFwuxaWIYxm48irM31Xjpy1R8Pz0cHV1oNgSxLGYn6dawC9ann36KsrIyLFy4sMHXLFy4ELGxscb3arUa3t7ezRFeoxWoq3CpSAMeD+jnR0/ShNyL1NYG30wKw+gNClwq0hgTdQdHCdehEWJkdpLu1KlTk324m5sbBAIBCgoKTI4XFBRAJpPVe41MJrtn+bqfBQUF8PDwMCkTHBwMoHYTEIVCAbFYbHKf0NBQjBs3Dlu2bLnjc8Vi8R3lLU3daNWenlJI7WzuU5oQ4uYgxneT5Ri5XoErtyow/ss07JjaDy72Iq5DIwTAA2xVefbsWSQmJmLPnj0mL3OIRCKEhIQgKSnJeMxgMCApKQnh4eH1XhMeHm5SHgD2799vLO/n5weZTGZSRq1WIzU11VgmPj4eJ06cwPHjx3H8+HHjFK6dO3fivffeM6sOlsTY1E390YQ0mIfUFlsny9HBUYycgjK8vCkNZVXVXIdFCIBGPElfvnwZw4YNw6lTp4x90QCMmziY0ycNALGxsZgwYQJCQ0MRFhaGjz/+GBqNBhMnTgQAxMTEwMvLC3FxcQBqd+EaOHAg1qxZg6FDh2LHjh1IT0/Hxo0bjXHMnTsX7777Lrp27Qo/Pz8sWbIEnp6eiI6OBgD4+PiYxODgULtLVOfOndGxY0dz/0gsAmPsn0Fj1B9NiFk6udpj62Q5Rm1Q4MR1FSZtSceWiWGwFQm4Do20cWY/Sc+ZMwd+fn4oLCyEnZ0dzpw5g8OHDyM0NBR//vmn2QGMHj0aH3zwAZYuXYrg4GAcP34ciYmJxoFfeXl5uHnzprF8REQEtm3bho0bNyIoKAg//PADEhIS0LNnT2OZ+fPnY/bs2Zg6dSr69u2L8vJyJCYmQiJpvX1N10oqcaO0EkI+D319XbgOhxCr09XdEd9OksNRLERabgmmfpuOqmrzHjoIaWo8xszbKcPNzQ0HDhxAr169IJVKkZaWhocffhgHDhzAvHnzkJWV1VyxWhS1Wg2pVAqVSgUnJyeuw8GOtDy89dMp9PV1wa7pEVyHQ4jVSr9Sgpiv01Ch02PgQ+2xYXwIJDb0RE3urznygtlP0nq9Ho6OjgBqE3Z+fj6A2gFlOTk5TRIUMV+ycX40NXUT8iBCfdvh65f7wtZGgEPnizDjuwxoa+iJmnDD7CTds2dPnDhxAgAgl8uxatUqJCcn45133oG/v3+TB0jujzEGxSUaNEZIU+nn74qvXg6FxIaPgzlFmLk1E7oaA9dhkTbI7CS9ePFiGAy1v6zvvPMOcnNz8cgjj2Dfvn2Ij49v8gDJ/V0oLEdxuQ4SGz6CfZy5DoeQViGisxu+mtAXYiEff5wrxKxtmajWU6ImLcvsJB0VFYXhw4cDALp06YLs7GwUFxejsLAQjz/+eJMHSO6vbupVX992EAup74yQptK/ixu+iAmFSMjH72cLMHtbFiVq0qIaPU/64sWL+O2331BZWYl27do1ZUzETCnUH01Is3n0ofbYOD4EIgEfiWeUmLvjOGooUZMWYnaSvnXrFgYPHoyHHnoITz/9tHF61KRJkzBv3rwmD5Dcm97AcPQybapBSHN67OEO2PB3ov7vqZuYu5MSNWkZZifp119/HTY2NsjLyzPZPWr06NFITExs0uDI/Z3JV6GsqgaOEiF6ekm5DoeQVmtQQAese6kPbAQ87D15E/N2nYDeYNYMVkLMZnaS/v3337Fy5co7Vubq2rUrrl692mSBkYZJvlj7FN3P3xUCPo/jaAhp3QZ3c8faF/tAyOfh5+P5eIMSNWlmZidpjUZT7/7LJSUlFr8BRWuUQlOvCGlRT/aQ4bMX+0DA52F31g3M/+EkJWrSbMxO0o888gi++eYb43sejweDwYBVq1Zh0KBBTRocuTddjQHHrpQAoEFjhLSkIT1l+HRsbwj4PPyYeR1v7DpBfdSkWZi9wcaqVaswePBgpKenQ6fTYf78+Thz5gxKSkqQnJzcHDGSu8jKu42qagPcHER4yN2B63AIaVOeDvQAY8BrO7KwO+sGagwMH44Kgo2g0ZNmCLlDo1YcO3/+PAYMGIDnn38eGo0Gw4cPR1ZWFjp37twcMZK7qJt6Fd7ZzbgLGSGk5Qzt5YG1L9YOJvvlRD5mb8uilclIk2rUP/mkUinefvttfP/999i3bx/effdd6PV6TJ06tanjI/eguERTrwjh2pCeMqx/6Z951K9upbW+SdNpsnaZW7du4auvvmqq25H7qNDVIOvabQCUpAnh2uBu7vhiQqhxCdGp32TQNpekSVDniZU6duU2qvUMXs628Gl352h7QkjLGvhQe2z61+5Zk7eko1JHiZo8GErSVurfU6+oP5oQyxDRxQ2bJ/aFvUiAIxeL8fKmNGi0NVyHRawYJWkrZeyP7kJN3YRYErm/K76ZFAZHsRCpuSWI+ToN6qpqrsMiVqrBU7Dqdr66m9LS0geNhTSQqqIap26oAND8aEIsUUindvh2shwxX6Ui4+ptjN14FN+8EgZXB1rwiZinwU/SUqn0nq9OnTohJiamOWMlfzuaewuMAZ3b28PdScJ1OISQegR7O2P71H5wcxDhTL4aozYocFNVyXVYxMo0+El606ZNzRkHMYOCtqYkxCr08JTi+2nheOnLVFwq0uCFdQp8N1kOPzd7rkMjVoL6pK0QrddNiPXwb++AXTMi4O9mjxullRi5XoFzN9Vch0WsBCVpK1NYVoXzBeXg8Wp3viKEWD4vZ1vsnBaObh5OKC7XYvQGBTLzbnMdFrEClKStTF1Td3cPJ7jYiziOhhDSUO0dxdgxpR/6+DhDXVWDl75MRfLFYq7DIhaOkrSVoaVACbFeUjsbfDdZjke6uqFCp8fETcfw2xkl12ERC0ZJ2sok1/VHd6FBY4RYIzuREF9OCMWQHjLo9AbM+C4DO9LyuA6LWChK0lbkWkkFrpVUQsjnoa9vO67DIYQ0klgowGcv9sbIkI4wMOCtn07hkz8ugDHGdWjEwlCStiJ1Td1B3s5wEJu9FTghxIIIBXyseqEXZg3qAgD46I/zWLT7NGr0tNUl+QclaStCU68IaV14PB7eiHoY/4nuCR4P2J6Wh+nfZdLGHMSIkrSVYIwhmRYxIaRVGt+vE9aNC4FIyMcf5wow7sujuK3RcR0WsQCUpK3EpaJyFJVpIRby0dvHmetwCCFNbEhPGbZOlkNqa4PMvFKMWJ+CayUVXIdFOGYRSXrt2rXw9fWFRCKBXC5HWlraPcvv2rULAQEBkEgkCAwMxL59+0zOM8awdOlSeHh4wNbWFpGRkbhw4YLx/JUrVzBp0iT4+fnB1tYWnTt3xrJly6DTWe6/XFP+fooO9XWBxEbAcTSEkObQ17cdfpgeDk+pBJeLNBi+LgVn8lVch0U4xHmS3rlzJ2JjY7Fs2TJkZmYiKCgIUVFRKCwsrLd8SkoKxo4di0mTJiErKwvR0dGIjo7G6dOnjWVWrVqF+Ph4rF+/HqmpqbC3t0dUVBSqqqoAANnZ2TAYDNiwYQPOnDmDjz76COvXr8eiRYtapM6NUbfoATV1E9K6dXV3xE+v9keAzBFFZVqMWq/Awez6vw9J68djHI/5l8vl6Nu3Lz777DMAgMFggLe3N2bPno233nrrjvKjR4+GRqPB3r17jcf69euH4OBgrF+/HowxeHp6Yt68eXjjjTcAACqVCu7u7ti8eTPGjBlTbxyrV6/GunXrcPny5QbFrVarIZVKoVKp4OTkZG61zaI3MPT5z36oKqux+9UI9PZxadbPI4RwT1VZjenfZkBx+Rb4PGDx0O6Y2N8XPB6P69DIXTRHXuD0SVqn0yEjIwORkZHGY3w+H5GRkVAoFPVeo1AoTMoDQFRUlLF8bm4ulEqlSRmpVAq5XH7XewK1ibxdu7vPPdZqtVCr1SavlnLuphqqymo4iIUI9JK22OcSQrgjtbXBllfCMDrUGwYGvLP3LBYnnEY1TdFqUzhN0sXFxdDr9XB3dzc57u7uDqWy/qXylErlPcvX/TTnnhcvXsSnn36KadOm3TXWuLg4k/2zvb297125JlQ39Uru1w5CAec9FISQFiIS8vH+iEC8/XQ38HjA1tQ8TNx0DKrKaq5DIy2kzX/j37hxA0OGDMHIkSMxZcqUu5ZbuHAhVCqV8XXt2rUWizH54t9Tr2gpUELaHB6PhymP+mPj+FDYiQQ4crEYwz9PxtVbGq5DIy2A0yTt5uYGgUCAgoICk+MFBQWQyWT1XiOTye5Zvu5nQ+6Zn5+PQYMGISIiAhs3brxnrGKxGE5OTiavlqCrMeDYlRIAtIgJIW3ZE93dsWt6ODykElwq0iB6bTJSL9/iOizSzDhN0iKRCCEhIUhKSjIeMxgMSEpKQnh4eL3XhIeHm5QHgP379xvL+/n5QSaTmZRRq9VITU01ueeNGzfw2GOPISQkBJs2bQKfb5mNCievl6JCp0c7exEednfkOhxCCId6eErx88z+COooxe2Karz0VSq+T2+5Vj3S8jjPTLGxsfjiiy+wZcsWnDt3DjNmzIBGo8HEiRMBADExMVi4cKGx/Jw5c5CYmIg1a9YgOzsby5cvR3p6OmbNmgWgtmlo7ty5ePfdd7Fnzx6cOnUKMTEx8PT0RHR0NIB/ErSPjw8++OADFBUVQalU3rXPmkt1Td3hnV3B59OoTkLaug5OEuyYGo6nA2Wo1jPM/+Eklu85QwPKWinOd2kYPXo0ioqKsHTpUiiVSgQHByMxMdE48CsvL8/kKTciIgLbtm3D4sWLsWjRInTt2hUJCQno2bOnscz8+fOh0WgwdepUlJaWYsCAAUhMTIREIgFQ++R98eJFXLx4ER07djSJx9J2oaH1ugkh/8tWJMBnY/vgkw4X8EnSBWxOuYKzN9VY+2IftHcUcx0eaUKcz5O2Vi0xT7pSp0fQit+h0xtw8I3H4Odm3yyfQwixXr+fUSL2+xMo19bAQyrB+pdCEOTtzHVYbVKrmydN7i3j6m3o9AZ4SCXwdbXjOhxCiAV6socMCTP7w7+9PW6qqjBygwLfH6N+6taCkrQFS770z1KgtMoQIeRuunRwwM8z+yOymzt0NQbM//EkFvxwElXVtOWltaMkbcFSjFtTUn80IeTeHCU22Dg+BLFPPAQeD9iZfg3Ra5Nxuaic69DIA6AkbaHUVdU4db0UABDRhZI0IeT++HweXhvcFd9NksPNQYRsZRme+ywZe0/mcx0aaSRK0hYq9XIJDAzwd7OHh9SW63AIIVakfxc3/Pe1RxDm1w7l2hrM2paFZT+fhraGmr+tDSVpC1U39SqcmroJIY3g7iTBtslyvPpYZwDAFsVVDFubgouF1PxtTShJWyiFsT+a1usmhDSOUMDH/CEB2PRyX7SzF+HsTTWe+fQvbE29anFrQpD6UZK2QMXlWmQrywAA/fzvvn0mIYQ0xKCADkic8wge6eqGqmoD3t59GtO+zcBtjY7r0Mh9UJK2QHVP0d08nODqQKsHEUIeXAcnCbZMDMPiod1gI+Dh97MFGPLJYRy5UMx1aOQeKElbIJp6RQhpDnw+D5Mf8cfuV/ujc3t7FKi1eOmrVCxOOAWNtobr8Eg9KElbIAWt100IaUY9vaTYO/sRjO/XCQDw3dE8DPnksLEVj1gOStIW5kZpJa7cqoCAz0OYH/VHE0Kah61IgP9E98TWyXJ4OdviWkklxn5xFMv3nEGFjp6qLQUlaQuTcrH2KbpXRykcJTYcR0MIae36d3FD4txHMDbMBwCwOeUKhnz8F/66UMRxZASgJG1xFNQfTQhpYY4SG8QND8Q3r4TBUypBXkkFxn+Vhrk7slBcruU6vDaNkrQFYYwZN9XoT/OjCSEt7NGH2uO31x/FyxG+4PGAhOP5GLzmEHak5cFgoHnVXKAkbUEuF2tQoNZCJOSjTycXrsMhhLRBjhIbLH+uBxJe7Y8enk5QVVbjrZ9OYfRGBc7dVHMdXptDSdqC1E29CvFxgcRGwHE0hJC2LMjbGT/P7I/FQ7vBTiTAsSu3MTT+L7y9+xRKaBGUFkNJ2oLQ1CtCiCURCviY/Ig/9scOxNBeHjAwYGtqHh5bfRBfH8lFtd7AdYitHiVpC2EwsH8GjXWh/mhCiOXwcrbF2hf7YOfUfuju4QR1VQ3e2XsWQz4+jAPZBbQOeDOiJG0hzinVuF1RDXuRAL06SrkOhxBC7iD3d8UvswcgbnggXO1FuFSkwSub0zFyvQJpuSVch9cqUZK2EHVP0WF+7WAjoP8thBDLJODzMDbMBwfffAzTBvpDLOQj/eptjNqgwMub0nAmX8V1iK0KZQMLkfz3Iib9qambEGIFnCQ2WPhUNxyePwjj5D4Q8nn4M6cIQ+OPYObWTJzNp5HgTYGStAWo1huMTUXhNGiMEGJF3J0keG9YIP6IHYjngz3B4wH/PXUTT8f/hVc2H0PGVWoGfxCUpC3AyesqaHR6ONvZoJvMietwCCHEbL5u9vhkTG/8OucRPBfkCT4POJBdiBHrFBi9QYFD54togFkjUJK2AHVTr8L9XcHn8ziOhhBCGi9A5oT4sb1xYN5jGBvmDRsBD6m5JZjwdRqe+OgwvlVcQTlti9lglKQtQPJFmnpFCGldfN3sETe8Fw7PH4RX+vvBXiTAxcJyLPn5DPr9XxKW7zmDS0XlXIdp8XiM2h8aRa1WQyqVQqVSwcmp8U3UVdV69FrxO3Q1BiTNG4jO7R2aMEpCCLEMZVXV+DHjOr5RXMXlYo3xeJhfO7zQpyOeCpRZ/c5/TZUX/o2SdCM11f+MlIvFePHLVLg7iXF04WDweNTcTQhpvQwGhiMXi/GN4gqSsgtRl4EkNnwM6SHD8D4d0b+LGwRW2PXXHEla2CR3IY32712vKEETQlo7Pp+HRx9qj0cfao/80krszrqBHzOv43KRBgnH85FwPB9uDiI80d0dT3aXIaKLK8TCtruXASVpjtVtqkFTrwghbY2nsy1mDuqCVx/rjOPXSvFT5g3sOZGP4nIdtqddw/a0a7AXCfBYQAc82d0d4Z1d0cFRwnXYLcoiBo6tXbsWvr6+kEgkkMvlSEtLu2f5Xbt2ISAgABKJBIGBgdi3b5/JecYYli5dCg8PD9ja2iIyMhIXLlwwKVNSUoJx48bByckJzs7OmDRpEsrLW3YQQ1lVNU5er12dh5I0IaSt4vF46O3jgv9E90T64kh8N0mO8f06wd1JDI1Oj/+evIk5O44j7L0kPPnRISzfcwa/nVFCVVnNdejNjvM+6Z07dyImJgbr16+HXC7Hxx9/jF27diEnJwcdOnS4o3xKSgoeffRRxMXF4ZlnnsG2bduwcuVKZGZmomfPngCAlStXIi4uDlu2bIGfnx+WLFmCU6dO4ezZs5BIav8V9tRTT+HmzZvYsGEDqqurMXHiRPTt2xfbtm1rUNxN0fdwILsAr2xORydXOxx6c1Cj7kEIIa2VwcBw8oYKv51R4vD5Ipy9qca/MxaPB/i72SPQS4rAjs7o1VGK7h5OsBdz00jcKgeOyeVy9O3bF5999hkAwGAwwNvbG7Nnz8Zbb711R/nRo0dDo9Fg7969xmP9+vVDcHAw1q9fD8YYPD09MW/ePLzxxhsAAJVKBXd3d2zevBljxozBuXPn0L17dxw7dgyhoaEAgMTERDz99NO4fv06PD097xt3U/zP+M/es/jqSC7Ghvkgbnhgo+5BCCFtxW2NDkcv30LKpVtIuVSMS0WaO8rweYCXiy18Xe3h52Zv/NnRxRauDmI429o023oUrW7gmE6nQ0ZGBhYuXGg8xufzERkZCYVCUe81CoUCsbGxJseioqKQkJAAAMjNzYVSqURkZKTxvFQqhVwuh0KhwJgxY6BQKODs7GxM0AAQGRkJPp+P1NRUDBs2rAlreXchnVyQX1qJxwPubDEghBBiysVehKcCPfBUoAcAoLhci1M3VDh1XYWT11U4fUMFpboK10oqca2kEn9dKL7jHgI+D+3sRXC1F6GdvQh2IiHsRALYiQSIGx5ocQN4OU3SxcXF0Ov1cHd3Nznu7u6O7Ozseq9RKpX1llcqlcbzdcfuVeZ/m9KFQiHatWtnLPO/tFottFqt8b1a/eCLxz8d6IGn//5lI4QQYh43BzEGPdwBgx7+5/u8qEyLy0XluHJLg9ziClwp1uDKLQ1uqqqgqqyG3sBQVKZFUZnW5F4iIR/vj+jV0lW4Lxrd3UBxcXFYsWIF12EQQgi5h/aOYrR3FEPuf+dgXF2NAbcrdCgu16K4XIfSCh0qdHpotDUwWOiSIZwmaTc3NwgEAhQUFJgcLygogEwmq/camUx2z/J1PwsKCuDh4WFSJjg42FimsLDQ5B41NTUoKSm56+cuXLjQpJldrVbD29u7AbUkhBBiCURCPtydJHB3sp5pXJxOwRKJRAgJCUFSUpLxmMFgQFJSEsLDw+u9Jjw83KQ8AOzfv99Y3s/PDzKZzKSMWq1GamqqsUx4eDhKS0uRkZFhLHPgwAEYDAbI5fJ6P1csFsPJycnkRQghhDQrxrEdO3YwsVjMNm/ezM6ePcumTp3KnJ2dmVKpZIwxNn78ePbWW28ZyycnJzOhUMg++OADdu7cObZs2TJmY2PDTp06ZSzz/vvvM2dnZ/bzzz+zkydPsueff575+fmxyspKY5khQ4aw3r17s9TUVHbkyBHWtWtXNnbs2AbHrVKpGACmUqma4E+BEEKItWuOvMB5n/To0aNRVFSEpUuXQqlUIjg4GImJicaBX3l5eeDz/3ngj4iIwLZt27B48WIsWrQIXbt2RUJCgnGONADMnz8fGo0GU6dORWlpKQYMGIDExETjHGkA2Lp1K2bNmoXBgweDz+djxIgRiI+Pb7mKE0IIIffB+Txpa9Uc8+EIIYRYr+bICxaxLCghhBBC7kRJmhBCCLFQlKQJIYQQC8X5wDFrVdeV3xQrjxFCCLF+dfmgKYd6UZJupLKyMgCgBU0IIYSYKCsrg1QqbZJ70ejuRjIYDMjPz4ejo2OjF2SvW7Xs2rVrrWaEONXJOlCdrAPVyTrU1SkvLw88Hg+enp4mU4cfBD1JNxKfz0fHjh2b5F6tcQUzqpN1oDpZB6qTdZBKpU1eJxo4RgghhFgoStKEEEKIhaIkzSGxWIxly5ZBLBZzHUqToTpZB6qTdaA6WYfmrBMNHCOEEEIsFD1JE0IIIRaKkjQhhBBioShJE0IIIRaKkjSH1q5dC19fX0gkEsjlcqSlpXEdUoPFxcWhb9++cHR0RIcOHRAdHY2cnByTMlVVVZg5cyZcXV3h4OCAESNGoKCggKOIzfP++++Dx+Nh7ty5xmPWWJ8bN27gpZdegqurK2xtbREYGIj09HTjecYYli5dCg8PD9ja2iIyMhIXLlzgMOJ70+v1WLJkCfz8/GBra4vOnTvjP//5j8kyjJZep8OHD+PZZ5+Fp6cneDweEhISTM43JP6SkhKMGzcOTk5OcHZ2xqRJk1BeXt6CtTB1rzpVV1djwYIFCAwMhL29PTw9PRETE4P8/HyTe1hTnf7X9OnTwePx8PHHH5scb4o6UZLmyM6dOxEbG4tly5YhMzMTQUFBiIqKQmFhIdehNcihQ4cwc+ZMHD16FPv370d1dTWefPJJaDQaY5nXX38dv/zyC3bt2oVDhw4hPz8fw4cP5zDqhjl27Bg2bNiAXr16mRy3tvrcvn0b/fv3h42NDX799VecPXsWa9asgYuLi7HMqlWrEB8fj/Xr1yM1NRX29vaIiopCVVUVh5Hf3cqVK7Fu3Tp89tlnOHfuHFauXIlVq1bh008/NZax9DppNBoEBQVh7dq19Z5vSPzjxo3DmTNnsH//fuzduxeHDx/G1KlTW6oKd7hXnSoqKpCZmYklS5YgMzMTP/30E3JycvDcc8+ZlLOmOv3b7t27cfToUXh6et5xrknqxAgnwsLC2MyZM43v9Xo98/T0ZHFxcRxG1XiFhYUMADt06BBjjLHS0lJmY2PDdu3aZSxz7tw5BoApFAquwryvsrIy1rVrV7Z//342cOBANmfOHMaYddZnwYIFbMCAAXc9bzAYmEwmY6tXrzYeKy0tZWKxmG3fvr0lQjTb0KFD2SuvvGJybPjw4WzcuHGMMeurEwC2e/du4/uGxH/27FkGgB07dsxY5tdff2U8Ho/duHGjxWK/m/+tU33S0tIYAHb16lXGmPXW6fr168zLy4udPn2aderUiX300UfGc01VJ3qS5oBOp0NGRgYiIyONx/h8PiIjI6FQKDiMrPFUKhUAoF27dgCAjIwMVFdXm9QxICAAPj4+Fl3HmTNnYujQoSZxA9ZZnz179iA0NBQjR45Ehw4d0Lt3b3zxxRfG87m5uVAqlSZ1kkqlkMvlFluniIgIJCUl4fz58wCAEydO4MiRI3jqqacAWGed/q0h8SsUCjg7OyM0NNRYJjIyEnw+H6mpqS0ec2OoVCrweDw4OzsDsM46GQwGjB8/Hm+++SZ69Ohxx/mmqhOt3c2B4uJi6PV6uLu7mxx3d3dHdnY2R1E1nsFgwNy5c9G/f3/07NkTAKBUKiESiYx/Ceu4u7tDqVRyEOX97dixA5mZmTh27Ngd56yxPpcvX8a6desQGxuLRYsW4dixY3jttdcgEokwYcIEY9z1/R5aap3eeustqNVqBAQEQCAQQK/X47333sO4ceMAwCrr9G8NiV+pVKJDhw4m54VCIdq1a2cVdayqqsKCBQswduxY4zrX1linlStXQigU4rXXXqv3fFPViZI0eWAzZ87E6dOnceTIEa5DabRr165hzpw52L9/PyQSCdfhNAmDwYDQ0FD83//9HwCgd+/eOH36NNavX48JEyZwHF3jfP/999i6dSu2bduGHj164Pjx45g7dy48PT2ttk5tSXV1NUaNGgXGGNatW8d1OI2WkZGBTz75BJmZmY3eBbGhqLmbA25ubhAIBHeMDC4oKIBMJuMoqsaZNWsW9u7di4MHD5rsCiaTyaDT6VBaWmpS3lLrmJGRgcLCQvTp0wdCoRBCoRCHDh1CfHw8hEIh3N3drao+AODh4YHu3bubHOvWrRvy8vIAwBi3Nf0evvnmm3jrrbcwZswYBAYGYvz48Xj99dcRFxcHwDrr9G8NiV8mk90xwLSmpgYlJSUWXce6BH316lXs37/fZLcoa6vTX3/9hcLCQvj4+Bi/L65evYp58+bB19cXQNPViZI0B0QiEUJCQpCUlGQ8ZjAYkJSUhPDwcA4jazjGGGbNmoXdu3fjwIED8PPzMzkfEhICGxsbkzrm5OQgLy/PIus4ePBgnDp1CsePHze+QkNDMW7cOON/W1N9AKB///53TIs7f/48OnXqBADw8/ODTCYzqZNarUZqaqrF1qmiouKOfXoFAgEMBgMA66zTvzUk/vDwcJSWliIjI8NY5sCBAzAYDJDL5S0ec0PUJegLFy7gjz/+gKurq8l5a6vT+PHjcfLkSZPvC09PT7z55pv47bffADRhnRo/3o08iB07djCxWMw2b97Mzp49y6ZOncqcnZ2ZUqnkOrQGmTFjBpNKpezPP/9kN2/eNL4qKiqMZaZPn858fHzYgQMHWHp6OgsPD2fh4eEcRm2ef4/uZsz66pOWlsaEQiF777332IULF9jWrVuZnZ0d++6774xl3n//febs7Mx+/vlndvLkSfb8888zPz8/VllZyWHkdzdhwgTm5eXF9u7dy3Jzc9lPP/3E3Nzc2Pz5841lLL1OZWVlLCsri2VlZTEA7MMPP2RZWVnGkc4NiX/IkCGsd+/eLDU1lR05coR17dqVjR07lqsq3bNOOp2OPffcc6xjx47s+PHjJt8XWq3WKutUn/8d3c1Y09SJkjSHPv30U+bj48NEIhELCwtjR48e5TqkBgNQ72vTpk3GMpWVlezVV19lLi4uzM7Ojg0bNozdvHmTu6DN9L9J2hrr88svv7CePXsysVjMAgIC2MaNG03OGwwGtmTJEubu7s7EYjEbPHgwy8nJ4Sja+1Or1WzOnDnMx8eHSSQS5u/vz95++22TL3tLr9PBgwfr/bszYcIExljD4r916xYbO3Ysc3BwYE5OTmzixImsrKyMg9rUuledcnNz7/p9cfDgQausU33qS9JNUSfaBYsQQgixUNQnTQghhFgoStKEEEKIhaIkTQghhFgoStKEEEKIhaIkTQghhFgoStKEEEKIhaIkTQghhFgoStKEEEKIhaIkTQghhFgoStKEEKOioiLMmDEDPj4+EIvFkMlkiIqKQnJyMgCAx+MhISGB2yAJaUNoP2lCiNGIESOg0+mwZcsW+Pv7o6CgAElJSbh16xbXoRHSJtHa3YQQAEBpaSlcXFzw559/YuDAgXec9/X1xdWrV43vO3XqhCtXrgAAfv75Z6xYsQJnz56Fp6cnJkyYgLfffhtCYe1zAI/Hw+eff449e/bgzz//hIeHB1atWoUXXnihRepGiLWi5m5CCADAwcEBDg4OSEhIgFarveP8sWPHAACbNm3CzZs3je//+usvxMTEYM6cOTh79iw2bNiAzZs347333jO5fsmSJRgxYgROnDiBcePGYcyYMTh37lzzV4wQK0ZP0oQQox9//BFTpkxBZWUl+vTpg4EDB2LMmDHo1asXgNon4t27dyM6Otp4TWRkJAYPHoyFCxcaj3333XeYP38+8vPzjddNnz4d69atM5bp168f+vTpg88//7xlKkeIFaInaUKI0YgRI5Cfn489e/ZgyJAh+PPPP9GnTx9s3rz5rtecOHEC77zzjvFJ3MHBAVOmTMHNmzdRUVFhLBceHm5yXXh4OD1JE3IfNHCMEGJCIpHgiSeewBNPPIElS5Zg8uTJWLZsGV5++eV6y5eXl2PFihUYPnx4vfcihDQePUkTQu6pe/fu0Gg0AAAbGxvo9XqT83369EFOTg66dOlyx4vP/+cr5ujRoybXHT16FN26dWv+ChBixehJmhACALh16xZGjhyJV155Bb169YKjoyPS09OxatUqPP/88wBqR3gnJSWhf//+EIvFcHFxwdKlS/HMM8/Ax8cHL7zwAvh8Pk6cOIHTp0/j3XffNd5/165dCA0NxYABA7B161akpaXhq6++4qq6hFgFGjhGCAEAaLVaLF++HL///jsuXbqE6upqeHt7Y+TIkVi0aBFsbW3xyy+/IDY2FleuXIGXl5dxCtZvv/2Gd955B1lZWbCxsUFAQAAmT56MKVOmAKgdOLZ27VokJCTg8OHD8PDwwMqVKzFq1CgOa0yI5aMkTQhpdvWNCieE3B/1SRNCCCEWipI0IYQQYqFo4BghpNlRrxohjUNP0oQQQoiFoiRNCCGEWChK0oQQQoiFoiRNCCGEWChK0oQQQoiFoiRNCCGEWChK0oQQQoiFoiRNCCGEWChK0oQQQoiF+n9EYrqk9d2ISwAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(5, 3))\n", "plt.ylabel(\"Learning rate\")\n", "plt.xlabel(\"Step\")\n", "plt.plot(range(total_training_steps), track_lrs)\n", "plt.tight_layout(); plt.savefig(\"2.pdf\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "e7512808-b48d-4146-86a1-5931b1e3aec1", "metadata": {}, "source": [ "## D.3 Gradient clipping" ] }, { "cell_type": "markdown", "id": "c0a74f76-8d2b-4974-a03c-d645445cdc21", "metadata": {}, "source": [ "- Gradient clipping is yet another technique used to stabilize the training when training LLMs\n", "- By setting a threshold, gradients exceeding this limit are scaled down to a maximum magnitude to ensure that the updates to the model's parameters during backpropagation remain within a manageable range\n", "- For instance, using the `max_norm=1.0` setting in PyTorch's `clip_grad_norm_` method means that the norm of the gradients is clipped such that their maximum norm does not exceed 1.0\n", "- the \"norm\" refers to a measure of the gradient vector's length (or magnitude) in the parameter space of the model\n", "- Specifically, it's the L2 norm, also known as the Euclidean norm\n", "- Mathematically, for a vector $\\mathbf{v}$ with components $\\mathbf{v} = [v_1, v_2, \\ldots, v_n]$, the L2 norm is defined as:\n", "$$\n", "\\| \\mathbf{v} \\|_2 = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}\n", "$$" ] }, { "cell_type": "markdown", "id": "d44838a6-4322-47b2-a935-c00d3a88355f", "metadata": {}, "source": [ "- The L2 norm is calculated similarly for matrices.\n", "- Let's assume our gradient matrix is:\n", "$$\n", "G = \\begin{bmatrix}\n", "1 & 2 \\\\\n", "2 & 4\n", "\\end{bmatrix}\n", "$$\n", "\n", "- And we want to clip these gradients with a `max_norm` of 1.\n", "\n", "- First, we calculate the L2 norm of these gradients:\n", "$$\n", "\\|G\\|_2 = \\sqrt{1^2 + 2^2 + 2^2 + 4^2} = \\sqrt{25} = 5\n", "$$\n", "\n", "- Since $\\|G\\|_2 = 5$ is greater than our `max_norm` of 1, we need to scale down the gradients so that their norm is exactly 1. The scaling factor is calculated as $\\frac{max\\_norm}{\\|G\\|_2} = \\frac{1}{5}$.\n", "\n", "- Therefore, the scaled gradient matrix $G'$ will be as follows:\n", "$$\n", "G' = \\frac{1}{5} \\times G = \\begin{bmatrix}\n", "\\frac{1}{5} & \\frac{2}{5} \\\\\n", "\\frac{2}{5} & \\frac{4}{5}\n", "\\end{bmatrix}\n", "$$" ] }, { "cell_type": "markdown", "id": "eeb0c3c1-2cff-46f5-8127-24412184428c", "metadata": {}, "source": [ "- Let's see this in action\n", "- First, we initialize a new model and calculate the loss for a training batch like we would do in the regular training loop" ] }, { "cell_type": "code", "execution_count": 9, "id": "e199e1ff-58c4-413a-855e-5edbe9292649", "metadata": {}, "outputs": [], "source": [ "from previous_chapters import calc_loss_batch\n", "\n", "torch.manual_seed(123)\n", "model = GPTModel(GPT_CONFIG_124M)\n", "\n", "loss = calc_loss_batch(input_batch, target_batch, model, device)\n", "loss.backward()" ] }, { "cell_type": "markdown", "id": "76b60f3a-15ec-4846-838d-fdef3df99899", "metadata": {}, "source": [ "- If we call `.backward()`, PyTorch will calculate the gradients and store them in a `.grad` attribute for each weight (parameter) matrix\n", "- Let's define a utility function to calculate the highest gradient based on all model weights" ] }, { "cell_type": "code", "execution_count": 10, "id": "e70729a3-24d1-411d-a002-2529cd3a8a9e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor(0.0411)\n" ] } ], "source": [ "def find_highest_gradient(model):\n", " max_grad = None\n", " for param in model.parameters():\n", " if param.grad is not None:\n", " grad_values = param.grad.data.flatten()\n", " max_grad_param = grad_values.max()\n", " if max_grad is None or max_grad_param > max_grad:\n", " max_grad = max_grad_param\n", " return max_grad\n", "\n", "print(find_highest_gradient(model))" ] }, { "cell_type": "markdown", "id": "734f30e6-6b24-4d4b-ae91-e9a4b871113f", "metadata": {}, "source": [ "- Applying gradient clipping, we can see that the largest gradient is now substantially smaller:" ] }, { "cell_type": "code", "execution_count": 11, "id": "fa81ef8b-4280-400f-a93e-5210f3e62ff0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor(0.0185)\n" ] } ], "source": [ "torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n", "print(find_highest_gradient(model))" ] }, { "cell_type": "markdown", "id": "b62c2af0-dac3-4742-be4b-4292c6753099", "metadata": {}, "source": [ "## D.4 The modified training function" ] }, { "cell_type": "markdown", "id": "76715332-94ec-4185-922a-75cb420819d5", "metadata": {}, "source": [ "- Now let's add the three concepts covered above (learning rate warmup, cosine decay, and gradient clipping) to the `train_model_simple` function covered in chapter 5 to create the more sophisticated `train_model` function below:" ] }, { "cell_type": "code", "execution_count": 12, "id": "46eb9c84-a293-4016-a523-7ad726e171e9", "metadata": {}, "outputs": [], "source": [ "from previous_chapters import evaluate_model, generate_and_print_sample\n", "\n", "\n", "def train_model(model, train_loader, val_loader, optimizer, device, n_epochs,\n", " eval_freq, eval_iter, start_context, warmup_steps=10,\n", " initial_lr=3e-05, min_lr=1e-6):\n", "\n", " train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []\n", " tokens_seen, global_step = 0, -1\n", "\n", " # Retrieve the maximum learning rate from the optimizer\n", " peak_lr = optimizer.param_groups[0][\"lr\"]\n", "\n", " # Calculate the total number of iterations in the training process\n", " total_training_steps = len(train_loader) * n_epochs\n", "\n", " # Calculate the learning rate increment during the warmup phase\n", " lr_increment = (peak_lr - initial_lr) / warmup_steps\n", "\n", " for epoch in range(n_epochs):\n", " model.train()\n", " for input_batch, target_batch in train_loader:\n", " optimizer.zero_grad()\n", " global_step += 1\n", "\n", " # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n", " if global_step < warmup_steps:\n", " # Linear warmup\n", " lr = initial_lr + global_step * lr_increment \n", " else:\n", " # Cosine annealing after warmup\n", " progress = ((global_step - warmup_steps) / \n", " (total_training_steps - warmup_steps))\n", " lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n", "\n", " # Apply the calculated learning rate to the optimizer\n", " for param_group in optimizer.param_groups:\n", " param_group[\"lr\"] = lr\n", " track_lrs.append(lr) # Store the current learning rate\n", "\n", " # Calculate and backpropagate the loss\n", " loss = calc_loss_batch(input_batch, target_batch, model, device)\n", " loss.backward()\n", "\n", " # Apply gradient clipping after the warmup phase to avoid exploding gradients\n", " if global_step > warmup_steps:\n", " torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n", " \n", " optimizer.step()\n", " tokens_seen += input_batch.numel()\n", "\n", " # Periodically evaluate the model on the training and validation sets\n", " if global_step % eval_freq == 0:\n", " train_loss, val_loss = evaluate_model(\n", " model, train_loader, val_loader,\n", " device, eval_iter\n", " )\n", " train_losses.append(train_loss)\n", " val_losses.append(val_loss)\n", " track_tokens_seen.append(tokens_seen)\n", " # Print the current losses\n", " print(f\"Ep {epoch+1} (Iter {global_step:06d}): \"\n", " f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n", "\n", " # Generate and print a sample from the model to monitor progress\n", " generate_and_print_sample(\n", " model, train_loader.dataset.tokenizer,\n", " device, start_context\n", " )\n", "\n", " return train_losses, val_losses, track_tokens_seen, track_lrs" ] }, { "cell_type": "code", "execution_count": 13, "id": "55fcd247-ba9d-4b93-a757-0f7ce04fee41", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Ep 1 (Iter 000000): Train loss 10.934, Val loss 10.939\n", "Ep 1 (Iter 000005): Train loss 8.529, Val loss 8.843\n", "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n", "Ep 2 (Iter 000010): Train loss 6.400, Val loss 6.825\n", "Ep 2 (Iter 000015): Train loss 6.116, Val loss 6.861\n", "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n", "Ep 3 (Iter 000020): Train loss 6.033, Val loss 6.904\n", "Ep 3 (Iter 000025): Train loss 5.915, Val loss 6.851\n", "Every effort moves you, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the\n", "Ep 4 (Iter 000030): Train loss 5.375, Val loss 6.815\n", "Ep 4 (Iter 000035): Train loss 5.001, Val loss 6.701\n", "Every effort moves you. \n", "Ep 5 (Iter 000040): Train loss 4.993, Val loss 6.641\n", "Every effort moves you of the \n", "Ep 6 (Iter 000045): Train loss 3.872, Val loss 6.508\n", "Ep 6 (Iter 000050): Train loss 3.636, Val loss 6.466\n", "Every effort moves you know to see and in spite of his pictures--I looked up the end of his pictures--and by me to see it was. \n", "Ep 7 (Iter 000055): Train loss 3.803, Val loss 6.266\n", "Ep 7 (Iter 000060): Train loss 1.846, Val loss 6.334\n", "Every effort moves you know it was his pictures--I must have to the fact the picture--had not till, when I felt to me to have that, in the moment--as Jack himself, as once one had been the fact, and his own he was his\n", "Ep 8 (Iter 000065): Train loss 1.565, Val loss 6.278\n", "Ep 8 (Iter 000070): Train loss 2.342, Val loss 6.286\n", "Every effort moves you know,\" was one of the axioms he said--as such--had not to my work, and in fact, and that, in the moment--as Jack himself, as once one had longed to say: \"Be dissatisfied with your\n", "Ep 9 (Iter 000075): Train loss 1.043, Val loss 6.323\n", "Ep 9 (Iter 000080): Train loss 0.780, Val loss 6.375\n", "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" \"Oh, I felt him back his head to look up at the honour being _mine_--because he had always his\n", "Ep 10 (Iter 000085): Train loss 0.493, Val loss 6.467\n", "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" \"Oh, and I looked at the donkey again. I saw that, and down the room, when I\n", "Ep 11 (Iter 000090): Train loss 0.434, Val loss 6.547\n", "Ep 11 (Iter 000095): Train loss 0.250, Val loss 6.602\n", "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He placed them at my elbow and as I turned, my eye fell on a small picture\n", "Ep 12 (Iter 000100): Train loss 0.242, Val loss 6.631\n", "Ep 12 (Iter 000105): Train loss 0.237, Val loss 6.630\n", "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", "Ep 13 (Iter 000110): Train loss 0.144, Val loss 6.664\n", "Ep 13 (Iter 000115): Train loss 0.118, Val loss 6.677\n", "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", "Ep 14 (Iter 000120): Train loss 0.109, Val loss 6.686\n", "Ep 14 (Iter 000125): Train loss 0.109, Val loss 6.698\n", "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", "Ep 15 (Iter 000130): Train loss 0.101, Val loss 6.707\n", "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n" ] } ], "source": [ "torch.manual_seed(123)\n", "model = GPTModel(GPT_CONFIG_124M)\n", "model.to(device)\n", "\n", "peak_lr = 5e-4\n", "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)\n", "\n", "n_epochs = 15\n", "train_losses, val_losses, tokens_seen, lrs = train_model(\n", " model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,\n", " eval_freq=5, eval_iter=1, start_context=\"Every effort moves you\",\n", " warmup_steps=10, initial_lr=1e-5, min_lr=1e-5\n", ")" ] }, { "cell_type": "markdown", "id": "827e8d5e-0872-4b90-98ac-200c80ee2d53", "metadata": {}, "source": [ "- Looking at the results above, we can see that the model starts out generating incomprehensible strings of words, whereas, towards the end, it's able to produce grammatically more or less correct sentences\n", "- If we were to check a few passages it writes towards the end, we would find that they are contained in the training set verbatim -- it simply memorizes the training data\n", "- Note that the overfitting here occurs because we have a very, very small training set, and we iterate over it so many times\n", " - The LLM training here primarily serves educational purposes; we mainly want to see that the model can learn to produce coherent text\n", " - Instead of spending weeks or months on training this model on vast amounts of expensive hardware, we load the pretrained weights" ] }, { "cell_type": "markdown", "id": "9decec45-4fdf-4ff6-85a7-1806613f8af7", "metadata": {}, "source": [ "- A quick check that the learning rate behaves as intended" ] }, { "cell_type": "code", "execution_count": 14, "id": "d8ebb8d2-8308-4a83-a2a6-730c3bf84452", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEmCAYAAACdy8LUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABQqklEQVR4nO3deVxUVf8H8M8dhhnWmRERBpTN3BUVRSZcM0k0yq1SiVzQxMxSs0UtlceeDLc2zdzyJ/Zkbi1quBSBubEK7ghikuAyICADyD5zfn8QtyZRGR28M/B9v17z4uGe7535Hp7kyzn3nHs5xhgDIYQQQsyKSOgECCGEEGI4KuCEEEKIGaICTgghhJghKuCEEEKIGaICTgghhJghKuCEEEKIGaICTgghhJghKuCEEEKIGRILnUBTptPpcOPGDdjb24PjOKHTIYQQIgDGGEpKSuDq6gqRyHjjZirgjejGjRtwc3MTOg1CCCEmICcnB23atDHa+1EBb0T29vYAav9Pk8lkAmdDCCFECMXFxXBzc+NrgrFQAW9EddPmMpmMCjghhDRzxr6USovYCCGEEDNEBZwQQggxQ1TACSGEEDMkeAFfu3YtPD09YWVlBZVKhaSkpPvG7969G506dYKVlRW8vb1x4MABvXbGGBYvXgwXFxdYW1sjICAAmZmZejFLly5F3759YWNjA4VCUe/nZGdnIygoCDY2NnBycsK7776LmpqaR+orIYQQYiyCFvCdO3di7ty5CA8PR2pqKnr06IHAwEDk5eXVGx8XF4fg4GBMnToVp06dwqhRozBq1CicP3+ej1mxYgVWr16N9evXIzExEba2tggMDERFRQUfU1VVhZdeegkzZsyo93O0Wi2CgoJQVVWFuLg4bN26FZGRkVi8eLFxfwCEEELIw2IC8vPzYzNnzuS/12q1zNXVlUVERNQbP3bsWBYUFKR3TKVSsenTpzPGGNPpdEypVLKVK1fy7UVFRUwqlbLt27ff9X5btmxhcrn8ruMHDhxgIpGIqdVq/ti6deuYTCZjlZWVDe6fRqNhAJhGo2nwOYQQQpqWxqoFgm0jq6qqQkpKChYsWMAfE4lECAgIQHx8fL3nxMfHY+7cuXrHAgMDsWfPHgBAVlYW1Go1AgIC+Ha5XA6VSoX4+HiMHz++QbnFx8fD29sbzs7Oep8zY8YMXLhwAT4+PvWeV1lZicrKSv774uLiBn2eKUi8UoCIg+nQ6hgsLThIxCJYWoggFYsgs7KEwkYCB9u6rxI42UvRpoUNnOylEInoLnOEEPK4CVbA8/PzodVq9YokADg7OyM9Pb3ec9Rqdb3xarWab687dq+YhrjX5/zzM+oTERGBJUuWNPhzTMn/Eq7idE6RwedJLERo3cIabVpYw83BBh2d7dFRaY9OSnsobCTGT5QQQggAupGLUS1YsEBvhqDu7jvm4NrtcgDAm0+3Q1dXOaq1OlRrdais0UFTXo3bZVUoulONwrIq3L5TBXVxBW5qKlCl1SEr/w6y8u/c9Z7OMik6KWXo6aZAb48W6OmugMzK8nF3jRBCmiTBCrijoyMsLCyQm5urdzw3NxdKpbLec5RK5X3j677m5ubCxcVFL6Znz54Nzk2pVN61Gr7uc++VGwBIpVJIpdIGf44pqSvggV2V6NZa3qBzarQ6qIsrkFNYjmu3y5CVfwcZ6hKkq0twvagcucWVyC2+hSOXbgEAOA7o4GSPXh4t8GRbB/Rv54iWdub58yKEEKEJVsAlEgl69+6NmJgYjBo1CkDt07tiYmLwxhtv1HuOv78/YmJiMGfOHP5YdHQ0/P39AQBeXl5QKpWIiYnhC3ZxcTESExPvueL8Xp+zdOlS5OXlwcnJif8cmUyGLl26GN5ZE1dRrUV+ae21+zYtrBt8nthChDYtbNCmhQ2AlnptJRXVuJRbggs3inEquwgpV28ju7AMGbklyMgtwfakbABAt9YyDGjfCgPaO6K3RwtIxRZG6xchhDRlgk6hz507F5MmTYKvry/8/Pzw+eef486dOwgNDQUATJw4Ea1bt0ZERAQAYPbs2Rg0aBA++eQTBAUFYceOHTh58iQ2btwIoPY+s3PmzMFHH32E9u3bw8vLC4sWLYKrqyv/RwJQu8e7sLAQ2dnZ0Gq1OH36NACgXbt2sLOzw9ChQ9GlSxdMmDABK1asgFqtxsKFCzFz5kyzHWHfT93o204qhtzaOFPc9laW6O3hgN4eDphY+/cVbpVUIjX7Nk7+WYjjlwtw8WYxzl+vfa37/Q/YS8V4urMThnVVYlDHVrCR0BUeQgi5F0F/Q44bNw63bt3C4sWLoVar0bNnTxw6dIhfMJadna337NS+ffviu+++w8KFC/H++++jffv22LNnD7p168bHvPfee7hz5w7CwsJQVFSE/v3749ChQ7CysuJjFi9ejK1bt/Lf160qP3z4MJ566ilYWFggKioKM2bMgL+/P2xtbTFp0iR8+OGHjf0jEcS122UAakffjfnc8lb2UgR2VSKwa+1liLySChzPzMexzHwcy7yF/NIq7D19A3tP34CVpQiDOrTCs94ueKaLMxVzQgj5F44xxoROoqkqLi6GXC6HRqMx6aeRfZtwFQv3nEdAZyd8PamPIDnodAyncopw6PxNHDyv5mcFAMBWYoHh3i54oVcbqLwcaNsaIcSsNFYtoGEN4Ytl7bVsYYhEHHp7tEBvjxZ4/9nOuHCjGIfOq7HvzA1kF5bh+5Rr+D7lGlorrDGmV2uM9XWDm4Nw+RJCiNCogBO9KXRTwHEcurWWo1trOd4e2gEpV2/jh9RriDpzE9eLyrEm9jK+PHwZT3d0wgR/Dwxs34pG5YSQZocKODGJEfi9cBwHX08H+Ho6IPz5rohOy8XO5Bwcv5yPmPQ8xKTnwaOlDV5ReWCsrxvkNrTPnBDSPFABJ/8o4KYxAr8XK0sLPN/DFc/3cMUft0qxLSEbu1NycLWgDEsPXMRnv11CsJ87pvT3QmuFafeFEEIeFS1ia0TmsIitolqLTosOAQDOLB5qdiPYsqoa7Dt9A5FxfyJdXQIAEIs4jOjhirBBbdFJaZo/d0JI80GL2EijqBt920vFkFmb338ONhIxxvu5Y1wfNxzNzMeGI38g7o8C/HjqOn48dR2DO7bCnIAO6OGmEDpVQggxKvP7jU2Mqm4BW+tG3gPe2DiOw6AOrTCoQyucvVaEDUev4OC5mziccQuHM25hSCcnvPVMhwbfJpYQQkyd6MEhpCkz5QVsD6t7GwXWvtwLsW8/hRd6tYGIA2LS8/DcmuOY9s1JXLihETpFQgh5ZFTAmzlzWcD2MDwdbfHJ2B74be4gjPZpDREHRKflImj1cczecQo5hWVCp0gIIQ+NCngzZ2p7wBtD21Z2+GxcT/z61iCM6OEKANh7+gaGfHIES/enoaisSuAMCSHEcFTAm7mmOIV+L+2c7LA62AdRb/ZHv3YtUaXVYdOxLAxa+Ts2Hb2Cyhqt0CkSQkiDUQFv5pryFPq9dGstx7dTVYgM7YNOSntoyqux9MBFBH52FLHpuQ9+A0IIMQFUwJux8qq/nwPu1gxG4P/EcRye6uiE/bMGYOWL3eFkL8WfBWWYEnkSUyKT8Wf+HaFTJISQ+6IC3oxdL6q9/m2ue8CNwULE4SVfN8S+8xSmD2oLSwsOsel5GPrZUaw4lI47lTVCp0gIIfWiAt6M5fw1fW7ue8CNwU4qxoLhnXFozkAM7NAKVVodvvr9Dwz97ChiLtK0OiHE9FABb8aa0wK2hnqilR22hvbBxgm90aaFNa4XlWPq1pOYuS0VecUVQqdHCCE8KuDNWHPYQvYwOI7D0K5K/PrWQEwf2BYWIg77z93EkE+P4NuEq9Dp6PEBhBDhUQFvxprjCnRD2EjEWPBsZ+x7ox96tJGjpKIGC/ecx9gN8bhyq1To9AghzRwV8GaMptAbpqurHD++3g//eb4LbCUWOHn1NoZ/cQxfH7sCLY3GCSECoQLejF2nKfQGsxBxmNzPC7/OHYQB7R1RWaPDR/svYhyNxgkhAqEC3kzV7gGvvYVoc9sD/ihaK6zxzRQ/fDza+67ROF0bJ4Q8TlTAmyl+D7iVGHIbS4GzMS8cx+FllTt+eWsg+rf7ezQe8nUibhSVC50eIaSZoALeTOXQ9e9H1qaFDf431Q9LR3eDtaUF4q8UYNjnRxF19obQqRFCmgEq4M0UrUA3Do7jEKLywIHZA9CjjRzFFTV447tTmLvzNEoqqoVOjxDShFEBb6ZoD7hxeTna4vsZfTHr6XYQccCPp65j+BfHkHL1ttCpEUKaKCrgzRRtITM+SwsR5g7tiF3T/eHmYI1rt8sxbkM8Nhz5gxa4EUKMjgp4M0VT6I3H19MBB2YNwIgerqjRMUQcTMer35zE7TtVQqdGCGlCqIA3U7QHvHHZW1nii/E9ETHGGxKxCLHpeXh29TGkXC0UOjVCSBNBBbwZ+ucecJpCbzwcxyHYzx17Xu+Hto62uKmpwNgNCVhPU+qEECOgAt4M6e0Bt6Y94I2ti6sM+97sj5E9XaHVMSw7mI4pW5NRSFPqhJBHQAW8GaI94I+fnVSMz8f1xLIx3pCKRfg94xaepVXqhJBHIHgBX7t2LTw9PWFlZQWVSoWkpKT7xu/evRudOnWClZUVvL29ceDAAb12xhgWL14MFxcXWFtbIyAgAJmZmXoxhYWFCAkJgUwmg0KhwNSpU1Faqn8/619++QVPPvkk7O3t0apVK7zwwgv4888/jdJnoV0rpOvfQuA4DuP93LFnZu2Uurq4AuM3xuO7xGyhUyOEmCFBC/jOnTsxd+5chIeHIzU1FT169EBgYCDy8vLqjY+Li0NwcDCmTp2KU6dOYdSoURg1ahTOnz/Px6xYsQKrV6/G+vXrkZiYCFtbWwQGBqKiooKPCQkJwYULFxAdHY2oqCgcPXoUYWFhfHtWVhZGjhyJp59+GqdPn8Yvv/yC/Px8jBkzpvF+GI8RrUAXVmeX2in1Z72VqNYyvP/TOSz48Swqa7RCp0YIMSdMQH5+fmzmzJn891qtlrm6urKIiIh648eOHcuCgoL0jqlUKjZ9+nTGGGM6nY4plUq2cuVKvr2oqIhJpVK2fft2xhhjaWlpDABLTk7mYw4ePMg4jmPXr19njDG2e/duJhaLmVar5WP27dvHOI5jVVVVDe6fRqNhAJhGo2nwOY/D69+mMI95UezrY1eETqVZ0+l0bO3hTOY5P4p5zItiI788zm4WlQudFiHEyBqrFgg2Aq+qqkJKSgoCAgL4YyKRCAEBAYiPj6/3nPj4eL14AAgMDOTjs7KyoFar9WLkcjlUKhUfEx8fD4VCAV9fXz4mICAAIpEIiYmJAIDevXtDJBJhy5Yt0Gq10Gg0+N///oeAgABYWpr/oi+6C5tp4DgOrz/VDlsm94Hc2hKnc4rw3JrjSP6TtpoRQh5MsAKen58PrVYLZ2dnvePOzs5Qq9X1nqNWq+8bX/f1QTFOTk567WKxGA4ODnyMl5cXfv31V7z//vuQSqVQKBS4du0adu3add8+VVZWori4WO9limgK3bQ81dEJ+97oh05Ke+SXViJ4YwK+if8TjNFWM0LIvQm+iM0UqdVqTJs2DZMmTUJycjKOHDkCiUSCF1988b6/VCMiIiCXy/mXm5vbY8y6YcqqalBwh/aAmxqPlrb48fW+eK67C2p0DIv3XsC7359FRTVdFyeE1E+wAu7o6AgLCwvk5ubqHc/NzYVSqaz3HKVSed/4uq8Pivn3IrmamhoUFhbyMWvXroVcLseKFSvg4+ODgQMH4ttvv0VMTAw/zV6fBQsWQKPR8K+cnJwH/Rgeu+t/jb5pD7jpsZGIsSbYB+8/2wkiDvg+5Rpe3pSA/NJKoVMjhJggwQq4RCJB7969ERMTwx/T6XSIiYmBv79/vef4+/vrxQNAdHQ0H+/l5QWlUqkXU1xcjMTERD7G398fRUVFSElJ4WNiY2Oh0+mgUqkAAGVlZRCJ9H80FhYWfI73IpVKIZPJ9F6mhh5iYto4jkPYwCfwzRQVZFZipGYXYeSXJ5CuNs3LMYQQARl1SZyBduzYwaRSKYuMjGRpaWksLCyMKRQKplarGWOMTZgwgc2fP5+PP3HiBBOLxWzVqlXs4sWLLDw8nFlaWrJz587xMcuWLWMKhYLt3buXnT17lo0cOZJ5eXmx8vK/V/cOGzaM+fj4sMTERHb8+HHWvn17FhwczLfHxMQwjuPYkiVL2KVLl1hKSgoLDAxkHh4erKysrMH9M8VV6N/EZTGPeVHs1a3JDw4mgrqcV8IGrYhlHvOiWJdFB1nMRbXQKRFCHkJj1QJBCzhjjK1Zs4a5u7sziUTC/Pz8WEJCAt82aNAgNmnSJL34Xbt2sQ4dOjCJRMK6du3K9u/fr9eu0+nYokWLmLOzM5NKpWzIkCEsIyNDL6agoIAFBwczOzs7JpPJWGhoKCspKdGL2b59O/Px8WG2trasVatWbMSIEezixYsG9c0UC/jH+9OYx7wo9p9954VOhTRAYWklG7chjnnMi2Je82u3/ul0OqHTIoQYoLFqAccYLXVtLMXFxZDL5dBoNCYznT5zWyr2n7uJRc91wdT+XkKnQxqgqkaHxXvPY0dy7ZqKYD93fDiyKywtaA0qIeagsWoB/QZoZmgPuPmRiEWIGOONhUGdwXHA9qRsTPq/JGjKqoVOjRAiICrgzUzdIjY3WsRmVjiOw6sD2mLTBF/YSiwQ90cBRn91AlcL7gidGiFEIFTAm5F/7gFvTSNwsxTQxRnfz+iL1gprXMm/gzFfxeF0TpHQaRFCBEAFvBmp2wMuoz3gZq2ziww/zeyLbq1lKLhThfEb4xGdlvvgEwkhTQoV8GaE9oA3HU72VtgZ5o+nOrZCRbUO0/93Ev+L/1PotAghjxEV8GaEFrA1LbZSMb6e6IvxfdygY8CivRew7GA6dDraWEJIc0AFvBmhEXjTI7aoXaH+9jMdAADrj/yBOTtP07PFCWkGqIA3I/QUsqaJ4zi8OaQ9PnmpB8QiDvvO3MDEzbTNjJCmjgp4M5JDU+hN2gu922BLaB/YScVIzCrEi+vjcKOoXOi0CCGNhAp4M0JT6E3fgPatsPs1fyhlVsjMK8WL6+JwOa9U6LQIIY2ACngzcaeyBoW0B7xZ6Owiww+v90XbVra4oanAS+vjcCr7ttBpEUKM7KEKeE1NDX777Tds2LABJSUlAIAbN26gtJT+0jdV14toD3hz0lphje9f64sebeS4XVaNlzcl4silW0KnRQgxIoML+NWrV+Ht7Y2RI0di5syZuHWr9pfC8uXL8c477xg9QWIcf28ho+nz5sLBVoLvpj2JAe0dUV6txdTIZOw9fV3otAghRmJwAZ89ezZ8fX1x+/ZtWFv/PRU7evRoxMTEGDU5Yjy0Ar15spWKsXlSHzzfwxU1OobZO05jy4ksodMihBiB2NATjh07hri4OEgkEr3jnp6euH6d/ro3VbSArfmSiEX4YlxPONhYYmv8VSz5OQ2Fd6ow95kO4DhO6PQIIQ/J4BG4TqeDVnv3TSKuXbsGe3t7oyRFjI/uwta8iUQc/jOiK3/DlzWxl/H+T+ehpbu2EWK2DC7gQ4cOxeeff85/z3EcSktLER4ejmeffdaYuREjoil0UnfDl49He0P013PF5+w8jWqtTujUCCEPweAp9E8++QSBgYHo0qULKioq8PLLLyMzMxOOjo7Yvn17Y+RIjICm0Emdl1XukFtbYvaOU/j5zA2UV9Xgy5d7wcrSQujUCCEGMLiAt2nTBmfOnMHOnTtx5swZlJaWYurUqQgJCdFb1EZMB+0BJ/8W1N0FNhILvPZtCn67mIcpkcnYNNEXtlKDfyUQQgTCMcYMugh29OhR9O3bF2Kx/j/0mpoaxMXFYeDAgUZN0JwVFxdDLpdDo9FAJpMJlsel3BIM/ewoZFZinP1PoGB5ENMT/0cBXt2ajDtVWvi4KxA52Q9yG7pPACHG1Fi1wOBr4IMHD0ZhYeFdxzUaDQYPHmyUpIhx0R5wci/+T7TEtmlPQm5tiVPZRRi/KQH5pZVCp0UIaQCDCzhjrN6tJwUFBbC1tTVKUsS4aAEbuZ+ebgrsnP4kHO2kuHizGGM3xNNDUAgxAw2+4DVmzBgAtStZJ0+eDKlUyrdptVqcPXsWffv2NX6G5JHVFXA3BxqBk/p1Usqwa/qTeOXrRFy5dQcvrY/Hd9NU8GhJf5QTYqoaPAKXy+WQy+VgjMHe3p7/Xi6XQ6lUIiwsDN9++21j5koeEu0BJw3RtpUdds/oC8+WNrheVI6X1sfjUm6J0GkRQu6hwSPwLVu2AKi949o777xD0+VmhLaQkYZqrbDGrtf8MXFzEtLVJRi7IR7fTPFD9zYKoVMjhPyLwdfAw8PDqXibGboGTgzhZG+FHWFPooebAkV/Pcks+c+7F64SQoT1UJs+v//+e+zatQvZ2dmoqqrSa0tNTTVKYsQ4aA84eRgKGwm2varC1MhkJGYVYuLmJGye7Iu+TzgKnRoh5C8Gj8BXr16N0NBQODs749SpU/Dz80PLli1x5coVDB8+vDFyJI+gbvQtt7aEzIr295KGs5OKsXWKH/840tAtyThKzxQnxGQYXMC/+uorbNy4EWvWrIFEIsF7772H6OhozJo1CxqNpjFyJI+AFrCRR2FlaYFNE33xdCcnVNbo8OrWk4hNzxU6LUIIHqKAZ2dn89vFrK2tUVJSu0p1woQJdC90E0TXv8mjsrK0wPpXemNoF2dUaXWY/r8U/HJBLXRahDR7BhdwpVLJ34nN3d0dCQkJAICsrCwYeFdW8hjQXdiIMUjEIqwN6YWg7i6o1jLM3JaK/WdvCp0WIc2awQX86aefxr59+wAAoaGheOutt/DMM89g3LhxGD16tNETJI+GRuDEWCwtRPhiXE+M6umKGh3Dm9tTsff0daHTIqTZMriAb9y4ER988AEAYObMmfi///s/dO7cGR9++CHWrVtncAJr166Fp6cnrKysoFKpkJSUdN/43bt3o1OnTrCysoK3tzcOHDig184Yw+LFi+Hi4gJra2sEBAQgMzNTL6awsBAhISGQyWRQKBSYOnUqSktL73qfVatWoUOHDpBKpWjdujWWLl1qcP+ERnvAiTGJLUT4ZGxPvNi7DXQMmLPzNL5PuSZ0WoQ0T8wA1dXVbMmSJSwnJ8eQ0+5px44dTCKRsP/7v/9jFy5cYNOmTWMKhYLl5ubWG3/ixAlmYWHBVqxYwdLS0tjChQuZpaUlO3fuHB+zbNkyJpfL2Z49e9iZM2fYiBEjmJeXFysvL+djhg0bxnr06MESEhLYsWPHWLt27VhwcLDeZ7355pusY8eObO/evezKlSvs5MmT7NdffzWofxqNhgFgGo3GoPOMqeeSX5jHvCiWdkO4HEjTo9Xq2IIfzzKPeVHMc34U+y7xqtApEWKyGqsWGFTAGWPM1taWZWVlGeXD/fz82MyZM/nvtVotc3V1ZREREfXGjx07lgUFBekdU6lUbPr06YwxxnQ6HVMqlWzlypV8e1FREZNKpWz79u2MMcbS0tIYAJacnMzHHDx4kHEcx65fv87HiMVilp6e/kj9E7qAl1RUM495UcxjXhTTlFcJkgNpunQ6HQvfe57/b2xrXJbQKRFikhqrFhg8hT5kyBAcOXLkkUf+VVVVSElJQUBAAH9MJBIhICAA8fHx9Z4THx+vFw8AgYGBfHxWVhbUarVejFwuh0ql4mPi4+OhUCjg6+vLxwQEBEAkEiExMREA8PPPP6Nt27aIioqCl5cXPD098eqrr9b7GNV/qqysRHFxsd5LSNdpDzhpRBzHIfz5Lpg2wAsAsHjvBXx97IrAWRHSfBh8J7bhw4dj/vz5OHfuHHr37n3XbVVHjBjRoPfJz8+HVquFs7Oz3nFnZ2ekp6fXe45ara43Xq1W8+11x+4X4+TkpNcuFovh4ODAx1y5cgVXr17F7t278c0330Cr1eKtt97Ciy++iNjY2Hv2KSIiAkuWLHlQ1x8b2gNOGhvHcXj/2c61q9QP/4GP9l+EjjGEDXxC6NQIafIMLuCvv/46AODTTz+9q43jOGi12kfPSmA6nQ6VlZX45ptv0KFDBwDA5s2b0bt3b2RkZKBjx471nrdgwQLMnTuX/764uBhubm6PJef60Ap08jhwHId3hnaEhUiE1TGZ+PhA7R/gVMQJaVwGT6HrdLp7vgwp3o6OjrCwsEBurv5dnXJzc6FUKus9R6lU3je+7uuDYvLy8vTaa2pqUFhYyMe4uLhALBbzxRsAOnfuDKD2Rjb3IpVKIZPJ9F5Coj3g5HHhOA5zn+mA2UPaAwA+PpCODUf+EDgrQpo2gwu4sUgkEvTu3RsxMTH8MZ1Oh5iYGPj7+9d7jr+/v148AERHR/PxXl5eUCqVejHFxcVITEzkY/z9/VFUVISUlBQ+JjY2FjqdDiqVCgDQr18/1NTU4I8//v4FdOnSJQCAh4fHo3T7saIROHnc3nqmA+YE1BbxiIPpWE9FnJDGY9QlcQbasWMHk0qlLDIykqWlpbGwsDCmUCiYWq1mjDE2YcIENn/+fD7+xIkTTCwWs1WrVrGLFy+y8PDwereRKRQKtnfvXnb27Fk2cuTIereR+fj4sMTERHb8+HHWvn17vW1kWq2W9erViw0cOJClpqaykydPMpVKxZ555hmD+if0KvTnVh9jHvOi2K8X1IJ8Pmm+Po++xK9O/+rwZaHTIURQJrONzNjWrFnD3N3dmUQiYX5+fiwhIYFvGzRoEJs0aZJe/K5du1iHDh2YRCJhXbt2Zfv379dr1+l0bNGiRczZ2ZlJpVI2ZMgQlpGRoRdTUFDAgoODmZ2dHZPJZCw0NJSVlJToxVy/fp2NGTOG2dnZMWdnZzZ58mRWUFBgUN+ELuC0B5wI6Yvf/i7iaw9nCp0OIYJprFrAMUY3MG8sxcXFkMvl0Gg0j/16eGllDbqF/wIAOPefobCnbWREAGtiMvFJdO3lp3cDO2Lm4HYCZ0TI49dYtUCwa+CkcdXtAVfYWFLxJoJ5c0h7vP1M7WLQlb9kYO3hywJnREjTYfA2snvdnITjOEilUkgkkkdOijw62gNOTMWbQ9qD44BVv17Cyl8ywBjDG0+3FzotQsyewQVcoVCA47h7trdp0waTJ09GeHg4RCIa4AuFX4GuoC1kRHhvPN0eHMdh5S8ZWPXrJTBWW9gJIQ/P4AIeGRmJDz74AJMnT4afnx8AICkpCVu3bsXChQtx69YtrFq1ClKpFO+//77REyYNQyNwYmrqrn+v/CWDvy5ORZyQh2dwAd+6dSs++eQTjB07lj/2/PPPw9vbGxs2bEBMTAzc3d2xdOlSKuACyimkPeDE9Mwc3A4cB6w4VFvEGYBZVMQJeSgGz3HHxcXBx8fnruM+Pj78A0P69+9/3zuWkcZ3rYjuwkZM0+tPtcO8YZ0AAJ9GX8IXv2UKnBEh5sngAu7m5obNmzffdXzz5s38fb8LCgrQokWLR8+OPDT+GrgDjcCJ6Znx1BOYP7y2iH/22yV8/tslgTMixPwYPIW+atUqvPTSSzh48CD69OkDADh58iTS09Px/fffAwCSk5Mxbtw442ZKGqykohpFZdUAgNYKKuDENL026AlwqL3l6ue/ZYKx2luxEkIaxuACPmLECKSnp2PDhg38/cGHDx+OPXv2wNPTEwAwY8YMoyZJDHO9iPaAE/MwfdAT4Ljah598EVM7lU5FnJCGMbiAA7UPDVm2bJmxcyFGco0WsBEzUvfYUSrihBjmoQp4UVERkpKSkJeXB51Op9c2ceJEoyRGHh6/hYz2gBMzETbwCXDgsPTARSrihDSQwQX8559/RkhICEpLSyGTyfRu6sJxHBVwE0CPESXmaNrAtgDAF3EG4K2A9ve9cRQhzZnBq9DffvttTJkyBaWlpSgqKsLt27f5V2FhYWPkSAxEBZyYq2kD2+KDZzsDAFbHZOKz3zJBz1sipH4GF/Dr169j1qxZsLGh6VlTRXvAiTmbNrAtFgZRESfkQQwu4IGBgTh58mRj5EKMhPaAE3P36gAq4oQ8iMHXwIOCgvDuu+8iLS0N3t7esLTU36Y0YsQIoyVHDEd7wElT8eqA2mviH+2/iNUxmQBjeOuZDnRNnJC/GFzAp02bBgD48MMP72rjOA5arfbRsyIPjfaAk6ZEr4jH1j5LnIo4IbUMLuD/3jZGTAvtASdNDRVxQupHD+xuYmgPOGmK9K6Jx17Gp9GX6Jo4afYaNAJfvXo1wsLCYGVlhdWrV983dtasWUZJjDwc2kJGmqp/jsTX/DUSn0sjcdKMNaiAf/bZZwgJCYGVlRU+++yze8ZxHEcFXGBUwElTRkWckL81qIBnZWXV+7+J6anbA+7mQFPopGl6dUBbcByH/0alUREnzRpdA29i/h6BUwEnTdfU/l5Y9FwXAMCa2Mv45Fe6Jk6aH4NXoWu1WkRGRiImJqbeh5nExsYaLTliGL094DSFTpq4qf29AAD/jUrDl4drR+JvD6WROGk+DC7gs2fPRmRkJIKCgtCtWzf6x2JC6kbfLWwsYSd9qAfNEWJWpvb3AgfgQyripBky+Lf8jh07sGvXLjz77LONkQ95BDR9TpqjKX+NxOuKOAPDO0M7UhEnTZ7B18AlEgnatWvXGLmQR8TvAafpc9LMTOnvhcV/XRNfe/gPrPo1g66JkybvoR4n+sUXX9A/DhNEW8hIczalvxfCn6ciTpoPg6fQjx8/jsOHD+PgwYPo2rXrXQ8z+fHHH42WHDHM3yNwmkInzVNov9rp9CU/p2Ht4T/AGPBuIE2nk6bJ4AKuUCgwevToxsiFPCIagROiX8S/+v0PAFTESdNkUAGvqanB4MGDMXToUCiVysbKiTwkWsRGSC0q4qQ5MOgauFgsxmuvvYbKykqjJrF27Vp4enrCysoKKpUKSUlJ943fvXs3OnXqBCsrK3h7e+PAgQN67YwxLF68GC4uLrC2tkZAQAAyMzP1YgoLCxESEgKZTAaFQoGpU6eitLS03s+7fPky7O3toVAoHqmfjam4ohqactoDTkid0H5e+M9f18S/+v0PrPyFromTpsXgRWx+fn44deqU0RLYuXMn5s6di/DwcKSmpqJHjx4IDAxEXl5evfFxcXEIDg7G1KlTcerUKYwaNQqjRo3C+fPn+ZgVK1Zg9erVWL9+PRITE2Fra4vAwEBUVFTwMSEhIbhw4QKio6MRFRWFo0ePIiws7K7Pq66uRnBwMAYMGGC0PjeG67QHnJC7TP5XEV92KJ2KOGk6mIF27tzJ2rZty9asWcPi4uLYmTNn9F6G8vPzYzNnzuS/12q1zNXVlUVERNQbP3bsWBYUFKR3TKVSsenTpzPGGNPpdEypVLKVK1fy7UVFRUwqlbLt27czxhhLS0tjAFhycjIfc/DgQcZxHLt+/bree7/33nvslVdeYVu2bGFyudygvmk0GgaAaTQag857GL9eUDOPeVHsudXHGv2zCDE3kSeymMe8KOYxL4r9Z995ptPphE6JNCONVQsMHoGPHz8eWVlZmDVrFvr164eePXvCx8eH/2qIqqoqpKSkICAggD8mEokQEBCA+Pj4es+Jj4/XiweAwMBAPj4rKwtqtVovRi6XQ6VS8THx8fFQKBTw9fXlYwICAiASiZCYmMgfi42Nxe7du7F27doG9aeyshLFxcV6r8eF9oATcm+T+nrio1HdAABbTvyJRXvPQ6ejkTgxbwbPtRrzaWT5+fnQarVwdnbWO+7s7Iz09PR6z1Gr1fXGq9Vqvr3u2P1inJyc9NrFYjEcHBz4mIKCAkyePBnffvstZDJZg/oTERGBJUuWNCjW2GgFOiH398qTHpBYiDDvx7P4NiEbNVqGj0d7QySihW3EPBlcwD08PBojD5Mzbdo0vPzyyxg4cGCDz1mwYAHmzp3Lf19cXAw3N7fGSO8utAeckAcb28cNlmIOb+86gx3JOajS6rDyxR6woCJOzNBDr3ZKS0tDdnY2qqqq9I6PGDGiwe/h6OgICwsL5Obm6h3Pzc295zY1pVJ53/i6r7m5uXBxcdGL6dmzJx/z70VyNTU1KCws5M+PjY3Fvn37sGrVKgC1K9t1Oh3EYjE2btyIKVOm3JWbVCqFVCptaPeNikbghDTMaJ82EItEmLPzNH5MvY5qLcNnY3tAbEFPVybmxeACfuXKFYwePRrnzp0Dx3H8is66/ZVarbbB7yWRSNC7d2/ExMRg1KhRAACdToeYmBi88cYb9Z7j7++PmJgYzJkzhz8WHR0Nf39/AICXlxeUSiViYmL4gl1cXIzExETMmDGDf4+ioiKkpKSgd+/eAGoLtk6ng0qlAlB7nfyffdm7dy+WL1+OuLg4tG7dusF9fFxoDzghDfd8D1dYWnB447tT+PnMDdRodfhivA8kYirixIwYuurtueeeYyNHjmS3bt1idnZ2LC0tjR07doz5+fmxo0ePGryKbseOHUwqlbLIyEiWlpbGwsLCmEKhYGq1mjHG2IQJE9j8+fP5+BMnTjCxWMxWrVrFLl68yMLDw5mlpSU7d+4cH7Ns2TKmUCjY3r172dmzZ9nIkSOZl5cXKy8v52OGDRvGfHx8WGJiIjt+/Dhr3749Cw4OvmeeprwKXVNexa+wLamobtTPIqQpib6gZu3fP8A85kWxqZFJrKK6RuiUSBPUWLXA4ALesmVLfruYTCZj6enpjDHGYmJiWM+ePR8qiTVr1jB3d3cmkUiYn58fS0hI4NsGDRrEJk2apBe/a9cu1qFDByaRSFjXrl3Z/v379dp1Oh1btGgRc3Z2ZlKplA0ZMoRlZGToxRQUFLDg4GBmZ2fHZDIZCw0NZSUlJffM0ZQLeNoNDfOYF8V6LvmlUT+HkKbocHou6/BBbRGfuDmRlVdRESfG1Vi1gGPMsLsatGjRAqmpqfDy8sITTzyBr7/+GoMHD8Yff/wBb29vlJWVNcZEgVkqLi6GXC6HRqNp8Er2hxGdlotp35yEd2s5fn6zf6N9DiFN1YnL+Zi6NRkV1Tr0b+eITRN9YS2xEDot0kQ0Vi0w+IJPt27dcObMGQCASqXCihUrcOLECXz44Ydo27at0RIjDZdTSHvACXkU/do5IjLUDzYSCxy/nI9JW5JQWlkjdFqE3JfBBXzhwoXQ6XQAgA8//BBZWVkYMGAADhw4gNWrVxs9QfJgdQvY3BxoARshD+vJti3xv6l+sJeKkZRViJCvE1FUVvXgEwkRiMEFPDAwEGPGjAEAtGvXDunp6cjPz0deXh6efvppoydIHozuwkaIcfT2cMC2aSq0sLHEmZwijNuQgLySigefSIgAHnrPxOXLl/HLL7+gvLwcDg4OxsyJGIj2gBNiPN3bKLBzuj9a2UuRkVuCcRsScL2oXOi0CLmLwQW8oKAAQ4YMQYcOHfDss8/i5s2bAICpU6fi7bffNnqC5MHoLmyEGFcHZ3vsnu6P1gprZOXfwUvr4nDlVv2PGyZEKAYX8LfeeguWlpbIzs6Gjc3fBWPcuHE4dOiQUZMjD6Ypr0ZxRe1im9YKGoETYiyejrb4foY/2rayxQ1NBcZuSMDFm4/vAUWEPIjBBfzXX3/F8uXL0aZNG73j7du3x9WrV42WGGmYuueAO9hKYEvPASfEqFzk1tg13R+dXWTIL63E+I0JOJV9W+i0CAHwEAX8zp07eiPvOoWFhYLdB7w5owVshDQuRzspdkx7Er3cFdCUV+OVrxMR/0eB0GkRYngBHzBgAL755hv+e47joNPpsGLFCgwePNioyZEHowVshDQ+uY0l/jdVhX7tWuJOlRaTtyThcHreg08kpBEZXMBXrFiBjRs3Yvjw4aiqqsJ7772Hbt264ejRo1i+fHlj5Ejugx5iQsjjYSsVY/OkPgjo7ITKGh2mfXMSUWdvCJ0WacYe6k5sly5dQv/+/TFy5EjcuXMHY8aMwalTp/DEE080Ro7kPmgKnZDHx8rSAute6Y0RPVxRo2OYtf0UdiRlC50WaaYeatWTXC7HBx98oHfs2rVrCAsLw8aNG42SGGkYmkIn5PGytBDhs3E9YSu1wPakHMz/8RwK7lTh9aee4B+rTMjjYLSH3xYUFGDz5s3GejvSQLQHnJDHz0LE4ePR3nj9qdpZx5W/ZOC/UReh0xn0bChCHgk9vd6M0R5wQoTDcRzeG9YJC4M6AwD+70QW5u46jWqtTuDMSHNBBdyM0R5wQoT36oC2+GxcD4hFHPacvoFp35xEWRU9yYw0PirgZowWsBFiGkb7tMGmSb6wshTh94xbCPk6Ebfv0JPMSONq8LCt7glk91JUVPSouRAD0QI2QkzH4I5O2Pbqk5gSmYxT2UV4aUM8vpniB1e6vEUaSYNH4HK5/L4vDw8PTJw4sTFzJf9Ce8AJMS29PVpg92v+UMqscDmvFC+ui8PlvBKh0yJNVINH4Fu2bGnMPMhDoCl0QkxPB2d7/PB6X0zYnIgrt+5gzFdx2DTRF6q2LYVOjTQxdA3cjOXQFDohJqm1whrfv9YXvdwVKK6owYTNSfj5DN21jRgXFXAzRnvACTFdDrYSfDftSQzrqkSVVoc3t5/CxqN/gDHaK06Mgwq4mdKUV6OE9oATYtKsLC2wNqQXQvt5AgA+PpCO/+y7AC3d8IUYARVwM1U3+m5Je8AJMWkWIg7hz3flb/iyNf4qXvs2BeVVWoEzI+aOCriZoi1khJiXVwe0xdqXe0EiFiE6LRfBmxJQUFopdFrEjFEBN1O0hYwQ8xPU3QXbXlVBbm2J0zlFeGFdHK7cKhU6LWKmqICbKdpCRoh56uPpgB9m9EWbFtb4s6AMo7+KQ9zlfKHTImaICriZoil0QsxXOyc7/PR6P/RyV0BTXo2J/5eE7xLpueLEMFTAzRRNoRNi3lrZS/HdtCcxsqcranQM7/90Dh/+nEYr1EmDUQE3UzSFToj5s7K0wOfjeuLtZzoAqH0k6bRvTqKkolrgzIg5oAJuhvT2gFMBJ8SscRyHN4e0x9qXe8HKUoTY9Dy8uC4eOYVlQqdGTBwVcDP0zz3gNhLaA05IUxDU3QW7pvvDyV6KjNwSjFp7AilXC4VOi5gwKuBmiBawEdI0dW+jwN43+qGrqwwFd6oQvDERO5NpcRupn0kU8LVr18LT0xNWVlZQqVRISkq6b/zu3bvRqVMnWFlZwdvbGwcOHNBrZ4xh8eLFcHFxgbW1NQICApCZmakXU1hYiJCQEMhkMigUCkydOhWlpX/vx/z9998xcuRIuLi4wNbWFj179sS2bduM1+lHQAvYCGm6XOTW2P2aP38P9Xk/nMMHP51DVY1O6NSIiRG8gO/cuRNz585FeHg4UlNT0aNHDwQGBiIvL6/e+Li4OAQHB2Pq1Kk4deoURo0ahVGjRuH8+fN8zIoVK7B69WqsX78eiYmJsLW1RWBgICoqKviYkJAQXLhwAdHR0YiKisLRo0cRFham9zndu3fHDz/8gLNnzyI0NBQTJ05EVFRU4/0wGogWsBHStNlIxPgqpBfeGdoBHAdsS8xG8KYE5BVXPPhk0nwwgfn5+bGZM2fy32u1Wubq6soiIiLqjR87diwLCgrSO6ZSqdj06dMZY4zpdDqmVCrZypUr+faioiImlUrZ9u3bGWOMpaWlMQAsOTmZjzl48CDjOI5dv379nrk+++yzLDQ0tMF902g0DADTaDQNPqchXt2azDzmRbFv4rKM+r6EENMTezGXdQs/xDzmRbE+H0Wzk38WCp0SMVBj1QJBR+BVVVVISUlBQEAAf0wkEiEgIADx8fH1nhMfH68XDwCBgYF8fFZWFtRqtV6MXC6HSqXiY+Lj46FQKODr68vHBAQEQCQSITEx8Z75ajQaODg43LO9srISxcXFeq/GQFPohDQfgzs5Yd8b/dHeyQ55JZUYvzEe2xKvCp0WMQGCFvD8/HxotVo4OzvrHXd2doZara73HLVafd/4uq8PinFyctJrF4vFcHBwuOfn7tq1C8nJyQgNDb1nfyIiIiCXy/mXm5vbPWMfBU2hE9K8eDnaYs/MfnjWW4lqLcMHP53Hgh/PorKGnmjWnAl+DdwcHD58GKGhodi0aRO6du16z7gFCxZAo9Hwr5ycHKPnQnvACWmebKVirH25F+YN6wSOA7Yn5WDshgT+D3rS/AhawB0dHWFhYYHc3Fy947m5uVAqlfWeo1Qq7xtf9/VBMf9eJFdTU4PCwsK7PvfIkSN4/vnn8dlnn2HixIn37Y9UKoVMJtN7GVvdzR1oDzghzQ/HcZjx1BOIDPWD3NoSZ3KKELT6OKLTch98MmlyBC3gEokEvXv3RkxMDH9Mp9MhJiYG/v7+9Z7j7++vFw8A0dHRfLyXlxeUSqVeTHFxMRITE/kYf39/FBUVISUlhY+JjY2FTqeDSqXij/3+++8ICgrC8uXL9VaoC4n2gBNCBnVohag3+6OHW+3DUKZ9cxIfRaXRVrPmxqhL4h7Cjh07mFQqZZGRkSwtLY2FhYUxhULB1Go1Y4yxCRMmsPnz5/PxJ06cYGKxmK1atYpdvHiRhYeHM0tLS3bu3Dk+ZtmyZUyhULC9e/eys2fPspEjRzIvLy9WXl7OxwwbNoz5+PiwxMREdvz4cda+fXsWHBzMt8fGxjIbGxu2YMECdvPmTf5VUFDQ4L41xsrDTUf/YB7zotjr36YY7T0JIeapslrLPvz5AvOYF8U85kWxEV8eZ9kFd4ROi/xLY61CF7yAM8bYmjVrmLu7O5NIJMzPz48lJCTwbYMGDWKTJk3Si9+1axfr0KEDk0gkrGvXrmz//v167Tqdji1atIg5OzszqVTKhgwZwjIyMvRiCgoKWHBwMLOzs2MymYyFhoaykpISvn3SpEkMwF2vQYMGNbhfjfF/Wvje88xjXhT7eH+a0d6TEGLefjl/k3n/tdXMO/wQ++X8TaFTIv/QWAWcY4zRs+saSXFxMeRyOTQajdGuh7+69SR+u5iL/47sign+nkZ5T0KI+cspLMOb20/hdE4RAGBKPy/MH94JEjGtVRZaY9QCgFahmx1+C5kD7QEnhPzNzcEGu6b7Y9oALwC1jyZ9cX0crtwqfcCZxFxRATcjjDFc/2sRmxstYiOE/ItELMIHQV3w9URfyK0tcfaaBkGrj+O7xGzQZGvTQwXcjBSX16Ck8q894AoagRNC6hfQxRmH5gxAv3YtUV6txfs/ncO0b1JQUFopdGrEiKiAm5Gcv6bPHe0ksJZYCJwNIcSUucit8b8pKiwM6gyJhQi/XcxF4OfHcDi9/gdFEfNDBdyM1O0Bb033QCeENIBIxOHVAW2xZ2Y/dHC2Q35pJUIjkzH/h7MoqagWOj3yiKiAmxG6Bzoh5GF0cZVh3xv9MaWfFzgO2JGcg8DPjuJY5i2hUyOPgAq4GaG7sBFCHpaVpQUWP98FO6Y9CXcHG9zQVGDC5iQs+PEcjcbNFBVwM0KPESWEPCpV25Y4NGcAJvf1BABsT8rGsM+P4fcMujZubqiAmxGaQieEGIONRIz/jOiK7dOehJuDNa4XlWPylmS8uf0UbpXQSnVzQQXcTNAecEKIsfk/0RKHZg/Eq/29IOKAn8/cwJBPfseOpGzodLRv3NRRATcTtAecENIYbKViLHyuC/bO7I9urWUorqjB/B/PYfzGBGTmlgidHrkPKuBmgvaAE0Iak3cbOfa83g8LgzrD2tICSX8WYvgXx/BRVBotcjNRVMDNBO0BJ4Q0NrGFCK8OaIvouQPxTBdn1OgYvj6ehcGrjuD7lGs0rW5iqICbCVrARgh5XNq0sMGmib6IDO2Dto62yC+txDu7z+CF9XE4e61I6PTIX6iAmwnaA04Iedye6uiEQ3MGYsHwTrCVWOBUdhFGfHkCs3ecQk5hmdDpNXtUwM3E3yNwmkInhDw+ErEI0wc9gdh3nsIYn9YAgL2nb2DIJ0ewdH8aisqqBM6w+aICbiZoBE4IEZKzzAqfjuuJqDf7o1+7lqjS6rDpWBYGrfwdG4/+gfIqrdApNjtUwM0AY4wv4LQHnBAipG6t5fh2qgqRoX3QSWkPTXk1Pj6QjoErD+P/jmehopoK+eNCBdwMaMqrUUp7wAkhJoLjODzV0Qn7Zw3Ayhe7o00La9wqqcSHUWkYtPIwtsb9SYX8MaACbgbqRt+0B5wQYkosRBxe8nVD7NtP4ePR3mitsEZucSXC913A4FW/Y/PxLNz5a/BBjI8KuBmoW8BGe8AJIaZIIhbhZZU7Yt8ZhI9GdYOL3Ao3NRX4b1Qa+i6LxSe/ZiC/lO6xbmxUwM0ALWAjhJgDqdgCrzzpgd/frR2ReznaQlNejTWxl9FvWSw++Okc/rhVKnSaTYZY6ATIg/29gI1G4IQQ0ycVW+BllTvG9XFDdJoa645cwZmcImxLzMa2xGz0b+eICf4eGNLJCWILGkc+LCrgZoDuwkYIMUcWIg7DurkgsKsSSVmF2HTsCmLS83D8cj6OX86Hi9wKL/u5Y5yfG5zsrYRO1+xQATcDNIVOCDFnHMdB1bYlVG1bIqewDN8lZWNncg5uairwSfQlfB6TiYHtHfFC7zYI6OwMK0tarNsQVMBN3D/3gNNd2Agh5s7NwQbzhnXCnID2OHDuJv4XfxWp2UU4nHELhzNuwd5KjOe6u+KFXq3Ry70FRCJO6JRNFhVwE/fPPeA0AieENBVSsQVG+7TBaJ82+ONWKX5KvY4fU6/hhqYC25OysT0pG84yKQK7KjGsqxJ+Xg50vfxfOMYYPR+ukRQXF0Mul0Oj0UAmkz3Ue5y/rsFza47D0U6KkwsDjJwhIYSYDp2OIeFKAb5PvYboC7ko+cce8hY2lnimizMGd3RC3yccIbexFDBTwxijFtSHRuAmjhawEUKaC5GIQ992jujbzhGVNVrEXS7AofNq/Jqmxu2yauw6eQ27Tl6DiAN6uikwoH0rDOzgiO5tFLBshqNzKuAmjhawEUKaI6nYAoM7OWFwJycs1XZD0p+FiE7LxbHMfFzOK0VqdhFSs4vwRUwmrCxF6N5Ggd4eLdDbvQV6ebSAg61E6C40OirgJo4WsBFCmjuxhQh9n3BE3yccAQA3ispxPDMfRzNv4fjlfBSVVSMpqxBJWYX8OW4O1ujoLEMnpT06/vXycrRtUiN1k+jJ2rVr4enpCSsrK6hUKiQlJd03fvfu3ejUqROsrKzg7e2NAwcO6LUzxrB48WK4uLjA2toaAQEByMzM1IspLCxESEgIZDIZFAoFpk6ditJS/TsEnT17FgMGDICVlRXc3NywYsUK43TYADSFTggh+lwV1hjbxw1fvtwLqQufwW9zB2HFC90xztcN7Z3sAAA5heX47WIuvjx8GW9uP4Whnx1F50WHMGBFLII3JuC9789gdUwmfki5ht8z8nAmpwg5hWUoqaiGuSwNE3wEvnPnTsydOxfr16+HSqXC559/jsDAQGRkZMDJyemu+Li4OAQHByMiIgLPPfccvvvuO4waNQqpqano1q0bAGDFihVYvXo1tm7dCi8vLyxatAiBgYFIS0uDlVXtzQJCQkJw8+ZNREdHo7q6GqGhoQgLC8N3330HoHbRwdChQxEQEID169fj3LlzmDJlChQKBcLCwh7bz4em0Akh5N5EIg7tnOzQzskOY/u4AQA0ZdVIu1mMDHUxMnJLkK4uwSV1Ce5UaZFTWI6cwnLEX7n3e1pacLCVimEltoC1xAJhA9si2M/9MfWo4QRfha5SqdCnTx98+eWXAACdTgc3Nze8+eabmD9//l3x48aNw507dxAVFcUfe/LJJ9GzZ0+sX78ejDG4urri7bffxjvvvAMA0Gg0cHZ2RmRkJMaPH4+LFy+iS5cuSE5Ohq+vLwDg0KFDePbZZ3Ht2jW4urpi3bp1+OCDD6BWqyGR1F5LmT9/Pvbs2YP09PQG9e1RVx4yxuD9n19RWlmD3+YOQru//rIkhBBiGJ2OQV1cgetF5bh2uww5hbVfrxeVo/BONYrKqnC7rAoV1bq7zl0Y1BmvDmj70J/dJFehV1VVISUlBQsWLOCPiUQiBAQEID4+vt5z4uPjMXfuXL1jgYGB2LNnDwAgKysLarUaAQF/b7mSy+VQqVSIj4/H+PHjER8fD4VCwRdvAAgICIBIJEJiYiJGjx6N+Ph4DBw4kC/edZ+zfPly3L59Gy1atLgrt8rKSlRW/v3EneLiYsN+IPXYEfYkrt0uh5sDjcAJIeRhiUQcXBXWcFVYo4+nwz3jyqu0uF1WhbIqLSqqtSiv1prsDKigBTw/Px9arRbOzs56x52dne85ylWr1fXGq9Vqvr3u2P1i/j09LxaL4eDgoBfj5eV113vUtdVXwCMiIrBkyZJ7d9hAHMehW2s5urWWG+09CSGE3Ju1xALWEtMs2P9mEovYmooFCxZAo9Hwr5ycHKFTIoQQ0kQJWsAdHR1hYWGB3NxcveO5ublQKpX1nqNUKu8bX/f1QTF5eXl67TU1NSgsLNSLqe89/vkZ/yaVSiGTyfRehBBCSGMQtIBLJBL07t0bMTEx/DGdToeYmBj4+/vXe46/v79ePABER0fz8V5eXlAqlXoxxcXFSExM5GP8/f1RVFSElJQUPiY2NhY6nQ4qlYqPOXr0KKqrq/U+p2PHjvVOnxNCCCGPFRPYjh07mFQqZZGRkSwtLY2FhYUxhULB1Go1Y4yxCRMmsPnz5/PxJ06cYGKxmK1atYpdvHiRhYeHM0tLS3bu3Dk+ZtmyZUyhULC9e/eys2fPspEjRzIvLy9WXl7OxwwbNoz5+PiwxMREdvz4cda+fXsWHBzMtxcVFTFnZ2c2YcIEdv78ebZjxw5mY2PDNmzY0OC+aTQaBoBpNJpH+RERQggxY41VCwQv4IwxtmbNGubu7s4kEgnz8/NjCQkJfNugQYPYpEmT9OJ37drFOnTowCQSCevatSvbv3+/XrtOp2OLFi1izs7OTCqVsiFDhrCMjAy9mIKCAhYcHMzs7OyYTCZjoaGhrKSkRC/mzJkzrH///kwqlbLWrVuzZcuWGdQvKuCEEEIaqxYIvg+8KWusvX+EEELMR2PVAlqFTgghhJghwW+l2pTVTW4Y44YuhBBCzFNdDTD2hDcV8EZUUlICAHBzcxM4E0IIIUIrKSmBXG68G3PRNfBGpNPpcOPGDdjb24PjuId6j+LiYri5uSEnJ6fJXEdvan1qav0Bml6fmlp/AOqTOajrT3Z2NjiOg6urK0Qi4125phF4IxKJRGjTpo1R3qsp3himqfWpqfUHaHp9amr9AahP5kAulzdKf2gRGyGEEGKGqIATQgghZogKuImTSqUIDw+HVCoVOhWjaWp9amr9AZpen5pafwDqkzlo7P7QIjZCCCHEDNEInBBCCDFDVMAJIYQQM0QFnBBCCDFDVMAJIYQQM0QF3MStXbsWnp6esLKygkqlQlJSktApNUhERAT69OkDe3t7ODk5YdSoUcjIyNCLqaiowMyZM9GyZUvY2dnhhRdeQG5urkAZG2bZsmXgOA5z5szhj5ljf65fv45XXnkFLVu2hLW1Nby9vXHy5Em+nTGGxYsXw8XFBdbW1ggICEBmZqaAGd+fVqvFokWL4OXlBWtrazzxxBP473//q3cPalPu09GjR/H888/D1dUVHMdhz549eu0Nyb2wsBAhISGQyWRQKBSYOnUqSktLH2Mv9N2vT9XV1Zg3bx68vb1ha2sLV1dXTJw4ETdu3NB7D3Pq07+99tpr4DgOn3/+ud5xY/SJCrgJ27lzJ+bOnYvw8HCkpqaiR48eCAwMRF5entCpPdCRI0cwc+ZMJCQkIDo6GtXV1Rg6dCju3LnDx7z11lv4+eefsXv3bhw5cgQ3btzAmDFjBMy6YZKTk7FhwwZ0795d77i59ef27dvo168fLC0tcfDgQaSlpeGTTz5BixYt+JgVK1Zg9erVWL9+PRITE2Fra4vAwEBUVFQImPm9LV++HOvWrcOXX36JixcvYvny5VixYgXWrFnDx5hyn+7cuYMePXpg7dq19bY3JPeQkBBcuHAB0dHRiIqKwtGjRxEWFva4unCX+/WprKwMqampWLRoEVJTU/Hjjz8iIyMDI0aM0Iszpz79008//YSEhAS4urre1WaUPhn16eLEqPz8/NjMmTP577VaLXN1dWURERECZvVw8vLyGAB25MgRxhhjRUVFzNLSku3evZuPuXjxIgPA4uPjhUrzgUpKSlj79u1ZdHQ0GzRoEJs9ezZjzDz7M2/ePNa/f/97tut0OqZUKtnKlSv5Y0VFRUwqlbLt27c/jhQNFhQUxKZMmaJ3bMyYMSwkJIQxZl59AsB++ukn/vuG5J6WlsYAsOTkZD7m4MGDjOM4dv369ceW+738u0/1SUpKYgDY1atXGWPm26dr166x1q1bs/PnzzMPDw/22Wef8W3G6hONwE1UVVUVUlJSEBAQwB8TiUQICAhAfHy8gJk9HI1GAwBwcHAAAKSkpKC6ulqvf506dYK7u7tJ92/mzJkICgrSyxswz/7s27cPvr6+eOmll+Dk5AQfHx9s2rSJb8/KyoJardbrk1wuh0qlMtk+9e3bFzExMbh06RIA4MyZMzh+/DiGDx8OwDz7VKchucfHx0OhUMDX15ePCQgIgEgkQmJi4mPP+WFoNBpwHAeFQgHAPPuk0+kwYcIEvPvuu+jatetd7cbqEz3MxETl5+dDq9XC2dlZ77izszPS09MFyurh6HQ6zJkzB/369UO3bt0AAGq1GhKJhP9HWsfZ2RlqtVqALB9sx44dSE1NRXJy8l1t5tifK1euYN26dZg7dy7ef/99JCcnY9asWZBIJJg0aRKfd33/DZpqn+bPn4/i4mJ06tQJFhYW0Gq1WLp0KUJCQgDALPtUpyG5q9VqODk56bWLxWI4ODiYfP+A2nUk8+bNQ3BwMP/wD3Ps0/LlyyEWizFr1qx6243VJyrgpNHNnDkT58+fx/Hjx4VO5aHl5ORg9uzZiI6OhpWVldDpGIVOp4Ovry8+/vhjAICPjw/Onz+P9evXY9KkSQJn93B27dqFbdu24bvvvkPXrl1x+vRpzJkzB66urmbbp+aiuroaY8eOBWMM69atEzqdh5aSkoIvvvgCqampD/0Y6YaiKXQT5ejoCAsLi7tWMefm5kKpVAqUleHeeOMNREVF4fDhw3qPVlUqlaiqqkJRUZFevKn2LyUlBXl5eejVqxfEYjHEYjGOHDmC1atXQywWw9nZ2az6AwAuLi7o0qWL3rHOnTsjOzsbAPi8zem/wXfffRfz58/H+PHj4e3tjQkTJuCtt95CREQEAPPsU52G5K5UKu9a5FpTU4PCwkKT7l9d8b569Sqio6P1Hr1pbn06duwY8vLy4O7uzv+uuHr1Kt5++214enoCMF6fqICbKIlEgt69eyMmJoY/ptPpEBMTA39/fwEzaxjGGN544w389NNPiI2NhZeXl1577969YWlpqde/jIwMZGdnm2T/hgwZgnPnzuH06dP8y9fXFyEhIfz/Nqf+AEC/fv3u2tp36dIleHh4AAC8vLygVCr1+lRcXIzExEST7VNZWRlEIv1faxYWFtDpdADMs091GpK7v78/ioqKkJKSwsfExsZCp9NBpVI99pwboq54Z2Zm4rfffkPLli312s2tTxMmTMDZs2f1fle4urri3XffxS+//ALAiH16+LV3pLHt2LGDSaVSFhkZydLS0lhYWBhTKBRMrVYLndoDzZgxg8nlcvb777+zmzdv8q+ysjI+5rXXXmPu7u4sNjaWnTx5kvn7+zN/f38BszbMP1ehM2Z+/UlKSmJisZgtXbqUZWZmsm3btjEbGxv27bff8jHLli1jCoWC7d27l509e5aNHDmSeXl5sfLycgEzv7dJkyax1q1bs6ioKJaVlcV+/PFH5ujoyN577z0+xpT7VFJSwk6dOsVOnTrFALBPP/2UnTp1il+R3ZDchw0bxnx8fFhiYiI7fvw4a9++PQsODhaqS/ftU1VVFRsxYgRr06YNO336tN7visrKSrPsU33+vQqdMeP0iQq4iVuzZg1zd3dnEomE+fn5sYSEBKFTahAA9b62bNnCx5SXl7PXX3+dtWjRgtnY2LDRo0ezmzdvCpe0gf5dwM2xPz///DPr1q0bk0qlrFOnTmzjxo167Tqdji1atIg5OzszqVTKhgwZwjIyMgTK9sGKi4vZ7Nmzmbu7O7OysmJt27ZlH3zwgV4xMOU+HT58uN5/N5MmTWKMNSz3goICFhwczOzs7JhMJmOhoaGspKREgN7Uul+fsrKy7vm74vDhw2bZp/rUV8CN0Sd6nCghhBBihugaOCGEEGKGqIATQgghZogKOCGEEGKGqIATQgghZogKOCGEEGKGqIATQgghZogKOCGEEGKGqIATQgghZogKOCHknm7duoUZM2bA3d0dUqkUSqUSgYGBOHHiBACA4zjs2bNH2CQJaabocaKEkHt64YUXUFVVha1bt6Jt27bIzc1FTEwMCgoKhE6NkGaPRuCEkHoVFRXh2LFjWL58OQYPHgwPDw/4+flhwYIFGDFiBP9oxNGjR4PjOP57ANi7dy969eoFKysrtG3bFkuWLEFNTQ3fznEc1q1bh+HDh8Pa2hpt27bF999/z7dXVVXhjTfegIuLC6ysrODh4cE/EpQQUosKOCGkXnZ2drCzs8OePXtQWVl5V3tycjIAYMuWLbh58yb//bFjxzBx4kTMnj0baWlp2LBhAyIjI7F06VK98xctWoQXXngBZ86cQUhICMaPH4+LFy8CAFavXo19+/Zh165dyMjIwLZt2/T+QCCEAPQwE0LIPf3www+YNm0aysvL0atXLwwaNAjjx49H9+7dAdSOpH/66SeMGjWKPycgIABDhgzBggUL+GPffvst3nvvPdy4cYM/77XXXsO6dev4mCeffBK9evXCV199hVmzZuHChQv47bffwHHc4+ksIWaGRuCEkHt64YUXcOPGDezbtw/Dhg3D77//jl69eiEyMvKe55w5cwYffvghP4K3s7PDtGnTcPPmTZSVlfFx/v7+euf5+/vzI/DJkyfj9OnT6NixI2bNmoVff/21UfpHiDmjAk4IuS8rKys888wzWLRoEeLi4jB58mSEh4ffM760tBRLlizB6dOn+de5c+eQmZkJKyurBn1mr169kJWVhf/+978oLy/H2LFj8eKLLxqrS4Q0CVTACSEG6dKlC+7cuQMAsLS0hFar1Wvv1asXMjIy0K5du7teItHfv3ISEhL0zktISEDnzp3572UyGcaNG4dNmzZh586d+OGHH1BYWNiIPSPEvNA2MkJIvQoKCvDSSy9hypQp6N69O+zt7XHy5EmsWLECI0eOBAB4enoiJiYG/fr1g1QqRYsWLbB48WI899xzcHd3x4svvgiRSIQzZ87g/Pnz+Oijj/j33717N3x9fdG/f39s27YNSUlJ2Lx5MwDg008/hYuLC3x8fCASibB7924olUooFAohfhSEmCZGCCH1qKioYPPnz2e9evVicrmc2djYsI4dO7KFCxeysrIyxhhj+/btY+3atWNisZh5eHjw5x46dIj17duXWVtbM5lMxvz8/NjGjRv5dgBs7dq17JlnnmFSqZR5enqynTt38u0bN25kPXv2ZLa2tkwmk7EhQ4aw1NTUx9Z3QswBrUInhDx29a1eJ4QYhq6BE0IIIWaICjghhBBihmgRGyHksaMrd4Q8OhqBE0IIIWaICjghhBBihqiAE0IIIWaICjghhBBihqiAE0IIIWaICjghhBBihqiAE0IIIWaICjghhBBihqiAE0IIIWbo/wFVGCrhrpUbVgAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(5, 3))\n", "plt.plot(range(len(lrs)), lrs)\n", "plt.ylabel(\"Learning rate\")\n", "plt.xlabel(\"Steps\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "a2f85b01-859b-4454-a3a3-c7ef593735a6", "metadata": {}, "source": [ "- And a quick look at the loss curves" ] }, { "cell_type": "code", "execution_count": 15, "id": "445d8155-6eae-4b50-a381-d0820ebc27cc", "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/ipykernel_34986/3589549395.py:5: UserWarning: The figure layout has changed to tight\n", " plt.tight_layout(); plt.savefig(\"3.pdf\")\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABY0ElEQVR4nO3dd1wT9/8H8NclgRBGwpApU0VARERRimiXVLTW1lVtf9TiaK2Kq1ar1mq1Vq2j1jqq1bb6bevswFrrwq24BwgFcSFDGSqyIUDy+f1xEIgTELgE38/H4x7kPve5u/cn0bxzd5+7D8cYYyCEEEKIzhEJHQAhhBBCHo2SNCGEEKKjKEkTQgghOoqSNCGEEKKjKEkTQgghOoqSNCGEEKKjKEkTQgghOoqSNCGEEKKjKEkTQgghOoqSNCE67ObNm+A4DtHR0UKHQggRACVpQhoYx3FPnGbPni10iIQQHSUROgBCmrr09HTN661bt2LWrFlITEzUlJmamgoRFiFED9CRNCENzM7OTjMpFApwHKeZt7GxwdKlS+Ho6AipVIr27dtjz549j92WSqXC8OHD4enpiZSUFADA33//jQ4dOsDIyAgtWrTAnDlzUF5erlmH4zj8+OOP6NevH4yNjeHu7o4dO3Zolt+/fx+hoaGwtraGTCaDu7s71q9f/9gY/vjjD/j4+EAmk8HKygrBwcEoLCzULP/xxx/h5eUFIyMjeHp64vvvv9daPzU1FYMGDYK5uTksLS3x1ltv4ebNm5rlQ4cORd++fbFkyRLY29vDysoK4eHhKCsrq/F7TkiTwQghjWb9+vVMoVBo5pcuXcrkcjnbvHkzu3z5Mvv000+ZgYEBu3LlCmOMsaSkJAaAXbx4kZWUlLB+/foxPz8/lpWVxRhj7OjRo0wul7MNGzaw69evs3379jFXV1c2e/ZszT4AMEdHR7Zp0yZ29epVNn78eGZqasru3bvHGGMsPDyctW/fnp09e5YlJSWxyMhItmPHjkfGf/v2bSaRSNjSpUtZUlISu3TpElu1ahXLz89njDH222+/MXt7e/bnn3+yGzdusD///JNZWlqyDRs2MMYYKy0tZV5eXmz48OHs0qVLLD4+nv3f//0f8/DwYEqlkjHGWFhYGJPL5WzUqFEsISGB/fPPP8zY2JitXbu2fj8MQvQAJWlCGtGDSdrBwYHNmzdPq06nTp3YmDFjGGNVSfrYsWOse/furGvXriwnJ0dTt3v37mz+/Pla6//666/M3t5eMw+Aff7555r5goICBoDt3r2bMcZYnz592LBhw2oU//nz5xkAdvPmzUcub9myJdu0aZNW2dy5c1lgYKAmNg8PD6ZWqzXLlUolk8lkbO/evYwxPkm7uLiw8vJyTZ23336bDR48uEYxEtKU0DVpQgSSl5eH27dvIygoSKs8KCgIMTExWmXvvvsuHB0dcfDgQchkMk15TEwMoqKiMG/ePE2ZSqVCSUkJioqKYGxsDABo166dZrmJiQnkcjmysrIAAKNHj8aAAQNw4cIF9OjRA3379kWXLl0eGbOvry+6d+8OHx8fhISEoEePHhg4cCAsLCxQWFiI69evY8SIEfjwww8165SXl0OhUGjivXbtGszMzLS2W1JSguvXr2vmvb29IRaLNfP29vaIjY19wrtJSNNESZoQPfD666/jt99+w8mTJ/Hqq69qygsKCjBnzhz079//oXWMjIw0rw0MDLSWcRwHtVoNAOjVqxeSk5Oxa9cuREZGonv37ggPD8eSJUse2qZYLEZkZCROnDiBffv2YcWKFZgxYwZOnz6t+UGwbt06BAQEPLReZbwdO3bExo0bH9q2tbV1jeIl5HlCSZoQgcjlcjg4OCAqKgovvfSSpjwqKgqdO3fWqjt69Gi0bdsWb775Jv79919N/Q4dOiAxMRGtWrV6plisra0RFhaGsLAwdOvWDVOmTHlkkgb4hBkUFISgoCDMmjULLi4uiIiIwKRJk+Dg4IAbN24gNDT0ket26NABW7duhY2NDeRy+TPFTMjzgJI0IQKaMmUKvvjiC7Rs2RLt27fH+vXrER0d/cgjzXHjxkGlUuGNN97A7t270bVrV8yaNQtvvPEGnJ2dMXDgQIhEIsTExCAuLg5fffVVjWKYNWsWOnbsCG9vbyiVSuzcuRNeXl6PrHv69GkcOHAAPXr0gI2NDU6fPo07d+5o6s+ZMwfjx4+HQqFAz549oVQqce7cOdy/fx+TJk1CaGgoFi9ejLfeegtffvklHB0dkZycjL/++guffvopHB0d6/5mEtIEUZImREDjx49Hbm4uPvnkE2RlZaFNmzbYsWMH3N3dH1l/4sSJUKvVeP3117Fnzx6EhIRg586d+PLLL7Fw4UIYGBjA09MTH3zwQY1jMDQ0xPTp03Hz5k3IZDJ069YNW7ZseWRduVyOo0ePYtmyZcjLy4OLiwu++eYb9OrVCwDwwQcfwNjYGIsXL8aUKVNgYmICHx8fTJw4EQBgbGyMo0ePYurUqejfvz/y8/PRvHlzdO/enY6sCXkEjjHGhA6CEEIIIQ+jh5kQQgghOoqSNCGEEKKjKEkTQgghOoqSNCGEEKKjKEkTQgghOoqSNCGEEKKjKEk/wapVq+Dq6gojIyMEBATgzJkzgsZz9OhR9OnTBw4ODuA4Dtu3b9dazhjDrFmzYG9vD5lMhuDgYFy9elWrTnZ2NkJDQyGXy2Fubo4RI0agoKBAq86lS5fQrVs3GBkZwcnJCYsWLXoolt9//x2enp4wMjKCj48Pdu3a9UxtW7BgATp16gQzMzPY2Nigb9++WmMuA/zzncPDw2FlZQVTU1MMGDAAmZmZWnVSUlLQu3dvGBsbw8bGBlOmTNEathEADh8+jA4dOkAqlaJVq1bYsGHDQ/HU52e/evVqtGvXDnK5HHK5HIGBgdi9e7fet+tRvv76a3Acp7kvWp/bN3v2bHAcpzV5enrqfbsq3bp1C++99x6srKwgk8ng4+ODc+fOaZbr6/eJq6vrQ58bx3EIDw8HoIefm7Dje+iuLVu2MENDQ/bzzz+z//77j3344YfM3NycZWZmChbTrl272IwZM9hff/3FALCIiAit5V9//TVTKBRs+/btLCYmhr355pvMzc2NFRcXa+r07NmT+fr6slOnTrFjx46xVq1asXfffVezPDc3l9na2rLQ0FAWFxfHNm/ezGQyGfvhhx80daKiophYLGaLFi1i8fHx7PPPP2cGBgYsNja2zm0LCQlh69evZ3FxcSw6Opq9/vrrzNnZmRUUFGjqjBo1ijk5ObEDBw6wc+fOsRdeeIF16dJFs7y8vJy1bduWBQcHs4sXL7Jdu3axZs2asenTp2vq3LhxgxkbG7NJkyax+Ph4tmLFCiYWi9mePXs0der7s9+xYwf7999/2ZUrV1hiYiL77LPPmIGBAYuLi9Prdj3ozJkzzNXVlbVr145NmDBBU66v7fviiy+Yt7c3S09P10x37tzR+3Yxxlh2djZzcXFhQ4cOZadPn2Y3btxge/fuZdeuXdPU0dfvk6ysLK3PLDIykgFghw4dYozp3+dGSfoxOnfuzMLDwzXzKpWKOTg4sAULFggYVZUHk7RarWZ2dnZs8eLFmrKcnBwmlUrZ5s2bGWOMxcfHMwDs7Nmzmjq7d+9mHMexW7duMcYY+/7775mFhYVmbF/GGJs6dSrz8PDQzA8aNIj17t1bK56AgAD20Ucf1Vv7srKyGAB25MgRTVsMDAzY77//rqmTkJDAALCTJ08yxvgfMSKRiGVkZGjqrF69msnlck17Pv30U+bt7a21r8GDB7OQkBDNfGN89hYWFuzHH39sMu3Kz89n7u7uLDIykr300kuaJK3P7fviiy+Yr6/vI5fpc7sY4/9Pd+3a9bHLm9L3yYQJE1jLli2ZWq3Wy8+NTnc/QmlpKc6fP4/g4GBNmUgkQnBwME6ePClgZI+XlJSEjIwMrZgVCgUCAgI0MZ88eRLm5ubw9/fX1AkODoZIJMLp06c1dV588UUYGhpq6oSEhCAxMRH379/X1Km+n8o69fne5ObmAgAsLS0BAOfPn0dZWZnWfj09PeHs7KzVPh8fH9ja2mrFlZeXh//++69GsTf0Z69SqbBlyxYUFhYiMDCwybQrPDwcvXv3figGfW/f1atX4eDggBYtWiA0NBQpKSlNol07duyAv78/3n77bdjY2MDPzw/r1q3TLG8q3yelpaX47bffMHz4cHAcp5efGyXpR7h79y5UKpXWhwQAtra2yMjIECiqJ6uM60kxZ2RkwMbGRmu5RCKBpaWlVp1HbaP6Ph5Xp77eG7VajYkTJyIoKAht27bV7NPQ0BDm5uZPbF9dY8/Ly0NxcXGDffaxsbEwNTWFVCrFqFGjEBERgTZt2uh9uwBgy5YtuHDhAhYsWPDQMn1uX0BAADZs2IA9e/Zg9erVSEpKQrdu3ZCfn6/X7QKAGzduYPXq1XB3d8fevXsxevRojB8/Hv/73/+04tP375Pt27cjJycHQ4cO1exL3z43GmCD6Jzw8HDExcXh+PHjQodSbzw8PBAdHY3c3Fz88ccfCAsLw5EjR4QO65mlpqZiwoQJiIyM1Bq/uimoHDQEANq1a4eAgAC4uLhg27ZtkMlkAkb27NRqNfz9/TF//nwAgJ+fH+Li4rBmzRqEhYUJHF39+emnn9CrVy84ODgIHUqd0ZH0IzRr1gxisfihHn+ZmZmws7MTKKonq4zrSTHb2dkhKytLa3l5eTmys7O16jxqG9X38bg69fHejB07Fjt37sShQ4e0hi20s7NDaWkpcnJynti+usYul8shk8ka7LM3NDREq1at0LFjRyxYsAC+vr747rvv9L5d58+fR1ZWFjp06ACJRAKJRIIjR45g+fLlkEgksLW11ev2VWdubo7WrVvj2rVrev+52dvbo02bNlplXl5emtP5TeH7JDk5Gfv379caEU4fPzdK0o9gaGiIjh074sCBA5oytVqNAwcOIDAwUMDIHs/NzQ12dnZaMefl5eH06dOamAMDA5GTk4Pz589r6hw8eBBqtRoBAQGaOkePHkVZWZmmTmRkJDw8PGBhYaGpU30/lXWe5b1hjGHs2LGIiIjAwYMH4ebmprW8Y8eOMDAw0NpvYmIiUlJStNoXGxur9cURGRkJuVyu+UJ6WuyN9dmr1WoolUq9b1f37t0RGxuL6OhozeTv74/Q0FDNa31uX3UFBQW4fv067O3t9f5zCwoKeugWxytXrsDFxQWA/n+fAMD69ethY2OD3r17a8r08nOrVTez58iWLVuYVCplGzZsYPHx8WzkyJHM3Nxcq8dfY8vPz2cXL15kFy9eZADY0qVL2cWLF1lycjJjjL9lwtzcnP3999/s0qVL7K233nrkLRN+fn7s9OnT7Pjx48zd3V3rlomcnBxma2vLhgwZwuLi4tiWLVuYsbHxQ7dMSCQStmTJEpaQkMC++OKLZ74Fa/To0UyhULDDhw9r3T5RVFSkqTNq1Cjm7OzMDh48yM6dO8cCAwNZYGCgZnnlrRM9evRg0dHRbM+ePcza2vqRt05MmTKFJSQksFWrVj3y1on6/OynTZvGjhw5wpKSktilS5fYtGnTGMdxbN++fXrdrsep3rtbn9v3ySefsMOHD7OkpCQWFRXFgoODWbNmzVhWVpZet4sx/nY5iUTC5s2bx65evco2btzIjI2N2W+//aapo8/fJyqVijk7O7OpU6c+tEzfPjdK0k+wYsUK5uzszAwNDVnnzp3ZqVOnBI3n0KFDDMBDU1hYGGOMv21i5syZzNbWlkmlUta9e3eWmJiotY179+6xd999l5mamjK5XM6GDRvG8vPzterExMSwrl27MqlUypo3b86+/vrrh2LZtm0ba926NTM0NGTe3t7s33//faa2PapdANj69es1dYqLi9mYMWOYhYUFMzY2Zv369WPp6ela27l58ybr1asXk8lkrFmzZuyTTz5hZWVlWnUOHTrE2rdvzwwNDVmLFi209lGpPj/74cOHMxcXF2ZoaMisra1Z9+7dNQlan9v1OA8maX1t3+DBg5m9vT0zNDRkzZs3Z4MHD9a6j1hf21Xpn3/+YW3btmVSqZR5enqytWvXai3X5++TvXv3MgAPxcuY/n1uHGOM1e7YmxBCCCGNga5JE0IIITqKkjQhhBCioyhJE0IIITqKkjQhhBCioyhJE0IIITqKkjQhhBCioyhJP4FSqcTs2bOhVCqFDqVBNOX2Udv0E7VNP1HbGg7dJ/0EeXl5UCgUyM3NhVwuFzqceteU20dt00/UNv1EbWs4dCRNCCGE6ChK0oQQQoiOavLjSZeXl+PixYuwtbWFSFS73yT5+fkAgFu3biEvL68hwhNUU24ftU0/Udv00/PQttTUVBQVFcHPzw8SSeOlziZ/Tfrs2bPo3Lmz0GEQQghpAs6cOYNOnTo12v6a/JG0ra0tAP6Ntbe3FzgaQggh+ig9PR2dO3fW5JTG0uSTdOUpbnt7ezg6OgocDSGEEH1W28umz7y/Rt0bIYQQQmqMkjQhhBCioyhJE0IIITqqyV+TJoQ0bWq1GqWlpUKHQZoAQ0PDRr/m/DSUpGuoQFmOAwmZeNPXARzHCR0OIQRAaWkpkpKSoFarhQ6FNAEikQhubm4wNDQUOhQNStI1UKZS47VF+/BC8VG0LnsNXp26Cx0SIc89xhjS09MhFovh5OSkc0dARL+o1Wrcvn0b6enpcHZ21pmDMUrSNWAgFmGe+Q68qtqM+KOxACVpQgRXXl6OoqIiODg4wNjYWOhwSBNgbW2N27dvo7y8HAYGBkKHA4A6jtWYRdcRAADPvCiUZF4TOBpCiEqlAgCdOjVJ9Fvlv6XKf1u6gJJ0Dfn6+uOkqANEHEPavuVCh0MIqaArpyWJ/tPFf0uUpGtIJOKQ7hkGAHC48TugLBA4IkIIIU0dJela8H91IK6r7WHMipBz6hehwyGEEACAq6srli1bVuP6hw8fBsdxyMnJabCYAGDDhg0wNzdv0H00dZSka8G5mSmOmPcDAKhP/wDQbR+EkFrgOO6J0+zZs+u03bNnz2LkyJE1rt+lSxekp6dDoVDUaX+k8VDv7lqy6BKGvN3rYVl0E+z6QXDuwUKHRAjRE+np6ZrXW7duxaxZs5CYmKgpMzU11bxmjEGlUtVo7GJra+taxWFoaAg7O7tarUOEQUfStdTDrxUi8AoAIO/ISoGjIYToEzs7O82kUCjAcZxm/vLlyzAzM8Pu3bvRsWNHSKVSHD9+HNevX8dbb70FW1tbmJqaolOnTti/f7/Wdh883c1xHH788Uf069cPxsbGcHd3x44dOzTLHzzdXXlaeu/evfDy8oKpqSl69uyp9aOivLwc48ePh7m5OaysrDB16lSEhYWhb9++tXoPVq9ejZYtW8LQ0BAeHh749ddfNcsYY5g9ezacnZ0hlUrh4OCA8ePHa5Z///33cHd3h5GREWxtbTFw4MBa7VsfUZKuJROpBLfc34OacVCkHQLuXRc6JEII+C/4otJyQSbGWL21Y9q0afj666+RkJCAdu3aoaCgAK+//joOHDiAixcvomfPnujTpw9SUlKeuJ05c+Zg0KBBuHTpEl5//XWEhoYiOzv7sfWLioqwZMkS/Prrrzh69ChSUlIwefJkzfKFCxdi48aNWL9+PaKiopCXl4ft27fXqm0RERGYMGECPvnkE8TFxeGjjz7CsGHDcOjQIQDAn3/+iW+//RY//PADrl69iu3bt8PHxwcAcO7cOYwfPx5ffvklEhMTsWfPHrz44ou12r8+otPddfBK4As4mNgeweKLKD+5BpI3FgsdEiHPveIyFdrM2ivIvuO/DIGxYf18nX755Zd47bXXNPOWlpbw9fXVzM+dOxcRERHYsWMHxo4d+9jtDB06FO+++y4AYP78+Vi+fDnOnDmDnj17PrJ+WVkZ1qxZg5YtWwIAxo4diy+//FKzfMWKFZg+fTr69eP75axcuRK7du2qVduWLFmCoUOHYsyYMQCASZMm4dSpU1iyZAleeeUVpKSkwM7ODsHBwTAwMICzszM6d+4MAEhJSYGJiQneeOMNmJmZwcXFBX5+frXavz4S9Ej66NGj6NOnDxwc+OdhP/irjDGGWbNmwd7eHjKZDMHBwbh69aowwVYT4GaJnbI3AQAseiNQkidwRISQpsLf319rvqCgAJMnT4aXlxfMzc1hamqKhISEpx5Jt2vXTvPaxMQEcrkcWVlZj61vbGysSdAAYG9vr6mfm5uLzMxMTcIEALFYjI4dO9aqbQkJCQgKCtIqCwoKQkJCAgDg7bffRnFxMVq0aIEPP/wQERERKC8vBwC89tprcHFxQYsWLTBkyBBs3LgRRUVFtdq/PhL0SLqwsBC+vr4YPnw4+vfv/9DyRYsWYfny5fjf//4HNzc3zJw5EyEhIYiPj4eRkZEAEfNEIg4u/r1x9fiPcFRlwyA9BnDrJlg8hBBAZiBG/Jchgu27vpiYmGjNT548GZGRkViyZAlatWoFmUyGgQMHPnXkrwcfa8lx3BMHInlU/fo8jV8TTk5OSExMxP79+xEZGYkxY8Zg8eLFOHLkCMzMzHDhwgUcPnwY+/btw6xZszB79mycPXu2Sd/mJeiRdK9evfDVV19pTp9UxxjDsmXL8Pnnn+Ott95Cu3bt8Msvv+D27du1vg7SEAZ0dMK4snEIKFmB2xb+T1+BENKgOI6DsaFEkKkhn1QVFRWFoUOHol+/fvDx8YGdnR1u3rzZYPt7FIVCAVtbW5w9e1ZTplKpcOHChVptx8vLC1FRUVplUVFRaNOmjWZeJpOhT58+WL58OQ4fPoyTJ08iNjYWACCRSBAcHIxFixbh0qVLuHnzJg4ePPgMLdN9OntNOikpCRkZGQgOrrrFSaFQICAgACdPnsQ777zzyPWUSiWUSqVmPj8/v0Hic7YyhsK1PS4nZSPi4i2Ev9KqQfZDCHm+ubu746+//kKfPn3AcRxmzpwpyNCc48aNw4IFC9CqVSt4enpixYoVuH//fq1+oEyZMgWDBg2Cn58fgoOD8c8//+Cvv/7S9FbfsGEDVCoVAgICYGxsjN9++w0ymQwuLi7YuXMnbty4gRdffBEWFhbYtWsX1Go1PDw8GqrJOkFne3dnZGQAAGxtbbXKbW1tNcseZcGCBVAoFJqp+i+0+jawoyMA4I9zqWCZ/zXYfgghz6+lS5fCwsICXbp0QZ8+fRASEoIOHTo0ehxTp07Fu+++i/fffx+BgYEwNTVFSEhIrS499u3bF9999x2WLFkCb29v/PDDD1i/fj1efvllAIC5uTnWrVuHoKAgtGvXDvv378c///wDKysrmJub46+//sKrr74KLy8vrFmzBps3b4a3t3cDtVg3cKyxLzo8BsdxiIiI0Nxzd+LECQQFBeH27duwt7fX1Bs0aBA4jsPWrVsfuZ0Hj6Rv3bqFNm3aIDU1FY6OjvUac6GyHF3m7cH/MBPtRTeA8LOAdet63Qch5NFKSkqQlJQENzc3QfuoPK/UajW8vLwwaNAgzJ07V+hw6sWT/k2lpaXBycmpQXLJk+jskXTl03AyMzO1yjMzM5/4pBypVAq5XK6ZzMzMGixGE6kEwW2dkMUsUMYZAunRDbYvQggRUnJyMtatW4crV64gNjYWo0ePRlJSEv7v//5P6NCaNJ1N0m5ubrCzs8OBAwc0ZXl5eTh9+jQCAwMFjEzbwI6OmFv+Hl5Vf49izwFCh0MIIQ1CJBJhw4YN6NSpE4KCghAbG4v9+/fDy8tL6NCaNEE7jhUUFODatWua+aSkJERHR8PS0hLOzs6YOHEivvrqK7i7u2tuwXJwcKj1Y+gaUoCbJZi5K9LuF2NffAbeat9c6JAIIaTeOTk5PdQzmzQ8QY+kz507Bz8/P81TYyZNmgQ/Pz/MmjULAPDpp59i3LhxGDlyJDp16oSCggLs2bNHp64/iUQcBnSo6EB2Pg1IvwSoVQJHRQghpCkQNEm//PLLYIw9NG3YsAEA35nsyy+/REZGBkpKSrB//360bq17HbMqk/T7N6cDP3QDru4TOCJCCCFNgc5ek9YnzlbGCHCzxHVW0Qv99BphAyKEENIkUJKuJwM7OuI31WtQQQTcOAxkXRY6JEIIIXqOknQ9ed3HHtkGdohUVTxw/swPwgZECCFE71GSricmUgl6tbXH+vKKYeBitgDF94UNihBCiF6jJF2PBnZ0xGnmiUTmApQVARd+FTokQkgT9PLLL2PixImaeVdXVyxbtuyJ6zxqOOC6qK/tPMns2bPRvn37Bt2HvqAkXY8C3CzhaGGMn8p78AVn1tHtWIQQjT59+qBnz56PXHbs2DFwHIdLly7Vertnz57FyJEjnzU8LY9LlOnp6ejVq1e97os8HiXpelR5z/TfqiDki8yA3BQgcbfQYRFCdMSIESMQGRmJtLS0h5atX78e/v7+aNeuXa23a21tDWNj4/oI8ans7OwglUobZV+EknS9G9jREUoY4tfSV/gCuh2LEFLhjTfegLW1teZZEJUKCgrw+++/Y8SIEbh37x7effddNG/eHMbGxvDx8cHmzZufuN0HT3dfvXoVL774IoyMjNCmTRtERkY+tM7UqVPRunVrGBsbo0WLFpg5cybKysoA8ENGzpkzBzExMeA4DhzHaT2/ovrp7tjYWLz66quQyWSwsrLCyJEjUVBQoFk+dOhQ9O3bF0uWLIG9vT2srKwQHh6u2VdNqNVqfPnll3B0dIRUKkX79u2xZ88ezfLS0lKMHTsW9vb2MDIygouLCxYsWAAAYIxh9uzZcHZ2hlQqhYODA8aPH1/jfQtNZ8eT1ldOlsZ4oYUlfr3xGkZJ/oXo5jEg8z/AtmkPp0aIzigtrP06Yikgrvg6VJUDKiXAiQAD2dO3a2hS491IJBK8//772LBhA2bMmKEZi/n333+HSqXCu+++i4KCAnTs2BFTp06FXC7Hv//+iyFDhqBly5bo3LnzU/ehVqvRv39/2Nra4vTp08jNzdW6fl3JzMwMGzZsgIODA2JjY/Hhhx/CzMwMn376KQYPHoy4uDjs2bNHM9azQqF4aBuFhYUICQlBYGAgzp49i6ysLHzwwQcYO3as1g+RQ4cOwd7eHocOHcK1a9cwePBgtG/fHh9++GGN3rfvvvsO33zzDX744Qf4+fnh559/xptvvon//vsP7u7uWL58OXbs2IFt27bB2dkZqampSE1NBQD8+eef+Pbbb7FlyxZ4e3sjIyMDMTExNdqvLqAk3QAGdnTC5BvZOCp5AS+XRwGnfwDeXC50WIQ8H+Y71H6dtzcA3v3415f/AX4fCrh0BYb9W1VnmQ9QdO/hdWfn1mpXw4cPx+LFi3HkyBHNOMrr16/HgAEDoFAooFAoMHnyZE39cePGYe/evdi2bVuNkvT+/ftx+fJl7N27Fw4O/Hsxf/78h64jf/7555rXrq6umDx5MrZs2YJPP/0UMpkMpqamkEgkTxx1cNOmTSgpKcEvv/wCExP+x8rKlSvRp08fLFy4ELa2tgAACwsLrFy5EmKxGJ6enujduzcOHDhQ4yS9ZMkSTJ06Fe+88w4AYOHChTh06BCWLVuGVatWISUlBe7u7ujatSs4joOLi4tm3ZSUFNjZ2SE4OBgGBgZwdnau0fuoK+h0dwPo1dYOxoZirCzszhdc2gaU5AkbFCFEJ3h6eqJLly74+eefAQDXrl3DsWPHMGLECACASqXC3Llz4ePjA0tLS5iammLv3r1ISUmp0fYTEhLg5OSkSdAAHjly4NatWxEUFAQ7OzuYmpri888/r/E+qu/L19dXk6ABICgoCGq1GomJiZoyb29viMVizby9vT2ysrJqtI+8vDzcvn0bQUFBWuVBQUFISEgAwJ9Sj46OhoeHB8aPH499+6oezfz222+juLgYLVq0wIcffoiIiAiUl5fXqp1CoiPpBmAileB1H3v8cb4c+22HI7j/B4CRXOiwCHk+fHa79uuIq3WE8uzDb4N74BhmYuyzxVXNiBEjMG7cOKxatQrr169Hy5Yt8dJLLwEAFi9ejO+++w7Lli2Dj48PTExMMHHiRJSWltbb/k+ePInQ0FDMmTMHISEhUCgU2LJlC7755pt620d1BgYGWvMcx0GtVtfb9jt06ICkpCTs3r0b+/fvx6BBgxAcHIw//vgDTk5OSExMxP79+xEZGYkxY8ZozmQ8GJcuoiPpBjKwoyMADh9n9ESxhafQ4RDy/DA0qf0krna8IpbwZdWvRz9pu3UwaNAgiEQibNq0Cb/88guGDx+uuT4dFRWFt956C++99x58fX3RokULXLlypcbb9vLyQmpqKtLT0zVlp06d0qpz4sQJuLi4YMaMGfD394e7uzuSk5O1m2toCJXqybeQenl5ISYmBoWFVdfro6KiIBKJ4OHhUeOYn0Qul8PBweGhYTKjoqLQpk0brXqDBw/GunXrsHXrVvz555/Izs4GAMhkMvTp0wfLly/H4cOHcfLkScTG1t+ProZESbqBdHa1hJOlDPnKcuyLz+AL6Z5pQggAU1NTDB48GNOnT0d6ejqGDh2qWebu7o7IyEicOHECCQkJ+Oijj5CZmVnjbQcHB6N169YICwtDTEwMjh07hhkzZmjVcXd3R0pKCrZs2YLr169j+fLliIiI0Krj6uqKpKQkREdH4+7du1AqlQ/tKzQ0FEZGRggLC0NcXBwOHTqEcePGYciQIZrr0fVhypQpWLhwIbZu3YrExERMmzYN0dHRmDBhAgBg6dKl2Lx5My5fvowrV67g999/h52dHczNzbFhwwb89NNPiIuLw40bN/Dbb79BJpNpXbfWZZSkG0j1caYPnI4GIkYBP70GMCZsYIQQnTBixAjcv38fISEhWtePP//8c3To0AEhISF4+eWXYWdnh759+9Z4uyKRCBERESguLkbnzp3xwQcfYN68eVp13nzzTXz88ccYO3Ys2rdvjxMnTmDmzJladQYMGICePXvilVdegbW19SNvAzM2NsbevXuRnZ2NTp06YeDAgejevTtWrlxZuzfjKcaPH49Jkybhk08+gY+PD/bs2YMdO3bA3d0dAN9TfdGiRfD390enTp1w8+ZN7Nq1CyKRCObm5li3bh2CgoLQrl077N+/H//88w+srKzqNcaGwjHWtLNGWloanJyckJqaCkdHx0bdd2p2EbotOgRzrgAXTCZAVF4MfHAQcOzYqHEQ0hSVlJQgKSkJbm5uMDIyEjoc0gQ86d+UULmEjqQbUOU90znMFJFunwIfUoImhBBSc5SkG1hoAH/d4/Ob7VBq6ydwNIQQQvQJJekG1rOtHWzMpLiTr8TuuIrelmUlwgZFCCFEL1CSbmAGYpHmaHrziavAPxOBbzyAwrvCBkYIIUTn6XSSVqlUmDlzJtzc3CCTydCyZUvMnTsX+tbX7d0AJxiIOZxKKUBRygWgJAc4+6PQYRFCCNFxOp2kFy5ciNWrV2PlypVISEjAwoULsWjRIqxYsULo0GrFxswIr/vYA+Dwl7QvX3hmHZ32JqQe6NuPdqK7dPHfkk4/FvTEiRN466230Lt3bwD8zfWbN2/GmTNnBI6s9t4PdMXf0bcx72ZrvGvpCHFeGnBpK9AxTOjQCNFLBgYG4DgOd+7cgbW1teaJXYTUBWMMd+7cAcdxOvW4UJ1O0l26dMHatWtx5coVtG7dGjExMTh+/DiWLl0qdGi11sHZHD7NFYi9lYvT1m+jS963wMlVQIf3AfpyIaTWxGIxHB0dkZaWhps3bwodDmkCOI6Do6Oj1mAgQtPpJD1t2jTk5eXB09MTYrEYKpUK8+bNQ2ho6GPXUSqVWo+vy8/Pb4xQn4rjOLwf6IIpf1zC7LSO2GtoBu5uInDtAOAeLHR4hOglU1NTuLu7o6ysTOhQSBNgYGCgUwka0PEkvW3bNmzcuBGbNm2Ct7c3oqOjMXHiRDg4OCAs7NGniRcsWIA5c+Y0cqQ108fXAfN3JeBKbhmSfQbA9eoG4OQKStKEPAOxWKxzX6yE1Bed7jg2ZcoUTJs2De+88w58fHwwZMgQfPzxx1iwYMFj15k+fTpyc3M1U3x8fCNG/GRGBmK809kZALA071V+KLwbh4GMOGEDI4QQopN0OkkXFRVBJNIOUSwWP3EcUqlUCrlcrpnMzMwaOsxaee8FF4g4YEeyBPkt+A5xOLlK2KAIIYToJJ1O0n369MG8efPw77//4ubNm4iIiMDSpUvRr18/oUOrs+bmMrzWhh/CbaPoDb4w9ncgP0PAqAghhOginU7SK1aswMCBAzFmzBh4eXlh8uTJ+OijjzB37lyhQ3smYV1cAQDLE81R7hgAqMv4+6YJIYSQanQ6SZuZmWHZsmVITk5GcXExrl+/jq+++gqGhoZCh/ZMAltYwd3GFEWlKhyxHMQXnvsJKC0UNjBCCCE6RaeTdFPFcRzerziann/dDaxNX+CNbwGxVNC4CCGE6BZK0gLp79ccZlIJrt8rwRHfxYB3P0Cs03fEEUIIaWSUpAViIpVgoL8jAOCXk8kCR0MIIUQXUZIW0PuBrgCAQ4lZSEnPAI4vA/4YIWhMhBBCdAclaQG5NTPBS62twRgQcfIycHAuEPcHcOuC0KERQgjRAZSkBTa0ogPZT5eUKOsyCei7GrD1FjYoQgghOoGStMBeam0NFytj5JWU43ezIUD7/wMk1MubEEIIJWnBiUQchrzgAgD434mbOjnoOCGEEGFQktYBb/s7QWYgRmJmPs5czwRO/wCs6QaU5AkdGiGEEAFRktYBCpkB+nVoDgD438lU4OyPQMYl4OJvDbtjxoCyYqDwLnD/JpAZD2Tf4OfLSxt234QQQp6Knp6hI94PdMGm0ynYm3AHOb0+hPmBKcDp1UDnDwFVKZ9My4r4v6WFFfPFgKEJ4BJYtaETK4Gie8ALYwBTa74sehNw4VegtIBft3IqKwTYY0YUM3cGJsZWzUeMBvJuAcGzgeYd+LKMWCDpKCA14+MwMK42ySqmiteGJoDYoEHeO0IIaaooSesITzs5XmhhiVM3srE+vzM+llkCOSnA3GZPXtGxM/BBZNX8yZVAfjrg3bcqSeenAyknnrwdAxPAwAgoK+GTt1ShvTz1NJB9HSgvqSpLPgHs/azGbYRIAsibAxMvVZUd/hrITQP8h1clf1U5P9a2iE70EEKeb5SkdUhYoCtO3cjGr+fvYOzLE2Bw4AvtChKjiqNTk6ojVatW2nV83+WPuI3Mq8o83wAsWwKGpvwRrWYyrToCrp4QVeVAebH2dnsvAQruAM08qsosWwBtBwLKPKC0qOpIX/O3WPtoXV3+8JH75X/5U/ueb1SVxW8H/g4HFE78Eb2FC2Duov3a2ArguNq8vYSQpoIx/ruEqQG1quo1q3zNtMvl9kJHXGeUpHXIa21sYa8wQnpuCXYYD8SAyf/HH1EayACJrGZHlsFfPFxm7cFPNSWWAGIz7bKWrz5cz/01fnoSxipO11ckblWZ9vJuk4A7VwC7tlVlOcn8Efu9q/z0KCIDwEgByMz5HySWbsCAH6uWx/3J/3Bo1R2QO/BlZRU/PAxkT46ZEH2nKuf/zzEVILOoKk89C5TkAk6d+P8/AJB2Hkg5yf+IrvwhXflaXc4nO7VKu0xmAfSoNmRw5BfAvWvAi5MBBz++7NoB4MTyimTJ+Fg0ibNyexX7qpwXS4GxZ6q2u+194NpBfgCidm/zZVf2ApsG1fy94MTAF9l1ex91ACVpHSIRi/DeCy5YvDcRv5y8iQEduwod0rPjOP6+b4lU+8uikne/h8u6jAe8+/On+3OS+b/3k6vm89P5MbiL7vITwH/xVHf0GyDrP2BIRFWSvrQN+Gc8f4Ru2YI/C2HVCrBqyf81d6br5qTmGOPPIpXkAQrHqjM7N6OAzP+A0nz+8lF5CVCu5M9OlSv5+QfLm/sDbyyt2vaydnwfkg8O8D9AAeDoYv7OD42K/VXul7GKbRfz/z8AoHlH4MODVav8MQzITeXLmnesiPcYsP8RP+6fROGknaSTjgK3LwB+Q6rKCjKBG4drt12JkfZ8WQn/Pla/zMbV4jIYJ6pdfR1ESVrHvNPJCd/tv4qYtFxcTLkPP+dHJLamTmzAfzFVfjk9qKyE7xxXkgMU5/B/RQ/8U3brBiia89fAKykrbmnLTeWnpCPa64gk/Kl0TfJuAVh7Aq5N4McSebLyUqDwDlCYxV/Wqf66OJv/EViSy/9beKWiH0ZZMfC1M/96ehrfgRIAYjbV/s6M6penAKAom09O1S8PlRbycdVG2QOXraw9+R/L1f+/2LYF2g3mjzhFYn6Z5q+kon+IpNokejjeoAn8D2Ybr6oy50Cg31p+W5yo4q+42vYq/nLV9ynW3u4b3/IJ2sS6qsztRWDytYrtcVXb1OxDVFHGNYlLYhxr4k/PSEtLg5OTE1JTU+Ho6Ch0ODUyaVs0/rpwC/38muPbwe2FDqfpYIz/8rt3jZ+yr1e8vs5PD16HBwBrLyD8VNX8ulf5L+vBv1V9ISXuBi7v1L7mb1Dx19gScOig19fE9F65Erh+ECjIAjq8X/XFvX8OkPAPn/hKcmq2Lc83gHc28q8ZA+Za89ubcKnqMz73M38EaWjGd8aUVJsenJdI+csvJtaAQ/uq/dy5widoyxaAxJAvy8/kf5yCVe2ff1G1nkRW7c4KGb+PJpCodIFQuYSOpHXQ0C6u+OvCLey8dBuf9vSAvYKuodYLjgNMrPjJOUB7mVrNn0Z/MHGbO2vXy07ij6xQ7YvvdvTTj5zkjvx1QMfOgFNnwM6HHv9aV2o1n6zy04H8jKq/BRlV824vAq99yddXlQGb3+Fft+1fdcRbdFe7z4NIwidLk2aAiQ1gasPPG1vx12+NFHzHxUocxx9BGzxwitZ/OD89C+vWD5eZ2fITea5QktZB7RzN0d7JHNGpOXhx0SG80MIKId526NHGFjZyo6dvgNSeSMSfHlc057/gH+f9v/lrhdWTd4uX+YSruQe92v3oebf5a+N5acB/acB/Efw6Dn7AyMNV2yi8x/94qC3G+P0VZfM/HgzNgGYVPf7Vav7apC79GKjs0FRWBCjz+dgq38tyJf8gH2U+8OKUqlOfRxfzZytK8vhLFkX3+E5GT1L99KjUFHDuAhjJ+X1UJumAUYDPIL6uqQ1/Cre2t/09mKAJqWeUpHXUV33bYuLWaFzLKsCxq3dx7OpdzPw7Dn5O5gjxtkOItx1cm5kIHebzx77dw2UugdoPlHmQsoDvVJN6Bkg7y/+t7AEL8IljqSd/9DbycNX97UlHgZxUPvlWJuGie0DRfe0yVbWnwwWOBULm8a/zbwPfevP3vE9Pqapz+GvgzmVAKq84QpTzdTiu2q0sD0x27fie8pXtifqOL+8+s2q7Ucv5++lLC6sScWlR1a14ZcXasQL8tdD+a6vmK++7DxjF99wH+Pfg1vkH3lSOT65mdoCZ/QN/7QCLB/ozDN/98OdCo80RPaDzSfrWrVuYOnUqdu/ejaKiIrRq1Qrr16+Hv7+/0KE1qLbNFdg/6SVcyyrAvvgM7P0vEzGpObiQwk8Ldl+Gh60ZenjbIsTbDt4OcnB07Uk3SU35o/PKI/TKx7FWunuFPzIsL+ZPtVaK/IJP7jUhlvKnZS1cq8oqe7w/2GM96RiQfLx2bfAfUZWky0uAo4v4169+XnXN89Z5/tp8jXD8Ea2oWmwSKZ+0DYwf2PcwoHXPih8TZnw7TW2pJz55LtSp41hqaio4jtNcPD9z5gw2bdqENm3aYOTIkfUW3P379+Hn54dXXnkFo0ePhrW1Na5evYqWLVuiZcuWNdqGPnYce5z03GJExmdi738ZOHUjGyp11UfX3FymSdidXC0hFlHC1ivKAv72supHd7un8Qnc2BKQWfLJydiS751rXDEvs+RfGxg/3EFIreZPD5cVVd2GBvD3md6/WdVjuSSXr8dYtd6xomq9ZEV8b3nfd6piPTCHLw9ZUHWK+Np+/ja5yoftGBprPyq2+rxESh2aiF4RKpfUKUl369YNI0eOxJAhQ5CRkQEPDw94e3vj6tWrGDduHGbNmlUvwU2bNg1RUVE4duxYnbfRlJJ0dTlFpTiQkIV98Rk4cuUOSsqqbtUwNzaAq5UJrM2kaGYqhbWZFNamhlrzzUylMJHq/IkUQgjRCXqVpC0sLHDq1Cl4eHhg+fLl2Lp1K6KiorBv3z6MGjUKN27cqJfg2rRpg5CQEKSlpeHIkSNo3rw5xowZgw8//LDG22iqSbq64lIVjl69g73/ZeBAQhZyi8uevhIAY0NxtaTNJ3EHcxmcLY3hZGEMZ0tjmBsb0Gl0QshzT69uwSorK4NUyvcY3b9/P958800AgKenJ9LT0+stuBs3bmD16tWYNGkSPvvsM5w9exbjx4+HoaEhwsLCHrmOUqmEUqnUzOfn59dbPLpKZijWdCYrU6nx3+08ZOSW4E6BEnfzldp/C5S4k69ESZkaRaUqpGQXISW76LHbNpVK4GRpDCcLGZws+cTtZMknckcLYxgZiB+7LiGEkGdTpyTt7e2NNWvWoHfv3oiMjMTcufzj4W7fvg0rqzrcRvIYarUa/v7+mD9/PgDAz88PcXFxWLNmzWOT9IIFCzBnzpx6i0HfGIhFaO9kDjg9vg5jDIWlKtzJr0radwuUyMpTIu1+EVLvFyM1uwhZ+UoUKMuRkJ6HhPS8R27L2kwKZ0tjhHjbYmgXNxhK9PsRfIQQokvqlKQXLlyIfv36YfHixQgLC4Ovry8AYMeOHejcuXO9BWdvb482bdpolXl5eeHPP/987DrTp0/HpEmTNPO3bt16aBvPO47jYCqVwFQqgdsTbuMqKVPxSTu7GKn3i5ByrwiplfPZRchXluNOPp/kzyffx5azqfjyzbbo6v6U4TUJIYTUSJ2S9Msvv4y7d+8iLy8PFhZVz5YeOXIkjI2Nn7Bm7QQFBSExMVGr7MqVK3BxcXnMGoBUKtWcigeAvLxHHwGSpzMyEKOVjRla2Zg9tIwxhtziMqRmFyM6LQff7b+CG3cK8d5Pp/G6jx0+790GDub0pDRCCHkWdTo3WVxcDKVSqUnQycnJWLZsGRITE2FjY1NvwX388cc4deoU5s+fj2vXrmHTpk1Yu3YtwsPD620fpG44joO5sSF8HBUY8oILDnzyMoZ2cYWIA3bFZqD7N0fw/eFrKC1XP31jhBBCHqlOSfqtt97CL7/8AgDIyclBQEAAvvnmG/Tt2xerV6+ut+A6deqEiIgIbN68GW3btsXcuXOxbNkyhIaG1ts+SP1QyAww+01v7BzXDZ1cLVBcpsKiPYnouewojl6p5cg9hBBCANQxSV+4cAHdunUDAPzxxx+wtbVFcnIyfvnlFyxfvrxeA3zjjTcQGxuLkpISJCQk1Or2K9L42jjIse2jQCwd5ItmplLcuFuI938+g9G/ncetnEeMMkUIIeSx6pSki4qKYGbGX6fct28f+vfvD5FIhBdeeAHJycn1GiDRPxzHoX8HRxyc/BKGB7lBLOKwOy4Dwd8cwapD16AsVwkdIiGE6IU6JelWrVph+/btSE1Nxd69e9GjRw8AQFZWFuRyeb0GSPSX3MgAs/q0wb/ju6KzqyWKy1RYvDcRPZcdwxE6BU4IIU9VpyQ9a9YsTJ48Ga6urujcuTMCA/kRgPbt2wc/P7+nrE2eN552cmz96AUsG9we1mZSJN0tRNjPZ/DRr+eQcu/xD1IhhJDnXZ0eCwoAGRkZSE9Ph6+vL0QVD9g/c+YM5HI5PD096zXIZ/E8PBZUn+SXlGHZ/qvYcOImVGoGEQe86mmD915wwYvu1hDRwCCEEB2kV8/uri4tLQ0AdDYBUpLWTYkZ+fjq33gcu3pXU+ZiZYz3Alzwtr8jzI0NBYyOEEK0CZVL6nS6W61W48svv4RCoYCLiwtcXFxgbm6OuXPnQq2m+2LJ03nYmeHXEQHYP+klDO3iCjOpBMn3ijBvVwIC5h/A5N9jEJOaI3SYhBAiqDo9cWzGjBn46aef8PXXXyMoKAgAcPz4ccyePRslJSWYN29evQZJmq5WNqaY/aY3Pu3pgb+jb+OXk8lISM/DH+fT8Mf5NLSreFhKH18HGsyDEPLcqdPpbgcHB6xZs0Yz+lWlv//+G2PGjMGtW7fqLcBnRae79QtjDBdScvDbqWT8eykdpSr+zIxCZoBB/o4IDXCB6xOeN04IIQ1Br4aqzM7OfmTnME9PT2RnZz9zUOT5xXEcOrpYoKOLBT7v7YWt51Kx8VQKbuUUY92xJKw7loQXW1vjpdbWkIg4iEQcRBwg5jiIuGrzIg4cV7WM4ziIRRy87M3gaFF/z5cnhJCGVKck7evri5UrVz70dLGVK1eiXbt29RIYIVamUox5uRU+erElDidm4ddTyThy5Q6OVkx1YSDm8MvwAAS2rL8hVQkhpKHUKUkvWrQIvXv3xv79+zX3SJ88eRKpqanYtWtXvQZIiFjEobuXLbp72SL5XiG2nk1FSnYRGANUagY1q5yq5h+1LLuwFEl3CxG+6QL+GdcVzWmULkKIjqtTkn7ppZdw5coVrFq1CpcvXwYA9O/fHyNHjsRXX32lea43IfXNxcoEn/as2334JWUqDFxzAnG38jDyl3P4Y1QXyAypMxohRHc9833S1cXExKBDhw5QqXTn2czUcYxUdyunGG+uOI57haXo294B3w5uD46jB6gQQp5Mr+6TJkRfNTeXYVVoB4hFHLZH38ZPx5OEDokQQh6LkjR57rzQwgqz3mgDAJi/KwHHrtJgH4QQ3URJmjyX3g90wdsdHaFmwNhNF2mgD0KITqpVx7H+/fs/cXlOTs6zxEJIo+E4DnP7tsWVrALEpOZg5K/n8OfoLjCR1qkvJSGENIhaHUkrFIonTi4uLnj//fcbKlZC6pWRgRg/vNcRzUyluJyRjyl/xKAe+1ESQsgzq9Vhw/r16xsqDkIEYacwwpr3OuDddaewKzYD3x++jvBXWgkdFiGEAKBr0oTA39USc95sCwBYsi8Rhy5nCRwRIYTwKEkTAuD/ApwRGuAMxoDxWy7ixp0CoUMihBD9StJff/01OI7DxIkThQ6FNEFf9PGGv4sF8kvKMfLX88gvKRM6JELIc05vkvTZs2fxww8/0AAepMEYSkT4/r0OsJMb4VpWASZti4FaTR3JCCHC0YskXVBQgNDQUKxbtw4WFhZCh0OaMBszI/wwpCMMJSJExmdi+cGrQodECHmO6UWSDg8PR+/evREcHPzUukqlEnl5eZopPz+/ESIkTYmvkznm9eU7ki3bfxV7/8sQOCJCyPNK55P0li1bcOHCBSxYsKBG9RcsWKB173abNm0aOELSFL3t74ShXVwBAJO2RuNqJv3YI4Q0Pp1O0qmpqZgwYQI2btwIIyOjGq0zffp05Obmaqb4+PgGjpI0VTN6e+GFFpYoLFVh6PqzWHXoGqJTc6Ci69SEkEZSr0NV1rft27ejX79+EIurxvxVqVTgOA4ikQhKpVJr2aPQUJXkWdwrUOLNlVG4lVOsKTMzkiCwhRWCWjVDUCsrtLQ2febhLstUalzLKkBCeh7ib+fhflEZPu3pAVt5zX6cEkIallC5RKcfVNy9e3fExsZqlQ0bNgyenp6YOnXqUxM0Ic/KylSKf8Z1xT8xtxF17S5O3riH/JJy7IvPxL74TACArVyKLi2boUtLPnE7mMueuM3cojLEp+fxCbkiKV/LKkCpSq1Vj+OAJW/7NljbCCG6T6eTtJmZGdq2batVZmJiAisrq4fKCWkoliaGCOviirAurihXqRF3Ow9R1+7ixPW7OHvzPjLzlIi4eAsRF28BAFo0M0GXVlYIatkM7ramuJZVgPjbeYhPz0dCep7WUXl1ZlIJvOzlcLI0xp8X0vB39C180qM17BVPTvqEkKZLp5M0IbpGIhahvZM52juZI/yVVigpU+FC8n1EXb+L49fuITYtBzfuFuLG3UL8dirlsdtxtJDBy16ONvZyeNnL4e0gh6OFTHPaPPV+Ec4kZWN91E189rpXYzWPEKJj9C5JHz58WOgQCNEwMhCjS6tm6NKqGaaEALnFZTh94x5OXL+HqGt3kXq/CK1sTDXJuI29HJ72cihkBk/c7qiXWuBMUjY2nU5B+CutnlqfENI06V2SJkSXKWQG6OFthx7eds+0nZdb26C1rSmuZBZg0+kUjH65ZT1FSAjRJzp9CxYhzyuRiMPIF/nE/HNUEpTlKoEjIoQIgZI0ITrqTV8H2MmNcCdfie0VndIIIc8XStKE6ChDiQgjuroBAH44eoMG+yDkOURJmhAd9k5nJ5gZSXDjTiH2J2QKHQ4hpJFRkiZEh5kZGeC9F1wAAGuP3hA4GkJIY6MkTYiOG9bFFYZiEc4l38e5m9lCh0MIaUSUpAnRcTZyI/Tv0BwAf22aEPL8oCRNiB748MUW4DggMj4T17IKhA6HENJIKEkTogdaWpsi2MsWALCOjqYJeW5QkiZET4x6qQUAIOLiLWTmlQgcDSGkMVCSJkRPdHSxhL+LBUpVaqyPuil0OISQRkBJmhA98tFL/KNCN55KRn5JmcDREEIaGiVpQvRId08btLQ2Qb6yHJvPPH4oTEJI00BJmhA9IhJx+Khi4I2fjiehtFwtcESEkIZESZoQPfOWnwNszKTIzFPi72gaeIOQpoySNCF6RioRY3jFwBtraeANQpo0StKE6KH/C3CGqVSCq1kFOJSYJXQ4hJAGQkmaED0kNzJAaIAzAOCHI8I83CQhPQ/v/3wGW89SBzZCGgolaUL01LAgNxiIOZy5mY0LKfcbdd+HE7MwcPUJHL1yB59FxOFiI++fkOeFTifpBQsWoFOnTjAzM4ONjQ369u2LxMREocMiRCfYKYzQtz0/8MbaRjya3ng6GSP+dw6FpSqYSSVQqRkmbYtBUWl5o8VAyPNCp5P0kSNHEB4ejlOnTiEyMhJlZWXo0aMHCgsLhQ6NEJ0w8kX+UaF74zNw407DDryhVjPM35WAGRFxUKkZBnRwxKEpL8NeYYSku4WYvyuhQfdPyPNIp5P0nj17MHToUHh7e8PX1xcbNmxASkoKzp8/L3RohOgEd1szBHvZgDFg3bGkBttPSZkK4ZsuYG3F4B6TXmuNJW+3QzNTKZa87QsA+O1UCnViI6Se6XSSflBubi4AwNLSUuBICNEdlY8K/fNCGrLy63/gjTv5Sryz9hR2x2XAUCzCssHtMb67OziOAwAEtWqG4UH8LWGf/nEJ2YWl9R4DIc8rvUnSarUaEydORFBQENq2bfvYekqlEnl5eZopPz+/EaMkpPH5u1igg7M5SsvV+N+Jm/W67WtZ+ej3fRSiU3NgbmyAX0d0Rl+/5g/V+7SnB1rZmOJOvhIzImLBGN27TUh90JskHR4ejri4OGzZsuWJ9RYsWACFQqGZ2rRp00gREiIMjuM0R9O/nkxGgbJ+OnCduH4X/b8/gbT7xXCxMsZfo7sgoIXVI+saGYixbHB7SEQcdsdlIOIiPQmNkPqgF0l67Nix2LlzJw4dOgRHR8cn1p0+fTpyc3M1U3x8fCNFSYhwXvOyRYtmJsgrKUffVVH4Zl8iYtNy63xE+8f5NLz/0xnklZSjo4sFIsYEoYW16RPXadtcgY9faw0A+OLv/5B2v6hO+yaEVNHpJM0Yw9ixYxEREYGDBw/Czc3tqetIpVLI5XLNZGZm1giREiIskYjDjN5eMBSLcC2rACsOXkOflccR9PVBfPF3HI5fvYsy1dMH42CMYem+REz+PQblaoY32tlj4wcBsDQxrFEcH73YAh1dLJCvLMcn22LokaWEPCOO6fDFozFjxmDTpk34+++/4eHhoSlXKBSQyWQ12kZaWhqcnJyQmpr61KNwQvRdTlEpDiVmYd9/mThy5Q6KSlWaZXIjCV71tEEPbzu82NoaplKJ1rrKchU+/eMS/o6+DQAIf6UlPnnNAyIRV6sYku8Votd3x1BUqsKM173wYcVtYoToM6FyiU4n6creow9av349hg4dWqNtUJImz6uSMhWirt1FZHwmIuMzca9ar2tDsQhBrazQw9sO3b1sYCAS4aNfz+PMzWxIRBzm9WuLwZ2c67zvzWdSMP2vWBiKRfhnXFd42NEZLaLfKEk3EErShAAqNcPFlPvYF5+Jff9l4Oa9quvFHMc/Czy3uAxmUglWv9cRXd2bPdP+GGP44H/ncOByFrzs5dge3gVSifhZm0GIYITKJTp9TZoQUj/EIg7+rpb47HUvHJr8MiI/fhFTQjzg66gAY0BucRmam8vw55guz5ygAf4s2NcD2sHSxBAJ6Xn4NvJqPbSCkOeP5OlVCCFNCcdxcLc1g7utGcJfaYWM3BKcvZmNoFbNatxBrCaszaRY0N8HH/16Hj8cvY5XPW3Q2Y0eRERIbdCRNCHPOTuFEfr4OtRrgq4U4m2Htzs6gjFg0rZo5JeU1fs+CGnKKEkTQhrUrD5t4GghQ9r9YszdSc8tIKQ2KEkTQhqUmZEBlg5qD44Dtp1Lw97/MoQOiRC9QUmaENLgOrtZaobVnP5XLO7kKwWOiBD9QEmaENIoJr3WGp52ZsguLMW0Py/RIByE1AAlaUJIo5BKxFj2TnsYikU4cDkL30ZeaZChNQlpSihJE0IajaedHFNC+Ef8Lj94DZ3nHUCfFcexNPIKolNz6FnfhDyA7pMmhDSqEV3dwHHAjpjbuJSWi9hb/LT8wFU0MzXES61t8IqnNbq5W0MhMxA6XEIERY8FJYQIJiu/BIcT7+DQ5Swcu3pXayxsiYhDRxcLvOppg1c9bdDKxvSxz/MnpKHRs7sbCCVpQvRDabka525m41BiFg5ezsL1O4Vayx0tZHjV0wbdvWwR2MIKhhK6WkcaDyXpBkJJmhD9lHKvCAcvZ+Jg4h2cunEPpeVV42GbGUnQ3dMGId52eMnDGsaGdOWONCxK0g2EkjQh+q+otBxR1+7h4OUs7E/I1LrPWioR4cXW1gjxtkOwlw3Mjev/8aaEUJJuIJSkCWla1GqGi6n3sScuA3v/y0RKdtWwm2IRhxdaWKKntx16eNvBVm4kYKSkKaEk3UAoSRPSdDHGkJCej73/ZWDvfxm4nJGvtdzP2Rwh3nYI8baDWzMTgaIkTQEl6QZCSZqQ58fNu4WahH0hJUdrmY2ZFObGBlDIDCA3qvhbORlJNPOa5cZ8ualUQr3KiWC5hHpbEEKaDNdmJvjopZb46KWWyMwrwb74TOz7LwMnr99DVr4SWXV4ZriJoRjezRXwdVTAx9Ecvo4KOFsaU+ImjYKSNCGkSbKVG2HICy4Y8oILcovLkJpdhNziMuQWlyGv8m9J5Xz5Q/N5xWUoValRWKrCmaRsnEnK1mxbITNAO0cFfJor0M5RgXaO5rBXGFHiJvWOkjQhpMlTyAygaK6o1TqMMSjL1Ui+V4RLaTm4lJaLS7dykXA7D7nFZTh29S6OXb2rqd/M1BDtHM01idvdxgwGEg4ijgPHAWKOfy3iOHAiQMRxEFcs48sr/ooo0ZMqepGkV61ahcWLFyMjIwO+vr5YsWIFOnfuLHRYhJAmjOM4GBmI4WFnBg87M7zt7wSAf+jKlcx8PmlXJO/EzHzcLSjFwcv8g1iefd8AB2gSPMdxWvOiivnKZSIOEItEaGZqCFu5EezkRrCVS2GjeW0EW4UUViZSiOlHgF7R+SS9detWTJo0CWvWrEFAQACWLVuGkJAQJCYmwsbGRujwCCHPGUOJCG2bK9C2uQL/F+AMACgpUyE+PQ+xabmISctBbFouUrKLwBigYgxqxlCbLrqMAQyAWrNSzVa+W6B8qId7dWIRB2tTKWwVRrA1k8JWbgQbMymMpRIYSkSQikWQGohgqPkr5sslIq2/fF0xxOKqswFiUdVrOu1ff3S+d3dAQAA6deqElStXAgDUajWcnJwwbtw4TJs27anrU+9uQoguYBWJWs0YVNVeqyv/qqteM1ZRH1XzVeUAA1+XVawP8Kfm7+QrkZWnRGZeCTLySpBZ8TozrwR3C5RorEHGqp+6F1U/1V8x/7gk/rR0VHnWgKu4VKB5LeIe2id/1oGDuOLSwraPAmFkIK5zm6h39yOUlpbi/PnzmD59uqZMJBIhODgYJ0+eFDAyQgipHa7yVDU4Qb54y1Vq3C0o1STwrIoknpVfgpIyNZTlKpSWq1GqUkNZxv8tLVdDWV71t3qdJ+XTyh8bjfaroAnT6SR99+5dqFQq2NraapXb2tri8uXLj1xHqVRCqay6zSI///Gnfggh5HkhEYtgpzCCncIIvs+4LcYYylT8aXyVmlWcCag6tV95VkCleV1Zr/opfG2POrZ+8IC7Mu9X7lP7cgKDSl15doKPR82q6hmK9XNAFp1O0nWxYMECzJkzR+gwCCGkyeI4DoYSuu7cGHT6p0WzZs0gFouRmZmpVZ6ZmQk7O7tHrjN9+nTk5uZqpvj4+MYIlRBCCKl3Op2kDQ0N0bFjRxw4cEBTplarceDAAQQGBj5yHalUCrlcrpnMzMwaK1xCCCGkXun86e5JkyYhLCwM/v7+6Ny5M5YtW4bCwkIMGzZM6NAIIYSQBqXzSXrw4MG4c+cOZs2ahYyMDLRv3x579ux5qDMZIYQQ0tTofJIGgLFjx2Ls2LFCh0EIIYQ0Kr1I0s9CrVYDANLT0wWOhBBCiL6qzCGVOaWxNPkkXdkznJ71TQgh5FllZmbC2dm50fan848FfVbl5eW4ePEibG1tIRLpdGd2APzDV9q0aYP4+Pgm0zO9qbWpqbUHaHptovboPn1rk1qtRmZmJvz8/CCRNN7xbZNP0vomLy8PCoUCubm5kMvlQodTL5pam5pae4Cm1yZqj+5rim1qCLp/aEkIIYQ8pyhJE0IIITqKkrSOkUql+OKLLyCVSoUOpd40tTY1tfYATa9N1B7d1xTb1BDomjQhhBCio+hImhBCCNFRlKQJIYQQHUVJmhBCCNFRlKR1xIIFC9CpUyeYmZnBxsYGffv2RWJiotBh1Zuvv/4aHMdh4sSJQofyTG7duoX33nsPVlZWkMlk8PHxwblz54QOq05UKhVmzpwJNzc3yGQytGzZEnPnzoU+dVM5evQo+vTpAwcHB3Ach+3bt2stZ4xh1qxZsLe3h0wmQ3BwMK5evSpMsDXwpPaUlZVh6tSp8PHxgYmJCRwcHPD+++/j9u3bwgVcA0/7jKobNWoUOI7DsmXLGi0+XUdJWkccOXIE4eHhOHXqFCIjI1FWVoYePXqgsLBQ6NCe2dmzZ/HDDz+gXbt2QofyTO7fv4+goCAYGBhg9+7diI+PxzfffAMLCwuhQ6uThQsXYvXq1Vi5ciUSEhKwcOFCLFq0CCtWrBA6tBorLCyEr68vVq1a9cjlixYtwvLly7FmzRqcPn0aJiYmCAkJQUlJSSNHWjNPak9RUREuXLiAmTNn4sKFC/jrr7+QmJiIN998U4BIa+5pn1GliIgInDp1Cg4ODo0UmZ5gRCdlZWUxAOzIkSNCh/JM8vPzmbu7O4uMjGQvvfQSmzBhgtAh1dnUqVNZ165dhQ6j3vTu3ZsNHz5cq6x///4sNDRUoIieDQAWERGhmVer1czOzo4tXrxYU5aTk8OkUinbvHmzABHWzoPteZQzZ84wACw5OblxgnpGj2tTWloaa968OYuLi2MuLi7s22+/bfTYdBUdSeuo3NxcAIClpaXAkTyb8PBw9O7dG8HBwUKH8sx27NgBf39/vP3227CxsYGfnx/WrVsndFh11qVLFxw4cABXrlwBAMTExOD48ePo1auXwJHVj6SkJGRkZGj921MoFAgICMDJkycFjKz+5ObmguM4mJubCx1KnanVagwZMgRTpkyBt7e30OHonCY/CpY+UqvVmDhxIoKCgtC2bVuhw6mzLVu24MKFCzh79qzQodSLGzduYPXq1Zg0aRI+++wznD17FuPHj4ehoSHCwsKEDq/Wpk2bhry8PHh6ekIsFkOlUmHevHkIDQ0VOrR6kZGRAQCwtbXVKre1tdUs02clJSWYOnUq3n33Xb1+9vXChQshkUgwfvx4oUPRSZSkdVB4eDji4uJw/PhxoUOps9TUVEyYMAGRkZEwMjISOpx6oVar4e/vj/nz5wMA/Pz8EBcXhzVr1uhlkt62bRs2btyITZs2wdvbG9HR0Zg4cSIcHBz0sj3Pk7KyMgwaNAiMMaxevVrocOrs/Pnz+O6773DhwgVwHCd0ODqJTnfrmLFjx2Lnzp04dOgQHB0dhQ6nzs6fP4+srCx06NABEokEEokER44cwfLlyyGRSKBSqYQOsdbs7e3Rpk0brTIvLy+kpKQIFNGzmTJlCqZNm4Z33nkHPj4+GDJkCD7++GMsWLBA6NDqhZ2dHYCqMeUrZWZmapbpo8oEnZycjMjISL0+ij527BiysrLg7Oys+Z5ITk7GJ598AldXV6HD0wl0JK0jGGMYN24cIiIicPjwYbi5uQkd0jPp3r07YmNjtcqGDRsGT09PTJ06FWKxWKDI6i4oKOih2+KuXLkCFxcXgSJ6NkVFRQ+NsS4Wi6FWqwWKqH65ubnBzs4OBw4cQPv27QHwwyOePn0ao0ePFja4OqpM0FevXsWhQ4dgZWUldEjPZMiQIQ/1VwkJCcGQIUMwbNgwgaLSLZSkdUR4eDg2bdqEv//+G2ZmZpprZgqFAjKZTODoas/MzOyh6+kmJiawsrLS2+vsH3/8Mbp06YL58+dj0KBBOHPmDNauXYu1a9cKHVqd9OnTB/PmzYOzszO8vb1x8eJFLF26FMOHDxc6tBorKCjAtWvXNPNJSUmIjo6GpaUlnJ2dMXHiRHz11Vdwd3eHm5sbZs6cCQcHB/Tt21e4oJ/gSe2xt7fHwIEDceHCBezcuRMqlUrzPWFpaQlDQ0Ohwn6ip31GD/7QMDAwgJ2dHTw8PBo7VN0kdPdywgPwyGn9+vVCh1Zv9P0WLMYY++eff1jbtm2ZVCplnp6ebO3atUKHVGd5eXlswoQJzNnZmRkZGbEWLVqwGTNmMKVSKXRoNXbo0KFH/r8JCwtjjPG3Yc2cOZPZ2toyqVTKunfvzhITE4UN+gme1J6kpKTHfk8cOnRI6NAf62mf0YPoFixtNAoWIYQQoqOo4xghhBCioyhJE0IIITqKkjQhhBCioyhJE0IIITqKkjQhhBCioyhJE0IIITqKkjQhhBCioyhJE0IIITqKkjQhpMY4jsP27duFDoOQ5wYlaUL0xNChQ8Fx3ENTz549hQ6NENJAaIANQvRIz549sX79eq0yqVQqUDSEkIZGR9KE6BGpVAo7OzutycLCAgB/Knr16tXo1asXZDIZWrRogT/++ENr/djYWLz66quQyWSwsrLCyJEjUVBQoFXn559/hre3N6RSKezt7TF27Fit5Xfv3kW/fv1gbGwMd3d37NixQ7Ps/v37CA0NhbW1NWQyGdzd3R/6UUEIqTlK0oQ0ITNnzsSAAQMQExOD0NBQvPPOO0hISAAAFBYWIiQkBBYWFjh79ix+//137N+/XysJr169GuHh4Rg5ciRiY2OxY8cOtGrVSmsfc+bMwaBBg3Dp0iW8/vrrCA0NRXZ2tmb/8fHx2L17NxISErB69Wo0a9as8d4AQpoaoYfhIoTUTFhYGBOLxczExERrmjdvHmOMH+501KhRWusEBASw0aNHM8YYW7t2LbOwsGAFBQWa5f/++y8TiUQsIyODMcaYg4MDmzFjxmNjAMA+//xzzXxBQQEDwHbv3s0YY6xPnz5s2LBh9dNgQgija9KE6JFXXnkFq1ev1iqztLTUvA4MDNRaFhgYiOjoaABAQkICfH19YWJiolkeFBQEtVqNxMREcByH27dvo3v37k+MoV27dprXJiYmkMvlyMrKAgCMHj0aAwYMwIULF9CjRw/07dsXXbp0qVNbCSHUcYwQvWJiYvLQ6ef6IpPJalTPwMBAa57jOKjVagBAr169kJycjF27diEyMhLdu3dHeHg4lixZUu/xEvI8oGvShDQhp06demjey8sLAODl5YWYmBgUFhZqlkdFRUEkEsHDwwNmZmZwdXXFgQMHnikGa2trhIWF4bfffsOyZcuwdu3aZ9oeIc8zOpImRI8olUpkZGRolUkkEk3nrN9//x3+/v7o2rUrNm7ciDNnzuCnn34CAISGhuKLL75AWFgYZs+ejTt37mDcuHEYMmQIbG1tAQCzZ8/GqFGjYGNjg169eiE/Px9RUVEYN25cjeKbNWsWOnbsCG9vbyiVSuzcuVPzI4EQUnuUpAnRI3v27IG9vb1WmYeHBy5fvgyA73m9ZcsWjBkzBvb29ti8eTPatGkDADA2NsbevXsxYcIEdOrUCcbGxhgwYACWLl2q2VZYWBhKSkrw7bffYvLkyWjWrBkGDhxY4/gMDQ0xffp03Lx5EzKZDN26dcOWLVvqoeWEPJ84xhgTOghCyLPjOA4RERHo27ev0KEQQuoJXZMmhBBCdBQlaUIIIURH0TVpQpoIunJFSNNDR9KEEEKIjqIkTQghhOgoStKEEEKIjqIkTQghhOgoStKEEEKIjqIkTQghhOgoStKEEEKIjqIkTQghhOgoStKEEEKIjvp/UBeKXn5NbVkAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from previous_chapters import plot_losses\n", "\n", "epochs_tensor = torch.linspace(1, n_epochs, len(train_losses))\n", "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)\n", "plt.tight_layout(); plt.savefig(\"3.pdf\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "c16fa614-67e1-4254-8b7e-c3e2f690c29c", "metadata": {}, "source": [ "- Note that the model is overfitting here because the dataset is kept very small for educational purposes (so that the code can be executed on a laptop computer)\n", "- For a longer pretraining run on a much larger dataset, see [../../ch05/03_bonus_pretraining_on_gutenberg](../../ch05/03_bonus_pretraining_on_gutenberg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" } }, "nbformat": 4, "nbformat_minor": 5 }