From 6a585e08bc7f32fb1d21b2e45955f22f7f1b0ff5 Mon Sep 17 00:00:00 2001 From: rasbt Date: Mon, 11 Mar 2024 07:07:36 -0500 Subject: [PATCH] Add appendix D --- README.md | 31 +- .../01_main-chapter-code/appendix-D.ipynb | 738 ++++++++++++++++++ .../01_main-chapter-code/previous_chapters.py | 318 ++++++++ .../01_main-chapter-code/the-verdict.txt | 165 ++++ 4 files changed, 1236 insertions(+), 16 deletions(-) create mode 100644 appendix-D/01_main-chapter-code/appendix-D.ipynb create mode 100644 appendix-D/01_main-chapter-code/previous_chapters.py create mode 100644 appendix-D/01_main-chapter-code/the-verdict.txt diff --git a/README.md b/README.md index c223930..fabf0fa 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ This repository contains the code for coding, pretraining, and finetuning a GPT- -In [*Build a Large Language Model (from Scratch)*](http://mng.bz/orYv), you'll discover how LLMs work from the inside out. In this book, I'll guide you step by step through creating your own LLM, explaining each stage with clear text, diagrams, and examples. +In [*Build a Large Language Model (From Scratch)*](http://mng.bz/orYv), you'll discover how LLMs work from the inside out. In this book, I'll guide you step by step through creating your own LLM, explaining each stage with clear text, diagrams, and examples. The method described in this book for training and developing your own small-but-functional model for educational purposes mirrors the approach used in creating large-scale foundational models such as those behind ChatGPT. @@ -31,21 +31,20 @@ Alternatively, you can view this and other files on GitHub at [https://github.co

-| Chapter Title | Main Code (for quick access) | All Code + Supplementary | -|------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|-------------------------------| -| Ch 1: Understanding Large Language Models | No code | No code | -| Ch 2: Working with Text Data | - [ch02.ipynb](ch02/01_main-chapter-code/ch02.ipynb)
- [dataloader.ipynb](ch02/01_main-chapter-code/dataloader.ipynb) (summary)
- [exercise-solutions.ipynb](ch02/01_main-chapter-code/exercise-solutions.ipynb) | [./ch02](./ch02) | -| Ch 3: Coding Attention Mechanisms | - [ch03.ipynb](ch03/01_main-chapter-code/ch03.ipynb)
- [multihead-attention.ipynb](ch03/01_main-chapter-code/multihead-attention.ipynb) (summary)
- [exercise-solutions.ipynb](ch03/01_main-chapter-code/exercise-solutions.ipynb)| [./ch03](./ch03) | -| Ch 4: Implementing a GPT Model from Scratch | - [ch04.ipynb](ch04/01_main-chapter-code/ch04.ipynb)
- [gpt.py](ch04/01_main-chapter-code/gpt.py) (summary)
- [exercise-solutions.ipynb](ch04/01_main-chapter-code/exercise-solutions.ipynb) | [./ch04](./ch04) | -| Ch 5: Pretraining on Unlabeled Data | Q1 2024 | ... | -| Ch 6: Finetuning for Text Classification | Q2 2024 | ... | -| Ch 7: Finetuning with Human Feedback | Q2 2024 | ... | -| Ch 8: Using Large Language Models in Practice | Q2/3 2024 | ... | -| Appendix A: Introduction to PyTorch | - [code-part1.ipynb](appendix-A/03_main-chapter-code/code-part1.ipynb)
- [code-part2.ipynb](appendix-A/03_main-chapter-code/code-part2.ipynb)
- [DDP-script.py](appendix-A/03_main-chapter-code/DDP-script.py)
- [exercise-solutions.ipynb](appendix-A/03_main-chapter-code/exercise-solutions.ipynb) | [./appendix-A](./appendix-A) | -| Appendix B: References and Further Reading | No code | | -| Appendix C: Exercises | No code | | - - +| Chapter Title | Main Code (for quick access) | All Code + Supplementary | +|------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|-------------------------------| +| Ch 1: Understanding Large Language Models | No code | - | +| Ch 2: Working with Text Data | - [ch02.ipynb](ch02/01_main-chapter-code/ch02.ipynb)
- [dataloader.ipynb](ch02/01_main-chapter-code/dataloader.ipynb) (summary)
- [exercise-solutions.ipynb](ch02/01_main-chapter-code/exercise-solutions.ipynb) | [./ch02](./ch02) | +| Ch 3: Coding Attention Mechanisms | - [ch03.ipynb](ch03/01_main-chapter-code/ch03.ipynb)
- [multihead-attention.ipynb](ch03/01_main-chapter-code/multihead-attention.ipynb) (summary)
- [exercise-solutions.ipynb](ch03/01_main-chapter-code/exercise-solutions.ipynb)| [./ch03](./ch03) | +| Ch 4: Implementing a GPT Model from Scratch | - [ch04.ipynb](ch04/01_main-chapter-code/ch04.ipynb)
- [gpt.py](ch04/01_main-chapter-code/gpt.py) (summary)
- [exercise-solutions.ipynb](ch04/01_main-chapter-code/exercise-solutions.ipynb) | [./ch04](./ch04) | +| Ch 5: Pretraining on Unlabeled Data | Q1 2024 | ... | +| Ch 6: Finetuning for Text Classification | Q2 2024 | ... | +| Ch 7: Finetuning with Human Feedback | Q2 2024 | ... | +| Ch 8: Using Large Language Models in Practice | Q2/3 2024 | ... | +| Appendix A: Introduction to PyTorch | - [code-part1.ipynb](appendix-A/03_main-chapter-code/code-part1.ipynb)
- [code-part2.ipynb](appendix-A/03_main-chapter-code/code-part2.ipynb)
- [DDP-script.py](appendix-A/03_main-chapter-code/DDP-script.py)
- [exercise-solutions.ipynb](appendix-A/03_main-chapter-code/exercise-solutions.ipynb) | [./appendix-A](./appendix-A) | +| Appendix B: References and Further Reading | No code | - | +| Appendix C: Exercises | No code | - | +| Appendix D: Adding Bells and Whistles to the Training Loop | - [appendix-D.ipynb](appendix-D/01_main-chapter-code/appendix-D.ipynb) | [./appendix-D](./appendix-D) |
> [!TIP] diff --git a/appendix-D/01_main-chapter-code/appendix-D.ipynb b/appendix-D/01_main-chapter-code/appendix-D.ipynb new file mode 100644 index 0000000..4f34140 --- /dev/null +++ b/appendix-D/01_main-chapter-code/appendix-D.ipynb @@ -0,0 +1,738 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "af53bcb1-ff9d-49c7-a0bc-5b8d32ff975b", + "metadata": {}, + "source": [ + "## Appendix D: Adding Bells and Whistles to the Training Loop" + ] + }, + { + "cell_type": "markdown", + "id": "4f58c142-9434-49af-b33a-356b80a45b86", + "metadata": {}, + "source": [ + "- In this appendix, we add a few more advanced features to the training function, which are used in typical pretraining and finetuning; finetuning is covered in chapters 6 and 7\n", + "- The next three sections below discuss learning rate warmup, cosine decay, and gradient clipping\n", + "- The final section adds these techniques to the training function" + ] + }, + { + "cell_type": "markdown", + "id": "744def4f-c03f-42ee-97bb-5d7d5b89b723", + "metadata": {}, + "source": [ + "- We start by initializing a model reusing the code from chapter 5:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "8755bd5e-bc06-4e6e-9e63-c7c82b816cbe", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "torch version: 2.2.1\n", + "tiktoken version: 0.5.1\n" + ] + } + ], + "source": [ + "from importlib.metadata import version\n", + "import torch\n", + "import tiktoken\n", + "\n", + "print(\"torch version:\", version(\"torch\"))\n", + "print(\"tiktoken version:\", version(\"tiktoken\"))\n", + "\n", + "\n", + "from previous_chapters import GPTModel\n", + "\n", + "GPT_CONFIG_124M = {\n", + " \"vocab_size\": 50257, # Vocabulary size\n", + " \"ctx_len\": 256, # Shortened context length (orig: 1024)\n", + " \"emb_dim\": 768, # Embedding dimension\n", + " \"n_heads\": 12, # Number of attention heads\n", + " \"n_layers\": 12, # Number of layers\n", + " \"drop_rate\": 0.1, # Dropout rate\n", + " \"qkv_bias\": False # Query-key-value bias\n", + "}\n", + "\n", + "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", + "\n", + "torch.manual_seed(123)\n", + "model = GPTModel(GPT_CONFIG_124M)\n", + "model.eval(); # Disable dropout during inference" + ] + }, + { + "cell_type": "markdown", + "id": "51574e57-a098-412c-83e8-66dafa5a0b99", + "metadata": {}, + "source": [ + "- Next, using the same code we used in chapter 5, we initialize the data loaders:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "386ca110-2bb4-42f1-bd54-8836df80acaa", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import urllib.request\n", + "\n", + "file_path = \"the-verdict.txt\"\n", + "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n", + "\n", + "if not os.path.exists(file_path):\n", + " with urllib.request.urlopen(url) as response:\n", + " text_data = response.read().decode('utf-8')\n", + " with open(file_path, \"w\", encoding=\"utf-8\") as file:\n", + " file.write(text_data)\n", + "else:\n", + " with open(file_path, \"r\", encoding=\"utf-8\") as file:\n", + " text_data = file.read()" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "ae96992b-536a-4684-a924-658b9ffb7e9c", + "metadata": {}, + "outputs": [], + "source": [ + "from previous_chapters import create_dataloader_v1\n", + "\n", + "# Train/validation ratio\n", + "train_ratio = 0.90\n", + "split_idx = int(train_ratio * len(text_data))\n", + "\n", + "\n", + "torch.manual_seed(123)\n", + "\n", + "train_loader = create_dataloader_v1(\n", + " text_data[:split_idx],\n", + " batch_size=2,\n", + " max_length=GPT_CONFIG_124M[\"ctx_len\"],\n", + " stride=GPT_CONFIG_124M[\"ctx_len\"],\n", + " drop_last=True,\n", + " shuffle=True\n", + ")\n", + "\n", + "val_loader = create_dataloader_v1(\n", + " text_data[split_idx:],\n", + " batch_size=2,\n", + " max_length=GPT_CONFIG_124M[\"ctx_len\"],\n", + " stride=GPT_CONFIG_124M[\"ctx_len\"],\n", + " drop_last=False,\n", + " shuffle=False\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "939c08d8-257a-41c6-b842-019f7897ac74", + "metadata": {}, + "source": [ + "## D.1 Learning rate warmup" + ] + }, + { + "cell_type": "markdown", + "id": "7fafcd30-ddf7-4a9f-bcf4-b13c052b3133", + "metadata": {}, + "source": [ + "- When training complex models like LLMs, implementing learning rate warmup can help stabilize the training\n", + "- In learning rate warmup, we gradually increase the learning rate from a very low value (`initial_lr`) to a user-specified maximum (`peak_lr`)\n", + "- This way, the model will start the training with small weight updates, which helps decrease the risk of large destabilizing updates during the training" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "2bb4790b-b8b6-4e9e-adf4-704a04b31ddf", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "135\n" + ] + } + ], + "source": [ + "n_epochs = 15\n", + "peak_lr = 0.01\n", + "initial_lr = 0.0001\n", + "\n", + "optimizer = torch.optim.AdamW(model.parameters(), lr=peak_lr, weight_decay=0.1)\n", + "total_training_steps = len(train_loader) * n_epochs\n", + "\n", + "print(total_training_steps)" + ] + }, + { + "cell_type": "markdown", + "id": "5bf3a8da-abc4-4b80-a5d8-f1cc1c7cc5f3", + "metadata": {}, + "source": [ + "- Typically, the number of warmup steps is between 10% and 20% of the total number of steps\n", + "- We can compute the increment as the difference between the `peak_lr` and `initial_lr` divided by the number of warmup steps" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "e075f80e-a398-4809-be1d-8019e1d31c90", + "metadata": {}, + "outputs": [], + "source": [ + "warmup_steps = 20\n", + "lr_increment = (peak_lr - initial_lr) / warmup_steps\n", + "\n", + "global_step = -1\n", + "track_lrs = []\n", + "\n", + "for epoch in range(n_epochs):\n", + " for input_batch, target_batch in train_loader:\n", + " optimizer.zero_grad()\n", + " global_step += 1\n", + " \n", + " if global_step < warmup_steps:\n", + " lr = initial_lr + global_step * lr_increment\n", + " else:\n", + " lr = peak_lr\n", + " \n", + " # Apply the calculated learning rate to the optimizer\n", + " for param_group in optimizer.param_groups:\n", + " param_group[\"lr\"] = lr\n", + " track_lrs.append(optimizer.param_groups[0][\"lr\"])\n", + " \n", + " # Calculate loss and update weights" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "cb6da121-eeed-4023-bdd8-3666c594b4ed", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAGwCAYAAACNeeBZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9s0lEQVR4nO3dfVyV9eH/8fdB5cYb8G6BCCoVm5UoCYqY+7omRWUryjV0ftPM6dqsNCynplCtxrIsZ1nkfn2zfZfTuZWV32I5rKxJqIgKmTctCxUP3g1QSu7O9fvDnSNnonHkHK5z83o+HjyM63zO4XN9VvLedb3P51gMwzAEAACAixJk9gQAAAB8GWEKAACgDQhTAAAAbUCYAgAAaAPCFAAAQBsQpgAAANqAMAUAANAGHc2egK+y2WyqqKhQt27dZLFYzJ4OAABoBcMwdPLkSUVHRysoyD3XlAhTF6miokKxsbFmTwMAAFyEAwcOKCYmxi2vRZi6SN26dZN05n+M8PBwk2cDAABao6amRrGxsY7f4+5AmLpI9lt74eHhhCkAAHyMOys6FNABAADagDAFAADQBoQpAACANiBMAQAAtAFhCgAAoA0IUwAAAG1AmAIAAGgDwhQAAEAbEKYAAADagDAFAADQBqaHqWXLlmnAgAEKDQ1VSkqKNm/efMHxa9as0cCBAxUaGqqEhAS98847To+//vrruv7669WrVy9ZLBZt3779nNc4ffq0ZsyYoV69eqlr164aN26cKisr3XlaAAAgQJgaplavXq2srCzl5ORo27ZtGjJkiNLT03XkyJEWx2/atEkTJkzQ1KlTVVJSooyMDGVkZKisrMwxpra2VqNGjdKTTz553p/7wAMP6O2339aaNWv04YcfqqKiQrfffrvbzw8AAPg/i2EYhlk/PCUlRcOGDdPzzz8vSbLZbIqNjdV9992nuXPnnjM+MzNTtbW1WrdunePYiBEjlJiYqLy8PKexX375peLi4lRSUqLExETH8erqan3nO9/RypUr9eMf/1iStHv3bl1xxRUqLCzUiBEjWjX3mpoaRUREqLq6mg869qDqrxt0sq7B7GkAAEzQu2uIQjt1cOtreuL3d0e3vMpFqK+vV3FxsebNm+c4FhQUpLS0NBUWFrb4nMLCQmVlZTkdS09P19q1a1v9c4uLi9XQ0KC0tDTHsYEDB6pfv34XDFN1dXWqq6tzfF9TU9Pqn4mLs+mfx/Tf/69INtPiPgDATH+4e7j+67vfMXsa38q0MHXs2DE1NTUpMjLS6XhkZKR2797d4nOsVmuL461Wa6t/rtVqVXBwsLp37+7S6+Tm5urRRx9t9c9B2/191xHZDKlDkEUdgyxmTwcA0M6CLL7xd79pYcrXzJs3z+mqWE1NjWJjY02ckf8rO1QtSVo0brDGJcWYPBsAAFpmWpjq3bu3OnTocM676CorKxUVFdXic6Kiolwaf77XqK+vV1VVldPVqW97nZCQEIWEhLT656BtmmyGPq04E6YSYiJMng0AAOdn2rv5goODlZSUpIKCAscxm82mgoICpaamtvic1NRUp/GStH79+vOOb0lSUpI6derk9Dp79uxReXm5S68Dz9p/7JRq65sU1qmDLvtOV7OnAwDAeZl6my8rK0uTJ09WcnKyhg8friVLlqi2tlZTpkyRJE2aNEl9+/ZVbm6uJGnmzJkaPXq0Fi9erLFjx2rVqlXaunWrli9f7njNEydOqLy8XBUVFZLOBCXpzBWpqKgoRUREaOrUqcrKylLPnj0VHh6u++67T6mpqa1+Jx88r/Tft/iujA5XB/pSAAAvZmqYyszM1NGjR5WdnS2r1arExETl5+c7Subl5eUKCjp78WzkyJFauXKlFixYoPnz5ys+Pl5r167VoEGDHGPeeustRxiTpPHjx0uScnJy9Mgjj0iSnn32WQUFBWncuHGqq6tTenq6XnjhhXY4Y7RW6cEz75ZM6MstPgCAdzN1nylfxj5TnnVH3iZt+fJfWnzHEMrnAAC38cTvb9M/Tgb4T2fK5/++MkX5HADg5QhT8Dr7j53S15TPAQA+gjAFr2Mvn19F+RwA4AMIU/A6Ow+eCVODKJ8DAHwAYQpex77zOe/kAwD4AsIUvErz8vlgyucAAB9AmIJX+eLomfJ55+AOupTyOQDABxCm4FUcO5/3oXwOAPANhCl4FXuYonwOAPAVhCl4FXv5nL4UAMBXEKbgNZpshsoO8Zl8AADfQpiC1/ji6Cl900D5HADgWwhT8BrsfA4A8EWEKXgNyucAAF9EmILXKD3IzucAAN9DmIJXaL7zOWEKAOBLCFPwCpTPAQC+ijAFr7DzIOVzAIBvIkzBK1A+BwD4KsIUvIJ953P6UgAAX0OYgumal8/5GBkAgK8hTMF0/2xWPo/rTfkcAOBbCFMwXSnlcwCADyNMwXSljr5Ud3MnAgDARSBMwXSO8nlMuMkzAQDAdYQpmIqdzwEAvo4wBVNRPgcA+DrCFExlL58Pio6gfA4A8EmEKZiKnc8BAL6OMAVTlVI+BwD4OMIUTNNkM7SL8jkAwMcRpmAae/m8C+VzAIAPI0zBNDsdO59TPgcA+C7CFExTRvkcAOAHCFMwjb18PjiGMAUA8F2EKZiiscmmTyu4MgUA8H2EKZjin0drdbrBpi7BHXRp7y5mTwcAgItGmIIp7Lf4roqOUBDlcwCADyNMwRRljs06ucUHAPBthCmYYufBKkls1gkA8H2EKbS7xiabdh0+s/M55XMAgK8jTKHdUT4HAPgTwhTanaN83pfyOQDA9xGm0O5K6UsBAPwIYQrtzn5lijAFAPAHhCm0q+blc7ZFAAD4A8IU2tXnR0/pdINNXUM6Kq4X5XMAgO8jTKFdlR48c4vvyuhwyucAAL9AmEK7KqMvBQDwM4QptCt7+XwwfSkAgJ8gTKHdsPM5AMAfEabQbiifAwD8EWEK7cZePr+K8jkAwI8QptBuKJ8DAPwRYQrtZqc9TFE+BwD4EcIU2kVjk02fUT4HAPghwhTaBeVzAIC/Mj1MLVu2TAMGDFBoaKhSUlK0efPmC45fs2aNBg4cqNDQUCUkJOidd95xetwwDGVnZ6tPnz4KCwtTWlqa9u3b5zRm7969uvXWW9W7d2+Fh4dr1KhRev/9991+bjhrJ+VzAICfMjVMrV69WllZWcrJydG2bds0ZMgQpaen68iRIy2O37RpkyZMmKCpU6eqpKREGRkZysjIUFlZmWPMokWLtHTpUuXl5amoqEhdunRRenq6Tp8+7Rhz8803q7GxURs2bFBxcbGGDBmim2++WVar1ePnHKgonwMA/JXFMAzDrB+ekpKiYcOG6fnnn5ck2Ww2xcbG6r777tPcuXPPGZ+Zmana2lqtW7fOcWzEiBFKTExUXl6eDMNQdHS0Zs+erQcffFCSVF1drcjISK1YsULjx4/XsWPH9J3vfEcbN27U97//fUnSyZMnFR4ervXr1ystLa1Vc6+pqVFERISqq6sVHh7e1qXwe7e98A+VlFfpd+MTdWtiX7OnAwAIUJ74/W3alan6+noVFxc7hZegoCClpaWpsLCwxecUFhaeE3bS09Md4/fv3y+r1eo0JiIiQikpKY4xvXr10ve+9z394Q9/UG1trRobG/XSSy/pkksuUVJS0nnnW1dXp5qaGqcvtE5jk027Ks6sF1emAAD+xrQwdezYMTU1NSkyMtLpeGRk5Hlvt1mt1guOt/95oTEWi0V///vfVVJSom7duik0NFTPPPOM8vPz1aNHj/PONzc3VxEREY6v2NhY1044gO07ckp1jWfK5wMonwMA/IzpBfT2ZhiGZsyYoUsuuUQfffSRNm/erIyMDP3oRz/S4cOHz/u8efPmqbq62vF14MCBdpy1b7N/uDHlcwCAPzItTPXu3VsdOnRQZWWl0/HKykpFRUW1+JyoqKgLjrf/eaExGzZs0Lp167Rq1Spdc801Gjp0qF544QWFhYXp1VdfPe98Q0JCFB4e7vSF1rGXzwezWScAwA+ZFqaCg4OVlJSkgoICxzGbzaaCggKlpqa2+JzU1FSn8ZK0fv16x/i4uDhFRUU5jampqVFRUZFjzNdffy3pTD+ruaCgINlstrafGM5h3xaBzToBAP6oo5k/PCsrS5MnT1ZycrKGDx+uJUuWqLa2VlOmTJEkTZo0SX379lVubq4kaebMmRo9erQWL16ssWPHatWqVdq6dauWL18u6UwfatasWXr88ccVHx+vuLg4LVy4UNHR0crIyJB0JpD16NFDkydPVnZ2tsLCwvT73/9e+/fv19ixY01ZB3/WfOdzyucAAH9kapjKzMzU0aNHlZ2dLavVqsTEROXn5zsK5OXl5U5XkEaOHKmVK1dqwYIFmj9/vuLj47V27VoNGjTIMWbOnDmqra3V9OnTVVVVpVGjRik/P1+hoaGSztxezM/P18MPP6wf/vCHamho0FVXXaU333xTQ4YMad8FCACUzwEA/s7UfaZ8GftMtc6ftx7QnL/s1IhLe2rV9JZv3wIA0F78ap8pBIbSg+x8DgDwb4QpeJR9WwTK5wAAf0WYgsc0L58Pjulu7mQAAPAQwhQ8xl4+7xbSUf17djZ7OgAAeARhCh5j70td1ZedzwEA/oswBY+x96UonwMA/BlhCh7jCFP0pQAAfowwBY9oaLJpFzufAwACAGEKHrGv8pTqKZ8DAAIAYQoeUXaI8jkAIDAQpuAR9r4U+0sBAPwdYQoesZOdzwEAAYIwBbdraLbzOeVzAIC/I0zB7Rzl81DK5wAA/0eYgtvZy+eDoiMonwMA/B5hCm6381CVJCkhhlt8AAD/R5iC25UeOtOXonwOAAgEhCm4VfPy+WDCFAAgABCm4FZ7K0+eLZ/3onwOAPB/hCm4VfPyucVC+RwA4P8IU3Ar+87nlM8BAIGCMAW3spfP2awTABAoCFNwG3Y+BwAEIsIU3IbyOQAgEBGm4Db28nlCX8rnAIDAQZiC2+w8eDZMAQAQKAhTcBvHtgiEKQBAACFMwS0ammz6zHpSElemAACBhTAFt6B8DgAIVIQpuEXpQcrnAIDARJiCW5QeonwOAAhMhCm4RRkfIwMACFCEKbRZfSPlcwBA4CJMoc3s5fPw0I7q15PyOQAgsBCm0GbN95eifA4ACDSEKbRZKX0pAEAAI0yhzXgnHwAgkBGm0Cb1jTbtPkz5HAAQuAhTaJO9lSdV30T5HAAQuAhTaJPm+0tRPgcABCLCFNpkZ7N38gEAEIgIU2iTMsrnAIAAR5jCRWtePh/ct7u5kwEAwCSEKVw0e/k8IqyTYnuGmT0dAABMQZjCRSt19KXCKZ8DAAIWYQoXrZTyOQAAhClcPHv5nL4UACCQEaZwUdj5HACAMy4qTDU2Nurvf/+7XnrpJZ08eeYXakVFhU6dOuXWycF7UT4HAOCMjq4+4auvvtINN9yg8vJy1dXV6brrrlO3bt305JNPqq6uTnl5eZ6YJ7wM5XMAAM5w+crUzJkzlZycrH/9618KCzt7ReK2225TQUGBWycH71Xq2Kyzu7kTAQDAZC5fmfroo4+0adMmBQcHOx0fMGCADh065LaJwbuVHmTncwAApIu4MmWz2dTU1HTO8YMHD6pbt25umRS8W32jTXuslM8BAJAuIkxdf/31WrJkieN7i8WiU6dOKScnRzfddJM75wYvRfkcAICzXL7Nt3jxYqWnp+vKK6/U6dOn9dOf/lT79u1T79699ac//ckTc4SX2dnsFh/lcwBAoHM5TMXExGjHjh1avXq1duzYoVOnTmnq1KmaOHGiUyEd/oudzwEAOMvlMLVx40aNHDlSEydO1MSJEx3HGxsbtXHjRv3Xf/2XWycI71N2iPI5AAB2Lnemrr32Wp04ceKc49XV1br22mtdnsCyZcs0YMAAhYaGKiUlRZs3b77g+DVr1mjgwIEKDQ1VQkKC3nnnHafHDcNQdna2+vTpo7CwMKWlpWnfvn3nvM7//d//KSUlRWFhYerRo4cyMjJcnnsgqmts0m5rjSRpcAxhCgAAl8OUYRgt9mSOHz+uLl26uPRaq1evVlZWlnJycrRt2zYNGTJE6enpOnLkSIvjN23apAkTJmjq1KkqKSlRRkaGMjIyVFZW5hizaNEiLV26VHl5eSoqKlKXLl2Unp6u06dPO8b89a9/1Z133qkpU6Zox44d+sc//qGf/vSnLs09UO21nlJDk6GIsE6K6cFtXQAALIZhGK0ZePvtt0uS3nzzTd1www0KCQlxPNbU1KSdO3fqe9/7nvLz81v9w1NSUjRs2DA9//zzks5suxAbG6v77rtPc+fOPWd8ZmamamtrtW7dOsexESNGKDExUXl5eTIMQ9HR0Zo9e7YefPBBSWeumEVGRmrFihUaP368GhsbNWDAAD366KOaOnVqq+f6n2pqahQREaHq6mqFh4df9Ov4mpVF5Zr/RqlGXd5bf/xZitnTAQDAJZ74/d3qK1MRERGKiIiQYRjq1q2b4/uIiAhFRUVp+vTp+uMf/9jqH1xfX6/i4mKlpaWdnUxQkNLS0lRYWNjicwoLC53GS1J6erpj/P79+2W1Wp3GREREKCUlxTFm27ZtOnTokIKCgnT11VerT58+uvHGG52ubrWkrq5ONTU1Tl+ByLHzObf4AACQ5EIB/ZVXXpF0ZqfzBx980OVbev/p2LFjampqUmRkpNPxyMhI7d69u8XnWK3WFsdbrVbH4/Zj5xvzxRdfSJIeeeQRPfPMMxowYIAWL16sH/zgB9q7d6969uzZ4s/Ozc3Vo48+6uJZ+h/K5wAAOHO5M5WTk9PmIGUmm80mSXr44Yc1btw4JSUl6ZVXXpHFYtGaNWvO+7x58+apurra8XXgwIH2mrLXaF4+J0wBAHCGy1sjSNJf/vIX/fnPf1Z5ebnq6+udHtu2bVurXqN3797q0KGDKisrnY5XVlYqKiqqxedERUVdcLz9z8rKSvXp08dpTGJioiQ5jl955ZWOx0NCQnTppZeqvLz8vPMNCQlx6okFIsrnAACcy+UrU0uXLtWUKVMUGRmpkpISDR8+XL169dIXX3yhG2+8sdWvExwcrKSkJBUUFDiO2Ww2FRQUKDU1tcXnpKamOo2XpPXr1zvGx8XFKSoqymlMTU2NioqKHGOSkpIUEhKiPXv2OMY0NDToyy+/VP/+/Vs9/0Bk70sNjmHncwAA7Fy+MvXCCy9o+fLlmjBhglasWKE5c+bo0ksvVXZ2dov7T11IVlaWJk+erOTkZA0fPlxLlixRbW2tpkyZIkmaNGmS+vbtq9zcXEnSzJkzNXr0aC1evFhjx47VqlWrtHXrVi1fvlzSmc8JnDVrlh5//HHFx8crLi5OCxcuVHR0tGMfqfDwcN1zzz3KyclRbGys+vfvr6eeekqSdMcdd7i6HAGl9FCVJHY+BwCgOZfDVHl5uUaOHClJCgsL08mTJyVJd955p0aMGOHY5qA1MjMzdfToUWVnZ8tqtSoxMVH5+fmOAnl5ebmCgs5ePBs5cqRWrlypBQsWaP78+YqPj9fatWs1aNAgx5g5c+aotrZW06dPV1VVlUaNGqX8/HyFhoY6xjz11FPq2LGj7rzzTn3zzTdKSUnRhg0b1KNHD1eXI6CUUj4HAOAcrd5nyu7SSy/VX//6V1199dVKTk7WtGnT9POf/1zvvfeexo8f7/LVKV8VaPtM1TU2aVDO39TQZOijOdcqtmdns6cEAIDLTN1nyu6HP/yh3nrrLUnSlClT9MADD+i6665TZmambrvtNrdMCt7HXj7v3pnyOQAAzbl8m2/58uWO7QVmzJihXr16adOmTbrlllv085//3O0ThHfY+e++VEJfyucAADTnUphqbGzUb37zG919992KiYmRJI0fP17jx4/3yOTgPeybdVI+BwDAmUu3+Tp27KhFixapsbHRU/OBl3Jsi0CYAgDAicudqTFjxujDDz/0xFzgpeoam7THeuZdm1yZAgDAmcudqRtvvFFz585VaWmpkpKSzvlomVtuucVtk4N32GM9SfkcAIDzcDlM/fKXv5QkPfPMM+c8ZrFY1NTU1PZZwas031+K8jkAAM5cDlP2d/IhcJSxWScAAOflcmcKgWfnQcIUAADnQ5jCBdU1NmlvJeVzAADOhzCFC6J8DgDAhRGmcEGUzwEAuDDCFC6olL4UAAAX5PK7+Wpqalo8brFYFBISouDg4DZPCt6jlHfyAQBwQS6Hqe7du1/wdk9MTIzuuusu5eTkKCiIC1++7HTD2fJ5QgxhCgCAlrgcplasWKGHH35Yd911l4YPHy5J2rx5s1599VUtWLBAR48e1dNPP62QkBDNnz/f7RNG+7GXz3t07qS+3SmfAwDQEpfD1KuvvqrFixfrJz/5iePYj370IyUkJOill15SQUGB+vXrpyeeeIIw5ePst/gGUT4HAOC8XL4Pt2nTJl199dXnHL/66qtVWFgoSRo1apTKy8vbPjuYip3PAQD4di6HqdjYWL388svnHH/55ZcVGxsrSTp+/Lh69OjR9tnBVPYrU4PpSwEAcF4u3+Z7+umndccdd+jdd9/VsGHDJElbt27V7t279Ze//EWStGXLFmVmZrp3pmhXpxuatMfKzucAAHwbl8PULbfcot27d+ull17S3r17JUk33nij1q5dqwEDBkiSfvGLX7h1kmh/e6wn1WijfA4AwLdxOUxJUlxcnH7729+6ey7wIpTPAQBonYsKU1VVVdq8ebOOHDkim83m9NikSZPcMjGYq4y+FAAAreJymHr77bc1ceJEnTp1SuHh4U5XLSwWC2HKT+zkY2QAAGgVl9/NN3v2bN199906deqUqqqq9K9//cvxdeLECU/MEe2s+c7nlM8BALgwl8PUoUOHdP/996tz586emA+8gL183rNLMOVzAAC+hcthKj09XVu3bvXEXOAldlI+BwCg1VzuTI0dO1YPPfSQdu3apYSEBHXq1Mnp8VtuucVtk4M5yhx9qXCTZwIAgPdzOUxNmzZNkvTYY4+d85jFYlFTU1PbZwVTlfIxMgAAtJrLYeo/t0KAf2lePk+I6W7uZAAA8AEud6bg33Y3K59HR4SaPR0AALxeq65MLV26VNOnT1doaKiWLl16wbH333+/WyYGc7DzOQAArmlVmHr22Wc1ceJEhYaG6tlnnz3vOIvFQpjycfby+WD6UgAAtEqrwtT+/ftb/Gf4n+ZXpgAAwLejMwUH5/I5YQoAgNZw+d18TU1NWrFihQoKClr8oOMNGza4bXJoX5TPAQBwncthaubMmVqxYoXGjh2rQYMGUVL2I833l+J/VwAAWsflMLVq1Sr9+c9/1k033eSJ+cBEpQerJLFZJwAArnC5MxUcHKzLL7/cE3OByUoP1UiifA4AgCtcDlOzZ8/W7373OxmG4Yn5wCSnG5q0j/I5AAAuc/k238cff6z3339f7777rq666qpzPuj49ddfd9vk0H7s5fNelM8BAHCJy2Gqe/fuuu222zwxF5jI3pdi53MAAFzjUphqbGzUtddeq+uvv15RUVGemhNM0PydfAAAoPVc6kx17NhR99xzj+rq6jw1H5jEXj6nLwUAgGtcLqAPHz5cJSUlnpgLTOK08zlXpgAAcInLnalf/vKXmj17tg4ePKikpCR16dLF6fHBgwe7bXJoH58drlHTv8vnfSifAwDgEpfD1Pjx4yVJ999/v+OYxWKRYRiyWCxqampy3+zQLsqafbgx5XMAAFzjcpjav3+/J+YBE9nL54PpSwEA4DKXw1T//v09MQ+YaOfBs1emAACAa1wOU3a7du1SeXm56uvrnY7fcsstbZ4U2s/phibtO3JKEuVzAAAuhsth6osvvtBtt92m0tJSR1dKkqNrQ2fKt1A+BwCgbVzeGmHmzJmKi4vTkSNH1LlzZ3366afauHGjkpOT9cEHH3hgivAke/k8IYbyOQAAF8PlK1OFhYXasGGDevfuraCgIAUFBWnUqFHKzc3V/fffzx5UPsbel+IWHwAAF8flK1NNTU3q1q2bJKl3796qqKiQdKaYvmfPHvfODh5XeojyOQAAbeHylalBgwZpx44diouLU0pKihYtWqTg4GAtX75cl156qSfmCA9pXj5nWwQAAC6Oy2FqwYIFqq2tlSQ99thjuvnmm/X9739fvXr10urVq90+QXjOrn+Xz3t3DVZUOOVzAAAuhsthKj093fHPl19+uXbv3q0TJ06oR48eFJh9DDufAwDQdi53puw+//xz/e1vf9M333yjnj17tmkSy5Yt04ABAxQaGqqUlBRt3rz5guPXrFmjgQMHKjQ0VAkJCXrnnXecHjcMQ9nZ2erTp4/CwsKUlpamffv2tfhadXV1SkxMlMVi0fbt29t0Hr6mlPI5AABt5nKYOn78uMaMGaPvfve7uummm3T48GFJ0tSpUzV79myXJ7B69WplZWUpJydH27Zt05AhQ5Senq4jR460OH7Tpk2aMGGCpk6dqpKSEmVkZCgjI0NlZWWOMYsWLdLSpUuVl5enoqIidenSRenp6Tp9+vQ5rzdnzhxFR0e7PG9/YC+fE6YAALh4LoepBx54QJ06dVJ5ebk6d+7sOJ6Zman8/HyXJ/DMM89o2rRpmjJliq688krl5eWpc+fO+p//+Z8Wx//ud7/TDTfcoIceekhXXHGFfv3rX2vo0KF6/vnnJZ25KrVkyRItWLBAt956qwYPHqw//OEPqqio0Nq1a51e691339V7772np59+2uV5+zqnnc8pnwMAcNFcDlPvvfeennzyScXExDgdj4+P11dffeXSa9XX16u4uFhpaWlnJxQUpLS0NBUWFrb4nMLCQqfx0pkel338/v37ZbVancZEREQoJSXF6TUrKys1bdo0/e///q9TKDyfuro61dTUOH35MsrnAAC4h8thqra2tsXwceLECYWEhLj0WseOHVNTU5MiIyOdjkdGRspqtbb4HKvVesHx9j8vNMYwDN1111265557lJyc3Kq55ubmKiIiwvEVGxvbqud5q7Jmt/gonwMAcPFcDlPf//739Yc//MHxvcVikc1m06JFi3Tttde6dXKe8txzz+nkyZOaN29eq58zb948VVdXO74OHDjgwRl6HuVzAADcw+WtERYtWqQxY8Zo69atqq+v15w5c/Tpp5/qxIkT+sc//uHSa/Xu3VsdOnRQZWWl0/HKykpFRUW1+JyoqKgLjrf/WVlZqT59+jiNSUxMlCRt2LBBhYWF51xJS05O1sSJE/Xqq6+e83NDQkJcvvLmzdj5HAAA93D5ytSgQYO0d+9ejRo1Srfeeqtqa2t1++23q6SkRJdddplLrxUcHKykpCQVFBQ4jtlsNhUUFCg1NbXF56SmpjqNl6T169c7xsfFxSkqKsppTE1NjYqKihxjli5dqh07dmj79u3avn27Y2uF1atX64knnnDpHHwR5XMAANzH5StT0plC98MPP+x07ODBg5o+fbqWL1/u0mtlZWVp8uTJSk5O1vDhw7VkyRLV1tZqypQpkqRJkyapb9++ys3NlSTNnDlTo0eP1uLFizV27FitWrVKW7dudfxci8WiWbNm6fHHH1d8fLzi4uK0cOFCRUdHKyMjQ5LUr18/pzl07dpVknTZZZedU6z3R2fL5yGUzwEAaKOLClMtOX78uF5++WWXw1RmZqaOHj2q7OxsWa1WJSYmKj8/31EgLy8vV1DQ2QtoI0eO1MqVK7VgwQLNnz9f8fHxWrt2rQYNGuQYM2fOHNXW1mr69OmqqqrSqFGjlJ+fr9BQgoPUvC8VTvkcAIA2shiGYbjjhXbs2KGhQ4eqqanJHS/n9WpqahQREaHq6mqFh4ebPR2XPLhmh/5SfFD3//ByZV3/PbOnAwBAu/HE7++L/jgZ+K4yyucAALgNYSrAfFN/tnw+OKa7uZMBAMAPtLozdfvtt1/w8aqqqrbOBe2gefk8Mtx/tnoAAMAsrQ5TEREXviUUERGhSZMmtXlC8KyzO59TPgcAwB1aHaZeeeUVT84D7cS+WWcCt/gAAHALOlMBho+RAQDAvQhTAeRM+fykJMIUAADuQpgKILsO18hmiPI5AABuRJgKIPby+eCYCMrnAAC4CWEqgOw8yGadAAC4G2EqgJzdFoEwBQCAuxCmAgTlcwAAPIMwFSDs5fPvdKN8DgCAOxGmAkTpwSpJZ65KUT4HAMB9CFMBovRQjSTK5wAAuBthKkA4tkUgTAEA4FaEqQDwdX3j2fJ5DGEKAAB3IkwFgM+cyuehZk8HAAC/QpgKAHy4MQAAnkOYCgD28jlhCgAA9yNMBYDSQ1WSCFMAAHgCYcrPfV3fqM+PnJJE+RwAAE8gTPk5e/n8EsrnAAB4BGHKz1E+BwDAswhTfm7nvzfrZOdzAAA8gzDl5+w7n3NlCgAAzyBM+THK5wAAeB5hyo/tqqB8DgCApxGm/Fgpt/gAAPA4wpQfK6V8DgCAxxGm/Ji9fD6YvhQAAB5DmPJTTuVzrkwBAOAxhCk/1bx8fgnlcwAAPIYw5adKucUHAEC7IEz5KfvHyFA+BwDAswhTfoptEQAAaB+EKT/0dX2j/nmU8jkAAO2BMOWH7OXzyHDK5wAAeBphyg/tPMgtPgAA2gthyg+VsfM5AADthjDlhyifAwDQfghTfqa2jvI5AADtiTDlZ3YdpnwOAEB7Ikz5mVLK5wAAtCvClJ8pc/Slups7EQAAAgRhys/stIepmHCTZwIAQGAgTPmR5uVztkUAAKB9EKb8yK7DNTLs5fNulM8BAGgPhCk/crZ83t3ciQAAEEAIU36EzToBAGh/hCk/Ukr5HACAdkeY8hOUzwEAMAdhyk/Yy+dR4aGUzwEAaEeEKT+x89/lc65KAQDQvghTfqKM8jkAAKYgTPkJe/l8cAxhCgCA9kSY8gOnKJ8DAGAarwhTy5Yt04ABAxQaGqqUlBRt3rz5guPXrFmjgQMHKjQ0VAkJCXrnnXecHjcMQ9nZ2erTp4/CwsKUlpamffv2OR7/8ssvNXXqVMXFxSksLEyXXXaZcnJyVF9f75Hz87RdFWfL59/pFmL2dAAACCimh6nVq1crKytLOTk52rZtm4YMGaL09HQdOXKkxfGbNm3ShAkTNHXqVJWUlCgjI0MZGRkqKytzjFm0aJGWLl2qvLw8FRUVqUuXLkpPT9fp06clSbt375bNZtNLL72kTz/9VM8++6zy8vI0f/78djlnd7Pf4uOqFAAA7c9iGIZh5gRSUlI0bNgwPf/885Ikm82m2NhY3XfffZo7d+454zMzM1VbW6t169Y5jo0YMUKJiYnKy8uTYRiKjo7W7Nmz9eCDD0qSqqurFRkZqRUrVmj8+PEtzuOpp57Siy++qC+++KLFx+vq6lRXV+f4vqamRrGxsaqurlZ4uLmbZD6werveKDmkrOu+q/vHxJs6FwAAvFlNTY0iIiLc+vvb1CtT9fX1Ki4uVlpamuNYUFCQ0tLSVFhY2OJzCgsLncZLUnp6umP8/v37ZbVancZEREQoJSXlvK8pnQlcPXv2PO/jubm5ioiIcHzFxsa26hzbw86DVZJ4Jx8AAGYwNUwdO3ZMTU1NioyMdDoeGRkpq9Xa4nOsVusFx9v/dOU1P//8cz333HP6+c9/ft65zps3T9XV1Y6vAwcOXPjk2smpukZ9caxWErf5AAAwQ0ezJ2C2Q4cO6YYbbtAdd9yhadOmnXdcSEiIQkK8r9xtL5/3iaB8DgCAGUy9MtW7d2916NBBlZWVTscrKysVFRXV4nOioqIuON7+Z2tes6KiQtdee61Gjhyp5cuXt+lczGK/xcdVKQAAzGFqmAoODlZSUpIKCgocx2w2mwoKCpSamtric1JTU53GS9L69esd4+Pi4hQVFeU0pqamRkVFRU6veejQIf3gBz9QUlKSXnnlFQUFmf7GxovCzucAAJjL9Nt8WVlZmjx5spKTkzV8+HAtWbJEtbW1mjJliiRp0qRJ6tu3r3JzcyVJM2fO1OjRo7V48WKNHTtWq1at0tatWx1XliwWi2bNmqXHH39c8fHxiouL08KFCxUdHa2MjAxJZ4NU//799fTTT+vo0aOO+Zzvipi3KiVMAQBgKtPDVGZmpo4ePars7GxZrVYlJiYqPz/fUSAvLy93umo0cuRIrVy5UgsWLND8+fMVHx+vtWvXatCgQY4xc+bMUW1traZPn66qqiqNGjVK+fn5Cg0NlXTmStbnn3+uzz//XDExMU7zMXmnCJdQPgcAwHym7zPlqzyxT4Wrir44rszln6hPRKgK540xZQ4AAPgSv9tnCm3DzucAAJiPMOXDKJ8DAGA+wpQPc5TPYwhTAACYhTDlo5qXz7kyBQCAeQhTPurTQ9WOnc97d2XncwAAzEKY8lHsLwUAgHcgTPkowhQAAN6BMOWjHNsiUD4HAMBUhCkfdKquUfspnwMA4BUIUz7IXj6PpnwOAIDpCFM+iJ3PAQDwHoQpH0T5HAAA70GY8kGUzwEA8B6EKR9z8nSDvjhK+RwAAG9BmPIxn1bUSKJ8DgCAtyBM+ZgyyucAAHgVwpSPsfelBtOXAgDAKxCmfEzpQa5MAQDgTQhTPuTk6QZ9wc7nAAB4FcKUD2lePu9F+RwAAK9AmPIh9vJ5An0pAAC8BmHKh+w8yM7nAAB4G8KUD2FbBAAAvA9hykdQPgcAwDsRpnxE2aEz5fO+3cMonwMA4EUIUz7i7C2+cJNnAgAAmiNM+Qj7zufc4gMAwLsQpnzE2W0Rups7EQAA4IQw5QNqKJ8DAOC1CFM+4NNm5fOeXYJNng0AAGiOMOUDKJ8DAOC9CFM+wF4+H0xfCgAAr0OY8gGl7HwOAIDXIkx5uZrTDdpP+RwAAK9FmPJylM8BAPBuhCkvV3qoShJXpQAA8FaEKS9X+u8rUwkxhCkAALwRYcrLlVE+BwDAqxGmvBjlcwAAvB9hyovZr0pRPgcAwHsRpryY48ONuSoFAIDXIkx5McrnAAB4P8KUFys9WCWJK1MAAHgzwpSXqjndoC+Pfy2JMAUAgDcjTHmp5uXzHpTPAQDwWoQpL2UPU4PpSwEA4NUIU15q50E26wQAwBcQprwU2yIAAOAbCFNeiPI5AAC+gzDlhexXpWJ6UD4HAMDbEaa8UOlBbvEBAOArCFNeqPQQ5XMAAHwFYcoLsS0CAAC+gzDlZaq/OVs+HxRNmAIAwNsRprzMp5TPAQDwKYQpL1PK/lIAAPgUwpSXcYQp+lIAAPgErwhTy5Yt04ABAxQaGqqUlBRt3rz5guPXrFmjgQMHKjQ0VAkJCXrnnXecHjcMQ9nZ2erTp4/CwsKUlpamffv2OY05ceKEJk6cqPDwcHXv3l1Tp07VqVOn3H5urjrd0KQOQRauTAEA4CNMD1OrV69WVlaWcnJytG3bNg0ZMkTp6ek6cuRIi+M3bdqkCRMmaOrUqSopKVFGRoYyMjJUVlbmGLNo0SItXbpUeXl5KioqUpcuXZSenq7Tp087xkycOFGffvqp1q9fr3Xr1mnjxo2aPn26x8/32/y/ycNU9ki6UuJ6mT0VAADQChbDMAwzJ5CSkqJhw4bp+eeflyTZbDbFxsbqvvvu09y5c88Zn5mZqdraWq1bt85xbMSIEUpMTFReXp4Mw1B0dLRmz56tBx98UJJUXV2tyMhIrVixQuPHj9dnn32mK6+8Ulu2bFFycrIkKT8/XzfddJMOHjyo6Ojoc35uXV2d6urqHN/X1NQoNjZW1dXVCg8Pd+uaAAAAz6ipqVFERIRbf3+bemWqvr5excXFSktLcxwLCgpSWlqaCgsLW3xOYWGh03hJSk9Pd4zfv3+/rFar05iIiAilpKQ4xhQWFqp79+6OICVJaWlpCgoKUlFRUYs/Nzc3VxEREY6v2NjYiztpAADgV0wNU8eOHVNTU5MiIyOdjkdGRspqtbb4HKvVesHx9j+/bcwll1zi9HjHjh3Vs2fP8/7cefPmqbq62vF14MCBVp4lAADwZx3NnoCvCAkJUUhIiNnTAAAAXsbUK1O9e/dWhw4dVFlZ6XS8srJSUVFRLT4nKirqguPtf37bmP8suDc2NurEiRPn/bkAAAAtMTVMBQcHKykpSQUFBY5jNptNBQUFSk1NbfE5qampTuMlaf369Y7xcXFxioqKchpTU1OjoqIix5jU1FRVVVWpuLjYMWbDhg2y2WxKSUlx2/kBAAD/Z/ptvqysLE2ePFnJyckaPny4lixZotraWk2ZMkWSNGnSJPXt21e5ubmSpJkzZ2r06NFavHixxo4dq1WrVmnr1q1avny5JMlisWjWrFl6/PHHFR8fr7i4OC1cuFDR0dHKyMiQJF1xxRW64YYbNG3aNOXl5amhoUH33nuvxo8f3+I7+QAAAM7H9DCVmZmpo0ePKjs7W1arVYmJicrPz3cUyMvLyxUUdPYC2siRI7Vy5UotWLBA8+fPV3x8vNauXatBgwY5xsyZM0e1tbWaPn26qqqqNGrUKOXn5ys0NNQx5rXXXtO9996rMWPGKCgoSOPGjdPSpUvb78QBAIBfMH2fKV/liX0qAACAZ/ndPlMAAAC+jjAFAADQBoQpAACANiBMAQAAtAFhCgAAoA1M3xrBV9nfBFlTU2PyTAAAQGvZf2+7czMDwtRFOnnypCQpNjbW5JkAAABXnTx5UhEREW55LfaZukg2m00VFRXq1q2bLBaL2163pqZGsbGxOnDgAPtXifVojrU4i7VwxnqcxVqcxVqc1XwtunXrppMnTyo6OtppU/C24MrURQoKClJMTIzHXj88PDzg/+VvjvU4i7U4i7VwxnqcxVqcxVqcZV8Ld12RsqOADgAA0AaEKQAAgDYgTHmZkJAQ5eTkKCQkxOypeAXW4yzW4izWwhnrcRZrcRZrcZan14ICOgAAQBtwZQoAAKANCFMAAABtQJgCAABoA8IUAABAGxCmvMyyZcs0YMAAhYaGKiUlRZs3bzZ7Sh6Xm5urYcOGqVu3brrkkkuUkZGhPXv2OI05ffq0ZsyYoV69eqlr164aN26cKisrTZpx+/ntb38ri8WiWbNmOY4F0locOnRI//3f/61evXopLCxMCQkJ2rp1q+NxwzCUnZ2tPn36KCwsTGlpadq3b5+JM/acpqYmLVy4UHFxcQoLC9Nll12mX//6106fL+av67Fx40b96Ec/UnR0tCwWi9auXev0eGvO+8SJE5o4caLCw8PVvXt3TZ06VadOnWrHs3CPC61FQ0ODfvWrXykhIUFdunRRdHS0Jk2apIqKCqfX8Je1kL79343m7rnnHlksFi1ZssTpuDvWgzDlRVavXq2srCzl5ORo27ZtGjJkiNLT03XkyBGzp+ZRH374oWbMmKFPPvlE69evV0NDg66//nrV1tY6xjzwwAN6++23tWbNGn344YeqqKjQ7bffbuKsPW/Lli166aWXNHjwYKfjgbIW//rXv3TNNdeoU6dOevfdd7Vr1y4tXrxYPXr0cIxZtGiRli5dqry8PBUVFalLly5KT0/X6dOnTZy5Zzz55JN68cUX9fzzz+uzzz7Tk08+qUWLFum5555zjPHX9aitrdWQIUO0bNmyFh9vzXlPnDhRn376qdavX69169Zp48aNmj59enudgttcaC2+/vprbdu2TQsXLtS2bdv0+uuva8+ePbrlllucxvnLWkjf/u+G3RtvvKFPPvlE0dHR5zzmlvUw4DWGDx9uzJgxw/F9U1OTER0dbeTm5po4q/Z35MgRQ5Lx4YcfGoZhGFVVVUanTp2MNWvWOMZ89tlnhiSjsLDQrGl61MmTJ434+Hhj/fr1xujRo42ZM2cahhFYa/GrX/3KGDVq1Hkft9lsRlRUlPHUU085jlVVVRkhISHGn/70p/aYYrsaO3ascffddzsdu/32242JEycahhE46yHJeOONNxzft+a8d+3aZUgytmzZ4hjz7rvvGhaLxTh06FC7zd3d/nMtWrJ582ZDkvHVV18ZhuG/a2EY51+PgwcPGn379jXKysqM/v37G88++6zjMXetB1emvER9fb2Ki4uVlpbmOBYUFKS0tDQVFhaaOLP2V11dLUnq2bOnJKm4uFgNDQ1OazNw4ED169fPb9dmxowZGjt2rNM5S4G1Fm+99ZaSk5N1xx136JJLLtHVV1+t3//+947H9+/fL6vV6rQWERERSklJ8bu1kKSRI0eqoKBAe/fulSTt2LFDH3/8sW688UZJgbcedq0578LCQnXv3l3JycmOMWlpaQoKClJRUVG7z7k9VVdXy2KxqHv37pICby1sNpvuvPNOPfTQQ7rqqqvOedxd68EHHXuJY8eOqampSZGRkU7HIyMjtXv3bpNm1f5sNptmzZqla665RoMGDZIkWa1WBQcHO/4ysIuMjJTVajVhlp61atUqbdu2TVu2bDnnsUBaiy+++EIvvviisrKyNH/+fG3ZskX333+/goODNXnyZMf5tvTfjL+thSTNnTtXNTU1GjhwoDp06KCmpiY98cQTmjhxoiQF3HrYtea8rVarLrnkEqfHO3bsqJ49e/r12pw+fVq/+tWvNGHCBMcHHQfaWjz55JPq2LGj7r///hYfd9d6EKbgVWbMmKGysjJ9/PHHZk/FFAcOHNDMmTO1fv16hYaGmj0dU9lsNiUnJ+s3v/mNJOnqq69WWVmZ8vLyNHnyZJNn1/7+/Oc/67XXXtPKlSt11VVXafv27Zo1a5aio6MDcj1wYQ0NDfrJT34iwzD04osvmj0dUxQXF+t3v/udtm3bJovF4tGfxW0+L9G7d2916NDhnHdlVVZWKioqyqRZta97771X69at0/vvv6+YmBjH8aioKNXX16uqqsppvD+uTXFxsY4cOaKhQ4eqY8eO6tixoz788EMtXbpUHTt2VGRkZMCsRZ8+fXTllVc6HbviiitUXl4uSY7zDZT/Zh566CHNnTtX48ePV0JCgu6880498MADys3NlRR462HXmvOOioo65408jY2NOnHihF+ujT1IffXVV1q/fr3jqpQUWGvx0Ucf6ciRI+rXr5/j79OvvvpKs2fP1oABAyS5bz0IU14iODhYSUlJKigocByz2WwqKChQamqqiTPzPMMwdO+99+qNN97Qhg0bFBcX5/R4UlKSOnXq5LQ2e/bsUXl5ud+tzZgxY1RaWqrt27c7vpKTkzVx4kTHPwfKWlxzzTXnbJGxd+9e9e/fX5IUFxenqKgop7WoqalRUVGR362FdOadWkFBzn9ld+jQQTabTVLgrYdda847NTVVVVVVKi4udozZsGGDbDabUlJS2n3OnmQPUvv27dPf//539erVy+nxQFqLO++8Uzt37nT6+zQ6OloPPfSQ/va3v0ly43pcfG8e7rZq1SojJCTEWLFihbFr1y5j+vTpRvfu3Q2r1Wr21DzqF7/4hREREWF88MEHxuHDhx1fX3/9tWPMPffcY/Tr18/YsGGDsXXrViM1NdVITU01cdbtp/m7+QwjcNZi8+bNRseOHY0nnnjC2Ldvn/Haa68ZnTt3Nv74xz86xvz2t781unfvbrz55pvGzp07jVtvvdWIi4szvvnmGxNn7hmTJ082+vbta6xbt87Yv3+/8frrrxu9e/c25syZ4xjjr+tx8uRJo6SkxCgpKTEkGc8884xRUlLieIdaa877hhtuMK6++mqjqKjI+Pjjj434+HhjwoQJZp3SRbvQWtTX1xu33HKLERMTY2zfvt3p79O6ujrHa/jLWhjGt/+78Z/+8918huGe9SBMeZnnnnvO6NevnxEcHGwMHz7c+OSTT8yeksdJavHrlVdecYz55ptvjF/+8pdGjx49jM6dOxu33XabcfjwYfMm3Y7+M0wF0lq8/fbbxqBBg4yQkBBj4MCBxvLly50et9lsxsKFC43IyEgjJCTEGDNmjLFnzx6TZutZNTU1xsyZM41+/foZoaGhxqWXXmo8/PDDTr8k/XU93n///Rb/jpg8ebJhGK077+PHjxsTJkwwunbtaoSHhxtTpkwxTp48acLZtM2F1mL//v3n/fv0/fffd7yGv6yFYXz7vxv/qaUw5Y71sBhGs+1zAQAA4BI6UwAAAG1AmAIAAGgDwhQAAEAbEKYAAADagDAFAADQBoQpAACANiBMAQAAtAFhCgAAoA0IUwAAAG1AmALg144ePapf/OIX6tevn0JCQhQVFaX09HT94x//kCRZLBatXbvW3EkC8GkdzZ4AAHjSuHHjVF9fr1dffVWXXnqpKisrVVBQoOPHj5s9NQB+gs/mA+C3qqqq1KNHD33wwQcaPXr0OY8PGDBAX331leP7/v3768svv5Qkvfnmm3r00Ue1a9cuRUdHa/LkyXr44YfVseOZ/w9qsVj0wgsv6K233tIHH3ygPn36aNGiRfrxj3/cLucGwHtwmw+A3+ratau6du2qtWvXqq6u7pzHt2zZIkl65ZVXdPjwYcf3H330kSZNmqSZM2dq165deumll7RixQo98cQTTs9fuHChxo0bpx07dmjixIkaP368PvvsM8+fGACvwpUpAH7tr3/9q6ZNm6ZvvvlGQ4cO1ejRozV+/HgNHjxY0pkrTG+88YYyMjIcz0lLS9OYMWM0b948x7E//vGPmjNnjioqKhzPu+eee/Tiiy86xowYMUJDhw7VCy+80D4nB8ArcGUKgF8bN26cKioq9NZbb+mGG27QBx98oKFDh2rFihXnfc6OHTv02GOPOa5sde3aVdOmTdPhw4f19ddfO8alpqY6PS81NZUrU0AAooAOwO+Fhobquuuu03XXXaeFCxfqZz/7mXJycnTXXXe1OP7UqVN69NFHdfvtt7f4WgDQHFemAAScK6+8UrW1tZKkTp06qampyenxoUOHas+ePbr88svP+QoKOvvX5ieffOL0vE8++URXXHGF508AgFfhyhQAv3X8+HHdcccduvvuuzV48GB169ZNW7du1aJFi3TrrbdKOvOOvoKCAl1zzTUKCQlRjx49lJ2drZtvvln9+vXTj3/8YwUFBWnHjh0qKyvT448/7nj9NWvWKDk5WaNGjdJrr72mzZs36+WXXzbrdAGYhAI6AL9VV1enRx55RO+9957++c9/qqGhQbGxsbrjjjs0f/58hYWF6e2331ZWVpa+/PJL9e3b17E1wt/+9jc99thjKikpUadOnTRw4ED97Gc/07Rp0ySdKaAvW7ZMa9eu1caNG9WnTx89+eST+slPfmLiGQMwA2EKAC5CS+8CBBCY6EwBAAC0AWEKAACgDSigA8BFoCEBwI4rUwAAAG1AmAIAAGgDwhQAAEAbEKYAAADagDAFAADQBoQpAACANiBMAQAAtAFhCgAAoA3+P8MbL070/c55AAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "\n", + "plt.ylabel(\"Learning rate\")\n", + "plt.xlabel(\"Step\")\n", + "plt.plot(range(total_training_steps), track_lrs);" + ] + }, + { + "cell_type": "markdown", + "id": "7b3996b6-3f7a-420a-8584-c5760249f3d8", + "metadata": {}, + "source": [ + "## D.2 Cosine decay" + ] + }, + { + "cell_type": "markdown", + "id": "c5216214-de79-40cf-a733-b1049a73023c", + "metadata": {}, + "source": [ + "- Another popular technique for training complex deep neural networks is cosine decay, which also adjusts the learning rate across training epochs\n", + "- In cosine decay, the learning rate follows a cosine curve, decreasing from its initial value to near zero following a half-cosine cycle\n", + "- This gradual reduction is designed to slow the pace of learning as the model begins to improve its weights; it reduces the risk of overshooting minima as the training progresses, which is crucial for stabilizing the training in its later stages\n", + "- Cosine decay is often preferred over linear decay for its smoother transition in learning rate adjustments, but linear decay is also used in practice (for example, [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838))" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "4e8d2068-a057-4abf-b478-f02cc37191f6", + "metadata": {}, + "outputs": [], + "source": [ + "import math\n", + "\n", + "min_lr = 0.1 * initial_lr\n", + "track_lrs = []\n", + "\n", + "lr_increment = (peak_lr - initial_lr) / warmup_steps\n", + "global_step = -1\n", + "\n", + "for epoch in range(n_epochs):\n", + " for input_batch, target_batch in train_loader:\n", + " optimizer.zero_grad()\n", + " global_step += 1\n", + " \n", + " # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n", + " if global_step < warmup_steps:\n", + " # Linear warmup\n", + " lr = initial_lr + global_step * lr_increment \n", + " else:\n", + " # Cosine annealing after warmup\n", + " progress = ((global_step - warmup_steps) / \n", + " (total_training_steps - warmup_steps))\n", + " lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n", + " \n", + " # Apply the calculated learning rate to the optimizer\n", + " for param_group in optimizer.param_groups:\n", + " param_group[\"lr\"] = lr\n", + " track_lrs.append(optimizer.param_groups[0][\"lr\"])\n", + " \n", + " # Calculate loss and update weights" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "0e779e33-8a44-4984-bb23-be0603dc4158", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAGwCAYAAACNeeBZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABiTUlEQVR4nO3deVhUZf8G8HsWZoZ12GRHwaVcAFFQxDRbKDJbSDM1SzPTMtewLM20fO1nWpaZmlqWVppmb5mZWYaVG6IIIoi4JIqCwyoM+zJzfn8go7yigSxnlvtzXXP5cuaZ4TvnNbh9zvc8j0QQBAFEREREdFukYhdAREREZMoYpoiIiIiagWGKiIiIqBkYpoiIiIiagWGKiIiIqBkYpoiIiIiagWGKiIiIqBnkYhdgqvR6PbKysmBvbw+JRCJ2OURERNQIgiCguLgYXl5ekEpbZk6JYeo2ZWVlwdfXV+wyiIiI6DZcvHgRPj4+LfJeDFO3yd7eHkDt/xkODg4iV0NERESNodVq4evra/g93hIYpm5T3aU9BwcHhikiIiIT05ItOmxAJyIiImoGhikiIiKiZmCYIiIiImoGhikiIiKiZmCYIiIiImoGhikiIiKiZmCYIiIiImoGhikiIiKiZmCYIiIiImoGhikiIiKiZhA9TK1cuRJ+fn5QqVQICwvD4cOHbzl+69at6Nq1K1QqFQIDA7Fz5856z//www948MEH4eLiAolEgmPHjt3wHhUVFZg8eTJcXFxgZ2eHYcOGITs7uyU/FhEREVkIUcPUli1bEB0djfnz5yMhIQE9e/ZEZGQkcnJyGhx/8OBBjBo1CuPHj0diYiKioqIQFRWFlJQUw5jS0lIMGDAAixcvvun3feWVV/Dzzz9j69at+Pvvv5GVlYWhQ4e2+OcjIiIi8ycRBEEQ65uHhYWhT58+WLFiBQBAr9fD19cXU6dOxRtvvHHD+BEjRqC0tBQ7duwwHOvXrx+Cg4OxevXqemPPnz8Pf39/JCYmIjg42HC8qKgI7dq1w6ZNm/Dkk08CANLS0tCtWzfExsaiX79+japdq9VCrVajqKiIGx23oqKyapRV10AulcJKJoHKSgalXNqiG1QSEZHlaI3f3/IWeZfbUFVVhaNHj2L27NmGY1KpFBEREYiNjW3wNbGxsYiOjq53LDIyEtu2bWv09z169Ciqq6sRERFhONa1a1e0b9/+lmGqsrISlZWVhq+1Wm2jvyfdnoP/5OGZz+Og/5+4byWTwE4ph4O1FVztlGhnp0Q7eyW8HK3h62yN9s426OBiC7W1lTiFExGRRREtTOXl5UGn08Hd3b3ecXd3d6SlpTX4Go1G0+B4jUbT6O+r0WigUCjg6OjYpPdZtGgR3nnnnUZ/H2q+P1JzoBcAiQS4fv60WifgSlk1rpRV40J+2U1f72avxB3u9rjD3R4B3g4I8lHD39UOMilntYiIqOWIFqZMzezZs+vNimm1Wvj6+opYkflLziwEAHzwZE8M7e2NGr2AimodSiprUFxRg6LyauQVVyK3pBLZ2gpkXinHxSvluFhQhpziSsNj/9k8w3vaKmQIbu+Ivn4u6OPvhN7tnaCykon0CYmIyByIFqZcXV0hk8luuIsuOzsbHh4eDb7Gw8OjSeNv9h5VVVUoLCysNzv1b++jVCqhVCob/X2oeXR6ASmZtZdSg3zUkEgksJJJYCWTwl5lBU/1rV9fXFGNszklOJNdgpMaLZIvFeFElhalVTocOJuPA2fzAQAKuRR9/Zxx9x2uuPuOdrjT3Z79WERE1CSihSmFQoGQkBDExMQgKioKQG0DekxMDKZMmdLga8LDwxETE4MZM2YYju3evRvh4eGN/r4hISGwsrJCTEwMhg0bBgA4deoUMjIymvQ+1LrO5ZagvFoHG4UMHdvZNfn19ior9GrvhF7tnQzHanR6nMkpQfyFKziSXoDD6QXQaCuw/2we9p/Nw//tTIOPkzUe7O6BB3u4o4+fMy8JEhHRvxL1Ml90dDTGjh2L0NBQ9O3bF8uWLUNpaSnGjRsHABgzZgy8vb2xaNEiAMD06dMxaNAgLF26FEOGDMHmzZsRHx+PtWvXGt6zoKAAGRkZyMrKAlAblIDaGSkPDw+o1WqMHz8e0dHRcHZ2hoODA6ZOnYrw8PBG38lHre/4pSIAQA8vhxYLNHKZFN08HdDN0wHP9usAQRBwLq8Uf5/Kxd4zuTh0Lh+XrpTjiwPp+OJAOlztFHgkyAuP9vRC7/aOnLEiIqIGiRqmRowYgdzcXMybNw8ajQbBwcHYtWuXock8IyMDUum1pbD69++PTZs2Ye7cuZgzZw66dOmCbdu2ISAgwDBm+/bthjAGACNHjgQAzJ8/H2+//TYA4KOPPoJUKsWwYcNQWVmJyMhIrFq1qg0+MTVWcmZtmAr0dmy17yGRSNCpnR06tbPD8wP8UV6lw94zufj9RDZi0rKRV1KF9QfPY/3B8/B1tsaTvX3xZKgPvB2tW60mIiIyPaKuM2XKuM5U6xq66gASMgqxbEQwonp5t/n3r9bpsf9MHrYnZeG3ExqUVekA1N5ZOLBLOzwT1h73d3PnZUAiIhNjVutMEd1MjU6P1Mu1zeeBPv/Sad5KrGRS3NvVDfd2dUN5lQ67TlzGd0cuIfZcPvaezsXe07no4GKD5/r7YXioL+yU/E+JiMhScWbqNnFmqvWkabR4aNk+2CnlOD7/QUiNaPbnQn4pvj18Ed8ezkBReTUAwF4px1N9fPFcfz/4OtuIXCEREd1Ka/z+Fn2jY6L/dX3zuTEFKQDo4GKLNwZ3Rezs+7AwKgAd29miuLIG6/anY9D7f+Klr48i5Wq/FxERWQZemyCjk3w1TAWJdImvMWwUcjzTrwOe7tsef5/JxRf707HvTB52ndBg1wkNInu445UH7kBXD85aEhGZO4YpMjqGO/l8HMUtpBGkUgnuvdMN997phtPZxVj151n8lJSF305k47cT2RgS5IlXIrqgs5u92KUSEVEr4WU+MirV1zefexvvzFRD7nC3x7KRvfD7jLsxJMgTAPDL8ct44KO9mLE5EedyS0SukIiIWgPDFBmV09nFqKrRw14lRwcTbebu4m6PlU/3xq/TByKyhzsEAdh2LAsPfrQX7/x8AkVl1WKXSERELYhhioxKimGxTrXRNZ83VTdPB6x5NhQ7pg7AfV3dUKMX8OWB87jngz/xzaEL0Ol5Iy0RkTlgmCKjUncnn1jrS7WGAG81vniuD74e3xdd3Oxwpawac7elYMjyfTj4T57Y5RERUTMxTJFRSb5uZsrcDOzSDr9OH4h3HusBtbUV0jTFePqzOLz09VFcLCgTuzwiIrpNDFNkNKpq9Ei7XAwACGrFPfnEJJdJMba/H/569R6MCe8AqQTYdUKDBz76G5/vO8dLf0REJohhiozG6exiVOn0UFtbwdfZvDcTdrJVYMHjAfh1+t0I83dGRbUeC385iaGrDuDk1bsZiYjINDBMkdEw9Et5qyGRmHbzeWPd6WGPzRP7YdHQQNir5Ei6VIRHP9mPpb+fQmWNTuzyiIioERimyGgkZxYCMK/m88aQSCQY1bc9/ogehAe7u6NGL+CTPWfx8Mf7EH++QOzyiIjoXzBMkdGom5kKMsPm88Zwd1BhzbMh+HR0b7jaKfFPbimGr4nFgp9TUVHNWSoiImPFMEVGoaJah9PZtc3nljYzdT2JRILBgZ6IiR6Ep0J9IAjAFwfSEbXyAE5pisUuj4iIGsAwRUbhlKYY1ToBTjZW8HY07+bzxlDbWGHJkz2xbmwoXGwVSNMU49EV+/HlgXQIAu/4IyIyJgxTZBSOX7e5saU0nzfG/d3csWvG3bj3znaoqtHjnZ9TMfbLI8jRVohdGhERXcUwRUYh+VIhAMvtl7qVdvZKfPFcHyx4vAeUcin2ns7FQx/vw+8nNGKXRkREYJgiI5GcWbu2kiX3S92KRCLBmHA/7Jg6AN08HVBQWoWJXx/Fgp9TUa3Ti10eEZFFY5gi0dVrPufM1C11cbfHtsn9MWGgP4Da5vSRaw9BU8TLfkREYmGYItGlXtZCpxfgaqeAp1oldjlGTymX4c0h3bH22RDYq+Q4euFK7abJZ7lpMhGRGBimSHQpmZa38nlLeLCHh+GyX35pFZ5ZF4eVf56Fnvv7ERG1KYYpEp1hGxkfR3ELMUEdXGzx48v9MTzEB3oBeP+3U5jwVTyKyqrFLo2IyGIwTJHokq/bk4+aTmUlw/vDe2LxsEAo5FLEpOXg0RX7DX1oRETUuhimSFRlVTU4k1P7Sz+Id/I1y4g+7fHDpP7wdbZGRkEZhq46iD1p2WKXRURk9himSFQnL2uhFwA3eyXcHdh83lwB3mr8NHkAwvydUVJZg/Eb4vH5vnNcNZ2IqBUxTJGojvMSX4tztlXg6/FhGNXXF4IALPzlJF7/73FU1XA9KiKi1sAwRaIy9EvxEl+LUsil+L8nAjH/0e6QSoDv4i/hmc/jkF9SKXZpRERmh2GKRFW3Jx/7pVqeRCLBuLv88eW4vrBXyXH4fAEeX3kApzRsTCciakkMUySa0soa/JNbAqC214dax6A72uHHl++Cn4sNLl0px5OfHsTBf7jAJxFRS2GYItGcyNJCEAAPBxXc7Nl83po6u9lh2+S70NffGcWVNXjuiyPYnpQldllERGaBYYpEc/xSIQD2S7UVRxsFvnq+L4YEeqJKp8e0bxPx+b5zYpdFRGTyGKZINHXbyATxEl+bUVnJ8MmoXhh3lx+A2jv9Fvycyi1oiIiagWGKRFPXfM6ZqbYllUow75HuePPhbgCALw6kY+rmRFRU60SujIjINDFMkSiKK6pxLrcUANeYEoNEIsGEuzvi45HBsJJJ8MvxyxjzxWEUlXNPPyKipmKYIlGkZGoBAN6O1nCxU4pcjeV6PNgbG8b1hb1SjsPpBRi59hDyuBYVEVGTMEyRKOr6pTgrJb7+nV2x5cVwuNopcfKyFk+tiUVWYbnYZRERmQyGKRIF+6WMS3cvB3z3Yj94qVU4l1uK4atjcT6vVOyyiIhMAsMUiSK5blkEzkwZjY7t7LB1Un/4u9ois7Acw9fEcrV0IqJGYJiiNldUXo3z+WUAGKaMjbejNb57MRxdPeyRW1yJEWtjkXSxUOyyiIiMGsMUtbkTVy/x+Tpbw8lWIXI19L/a2SuxZWI4gn0dUVhWjdGfxyHuXL7YZRERGS2GKWpzx9l8bvTUNlb45oUwhHd0QUllDcZ8cRj7zuSKXRYRkVFimKI2l3ypLkw5ilsI3ZKdUo4vx/XB/V3dUFmjxwsb4rH3NAMVEdH/YpiiNnc8sxAAEMQ7+YyeykqGT58JQUS3q4HqKwYqIqL/xTBFbaqwrAoXC2rXMArwYpgyBQq5FKtGhyCimzuqrgaqvxmoiIgMGKaoTSVf7Zfq4GIDtY2VyNVQY9UGqt54oHttoJrwVTz+OpUjdllEREaBYYra1PFLbD43VQq5FCuf7o0HrwaqiV8fZaAiIgLDFLWxum1k2C9lmhRyKVZcH6i+Ooo/GaiIyMIxTFGbOs47+UxeXaCK7OGOKp0eL359FAfO5oldFhGRaBimqM3kl1Qi8+oGuj28HUSuhpqjLlDV9VC9sCEe8ecLxC6LiEgUDFPUZuqazzu62sJBxeZzU2clk2LF070wsIsryqt1GPflERy/uuciEZElYZiiNlPXLxXIfimzoZTLsPbZUPT1d0bx1ZXSuTkyEVkahilqM7yTzzxZK2T44rk+6HndXn7nckvELouIqM0wTFGbSeaefGbLTinHhnF90NXDHnkllRj9eRwuFpSJXRYRUZtgmKI2kVtcictFFZBIgB4MU2bJ0UaBb14IQ6d2trhcVIHRn8dBU1QhdllERK2OYYraRF2/VKd2drBTykWuhlqLq50SG1/oB19na2QUlOHZdXEoLKsSuywiolYlephauXIl/Pz8oFKpEBYWhsOHD99y/NatW9G1a1eoVCoEBgZi586d9Z4XBAHz5s2Dp6cnrK2tERERgTNnztQbc/r0aTz++ONwdXWFg4MDBgwYgD///LPFPxtdw34py+GhVmHTC/3g7qDEmZwSPL/+CMqqasQui4io1YgaprZs2YLo6GjMnz8fCQkJ6NmzJyIjI5GT0/CKygcPHsSoUaMwfvx4JCYmIioqClFRUUhJSTGMWbJkCZYvX47Vq1cjLi4Otra2iIyMREXFtcsNjzzyCGpqarBnzx4cPXoUPXv2xCOPPAKNRtPqn9lSJWcWAmCYshS+zjb46vkwOKjkSMgoxMsbE1Ct04tdFhFRq5AIgiCI9c3DwsLQp08frFixAgCg1+vh6+uLqVOn4o033rhh/IgRI1BaWoodO3YYjvXr1w/BwcFYvXo1BEGAl5cXZs6ciVdffRUAUFRUBHd3d6xfvx4jR45EXl4e2rVrh71792LgwIEAgOLiYjg4OGD37t2IiIhosNbKykpUVlYavtZqtfD19UVRUREcHLgA5b8J+78/kK2txPcvhSPUz1nscqiNxJ8vwDPr4lBRrccTvbyxdHhPSKUSscsiIgum1WqhVqtb9Pe3aDNTVVVVOHr0aL3wIpVKERERgdjY2AZfExsbe0PYiYyMNIxPT0+HRqOpN0atViMsLMwwxsXFBXfeeSe++uorlJaWoqamBmvWrIGbmxtCQkJuWu+iRYugVqsND19f39v+7JYmW1uBbG0lpBKguxeDpyUJ9XPGqtG9IZNK8GNiJhb+chIi/vuNiKhViBam8vLyoNPp4O7uXu+4u7v7TS+3aTSaW46v+/NWYyQSCf744w8kJibC3t4eKpUKH374IXbt2gUnJ6eb1jt79mwUFRUZHhcvXmzaB7ZgyVf7pTq72cFGweZzS3NfV3e8/2QQAOCLA+lY9dc/IldERNSyRG9Ab2uCIGDy5Mlwc3PDvn37cPjwYURFReHRRx/F5cuXb/o6pVIJBweHeg9qnOOZ3NzY0g3t7YO5Q7oBAN7/7RQ2H84QuSIiopYjWphydXWFTCZDdnZ2vePZ2dnw8PBo8DUeHh63HF/3563G7NmzBzt27MDmzZtx1113oXfv3li1ahWsra2xYcOGFvlsVF/dsghB3EbGor0wsCMm3dMJADDnx2T8doI3fBCReRAtTCkUCoSEhCAmJsZwTK/XIyYmBuHh4Q2+Jjw8vN54ANi9e7dhvL+/Pzw8POqN0Wq1iIuLM4wpK6tdlVkqrf/RpVIp9HrebdTSBEG4tiwCw5TFmxV5J0aE+kIvANO+TURCxhWxSyIiajZRL/NFR0fjs88+w4YNG3Dy5ElMmjQJpaWlGDduHABgzJgxmD17tmH89OnTsWvXLixduhRpaWl4++23ER8fjylTpgCo7YeaMWMGFi5ciO3btyM5ORljxoyBl5cXoqKiANQGMicnJ4wdOxZJSUk4ffo0XnvtNaSnp2PIkCFtfg7MnUZbgbySSsikEnT35KVRSyeRSPDuEwG49852qKzR44UN8UjPKxW7LCKiZhE1TI0YMQIffPAB5s2bh+DgYBw7dgy7du0yNJBnZGTU62Pq378/Nm3ahLVr16Jnz574/vvvsW3bNgQEBBjGzJo1C1OnTsXEiRPRp08flJSUYNeuXVCpVABqLy/u2rULJSUluO+++xAaGor9+/fjp59+Qs+ePdv2BFiAulmpLm52UFnJRK6GjIFcJsWKp3sj0FuNgtIqPPflYeSVVP77C4mIjJSo60yZstZYp8IcLf39FD7ZcxZPhfpgyZMMq3RNbnElhn56ABcLytHT1xGbJ/SDtYKBm4hal1mtM0WW4Vq/lKO4hZDRaWevxPpxfeFoY4Wki4WY+m0idHr+246ITA/DFLUaQRCQnMk9+ejmOrWzw+djQqGQS/HHyWy8vf0EF/UkIpPDMEWtJquoAgWlVZBLJejqYS92OWSkQv2c8fGIYEgkwNeHLmDN3nNil0RE1CQMU9Rqki8VAgDu9LBn8znd0uBAT7w1pDsA4L1f07DjeJbIFRERNR7DFLWaun4pLtZJjfH8AH88f5c/ACD6uySuQUVEJoNhilpNXb9UAPulqJHeHNINEd3cUFWjx8Sv4nGxoEzskoiI/hXDFLWK65vPg7gnHzWSTCrBxyN7obunA/JKqjB+wxFoK6rFLouI6JYYpqhVXLpSjsKyaihkUtzhYSd2OWRCbJVyrHsuFG72SpzOLsGUTYmo0XGrJyIyXgxT1Crq+qXu9LCHUs7mc2oaT7U11o3tA2srGfaezsXbP3PJBCIyXgxT1CqOZxYC4ObGdPsCfdRYNrJ2yYRvDmXgywPnxS6JiKhBDFPUKlIM/VIMU3T7Int4YM7gbgCA//ySij9Ss0WuiIjoRgxT1OIEQbhuGxmGKWqeFwb6Y1Tf9hAEYPrmRJzSFItdEhFRPQxT1OIu5JehuKIGCrkUd7hz5XNqHolEggWP90D/Ti4ordLhha+O4EppldhlEREZMExRizt+9RJfN08HWMn4V4yaz0omxcqne6O9sw0uFpTj5Y0JqOYdfkRkJPibjloc+6WoNTjZKvD52FDYKmSIPZeP/+xIFbskIiIADFPUCo5f3ZOP/VLU0u5wt8fHI3tBIgG+ir2AjXEXxC6JiIhhilqWXi8gJVMLAAjkzBS1goju7nj1wTsBAPN/OoG4c/kiV0RElo5hilrU+fxSlFTWQCmXoosbVz6n1vHyPZ3waE8v1OgFTNqYwD38iEhUDFPUour24+vh5QA5m8+plUgkEiwZFoRAbzUKSqsw4at4lFbWiF0WEVko/rajFlW3vlSQj6O4hZDZs1bIsHZMCFztlEjTFCP6u2PQ67nlDBG1PYYpalHJV8NUAPulqA14qq2x5tkQKGRS/HYiGx/HnBG7JCKyQAxT1GJ0egEnsupmphimqG2EdHDCu08EAAA+jjmDncmXRa6IiCwNwxS1mPS8EpRW6WBtJUOndmw+p7YzPNQX4wf4AwBmfpdkCPVERG2BYYpaTF2/VA8vB8ikEpGrIUsze3BXDOziivJqHSZ+dRT5JZVil0REFoJhiloMNzcmMcllUqwY1Rv+rrbILCzHtM2JqOGWM0TUBhimqMUYtpFhmCKRqG2ssPqZENgoZDhwNh/v/35K7JKIyAIwTFGLqNHpcSKrbuVzR3GLIYt2p4c9ljwZBABY8/c5/HKcDelE1LoYpqhF/JNbivJqHWwVMnR0tRW7HLJwjwR54cW7OwIAXvs+Caezi0WuiIjMGcMUtQjDyufeakjZfE5G4LXIO9G/kwvKqnR48euj0FZUi10SEZkphilqEcmXCgEAQVysk4yEXCbFJ6N6wdvRGul5pYjewhXSiah1MExRizieyTv5yPi42Cnx6TO9oZBL8cfJHKz486zYJRGRGWKYomar0emRamg+Z5gi4xLk44iFUbUrpH/0x2n8mZYjckVEZG4YpqjZzuSUoLJGD3ulHH4ubD4n4/NUqC9Gh7WHIADTNyfifF6p2CURkRlhmKJmu35zYzafk7Ga/2gP9GrvCG1FDV765ijKqmrELomIzATDFDXb8cxCAFysk4ybQi7Fp6ND4GqnRJqmGG/8NxmCwIZ0Imo+hilqtutnpoiMmYdahVWje0MulWB7UhbW7U8XuyQiMgMMU9QsVTV6nNTULojImSkyBX39nTF3SDcAwKJf0xB3Ll/kiojI1DFMUbOczi5GVY0eDio52jvbiF0OUaOM7e+HqGAv6PQCpnybiJziCrFLIiITxjBFzZJ83fpSEgmbz8k0SCQS/N/QQHRxs0NucSWmfZuIGp1e7LKIyEQxTFGzHL/aL8XNjcnU2Cjk+PSZENgoZDh0rgAf7j4tdklEZKIYpqhZUq7OTLFfikxRZzc7LB4WBABY9dc/+CM1W+SKiMgUMUzRbaus0SFNw5XPybQ92tMLz/X3AwBEf3cMFwvKxC2IiEwOwxTdtlOaYlTrBDjaWMHHyVrscohu25yHuyHYt3ZBz0kbj6KiWid2SURkQhim6LYZms+92XxOpk0hl2Ll6N5wsrFCSqYWC3akil0SEZkQhim6bXWLdbJfisyBt6M1lo3sBYkE2BSXgR8SLoldEhGZCIYpum28k4/MzaA72mHafV0AAHN+TDb0BBIR3QrDFN2WimodTmfXrnweyJkpMiPT7u+CgV1cUVGtx8vfJKC4olrskojIyDFM0W1J0xSjRi/AxVYBL7VK7HKIWoxMKsGyEcHwVKtwLq+UGyIT0b+6rTBVU1ODP/74A2vWrEFxce3sRFZWFkpKSlq0ODJeyZcKAXDlczJPLnZKrHi6dkPkX5Iv48sD58UuiYiMWJPD1IULFxAYGIjHH38ckydPRm5uLgBg8eLFePXVV1u8QDJOdf1SQVxfisxUSAcnvHl1Q+T/23kSCRlXRK6IiIxVk8PU9OnTERoaiitXrsDa+traQk888QRiYmJatDgyXnXLIgQwTJEZe66/H4YEeqJGL2DqpkQUllWJXRIRGaEmh6l9+/Zh7ty5UCgU9Y77+fkhMzOzxQoj41VepcOZnNpLukE+juIWQ9SKJBIJFg0LRAcXG2QWluPVrcfZP0VEN2hymNLr9dDpblwd+NKlS7C3t2+Rosi4pV7WQqcX0M5eCXcHpdjlELUqB5UVVj7dGwqZFH+czMa6/elil0RERqbJYerBBx/EsmXLDF9LJBKUlJRg/vz5ePjhh1uyNjJShuZzrnxOFiLAW423Hqntn3rv1zQksn+KiK7T5DC1dOlSHDhwAN27d0dFRQWefvppwyW+xYsXt0aNZGSSM7m5MVmeZ/p1wMOBHqjRC5iyKRFFZVx/iohqyZv6Ah8fHyQlJWHLli1ISkpCSUkJxo8fj9GjR9drSCfzlZxZCIDbyJBlkUgkeG9YEFIytcgoKMOr3ydh7bMhnJ0loqbPTO3duxcAMHr0aCxZsgSrVq3CCy+8ACsrK8NzTbFy5Ur4+flBpVIhLCwMhw8fvuX4rVu3omvXrlCpVAgMDMTOnTvrPS8IAubNmwdPT09YW1sjIiICZ86cueF9fvnlF4SFhcHa2hpOTk6Iiopqcu2WqLSyBmevNp9zZooszfX9U7tTs/EF158iItxGmLr33ntRUFBww/GioiLce++9TXqvLVu2IDo6GvPnz0dCQgJ69uyJyMhI5OTkNDj+4MGDGDVqFMaPH4/ExERERUUhKioKKSkphjFLlizB8uXLsXr1asTFxcHW1haRkZGoqKgwjPnvf/+LZ599FuPGjUNSUhIOHDiAp59+ukm1W6rUy1roBcDdQQk3B658TpYn0EeNuYb+qZM4drFQ3IKISHQSoYn3+UqlUmRnZ6Ndu3b1jp8+fRqhoaHQahu/MWhYWBj69OmDFStWAKi9U9DX1xdTp07FG2+8ccP4ESNGoLS0FDt27DAc69evH4KDg7F69WoIggAvLy/MnDnTsIBoUVER3N3dsX79eowcORI1NTXw8/PDO++8g/Hjxze61srKSlRWVhq+1mq18PX1RVFRERwcHBr9Pqbui/3pWLAjFRHd3PH52FCxyyEShSAImLwpATuTNfBxssYvUwdCbWMldllE1AharRZqtbpFf383emZq6NChGDp0KCQSCZ577jnD10OHDsXjjz+OyMhI9O/fv9HfuKqqCkePHkVERMS1YqRSREREIDY2tsHXxMbG1hsPAJGRkYbx6enp0Gg09cao1WqEhYUZxiQkJCAzMxNSqRS9evWCp6cnBg8eXG92qyGLFi2CWq02PHx9fRv9Wc1J3WKd7JciS1bXP9Xe2QaXrpTjte+TuP4UkQVrdJiqCxGCIMDe3r5esPDw8MDEiRPxzTffNPob5+XlQafTwd3dvd5xd3d3aDSaBl+j0WhuOb7uz1uNOXfuHADg7bffxty5c7Fjxw44OTnhnnvuafDyZZ3Zs2ejqKjI8Lh48WKjP6s5OX7dnnxEluz6/qnfU7O5fx+RBWv03XxffvklgNqVzl999VXY2tq2WlGtSa/XAwDefPNNDBs2DEDtZ/Px8cHWrVvx4osvNvg6pVIJpdKyF6gsqazBubxSAGw+JwJq/1Hx5pBumL/9BBb9ehIhHZzQ09dR7LKIqI01uQF9/vz5LRKkXF1dIZPJkJ2dXe94dnY2PDw8GnyNh4fHLcfX/XmrMZ6engCA7t27G55XKpXo2LEjMjIymvGJzN+JzCIIAuClVsHVzrKDJVGdMeEdMDjAA9W62j6qonKuP0VkaZocpgDg+++/x1NPPYV+/fqhd+/e9R6NpVAoEBISUm9zZL1ej5iYGISHhzf4mvDw8Bs2U969e7dhvL+/Pzw8POqN0Wq1iIuLM4wJCQmBUqnEqVOnDGOqq6tx/vx5dOjQodH1W6K6file4iO6RiKRYPGTQfB1tsalK+WYxf4pIovT5DC1fPlyjBs3Du7u7khMTETfvn3h4uKCc+fOYfDgwU16r+joaHz22WfYsGEDTp48iUmTJqG0tBTjxo0DAIwZMwazZ882jJ8+fTp27dqFpUuXIi0tDW+//Tbi4+MxZcoUALU/1GbMmIGFCxdi+/btSE5OxpgxY+Dl5WVYR8rBwQEvvfQS5s+fj99//x2nTp3CpEmTAADDhw9v6umwKMcv1TWfO4pbCJGRqeufspJJ8NuJbKw/eF7skoioDTV5BfRVq1Zh7dq1GDVqFNavX49Zs2ahY8eOmDdv3i0buBsyYsQI5ObmYt68edBoNAgODsauXbsMDeQZGRmQSq/lvf79+2PTpk2YO3cu5syZgy5dumDbtm0ICAgwjJk1axZKS0sxceJEFBYWYsCAAdi1axdUqmtrIr3//vuQy+V49tlnUV5ejrCwMOzZswdOTk5NPR0WpW5mKoD9UkQ3CPJxxJsPd8PbP6fi/3aeRO/27J8ishRNXmfKxsYGJ0+eRIcOHeDm5obdu3ejZ8+eOHPmDPr164f8/PzWqtWotMY6FcZMW1GNoLd/BwAkvPUAnG0VIldEZHwEQcCkbxKw64QGvs7W2DF1INTWXH+KyJiIus5UHQ8PD8MMVPv27XHo0CEAtWs8sU/AfKVcnZXycbJmkCK6iev7py4WlOON/x7nz0UiC9DkMHXfffdh+/btAIBx48bhlVdewQMPPIARI0bgiSeeaPECyTgkX+JinUSNoba2wopRtf1Tv6ZosDGOdwkTmbsm90ytXbvWsFbT5MmT4eLigoMHD+Kxxx676RpNZPrYL0XUeD19HfH6Q12x8JeTWLAjFSEdnNDN0/zbAYgsVZNmpmpqarBw4cJ6K5SPHDkSy5cvx9SpU6FQ8PKPuTJsI+PtKG4hRCZi/AB/3NfVDVU1ekzZlICyqhqxSyKiVtKkMCWXy7FkyRLU1PCHgiUpKqvGhfwyAFz5nKixJBIJ3n8yCO4OSvyTW4q3t58QuyQiaiVN7pm6//778ffff7dGLWSk6mal2jvbQG3DO5OIGsvFTollI3pBKgG+i7+En45lil0SEbWCJvdMDR48GG+88QaSk5MREhJyw9Yyjz32WIsVR8aBK58T3b7wTi6Yel8XfBxzBnN+SEZPH0f4uZrm3qZE1LAmh6mXX34ZAPDhhx/e8JxEIoFOp2t+VWRUkjMLAQBBvMRHdFum3tcZsefycTi9AFO+TcB/J/WHUi4TuywiaiFNvsyn1+tv+mCQMk9128hwZoro9shlUnw8MhiONlZIydRi8a+n/v1FRGQybmujY7IcV0qrcOlKOQAui0DUHJ5qa3zwZE8AwBcH0vFHarbIFRFRS2GYoluq65fyd7WFg4rN50TNEdHdHc/f5Q8AeO37JFwuKhe5IiJqCQxTdEuG5nPOShG1iNcH34kAbwdcKavG9M3HUKPTi10SETUTwxTd0vFLhQC4jQxRS1HKZfhkVG/YKmQ4nF6AT/acFbskImomhim6pZRMLQD2SxG1JH9XW/zf0EAAwCd7ziD2n3yRKyKi5mhymNJqtQ0+iouLUVVV1Ro1kkjySiqRWVgOiQTo4cV9xYha0uPB3hge4gO9AMzYkoiCUv78JDJVTQ5Tjo6OcHJyuuHh6OgIa2trdOjQAfPnzzdshkymq65fqqOrLezZfE7U4t55vAc6tbNFtrYSr25NgiAIYpdERLehyWFq/fr18PLywpw5c7Bt2zZs27YNc+bMgbe3Nz799FNMnDgRy5cvx3vvvdca9VIbSr66vlSQj6O4hRCZKRuFHCue7g2FXIo9aTlYtz9d7JKI6DY0eQX0DRs2YOnSpXjqqacMxx599FEEBgZizZo1iImJQfv27fHuu+9izpw5LVosta26mSn2SxG1nm6eDnjrke54a1sKFu9KQ19/Z/4DhsjENHlm6uDBg+jVq9cNx3v16oXY2FgAwIABA5CRkdH86khU12amGKaIWtMzYe3xUA8PVOsETP02EcUV1WKXRERN0OQw5evri3Xr1t1wfN26dfD19QUA5Ofnw8nJqfnVkWhytBXQaCsglQDdPdl8TtSaJBIJFg8LgrejNS7kl2HOjynsnyIyIU2+zPfBBx9g+PDh+PXXX9GnTx8AQHx8PNLS0vD9998DAI4cOYIRI0a0bKXUpuou8XVqZwdbZZP/mhBRE6ltrLB8VC88tSYWPydlYWBnVzzVx1fssoioEZo8M/XYY48hLS0NgwcPRkFBAQoKCjB48GCkpaXhkUceAQBMmjQJH374YYsXS23HsPI5L/ERtZmQDk6Y+eAdAIB521NwJrtY5IqIqDFua8rB39+fd+uZOUO/FJvPidrUS3d3Quw/+dh3Jg9TNiXipyl3QWUlE7ssIrqF2wpThYWFOHz4MHJycm5YT2rMmDEtUhiJRxAEHDfMTDmKWwyRhZFKJfjwqWAM/ngfTmUX4z87UvHuE4Fil0VEt9DkMPXzzz9j9OjRKCkpgYODAyQSieE5iUTCMGUGsrWVyC2uZPM5kUja2Svx0YieeHbdYWyMy0D/Tq4YEuQpdllEdBNN7pmaOXMmnn/+eZSUlKCwsBBXrlwxPAoKClqjRmpjdf1Sd7jbw1rBywtEYhjYpR0m3dMJAPDGD8dxsaBM5IqI6GaaHKYyMzMxbdo02NjYtEY9ZASSLxUCAALZL0UkqugH7kDv9o4orqjB1G8TUa3jNl1ExqjJYSoyMhLx8fGtUQsZibp+KS7WSSQuK5kUH4/sBQeVHMcuFuKD30+JXRIRNaDJPVNDhgzBa6+9htTUVAQGBsLKqv4GuI899liLFUdtTxAEpHAbGSKj4etsgyVPBuGlbxKw5u9zCO/ognvudBO7LCK6jkRo4jK7UunNJ7MkEgl0Ol2zizIFWq0WarUaRUVFcHAwnybtrMJy9H9vD+RSCVLeieQt2URG4q1tKfj60AW42Crw6/SBcHNQiV0SkUlqjd/fTb7Mp9frb/qwlCBlzo5futZ8ziBFZDzeHNINXT3skV9ahRlbjkGn53YzRMaiyWGKzFtyZiEA9ksRGRuVlQwrnu4NaysZDv6Tj1V/nhW7JCK6qlE9U8uXL8fEiROhUqmwfPnyW46dNm1aixRG4kjO1AJgvxSRMersZof/RAXg1a1J+OiP0wjr6IK+/s5il0Vk8RrVM+Xv74/4+Hi4uLjA39//5m8mkeDcuXMtWqCxMseeKUEQ0Ps/u3GlrBrbp9yFIK5+TmSUorccww+JmfBUq7Bz2kA42SrELonIZLTG7+9GzUylp6c3+L/JvFy6Uo4rZdWwkklwp4e92OUQ0U0siApA4sVCpOeV4rXvj+OzMSH1dqMgorbFnikyqFv5/E4PeyjlbD4nMlZ2SjlWPN0LCpkUf5zMxpcHzotdEpFFa/I6UzqdDuvXr0dMTEyDGx3v2bOnxYqjtlUXpgK9HcUthIj+VQ8vNd4c0g3zt5/Aol9Poo+fMwJ54wiRKJocpqZPn47169djyJAhCAgI4NSyGUm+xJXPiUzJmPAOOHA2D7+nZmPqtwnYMW0g7JRN/rFORM3U5P/qNm/ejO+++w4PP/xwa9RDIhEE4bqZKYYpIlMgkUiw5MkgnFi+H+fzy/Dmj8lYNiKY/8glamNN7plSKBTo3Llza9RCIrpYUI6i8mooZFLc4c7mcyJT4WijwPJRwZBJJfjpWBa2Hr0kdklEFqfJYWrmzJn4+OOP0cRdaMjIHb+6WGc3T3so5LwvgciUhHRwRvQDdwAA5v90AmdzikWuiMiyNPky3/79+/Hnn3/i119/RY8ePW7Y6PiHH35oseKo7dT1S7GBlcg0TRrUCbH/5GP/2TxM3piIn6bcxS2hiNpIk6cgHB0d8cQTT2DQoEFwdXWFWq2u9yDTVLcnXxDv5CMySVKpBB+O6AlXOwVOZRdjwY5UsUsishhNmpmqqanBvffeiwcffBAeHh6tVRO1Mb1eQEpWbZjiNjJEpsvNXoWPRgRjzBeHsSkuA3d1csWQIE+xyyIye02amZLL5XjppZdQWVnZWvWQCC4UlKG4ogZKuRRd3O3ELoeImmFgl3aYNKgTAOCN/x7HxYIykSsiMn9NvszXt29fJCYmtkYtJJLjlwoBAN29HGAlY/M5kal75YE7ENLBCcWVNZjybSKqavT//iIium1NbkB/+eWXMXPmTFy6dAkhISGwtbWt93xQUFCLFUdtw7BYJy/xEZkFK5kUH48MxsMf70PSxUJ88PspzHm4m9hlEZmtJoepkSNHAgCmTZtmOCaRSCAIAiQSCXQ6XctVR22ibrFO9ksRmQ8fJxu8P7wnXvz6KNbuPYfwTi649043scsiMktNDlPp6emtUQeJRK8XkJJZt42Mo7jFEFGLiuzhgbHhHbAh9gJmfpeEX6cPhLuDSuyyiMxOk8NUhw4dWqMOEsm5vFKUVulgbSVDp3a2//4CIjIpsx/uhiPnryD1shYzNh/DNy+EQSbldjNELem2d8RMTU1FRkYGqqqq6h1/7LHHml0UtZ26WanuXg6Qs/mcyOyorGRY8XQvPPLJfsSey8fKP89i2v1dxC6LyKw0OUydO3cOTzzxBJKTkw29UgAMG2uyZ8q01C3Wyc2NicxXx3Z2WBgVgOjvkrDsj9MI83dGWEcXscsiMhtNnoqYPn06/P39kZOTAxsbG5w4cQJ79+5FaGgo/vrrr1YokVpT8tU9+YK4jQyRWRva2wfDevtALwDTNx9DQWnVv7+IiBqlyWEqNjYWCxYsgKurK6RSKaRSKQYMGIBFixbVu8OPjJ9OLyAlUwuAYYrIEix4vAc6trOFRluB17YmccN6ohbS5DCl0+lgb28PAHB1dUVWVhaA2sb0U6dOtWx11KrO5ZagvFoHG4UM/q5c+ZzI3Nkq5VgxqjcUcili0nLwxYHzYpdEZBaaHKYCAgKQlJQEAAgLC8OSJUtw4MABLFiwAB07dmzxAqn11PVLBXipeXcPkYXo7uWAt4bULuD53q8ncexiobgFEZmBJoepuXPnQq+v3ZpgwYIFSE9Px8CBA7Fz504sX778topYuXIl/Pz8oFKpEBYWhsOHD99y/NatW9G1a1eoVCoEBgZi586d9Z4XBAHz5s2Dp6cnrK2tERERgTNnzjT4XpWVlQgODoZEIsGxY8duq35TVbdYZyAv8RFZlGf6dcDDgR6o1gmYvDEBRWXVYpdEZNKaHKYiIyMxdOhQAEDnzp2RlpaGvLw85OTk4L777mtyAVu2bEF0dDTmz5+PhIQE9OzZE5GRkcjJyWlw/MGDBzFq1CiMHz8eiYmJiIqKQlRUFFJSUgxjlixZguXLl2P16tWIi4uDra0tIiMjUVFRccP7zZo1C15eXk2u2xzU7cnHfikiyyKRSPDesCB0cLFBZmE5Xv2e/VNEzXHbCwudPXsWv/32G8rLy+Hs7HzbBXz44YeYMGECxo0bh+7du2P16tWwsbHBF1980eD4jz/+GA899BBee+01dOvWDf/5z3/Qu3dvrFixAkDtrNSyZcswd+5cPP744wgKCsJXX32FrKwsbNu2rd57/frrr/j999/xwQcf3Hb9pqpGp0fq5drmc24jQ2R5HFRWWPl0byhkUuxOzca6/dzdguh2NTlM5efn4/7778cdd9yBhx9+GJcvXwYAjB8/HjNnzmzSe1VVVeHo0aOIiIi4VpBUioiICMTGxjb4mtjY2HrjgdrZsrrx6enp0Gg09cao1WqEhYXVe8/s7GxMmDABX3/9NWxsbP611srKSmi12noPU3Y2twQV1XrYKeXwd+HK50SWKMBbjbce7Q4AeO/XNCRkXBG5IiLT1OQw9corr8DKygoZGRn1QsiIESOwa9euJr1XXl4edDod3N3d6x13d3eHRqNp8DUajeaW4+v+vNUYQRDw3HPP4aWXXkJoaGijal20aBHUarXh4evr26jXGStD87m3A6RsPieyWM+EtccjQZ6o0QuYsjEBV7j+FFGTNTlM/f7771i8eDF8fHzqHe/SpQsuXLjQYoW1pk8++QTFxcWYPXt2o18ze/ZsFBUVGR4XL15sxQpbX/Ilbm5MRLX9U4uGBsLf1RZZRRWYuTUJej37p4iaoslhqrS0tMHLYgUFBVAqlU16L1dXV8hkMmRnZ9c7np2dDQ8PjwZf4+HhccvxdX/easyePXsQGxsLpVIJuVyOzp07AwBCQ0MxduzYBr+vUqmEg4NDvYcpq7uTj/1SRGSvssKKp3tBIZdiT1oO1u47J3ZJRCalyWFq4MCB+OqrrwxfSyQS6PV6LFmyBPfee2+T3kuhUCAkJAQxMTGGY3q9HjExMQgPD2/wNeHh4fXGA8Du3bsN4/39/eHh4VFvjFarRVxcnGHM8uXLkZSUhGPHjuHYsWOGpRW2bNmCd999t0mfwRRVX9d8HsQwRUQAenip8fajPQAA7/92CkfOF4hcEZHpaPJGx0uWLMH999+P+Ph4VFVVYdasWThx4gQKCgpw4MCBJhcQHR2NsWPHIjQ0FH379sWyZctQWlqKcePGAQDGjBkDb29vLFq0CEDt3oCDBg3C0qVLMWTIEGzevBnx8fFYu3YtgNpwN2PGDCxcuBBdunSBv78/3nrrLXh5eSEqKgoA0L59+3o12NnVrv7dqVOnGy5fmqPT2cWoqtHDXiVHB5d/b74nIsswqq8v4tLz8dOxLEzdlIhfpg2Ai13TrjgQWaImh6mAgACcPn0aK1asgL29PUpKSjB06FBMnjwZnp6eTS5gxIgRyM3Nxbx586DRaBAcHIxdu3YZGsgzMjIglV6bQOvfvz82bdqEuXPnYs6cOejSpQu2bduGgIAAw5hZs2ahtLQUEydORGFhIQYMGIBdu3ZBpVI1uT5zlFK3WKe3GhIJm8+JqJZEIsH/PRGI5MwinMstxSvfJWH9c314kwrRv5AILbRS26VLl7BgwQLDDJG502q1UKvVKCoqMrn+qTd/TMbGuAy8OKgjZg/uJnY5RGRk0jRaRK08gIpqPV6LvBOT7+0sdklELaY1fn/f9qKd/ys/Px/r1q1rqbejVlTXfB7k7ShuIURklLp6OGDBY7Wz/Ut/P4VD5/JFrojIuLVYmCLTUFWjR9rlYgDcRoaIbm54qA+G9vaGXgCmfZuI3OJKsUsiMloMUxbmdHYxqnR6qK2t4ONkLXY5RGSkJBIJFkYFoIubHXKKK/HKlmPQcf0pogYxTFmY44bFOtl8TkS3ZqOQY9Xo3rC2kmH/2Tys2HNW7JKIjFKj7+YbOnToLZ8vLCxsbi3UBpIzCwHU3slHRPRvurjbY2FUAGZuTcKymNPo4+eE/p1dxS6LyKg0Okyp1bf+5atWqzFmzJhmF0St6/qZKSKixhgW4oO49Hx8F38J0zYfw87pA+Bmz6VmiOo0Okx9+eWXrVkHtYGKah1OZ9c2n3MbGSJqinceC0DSxSKcyi7G1E2J2PhCGOQydooQAeyZsiinNMWo1glwtlXA25HN50TUeNYKGVY90xu2Chni0gvw/m+nxC6JyGgwTFmQ41z5nIiaoVM7O7w/vCcAYM3ec9iVclnkioiMA8OUBUlhvxQRNdPDgZ54YYA/AODVrcdxLrdE5IqIxMcwZUHqZqbYL0VEzfH64K7o6+eMksoaTPomAWVVNWKXRCQqhikLcX3zOWemiKg5rGRSrHi6F1ztlDiVXYw3f0xBC23zSmSSGKYsROplLXR6Aa52Sng48JZmImoeNwcVVj7dCzKpBD8mZuKbuAyxSyISDcOUhUjJ5MrnRNSywjq64PWH7gQALPj5BI5dLBS3ICKRMExZiLrFOtkvRUQtacLAjniohweqdQJe/uYoCkqrxC6JqM0xTFmI5Lo7+RimiKgFSSQSvD88CP6utsgqqsD0zYncEJksDsOUBSirqsGZnNrm80A2nxNRC7NXWWH1MyGwtpJh35k8fPzHabFLImpTDFMW4ORlLfQC4GavhDubz4moFdzpYY9FQwMBAMv3nMWetGyRKyJqOwxTFoCbGxNRW4jq5Y0x4R0AAK9sScLFgjKRKyJqGwxTFqCuXyrQ21HcQojI7L05pBuCfR1RVF6NF78+ivIqndglEbU6hikLcDyTM1NE1DaUchlWje4NVzsFUi9r8fp/j3NBTzJ7DFNmrrSyBv9c3TuLyyIQUVvwcrTGyqd7Qy6VYHtSFj7fly52SUStimHKzJ3I0kIQAE+1Cu3slWKXQ0QWIqyjC956pDsAYNGvJ7H/TJ7IFRG1HoYpM3f8UiEAIJCzUkTUxsaEd8Cw3j7QC8DUbxPYkE5mi2HKzKWwX4qIRCKRSPDuEwEI8lHjShkb0sl8MUyZubrmc/ZLEZEYVFYyrH4mBC62tQ3pb/zAhnQyPwxTZqy4ohrncksB8DIfEYnHy9EaK0fXNqT/dCwL6/azIZ3MC8OUGUvJ1AIAvB2t4WLH5nMiEk+/ji6YO6QbAOD/dp7EgbNsSCfzwTBlxtgvRUTGZGx/P0ND+pRNbEgn88EwZcbYL0VExqSuIT3Qmw3pZF4YpsxY8tVlETgzRUTGQmUlw5pn2ZBO5oVhykwVlVfjfH7tFDqbz4nImNQ1pMuuNqRzhXQydQxTZurE1Ut8vs7WcLRRiFwNEVF91zekL/r1JP48lSNyRUS3j2HKTBk2N/Z2FLcQIqKbeK6/H0aE+kIvANM2JeJsTonYJRHdFoYpM5V8qTZMBbJfioiMlEQiwX+iAtDHzwnFlTV4YcMRFJZViV0WUZMxTJmpZMPMFMMUERkvhVyKT58JgbejNc7nl2HKpkTU6PRil0XUJAxTZqiwrAoZV9dv6cEwRURGztVOic/GhMJGIcP+s3lY+MtJsUsiahKGKTNUNyvl52IDtbWVyNUQEf277l4O+PCpYADA+oPnsSkuQ9yCiJqAYcoMHTf0SzmKWwgRURM8FOCBmQ/cAQCY91MKDp3LF7kiosZhmDJDKeyXIiITNeW+zni0pxdq9AImfXMUGfnccoaMH8OUGaqbmeI2MkRkaiQSCd5/MghBPrVbzjy/4QiKyqvFLovolhimzEx+SSUyC8sBAAHeDiJXQ0TUdCorGdY+GwoPBxXO5pRg8sYEVPMOPzJiDFNmpq75vGM7W9ir2HxORKbJQ63C52Ov3eE3f/sJ7uFHRothysywX4qIzEWAtxrLR/aCRAJsisvAuv3cw4+ME8OUmWG/FBGZk4ju7pg7pDsA4N2dJ/H7CY3IFRHdiGHKzBhWPueyCERkJp6/yw/P9GsPQQCmbz5mmIEnMhYMU2Ykt7gSl4sqIJEAPbzYfE5E5kEikeDtR3tgYBdXlFfrMH7DEVwuKhe7LCIDhikzUvevtU7t7GCrlItcDRFRy5HLpFg5ujfucLdDtrYS49fHo6SyRuyyiAAwTJmVun4pNp8TkTlyUFlh3dg+cLVTIPWylksmkNFgmDIjyZmFAIBAH4YpIjJPvs42WDe2D6ytZPj7dC7e/DGZSyaQ6BimzMi15nOGKSIyXz19HbHi6V6QSoDv4i9hecxZsUsiC8cwZSaytRXI1lZCKgG6ezJMEZF5u7+bO/4TFQAA+OiP0/gu/qLIFZElY5gyE8lX+6W6uNnDWiETuRoiotY3OqwDXr6nEwBg9g/J+Pt0rsgVkaVimDITx69e4mO/FBFZktci70RUsBd0egEvf3OUa1CRKBimzEQK+6WIyAJJJBIsebIn+ndyQWmVDuPWH8GlK2Vil0UWhmHKDAiCwG1kiMhiKeRSrH42BHe62yO3uBJjvziMgtIqscsiC2IUYWrlypXw8/ODSqVCWFgYDh8+fMvxW7duRdeuXaFSqRAYGIidO3fWe14QBMybNw+enp6wtrZGREQEzpw5Y3j+/PnzGD9+PPz9/WFtbY1OnTph/vz5qKoyzf/4NNoK5JVUQiaVoLsnVz4nIsvjoLLCl+P6wFOtwj+5pXh+/RGUclFPaiOih6ktW7YgOjoa8+fPR0JCAnr27InIyEjk5OQ0OP7gwYMYNWoUxo8fj8TERERFRSEqKgopKSmGMUuWLMHy5cuxevVqxMXFwdbWFpGRkaioqAAApKWlQa/XY82aNThx4gQ++ugjrF69GnPmzGmTz9zS6mal7nC3h8qKzedEZJm8HK3x1fN94WhjhWMXC/HSN0dRVcNFPan1SQSRVzsLCwtDnz59sGLFCgCAXq+Hr68vpk6dijfeeOOG8SNGjEBpaSl27NhhONavXz8EBwdj9erVEAQBXl5emDlzJl599VUAQFFREdzd3bF+/XqMHDmywTref/99fPrppzh37lyj6tZqtVCr1SgqKoKDg7izQUt/P4VP9pzFiFBfLH4ySNRaiIjElpBxBaM/i0N5tQ6P9vTCxyOCIZVKxC6LjERr/P4WdWaqqqoKR48eRUREhOGYVCpFREQEYmNjG3xNbGxsvfEAEBkZaRifnp4OjUZTb4xarUZYWNhN3xOoDVzOzs43fb6yshJarbbew1gY+qXYfE5EhN7tnbD62RDIpRL8nJSFd34+wVXSqVWJGqby8vKg0+ng7u5e77i7uzs0Gk2Dr9FoNLccX/dnU97z7Nmz+OSTT/Diiy/etNZFixZBrVYbHr6+vrf+cG1EEIRrK5+z+ZyICAAw6I52WPpUTwDAhtgLWLGHq6RT6xG9Z0psmZmZeOihhzB8+HBMmDDhpuNmz56NoqIiw+PiReNYbTerqAIFpVWwkknQ1dNe7HKIiIzG48HeePvR7gCApbtPY2PcBZErInMlaphydXWFTCZDdnZ2vePZ2dnw8PBo8DUeHh63HF/3Z2PeMysrC/feey/69++PtWvX3rJWpVIJBweHeg9jkHypEEBt87lSzuZzIqLrPXeXP6bd1xkAMHdbCn45flnkisgciRqmFAoFQkJCEBMTYzim1+sRExOD8PDwBl8THh5ebzwA7N692zDe398fHh4e9cZotVrExcXVe8/MzEzcc889CAkJwZdffgmp1DQn6er6pbhYJxFRw1554A6MDmsPQQCmb05EzMnsf38RUROIniCio6Px2WefYcOGDTh58iQmTZqE0tJSjBs3DgAwZswYzJ492zB++vTp2LVrF5YuXYq0tDS8/fbbiI+Px5QpUwDUroY7Y8YMLFy4ENu3b0dycjLGjBkDLy8vREVFAbgWpNq3b48PPvgAubm50Gg0N+2pMmZ1/VKB3o7iFkJEZKQkEgkWPB6Ax4O9UKMXMGljAg6czRO7LDIjcrELGDFiBHJzczFv3jxoNBoEBwdj165dhgbyjIyMerNG/fv3x6ZNmzB37lzMmTMHXbp0wbZt2xAQEGAYM2vWLJSWlmLixIkoLCzEgAEDsGvXLqhUKgC1M1lnz57F2bNn4ePjU68eU7rjo17zOWemiIhuSiaV4IPhPVFepcPvqdl4YUM8vh7fF6F+N7+Lm6ixRF9nylQZwzpTFwvKMHDJn1DIpEh5JxIKuegTjURERq2yRoeJXx3F36dzYa+UY+OEMAT5OIpdFrUhs1tnipqnrl+qq6c9gxQRUSMo5TKsfiYEYf7OKK6swZgvDiNNYzzrBpJp4m9gE3Y8sxAAEMj1pYiIGs1aIcO65/og2NcRhWXVeObzwziXWyJ2WWTCGKZMWAr7pYiIboudUo4N4/qiu6cD8koqMfrzOGTkl4ldFpkohikTJQjCtW1kODNFRNRkahsrfD2+Lzq72eFyUQVGro1loKLbwjBloi7kl6G4ogYKuRR3uHPlcyKi2+Fip8SmF8LQsZ0tsq4Gqgv5pWKXRSaGYcpEHb96ia+7pwOsZPy/kYjodrk5qLB5Qj90MgSqQwxU1CT8LWyi2C9FRNRy3BxU+HZibaC6fDVQnc9joKLGYZgyUcev7snHfikiopbhZl8/UI36jIGKGodhygTp9QJSMmvXReHMFBFRy6kLVNea0hmo6N8xTJmg8/mlKKmsgcpKis7t7MQuh4jIrLjZq/DthNpApdFWYMTaWJzNKRa7LDJiDFMmKPm65nM5m8+JiFpcO3slvp3QD13c7JCtrcRTaw4ZelWJ/hd/E5uguvWluJ8UEVHraWevxJYXwxHorUZBaRVGfXYI8ecLxC6LjBDDlAlKvhqmuI0MEVHrcrZVYOOEMPTxc0JxRQ2eXXcY+87kil0WGRmGKROj0ws4kcVlEYiI2oqDygpfPR+GQXe0Q3m1DuPXx+O3ExqxyyIjwjBlYtLzSlBapYONQoaObD4nImoT1goZPhsTisEBHqjS6fHyxgT8mHhJ7LLISDBMmZi6fqkeXg6QSSUiV0NEZDkUcik+GdULT4b4QKcXEP1dEr6KPS92WWQEGKZMzHFDv5SjuIUQEVkguUyKJcOC8Fx/PwgCMO+nE3j/tzQIgiB2aSQihikTw21kiIjEJZVKMP/R7ngl4g4AwMo//8HMrUmo1ulFrozEwjBlQmp0epzIql35nNvIEBGJRyKRYHpEFywZFgSZVIIfEjLx/PojKKmsEbs0EgHDlAn5J7cU5dU62Cpk6OhqK3Y5REQW76k+vvh8bCisrWTYdyYPI9bEIqe4QuyyqI0xTJmQupXPA7zVkLL5nIjIKNx7pxs2T+wHF1sFTmRpMXTVQfyTWyJ2WdSGGKZMSPKlQgDslyIiMjY9fR3xw8v94edig0tXyvHkpwdxhKulWwyGKRNy/LqZKSIiMi4dXGzx30n90dPXEVfKqvH0Z4ewNf6i2GVRG2CYMhE1Oj1Srzafc08+IiLj5GKnxLcTwjA4wAPVOgGvfX8c/7fzJHR6Lp1gzhimTMSZnBJU1uhhr5Kjg7ON2OUQEdFN2CjkWPl0b0y7rzMAYO3ec5j4VTyKK6pFroxaC8OUibh+c2M2nxMRGTepVILoB+/E8lG9oJRLEZOWg2GfHsTFgjKxS6NWwDBlIo5nFgKoDVNERGQaHuvphS0vhsPNXonT2SV4fOUBHE5nY7q5YZgyEYaZKd7JR0RkUoJ9HbF9ygAEeqtRUFqF0Z8fwoaD57kFjRlhmDIBVTV6nNQUAwCCuCcfEZHJ8VCr8N2L4XgkyBPVOgHzt5/AtM3HUMoV080Cw5QJOJ1djKoaPdTWVvB1tha7HCIiug3WChk+GdULbz3SHXKpBD8nZeHxlQdwNqdY7NKomRimTEDdyueB3mpIJGw+JyIyVRKJBOMH+GPzxH5wd1DibE4JHltxAD8nZYldGjUDw5QJMIQp9ksREZmFUD9n7Jg6EOEdXVBWpcPUbxPx9vYTqKrRi10a3QaGKRNQ13wexDv5iIjMRjt7Jb4e3xcv39MJALD+4Hk8tSYWF/JLRa6MmophyshV1uiQpqld+ZzbyBARmRe5TIpZD3XFZ2NC4aCS49jFQjz88T5sjb/Iu/1MCMOUkTulKUa1ToCTjRV8nNh8TkRkjh7o7o6d0weir78zSqt0eO3745iyKRGFZVVil0aNwDBl5K71Szmy+ZyIyIz5ONng2wn98FrknZBLJfgl+TIeWrYPB//JE7s0+hcMU0aO/VJERJZDJpVg8r2d8cPL/eHvaguNtgKjP4/Dop0nUVmjE7s8ugmGKSN3/GqYYr8UEZHlCPJxxC/TBmBUX18IArBm7zk89skBJGZcEbs0agDDlBGrqNbhdPbVlc+5LAIRkUWxUcixaGgQ1jwbAmdbBU5lF2Popwex4OdUlFVx5XRjwjBlxNI0xajRC3C1U8BTrRK7HCIiEkFkDw/8ET0IT/TyhiAAXxxIx4Mf7cXe07lil0ZXMUwZseRLhQC48jkRkaVztlXgoxHBWD+uD7wdrXHpSjnGfHEY0d8dw5VS3vEnNoYpI1bXLxXIfikiIgJwz51u+P2Vu/Fcfz9IJMAPCZl44KO/8V38Rej1XJdKLAxTRuz6ZRGIiIgAwFYpx9uP9cD3L/VHFzc75JVUYdb3xxG16gCOXigQuzyLxDBlpMqrdDiTUwKAzedERHSjkA5O+GXaQMx5uCvslHIcv1SEYZ/GYtq3icgqLBe7PIvCMGWkUi9rodMLaGevhLsDm8+JiOhGCrkUE+/uhD9fvQcj+/hCIgG2J2XhvqV/4eM/zqC8imtTtQWGKSNV13zOxTqJiOjftLNX4r1hQfh5ygD08XNCRbUeH/1xGvct/Qub4jJQrdOLXaJZY5gyUsmZtZsbB/ISHxERNVKAtxrfvRiOT0b1gpdahctFFZjzYzLuW/oXvou/iBqGqlbBMGWkkjMLAbBfioiImkYikeDRnl7Y8+o9mPdId7jaKXGxoByzvj+OiA//xo+Jl6DjnX8timHKCJVW1uDs1eZzbiNDRES3Q2Ulw/MD/LFv1r2Y83BXONsqcD6/DK9sScKDH9WGqqoazlS1BIYpI5R6WQu9AHg4qOBmz+ZzIiK6fdYKGSbe3Qn7Zt2LWQ/dCUcbK/yTW4pXtiRh4JI9WPnnWS782UwMU0YouW6xTl7iIyKiFmKrlOPlezpj36x78VrknXCzVyJbW4n3fzuFfotiMPuHZJy5uh8sNY1c7ALoRnWLdfJOPiIiamn2KitMvrczJgzsiF+Ss7BufzpSMrX49nAGvj2cgbvvaIfRYe1xX1c3WMk459IYDFNG6PjVZRECODNFREStRCGX4olePogK9saR81ewbv85/J6ajb2nc7H3dC5c7ZQY1tsbw0N90NnNXuxyjRrDlJEpqazBubxSANyTj4iIWp9EIkFff2f09XdGRn4ZNsZdwH8TLiGvpBJr9p7Dmr3n0N3TAY8Fe+HRnl7wdrQWu2SjIxEEgfdH3gatVgu1Wo2ioiI4ODi02PvGncvHiLWH4O1ojQNv3Ndi70tERNRY1To99qTlYGv8Rfx1Khc11y2l0NNHjQd7eODB7u7o7GYHiUQiYqVN1xq/vzkzZWQMmxtzVoqIiERiJZMisocHInt44EppFXamXMZPx7Jw5HwBki4VIelSEd7/7RR8na1xd5d2uPuOdujfyQX2KiuxSxcFw5SROc47+YiIyIg42SowOqwDRod1QE5xBf5IzcHvqRocPJuPiwXl2BiXgY1xGZBKgO5eDujr54K+/k7o6esIDweVyc1c3Q6jaNNfuXIl/Pz8oFKpEBYWhsOHD99y/NatW9G1a1eoVCoEBgZi586d9Z4XBAHz5s2Dp6cnrK2tERERgTNnztQbU1BQgNGjR8PBwQGOjo4YP348SkpKWvyzNVVheTUAzkwREZHxcbNX4emw9lg/ri8S5z2AdWNDMTa8A/xdbaEXgJRMLb44kI6XvklA+KI96PNuDJ5ffwTv/5aGn45lIjVLi4pq89t8WfSeqS1btmDMmDFYvXo1wsLCsGzZMmzduhWnTp2Cm5vbDeMPHjyIu+++G4sWLcIjjzyCTZs2YfHixUhISEBAQAAAYPHixVi0aBE2bNgAf39/vPXWW0hOTkZqaipUqtpFMAcPHozLly9jzZo1qK6uxrhx49CnTx9s2rSpUXW3Vs8UABSVV0NlJYVSLmvR9yUiImotmqIKHD5fgMPp+Yg/fwVnckpuum2Nh4MKvs7W8HWyQTsHJdrZKdHOXglHGwXsVXLYK+WwU8nhYquEQt6y8z6t8ftb9DAVFhaGPn36YMWKFQAAvV4PX19fTJ06FW+88cYN40eMGIHS0lLs2LHDcKxfv34IDg7G6tWrIQgCvLy8MHPmTLz66qsAgKKiIri7u2P9+vUYOXIkTp48ie7du+PIkSMIDQ0FAOzatQsPP/wwLl26BC8vrxu+b2VlJSorKw1fa7Va+Pr6tkqYIiIiMnXlVTqkXtYi+VIhTmWX4Ex2MU5nF0NbUdPo9/h8TCgiuru3aF1m14BeVVWFo0ePYvbs2YZjUqkUERERiI2NbfA1sbGxiI6OrncsMjIS27ZtAwCkp6dDo9EgIiLC8LxarUZYWBhiY2MxcuRIxMbGwtHR0RCkACAiIgJSqRRxcXF44oknbvi+ixYtwjvvvNOcj0tERGQxrBUyhHRwQkgHJ8MxQRBwpawaGQVluFhQhktXypFbXInckkrkFlegqLwGJZXVKKmoQXFFDexVptHaLWqVeXl50Ol0cHevnzrd3d2RlpbW4Gs0Gk2D4zUajeH5umO3GvO/lxDlcjmcnZ0NY/7X7Nmz64W4upkpIiIiahyJRAJnWwWcbRUI9nW85VhTWrnJNCKfEVAqlVAqlWKXQUREZBFM6S5AUe/mc3V1hUwmQ3Z2dr3j2dnZ8PDwaPA1Hh4etxxf9+e/jcnJyan3fE1NDQoKCm76fYmIiIgaImqYUigUCAkJQUxMjOGYXq9HTEwMwsPDG3xNeHh4vfEAsHv3bsN4f39/eHh41Buj1WoRFxdnGBMeHo7CwkIcPXrUMGbPnj3Q6/UICwtrsc9HRERE5k/0y3zR0dEYO3YsQkND0bdvXyxbtgylpaUYN24cAGDMmDHw9vbGokWLAADTp0/HoEGDsHTpUgwZMgSbN29GfHw81q5dC6B2WnDGjBlYuHAhunTpYlgawcvLC1FRUQCAbt264aGHHsKECROwevVqVFdXY8qUKRg5cmSDd/IRERER3YzoYWrEiBHIzc3FvHnzoNFoEBwcjF27dhkayDMyMiCVXptA69+/PzZt2oS5c+dizpw56NKlC7Zt22ZYYwoAZs2ahdLSUkycOBGFhYUYMGAAdu3aZVhjCgA2btyIKVOm4P7774dUKsWwYcOwfPnytvvgREREZBZEX2fKVLXmop1ERETUOlrj97dRbCdDREREZKoYpoiIiIiagWGKiIiIqBkYpoiIiIiagWGKiIiIqBkYpoiIiIiagWGKiIiIqBkYpoiIiIiaQfQV0E1V3VqnWq1W5EqIiIiosep+b7fkmuUMU7epuLgYAODr6ytyJURERNRUxcXFUKvVLfJe3E7mNun1emRlZcHe3h4SiaTF3ler1cLX1xcXL17kNjXg+bgez8U1PBf18Xxcw3NxDc/FNdefC3t7exQXF8PLy6ve3r/NwZmp2ySVSuHj49Nq7+/g4GDxf/mvx/NxDc/FNTwX9fF8XMNzcQ3PxTV156KlZqTqsAGdiIiIqBkYpoiIiIiagWHKyCiVSsyfPx9KpVLsUowCz8c1PBfX8FzUx/NxDc/FNTwX17T2uWADOhEREVEzcGaKiIiIqBkYpoiIiIiagWGKiIiIqBkYpoiIiIiagWHKyKxcuRJ+fn5QqVQICwvD4cOHxS6p1S1atAh9+vSBvb093NzcEBUVhVOnTtUbU1FRgcmTJ8PFxQV2dnYYNmwYsrOzRaq47bz33nuQSCSYMWOG4ZglnYvMzEw888wzcHFxgbW1NQIDAxEfH294XhAEzJs3D56enrC2tkZERATOnDkjYsWtR6fT4a233oK/vz+sra3RqVMn/Oc//6m3v5i5no+9e/fi0UcfhZeXFyQSCbZt21bv+cZ87oKCAowePRoODg5wdHTE+PHjUVJS0oafomXc6lxUV1fj9ddfR2BgIGxtbeHl5YUxY8YgKyur3nuYy7kA/v3vxvVeeuklSCQSLFu2rN7xljgfDFNGZMuWLYiOjsb8+fORkJCAnj17IjIyEjk5OWKX1qr+/vtvTJ48GYcOHcLu3btRXV2NBx98EKWlpYYxr7zyCn7++Wds3boVf//9N7KysjB06FARq259R44cwZo1axAUFFTvuKWciytXruCuu+6ClZUVfv31V6SmpmLp0qVwcnIyjFmyZAmWL1+O1atXIy4uDra2toiMjERFRYWIlbeOxYsX49NPP8WKFStw8uRJLF68GEuWLMEnn3xiGGOu56O0tBQ9e/bEypUrG3y+MZ979OjROHHiBHbv3o0dO3Zg7969mDhxYlt9hBZzq3NRVlaGhIQEvPXWW0hISMAPP/yAU6dO4bHHHqs3zlzOBfDvfzfq/Pjjjzh06BC8vLxueK5FzodARqNv377C5MmTDV/rdDrBy8tLWLRokYhVtb2cnBwBgPD3338LgiAIhYWFgpWVlbB161bDmJMnTwoAhNjYWLHKbFXFxcVCly5dhN27dwuDBg0Spk+fLgiCZZ2L119/XRgwYMBNn9fr9YKHh4fw/vvvG44VFhYKSqVS+Pbbb9uixDY1ZMgQ4fnnn693bOjQocLo0aMFQbCc8wFA+PHHHw1fN+Zzp6amCgCEI0eOGMb8+uuvgkQiETIzM9us9pb2v+eiIYcPHxYACBcuXBAEwXzPhSDc/HxcunRJ8Pb2FlJSUoQOHToIH330keG5ljofnJkyElVVVTh69CgiIiIMx6RSKSIiIhAbGytiZW2vqKgIAODs7AwAOHr0KKqrq+udm65du6J9+/Zme24mT56MIUOG1PvMgGWdi+3btyM0NBTDhw+Hm5sbevXqhc8++8zwfHp6OjQaTb1zoVarERYWZnbnAgD69++PmJgYnD59GgCQlJSE/fv3Y/DgwQAs73zUacznjo2NhaOjI0JDQw1jIiIiIJVKERcX1+Y1t6WioiJIJBI4OjoCsLxzodfr8eyzz+K1115Djx49bni+pc4HNzo2Enl5edDpdHB3d6933N3dHWlpaSJV1fb0ej1mzJiBu+66CwEBAQAAjUYDhUJh+GFQx93dHRqNRoQqW9fmzZuRkJCAI0eO3PCcJZ2Lc+fO4dNPP0V0dDTmzJmDI0eOYNq0aVAoFBg7dqzh8zb034y5nQsAeOONN6DVatG1a1fIZDLodDq8++67GD16NABY3Pmo05jPrdFo4ObmVu95uVwOZ2dnsz43FRUVeP311zFq1CjDRseWdi4WL14MuVyOadOmNfh8S50PhikyKpMnT0ZKSgr2798vdimiuHjxIqZPn47du3dDpVKJXY6o9Ho9QkND8X//938AgF69eiElJQWrV6/G2LFjRa6u7X333XfYuHEjNm3ahB49euDYsWOYMWMGvLy8LPJ80K1VV1fjqaeegiAI+PTTT8UuRxRHjx7Fxx9/jISEBEgkklb9XrzMZyRcXV0hk8luuCsrOzsbHh4eIlXVtqZMmYIdO3bgzz//hI+Pj+G4h4cHqqqqUFhYWG+8OZ6bo0ePIicnB71794ZcLodcLsfff/+N5cuXQy6Xw93d3WLOhaenJ7p3717vWLdu3ZCRkQEAhs9rKf/NvPbaa3jjjTcwcuRIBAYG4tlnn8Urr7yCRYsWAbC881GnMZ/bw8Pjhht5ampqUFBQYJbnpi5IXbhwAbt37zbMSgGWdS727duHnJwctG/f3vDz9MKFC5g5cyb8/PwAtNz5YJgyEgqFAiEhIYiJiTEc0+v1iImJQXh4uIiVtT5BEDBlyhT8+OOP2LNnD/z9/es9HxISAisrq3rn5tSpU8jIyDC7c3P//fcjOTkZx44dMzxCQ0MxevRow/+2lHNx11133bBExunTp9GhQwcAgL+/Pzw8POqdC61Wi7i4OLM7F0DtnVpSaf0f2TKZDHq9HoDlnY86jfnc4eHhKCwsxNGjRw1j9uzZA71ej7CwsDavuTXVBakzZ87gjz/+gIuLS73nLelcPPvsszh+/Hi9n6deXl547bXX8NtvvwFowfNx+33z1NI2b94sKJVKYf369UJqaqowceJEwdHRUdBoNGKX1qomTZokqNVq4a+//hIuX75seJSVlRnGvPTSS0L79u2FPXv2CPHx8UJ4eLgQHh4uYtVt5/q7+QTBcs7F4cOHBblcLrz77rvCmTNnhI0bNwo2NjbCN998Yxjz3nvvCY6OjsJPP/0kHD9+XHj88ccFf39/oby8XMTKW8fYsWMFb29vYceOHUJ6errwww8/CK6ursKsWbMMY8z1fBQXFwuJiYlCYmKiAED48MMPhcTERMMdao353A899JDQq1cvIS4uTti/f7/QpUsXYdSoUWJ9pNt2q3NRVVUlPPbYY4KPj49w7Nixej9PKysrDe9hLudCEP7978b/+t+7+QShZc4Hw5SR+eSTT4T27dsLCoVC6Nu3r3Do0CGxS2p1ABp8fPnll4Yx5eXlwssvvyw4OTkJNjY2whNPPCFcvnxZvKLb0P+GKUs6Fz///LMQEBAgKJVKoWvXrsLatWvrPa/X64W33npLcHd3F5RKpXD//fcLp06dEqna1qXVaoXp06cL7du3F1QqldCxY0fhzTffrPdL0lzPx59//tngz4ixY8cKgtC4z52fny+MGjVKsLOzExwcHIRx48YJxcXFInya5rnVuUhPT7/pz9M///zT8B7mci4E4d//bvyvhsJUS5wPiSBct3wuERERETUJe6aIiIiImoFhioiIiKgZGKaIiIiImoFhioiIiKgZGKaIiIiImoFhioiIiKgZGKaIiIiImoFhioiIiKgZGKaIiIiImoFhiojMWm5uLiZNmoT27dtDqVTCw8MDkZGROHDgAABAIpFg27Zt4hZJRCZNLnYBREStadiwYaiqqsKGDRvQsWNHZGdnIyYmBvn5+WKXRkRmgnvzEZHZKiwshJOTE/766y8MGjTohuf9/Pxw4cIFw9cdOnTA+fPnAQA//fQT3nnnHaSmpsLLywtjx47Fm2++Cbm89t+gEokEq1atwvbt2/HXX3/B09MTS5YswZNPPtkmn42IjAcv8xGR2bKzs4OdnR22bduGysrKG54/cuQIAODLL7/E5cuXDV/v27cPY8aMwfTp05Gamoo1a9Zg/fr1ePfdd+u9/q233sKwYcOQlJSE0aNHY+TIkTh58mTrfzAiMiqcmSIis/bf//4XEyZMQHl5OXr37o1BgwZh5MiRCAoKAlA7w/Tjjz8iKirK8JqIiAjcf//9mD17tuHYN998g1mzZiErK8vwupdeegmffvqpYUy/fv3Qu3dvrFq1qm0+HBEZBc5MEZFZGzZsGLKysrB9+3Y89NBD+Ouvv9C7d2+sX7/+pq9JSkrCggULDDNbdnZ2mDBhAi5fvoyysjLDuPDw8HqvCw8P58wUkQViAzoRmT2VSoUHHngADzzwAN566y288MILmD9/Pp577rkGx5eUlOCdd97B0KFDG3wvIqLrcWaKiCxO9+7dUVpaCgCwsrKCTqer93zv3r1x6tQpdO7c+YaHVHrtx+ahQ4fqve7QoUPo1q1b638AIjIqnJkiIrOVn5+P4cOH4/nnn0dQUBDs7e0RHx+PJUuW4PHHHwdQe0dfTEwM7rrrLiiVSjg5OWHevHl45JFH0L59ezz55JOQSqVISkpCSkoKFi5caHj/rVu3IjQ0FAMGDMDGjRtx+PBhrFu3TqyPS0QiYQM6EZmtyspKvP322/j999/xzz//oLq6Gr6+vhg+fDjmzJkDa2tr/Pzzz4iOjsb58+fh7e1tWBrht99+w4IFC5CYmAgrKyt07doVL7zwAiZMmACgtgF95cqV2LZtG/bu3QtPT08sXrwYTz31lIifmIjEwDBFRHQbGroLkIgsE3umiIiIiJqBYYqIiIioGdiATkR0G9ghQUR1ODNFRERE1AwMU0RERETNwDBFRERE1AwMU0RERETNwDBFRERE1AwMU0RERETNwDBFRERE1AwMU0RERETN8P9tmurV9DpCmQAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.ylabel(\"Learning rate\")\n", + "plt.xlabel(\"Step\")\n", + "plt.plot(range(total_training_steps), track_lrs);" + ] + }, + { + "cell_type": "markdown", + "id": "e7512808-b48d-4146-86a1-5931b1e3aec1", + "metadata": {}, + "source": [ + "## D.3 Gradient clipping" + ] + }, + { + "cell_type": "markdown", + "id": "c0a74f76-8d2b-4974-a03c-d645445cdc21", + "metadata": {}, + "source": [ + "- Gradient clipping is yet another technique used to stabilize the training when training LLMs\n", + "- By setting a threshold, gradients exceeding this limit are scaled down to a maximum magnitude to ensure that the updates to the model's parameters during backpropagation remain within a manageable range\n", + "- For instance, using the `max_norm=1.0` setting in PyTorch's `clip_grad_norm_` method means that the norm of the gradients is clipped such that their maximum norm does not exceed 1.0\n", + "- the \"norm\" refers to a measure of the gradient vector's length (or magnitude) in the parameter space of the model\n", + "- Specifically, it's the L2 norm, also known as the Euclidean norm\n", + "- Mathematically, for a vector $\\mathbf{v}$ with components $\\mathbf{v} = [v_1, v_2, \\ldots, v_n]$, the L2 norm is defined as:\n", + "$$\n", + "\\| \\mathbf{v} \\|_2 = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "d44838a6-4322-47b2-a935-c00d3a88355f", + "metadata": {}, + "source": [ + "- The L2 norm is calculated similarly for matrices.\n", + "- Let's assume our gradient matrix is:\n", + "$$\n", + "G = \\begin{bmatrix}\n", + "1 & 2 \\\\\n", + "2 & 4\n", + "\\end{bmatrix}\n", + "$$\n", + "\n", + "- And we want to clip these gradients with a `max_norm` of 1.\n", + "\n", + "- First, we calculate the L2 norm of these gradients:\n", + "$$\n", + "\\|G\\|_2 = \\sqrt{1^2 + 2^2 + 2^2 + 4^2} = \\sqrt{25} = 5\n", + "$$\n", + "\n", + "- Since $\\|G\\|_2 = 5$ is greater than our `max_norm` of 1, we need to scale down the gradients so that their norm is exactly 1. The scaling factor is calculated as $\\frac{max\\_norm}{\\|G\\|_2} = \\frac{1}{5}$.\n", + "\n", + "- Therefore, the scaled gradient matrix $G'$ will be as follows:\n", + "$$\n", + "G' = \\frac{1}{5} \\times G = \\begin{bmatrix}\n", + "\\frac{1}{5} & \\frac{2}{5} \\\\\n", + "\\frac{2}{5} & \\frac{4}{5}\n", + "\\end{bmatrix}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "eeb0c3c1-2cff-46f5-8127-24412184428c", + "metadata": {}, + "source": [ + "- Let's see this in action\n", + "- First, we initialize a new model and calculate the loss for a training batch like we would do in the regular training loop" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "e199e1ff-58c4-413a-855e-5edbe9292649", + "metadata": {}, + "outputs": [], + "source": [ + "from previous_chapters import calc_loss_batch\n", + "\n", + "torch.manual_seed(123)\n", + "model = GPTModel(GPT_CONFIG_124M)\n", + "\n", + "loss = calc_loss_batch(input_batch, target_batch, model, device)\n", + "loss.backward()" + ] + }, + { + "cell_type": "markdown", + "id": "76b60f3a-15ec-4846-838d-fdef3df99899", + "metadata": {}, + "source": [ + "- If we call `.backward()`, PyTorch will calculate the gradients and store them in a `.grad` attribute for each weight (parameter) matrix\n", + "- Let's define a utility function to calculate the highest gradient based on all model weights" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "e70729a3-24d1-411d-a002-2529cd3a8a9e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tensor(0.0373)\n" + ] + } + ], + "source": [ + "def find_highest_gradient(model):\n", + " max_grad = None\n", + " for param in model.parameters():\n", + " if param.grad is not None:\n", + " grad_values = param.grad.data.flatten()\n", + " max_grad_param = grad_values.max()\n", + " if max_grad is None or max_grad_param > max_grad:\n", + " max_grad = max_grad_param\n", + " return max_grad\n", + "\n", + "print(find_highest_gradient(model))" + ] + }, + { + "cell_type": "markdown", + "id": "734f30e6-6b24-4d4b-ae91-e9a4b871113f", + "metadata": {}, + "source": [ + "- Applying gradient clipping, we can see that the largest gradient is now substantially smaller:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "fa81ef8b-4280-400f-a93e-5210f3e62ff0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tensor(0.0166)\n" + ] + } + ], + "source": [ + "torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n", + "print(find_highest_gradient(model))" + ] + }, + { + "cell_type": "markdown", + "id": "b62c2af0-dac3-4742-be4b-4292c6753099", + "metadata": {}, + "source": [ + "## D.4 The modified training function" + ] + }, + { + "cell_type": "markdown", + "id": "76715332-94ec-4185-922a-75cb420819d5", + "metadata": {}, + "source": [ + "- Now let's add the three concepts covered above (learning rate warmup, cosine decay, and gradient clipping) to the `train_model_simple` function covered in chapter 5 to create the more sophisticated `train_model` function below:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "46eb9c84-a293-4016-a523-7ad726e171e9", + "metadata": {}, + "outputs": [], + "source": [ + "from previous_chapters import evaluate_model, generate_and_print_sample\n", + "\n", + "\n", + "def train_model(model, train_loader, val_loader, optimizer, device, n_epochs,\n", + " eval_freq, eval_iter, start_context, warmup_steps=10,\n", + " initial_lr=3e-05, min_lr=1e-6):\n", + "\n", + " train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []\n", + " tokens_seen, global_step = 0, -1\n", + "\n", + " # Retrieve the maximum learning rate from the optimizer\n", + " peak_lr = optimizer.param_groups[0][\"lr\"]\n", + "\n", + " # Calculate the total number of iterations in the training process\n", + " total_training_steps = len(train_loader) * n_epochs\n", + "\n", + " # Calculate the learning rate increment during the warmup phase\n", + " lr_increment = (peak_lr - initial_lr) / warmup_steps\n", + "\n", + " for epoch in range(n_epochs):\n", + " model.train()\n", + " for input_batch, target_batch in train_loader:\n", + " optimizer.zero_grad()\n", + " global_step += 1\n", + "\n", + " # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n", + " if global_step < warmup_steps:\n", + " # Linear warmup\n", + " lr = initial_lr + global_step * lr_increment \n", + " else:\n", + " # Cosine annealing after warmup\n", + " progress = ((global_step - warmup_steps) / \n", + " (total_training_steps - warmup_steps))\n", + " lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n", + "\n", + " # Apply the calculated learning rate to the optimizer\n", + " for param_group in optimizer.param_groups:\n", + " param_group[\"lr\"] = lr\n", + " track_lrs.append(lr) # Store the current learning rate\n", + "\n", + " # Calculate and backpropagate the loss\n", + " loss = calc_loss_batch(input_batch, target_batch, model, device)\n", + " loss.backward()\n", + "\n", + " # Apply gradient clipping after the warmup phase to avoid exploding gradients\n", + " if global_step > warmup_steps:\n", + " torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n", + " \n", + " optimizer.step()\n", + " tokens_seen += input_batch.numel()\n", + "\n", + " # Periodically evaluate the model on the training and validation sets\n", + " if global_step % eval_freq == 0:\n", + " train_loss, val_loss = evaluate_model(\n", + " model, train_loader, val_loader,\n", + " device, eval_iter\n", + " )\n", + " train_losses.append(train_loss)\n", + " val_losses.append(val_loss)\n", + " track_tokens_seen.append(tokens_seen)\n", + " # Print the current losses\n", + " print(f\"Ep {epoch+1} (Iter {global_step:06d}): \"\n", + " f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n", + "\n", + " # Generate and print a sample from the model to monitor progress\n", + " generate_and_print_sample(\n", + " model, train_loader.dataset.tokenizer,\n", + " device, start_context\n", + " )\n", + "\n", + " return train_losses, val_losses, track_tokens_seen, track_lrs" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "55fcd247-ba9d-4b93-a757-0f7ce04fee41", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Ep 1 (Iter 000000): Train loss 10.914, Val loss 10.940\n", + "Ep 1 (Iter 000005): Train loss 8.903, Val loss 9.313\n", + "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n", + "Ep 2 (Iter 000010): Train loss 7.362, Val loss 7.789\n", + "Ep 2 (Iter 000015): Train loss 6.273, Val loss 6.814\n", + "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n", + "Ep 3 (Iter 000020): Train loss 5.958, Val loss 6.609\n", + "Ep 3 (Iter 000025): Train loss 5.675, Val loss 6.592\n", + "Every effort moves you. \n", + "Ep 4 (Iter 000030): Train loss 5.607, Val loss 6.565\n", + "Ep 4 (Iter 000035): Train loss 5.063, Val loss 6.483\n", + "Every effort moves you, and, and the to the to the to the to the to the, and, and the, and the, and, and, and the, and the, and, and the, and, and, and the, and, and the\n", + "Ep 5 (Iter 000040): Train loss 4.384, Val loss 6.379\n", + "Every effort moves you, I was, and I had been. \"I, I had the picture, as a little's his pictures, I had been, I was his\n", + "Ep 6 (Iter 000045): Train loss 4.638, Val loss 6.306\n", + "Ep 6 (Iter 000050): Train loss 3.690, Val loss 6.196\n", + "Every effort moves you know the to me a little of his pictures--I had been. \"I was the's--and, I felt to see a little of his pictures--I had been. \"I of Jack's \"strong. \"I\n", + "Ep 7 (Iter 000055): Train loss 3.157, Val loss 6.148\n", + "Ep 7 (Iter 000060): Train loss 2.498, Val loss 6.157\n", + "Every effort moves you know it was not that, and he was to the fact of the of a and he was--his, the fact of the donkey, in the of the his head to have. \"I had been his pictures--and by his\n", + "Ep 8 (Iter 000065): Train loss 2.182, Val loss 6.178\n", + "Ep 8 (Iter 000070): Train loss 1.998, Val loss 6.193\n", + "Every effort moves you know,\" was not that my dear, his pictures--so handsome, in a so that he was a year after Jack's resolve had been his painting. \"Oh, I had the donkey. \"There were, with his\n", + "Ep 9 (Iter 000075): Train loss 1.824, Val loss 6.211\n", + "Ep 9 (Iter 000080): Train loss 1.742, Val loss 6.201\n", + "Every effort moves you know,\" was not that my hostess was \"interesting\": on that point I could have given Miss Croft the fact, and. \"Oh, as I turned, and down the room, in his\n", + "Ep 10 (Iter 000085): Train loss 1.285, Val loss 6.234\n", + "Every effort moves you?\" \"Yes--quite insensible to the fact with a little: \"Yes--and by me to me to have to see a smile behind his close grayish beard--as if he had the donkey. \"There were days when I\n", + "Ep 11 (Iter 000090): Train loss 1.256, Val loss 6.236\n", + "Ep 11 (Iter 000095): Train loss 0.803, Val loss 6.255\n", + "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", + "Ep 12 (Iter 000100): Train loss 0.731, Val loss 6.284\n", + "Ep 12 (Iter 000105): Train loss 0.889, Val loss 6.299\n", + "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", + "Ep 13 (Iter 000110): Train loss 0.703, Val loss 6.316\n", + "Ep 13 (Iter 000115): Train loss 0.517, Val loss 6.315\n", + "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", + "Ep 14 (Iter 000120): Train loss 0.594, Val loss 6.324\n", + "Ep 14 (Iter 000125): Train loss 0.481, Val loss 6.325\n", + "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", + "Ep 15 (Iter 000130): Train loss 0.529, Val loss 6.324\n", + "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n" + ] + } + ], + "source": [ + "torch.manual_seed(123)\n", + "model = GPTModel(GPT_CONFIG_124M)\n", + "model.to(device)\n", + "\n", + "peak_lr = 5e-4\n", + "optimizer = torch.optim.AdamW(model.parameters(), lr=peak_lr, weight_decay=0.1)\n", + "\n", + "n_epochs = 15\n", + "train_losses, val_losses, tokens_seen, lrs = train_model(\n", + " model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,\n", + " eval_freq=5, eval_iter=1, start_context=\"Every effort moves you\",\n", + " warmup_steps=10, initial_lr=1e-5, min_lr=1e-5\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "827e8d5e-0872-4b90-98ac-200c80ee2d53", + "metadata": {}, + "source": [ + "- Looking at the results above, we can see that the model starts out generating incomprehensible strings of words, whereas, towards the end, it's able to produce grammatically more or less correct sentences\n", + "- If we were to check a few passages it writes towards the end, we would find that they are contained in the training set verbatim -- it simply memorizes the training data\n", + "- Note that the overfitting here occurs because we have a very, very small training set, and we iterate over it so many times\n", + " - The LLM training here primarily serves educational purposes; we mainly want to see that the model can learn to produce coherent text\n", + " - Instead of spending weeks or months on training this model on vast amounts of expensive hardware, we load the pretrained weights" + ] + }, + { + "cell_type": "markdown", + "id": "9decec45-4fdf-4ff6-85a7-1806613f8af7", + "metadata": {}, + "source": [ + "- A quick check that the learning rate behaves as intended" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "d8ebb8d2-8308-4a83-a2a6-730c3bf84452", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAGdCAYAAADpBYyuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABWSElEQVR4nO3deVxU5f4H8M+ZGZhBVhEZQEHRcCc1EMIlK7lheUvKUonUzCvWldKsNCu1W90ozepa/iLrltV1yzIrXIrQNBVRwX3XTNxgUGQGULaZ5/eHzsgQLhjMmeXzfr3m5W3OM/A951XO5z7neb5HEkIIEBEREZGFQu4CiIiIiOwNAxIRERFRHQxIRERERHUwIBERERHVwYBEREREVAcDEhEREVEdDEhEREREdTAgEREREdWhkrsAe2IymXD69Gl4e3tDkiS5yyEiIqIbIIRAaWkpQkJCoFA0ztwPA1Itp0+fRmhoqNxlEBER0U04ceIEWrdu3Sg/iwGpFm9vbwCXLrCPj4/M1RAREdGNMBgMCA0NtXyPNwYGpFrMt9V8fHwYkIiIiBxMYy6P4SJtIiIiojoYkIiIiIjqYEAiIiIiqoMBiYiIiKgOBiQiIiKiOhiQiIiIiOpgQCIiIiKqgwGJiIiIqA4GJCIiIqI6biogzZ07F23btoVGo0FsbCy2bNlyzfFLly5Fp06doNFoEBkZiZUrV1odF0Jg+vTpCA4OhoeHB+Lj43H48GGrMcXFxUhOToaPjw/8/PwwZswYlJWVWY7/8ccfkCTpT6/NmzffzCkSERGRC2twQFqyZAkmTZqEGTNmIC8vD927d0dCQgJ0Ol294zdt2oSkpCSMGTMG27dvR2JiIhITE7Fnzx7LmJkzZ2LOnDlIT09HTk4OPD09kZCQgIqKCsuY5ORk7N27F5mZmcjIyMD69euRkpLyp9/3yy+/4MyZM5ZXVFRUQ0+RiIiIXJ1ooJiYGDF+/HjLPxuNRhESEiLS0tLqHT906FAxaNAgq/diY2PFuHHjhBBCmEwmERQUJGbNmmU5XlJSItRqtVi0aJEQQoh9+/YJAGLr1q2WMatWrRKSJIlTp04JIYQ4duyYACC2b9/e0FOy0Ov1AoDQ6/U3/TOIiIjItpri+7tBD6utqqpCbm4upk6danlPoVAgPj4e2dnZ9X4mOzsbkyZNsnovISEBy5cvBwAcO3YMBQUFiI+Ptxz39fVFbGwssrOzMXz4cGRnZ8PPzw/R0dGWMfHx8VAoFMjJycGDDz5oef+BBx5ARUUFOnTogMmTJ+OBBx646vlUVlaisrLS8s8Gg+HGLoSD2vpHMX7eWwA3pQJqlRJqNwXUqsv/W6WAj4cbAr3VaOmtRoCXGu4qLlEjIiLX1KCAdPbsWRiNRmi1Wqv3tVotDhw4UO9nCgoK6h1fUFBgOW5+71pjAgMDrQtXqeDv728Z4+XlhdmzZ6NPnz5QKBT49ttvkZiYiOXLl181JKWlpeFf//rXjZy6U3hh6U78ce7CDY9v3swNLS8HpjYtPNG+pRduCfRC+5aeCPH1gELReE9NJiIisicNCkj2LCAgwGqmqlevXjh9+jRmzZp11YA0depUq88YDAaEhoY2ea1yMJkETp6/CABIigmFJEmorDahssaIyhoTKqqN0F+sRlFpJYpKK1FjEjh/oRrnL1TjUGEZNh45Z/XzNG4KtAvwQoTWC91b+6FnmB+6hPhArVLKcXpERESNqkEBKSAgAEqlEoWFhVbvFxYWIigoqN7PBAUFXXO8+c/CwkIEBwdbjenRo4dlTN1F4DU1NSguLr7q7wWA2NhYZGZmXvW4Wq2GWq2+6nFncv5CFWpMApIEvDa4G9yUV799ZjKJS2GprBI6QyUKDBX442w5jujKcLSoDH+cK0dFtQn7zhiw74wB3+84DQBwVynQLcQHPcOao2eYH6Lb+CPIV2OrUyQiImo0DQpI7u7uiIqKQlZWFhITEwEAJpMJWVlZSE1NrfczcXFxyMrKwsSJEy3vZWZmIi4uDgAQHh6OoKAgZGVlWQKRwWBATk4OnnrqKcvPKCkpQW5urmVX2po1a2AymRAbG3vVenfs2GEVulxZoeHSWqsWnu7XDEcAoFBIaO7pjuae7uig9f7T8RqjCfnFF3C0qBz7zxiw40QJtuefx/kL1cjLL0FefollbEetN/p3bIn+HVoium1zzjAREZFDaPAttkmTJmHUqFGIjo5GTEwM3n//fZSXl2P06NEAgJEjR6JVq1ZIS0sDAEyYMAH9+/fH7NmzMWjQICxevBjbtm3DvHnzAACSJGHixIl44403EBERgfDwcEybNg0hISGWENa5c2cMHDgQY8eORXp6Oqqrq5Gamorhw4cjJCQEAPDFF1/A3d0dPXv2BAAsW7YMn332GT799NO/fJGcga70UsuEQO+/PqOjUirQrqUX2rX0wt+6XFo7JoTA8XMXsP3EeeQdL0Fe/nnsO2PAwcJSHCwsxbz1v8PDTYne7Vugf8eWGNBZi1Z+Hn+5FiIioqbQ4IA0bNgwFBUVYfr06SgoKECPHj2wevVqyyLr/Px8KBRXZih69+6NhQsX4pVXXsFLL72EiIgILF++HN26dbOMmTx5MsrLy5GSkoKSkhL07dsXq1evhkZz5ct8wYIFSE1NxYABA6BQKDBkyBDMmTPHqrbXX38dx48fh0qlQqdOnbBkyRI8/PDDDb4ozkh3eQZJ69M0txQlSULbAE+0DfDEgz1bAwDOl1dhw5GzWHeoCOsOFaGotBJZB3TIOqDD9O/3IqpNc/z91mAMigxGoA9vxRERkf2QhBBC7iLshcFggK+vL/R6PXx8fOQup1F9kHUYszMPYVh0KN5++Fab/34hBPafKcW6Q0VYe0CHrceLYf43T5KA2HB/3N89BPd2C4a/p7vN6yMiIsfVFN/fTrOLja5NV9q0M0jXI0kSuoT4oEuID566sz0KDRVYsesMMnadRl5+CTb/XozNvxdj+vd7MaBTIB6NDcMdES3ZSoCIiGTBgOQiCg2X1iC1tJNbWVofDZ7oG44n+obj5PkLWLHrDH7cdRp7Thnw875C/LyvEK2beyApJgyPRLdulLVTREREN4oByUUUmmeQvO2vrUHr5s0wrn97jOvfHocKS7EwJx/L8k7i5PmLmPXTQbyXeQh/66JFcmwb9LmlBSSJs0pERNS0GJBcRNHlGSStncwgXU0HrTdefaArpgzshBW7z2BhznHk5Zdg1Z4CrNpTgM7BPniyfzsMigyG6jrtCoiIiG4Wv2FcgMkkLGuQAmVag9RQHu5KPBzVGsv+2QerJvTDiNvboJm7EvvPGDBh8Q70n/UrPt94DBeqauQulYiInBADkgsortVFO8DLMQJSbZ2DffB6YjdsevFuPH9PB7TwdMepkov414/70PutNXg38xDOlVVe/wcRERHdIAYkF6CzdNFWX7eLtj3za+aO1LsjsPHFu/FGYje0adEMJReqMSfrMPrNXItZPx2A/kK13GUSEZETcNxvS7phhZYu2o43e1QfjZsSj93eBmueuxP/l3wbIlv54kKVEXPXHkW/mWvw4ZrDKK/krTciIrp5DEguoKiJu2jLRamQcF9kMH5I7YN5I6LQUesNQ0UN3vn5EO6YuRaf/vY7KqqNcpdJREQOiAHJBRQ6yA62myVJEu7pGoRVE/rhP8N7oG2LZjhXXoU3VuzHnbN+xZKt+TCa2DCeiIhuHAOSC3C2W2xXo1BIGNyjFTIn9cdbD0UixFeDAkMFpny7G4PnbsDWP4rlLpGIiBwEA5ILMC/SdpUHwropFRgeE4Y1z9+JVwZ1hrdGhT2nDHgkPRupC/NwquSi3CUSEZGdY0ByAeYu2s4+g1SXxk2Jf/Rrh1+fvxOPxoZBkoCMXWdw9zu/4t3MQ+yhREREV8WA5AJ0Tr4G6XpaeKnx5oORyHi6L2LD/VFZY8KcrMMYMHsdVu0+AyG4PomIiKwxIDk5k0mgyPwcNhcNSGZdQ3yxOOV2fJR8G1o398AZfQWeWpCHsV/m4jRvuxERUS0MSE7Ouou2u9zlyE6SJNwbGYxfJvXHM3ffAjelhF/2F+Jv767D/I3HuNuNiIgAMCA5PfMW/xaeaj7ctRaNmxKT7umIFc/0Q1Sb5iivMuLVH/fhoY82Yf8Zg9zlERGRzPiN6eR0pc7ZJLKxdNB6Y+m4OLyR2A3eahV2nijB3z/YgLdWHWCTSSIiF8aA5OTMC7RdbQdbQygUEh67vQ1+ea4/7u0WBKNJIH3dUdz/wQbsOaWXuzwiIpIBA5KT0xm4QPtGaX00+OixKMwbEYUALzUO68qQOHcj5mQdRo3RJHd5RERkQwxITs7SRZsB6Ybd0zUIPz97B+6LDEKNSeDdzEMY8tEmHNGVyV0aERHZCAOSkys0uGaTyL/K39Mdcx+9Df8Z3gM+GhV2ntRj0Jzf8NmGYzBxpxsRkdNjQHJyOvZAummSdOnZbj89ewf6RQSgssaE1zL24bH/5lh2BxIRkXNiQHJyV7pocwbpZgX7euDLJ2LwemI3eLgpsenoOdz3n9/w60Gd3KUREVETYUByYrW7aAd6cwbpr5AkCSNub4OMZ/qic7APzpVX4fHPtyJt5X5UcwE3EZHTYUByYuyi3fjat/TCd//sjZFxbQAAH6//HY+kZ+NE8QWZKyMiosbEgOTE2EW7aWjclHhtcDekPxYFH40KO06U4L45v2Hl7jNyl0ZERI2E35pO7EoPJK4/agoDuwVh5YR+uC3MD6UVNfjngjxM/34Pqmp4y42IyNExIDkxXal5gTbXHzWV1s2bYcm4OPzzzvYAgC+zjyPpk83c5UZE5OAYkJwYeyDZhptSgckDO+G/o6LhrVEh9/h5/P2DDdhyrFju0oiI6CYxIDkx8ywGu2jbxoDOWvyY2hcdtd4oKq3Eo59sxvyNxyAEG0sSETkaBiQndqVJJGeQbKVtgCe+G98b93cPQY1J4NUf9+HZJTtwscood2lERNQADEhOzNwkkj2QbKuZuwpzhvfAtL93gVIhYfmO03joo01sBUBE5EAYkJwYZ5DkI0kSxvQNx4J/xCLAyx37zxgweO5GbP2D65KIiBwBA5KTMpkEn8NmB25v1wI/Pt0X3Vr5oLi8Csmf5OCb3JNyl0VERNfBgOSkzpVXwXi5i3YLT3bRllOwrwe+HheHe7sFocpowvNLdyJt1X4YTVy8TURkrxiQnJS5B1KAF7to24Nm7irMffQ2PH33LQCAj9f9jnFf5aKsskbmyoiIqD785nRSOvZAsjsKhYTn7umI/wzvAXeVAr/sL8TDH23CyfNcvE1EZG8YkJyUuQcS1x/Zn8E9WmFJyu0I8FLjQEEpEuduxM4TJXKXRUREtTAgOSnuYLNvPcOa4/vUPugc7IOzZVUYPm8zsvYXyl0WERFdxoDkpMwzSC3ZA8lutfLzwNIn49AvIgAXq40Y++U2LMg5LndZREQEBiSnZX4OG2eQ7JuXWoXPHu+FR6JawySAl7/bg3d+OsjHkxARyYwByUkVXd7FpuUMkt1zUyow8+FbMWFABADgw7VH8NzSnaiqMclcGRGR62JAclLmGaRAziA5BEmS8OzfOuDtIZFQKiQsyzuFJ+ZvhaGiWu7SiIhcEgOSEzKZBIrK2EXbEQ3rFYZPR0WjmbsSG46cxbCPN1t6WhERke0wIDkhcxdtBbtoO6S7OgZiSUocArzU2H/GgEfSs/mgWyIiG2NAckLmHWwt2EXbYUW29sU3T8ahdXMPHD93AQ+nb8LBglK5yyIichn89nRCReyB5BTaBnji26d6o4PWC4WGSgz9OBt5+eflLouIyCUwIDkh8wxSIHewOTytjwZfj4tDzzA/6C9WI/mTHPx2uEjusoiInB4DkhNiDyTn4tfMHQv+EWtpKPnE/K1YseuM3GURETk1BiQnZN71xBkk59HMXYVPR0VjUGQwqo0CqYvysHhLvtxlERE5LQYkJ8QeSM5JrVJiTlJPPBobBiGAF5ftxheb/pC7LCIip8SA5IR07KLttJQKCf9O7Iax/cIBADN+2It564/KXBURkfNhQHJCOgObRDozSZLw0n2d8fTdtwAA3lx5AB9kHZa5KiIi58KA5GSMtbpo8xab85IkCc/d0xHP/a0DAGB25iHM+ukAH3JLRNRIGJCcTDG7aLuUpwdE4OX7OgMA5q49in+v2M+QRETUCBiQnIy5B1IAu2i7jLF3tMNrg7sCAD7dcAzTv98Lk4khiYjor+A3qJOxbPHn7TWXMjKuLd4eEglJAr7afByvfL+HIYmI6C+4qYA0d+5ctG3bFhqNBrGxsdiyZcs1xy9duhSdOnWCRqNBZGQkVq5caXVcCIHp06cjODgYHh4eiI+Px+HD1otOi4uLkZycDB8fH/j5+WHMmDEoKyur9/cdOXIE3t7e8PPzu5nTc2iWBdrcweZyhvUKw7tDu0MhAQtz8jHt+z283UZEdJMaHJCWLFmCSZMmYcaMGcjLy0P37t2RkJAAnU5X7/hNmzYhKSkJY8aMwfbt25GYmIjExETs2bPHMmbmzJmYM2cO0tPTkZOTA09PTyQkJKCiosIyJjk5GXv37kVmZiYyMjKwfv16pKSk/On3VVdXIykpCf369WvoqTkF9kBybQ/2bI13HukOSQIWMCQREd080UAxMTFi/Pjxln82Go0iJCREpKWl1Tt+6NChYtCgQVbvxcbGinHjxgkhhDCZTCIoKEjMmjXLcrykpESo1WqxaNEiIYQQ+/btEwDE1q1bLWNWrVolJEkSp06dsvrZkydPFo899pj4/PPPha+vb4POTa/XCwBCr9c36HP2ZOqyXaLNlAzx7s8H5S6FZPTNthOi7YsZos2UDPHKd7uFyWSSuyQioibTFN/fDZpBqqqqQm5uLuLj4y3vKRQKxMfHIzs7u97PZGdnW40HgISEBMv4Y8eOoaCgwGqMr68vYmNjLWOys7Ph5+eH6Ohoy5j4+HgoFArk5ORY3luzZg2WLl2KuXPn3tD5VFZWwmAwWL0cHXsgEQAMiWqNWQ93t6xJevWHvZxJIiJqgAYFpLNnz8JoNEKr1Vq9r9VqUVBQUO9nCgoKrjne/Of1xgQGBlodV6lU8Pf3t4w5d+4cHn/8ccyfPx8+Pj43dD5paWnw9fW1vEJDQ2/oc/bsynPYeIvN1T0c1Rozh9wKSQK+yD6Of/24jyGJiOgGOc0utrFjx+LRRx/FHXfcccOfmTp1KvR6veV14sSJJqzQNszb/DmDRADwSHQo3n7oUkiav+kPvJbBkEREdCMaFJACAgKgVCpRWFho9X5hYSGCgoLq/UxQUNA1x5v/vN6YuovAa2pqUFxcbBmzZs0avPPOO1CpVFCpVBgzZgz0ej1UKhU+++yzemtTq9Xw8fGxejkyo0ngbFkVAEDLRdp02dBel0ISAHy+8Q+8vfogQxIR0XU0KCC5u7sjKioKWVlZlvdMJhOysrIQFxdX72fi4uKsxgNAZmamZXx4eDiCgoKsxhgMBuTk5FjGxMXFoaSkBLm5uZYxa9asgclkQmxsLIBL65R27Nhheb322mvw9vbGjh078OCDDzbkNB3WufLKK120vRiQ6IqhvULx5oORAID0dUfxwZojMldERGTfVA39wKRJkzBq1ChER0cjJiYG77//PsrLyzF69GgAwMiRI9GqVSukpaUBACZMmID+/ftj9uzZGDRoEBYvXoxt27Zh3rx5AC49U2rixIl44403EBERgfDwcEybNg0hISFITEwEAHTu3BkDBw7E2LFjkZ6ejurqaqSmpmL48OEICQmxjKlt27ZtUCgU6Nat201fHEdjXqAd4KWGUiHJXA3Zm0djw3Cx2ojXM/bh3cxD0LgpkHJHe7nLIiKySw0OSMOGDUNRURGmT5+OgoIC9OjRA6tXr7Ysss7Pz4dCcWViqnfv3li4cCFeeeUVvPTSS4iIiMDy5cutgsvkyZNRXl6OlJQUlJSUoG/fvli9ejU0mivraBYsWIDU1FQMGDAACoUCQ4YMwZw5c/7KuTsd8wJtrj+iqxnTNxwV1UbM+ukg3lx5AB5uSoyIayt3WUREdkcSXIxgYTAY4OvrC71e75DrkRZtycfUZbsxoFMg/vt4L7nLITv2zk8H8eHaS7fZZj58K4ZGO/4OTiJyXU3x/e00u9joyi22QM4g0XU8d08HPNEnHADw4re78MPO0zJXRERkXxiQnEgheyDRDZIkCdP+3hmPxobBJIBnl+zAz3vr72VGROSKGJCciI49kKgBJEnCG4O74aGerWA0CaQu3I5NR8/KXRYRkV1gQHIiulLzY0Y4g0Q3RqGQMPPhW5HQVYsqowljv9iGXSdL5C6LiEh2DEhOxNxFO9CbM0h041RKBf4zvCfi2rVAeZURj3++FUeLyuQui4hIVgxITsJoEijiDBLdJI2bEvNGRiGylS+Ky6sw4tMcnC65KHdZRESyYUByEufKK2ESYBdtumneGjfMH90L7Vp64rS+AiP+m4Pi8iq5yyIikgUDkpNgF21qDC281PhqTCyCfTU4WlSO0Z9vQVlljdxlERHZHAOSkyjkDjZqJK38PPDVmBg0b+aGnSf1GPfVNlTWGOUui4jIphiQnAR3sFFjuiXQG/NHx8DTXYmNR85h4uIdMJrYdJ+IXAcDkpMwzyC15A42aiTdQ/0wb2Q03JUKrNpTgJe/2w0+mYiIXAUDkpPgDBI1hT63BGBOUg8oJGDx1hOY+dNBuUsiIrIJBiQnwS7a1FQGdgvGmw9GAgA++vUo5q0/KnNFRERNjwHJSRSaH1TL57BRExgeE4YpAzsBAN5ceQDf7zglc0VERE2LAclJ6Eo5g0RN68n+7fBEn3AAwPNLd2LTET63jYicFwOSE6jdRZszSNRUJEnCK4M6Y1BkMKqNAuO+ysX+Mwa5yyIiahIMSE7gXBm7aJNtKBQSZg/tjphwf5RW1mD051v5SBIickoMSE7AvIOtpTe7aFPT07gp8cmIaEQEeqHAUIHHP98C/cVqucsiImpUDEhOwNwDKZA9kMhGfJu5Yf4TMdD6qHGosAwpX7LbNhE5FwYkJ2DewcYeSGRLrfw88PnjMfBSq5BzrBjPfb0TJnbbJiInwYDkBMw72AK5g41srEuIDz4eEQU3pYSMXWeQtmq/3CURETUKBiQnwB5IJKc+twRg1sPdAQCf/HYMn204JnNFRER/HQOSEyhiDySSWWLPVpZGkq+v2IeVu8/IXBER0V/DgOQEuAaJ7MGT/dthxO1tIAQwcckObDlWLHdJREQ3jQHJCXAXG9kDSZLw6gNdcU8XLapqTBj75TYcLSqTuywiopvCgOTgjCaBs2WX1yBxBolkplRImJPUEz3D/KC/WI0n5m9FcXmV3GURETUYA5KDs+qi7cmARPLTuCnxychohPp74Pi5C0j5chsqqtkjiYgcCwOSgzOvP2IXbbInAV5qfP54L3hrVNh2/Dwmf7MLQrBHEhE5DgYkB6fjDjayU7cEeuPjx6KgUkj4YedpvJd5SO6SiIhuGAOSg2MPJLJnvW8JwJsPRQIA5qw5gm9yT8pcERHRjWFAcnCWHWycQSI7NTQ6FP+8sz0AYOqyXcg+ek7mioiIro8BycHpSi/3QOIWf7Jjz9/TEYNuDUa1UeDJ/+Vy+z8R2T0GJAens8wg8RYb2S+FQsLsR7pbtv+P/nwrzl1uT0FEZI8YkBycZQaJAYnsXO3t//nFF5DyVS63/xOR3WJAcnDsok2OpPb2/9zj5/HCN7tgMnH7PxHZHwYkB8Yu2uSIam///3Hnabz3C7f/E5H9YUByYOYu2kqFxC7a5FBqb///YM0RfLed2/+JyL4wIDkwcw+kAC93dtEmhzM0OhRPXd7+P+Wb3cg9fl7mioiIrmBAcmDm9Ufsok2O6oV7OuKeLlpUGU0Y99U2nDx/Qe6SiIgAMCA5NPMONi7QJkelUEh4b1gPdA72wdmyKvzji20or6yRuywiIgYkR1bIHkjkBDzVKnw6KhoBXmocKCjFxCU7uLONiGTHgOTALA+q5QwSObhWfh74eEQU3FUKZO4rxKyfD8pdEhG5OAYkB6YzsEkkOY+oNs0xc8itAICPfj2Kb/lgWyKSEQOSAyss5S02ci6JPVsh9a5bAABTl+1G7vFimSsiIlfFgOTAzDNIXKRNzmTS3zogoeulnW0pX+ZyZxsRyYIByUHVGE2WLtrc5k/OxLyzrUuwD86VX9rZVsadbURkYwxIDupceVWtLtrucpdD1KiauV/a2dbS+/LOtsXbYeTONiKyIQYkB2W+vdbSSw0Fu2iTEwrx88C8yzvbftmvw8zVB+QuiYhcCAOSg2IPJHIFPcOaY9bDl3a2fbz+d+5sIyKbYUByUJYdbFygTU5ucI9aO9u+240dJ0rkLYiIXAIDkoNiDyRyJZP+1gHxnbWoqjEh5cttlhlUIqKmwoDkoHScQSIXcmlnW3dEBHpBV1qJlK9yUVFtlLssInJiDEgOqpAzSORivDVu+HRUNHw93LDzRAle/m4PhODONiJqGgxIDsryHDb2QCIX0qaFJ+Y+ehuUCgnf5p3Efzcck7skInJSDEgOyjyD1NKbM0jkWvpGBODl+zoDAN5cuR/rDxXJXBEROSMGJAdUYzThHLtokwsb3actHolqDZMAUhfm4djZcrlLIiInw4DkgNhFm1ydJEl448FuuC3MD4aKGoz9chtKK6rlLouInMhNBaS5c+eibdu20Gg0iI2NxZYtW645funSpejUqRM0Gg0iIyOxcuVKq+NCCEyfPh3BwcHw8PBAfHw8Dh8+bDWmuLgYycnJ8PHxgZ+fH8aMGYOysjLL8YMHD+Kuu+6CVquFRqNBu3bt8Morr6C62vn+0jRvcWYXbXJlapUS6Y9FIchHgyO6MkxcvIOPIyGiRtPggLRkyRJMmjQJM2bMQF5eHrp3746EhATodLp6x2/atAlJSUkYM2YMtm/fjsTERCQmJmLPnj2WMTNnzsScOXOQnp6OnJwceHp6IiEhARUVV3qdJCcnY+/evcjMzERGRgbWr1+PlJQUy3E3NzeMHDkSP//8Mw4ePIj3338fn3zyCWbMmNHQU7R77IFEdEmgjwYfX34cSdYBHWb/fFDukojIWYgGiomJEePHj7f8s9FoFCEhISItLa3e8UOHDhWDBg2yei82NlaMGzdOCCGEyWQSQUFBYtasWZbjJSUlQq1Wi0WLFgkhhNi3b58AILZu3WoZs2rVKiFJkjh16tRVa3322WdF3759b/jc9Hq9ACD0ev0Nf0YO/9v8h2gzJUOMmb/1+oOJXMB3eSdFmykZos2UDPH9jqv/nUBEzqkpvr8bNINUVVWF3NxcxMfHW95TKBSIj49HdnZ2vZ/Jzs62Gg8ACQkJlvHHjh1DQUGB1RhfX1/ExsZaxmRnZ8PPzw/R0dGWMfHx8VAoFMjJyan39x45cgSrV69G//79G3KKDoE9kIisJfZshXF3tAMATP5mJ/ac0stcERE5ugYFpLNnz8JoNEKr1Vq9r9VqUVBQUO9nCgoKrjne/Of1xgQGBlodV6lU8Pf3/9Pv7d27NzQaDSIiItCvXz+89tprVz2fyspKGAwGq5cjKGIPJKI/mTywE+7s2BIV1ZceR1JUWil3SUTkwJxuF9uSJUuQl5eHhQsXYsWKFXjnnXeuOjYtLQ2+vr6WV2hoqA0rvXnmGaRA9kAislAqJPxneE+0a+mJ0/oKPPW/XFTVmOQui4gcVIMCUkBAAJRKJQoLC63eLywsRFBQUL2fCQoKuuZ485/XG1N3EXhNTQ2Ki4v/9HtDQ0PRpUsXJCUl4a233sKrr74Ko7H+ZzZNnToVer3e8jpx4sS1Tt9umHexcQaJyJqvhxs+GRkNb40K246fx4wf+DgSIro5DQpI7u7uiIqKQlZWluU9k8mErKwsxMXF1fuZuLg4q/EAkJmZaRkfHh6OoKAgqzEGgwE5OTmWMXFxcSgpKUFubq5lzJo1a2AymRAbG3vVek0mE6qrq2Ey1f//ItVqNXx8fKxejkB3+dZBINcgEf1J+5ZemJPUE5IELNpyAl9tPi53SUTkgFQN/cCkSZMwatQoREdHIyYmBu+//z7Ky8sxevRoAMDIkSPRqlUrpKWlAQAmTJiA/v37Y/bs2Rg0aBAWL16Mbdu2Yd68eQAuNXybOHEi3njjDURERCA8PBzTpk1DSEgIEhMTAQCdO3fGwIEDMXbsWKSnp6O6uhqpqakYPnw4QkJCAAALFiyAm5sbIiMjoVarsW3bNkydOhXDhg2Dm5tbY1wru1BjNOFsmfkWG2eQiOpzV8dAvDiwE9JWHcBrP+5DR603Ytu1kLssInIgDQ5Iw4YNQ1FREaZPn46CggL06NEDq1evtiyyzs/Ph0JxZWKqd+/eWLhwIV555RW89NJLiIiIwPLly9GtWzfLmMmTJ6O8vBwpKSkoKSlB3759sXr1amg0VwLAggULkJqaigEDBkChUGDIkCGYM2fOlRNRqfD222/j0KFDEEKgTZs2SE1NxbPPPntTF8ZenS2rgmAXbaLrSrmjHfaeNuCHnacxfmEefkjtixA/D7nLIiIHIQneoLcwGAzw9fWFXq+329ttu06W4IEPNyLYV4PsqQPkLofIrl2sMuKhjzZh/xkDurf2xZJxcdC4KeUui4gaWVN8fzvdLjZnxx1sRDfOw12JeSOi4NfMDTtP6vHKci7aJqIbw4DkYHSXeyAFcgcb0Q0J9W+GD5Nug0ICvsk9iS+zuWibiK6PAcnBcAaJqOH6RgRg6r2dAQCvZ+xDzu/nZK6IiOwdA5KD0bEHEtFN+Ue/cAzuEYIak8A/F+ThdMlFuUsiIjvGgORgzD2Q+Bw2ooaRJAlvPXQrugT74Fx5FZ78Xy4qqutvIktExIDkYMxdtNkDiajhPNyV+HhEFJo3c8Ouk3q8/B0XbRNR/RiQHIxlDRJnkIhuSqh/M3z46KVF29/mncQXm/6QuyQiskMMSA6kxmjCuXLzLTbOIBHdrD63BOCl+y4v2l6xH5u5aJuI6mBAciDmLtoqhQT/ZuyiTfRXjOkbjsQeITCaBMYvyMMpLtomoloYkByIef1RS281FApJ5mqIHJskSUirvWj7Ky7aJqIrGJAciHkHG5tEEjWO2ou2d5/S46XvdnPRNhEBYEByKFd2sHGBNlFjCfVvhrmP3galQsKyvFOYz0XbRAQGJIfCHkhETaP3LQGYem8nAMAbK/Yj+ygXbRO5OgYkB2Lpos0eSESNzmrR9kIu2iZydQxIDsRyi40zSESNzrxou2uID4rLqzDuq21ctE3kwhiQHAgXaRM1LfOibX9Pd+w5ZcBLy7hom8hVMSA5EEsXbS7SJmoyrZs3w4eP9ry0aHv7KXy+8Q+5SyIiGTAgOQh20Saynd7tr3Ta/vfK/dh09KzMFRGRrTEgOQh20SayrSf6tMWDPVvBaBJIXbgdJ89fkLskIrIhBiQHwS7aRLZ1adF2JLq1urRo+8n/sdM2kSthQHIQV3aw8fYaka1o3JT4eES0ZdH2VC7aJnIZDEgOwtIkkgu0iWyqlZ+HpdP2d9tP4TMu2iZyCQxIDkLHHkhEsolr3wIvX160/SYXbRO5BAYkB3FlBom32IjkMLpPWzzERdtELoMByUGY1yBxiz+RPCRJwpu1Fm2P+yoXF6u4aJvIWTEgOQhzk8iWvMVGJJvai7b3njZg6rJdXLRN5KQYkBwEb7ER2Yfai7aX7zjNRdtETooByQFU1+qizUXaRPLjom0i58eA5ADOllWyizaRnRndpy0euo2LtomcFQOSA9DVekgtu2gT2QdJkvDmg5GIbOXLRdtETogByQFYHjPCHWxEdkXjpkT6iCgu2iZyQgxIDqCQXbSJ7FbdRdv/3XBM7pKIqBEwIDmAIvZAIrJrtRdtp606gE1HuGibyNExIDmAwlprkIjIPtVetD1+YR5OFHPRNpEjY0ByALpSziAR2bvai7bPX6jmom0iB8eA5AAsM0jsgURk18yLtlt4umPfGS7aJnJkDEgOwDyDFMgu2kR2r5WfBz7kom0ih8eAZOcuddGuAgBoOYNE5BDi2rfAK4O4aJvIkTEg2bnaXbSbs4s2kcN4vDcXbRM5MgYkO1fILtpEDomLtokcGwOSndNd7oEUyB1sRA6n7qLtF7lom8hhMCDZOXMXbfZAInJMtRdtf89F20QOgwHJzunYRZvI4dVetP3myv3YyEXbRHaPAcnO6S6vQeIONiLHZl60bRJAKhdtE9k9BiQ7V8geSEROgYu2iRwLA5KdYxdtIuehcVPiYy7aJnIIDEh2rojPYSNyKiF+HpibzEXbRPaOAcmOVRtNOFt2qYs2d7EROY/b23HRNpG9Y0CyY2fLLt1ec1OyizaRs+GibSL7xoBkx8zrj1p6sYs2kbPhom0i+8aAZMcK2UWbyKnVXbQ95Vsu2iayFwxIdkxXyh5IRM7OvGhbpZDww87T+OS33+UuiYjAgGTXLM9hYw8kIqd2e7sWmPb3LgCAt1YdwK8HdTJXREQMSHas0PKYEc4gETm7kXFtMCw6FCYBPL1oO44WlcldEpFLY0CyY+ZbbFyDROT8JEnCa4ldEdWmOUorajD2y20wVFTLXRaRy2JAsmOWLtrsgUTkEtQqJdIfi0Kwrwa/F5VjwqLtMJq4aJtIDgxIdkxnYBdtIlfT0luNeSOioVYpsPZgEWb9dFDukohcEgOSnao2mnCu/FIXbQYkItcS2doXMx++FQCQvu4ovt9xSuaKiFzPTQWkuXPnom3bttBoNIiNjcWWLVuuOX7p0qXo1KkTNBoNIiMjsXLlSqvjQghMnz4dwcHB8PDwQHx8PA4fPmw1pri4GMnJyfDx8YGfnx/GjBmDsrIrixh//fVXDB48GMHBwfD09ESPHj2wYMGCmzk9u1BUWruLtpvM1RCRrQ3u0QpP9m8PAJj8zS7sOlkib0FELqbBAWnJkiWYNGkSZsyYgby8PHTv3h0JCQnQ6erflrpp0yYkJSVhzJgx2L59OxITE5GYmIg9e/ZYxsycORNz5sxBeno6cnJy4OnpiYSEBFRUVFjGJCcnY+/evcjMzERGRgbWr1+PlJQUq99z66234ttvv8WuXbswevRojBw5EhkZGQ09RbtgWaDtrYEksYs2kSt6IaEj7u4UiMoaE1K+zIWutOL6HyKiRiGJBrZtjY2NRa9evfDhhx8CAEwmE0JDQ/H000/jxRdf/NP4YcOGoby83Cqo3H777ejRowfS09MhhEBISAiee+45PP/88wAAvV4PrVaL+fPnY/jw4di/fz+6dOmCrVu3Ijo6GgCwevVq3HfffTh58iRCQkLqrXXQoEHQarX47LPPbujcDAYDfH19odfr4ePj05DL0uh+2luAcV/lokeoH5aP7yNrLUQkH0NFNR6cuxFHi8pxW5gfFqXcDrVKKXdZRHalKb6/GzSDVFVVhdzcXMTHx1/5AQoF4uPjkZ2dXe9nsrOzrcYDQEJCgmX8sWPHUFBQYDXG19cXsbGxljHZ2dnw8/OzhCMAiI+Ph0KhQE5OzlXr1ev18Pf3v+rxyspKGAwGq5e90LEHEhEB8NG44ZOR0fDWqJCXX4Lpy/fycSRENtCggHT27FkYjUZotVqr97VaLQoKCur9TEFBwTXHm/+83pjAwECr4yqVCv7+/lf9vV9//TW2bt2K0aNHX/V80tLS4Ovra3mFhoZedaytXXnMCBdoE7m6di298OGjt0EhAUu2ncAXm/6QuyQip+eUu9jWrl2L0aNH45NPPkHXrl2vOm7q1KnQ6/WW14kTJ2xY5bVZHlTLHkhEBKB/h5aYem9nAMDrK/Zj45GzMldE5NwaFJACAgKgVCpRWFho9X5hYSGCgoLq/UxQUNA1x5v/vN6YuovAa2pqUFxc/Kffu27dOtx///147733MHLkyGuej1qtho+Pj9XLXliaRHIGiYgu+0e/cDzYsxWMJoHxC/OQf+6C3CUROa0GBSR3d3dERUUhKyvL8p7JZEJWVhbi4uLq/UxcXJzVeADIzMy0jA8PD0dQUJDVGIPBgJycHMuYuLg4lJSUIDc31zJmzZo1MJlMiI2Ntbz366+/YtCgQXj77betdrg5It5iI6K6JElC2kOR6N7aFyUXqvGPL7eirLJG7rKInFKDb7FNmjQJn3zyCb744gvs378fTz31FMrLyy1rfUaOHImpU6daxk+YMAGrV6/G7NmzceDAAbz66qvYtm0bUlNTAVz6D37ixIl444038MMPP2D37t0YOXIkQkJCkJiYCADo3LkzBg4ciLFjx2LLli3YuHEjUlNTMXz4cMsOtrVr12LQoEF45plnMGTIEBQUFKCgoADFxcV/9RrJQsdbbERUD42bEh+PiEagtxqHCsvw7JIdMPFxJESNT9yEDz74QISFhQl3d3cRExMjNm/ebDnWv39/MWrUKKvxX3/9tejQoYNwd3cXXbt2FStWrLA6bjKZxLRp04RWqxVqtVoMGDBAHDx40GrMuXPnRFJSkvDy8hI+Pj5i9OjRorS01HJ81KhRAsCfXv3797/h89Lr9QKA0Ov1N34xmkBltVG0mZIh2kzJEOfKKmWthYjsU+7xYhHx0krRZkqGmP3TAbnLIZJVU3x/N7gPkjOzlz5Ip0suovdba+CmlHDojXvZKJKI6vVN7kk8v3QnAGBOUk880L3+nnBEzk72PkhkG1d2sLGLNhFd3cNRrZFyRzsAwAtLd2LHiRJ5CyJyIgxIdsjymBE2iSSi65gysBMGXH4cydgvt+GM/qLcJRE5BQYkO2Tpou3NHWxEdG1KhYT3h/dAR603ikorMfbLbbhQxZ1tRH8VA5IdutIDiTNIRHR93ho3fDoqGi083bHnlAHPfb2TO9uI/iIGJDtkfmI3eyAR0Y0K9W+G9BFRcFcqsGpPAd775ZDcJRE5NAYkO2SeQWrJHkhE1AC92vrjzYciAQAfrDmC73eckrkiIsfFgGSHzLvYOINERA31cFRrjOt/eWfbN7uwPf+8zBUROSYGJDtUZHnMCGeQiKjhJid0QnxnLapqTBj7ZS5Ol3BnG1FDMSDZmaoaE86VVwG41AeJiKihzDvbOgV542xZJf7xBXe2ETUUA5KdKSq7NHvkppTQvJmbzNUQkaPyUqvw6ahoBHi5Y98ZA5/ZRtRADEh2Rscu2kTUSFo3b4aPR0TDXanAT3sL8c7PB+UuichhMCDZGfZAIqLGFNWmOd4acmln2//9ehRLt52QuSIix8CAZGeKStlFm4ga10O3tUbqXbcAAKYu241NR8/KXBGR/WNAsjPmGSTuYCOixjTpbx1wf/cQ1JgEnvwqF0d0ZXKXRGTXGJDsjLkHUiB7IBFRI1IoJMx6+FZEtWkOQ0UNRs/fgnOXN4UQ0Z8xINkZ3eUeSIHsok1EjUzjpsS8EVEI82+GE8UXMfbLbaioNspdFpFdYkCyM5xBIqKm1MJLjc8e7wUfjQp5+SV4fikfbEtUHwYkO6NjF20iamK3BHohfUQUVAoJGbvO4N1MPtiWqC4GJDtSVWNC8eUu2tzFRkRNqXf7AKRdfrDth2uP4Gtu/yeywoBkR8xdtN2VCvixizYRNbFHokMt2/9fWrYbm45w+z+RGQOSHTGvP2rprWYXbSKyCavt///j9n8iMwYkO6JjDyQisjFu/yeqHwOSHdGVXnkOGxGRrdTd/v8Pbv8nYkCyJ+ZbbJxBIiJba+GlxuejL23/355fgmcWbYeR2//JhTEg2RGd5UG1nEEiIttr39ILn4yMhrtSgZ/3FeJfP+6FEAxJ5JoYkOxIIbtoE5HMYtu1wHvDekCSgC+zjyN93e9yl0QkCwYkO6Kz3GLjDBIRyWfQrcF4ZVAXAMDbqw9g+fZTMldEZHsMSHbE8hw2rkEiIpmN6RuOf/QNBwC88M1ObGSPJHIxDEh2gl20icjevHRfZ/z91mBUGwXGfZWLfacNcpdEZDMMSHaCXbSJyN4oFBJmD+2O2HB/lFVe6pF0quSi3GUR2QQDkp1gF20iskdqlRLzRkSjg9YLhYZKjPpsC/QXquUui6jJMSDZCR17IBGRnfJt5ob5o2Og9VHjiK4MY79iI0lyfgxIdsK8QJs72IjIHoX4eWD+6Bh4q1XYcqwYzy3dCRMbSZITY0CyE+ZbbOyBRET2qnOwDz4eEQU3pYQVu87gjRX72UiSnBYDkp0oZBdtInIAvW8JwDuPdAcAfLbxGBtJktNiQLITvMVGRI5icI9WePm+zgAuNZJctCVf5oqIGh8Dkp3Q8RYbETmQsXe0w1N3tgcAvPzdbqzafUbmiogaFwOSneAMEhE5mskJHTG8VyhMApiweAe7bZNTYUCyA5U1RksXbc4gEZGjkCQJ/34wEgO7BqHKaELKl9uw80SJ3GURNQoGJDtQVMou2kTkmJQKCf9J6oE+t7RAeZURj3++BUd0ZXKXRfSXMSDZgdoPqWUXbSJyNGqVEh+PiEb31r44f6EaI/6bw0eSkMNjQLIDXKBNRI7OS63C56Nj0L6lJ87oKzDivzk4d/kZk0SOiAHJDph7IHGBNhE5Mn9Pd3w1JhYhvhr8XlSOxz/firLKGrnLIropDEh2QFdqfg4bAxIRObYQPw98OSYW/p7u2H1Kj5Qv+dw2ckwMSHbAPIPUkrfYiMgJ3BLohfmje8HTXYlNR89hwuLtqDGa5C6LqEEYkOyA+TlsnEEiImdxa2s/fDIyGu5KBX7aW4jJ3+ziw23JoTAg2YEiS5NIziARkfPofUsAPny0J5QKCcu2n8LLy/fw4bbkMBiQ7EChZRcbZ5CIyLnc0zUI7w3rAUkCFm3Jx+sZ+xmSyCEwIMmsssaI8xeqAXAGiYic0wPdQ/D2kFsBAJ9tPIbZPx+SuSKi62NAkpmli7ZKAV8PdtEmIuc0NDoUrw/uCgD4cO0RfLjmsMwVEV0bA5LMzDvYAr3ZRZuInNuIuLZ46b5OAIB3fj6ET3/7XeaKiK6OAUlmReyBREQuJOWO9pgYHwEAeGPFfvxv83GZKyKqHwOSzGrPIBERuYIJAyIwrn87AMAry/dg8ZZ8mSsi+jMGJJmxBxIRuRpJkvDiwE54vHdbAMCLy3bj660n5C2KqA4GJJnpLi/SDuQONiJyIZIkYcb9XSwhacqyXfh6G0MS2Q8GJJmxBxIRuSpzSBoV1wZCAFO+3YWlDElkJxiQZKYzsIs2EbkuSZLw6gNdMfJySJr87S58k3tS7rKIGJDkpuMuNiJycZIk4V8PdMVjt4dBCOCFb3ZiWR5DEsnrpgLS3Llz0bZtW2g0GsTGxmLLli3XHL906VJ06tQJGo0GkZGRWLlypdVxIQSmT5+O4OBgeHh4ID4+HocPWzcRKy4uRnJyMnx8fODn54cxY8agrKzMcryiogKPP/44IiMjoVKpkJiYeDOnZlO1u2hzFxsRuTJJkvDaA92QHHspJD23lCGJ5NXggLRkyRJMmjQJM2bMQF5eHrp3746EhATodLp6x2/atAlJSUkYM2YMtm/fjsTERCQmJmLPnj2WMTNnzsScOXOQnp6OnJwceHp6IiEhARUVFZYxycnJ2Lt3LzIzM5GRkYH169cjJSXFctxoNMLDwwPPPPMM4uPjG3pasmAXbSKiKxQKCa8P7oZHa4UktgAguUiigU8NjI2NRa9evfDhhx8CAEwmE0JDQ/H000/jxRdf/NP4YcOGoby8HBkZGZb3br/9dvTo0QPp6ekQQiAkJATPPfccnn/+eQCAXq+HVqvF/PnzMXz4cOzfvx9dunTB1q1bER0dDQBYvXo17rvvPpw8eRIhISFWv/Pxxx9HSUkJli9f3qCLYTAY4OvrC71eDx8fnwZ99mbkHj+PIR9tQuvmHtgw5e4m/31ERI7AZBJ49ce9+DL7UhPJ1wZ3xci4tvIWRXatKb6/GzSDVFVVhdzcXKsZGoVCgfj4eGRnZ9f7mezs7D/N6CQkJFjGHzt2DAUFBVZjfH19ERsbaxmTnZ0NPz8/SzgCgPj4eCgUCuTk5DTkFKxUVlbCYDBYvWxJxx5IRER/olBcWpM0tl84AGD693sxb/1RmasiV9OggHT27FkYjUZotVqr97VaLQoKCur9TEFBwTXHm/+83pjAwECr4yqVCv7+/lf9vTciLS0Nvr6+lldoaOhN/6ybYe6BxB1sRETWJEnCS/d1xtN33wIAeHPlAXyQxQfcku249C62qVOnQq/XW14nTti2/wZ7IBERXZ0kSXjuno547m8dAACzMw/hnZ8OooErQ4huSoMCUkBAAJRKJQoLC63eLywsRFBQUL2fCQoKuuZ485/XG1N3EXhNTQ2Ki4uv+ntvhFqtho+Pj9XLlizPYeMMEhHRVT09IAIv39cZAPDh2iP494r9DEnU5BoUkNzd3REVFYWsrCzLeyaTCVlZWYiLi6v3M3FxcVbjASAzM9MyPjw8HEFBQVZjDAYDcnJyLGPi4uJQUlKC3Nxcy5g1a9bAZDIhNja2IadgVyw9kDiDRER0TWPvaIfXBncFAHy64Rhe+m43jCaGJGo6qoZ+YNKkSRg1ahSio6MRExOD999/H+Xl5Rg9ejQAYOTIkWjVqhXS0tIAABMmTED//v0xe/ZsDBo0CIsXL8a2bdswb948AJemUCdOnIg33ngDERERCA8Px7Rp0xASEmLpZdS5c2cMHDgQY8eORXp6Oqqrq5Gamorhw4db7WDbt28fqqqqUFxcjNLSUuzYsQMA0KNHj79wiZqOjjNIREQ3bGRcW6hVCkxdthuLtpyA4WIN3h3WHWqVUu7SyAk1OCANGzYMRUVFmD59OgoKCtCjRw+sXr3assg6Pz8fCsWVianevXtj4cKFeOWVV/DSSy8hIiICy5cvR7du3SxjJk+ejPLycqSkpKCkpAR9+/bF6tWrodFcmVlZsGABUlNTMWDAACgUCgwZMgRz5syxqu2+++7D8ePHLf/cs2dPALDbqdhCdtEmImqQYb3C4K1xw8TFO7Bi9xkYKqqR/lgUPNUN/jojuqYG90FyZrbsg1RZY0THV1YDAHZOvwe+zdgokojoRm04fBYpX23DhSojeoT6Yf7oXvBr5i53WSQT2fsgUeMx315zVyng48H/50NE1BB9IwKw4B+x8Gvmhh0nSjD042wU6Cuu/0GiG8SAJJMrD6lVQ5IkmashInI8PcOa4+txcdD6qHGosAwPp2/CsbPlcpdFToIBSSbmGSTuYCMiunkdtN745sneaNuiGU6ev4hH0jdh18kSucsiJ8CAJBNLk0juYCMi+ktC/Zth6ZO90SXYB2fLqjB83masPVD/A9SJbhQDkkzMjxlhF20ior+upbcaS8bdjn4RAbhQZcQ/vtyGRVvy5S6LHBgDkkzYRZuIqHF5a9zw2eO9MOS21jCaBKYu2413f+ajSejmMCDJhF20iYgan5tSgXceuRXPDIgAAMxZcwTPL92FqhqTzJWRo2FAkollkTabRBIRNSpJkjDpbx3w1kORUCokfJt3EmO+2IrSimq5SyMHwoAkE3MXbd5iIyJqGsNjwvDpqGg0c1fit8Nn8Uh6Nk6evyB3WeQgGJBkUFFtRMmFS/9PhrfYiIiazl0dA7EkJQ4BXmocKChF4tyNyD1+Xu6yyAEwIMmg6PIONjW7aBMRNbnI1r74IbUPOl9uA5A0bzO+235S7rLIzjEgyUBX6/Yau2gTETW9ED8PfPNkHO7pokWV0YRnl+zEzNUHYDJxhxvVjwFJBoXsok1EZHOeahXSH4vCP+9sDwD4v1+P4qkFubhQVSNzZWSPGJBkoDOYn8PGgEREZEsKhYTJAzvh3aHd4a5U4Ke9hXj4o2ycKrkod2lkZxiQZFB4eQ1SS2/uYCMiksNDt7XGopRYtPB0x74zBjzwwQZsOnpW7rLIjjAgyYA9kIiI5BfVxh/Lx/dBl2AfnCuvwmOf5mDe+qPsvE0AGJBkYVmkzRkkIiJZhfo3w7J/9sZDt7WCSQBvrjyA1IXbUVbJdUmujgFJBoVcg0REZDc0bkrMfqQ7Xh/cFSqFhBW7z+DBuRtxtKhM7tJIRgxIMtCVmm+xcQaJiMgeSJKEEXFtsWTc7Qj0VuOwrgyDP9yIn/YWyF0ayYQBycZqd9EO5DZ/IiK7EtXGHxnP9EVMW3+UVdZg3Fe5+PeKfXzYrQtiQLIxdtEmIrJvgd4aLBgbiyf6hAMAPvntGB75OBv55/gcN1fCgGRj5gXaWh8Nu2gTEdkpN6UC0+/vgo9HRMHXww07T5Rg0JzfsGLXGblLIxthQLIxcxdt7mAjIrJ/CV2DsHJCP0S1aY7SyhqMX5iHqct2o6LaKHdp1MQYkGyMO9iIiBxLKz8PLEm5HePvag9JAhZtycfgDzficGGp3KVRE2JAsjHzDrZA7mAjInIYKqUCLyR0wldPxCLAS42DhaX4+wcb8PnGY3zgrZNiQLIx8wwSd7ARETmevhEBWDWhH/pFBKCyxoR//bgPj/03h89yc0IMSDZWxB5IREQOraW3Gl8+EYPXB3eFxk2BTUfPYeB76/FN7kk+psSJMCDZGGeQiIgcn7mx5KoJd6BnmB9KK2vw/NKdSPkqF2fLKuUujxoBA5KNFRo4g0RE5CzCAzyxdFwcXkjoCDelhMx9hUh4bz1W7j7D2SQHx4BkQxXVRugvXu6izV1sREROQaVUYPxdt2D5+D7oqPXGufIq/HNBHsZ+mYvTXJvksBiQbMiqi7aGXbSJiJxJ1xBf/PB0H6TedQtUCgm/7C/E395dh/kbj8HInW4OhwHJhmr3QGIXbSIi56NWKfF8QkeseKYfbgvzQ3mVEa/+uA8PfbQJ+88Y5C6PGoAByYZ03MFGROQSOgZ545sne+P1xG7wVquw80QJ/v7BBqSt2o8LVTVyl0c3gAHJhriDjYjIdSgUEkbc3ga/PNcf93YLgtEk8PG63zFg9jp8v+MUF3HbOQYkG7I8h40zSERELkPro8FHj0Xhk5HRaOXngTP6CkxYvAMPp2dj18kSucujq2BAsiFdKZ/DRkTkqv7WRYus5/rj+Xs6wMNNidzj5/HAhxvx/NKd0F2+w0D2gwHJhnTmGSRvziAREbkijZsSqXdHYO3zd+Khnq0AAN/knsRd7/yK//v1CCqqjTJXSGYMSDbEGSQiIgKAIF8N3h3WA8v+2RvdQy/tdpu5+iDumLkWX2X/gaoak9wlujwGJBsq5AwSERHVcltYc3z3VG+8O7Q7Wvl5QFdaiWnf78Xds3/F0m0nUGNkUJILA5KNsIs2ERHVR6GQ8NBtrbHm+f54bXBXtPRW4+T5i3jhm1245/31yNh1GiY2mrQ5BiQbMXfR1rixizYREf2ZWqXEyLi2WP/CXZh6byf4NXPD70XlSF24HffN+Q3fbT+Jas4o2QwDko3U7oHELtpERHQ1Hu5KjOvfHr9NvgsT4yPgpVbhQEEpnl2yE3fO+hWfbTiG8ko2m2xqDEg2Yl5/xC7aRER0I7w1bpgY3wEbp9yNFxI6IsDLHadKLuK1jH3o/dYazP75IM6WVcpdptNiQLIR8w42rj8iIqKG8G3mhvF33YINU+7Gmw9Gom2LZtBfrMYHa46gz1trMPmbndhxooSduRsZF8PYCHewERHRX6FxU+LR2DAM6xWKn/cWIH3dUew8qcfX207i620n0SXYB8m3h2Fwj1bwUvPr/a/iFbQRc5dU9kAiIqK/QqmQcG9kMAZ2C8K24+exMCcfK3afwb4zBrz83R68uWI/HujRCsmxYega4sN1rzeJAclGdKVcg0RERI1HkiT0auuPXm39Mf3vXfBt3kks3JKP34vKsWhLPhZtycctgV64/9YQ/L17MNq39JK7ZIfCgGQjtXexERERNabmnu74R792GNM3HJt/L8bCLfn4aU8BjujK8N4vh/DeL4fQJdgHf+8ejPtvDUGofzO5S7Z7DEg2Umi5xcYZJCIiahqSJCGufQvEtW8BQ0U1MvcW4sddp7Hh8FnsO2PAvjMGzFx9ELe29sWdHQPRv0NL9Aj1g1LB23B1MSDZQEW1EYaKSz0rWnIGiYiIbMBH44YhUa0xJKo1zpdXYfXeAvy48zQ2/34Ou07qseukHnOyDsPXww19IwLQv0NL3NmhJXdbX8aAZAM6A7toExGRfJp7uiMpJgxJMWHQlVbg1wNFWHeoCL8dLoL+YjVW7DqDFbvOAADat/TEbWHN0TOsOXqG+aGD1tslZ5j4bW0D5h5IWh920SYiInkFemswtFcohvYKRY3RhJ0nS7Du4KXAtOuUHkeLynG0qBxLc08CADzdlege6oceoX7oFOyD9i090S7ACx7uSpnPpGkxINkAeyAREZE9UikViGrjj6g2/ph0T0cUl1dhe/55bM8vwfYT57EjvwTlVUZsOnoOm46es3xOkoBWfh5o39IL7Vt6IbylJ7TearS8/ArwUkPj5tgBigHJBiw72Hhfl4iI7Ji/pzsGdNZiQGctAMBoEjiiK0Ne/nnsPFGCI7oyHCkqQ8mFapw8fxEnz1/EukNF9f4sXw83tPRWw7+ZO5qplfB0V6GZuxL3Rgbh7k5aW57WTWFAsgFLDyQu0CYiIgeiVEjoGOSNjkHeSIoJs7xfXF6FI7oyHC0qw1FdGf44dwFFZZU4W1qJotJKVBlN0F+shv5i9Z9+ZruWXgxIdEnfWwKgVADRbf3lLoWIiOgv8/d0R0y4P2LC//y9JoSA4WINdKUV0JVWQn+xGuWVNbhQZUR5VQ1i2znGd6Ek+HQ7C4PBAF9fX+j1evj4+MhdDhEREd2Apvj+VtzMh+bOnYu2bdtCo9EgNjYWW7Zsueb4pUuXolOnTtBoNIiMjMTKlSutjgshMH36dAQHB8PDwwPx8fE4fPiw1Zji4mIkJyfDx8cHfn5+GDNmDMrKyqzG7Nq1C/369YNGo0FoaChmzpx5M6dHRERELq7BAWnJkiWYNGkSZsyYgby8PHTv3h0JCQnQ6XT1jt+0aROSkpIwZswYbN++HYmJiUhMTMSePXssY2bOnIk5c+YgPT0dOTk58PT0REJCAioqKixjkpOTsXfvXmRmZiIjIwPr169HSkqK5bjBYMA999yDNm3aIDc3F7NmzcKrr76KefPmNfQUiYiIyNWJBoqJiRHjx4+3/LPRaBQhISEiLS2t3vFDhw4VgwYNsnovNjZWjBs3TgghhMlkEkFBQWLWrFmW4yUlJUKtVotFixYJIYTYt2+fACC2bt1qGbNq1SohSZI4deqUEEKI//u//xPNmzcXlZWVljFTpkwRHTt2vOFz0+v1AoDQ6/U3/BkiIiKSV1N8fzdoBqmqqgq5ubmIj4+3vKdQKBAfH4/s7Ox6P5OdnW01HgASEhIs448dO4aCggKrMb6+voiNjbWMyc7Ohp+fH6Kjoy1j4uPjoVAokJOTYxlzxx13wN3d3er3HDx4EOfPn6+3tsrKShgMBqsXERERUYMC0tmzZ2E0GqHVWm/P02q1KCgoqPczBQUF1xxv/vN6YwIDA62Oq1Qq+Pv7W42p72fU/h11paWlwdfX1/IKDQ2t/8SJiIjIpdzUIm1nMXXqVOj1esvrxIkTcpdEREREdqBBASkgIABKpRKFhYVW7xcWFiIoKKjezwQFBV1zvPnP642puwi8pqYGxcXFVmPq+xm1f0ddarUaPj4+Vi8iIiKiBgUkd3d3REVFISsry/KeyWRCVlYW4uLi6v1MXFyc1XgAyMzMtIwPDw9HUFCQ1RiDwYCcnBzLmLi4OJSUlCA3N9cyZs2aNTCZTIiNjbWMWb9+Paqrq61+T8eOHdG8efOGnCYRERG5uoau6l68eLFQq9Vi/vz5Yt++fSIlJUX4+fmJgoICIYQQI0aMEC+++KJl/MaNG4VKpRLvvPOO2L9/v5gxY4Zwc3MTu3fvtox56623hJ+fn/j+++/Frl27xODBg0V4eLi4ePGiZczAgQNFz549RU5OjtiwYYOIiIgQSUlJluMlJSVCq9WKESNGiD179ojFixeLZs2aiY8//viGz4272IiIiBxPU3x/NzggCSHEBx98IMLCwoS7u7uIiYkRmzdvthzr37+/GDVqlNX4r7/+WnTo0EG4u7uLrl27ihUrVlgdN5lMYtq0aUKr1Qq1Wi0GDBggDh48aDXm3LlzIikpSXh5eQkfHx8xevRoUVpaajVm586dom/fvkKtVotWrVqJt956q0HnxYBERETkeJri+5uPGqmFjxohIiJyPHbzqBEiIiIiZ8aARERERFSHSu4C7In5biM7ahMRETkO8/d2Y64aYkCqpbS0FADYUZuIiMgBlZaWwtfXt1F+Fhdp12IymXD69Gl4e3tDkqRG/dkGgwGhoaE4ceKEyy8A57W4gtfiCl4La7weV/BaXMFrYc18PfLz8yFJEkJCQqBQNM7qIc4g1aJQKNC6desm/R3s2H0Fr8UVvBZX8FpY4/W4gtfiCl4La76+vo1+PbhIm4iIiKgOBiQiIiKiOhiQbEStVmPGjBlQq9VylyI7XosreC2u4LWwxutxBa/FFbwW1pryenCRNhEREVEdnEEiIiIiqoMBiYiIiKgOBiQiIiKiOhiQiIiIiOpgQLKBuXPnom3bttBoNIiNjcWWLVvkLqnJpaWloVevXvD29kZgYCASExNx8OBBqzEVFRUYP348WrRoAS8vLwwZMgSFhYUyVWw7b731FiRJwsSJEy3vudq1OHXqFB577DG0aNECHh4eiIyMxLZt2yzHhRCYPn06goOD4eHhgfj4eBw+fFjGipuG0WjEtGnTEB4eDg8PD7Rv3x6vv/661fOknPVarF+/Hvfffz9CQkIgSRKWL19udfxGzru4uBjJycnw8fGBn58fxowZg7KyMhueReO51vWorq7GlClTEBkZCU9PT4SEhGDkyJE4ffq01c9wlutxvX83anvyySchSRLef/99q/cb41owIDWxJUuWYNKkSZgxYwby8vLQvXt3JCQkQKfTyV1ak1q3bh3Gjx+PzZs3IzMzE9XV1bjnnntQXl5uGfPss8/ixx9/xNKlS7Fu3TqcPn0aDz30kIxVN72tW7fi448/xq233mr1vitdi/Pnz6NPnz5wc3PDqlWrsG/fPsyePRvNmze3jJk5cybmzJmD9PR05OTkwNPTEwkJCaioqJCx8sb39ttv46OPPsKHH36I/fv34+2338bMmTPxwQcfWMY467UoLy9H9+7dMXfu3HqP38h5JycnY+/evcjMzERGRgbWr1+PlJQUW51Co7rW9bhw4QLy8vIwbdo05OXlYdmyZTh48CAeeOABq3HOcj2u9++G2XfffYfNmzcjJCTkT8ca5VoIalIxMTFi/Pjxln82Go0iJCREpKWlyViV7el0OgFArFu3TgghRElJiXBzcxNLly61jNm/f78AILKzs+Uqs0mVlpaKiIgIkZmZKfr37y8mTJgghHC9azFlyhTRt2/fqx43mUwiKChIzJo1y/JeSUmJUKvVYtGiRbYo0WYGDRoknnjiCav3HnroIZGcnCyEcJ1rAUB89913ln++kfPet2+fACC2bt1qGbNq1SohSZI4deqUzWpvCnWvR322bNkiAIjjx48LIZz3elztWpw8eVK0atVK7NmzR7Rp00a89957lmONdS04g9SEqqqqkJubi/j4eMt7CoUC8fHxyM7OlrEy29Pr9QAAf39/AEBubi6qq6utrk2nTp0QFhbmtNdm/PjxGDRokNU5A653LX744QdER0fjkUceQWBgIHr27IlPPvnEcvzYsWMoKCiwuh6+vr6IjY11uuvRu3dvZGVl4dChQwCAnTt3YsOGDbj33nsBuNa1qO1Gzjs7Oxt+fn6Ijo62jImPj4dCoUBOTo7Na7Y1vV4PSZLg5+cHwLWuh8lkwogRI/DCCy+ga9eufzreWNeCD6ttQmfPnoXRaIRWq7V6X6vV4sCBAzJVZXsmkwkTJ05Enz590K1bNwBAQUEB3N3dLf9xm2m1WhQUFMhQZdNavHgx8vLysHXr1j8dc7Vr8fvvv+Ojjz7CpEmT8NJLL2Hr1q145pln4O7ujlGjRlnOub7/bpzterz44oswGAzo1KkTlEoljEYj/v3vfyM5ORkAXOpa1HYj511QUIDAwECr4yqVCv7+/k59bYBLaxanTJmCpKQkywNaXel6vP3221CpVHjmmWfqPd5Y14IBiZrc+PHjsWfPHmzYsEHuUmRx4sQJTJgwAZmZmdBoNHKXIzuTyYTo6Gi8+eabAICePXtiz549SE9Px6hRo2Suzra+/vprLFiwAAsXLkTXrl2xY8cOTJw4ESEhIS53LejGVFdXY+jQoRBC4KOPPpK7HJvLzc3Ff/7zH+Tl5UGSpCb9XbzF1oQCAgKgVCr/tBupsLAQQUFBMlVlW6mpqcjIyMDatWvRunVry/tBQUGoqqpCSUmJ1XhnvDa5ubnQ6XS47bbboFKpoFKpsG7dOsyZMwcqlQpardZlrgUABAcHo0uXLlbvde7cGfn5+QBgOWdX+O/mhRdewIsvvojhw4cjMjISI0aMwLPPPou0tDQArnUtaruR8w4KCvrTZpeamhoUFxc77bUxh6Pjx48jMzPTMnsEuM71+O2336DT6RAWFmb5+/T48eN47rnn0LZtWwCNdy0YkJqQu7s7oqKikJWVZXnPZDIhKysLcXFxMlbW9IQQSE1NxXfffYc1a9YgPDzc6nhUVBTc3Nysrs3BgweRn5/vdNdmwIAB2L17N3bs2GF5RUdHIzk52fK/XeVaAECfPn3+1PLh0KFDaNOmDQAgPDwcQUFBVtfDYDAgJyfH6a7HhQsXoFBY/zWsVCphMpkAuNa1qO1GzjsuLg4lJSXIzc21jFmzZg1MJhNiY2NtXnNTM4ejw4cP45dffkGLFi2sjrvK9RgxYgR27dpl9fdpSEgIXnjhBfz0008AGvFa3PzacroRixcvFmq1WsyfP1/s27dPpKSkCD8/P1FQUCB3aU3qqaeeEr6+vuLXX38VZ86csbwuXLhgGfPkk0+KsLAwsWbNGrFt2zYRFxcn4uLiZKzadmrvYhPCta7Fli1bhEqlEv/+97/F4cOHxYIFC0SzZs3E//73P8uYt956S/j5+Ynvv/9e7Nq1SwwePFiEh4eLixcvylh54xs1apRo1aqVyMjIEMeOHRPLli0TAQEBYvLkyZYxznotSktLxfbt28X27dsFAPHuu++K7du3W3Zl3ch5Dxw4UPTs2VPk5OSIDRs2iIiICJGUlCTXKf0l17oeVVVV4oEHHhCtW7cWO3bssPo7tbKy0vIznOV6XO/fjbrq7mITonGuBQOSDXzwwQciLCxMuLu7i5iYGLF582a5S2pyAOp9ff7555YxFy9eFP/85z9F8+bNRbNmzcSDDz4ozpw5I1/RNlQ3ILnatfjxxx9Ft27dhFqtFp06dRLz5s2zOm4ymcS0adOEVqsVarVaDBgwQBw8eFCmapuOwWAQEyZMEGFhYUKj0Yh27dqJl19+2epLz1mvxdq1a+v9O2LUqFFCiBs773PnzomkpCTh5eUlfHx8xOjRo0VpaakMZ/PXXet6HDt27Kp/p65du9byM5zlelzv34266gtIjXEtJCFqtWwlIiIiIq5BIiIiIqqLAYmIiIioDgYkIiIiojoYkIiIiIjqYEAiIiIiqoMBiYiIiKgOBiQiIiKiOhiQiIiIiOpgQCIiIiKqgwGJiIiIqA4GJCIiIqI6GJCIiIiI6vh/02lODEDK4jUAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(range(len(lrs)), lrs)\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e80ac790-f9c3-45b8-9ea4-d2e5bf8fbf28", + "metadata": {}, + "outputs": [], + "source": [ + "- And a quick look at the loss curves" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "445d8155-6eae-4b50-a381-d0820ebc27cc", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnUAAAHWCAYAAAARl3+JAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB0m0lEQVR4nO3dd3wT9f8H8NcladKku6WTTqDQwZ5CkS1DRAERRH4MURQEAQfiAsGFICriAOWr4EBRlKWyyoayV6G0lN0W6KCU7p3c74+jaUMLFGh7afp6Ph73aPK59c4Z2xd39/mcIIqiCCIiIiKq1RRyF0BERERED46hjoiIiMgCMNQRERERWQCGOiIiIiILwFBHREREZAEY6oiIiIgsAEMdERERkQVgqCMiIiKyAAx1RERERBaAoY6Iap1Lly5BEAQcP35c7lKIiMwGQx0RyUIQhDtOs2bNkrtEIqJaRSV3AURUNyUmJhpf//HHH5g5cyZiY2ONbba2tnKURURUa/FMHRHJwsPDwzg5ODhAEATjezc3N3z++efw9vaGRqNBy5YtsXHjxttuS6/XY+zYsQgKCkJ8fDwAYO3atWjdujWsra3RoEEDzJ49G8XFxcZ1BEHA//73PwwaNAg6nQ6BgYFYt26dcf6NGzcwYsQIuLq6QqvVIjAwEEuXLr1tDX/99ReaNWsGrVYLFxcX9OrVCzk5Ocb5//vf/xAcHAxra2sEBQXh22+/NVk/ISEBQ4cOhaOjI5ydnfHEE0/g0qVLxvljxozBwIEDMX/+fHh6esLFxQUTJ05EUVFRpY85EVk2hjoiMjtffvklPvvsM8yfPx8nTpxAnz598Pjjj+Ps2bPlli0oKMBTTz2F48ePY/fu3fD19cXu3bsxatQoTJkyBdHR0fjuu++wbNkyfPTRRybrzp49G0OHDsWJEyfw6KOPYsSIEUhLSwMAzJgxA9HR0diwYQNiYmKwaNEi1KtXr8J6ExMTMXz4cIwdOxYxMTHYsWMHBg8eDFEUAQDLly/HzJkz8dFHHyEmJgYff/wxZsyYgZ9++gkAUFRUhD59+sDOzg67d+9GREQEbG1t0bdvXxQWFhr3s337dpw/fx7bt2/HTz/9hGXLlmHZsmVVcciJyBKIREQyW7p0qejg4GB87+XlJX700Ucmy7Rr10586aWXRFEUxYsXL4oAxN27d4s9e/YUO3fuLKanpxuX7dmzp/jxxx+brP/LL7+Inp6exvcAxHfffdf4Pjs7WwQgbtiwQRRFURwwYID47LPPVqr+I0eOiADES5cuVTi/YcOG4m+//WbS9sEHH4gdO3Y01takSRPRYDAY5xcUFIharVbctGmTKIqiOHr0aNHPz08sLi42LvPUU0+Jw4YNq1SNRGT5eE8dEZmVzMxMXL16FWFhYSbtYWFhiIyMNGkbPnw4vL29sW3bNmi1WmN7ZGQkIiIiTM7M6fV65OfnIzc3FzqdDgDQvHlz43wbGxvY29sjJSUFADBhwgQ8+eSTOHr0KHr37o2BAweiU6dOFdbcokUL9OzZE82aNUOfPn3Qu3dvDBkyBE5OTsjJycH58+fx3HPPYdy4ccZ1iouL4eDgYKz33LlzsLOzM9lufn4+zp8/b3wfGhoKpVJpfO/p6YmTJ0/e4WgSUV3CUEdEtdajjz6KX3/9Ffv27UOPHj2M7dnZ2Zg9ezYGDx5cbh1ra2vjaysrK5N5giDAYDAAAPr164e4uDisX78e4eHh6NmzJyZOnIj58+eX26ZSqUR4eDj27t2LzZs346uvvsI777yDAwcOGAPkkiVL0KFDh3LrldTbpk0bLF++vNy2XV1dK1UvERFDHRGZFXt7e3h5eSEiIgJdu3Y1tkdERKB9+/Ymy06YMAFNmzbF448/jv/++8+4fOvWrREbG4tGjRo9UC2urq4YPXo0Ro8ejYcffhjTpk2rMNQBUsAKCwtDWFgYZs6cCT8/P6xevRqvvvoqvLy8cOHCBYwYMaLCdVu3bo0//vgDbm5usLe3f6CaiajuYqgjIrMzbdo0vPfee2jYsCFatmyJpUuX4vjx4xWeyXr55Zeh1+vx2GOPYcOGDejcuTNmzpyJxx57DL6+vhgyZAgUCgUiIyMRFRWFDz/8sFI1zJw5E23atEFoaCgKCgrw77//Ijg4uMJlDxw4gK1bt6J3795wc3PDgQMHcO3aNePys2fPxuTJk+Hg4IC+ffuioKAAhw8fxo0bN/Dqq69ixIgR+PTTT/HEE0/g/fffh7e3N+Li4rBq1Sq88cYb8Pb2vv+DSUR1BkMdEZmdyZMnIyMjA6+99hpSUlIQEhKCdevWITAwsMLlp06dCoPBgEcffRQbN25Enz598O+//+L999/H3LlzYWVlhaCgIDz//POVrkGtVuOtt97CpUuXoNVq8fDDD2PFihUVLmtvb49du3ZhwYIFyMzMhJ+fHz777DP069cPAPD8889Dp9Ph008/xbRp02BjY4NmzZph6tSpAACdToddu3Zh+vTpGDx4MLKyslC/fn307NmTZ+6IqNIEUbzZ556IiIiIai2OU0dERERkARjqiIiIiCwAQx0RERGRBWCoIyIiIrIADHVEREREFoChjoiIiMgCMNTdo2+++Qb+/v6wtrZGhw4dcPDgQblLqlK7du3CgAED4OXlBUEQsGbNGpP5oihi5syZ8PT0hFarRa9evXD27FmTZdLS0jBixAjY29vD0dERzz33HLKzs02WOXHiBB5++GFYW1vDx8cH8+bNK1fLypUrERQUBGtrazRr1gzr16+v8s97v+bMmYN27drBzs4Obm5uGDhwIGJjY02Wyc/Px8SJE+Hi4gJbW1s8+eSTSE5ONlkmPj4e/fv3h06ng5ubG6ZNm4bi4mKTZXbs2IHWrVtDo9GgUaNGWLZsWbl6zPV7uWjRIjRv3hz29vawt7dHx44dsWHDBuN8HqOKffLJJxAEwTiOHcBjBQCzZs2CIAgmU1BQkHE+j5GpK1eu4P/+7//g4uICrVaLZs2a4fDhw8b5/H0O+Pv7l/tOCYKAiRMnAqiF3ymRKm3FihWiWq0Wf/zxR/HUqVPiuHHjREdHRzE5OVnu0qrM+vXrxXfeeUdctWqVCEBcvXq1yfxPPvlEdHBwENesWSNGRkaKjz/+uBgQECDm5eUZl+nbt6/YokULcf/+/eLu3bvFRo0aicOHDzfOz8jIEN3d3cURI0aIUVFR4u+//y5qtVrxu+++My4TEREhKpVKcd68eWJ0dLT47rvvilZWVuLJkyer/RhURp8+fcSlS5eKUVFR4vHjx8VHH31U9PX1FbOzs43LjB8/XvTx8RG3bt0qHj58WHzooYfETp06GecXFxeLTZs2FXv16iUeO3ZMXL9+vVivXj3xrbfeMi5z4cIFUafTia+++qoYHR0tfvXVV6JSqRQ3btxoXMacv5fr1q0T//vvP/HMmTNibGys+Pbbb4tWVlZiVFSUKIo8RhU5ePCg6O/vLzZv3lycMmWKsZ3HShTfe+89MTQ0VExMTDRO165dM87nMSqVlpYm+vn5iWPGjBEPHDggXrhwQdy0aZN47tw54zL8fS6KKSkpJt+n8PBwEYC4fft2URRr33eKoe4etG/fXpw4caLxvV6vF728vMQ5c+bIWFX1uTXUGQwG0cPDQ/z000+Nbenp6aJGoxF///13URRFMTo6WgQgHjp0yLjMhg0bREEQxCtXroiiKIrffvut6OTkJBYUFBiXmT59utikSRPj+6FDh4r9+/c3qadDhw7iiy++WKWfsaqkpKSIAMSdO3eKoigdFysrK3HlypXGZWJiYkQA4r59+0RRlAK0QqEQk5KSjMssWrRItLe3Nx6bN954QwwNDTXZ17Bhw8Q+ffoY39e276WTk5P4v//9j8eoAllZWWJgYKAYHh4udu3a1RjqeKwk7733ntiiRYsK5/EYmZo+fbrYuXPn287n7/OKTZkyRWzYsKFoMBhq5XeKl18rqbCwEEeOHEGvXr2MbQqFAr169cK+fftkrKzmXLx4EUlJSSbHwMHBAR06dDAeg3379sHR0RFt27Y1LtOrVy8oFAocOHDAuEyXLl2gVquNy/Tp0wexsbG4ceOGcZmy+ylZxlyPdUZGBgDA2dkZAHDkyBEUFRWZfIagoCD4+vqaHKtmzZrB3d3duEyfPn2QmZmJU6dOGZe503GoTd9LvV6PFStWICcnBx07duQxqsDEiRPRv3//cp+Hx6rU2bNn4eXlhQYNGmDEiBGIj48HwGN0q3Xr1qFt27Z46qmn4ObmhlatWmHJkiXG+fx9Xl5hYSF+/fVXjB07FoIg1MrvFENdJaWmpkKv15v8hwMAd3d3JCUlyVRVzSr5nHc6BklJSXBzczOZr1Kp4OzsbLJMRdsou4/bLWOOx9pgMGDq1KkICwtD06ZNAUj1q9VqODo6mix767G63+OQmZmJvLy8WvG9PHnyJGxtbaHRaDB+/HisXr0aISEhPEa3WLFiBY4ePYo5c+aUm8djJenQoQOWLVuGjRs3YtGiRbh48SIefvhhZGVl8Rjd4sKFC1i0aBECAwOxadMmTJgwAZMnT8ZPP/0EgL/PK7JmzRqkp6djzJgxAGrn/3eqe1qaiMqZOHEioqKisGfPHrlLMUtNmjTB8ePHkZGRgb/++gujR4/Gzp075S7LrCQkJGDKlCkIDw+HtbW13OWYrX79+hlfN2/eHB06dICfnx/+/PNPaLVaGSszPwaDAW3btsXHH38MAGjVqhWioqKwePFijB49WubqzNMPP/yAfv36wcvLS+5S7hvP1FVSvXr1oFQqy/V6SU5OhoeHh0xV1aySz3mnY+Dh4YGUlBST+cXFxUhLSzNZpqJtlN3H7ZYxt2M9adIk/Pvvv9i+fTu8vb2N7R4eHigsLER6errJ8rceq/s9Dvb29tBqtbXie6lWq9GoUSO0adMGc+bMQYsWLfDll1/yGJVx5MgRpKSkoHXr1lCpVFCpVNi5cycWLlwIlUoFd3d3HqsKODo6onHjxjh37hy/T7fw9PRESEiISVtwcLDxcjV/n5uKi4vDli1b8PzzzxvbauN3iqGuktRqNdq0aYOtW7ca2wwGA7Zu3YqOHTvKWFnNCQgIgIeHh8kxyMzMxIEDB4zHoGPHjkhPT8eRI0eMy2zbtg0GgwEdOnQwLrNr1y4UFRUZlwkPD0eTJk3g5ORkXKbsfkqWMZdjLYoiJk2ahNWrV2Pbtm0ICAgwmd+mTRtYWVmZfIbY2FjEx8ebHKuTJ0+a/NIMDw+Hvb298Zfx3Y5DbfxeGgwGFBQU8BiV0bNnT5w8eRLHjx83Tm3btsWIESOMr3msysvOzsb58+fh6enJ79MtwsLCyg2zdObMGfj5+QHg7/NbLV26FG5ubujfv7+xrVZ+p+6pW0Udt2LFClGj0YjLli0To6OjxRdeeEF0dHQ06fVS22VlZYnHjh0Tjx07JgIQP//8c/HYsWNiXFycKIpSF3hHR0dx7dq14okTJ8Qnnniiwi7wrVq1Eg8cOCDu2bNHDAwMNOkCn56eLrq7u4sjR44Uo6KixBUrVog6na5cF3iVSiXOnz9fjImJEd977z2z6QIviqI4YcIE0cHBQdyxY4dJd/jc3FzjMuPHjxd9fX3Fbdu2iYcPHxY7duwoduzY0Ti/pCt87969xePHj4sbN24UXV1dK+wKP23aNDEmJkb85ptvKuwKb67fyzfffFPcuXOnePHiRfHEiRPim2++KQqCIG7evFkURR6jOynb+1UUeaxEURRfe+01cceOHeLFixfFiIgIsVevXmK9evXElJQUURR5jMo6ePCgqFKpxI8++kg8e/asuHz5clGn04m//vqrcRn+Ppfo9XrR19dXnD59erl5te07xVB3j7766ivR19dXVKvVYvv27cX9+/fLXVKV2r59uwig3DR69GhRFKVu8DNmzBDd3d1FjUYj9uzZU4yNjTXZxvXr18Xhw4eLtra2or29vfjss8+KWVlZJstERkaKnTt3FjUajVi/fn3xk08+KVfLn3/+KTZu3FhUq9ViaGio+N9//1Xb575XFR0jAOLSpUuNy+Tl5YkvvfSS6OTkJOp0OnHQoEFiYmKiyXYuXbok9uvXT9RqtWK9evXE1157TSwqKjJZZvv27WLLli1FtVotNmjQwGQfJcz1ezl27FjRz89PVKvVoqurq9izZ09joBNFHqM7uTXU8VhJw0B4enqKarVarF+/vjhs2DCTcdd4jEz9888/YtOmTUWNRiMGBQWJ33//vcl8/j6XbNq0SQRQ7rOLYu37TgmiKIr3dm6PiIiIiMwN76kjIiIisgAMdUREREQWgKGOiIiIyAIw1BERERFZAIY6IiIiIgvAUEdERERkARjq7kNBQQFmzZqFgoICuUsxazxOlcPjVHk8VpXD41Q5PE6Vx2NVOXIfJ45Tdx8yMzPh4OCAjIwM2Nvby12O2eJxqhwep8rjsaocHqfK4XGqPB6rypH7OPFMHREREZEFYKgjIiIisgAquQuobsXFxTh27Bjc3d2hUFRNhs3KygIAXLlyBZmZmVWyTUvE41Q5PE6Vx2NVOTxOlcPjVHk8VpVT9jilp6cjOTkZrVq1gkpVM3HL4u+pO3ToENq3by93GURERFQHHTx4EO3atauRfVn8mTp3d3cA0kH19PSUuRoiIiKqCxITE9G+fXtjDqkJFh/qSi65enp6wtvbW+ZqiIiIqC6pqlu/KrWvGtsTEREREVUbhjoiIiIiC8BQR0RERGQBLP6eOiIiorL0ej2KiorkLoNqOSsrKyiVSrnLMMFQR0REdYIoikhKSkJ6errcpZCFcHR0hIeHBwRBkLsUAAx1RERUR5QEOjc3N+h0OrP5Q0y1jyiKyM3NRUpKCgCYzZBpDHVERGTx9Hq9MdC5uLjIXQ5ZAK1WCwBISUmBm5ubWVyKZUcJIiKyeCX30Ol0OpkrIUtS8n0yl3s0GeqIiKjO4CVXqkrm9n1iqCMiIiKyAAx1REREdYi/vz8WLFhQ6eV37NgBQRCqvdfwsmXL4OjoWK37sHQMdURERGZIEIQ7TrNmzbqv7R46dAgvvPBCpZfv1KkTEhMT4eDgcF/7o5rD3q9ERERmKDEx0fj6jz/+wMyZMxEbG2tss7W1Nb4WRRF6vR4q1d3/rLu6ut5THWq1Gh4eHve0DsmDZ+oeUH6RHjsiz+Hs4S1yl0JERBbEw8PDODk4OEAQBOP706dPw87ODhs2bECbNm2g0WiwZ88enD9/Hk888QTc3d1ha2uLdu3aYcsW079Pt15+FQQB//vf/zBo0CDodDoEBgZi3bp1xvm3Xn4tuUy6adMmBAcHw9bWFn379jUJocXFxZg8eTIcHR3h4uKC6dOnY/To0Rg4cOA9HYNFixahYcOGUKvVaNKkCX755RfjPFEUMWvWLPj6+kKj0cDLywuTJ082zv/2228RGBgIa2truLu7Y8iQIfe079qIoe4BLV/7Hx5a9RA8N4wFigvlLoeIiCpJFEXkFhbX+CSKYpV9hjfffBOffPIJYmJi0Lx5c2RnZ+PRRx/F1q1bcezYMfTt2xcDBgxAfHz8Hbcze/ZsDB06FCdOnMCjjz6KESNGIC0t7bbL5+bmYv78+fjll1+wa9cuxMfH4/XXXzfOnzt3LpYvX46lS5ciIiICmZmZWLNmzT19ttWrV2PKlCl47bXXEBUVhRdffBHPPvsstm/fDgD4+++/8cUXX+C7777D2bNnsWbNGjRr1gwAcPjwYUyePBnvv/8+YmNjsXHjRnTp0uWe9l8b8fLrA2rasgMyTtrAXZ8O/dnNUAY/JndJRERUCXlFeoTM3FTj+41+vw906qr58/v+++/jkUceMb53dnZGixYtjO8/+OADrF69GuvWrcOkSZNuu50xY8Zg+PDhAICPP/4YCxcuxMGDB9G3b98Kly8qKsLixYvRsGFDAMCkSZPw/vvvG+d/9dVXeOuttzBo0CAAwNdff43169ff02ebP38+xowZg5deegkA8Oqrr2L//v2YP38+unfvjvj4eHh4eKBXr16wsrKCr68v2rdvDwCIj4+HjY0NHnvsMdjZ2cHPzw+tWrW6p/3XRjxT94DaBLhig/AwACBz/y93WZqIiKjqtG3b1uR9dnY2Xn/9dQQHB8PR0RG2traIiYm565m65s2bG1/b2NjA3t7e+Aisiuh0OmOgA6THZJUsn5GRgeTkZGPAAgClUok2bdrc02eLiYlBWFiYSVtYWBhiYmIAAE899RTy8vLQoEEDjBs3DqtXr0ZxcTEA4JFHHoGfnx8aNGiAkSNHYvny5cjNzb2n/ddGPFP3gFRKBRL9BwGX/oF9/BYgNw3QOctdFhER3YXWSono9/vIst+qYmNjY/L+9ddfR3h4OObPn49GjRpBq9ViyJAhKCy88+1BVlZWJu8FQYDBYLin5avysnJl+Pj4IDY2Flu2bEF4eDheeuklfPrpp9i5cyfs7Oxw9OhR7NixA5s3b8bMmTMxa9YsHDp0yKKHTeGZuioQ0vIhnDL4QSkWA6dWyV0OERFVgiAI0KlVNT5V51MIIiIiMGbMGAwaNAjNmjWDh4cHLl26VG37q4iDgwPc3d1x6NAhY5ter8fRo0fvaTvBwcGIiIgwaYuIiEBISIjxvVarxYABA7Bw4ULs2LED+/btw8mTJwEAKpUKvXr1wrx583DixAlcunQJ27Zte4BPZv54pq4KdG3siq8NXRCq+AUFR36Dpt3zcpdERER1UGBgIFatWoUBAwZAEATMmDHjjmfcqsvLL7+MOXPmoFGjRggKCsJXX32FGzdu3FOgnTZtGoYOHYpWrVqhV69e+Oeff7Bq1Spjb95ly5ZBr9ejQ4cO0Ol0+PXXX6HVauHn54d///0XFy5cQJcuXeDk5IT169fDYDCgSZMm1fWRzQLP1FUBR50acV79UCwqoEk6AqSek7skIiKqgz7//HM4OTmhU6dOGDBgAPr06YPWrVvXeB3Tp0/H8OHDMWrUKHTs2BG2trbo06cPrK2tK72NgQMH4ssvv8T8+fMRGhqK7777DkuXLkW3bt0AAI6OjliyZAnCwsLQvHlzbNmyBf/88w9cXFzg6OiIVatWoUePHggODsbixYvx+++/IzQ0tJo+sXkQxJq+CF7DLl++DB8fHyQkJMDb27va9vPdzvMI3DIWPZTHgS7TgB7vVtu+iIjo3uTn5+PixYsICAi4p2BBVcNgMCA4OBhDhw7FBx98IHc5VeZO36uayh9l8UxdFekZ7IZVeqkXrCFyBSDD6W4iIiJzEBcXhyVLluDMmTM4efIkJkyYgIsXL+KZZ56RuzSLxlBXRRq62uK0Q2dkilooMhKA+L1yl0RERCQLhUKBZcuWoV27dggLC8PJkyexZcsWBAcHy12aRWNHiSoiCAI6B/vgv4MPYbhqOxD5O+DfWe6yiIiIapyPj0+5nqtU/XimrgqVvQQrnloLFFr+QIdERERkHhjqqlD7AGfEWAXjL30XJIR9DCiqboBJIiIiojthqKtCGpUSnQPd8XrReKwqeghQaeQuiYiIiOoIhroq1iPYDQCw7fTtn5lHREREVNUY6qpY9yZSqEu+fBHZW+YBkX/IXBERERHVBQx1VczVToMWPo7oqTwG2z0fAXsXyl0SERER1QEMddWgZ5Ab/tV3QIy2NdBhPAciJiIi2XTr1g1Tp041vvf398eCBQvuuI4gCFizZs0D77uqtnMns2bNQsuWLat1H7UFQ1016BHkhkzY4smc6chv9gyg4GEmIqJ7M2DAAPTt27fCebt374YgCDhx4sQ9b/fQoUN44YUXHrQ8E7cLVomJiejXr1+V7otuj2mjGoR62cPdXoPcQj0OXEyTuxwiIqqFnnvuOYSHh+Py5cvl5i1duhRt27ZF8+bN73m7rq6u0Ol0VVHiXXl4eECj4UgQNYWhrhoIgoAeQe4AgAMnTgP7FwHx+2WuioiIapPHHnsMrq6uWLZsmUl7dnY2Vq5cieeeew7Xr1/H8OHDUb9+feh0OjRr1gy///77Hbd76+XXs2fPokuXLrC2tkZISAjCw8PLrTN9+nQ0btwYOp0ODRo0wIwZM1BUVAQAWLZsGWbPno3IyEgIggBBEIw133r59eTJk+jRowe0Wi1cXFzwwgsvIDs72zh/zJgxGDhwIObPnw9PT0+4uLhg4sSJxn1VhsFgwPvvvw9vb29oNBq0bNkSGzduNM4vLCzEpEmT4OnpCWtra/j5+WHOnDkAAFEUMWvWLPj6+kKj0cDLywuTJ0+u9L7lxseEVZOeQW74/WA8AmIWAVH/AU2fBHwfkrssIiK6VWHOva+j1ADKm39C9cWAvgAQFICV9s7bVdtUehcqlQqjRo3CsmXL8M4770AQBADAypUrodfrMXz4cGRnZ6NNmzaYPn067O3t8d9//2HkyJFo2LAh2rdvf9d9GAwGDB48GO7u7jhw4AAyMjJM7r8rYWdnh2XLlsHLywsnT57EuHHjYGdnhzfeeAPDhg1DVFQUNm7ciC1btgAAHBwcym0jJycHffr0QceOHXHo0CGkpKTg+eefx6RJk0yC6/bt2+Hp6Ynt27fj3LlzGDZsGFq2bIlx48ZV6rh9+eWX+Oyzz/Ddd9+hVatW+PHHH/H444/j1KlTCAwMxMKFC7Fu3Tr8+eef8PX1RUJCAhISEgAAf//9N7744gusWLECoaGhSEpKQmRkZKX2aw4Y6qpJWKN60KgU+Cm3I57S/Aec/g/IzwCsy3/RiYhIRh973fs6Ty0DQgdJr0//A6wcA/h1Bp79r3SZBc2A3Oum683KuKfdjB07Fp9++il27tyJbt26AZAuvT755JNwcHCAg4MDXn/9dePyL7/8MjZt2oQ///yzUqFuy5YtOH36NDZt2gQvL+k4fPzxx+Xug3v33XeNr/39/fH6669jxYoVeOONN6DVamFrawuVSgUPD4/b7uu3335Dfn4+fv75Z9jYSOH266+/xoABAzB37ly4u0tXuJycnPD1119DqVQiKCgI/fv3x9atWysd6ubPn4/p06fj6aefBgDMnTsX27dvx4IFC/DNN98gPj4egYGB6Ny5MwRBgJ+fn3Hd+Ph4eHh4oFevXrCysoKvr2+ljqO54OXXaqJVK9GpoQuixACk6RoAxflA9Fq5yyIiolokKCgInTp1wo8//ggAOHfuHHbv3o3nnnsOAKDX6/HBBx+gWbNmcHZ2hq2tLTZt2oT4+PhKbT8mJgY+Pj7GQAcAHTt2LLfcH3/8gbCwMHh4eMDW1hbvvvtupfdRdl8tWrQwBjoACAsLg8FgQGxsrLEtNDQUSmXpYzY9PT2RklK5Af0zMzNx9epVhIWFmbSHhYUhJiYGgHSJ9/jx42jSpAkmT56MzZs3G5d76qmnkJeXhwYNGmDcuHFYvXo1iouL7+lzyoln6qpRj2B3bI+9hvWKrvg/XAAiVwCtR8ldFhERlfX21XtfR1nm5v+gAdI2hFvOk0w9+WB13fTcc8/h5ZdfxjfffIOlS5eiYcOG6Nq1KwDg008/xZdffokFCxagWbNmsLGxwdSpU1FYWFgl+waAffv2YcSIEZg9ezb69OkDBwcHrFixAp999lmV7aMsKysrk/eCIMBQhUODtW7dGhcvXsSGDRuwZcsWDB06FL169cJff/0FHx8fxMbGYsuWLQgPD8dLL71kPFN6a13mSNYzdbt27cKAAQPg5eVV4Vg2oihi5syZ8PT0hFarRa9evXD27Fl5ir0PPYKkp0t8e701RAhAXARw45K8RRERkSm1zb1PyjLnRJQqqa3s/XS32+59GDp0KBQKBX777Tf8/PPPGDt2rPH+uoiICDzxxBP4v//7P7Ro0QINGjTAmTNnKr3t4OBgJCQkIDEx0di2f79px769e/fCz88P77zzDtq2bYvAwEDExcWZflS1Gnq9/q77ioyMRE5O6b2GERERUCgUaNKkSaVrvhN7e3t4eXkhIiLCpD0iIgIhISEmyw0bNgxLlizBH3/8gb///htpadJoFVqtFgMGDMDChQuxY8cO7Nu3DydPVk1Ar26yhrqcnBy0aNEC33zzTYXz582bh4ULF2Lx4sU4cOAAbGxs0KdPH+Tn59dwpfenvqMWQR52uCq64JrrzU4SJ/6UtygiIqpVbG1tMWzYMLz11ltITEzEmDFjjPMCAwMRHh6OvXv3IiYmBi+++CKSk5Mrve1evXqhcePGGD16NCIjI7F792688847JssEBgYiPj4eK1aswPnz57Fw4UKsXr3aZBl/f39cvHgRx48fR2pqKgoKCsrta8SIEbC2tsbo0aMRFRWF7du34+WXX8bIkSON99NVhWnTpmHu3Ln4448/EBsbizfffBPHjx/HlClTAACff/45fv/9d5w+fRpnzpzBypUr4eHhAUdHRyxbtgw//PADoqKicOHCBfz666/QarUm992ZM1lDXb9+/fDhhx9i0KBB5eaJoogFCxbg3XffxRNPPIHmzZvj559/xtWrV6t9dOqq1DNYOlu3QdlNaoj8HRBF+QoiIqJa57nnnsONGzfQp08fk/vf3n33XbRu3Rp9+vRBt27d4OHhgYEDB1Z6uwqFAqtXr0ZeXh7at2+P559/Hh999JHJMo8//jheeeUVTJo0CS1btsTevXsxY8YMk2WefPJJ9O3bF927d4erq2uFw6rodDps2rQJaWlpaNeuHYYMGYKePXvi66+/vreDcReTJ0/Gq6++itdeew3NmjXDxo0bsW7dOgQGBgKQevLOmzcPbdu2Rbt27XDp0iWsX78eCoUCjo6OWLJkCcLCwtC8eXNs2bIF//zzD1xcXKq0xuoiiKJ5JAxBELB69Wrjl/HChQto2LAhjh07ZjJKddeuXdGyZUt8+eWXldru5cuX4ePjg4SEBHh7e1dD5Xd2JO4Gnly0F27WxThgNQFCUQ7wXDjgU3t60xAR1Xb5+fm4ePEiAgICYG1tLXc5ZCHu9L2SI3+Ybe/XpKQkACh3Stbd3d04ryIFBQXIzMw0TllZWdVa59209HGEs40aKfkqXPPpIzVG3nlgSCIiIqJ7Zbah7n7NmTPHOHaPg4ODyY2RclAqBHRr4goACLfqJjVG/Q0Ul7/fgIiIiOh+mW2oKxnA8NYbPpOTk+84uOFbb72FjIwM4xQdHV2tdVZGz5uPDFt6xRuwry8NQnxm413WIiIiIqo8sw11AQEB8PDwwNatW41tmZmZOHDgQIUDI5bQaDSwt7c3TnZ2djVR7h11aVwPKoWAc6n5SG90s1NIzD/yFkVEREQWRdbBh7Ozs3Hu3Dnj+5Lu0M7OzvD19cXUqVPx4YcfIjAwEAEBAZgxYwa8vLzuqWePObCztkKHBs6IOHcdm7T9MGx4Z6BRL7nLIiIiIgsia6g7fPgwunfvbnz/6quvAgBGjx6NZcuW4Y033kBOTg5eeOEFpKeno3Pnzti4cWOt7LnUI8gdEeeuY12cCsMe6Xf3FYiIqMpV5ZMJiMzt+2Q2Q5pUF7mHNClxKTUH3ebvgEoh4NjMR2BnbSWNV3dzVHAiIqo+BoMBZ8+ehVKphKurK9RqtfGpDET3ShRFFBYW4tq1a9Dr9QgMDIRCYXpHmxz5g89+rSH+9WzQwNUGF67lYPfZVDx6/Sfg2HJgxErALUju8oiILJpCoUBAQAASExNx9ep9POuVqAI6nQ6+vr7lAp1cGOpqUM8gN1y4dhFbY1LwqD4SyIgHTq4Ees64+8pERPRA1Go1fH19UVxcfNfnlBLdjVKphEqlMqszvgx1NahHkDuW7L6IHbEp0I98GcpmTwFNeH8dEVFNEQQBVlZWsLKykrsUoipnHucL64i2/k6ws1bhek4hIhXBQNPBgJVW7rKIiIjIAjDU1SArpQJdG0tPl9gWkyJzNURERGRJGOpqWM9gNwDA1tMp0qPCdn0KfNcFKMiWuTIiIiKqzRjqaljXxm5QCEBMYiauZumB478BiZHA6X/lLo2IiIhqMYa6GuZso0ZrXycAwLbYa0CL4dKMyN9lrIqIiIhqO4Y6GfS4eQl22+kUoPlQqfHCTiDjioxVERERUW3GUCeDnkHuAICIc6nIs/EB/MIAiMDJP+UtjIiIiGothjoZNHa3RX1HLQqKDdh7PhVo8bQ0I3KF9OgwIiIionvEUCcDQRBMe8GGPAGorIFrp4HE4/IWR0RERLUSQ51MegTdvK8uJgWixh4I6i/NiFwhY1VERERUWzHUyeShBi7QWimRlJmP6MTM0l6wJ1cC+iJ5iyMiIqJah6FOJtZWSnQOrAcA2BqTAjToDti4AbnXgXNbZK6OiIiIahuGOhn1DCpzX51SVTq8yfHlMlZFREREtRFDnYy63wx1kQnpuJZVALR8RpoRuwHIviZjZURERFTbMNTJyN3eGs3qOwAAtsemAO6hQP02gK0HkHZe5uqIiIioNmGok1nZXrAAgGHLgaknAN+HZKyKiIiIahuGOpmVjFe3++w1FBTrAXtPQKGUuSoiIiKqbRjqZNbUywGudhrkFOpx8GJa6Qx9EXD1mHyFERERUa3CUCczhUJAjyY3e8GWXILNvAp8HgL82A/IS5evOCIiIqo1GOrMQA/jI8OSIYoiYOcJ6FwAjR2Qelbm6oiIiKg2UMldAAGdG9WDWqlAQloezl/LRiM3O2D474CDN6C0krs8IiIiqgV4ps4M2GhU6NjQBUCZS7DOAQx0REREVGkMdWaiZ3CZp0uUZdADKadlqIiIiIhqE4Y6M9H9ZmeJI3E3kJ5bKDWmxwMLmgM/PAIU5shYHREREZk7hjoz4eOsQxN3O+gNInaeufmIMHtvacy6gkwgeq28BRIREZFZY6gzIyW9YDdHJ0sNCgXQepT0+ujPMlVFREREtQFDnRl5tKknACA8Orn0EmzLEYCgAOL3AdfOyFgdERERmTOGOjPStL49gj3tUVhswJpjV6RGe08gsI/0+hjP1hEREVHFGOrMiCAIGNbWGwDwx+HL0kDEANBmtPTz+O9AcaFM1REREZE5Y6gzMwNb1YdapUBMYiairmRKjY0eAWw9gNxUIHa9vAUSERGRWWKoMzOOOjX6hnoAAP44HC81KlVAqxHSa3aYICIiogow1JmhYe18AABrj11FXqFeamw1Uvp5fps0fh0RERFRGQx1ZqhjAxf4OGuRVVCMDVGJUqNzABDQFYAIHFsua31ERERkfhjqzJBCIeCpNtLZuj8OJZTOKBmz7tiv0uPDiIiIiG5iqDNTQ9p4QxCAAxfTcCn15iPCgh4DtE5A5mXpMiwRERHRTQx1ZsrLUYsuga4AgD8P3zxbZ2UNhE0Fes4EPFvIVxwRERGZHYY6M/b0zQ4Tfx25jGK9QWrsPBV4+DXA1k2+woiIiMjsMNSZsZ7B7nCxUSMlqwA7Yq/JXQ4RERGZMYY6M6ZWKTCoVX0AwB+Hy3SYKC4Eov4GVk8ASp46QURERHUaQ52ZKxmzbtvpFKRk5UuN+kJg3WQg8jcgbq+M1REREZG5YKgzc4Hudmjl6wi9QcSqo1ekRo0t8NAEoOt0afw6IiIiqvMY6mqBYW2ls3V/HkqAWHK5tce7QPe3AXsvGSsjIiIic8FQVws81sILOrUSF1JzcOjSDbnLISIiIjPEUFcL2GpUeKy5J4BbnjChLwZOrwc2TGeHCSIiojqOoa6WKOkwsf5kIrLyi6TGgkxg5RjgwGIg8bhstREREZH8GOpqida+TmjoaoO8Ij3+iUyUGnXOQPAA6fWRn+QrjoiIiGTHUFdLCIJgPFtnMmZdm9HSz5N/AYU5MlRGRERE5oChrhYZ3NobKoWAyIR0nE7KlBr9OgNOAUBhFnBqjaz1ERERkXwY6mqRerYa9Ap2B1Cmw4RCAbQeKb0+ykuwREREdRVDXS1Tcgl29bErKCjWS40tRwCCEkg4AKSclrE6IiIikotZhzq9Xo8ZM2YgICAAWq0WDRs2xAcffFA6AG8d1KWxKzzsrZGeW4Tw6GSp0c4DaNxXen3sF/mKIyIiItmYdaibO3cuFi1ahK+//hoxMTGYO3cu5s2bh6+++kru0mSjVAgY0sYbwC1j1rUeJf2M/B0oLpChMiIiIpKTWYe6vXv34oknnkD//v3h7++PIUOGoHfv3jh48KDcpclq6M3Hhu05l4rLN3Klxka9ADtPIPc6cPo/GasjIiIiOZh1qOvUqRO2bt2KM2fOAAAiIyOxZ88e9OvXT+bK5OXrokPHBi4QReCvI5elRqUKaPV/0uujP8tXHBEREcnCrEPdm2++iaeffhpBQUGwsrJCq1atMHXqVIwYMeK26xQUFCAzM9M4ZWVl1WDFNefp9tLZupWHL0NvuHmPYUmou7AduHFJnsKIiIhIFmYd6v78808sX74cv/32G44ePYqffvoJ8+fPx08/3X7ojjlz5sDBwcE4hYSE1GDFNadPqAfsrVW4kp6HiHOpUqOTP9CgG6CwAi4flrM8IiIiqmGCaMZdSX18fPDmm29i4sSJxrYPP/wQv/76K06frnjojoKCAhQUlHYUuHLlCkJCQpCQkABvb+9qr7kmzVwbhZ/3xaF/c09880xrqfHaGenxYTb15C2OiIioDrt8+TJ8fHxqNH+Y9Zm63NxcKBSmJSqVShgMhtuuo9FoYG9vb5zs7Oyqu0zZlHSYCD+VjBs5hVKja2MGOiIiojrIrEPdgAED8NFHH+G///7DpUuXsHr1anz++ecYNGiQ3KWZhab1HRDqZY9CvQGrj10pv0BmYs0XRURERLIw61D31VdfYciQIXjppZcQHByM119/HS+++CI++OADuUszGyVPmPjjUELpoMyFucCP/YAFTYGsZBmrIyIioppi1qHOzs4OCxYsQFxcHPLy8nD+/Hl8+OGHUKvVcpdmNp5oUR8alQKxyVmIvJwhNap1gKEYEA1AXIS8BRIREVGNMOtQR3fnoLNCv6YeAG55wkT/z4BXTgFNB8tUGREREdUkhjoLMPTmJdh/Iq8it7BYavRsDth7yVgVERER1SSGOgvwUIALfJ11yC4oxvqTSeUXSDgIXDlS84URERFRjWGoswAKhYChbaUxcP4sewkWACL/AH54BFg7CdAXyVAdERER1QSGOgsxpI0PFAJw8FIazl/LLp0R+AigcwFSooH938pXIBEREVUrhjoL4eFgjW5N3AAAfx4uc7ZO5ww8cnMImB2fAOkJFaxNREREtR1DnQUpecLE30euoEhf5qkbLYYDvh2Bolxg45syVUdERETViaHOgvQMdkM9WzVSswuw/XRK6QyFAuj/OaBQAaf/BWI3ylckERERVQuGOgtipVRgcOubHSYO33KZ1T0EeOgl6fWGadJTJ4iIiMhiMNRZmJJLsNtjryE5M990ZtfpgL03kB4P7J4vQ3VERERUXRjqLEwjN1u08XOC3iDiryOXTWdqbIF+c6XXEQuBa7E1XyARERFVC4Y6CzTs5hMmVh5OgCiKpjOD+gOBfQBDEfDfa8Ct84mIiKhWYqizQP2becJGrcSl67k4cDHNdKYgAI/OA1Ra4NJu4MSf8hRJREREVYqhzgLZaFQY0EJ67usv++PKL+DkD3R5XXq9dTafNEFERGQBGOos1KiO/gCA/04k4uTljPILdJoMtPo/YORqQGlVs8URERFRlWOos1AhXvYY1Ko+AODj9THl761TqYEnvgFcm8hQHREREVU1hjoL9lrvxlArFdh34Tp2xF6788LJpwCDvmYKIyIioirHUGfBvJ10GBPmDwCYsyEGesNterpu/QBY3Bk4/GPNFUdERERViqHOwk3s1ggOWiucSc7G37eOW1fCzgMQDUBKdM0WR0RERFWGoc7COeis8HKPRgCAz8JjkVtYXH6htmOBsZuBx76o4eqIiIioqjDU1QEjO/rB20mL5MwC/LjnYvkFFErAt0PNF0ZERERVhqGuDtColJjWR+rlunjnBaRmF9x+4YwrwIbpQPEdliEiIiKzw1BXRwxo7oVm9R2QXVCMhVvPVryQwQD89BhwYLH0bFgiIiKqNRjq6giFQsBbjwYBAH47EI8L17IrWgjo9rb0evd8IK2CS7VERERklhjq6pBODeuhR5Abig0i5m2MrXihZkOAgK5AcT6wfhpw66DFREREZJYY6uqYN/sFQSEAG08l4UhcWvkFBAHo/xmgVAPnwoGYf2q+SCIiIrpnDHV1TGN3Owxt6wMA+Hj96fKPDwOAeoFA2BTp9cY3gYIKLtUSERGRWWGoq4NeeaQxtFZKHIm7gU2nkipe6OHXAEc/IPMKsGNOzRZIRERE94yhrg5yt7fGuIcDAABzN8aiSG8ov5CVFnh0vvR6/yIgKaoGKyQiIqJ7xVBXR73QtSFcbNS4mJqDFQfjK16ocW8geAAg6oH/XpWGPCEiIiKzxFBXR9lqVJjaKxAAsGDLWWTlF1W8YN9PACsbIOEAsOltIO1CDVZJRERElcVQV4c93d4XDerZ4HpOIb7fdZuw5uANdL85dt2BRcBvT9dcgURERFRpDHV1mJVSgTf6SgMSL9l9AUkZ+RUv2HEiMOh7oEF3oPnQ0vbCXODngcDBJYD+Nmf6iIiIqEYw1NVxfULd0dbPCflFBnwRfqbihQQBaDEMGLVG6hVb4uxm4MJ26ZFiClVpe3FhtdZMRERE5THU1XGCIOCtR4MBACuPJCA2KetuK5S+9n0I6P0R0HlKaXtxIbCgqXSZ9sRKoOAu2yMiIqIqwVBHaOPnhH5NPWAQgU82xFR+RTsPoNMkoN3zpW0J+4HsZODMBmDV88CnjYA/RwHRa4GivKovnoiIiAAw1NFNb/QNgkohYHvsNew9l3r/GwroArx0AOg6HXBuKD1DNnqtFOw+bQT8PQ6I3QBcPw+kJwDZ1/jECiIioiogiBU+J8pyXL58GT4+PkhISIC3t7fc5Zi199ZG4ad9cWhW3wFrJ4ZBoRDuvtKdiCKQdAKI+huIWgVkJFS8nHd74Pnw0vdftwPybgCj1gLuoVLbseXA0Z8BlUaalGpAZS29ttICahtp6BW1DrDSAWpbwMYFaNSrdLsZVwCFEtC5AEqrB/tsREREdyBH/lDdfRGqKyb3DMTfR6/g5JUM/HPiKp5oWf/BNigIgGcLaeo1G7h8SAp4seuBvHTpLJ6+UApmZeWkAnlpgKAsbUuPky7t3ot6jYFJZULd8iFASrQUFht0k9qO/w5s/1gKg2obwM4TcPIHnAMApwDAuQHg4AMo+b8KERGZN/6lIiMXWw0mdGuITzfFYt7GWPRt6gGNSnn3FStDEACf9tLUb25pu8EAGIpNl31+ixT4nPxL25oOAdybAsUFgL5Aml9cKP0sygOKcoHCbGmYlaJcoDAHsK8olArSGb0SualAxm2eqFFCoZKCXUnQcwsG2o+71yNARERUrRjqyMTYsAD8si8OV9Lz8PPeOIzr0qB6d6hQAAq1aZtLw/LLuTaWpgfx0j7pknBZzZ8GfDsBRTnSvX0Zl4EbF4G0i6U/9QXS6xsXb9ZyS6j7ZbAUJB/9FPBoJrXlpkmXeq0dHqxmIiKiSmKoIxNatRKv9m6MN/46ga+2ncVTbb3hqFPffcXaQrjlPkFbV2m6HYMByEo0DXplg5ooAgkHgcIs6T6/Eod/ALZ9KJ3hcwuRzu65h0qv6zUGVBZ0TImIyCww1FE5T7b2xg+7LyI2OQvf7jiPt2+OY1cnKRSAQ31p8u9c8TKj10qBr+zl4pzr0s+MBGk6u6nMNlWASyMp4LmHAG6hUuhz9JP2R0REdB/Y+5UqtD02Bc8uPQS1UoGtr3WFj7NO7pJqn7wbQEoMkHxK6qCRHC29L8ioePnG/YBnVpS+j9sL1Gsi9eIlIqJahb1fyWx0a+yKsEYuiDh3HZ9tjsWCp1vJXVLto3UC/DpJUwlRBDKv3Ax4p0qDXmqs6b2EuWnA0n7S67cuAxo76XX0WiksOvoCDr6AgzdgZV1zn4mIiMwWQx1VSBAEvNUvGI99tQdrjl/F8w83QNP6vOn/gQmCFMQcvIHGvUvb9UWmT9zISiq9nFsS6ADgwPdA3B7Tbdq6S/fuOfrcDHs+0qVcRx/ptca22j4OEZGRKALizRENDMXSmKEl9zEXZEmjF1jppCGkAOn3Xm4aALF0fenFLe/LtAHS77ySsUbzM6TRDtQ2pfc7G/TS0FhAmfuohfLvtY5ShzYLwlBHt9W0vgMGtaqP1ceu4OP1MVj+fAcIt3Y0oKqhtDIdENk9BJgSKQ3bUpZ/mDTYckYCkB4v9brNTpamK4cr3nbYVOCR2dLr/ExprECPZkD9NuU7jhBR1cq+Jv1/auteelY94wqQdl76/1tfcHOopkLTnyXjeJa0qW2BHu+UbnfjW9JZ/h7vAt5tpbbYjcDWm/+vo8z/2+WCDUrfK62AcdtKl930DhAXAXR5Awh6VGq7tAdYO7F0CCrjpJd+ivrStrLeulL6j8qNbwLHfgV6vgc8/KrUlnwK+L7rvR/TycekMUQBYPfnQMQC4KGJQN+PpbasROCL0Ltv55Vo6X5pC8JQR3f0Wu/G+O9EIvaev45Np5LRt6mH3CXVLbf2ku3+dulrUZT+lZsRLwW89ITSsJd+82dBBmDrVrrO9XPAv1Olp2pMO1/aHrVK+peue1PA3othjyyXvlgawsigB3TOpe1nNgMFmUDjvqVBJHYDcDb85hiYOaU/i/LLhLGim68LpfDlFgS8uKt0uz/0Am5cAp4Ll8bpBIBTq4HNZQJaZdh5moa6y4eBywdNn72dnyHdv3svlLf8jrl+Hrh6DMhJKW0rLpA+w726NeQBMDnjBgBCSeewCs6kmbwv02YSWBVS57Nbf2cJiptn+iy620A5DHV0R95OOowJ88f3uy5g/K9H8FhzT7zWuwkC6tncfWWqXoIgdaKwcQG8bnPPY34GTH4BKq2Ahj2k+/3K/hLcPAPIvCy91joDHk0B92Y3fzYFXIM4DAuVJ4rSGaWCLOkPa0lI0hdJoQMAfB8q/a6lxADZKQDE0j+4xp+45f3Ns0I2boBvB2l9gwGI+EIKUGFTSi/jHVsOnNtSeoarop8lA5QX50vrNOgmPV2mxKpxQH46MPFQ6ZiYV45KwxPdi1vPrqu00lQ24Ni4Sp2gVOrSRx4q1bc8AlENKDWlr60dTbfbZZp0f61Xy9K2Bt2AUetw98uZYumPW//91vUNoO1YqUd+ifptgLGbpf/GCuXNn6o7vL/ZZlWmg93jXwNPfGO6L6+WwHs3KjyMldbrPWkqy8H79tsVxdLjYIH/eGWoo7ua2isQ17IKsPrYFfx7IhEbopIwtK0PpvQMhIcDb9I3a7cOfuzRDBi52rRNXyx15kg6CaSekR7RdnGXNJVQqKQ/Qh5NpW24NwXqt+bgyvfLoL/lEtutQaQAcPKT7pEEpEt4p/+V/sC3HF66ncgV0qV3kz+qZSZl2fdW0h9bRz+gXiNp/eJC4NppKUCVDQdRq6SzugWZUmArN5VpLwkr7Z4H+n8mvS7IApb2lV7PSC29tWDXp9Ll/3sR9Bjgu1x6LQjS+I+iAWj7bGmoSzoBnFp1b9stew8rIIXPwhzTe6wCHpbO+KhtpEljJ/0see60UiN9tpIwplSbBhkAmFjB4w1bDJOmB1H2ntwSdu7S9CDqty7fpnUsDdb3y1wClCCYTy3VgKGO7kqnVuGLYS0x7uEGmL85FttOp+D3g/FYdfQyxnTyx/iuDeFkw7M4tZZSBTy5RHpdlA9ciwGSooDkqJs/T968rHNKmk78IS07bDkQ/Jj0OuZf4MhS6Sxgx4ml2z79n3Tmz6aedMnX2tE8xuIrygNyr0vPF7b3LG0/t+Xm5bU8aSoJWSWX2PSFpZfZ9DfbW42U/vgDQMIhYPO70v0+gxaVbndJT2ng6pLAZii6e429ZgOdp0qvM+Kly+YOPqah7uD3wJUj9/bZO04C+nwkvc5OBr57GFDbAW9fLl3m2K/A+a33tl19mTNUChXg3LD8H097L+msL0r+sJb9ifLtSjVQL7B0fUEA2oyR5pe9bBg8QDrmKo1p4DK+t5buRdXYSvemqW3KP3P6mT/Kf6aALtJEVEsw1FGlhXjZ48cx7XDoUhrmbTyNQ5du4LtdF/DbgXi80KUBxnYOgI2GX6lazcpaupRb9nKuKEr36hmD3knpZ8kj0QBpSJZzWwDbMvdcFuYCK54x3b6gkEKezuVm0Lv5WldPOuNhKAZajyo9Q3Vxt3T/kWcLoM1oqc2gB/4ae/PmbH0FN27rb4ax/NJgVpQPPPm/0rMbp9YAa8ZLIbTsmcs/x0hPB7kXvh1LQ11hFpCwX7rMV1beDSlEVkRQVBxEyp4F1bkATfqXf/pJ477SGVRDUekx0BffckxK2oukY2PnaboNWw/p8p4oloawwEek3tMaO0Bjf/MMle3N92XaSia1rWlYt7YHJh8t/1l7fyhND+KxL8q3+Xe+/eDgRHUI/wLTPWvn74w/X+yIHbHXMHfjaZxOysJn4Wfw075LmNS9EYZ38IVGZVndxOs0QZBClqNvaW+4WzV5VOrd5+hX2laUC3i3k8JMznWp04ZoAHJTpSk1tuJtNeheGupSYqR7mkIHlYY6CED0mnv/HGWDlpW1dDlSuOWsoXdbKQSWnNkpOctTcu9T2amkzbtd6fruzYChv5jegA8Aw3+XPntJYCsb3pSV+DXs5A8M/618e9c3Kv3xK+ToA7xewX+HhyY82HaJSBZm/0SJK1euYPr06diwYQNyc3PRqFEjLF26FG3btq3U+nyiRPUyGET8c+IqPtt8BvFpuQAAbyctXunVGANb1YdSYbn3LtA9Ki6U7tfLvS6NIZV7vXTKSZXOvClUQPsXS+/5unwEOLtZumk7dGDptg58V8EN2jdv0haUN0OZtemlNzvP0l6NZc9KERFVAznyh1mHuhs3bqBVq1bo3r07JkyYAFdXV5w9exYNGzZEw4YN774BMNTVlMJiA/44nICFW8/iWlYBAKCxuy1e790Ej4S4c3w7IiKqU2pNqEtISIAgCMYiDx48iN9++w0hISF44YUXqqy4N998ExEREdi9e/d9b4OhrmblFeqxbO8lLNpxDpn5Uq+4Vr6OeKNPEDo25DNMiYiobpAjf9xXN7RnnnkG27dvBwAkJSXhkUcewcGDB/HOO+/g/fffr7Li1q1bh7Zt2+Kpp56Cm5sbWrVqhSVLltxxnYKCAmRmZhqnrKx7vOmZHohWrcSEbg2x+40eeKlbQ1hbKXAsPh3Dl+zHyB8O4OTl2zzMnoiIiB7IfYW6qKgotG8vjYz9559/omnTpti7dy+WL1+OZcuWVVlxFy5cwKJFixAYGIhNmzZhwoQJmDx5Mn766afbrjNnzhw4ODgYp5CQkCqrhyrPQWeFN/oGYde07hj5kB9UCgG7z6ZiwNd7MHH5UZxLYdgmIiKqSvd1+dXW1hZRUVHw9/fH448/jrCwMEyfPh3x8fFo0qQJ8vLy7r6RSlCr1Wjbti327t1rbJs8eTIOHTqEffv2VbhOQUEBCgoKjO+vXLmCkJAQXn6VWdz1HHwRfgZrI68aB/PuHeKOl7o3QksfR1lrIyIiqmq15vJraGgoFi9ejN27dyM8PBx9+0ojh1+9ehUuLlV335Snp2e5M23BwcGIj4+/7ToajQb29vbGyc7Orsrqofvn52KDBU+3wvrJD6N3iDTi+eboZAz8JgIj/rcfe86mwoz77BAREZm9+wp1c+fOxXfffYdu3bph+PDhaNGiBQDpHriSy7JVISwsDLGxpmMonTlzBn5+frdZg8xdsKc9vh/VFuGvdMHg1tKQJxHnruP/fjiAgd9EYGNUIgwGhjsiIqJ7dd9Dmuj1emRmZsLJycnYdunSJeh0Ori5uVVJcYcOHUKnTp0we/ZsDB06FAcPHsS4cePw/fffY8SIEZXaBnu/mrfLN3KxZNcFrDiUgIJiAwCgoasNxndtiIGt6sNKaQaPlCIiIrpHtWZIk7y8PIiiCJ1OenBxXFwcVq9ejeDgYPTp06dKC/z333/x1ltv4ezZswgICMCrr76KcePGVXp9hrraITW7AEsjLuLnfXHIujkUipeDNcZ1aYCn2/lCq+YTKoiIqPaoNaGud+/eGDx4MMaPH4/09HQEBQXBysoKqamp+PzzzzFhgvk8YoahrnbJyi/Cr/vj8cOei0jNljq8ONuo8Wwnf4zq6A8HnZXMFRIREd1drekocfToUTz8sPQA67/++gvu7u6Ii4vDzz//jIULF1ZpgVS32FlbYUK3htgzvTs+HNgUPs5apOUU4rPwMwibuw1z1scgJTNf7jKJiIjMzn2FutzcXGOv0s2bN2Pw4MFQKBR46KGHEBcXV6UFUt1kbaXE/z3kh+2vdcOXT7dEkIcdsguK8d2uC+g8bzveXn0Scddz5C6TiIjIbNxXqGvUqBHWrFmDhIQEbNq0Cb179wYApKSkwN7evkoLpLpNpVTgiZb1sWHKw/hhdFu08XNCYbEBvx2IR/f5OzD592NYe/wKDl5MQ0JaLor0BrlLJiIikoXqflaaOXMmnnnmGbzyyivo0aMHOnbsCEA6a9eqVasqLZAIAARBQM9gd/QIcsPBi2n4dsd57DxzDesir2Jd5NUyywFudhp4Omjh5WgNLwctPB218HKwhpejFp6O1qhno4FCIcj4aYiIiKrefQ9pkpSUhMTERLRo0QIKhXTC7+DBg7C3t0dQUFCVFvkg2FHCcp26moFf98fjYmo2EjPykZiej8JKnKlTKxVwd9DAy0ErBT0Ha3g6auHtqMVDDVzY05aIiB5Yren9Wtbly5cBwGwDE0Nd3WEwiLieU4jEjDxcTc/D1fR86XVGPq6m5yExPR8pWfm409jGAfVs8PPY9vBx1tVc4UREZHHkyB/3dfnVYDDgww8/xGeffYbs7GwAgJ2dHV577TW88847xjN3RDVJoRDgaqeBq50Gzb0dK1ymSG9AcmY+EkuC3s2fV9PzcTzhBi6m5uDJRXvx83PtEeTB+0OJiKj2uK9Q98477+CHH37AJ598grCwMADAnj17MGvWLOTn5+Ojjz6q0iKJqoqVUgFvJx28ncqfiUvKyMfoHw8iNjkLQxfvww9j2qGdv7MMVRIREd27+7r86uXlhcWLF+Pxxx83aV+7di1eeuklXLlypcoKfFC8/Er3IiO3CM/9dAiH425Ao1Lgm2dao1eIu9xlERFRLVNrBh9OS0ursDNEUFAQ0tLSHrgoIrk46Kzwy3Md0DPIDQXFBrz46xH8eThB7rKIiIju6r5CXYsWLfD111+Xa//666/RvHnzBy6KSE5atRKLR7bBk629oTeIeOOvE1i88zwesE8RERFRtbqve+rmzZuH/v37Y8uWLcYx6vbt24eEhASsX7++SgskkoOVUoH5TzVHPVs1vtt1AZ9sOI3r2QV4q18wx7gjIiKzdF9n6rp27YozZ85g0KBBSE9PR3p6OgYPHoxTp07hl19+qeoaiWQhCALeejQYbz8q3WqwZPdFvL4ykk+tICIis/TA49SVFRkZidatW0Ov11fVJh8YO0pQVfj7yGW88fcJ6A0iujdxxTcjWkOnvq8T3UREVAfUmo4SRHXNk228sWRUG1hbKbA99hr+738HkJ5bKHdZRERERgx1RJXUI8gdy5/vAHtrFY7Gp+OpxfuQmJEnd1lEREQAGOqI7kkbP2esHN8J7vYanE3JxpPf7sW5lGy5yyIiIrq33q+DBw++4/z09PQHqYWoVmjiYYe/J3TCqB8P4sK1HDy1eC9+HNMOrXyd5C6NiIjqsHs6U+fg4HDHyc/PD6NGjaquWonMhreTDn+N74QW3g64kVuEZ5YcwM4z1+Qui4iI6rAq7f1qjtj7lapTTkExxv96BLvPpkKlEPDZ0BZ4omV9ucsiIiKZsfcrUS1jo1Hhh9HtMKCFF4oNIqasOI6lERflLouIiOoghjqiB6RWKfDlsJYY08kfADD7n2h8uuk0HytGREQ1iqGOqAooFALeGxCC13s3BgB8s/083lt3isGOiIhqDEMdURURBAGTegTi40HNIAjAz/vi8O2O83KXRUREdQRDHVEVe6aDL957LAQA8OmmWKw+dlnmioiIqC5gqCOqBmPCAjDu4QAAwBt/ncDec6kyV0RERJaOoY6omrzVLxj9m3uiSC/ixV+O4HRSptwlERGRBWOoI6omCoWAz55qgfb+zsgqKMaYHw/xWbFERFRtGOqIqpG1lRLfj2qDhq42SMrMx7NLDyEzv0jusoiIyAIx1BFVM0edGsuebQ9XOw1OJ2Vhwq9HUFhskLssIiKyMAx1RDXAx1mHpWPaQadWIuLcdbz59wmOYUdERFWKoY6ohjSt74BvR7SGUiFg1bEr+GzzGblLIiIiC8JQR1SDujVxw8eDmgIAvt5+Dr8diJe5IiIishQMdUQ1bFg7X0zuGQgAeHfNSWw7nSxzRUREZAkY6ohk8EqvQAxp4w2DCExcfgwnLqfLXRIREdVyDHVEMhAEAXMGN8PDgfWQV6TH2GWHEH89t1r3WVCsx/bTKbiazrHyiIgsEUMdkUyslAp8O6I1gj3tkZpdiDFLD+JGTmGV7ycpIx+fbY5F2Cfb8OyyQxiyaC+yC4qrfD9ERCQvhjoiGdlZW2HZs+3g5WCNC6k5GPfzYeQX6R94u6Io4tClNEz87Sg6z92Gr7adQ2q2FBiv3gx5RERkWRjqiGTmbm+NZWPbw85ahcNxN/DKH8dhMNzfGHb5RXr8cSge/RfuwVOL9+G/E4koNohoH+CMb0e0xtIx7QAAP+29xPv4iIgsjEruAogIaOxuh+9HtsXoHw9iQ1QSPvwvBjMHhFR6/cs3cvHL/jj8cSgB6bnSY8isrRQY2LI+RnX0R4iXvXHZJ1p6Ye3xq3jz75NYNykMKiX/bUdEZAkY6ojMRMeGLvj0qeaYsuI4foy4CC9Hazz/cIPbLi+KIvaev46f9l7ClphklJzc83bSYuRDfhjWzgeOOnW59WY8FoIdsdcQnZiJpRGXMK7L7fdBRES1B0MdkRl5omV9XE3Px9yNp/HR+hh4OWrxaDNPk2VyCoqx6tgV/Lz3Es6mZBvbOzeqh1Ed/dAz2B1KhXDbfdSz1eCdR4Pxxt8n8Hn4GfRt6gEfZ121fSYiIqoZDHVEZmZ81wa4mp6HX/bHYeofx+Fqp0E7f2dcSs3Bz/visPJIArLypd6rOrUST7b2xuhOfmjkZlfpfTzV1ht/H72MAxfTMGNtFJaOaQdBuH0QJCIi88dQR2RmBEHArMdDkZiRjy0xyXj+p8No5euIHbHXjMv4u+gwqqM/hrT1hr211X3t4+PBzdBvwW7siL2Gf08kYkALr6r8GEREVMN4hzSRGVIqBHw1vBVa+DgiI6/IGOi6NXHF0mfbYdtr3TC2c8B9BboSDV1tMbF7IwDA7H+ikXGzgwUREdVOPFNHZKa0aiV+GN0Ws9adgpudNUZ29ENAPZsq3cf4bg2wLvIKzl/LwScbYzBncPMq3T4REdUcnqkjMmP1bDX4+pnWmDkgpMoDHQBoVEpjkPv9YAIOXkyr8n0QEVHNYKgjquPaBzjj6XY+AIC3V59EQfGDP9GCiIhqHkMdEeGtfsGoZ6vGuZRsfLfzgtzlEBHRfWCoIyI46Kwwc0AoAODrbedw/lr2XdYgIiJzw1BHRACAAc090bWxKwr1Bryz+iRE8f6eP0tERPJgqCMiANLYdR8ObAprKwX2X0jDyiOX5S6JiIjuQa0KdZ988gkEQcDUqVPlLoXIIvk46/BKr8YAgI/XxyA1u0DmioiIqLJqTag7dOgQvvvuOzRvznG0iKrT2M4BCPa0R3puET78N1rucoiIqJJqRajLzs7GiBEjsGTJEjg5OcldDpFFs1Iq8MngZhAEYM3xq9h99trdVyIiItnVilA3ceJE9O/fH7169ZK7FKI6oYWPI0Z39AcAvLM6CnmFHLuOiMjcmX2oW7FiBY4ePYo5c+ZUavmCggJkZmYap6ysrGqukMgyvd6nCTwdrBGflouF287KXQ4REd2FWYe6hIQETJkyBcuXL4e1tXWl1pkzZw4cHByMU0hISDVXSWSZbDUqzH5cGrvu+10XEJOYKXNFRER0J4JoxoNRrVmzBoMGDYJSqTS26fV6CIIAhUKBgoICk3mAdKauoKC0x96VK1cQEhKChIQEeHt711jtRJZi/C9HsPFUElr6OOLvCZ2gVAhyl0REZPYuX74MHx+fGs0fqhrZy33q2bMnTp48adL27LPPIigoCNOnTy8X6ABAo9FAo9EY32dm8uwC0YOY9Xgo9pxLxfGEdCw/EIdRN++1IyIi82LWoc7Ozg5NmzY1abOxsYGLi0u5diKqHh4O1nijbxPMXHsK8zbGoneIBzwcKnc7BBER1RyzvqeOiMzDiA5+aOnjiOyCYry3LkrucoiIqAK1LtTt2LEDCxYskLsMojpFqRAwZ3AzqBQCNp1KxuZTSXKXREREt6h1oY6I5BHsaY9xXRoAAGauPYWs/CKZKyIiorIY6oio0qb0DISvsw5Jmfn4bPMZucshIqIyGOqIqNKsrZT4aJDUSemnfZdwPCFd3oKIiMiIoY6I7snDga4Y2NILogi8/PtRxF3PkbskIiICQx0R3YcZj4XAx1mLhLQ8DP52L05cTpe7JCKiOo+hjojumYutBn9P6IRQL3tczynE09/vx47YFLnLIiKq0xjqiOi+uNlZY8ULD6Fzo3rILdTj+Z8O468jl+Uui4iozmKoI6L7ZmdthR/HtMPAll4oNoh4fWUkvtl+Dmb8SGkiIovFUEdED0StUuDzoS3xYldpDLtPN8XivXWnoDcw2BER1SSGOiJ6YAqFgLf6BeO9ASEQBODnfXGYuPwo8ov0cpdGRFRnMNQRUZV5NiwAXw1vBbVSgY2nkjDqh4PIyOWTJ4iIagJDHRFVqceae+Gnse1hZ63CwUtpGLJ4L66m58ldFhGRxWOoI6Iq17GhC1aO7wh3ew3OpmRj8Ld7EZuUJXdZREQWjaGOiKpFkIc9Vr0UhkZutkjKzMeQxXux/8J1ucsiIrJYDHVEVG3qO2rx1/iOaOvnhKz8Yoz64SDWn0yUuywiIovEUEdE1cpRp8avz3dAn1B3FOoNmPjbUSyLuCh3WUREFoehjoiqnbWVEt+OaIORD/lBFIFZ/0Tjkw2nOUgxEVEVYqgjohqhVAh4/4lQTOvTBACweOd5vPZnJAqLDTJXRkRkGRjqiKjGCIKAid0b4dMhzaFUCFh17Aqe++kQsguK5S6NiKjWY6gjohr3VFsf/G90W2itlNh9NhVPf78P17IK5C6LiKhWY6gjIll0b+KGFS88BBcbNaKuZGLYd/twPZvBjojofjHUEZFsWvg44u8JnVDfUYsLqTkY+9Nh5PBSLBHRfWGoIyJZ+dezwU9j28NJZ4XIhHRMWH6UnSeIiO4DQx0Rya6Rmy1+HNMOWisldp25hjf+ioTBwOFOiIjuBUMdEZmFVr5OWPR/raFSCFhz/Co+Wh/DceyIiO4BQx0RmY1uTdwwb0hzAMAPey7i+10XZK6IiKj2YKgjIrMyuLU33nk0GAAwZ8Np/HXksswVERHVDgx1RGR2xnVpgBe7NAAATP/7BLadTpa5IiIi88dQR0RmaXrfIAxuXR96g4iXlh/FkbgbcpdERGTWGOqIyCwpFALmPtkc3Zq4Ir/IgLHLDuFscpbcZRERmS2GOiIyW1ZKBb4d0RotfRyRkVeEUT8exNX0PLnLIiIySwx1RGTWdGoVlo5ph4auNkjMyMeoHw8iPbdQ7rKIiMwOQx0RmT0nGzV+fq4DPOytcS4lG2OXHUJeoV7usoiIzApDHRHVCvUdtfj5ufawt1bhaHw6Jv52FEV6Pk6MiKgEQx0R1RqN3e3w45h20KgU2HY6BW+tOsmnThAR3cRQR0S1Slt/Z3zzTGsoFQL+OnIZczfGyl0SEZFZYKgjolqnV4g75gxqBgBYvPM8/rebjxMjImKoI6JaaWg7H7zRtwkA4MP/YrDm2BWZKyIikhdDHRHVWhO6NsSzYf4AgNdXRmLXmWvyFkREJCOGOiKqtQRBwIz+IXi8hReKDSLG/3oExxPS5S6LiEgWDHVEVKspFALmP9UCnRvVQ26hHmOXHcL5a9lyl0VEVOMY6oio1lOrFFg8sg2aezsgLacQT3wdgQVbziC7oFju0oiIagxDHRFZBFuN9DixFj6OyC4oxoItZ9Fl3nb8sOci8ov49AkisnwMdURkMVxsNVg9oRO+eaY1GtSzQVpOIT74Nxo9P9uJPw8noJhPoCAiC8ZQR0QWRaEQ0L+5Jza/0gWfDG4GD3trXEnPwxt/nUDfL3djY1Qin0JBRBaJoY6ILJJKqcDT7X2xY1o3vPNoMBx1VjiXko3xvx7FwG/3Yu+5VLlLJCKqUgx1RGTRrK2UGNelAXa90R2TezSCTq1EZEI6nvnfAfzf/w4gkkOgEJGFYKgjojrB3toKr/Zugp3TumNMJ39YKQXsOZeKJ76JwIRfj+BcCodBIaLajaGOiOoUVzsNZj0eim2vdcPg1vUhCMCGqCT0/mIn3vgrElfS8+QukYjovjDUEVGd5OOsw+dDW2LjlC54JMQdBhH48/BldJ+/Ax/8G43r2QVyl0hEdE8Y6oioTmviYYclo9pi1Uud8FADZxQWG/DDnovoMm87Fmw5g6z8IrlLJCKqFEG08L79ly9fho+PDxISEuDt7S13OURkxkRRxO6zqZi36TSirmQCANRKBTo1csEjIe7oFewOd3trmaskotpAjvzBUEdEdAuDQcSGqCQs2HIGZ2/pQNHCxxGPBLvhkRAPNHa3hSAIMlVJROaMoe4Wc+bMwapVq3D69GlotVp06tQJc+fORZMmTSq9DYY6IrpfoijiXEo2NkcnIzw6GcdvGf7E11lnPIPXzt8JKiXvaCEiCUPdLfr27Yunn34a7dq1Q3FxMd5++21ERUUhOjoaNjY2ldoGQx0RVZWUrHxsjUlBeHQy9pxLRWFx6WPHHHVW6NHEDY+EuOPhxq6w1ahkrJSI5MZQdxfXrl2Dm5sbdu7ciS5dulRqHYY6IqoOOQXF2H02FeHRydh2Ohk3cks7VPA+PCKSI3/Uqn9KZmRkAACcnZ1lroSI6jobjQp9m3qgb1MPFOsNOBJ3A+HRyQiPSUbc9VzsiL2GHbHX8M7qKLTwdsAjIe7oGeyOIA873odHRNWi1pypMxgMePzxx5Geno49e/bcdrmCggIUFJSOL3XlyhWEhITwTB0R1Yi73Yfn5WCNbkFu6NHEDZ0auUCnrlX/tiaiSuLl1zuYMGECNmzYgD179tzx4MyaNQuzZ88u185QR0RyKHsfXsS5VBSUuQ9PrVKgYwMX9AhyQ/cmbvB10clYKRFVJYa625g0aRLWrl2LXbt2ISAg4I7L8kwdEZmr/CI99p2/jm2nU7DtdEq5R5I1dLWRAl6QG9r5O8OKvWmJai2GuluIooiXX34Zq1evxo4dOxAYGHjP22BHCSIyRyWXaUsC3uG4G9AbSn8d22lU6BxYD92D3NCtiSvc7NjZgqg2YUeJW0ycOBG//fYb1q5dCzs7OyQlJQEAHBwcoNVqZa6OiOj+CYKAQHc7BLrb4cWuDZGRV4TdZ69h++lr2BGbgus5hdgQlYQNUdLvvebeDujexA09gtzQrL4DFAp2tiAiU2Z9pu52PcSWLl2KMWPGVGobPFNHRLWNwSDixJUMbDudgu2nU3DySobJ/Hq2arT2dYKfiw5+LjbST2cbeDlacwBkIjPBM3W3MOO8SURUbRQKAS19HNHSxxGvPtIYKVn52BF7DdtPp2D32VSkZhdic3RyufVUCgH1nbTwddbB/2bY83WWgp+vsw5atVKGT0NENcWsQx0REQFudtYY2tYHQ9v6oLBYGhPvTHIW4q7nIj4tB3HXcxGXlovCYoP0+noudp9NrWA7Gvi72MDXRQc/Z53008UGQR52sLZi4COq7RjqiIhqEbVKgY4NXdCxoYtJu8EgIjkrXwp613MRl5aDSyWvr+cgM78YKVkFSMkqwMFLaSbrejpY4+tnWqGNHwd2J6rNGOqIiCyAQiHA00ELTwctHmrgUm5+em4hLt0MePE3z+zFX8/FmZQsJGbkY9h3+/FmvyA81zmAT7wgqqUY6oiI6gBHnRotdWq09HE0ac8uKMabf5/AvycS8eF/MTh86QbmPdUc9tZW8hRKRPeN3aSIiOowW40KXw1vhfefCIWVUsDGU0l4/Ks9OHU14+4rE5FZYagjIqrjBEHAqI7+WDm+E+o7anHpei4Gf7sXfx5KkLs0IroHDHVERAQAaOnjiH9f7ozuTVxRUGzAG3+fwLSVkcgr1MtdGhFVAkMdEREZOdmo8cPodpjWpwkUArDyyGUM+jYCF1Nz5C6NiO6CoY6IiEwoFAImdm+EX5/vgHq2apxOysKAr/Zg/clEuUsjojtgqCMiogp1algP/01+GO39nZFdUIyXlh/F7H9OobDYIHdpRFQBhjoiIrotd3tr/DauA17s2gAAsDTiEoZ9vw9X0/NkroyIbsVQR0REd6RSKvBWv2AsGdUWdtYqHItPR/+Fu7HzzDW5SyOiMhjqiIioUh4Jccd/Lz+MpvXtcSO3CGOWHsQX4WegN4hyl0ZEYKgjIqJ74Ouiw1/jO+GZDr4QReDLrWcx+seDuJ5dIHdpRHUeQx0REd0TayslPh7UDF8MawGtlRJ7zqWi/8I9OHwpTe7SiOo0PvuViIjuy6BW3gj1csD4X4/gwrUcPP39fgxuXR8uthrYalTQqZWw0ahgo1bBRqO82aaSft58r1EpIAiC3B+FyCIw1BER0X1r7G6HdZM6461VJ/FP5FX8efjyPa2vVAjQqZXGEFgS/Gw0KrTzd8JznQOgUvKiElFlMNQREdEDsdWosPDplujfzAOnrmYiu6AYuQV6ZBcWI7egGDkFeuQUFiOnoBjZBXrkFhYj9+ajx/QGEVn5xcjKLy633S0xydh2OgVfDW8FN3vrmv5YRLUOQx0RET0wQRDQt6kn+jb1rNTyBoOI3CI9cgqKb06lwS+nUI/E9Dx8te0cDlxMw6MLd2Ph063QqVG9av4URLUbQx0REdU4hUKArUa6v+52Hglxx0vLj+J0Uhb+74cDmNqrMSZ1bwSFgvfgEVWENyoQEZFZauBqizUTwzCsrQ8MIvB5+BmMWXaIw6cQ3QZDHRERmS1rKyXmDmmO+U+1gLWVArvOXEP/hXtwJI7DpxDdiqGOiIjM3pA23lg7sTMauNogKTMfw77bjyW7LkAU+TQLohIMdUREVCs08ZCGTxnQwgvFBhEfrY/BC78cQUZekdylEZkFhjoiIqo1SoZP+WBgU6iVCoRHJ+Oxr3bj5OUMuUsjkh1DHRER1SqCIGDkQ374e0In+DhrkZCWhycX7cUv++N4OZbqNIY6IiKqlZp5O+DfSQ/jkRB3FOoNmLEmClNWHEd2QfmBjInqAoY6IiKqtRx0Vvh+ZBu82z8YKoWAdZFX8fjXexCblCV3aUQ1joMPExFRrSYIAp5/uAFa+jhi0m/HcOFaDp74Zg8+HNgMQ9p4y12eibScQuw7fx17zqVi3/lUKBQCega54ZEQD7Txc4KSAyvTAxBEC78B4fLly/Dx8UFCQgK8vc3rf24iIqpa17ML8Mqfkdh15hoAYFhbH8x+IhTWVkpZ6skr1OPQpTREnEvFnnOpiE7MxO3+6rrYqNEjyA29Qz3wcGA92WqmqiFH/mCoIyIii2IwiPhm+zl8seUMDCIQ5GGHOYObobG7HWzu8FiyqlCsN+DklQxEnEtFxLnrOBJ3A4V6g8kyjd1tEdaoHsIa1kOR3oDw6GRsPZ1iMjSLtZUCXQJd0TvUAz2C3OBso67WuqnqMdRVA4Y6IqK6ae+5VExecQyp2YXGNkedFeo7aqXJSfrp7aSF1802Zxs1BKHyl0BFUcT5azk3Q1wq9l24jqx8044ang7WCGtUD50b1UOnhi5ws7cut50ivQGHLqZhc3QywqOTcSU9zzhPIQDt/J3xSIg7eod4wNdFdx9Hg2oaQ101YKgjIqq7UjLz8fbqKBy4WD5sVURrpYSXozXqO+nKBD5r1HfUob6TFu52GqTlFCLifCr2nL2OiHOpSMrMN9mGvbUKnRrWQ1gjF4Q1qoeAejb3HBSjEzOx+ZQU8KITM03mB3nYoXeIOx4J8UDT+vb3tG2qOQx11YChjoiIACAzvwhX0/Nw5UYertz8eTk9z9iWklVw120oFQL0BtM/m2qVAu38ndCpoXQ2rml9hyrt8JCQlostMcnYfCoZBy+lmezfy8EavW6ewevQwBlWSg5qYS4Y6qoBQx0REVVGQbEeien5xsB3JT3P5HViRh6K9CIEAWhW38F4X1xbf6ca69SQnluIbadTsPlUMnaeuYa8Ir1xnp21ytiTtmsTV9hW8/2DdGcMddWAoY6IiKqCwSDiWnYBrK2UcNBayV0O8ov0iDiXivDoZGyJSTa5d1CtVCCskQt6h3qgZ7Ab3OzK38dH1UuO/MEYT0REVAkKhQD3Cjo5yMXaSomewe7oGewOvUHEsfgbCI9OxqZTSbh0PRfbY69he+w1CALQ2tfpZkcLdzRwtZW7dKomPFNHRERkQURRxLmUbGyOTsbmU0mIvJxhMr+Rmy16h7ijd6gHmtd3gKIK7/8r1huQmJGPhLRcJNzIRUKa1Is32NMeoV728HXWVen+zBkvv1YDhjoiIqrLEjPysCU6GZujk7Hv/HUUl+lo4W6vQa9gKeB1bOACterOHS1EUboEnZCWh8s3cqXwlpaHhBu5iE/LRWJGfrmOJGXZalQI9rRDiKc9Qr0cEOJlj0B3W2hUljfQMkNdNWCoIyIikmTkFWFHbAo2Rydjx+kU5BSW6WihUaFbkBt6h7gjoJ4NLt8oE9xu5BnPvuUXGe6wB6k3sLeTFj5OOvg4a1GsFxGTmImYpCwUFpdfV6UQ0MjN1hjyQr3sEexpXyX3LYqiiOyCYtzIKUJabiHScgqQllNk/DmpR6Nq61DCUFcNGOqIiIjKKyjWY+/568bx8FKz7z6kCwAIAuBpbw1vZx18nXXG8OZz87WbnabCS6zFegPOX8tBdGIGTl3JRHRiJk5dzTR5kkZZPs7a0jN6nvYIrW+PerYa3MgtRFpO6XQjpxDXb/5Myy0yCW43corKPdGjrG2vda22ewwZ6qoBQx0REdGdGQwijl9OvxnwknAjtwg+Tlp4lwltJQHOy1F718u0lSWKIq5m5OPUlQxjyIu+mmnyRI2qoLVSwtlGDWcbNZxs1HCxUcNJp8a4LgHwdNBW6b5KsPcrERER1TiFQkBrXye09nXCm/2Camy/giAYH9vWO9TD2J6eW4joRCngRV+Vwt65a9nQG6RxAp10ajjprOBio4GTjRWcbTRwvvWnTg1nWzWcdWpo1ZZ3z15FGOqIiIjIrDjq1OjUsB46NaxnbMsv0iO3UA8HrVWVPrHDkjDUERERkdmztlLW2JM7ais+JI6IiIjIAjDUEREREVkAhjoiIiIiC8BQR0RERGQBGOqIiIiILABDHREREZEFYKgjIiIisgAMdUREREQWoFaEum+++Qb+/v6wtrZGhw4dcPDgQblLIiIiIjIrZh/q/vjjD7z66qt47733cPToUbRo0QJ9+vRBSkqK3KURERERmQ2zD3Wff/45xo0bh2effRYhISFYvHgxdDodfvzxR7lLIyIiIjIbZh3qCgsLceTIEfTq1cvYplAo0KtXL+zbt6/CdQoKCpCZmWmcsrKyaqpcIiIiItmYdahLTU2FXq+Hu7u7Sbu7uzuSkpIqXGfOnDlwcHAwTiEhITVRKhEREZGsVHIXUNXeeustvPrqq8b3CQkJaNq0KRITE2WsioiIiOqSktxhMBhqbJ9mHerq1asHpVKJ5ORkk/bk5GR4eHhUuI5Go4FGozG+z83NBQC0b9+++golIiIiqkBycjJ8fX1rZF9mHerUajXatGmDrVu3YuDAgQCkxLt161ZMmjSpUtto1aoVDh48CHd3dygUZn21uUpkZWUhJCQE0dHRsLOzk7scs8JjUzEel9vjsbk9HpuK8bjcXl07NgaDAcnJyWjVqlWN7dOsQx0AvPrqqxg9ejTatm2L9u3bY8GCBcjJycGzzz5bqfVVKhXatWtXzVWaj8zMTABA/fr1YW9vL3M15oXHpmI8LrfHY3N7PDYV43G5vbp4bGrqDF0Jsw91w4YNw7Vr1zBz5kwkJSWhZcuW2LhxY7nOE0RERER1mdmHOgCYNGlSpS+3EhEREdVFln+TWR2j0Wjw3nvvmXQWIQmPTcV4XG6Px+b2eGwqxuNyezw21U8QRVGUuwgiIiIiejA8U0dERERkARjqiIiIiCwAQx0RERGRBWCosxBz5sxBu3btYGdnBzc3NwwcOBCxsbFyl2V2PvnkEwiCgKlTp8pdilm4cuUK/u///g8uLi7QarVo1qwZDh8+LHdZstLr9ZgxYwYCAgKg1WrRsGFDfPDBB6iLtx/v2rULAwYMgJeXFwRBwJo1a0zmi6KImTNnwtPTE1qtFr169cLZs2flKbaG3enYFBUVYfr06WjWrBlsbGzg5eWFUaNG4erVq/IVXIPu9r0pa/z48RAEAQsWLKix+iwZQ52F2LlzJyZOnIj9+/cjPDwcRUVF6N27N3JycuQuzWwcOnQI3333HZo3by53KWbhxo0bCAsLg5WVFTZs2IDo6Gh89tlncHJykrs0Wc2dOxeLFi3C119/jZiYGMydOxfz5s3DV199JXdpNS4nJwctWrTAN998U+H8efPmYeHChVi8eDEOHDgAGxsb9OnTB/n5+TVcac2707HJzc3F0aNHMWPGDBw9ehSrVq1CbGwsHn/8cRkqrXl3+96UWL16Nfbv3w8vL68aqqwOEMkipaSkiADEnTt3yl2KWcjKyhIDAwPF8PBwsWvXruKUKVPkLkl206dPFzt37ix3GWanf//+4tixY03aBg8eLI4YMUKmiswDAHH16tXG9waDQfTw8BA//fRTY1t6erqo0WjE33//XYYK5XPrsanIwYMHRQBiXFxczRRlJm53bC5fvizWr19fjIqKEv38/MQvvviixmuzRDxTZ6EyMjIAAM7OzjJXYh4mTpyI/v37o1evXnKXYjbWrVuHtm3b4qmnnoKbmxtatWqFJUuWyF2W7Dp16oStW7fizJkzAIDIyEjs2bMH/fr1k7ky83Lx4kUkJSWZ/D/l4OCADh06YN++fTJWZp4yMjIgCAIcHR3lLkV2BoMBI0eOxLRp0xAaGip3ORalVjxRgu6NwWDA1KlTERYWhqZNm8pdjuxWrFiBo0eP4tChQ3KXYlYuXLiARYsW4dVXX8Xbb7+NQ4cOYfLkyVCr1Rg9erTc5cnmzTffRGZmJoKCgqBUKqHX6/HRRx9hxIgRcpdmVpKSkgCg3CMb3d3djfNIkp+fj+nTp2P48OF15pmndzJ37lyoVCpMnjxZ7lIsDkOdBZo4cSKioqKwZ88euUuRXUJCAqZMmYLw8HBYW1vLXY5ZMRgMaNu2LT7++GMAQKtWrRAVFYXFixfX6VD3559/Yvny5fjtt98QGhqK48ePY+rUqfDy8qrTx4XuT1FREYYOHQpRFLFo0SK5y5HdkSNH8OWXX+Lo0aMQBEHuciwOL79amEmTJuHff//F9u3b4e3tLXc5sjty5AhSUlLQunVrqFQqqFQq7Ny5EwsXLoRKpYJer5e7RNl4enoiJCTEpC04OBjx8fEyVWQepk2bhjfffBNPP/00mjVrhpEjR+KVV17BnDlz5C7NrHh4eAAAkpOTTdqTk5ON8+q6kkAXFxeH8PBwnqUDsHv3bqSkpMDX19f4OzkuLg6vvfYa/P395S6v1uOZOgshiiJefvllrF69Gjt27EBAQIDcJZmFnj174uTJkyZtzz77LIKCgjB9+nQolUqZKpNfWFhYuWFvzpw5Az8/P5kqMg+5ublQKEz/vatUKmEwGGSqyDwFBATAw8MDW7duRcuWLQEAmZmZOHDgACZMmCBvcWagJNCdPXsW27dvh4uLi9wlmYWRI0eWu7e5T58+GDlyJJ599lmZqrIcDHUWYuLEifjtt9+wdu1a2NnZGe9pcXBwgFarlbk6+djZ2ZW7r9DGxgYuLi51/n7DV155BZ06dcLHH3+MoUOH4uDBg/j+++/x/fffy12arAYMGICPPvoIvr6+CA0NxbFjx/D5559j7NixcpdW47Kzs3Hu3Dnj+4sXL+L48eNwdnaGr68vpk6dig8//BCBgYEICAjAjBkz4OXlhYEDB8pXdA2507Hx9PTEkCFDcPToUfz777/Q6/XG38nOzs5Qq9VylV0j7va9uTXgWllZwcPDA02aNKnpUi2P3N1vqWoAqHBaunSp3KWZHQ5pUuqff/4RmzZtKmo0GjEoKEj8/vvv5S5JdpmZmeKUKVNEX19f0draWmzQoIH4zjvviAUFBXKXVuO2b99e4e+V0aNHi6IoDWsyY8YM0d3dXdRoNGLPnj3F2NhYeYuuIXc6NhcvXrzt7+Tt27fLXXq1u9v35lYc0qTqCKJYB4dJJyIiIrIw7ChBREREZAEY6oiIiIgsAEMdERERkQVgqCMiIiKyAAx1RERERBaAoY6IiIjIAjDUEREREVkAhjoiIiIiC8BQR0T0gARBwJo1a+Qug4jqOIY6IqrVxowZA0EQyk19+/aVuzQiohqlkrsAIqIH1bdvXyxdutSkTaPRyFQNEZE8eKaOiGo9jUYDDw8Pk8nJyQmAdGl00aJF6NevH7RaLRo0aIC//vrLZP2TJ0+iR48e0Gq1cHFxwQsvvIDs7GyTZX788UeEhoZCo9HA09MTkyZNMpmfmpqKQYMGQafTITAwEOvWrTPOu3HjBkaMGAFXV1dotVoEBgaWC6FERA+KoY6ILN6MGTPw5JNPIjIyEiNGjMDTTz+NmJgYAEBOTg769OkDJycnHDp0CCtXrsSWLVtMQtuiRYswceJEvPDCCzh58iTWrVuHRo0amexj9uzZGDp0KE6cOIFHH30UI0aMQFpamnH/0dHR2LBhA2JiYrBo0SLUq1ev5g4AEdUNIhFRLTZ69GhRqVSKNjY2JtNHH30kiqIoAhDHjx9vsk6HDh3ECRMmiKIoit9//73o5OQkZmdnG+f/999/okKhEJOSkkRRFEUvLy/xnXfeuW0NAMR3333X+D47O1sEIG7YsEEURVEcMGCA+Oyzz1bNByYiug3eU0dEtV737t2xaNEikzZnZ2fj644dO5rM69ixI44fPw4AiImJQYsWLWBjY2OcHxYWBoPBgNjYWAiCgKtXr6Jnz553rKF58+bG1zY2NrC3t0dKSgoAYMKECXjyySdx9OhR9O7dGwMHDkSnTp3u67MSEd0OQx0R1Xo2NjblLodWFa1WW6nlrKysTN4LggCDwQAA6NevH+Li4rB+/XqEh4ejZ8+emDhxIubPn1/l9RJR3cV76ojI4u3fv7/c++DgYABAcHAwIiMjkZOTY5wfEREBhUKBJk2awM7ODv7+/ti6desD1eDq6orRo0fj119/xYIFC/D9998/0PaIiG7FM3VEVOsVFBQgKSnJpE2lUhk7I6xcuRJt27ZF586dsXz5chw8eBA//PADAGDEiBF47733MHr0aMyaNQvXrl3Dyy+/jJEjR8Ld3R0AMGvWLIwfPx5ubm7o168fsrKyEBERgZdffrlS9c2cORNt2rRBaGgoCgoK8O+//xpDJRFRVWGoI6Jab+PGjfD09DRpa9KkCU6fPg1A6pm6YsUKvPTSS/D09MTvv/+OkJAQAIBOp8OmTZswZcoUtGvXDjqdDk8++SQ+//xz47ZGjx6N/Px8fPHFF3j99ddRr149DBkypNL1qdVqvPXWW7h06RK0Wi0efvhhrFixogo+ORFRKUEURVHuIoiIqosgCFi9ejUGDhwodylERNWK99QRERERWQCGOiIiIiILwHvqiMii8Q4TIqoreKaOiIiIyAIw1BERERFZAIY6IiIiIgvAUEdERERkARjqiIiIiCwAQx0RERGRBWCoIyIiIrIADHVEREREFoChjoiIiMgC/D8YPsYDaCYvWQAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from previous_chapters import plot_losses\n", + "\n", + "epochs_tensor = torch.linspace(1, n_epochs, len(train_losses))\n", + "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/appendix-D/01_main-chapter-code/previous_chapters.py b/appendix-D/01_main-chapter-code/previous_chapters.py new file mode 100644 index 0000000..ba18f50 --- /dev/null +++ b/appendix-D/01_main-chapter-code/previous_chapters.py @@ -0,0 +1,318 @@ +# This file collects all the relevant code that we covered thus far +# throughout Chapters 2-4. +# This file can be run as a standalone script. + +import tiktoken +import torch +import torch.nn as nn +from torch.utils.data import Dataset, DataLoader +import matplotlib.pyplot as plt + + +##################################### +# Chapter 2 +##################################### + +class GPTDatasetV1(Dataset): + def __init__(self, txt, tokenizer, max_length, stride): + self.tokenizer = tokenizer + self.input_ids = [] + self.target_ids = [] + + # Tokenize the entire text + token_ids = tokenizer.encode(txt) + + # Use a sliding window to chunk the book into overlapping sequences of max_length + for i in range(0, len(token_ids) - max_length, stride): + input_chunk = token_ids[i:i + max_length] + target_chunk = token_ids[i + 1: i + max_length + 1] + self.input_ids.append(torch.tensor(input_chunk)) + self.target_ids.append(torch.tensor(target_chunk)) + + def __len__(self): + return len(self.input_ids) + + def __getitem__(self, idx): + return self.input_ids[idx], self.target_ids[idx] + + +def create_dataloader_v1(txt, batch_size=4, max_length=256, + stride=128, shuffle=True, drop_last=True): + # Initialize the tokenizer + tokenizer = tiktoken.get_encoding("gpt2") + + # Create dataset + dataset = GPTDatasetV1(txt, tokenizer, max_length, stride) + + # Create dataloader + dataloader = DataLoader( + dataset, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last) + + return dataloader + + +##################################### +# Chapter 3 +##################################### + +class MultiHeadAttention(nn.Module): + def __init__(self, d_in, d_out, block_size, dropout, num_heads, qkv_bias=False): + super().__init__() + assert d_out % num_heads == 0, "d_out must be divisible by n_heads" + + self.d_out = d_out + self.num_heads = num_heads + self.head_dim = d_out // num_heads # Reduce the projection dim to match desired output dim + + self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias) + self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias) + self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias) + self.out_proj = nn.Linear(d_out, d_out) # Linear layer to combine head outputs + self.dropout = nn.Dropout(dropout) + self.register_buffer('mask', torch.triu(torch.ones(block_size, block_size), diagonal=1)) + + def forward(self, x): + b, num_tokens, d_in = x.shape + + keys = self.W_key(x) # Shape: (b, num_tokens, d_out) + queries = self.W_query(x) + values = self.W_value(x) + + # We implicitly split the matrix by adding a `num_heads` dimension + # Unroll last dim: (b, num_tokens, d_out) -> (b, num_tokens, num_heads, head_dim) + keys = keys.view(b, num_tokens, self.num_heads, self.head_dim) + values = values.view(b, num_tokens, self.num_heads, self.head_dim) + queries = queries.view(b, num_tokens, self.num_heads, self.head_dim) + + # Transpose: (b, num_tokens, num_heads, head_dim) -> (b, num_heads, num_tokens, head_dim) + keys = keys.transpose(1, 2) + queries = queries.transpose(1, 2) + values = values.transpose(1, 2) + + # Compute scaled dot-product attention (aka self-attention) with a causal mask + attn_scores = queries @ keys.transpose(2, 3) # Dot product for each head + + # Original mask truncated to the number of tokens and converted to boolean + mask_bool = self.mask.bool()[:num_tokens, :num_tokens] + + # Use the mask to fill attention scores + attn_scores.masked_fill_(mask_bool, -torch.inf) + + attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1) + attn_weights = self.dropout(attn_weights) + + # Shape: (b, num_tokens, num_heads, head_dim) + context_vec = (attn_weights @ values).transpose(1, 2) + + # Combine heads, where self.d_out = self.num_heads * self.head_dim + context_vec = context_vec.reshape(b, num_tokens, self.d_out) + context_vec = self.out_proj(context_vec) # optional projection + + return context_vec + + +##################################### +# Chapter 4 +##################################### + +class LayerNorm(nn.Module): + def __init__(self, emb_dim): + super().__init__() + self.eps = 1e-5 + self.scale = nn.Parameter(torch.ones(emb_dim)) + self.shift = nn.Parameter(torch.zeros(emb_dim)) + + def forward(self, x): + mean = x.mean(dim=-1, keepdim=True) + var = x.var(dim=-1, keepdim=True, unbiased=False) + norm_x = (x - mean) / torch.sqrt(var + self.eps) + return self.scale * norm_x + self.shift + + +class GELU(nn.Module): + def __init__(self): + super().__init__() + + def forward(self, x): + return 0.5 * x * (1 + torch.tanh( + torch.sqrt(torch.tensor(2.0 / torch.pi)) * + (x + 0.044715 * torch.pow(x, 3)) + )) + + +class FeedForward(nn.Module): + def __init__(self, cfg): + super().__init__() + self.layers = nn.Sequential( + nn.Linear(cfg["emb_dim"], 4 * cfg["emb_dim"]), + GELU(), + nn.Linear(4 * cfg["emb_dim"], cfg["emb_dim"]), + nn.Dropout(cfg["drop_rate"]) + ) + + def forward(self, x): + return self.layers(x) + + +class TransformerBlock(nn.Module): + def __init__(self, cfg): + super().__init__() + self.att = MultiHeadAttention( + d_in=cfg["emb_dim"], + d_out=cfg["emb_dim"], + block_size=cfg["ctx_len"], + num_heads=cfg["n_heads"], + dropout=cfg["drop_rate"], + qkv_bias=cfg["qkv_bias"]) + self.ff = FeedForward(cfg) + self.norm1 = LayerNorm(cfg["emb_dim"]) + self.norm2 = LayerNorm(cfg["emb_dim"]) + self.drop_resid = nn.Dropout(cfg["drop_rate"]) + + def forward(self, x): + # Shortcut connection for attention block + shortcut = x + x = self.norm1(x) + x = self.att(x) # Shape [batch_size, num_tokens, emb_size] + x = self.drop_resid(x) + x = x + shortcut # Add the original input back + + # Shortcut connection for feed-forward block + shortcut = x + x = self.norm2(x) + x = self.ff(x) + x = self.drop_resid(x) + x = x + shortcut # Add the original input back + + return x + + +class GPTModel(nn.Module): + def __init__(self, cfg): + super().__init__() + self.tok_emb = nn.Embedding(cfg["vocab_size"], cfg["emb_dim"]) + self.pos_emb = nn.Embedding(cfg["ctx_len"], cfg["emb_dim"]) + self.drop_emb = nn.Dropout(cfg["drop_rate"]) + + self.trf_blocks = nn.Sequential( + *[TransformerBlock(cfg) for _ in range(cfg["n_layers"])]) + + self.final_norm = LayerNorm(cfg["emb_dim"]) + self.out_head = nn.Linear(cfg["emb_dim"], cfg["vocab_size"], bias=False) + + def forward(self, in_idx): + batch_size, seq_len = in_idx.shape + tok_embeds = self.tok_emb(in_idx) + pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device)) + x = tok_embeds + pos_embeds # Shape [batch_size, num_tokens, emb_size] + x = self.drop_emb(x) + x = self.trf_blocks(x) + x = self.final_norm(x) + logits = self.out_head(x) + return logits + + +def generate_text_simple(model, idx, max_new_tokens, context_size): + # idx is (B, T) array of indices in the current context + for _ in range(max_new_tokens): + + # Crop current context if it exceeds the supported context size + # E.g., if LLM supports only 5 tokens, and the context size is 10 + # then only the last 5 tokens are used as context + idx_cond = idx[:, -context_size:] + + # Get the predictions + with torch.no_grad(): + logits = model(idx_cond) + + # Focus only on the last time step + # (batch, n_token, vocab_size) becomes (batch, vocab_size) + logits = logits[:, -1, :] + + # Get the idx of the vocab entry with the highest logits value + idx_next = torch.argmax(logits, dim=-1, keepdim=True) # (batch, 1) + + # Append sampled index to the running sequence + idx = torch.cat((idx, idx_next), dim=1) # (batch, n_tokens+1) + + return idx + + +##################################### +# Chapter 5 +#################################### + + +def calc_loss_batch(input_batch, target_batch, model, device): + input_batch, target_batch = input_batch.to(device), target_batch.to(device) + + logits = model(input_batch) + logits = logits.view(-1, logits.size(-1)) + loss = torch.nn.functional.cross_entropy(logits, target_batch.view(-1)) + return loss + + +def calc_loss_loader(data_loader, model, device, num_batches=None): + total_loss, batches_seen = 0., 0. + if num_batches is None: + num_batches = len(data_loader) + for i, (input_batch, target_batch) in enumerate(data_loader): + if i < num_batches: + loss = calc_loss_batch(input_batch, target_batch, model, device) + total_loss += loss.item() + batches_seen += 1 + else: + break + return total_loss / batches_seen + + +def evaluate_model(model, train_loader, val_loader, device, eval_iter): + model.eval() + with torch.no_grad(): + train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter) + val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter) + model.train() + return train_loss, val_loss + + +def generate_and_print_sample(model, tokenizer, device, start_context): + model.eval() + context_size = model.pos_emb.weight.shape[0] + encoded = text_to_token_ids(start_context, tokenizer).to(device) + with torch.no_grad(): + token_ids = generate_text_simple( + model=model, idx=encoded, + max_new_tokens=50, context_size=context_size) + decoded_text = token_ids_to_text(token_ids, tokenizer) + print(decoded_text.replace("\n", " ")) # Compact print format + model.train() + + +def plot_losses(epochs_seen, tokens_seen, train_losses, val_losses): + fig, ax1 = plt.subplots() + + # Plot training and validation loss against epochs + ax1.plot(epochs_seen, train_losses, label="Training loss") + ax1.plot(epochs_seen, val_losses, linestyle="-.", label="Validation loss") + ax1.set_xlabel("Epochs") + ax1.set_ylabel("Loss") + ax1.legend(loc="upper right") + + # Create a second x-axis for tokens seen + ax2 = ax1.twiny() # Create a second x-axis that shares the same y-axis + ax2.plot(tokens_seen, train_losses, alpha=0) # Invisible plot for aligning ticks + ax2.set_xlabel("Tokens seen") + + fig.tight_layout() # Adjust layout to make room + plt.show() + + +def text_to_token_ids(text, tokenizer): + encoded = tokenizer.encode(text) + encoded_tensor = torch.tensor(encoded).unsqueeze(0) # add batch dimension + return encoded_tensor + + +def token_ids_to_text(token_ids, tokenizer): + flat = token_ids.squeeze(0) # remove batch dimension + return tokenizer.decode(flat.tolist()) \ No newline at end of file diff --git a/appendix-D/01_main-chapter-code/the-verdict.txt b/appendix-D/01_main-chapter-code/the-verdict.txt new file mode 100644 index 0000000..6b651c7 --- /dev/null +++ b/appendix-D/01_main-chapter-code/the-verdict.txt @@ -0,0 +1,165 @@ +I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no great surprise to me to hear that, in the height of his glory, he had dropped his painting, married a rich widow, and established himself in a villa on the Riviera. (Though I rather thought it would have been Rome or Florence.) + +"The height of his glory"--that was what the women called it. I can hear Mrs. Gideon Thwing--his last Chicago sitter--deploring his unaccountable abdication. "Of course it's going to send the value of my picture 'way up; but I don't think of that, Mr. Rickham--the loss to Arrt is all I think of." The word, on Mrs. Thwing's lips, multiplied its _rs_ as though they were reflected in an endless vista of mirrors. And it was not only the Mrs. Thwings who mourned. Had not the exquisite Hermia Croft, at the last Grafton Gallery show, stopped me before Gisburn's "Moon-dancers" to say, with tears in her eyes: "We shall not look upon its like again"? + +Well!--even through the prism of Hermia's tears I felt able to face the fact with equanimity. Poor Jack Gisburn! The women had made him--it was fitting that they should mourn him. Among his own sex fewer regrets were heard, and in his own trade hardly a murmur. Professional jealousy? Perhaps. If it were, the honour of the craft was vindicated by little Claude Nutley, who, in all good faith, brought out in the Burlington a very handsome "obituary" on Jack--one of those showy articles stocked with random technicalities that I have heard (I won't say by whom) compared to Gisburn's painting. And so--his resolve being apparently irrevocable--the discussion gradually died out, and, as Mrs. Thwing had predicted, the price of "Gisburns" went up. + +It was not till three years later that, in the course of a few weeks' idling on the Riviera, it suddenly occurred to me to wonder why Gisburn had given up his painting. On reflection, it really was a tempting problem. To accuse his wife would have been too easy--his fair sitters had been denied the solace of saying that Mrs. Gisburn had "dragged him down." For Mrs. Gisburn--as such--had not existed till nearly a year after Jack's resolve had been taken. It might be that he had married her--since he liked his ease--because he didn't want to go on painting; but it would have been hard to prove that he had given up his painting because he had married her. + +Of course, if she had not dragged him down, she had equally, as Miss Croft contended, failed to "lift him up"--she had not led him back to the easel. To put the brush into his hand again--what a vocation for a wife! But Mrs. Gisburn appeared to have disdained it--and I felt it might be interesting to find out why. + +The desultory life of the Riviera lends itself to such purely academic speculations; and having, on my way to Monte Carlo, caught a glimpse of Jack's balustraded terraces between the pines, I had myself borne thither the next day. + +I found the couple at tea beneath their palm-trees; and Mrs. Gisburn's welcome was so genial that, in the ensuing weeks, I claimed it frequently. It was not that my hostess was "interesting": on that point I could have given Miss Croft the fullest reassurance. It was just because she was _not_ interesting--if I may be pardoned the bull--that I found her so. For Jack, all his life, had been surrounded by interesting women: they had fostered his art, it had been reared in the hot-house of their adulation. And it was therefore instructive to note what effect the "deadening atmosphere of mediocrity" (I quote Miss Croft) was having on him. + +I have mentioned that Mrs. Gisburn was rich; and it was immediately perceptible that her husband was extracting from this circumstance a delicate but substantial satisfaction. It is, as a rule, the people who scorn money who get most out of it; and Jack's elegant disdain of his wife's big balance enabled him, with an appearance of perfect good-breeding, to transmute it into objects of art and luxury. To the latter, I must add, he remained relatively indifferent; but he was buying Renaissance bronzes and eighteenth-century pictures with a discrimination that bespoke the amplest resources. + +"Money's only excuse is to put beauty into circulation," was one of the axioms he laid down across the Sevres and silver of an exquisitely appointed luncheon-table, when, on a later day, I had again run over from Monte Carlo; and Mrs. Gisburn, beaming on him, added for my enlightenment: "Jack is so morbidly sensitive to every form of beauty." + +Poor Jack! It had always been his fate to have women say such things of him: the fact should be set down in extenuation. What struck me now was that, for the first time, he resented the tone. I had seen him, so often, basking under similar tributes--was it the conjugal note that robbed them of their savour? No--for, oddly enough, it became apparent that he was fond of Mrs. Gisburn--fond enough not to see her absurdity. It was his own absurdity he seemed to be wincing under--his own attitude as an object for garlands and incense. + +"My dear, since I've chucked painting people don't say that stuff about me--they say it about Victor Grindle," was his only protest, as he rose from the table and strolled out onto the sunlit terrace. + +I glanced after him, struck by his last word. Victor Grindle was, in fact, becoming the man of the moment--as Jack himself, one might put it, had been the man of the hour. The younger artist was said to have formed himself at my friend's feet, and I wondered if a tinge of jealousy underlay the latter's mysterious abdication. But no--for it was not till after that event that the _rose Dubarry_ drawing-rooms had begun to display their "Grindles." + +I turned to Mrs. Gisburn, who had lingered to give a lump of sugar to her spaniel in the dining-room. + +"Why _has_ he chucked painting?" I asked abruptly. + +She raised her eyebrows with a hint of good-humoured surprise. + +"Oh, he doesn't _have_ to now, you know; and I want him to enjoy himself," she said quite simply. + +I looked about the spacious white-panelled room, with its _famille-verte_ vases repeating the tones of the pale damask curtains, and its eighteenth-century pastels in delicate faded frames. + +"Has he chucked his pictures too? I haven't seen a single one in the house." + +A slight shade of constraint crossed Mrs. Gisburn's open countenance. "It's his ridiculous modesty, you know. He says they're not fit to have about; he's sent them all away except one--my portrait--and that I have to keep upstairs." + +His ridiculous modesty--Jack's modesty about his pictures? My curiosity was growing like the bean-stalk. I said persuasively to my hostess: "I must really see your portrait, you know." + +She glanced out almost timorously at the terrace where her husband, lounging in a hooded chair, had lit a cigar and drawn the Russian deerhound's head between his knees. + +"Well, come while he's not looking," she said, with a laugh that tried to hide her nervousness; and I followed her between the marble Emperors of the hall, and up the wide stairs with terra-cotta nymphs poised among flowers at each landing. + +In the dimmest corner of her boudoir, amid a profusion of delicate and distinguished objects, hung one of the familiar oval canvases, in the inevitable garlanded frame. The mere outline of the frame called up all Gisburn's past! + +Mrs. Gisburn drew back the window-curtains, moved aside a _jardiniere_ full of pink azaleas, pushed an arm-chair away, and said: "If you stand here you can just manage to see it. I had it over the mantel-piece, but he wouldn't let it stay." + +Yes--I could just manage to see it--the first portrait of Jack's I had ever had to strain my eyes over! Usually they had the place of honour--say the central panel in a pale yellow or _rose Dubarry_ drawing-room, or a monumental easel placed so that it took the light through curtains of old Venetian point. The more modest place became the picture better; yet, as my eyes grew accustomed to the half-light, all the characteristic qualities came out--all the hesitations disguised as audacities, the tricks of prestidigitation by which, with such consummate skill, he managed to divert attention from the real business of the picture to some pretty irrelevance of detail. Mrs. Gisburn, presenting a neutral surface to work on--forming, as it were, so inevitably the background of her own picture--had lent herself in an unusual degree to the display of this false virtuosity. The picture was one of Jack's "strongest," as his admirers would have put it--it represented, on his part, a swelling of muscles, a congesting of veins, a balancing, straddling and straining, that reminded one of the circus-clown's ironic efforts to lift a feather. It met, in short, at every point the demand of lovely woman to be painted "strongly" because she was tired of being painted "sweetly"--and yet not to lose an atom of the sweetness. + +"It's the last he painted, you know," Mrs. Gisburn said with pardonable pride. "The last but one," she corrected herself--"but the other doesn't count, because he destroyed it." + +"Destroyed it?" I was about to follow up this clue when I heard a footstep and saw Jack himself on the threshold. + +As he stood there, his hands in the pockets of his velveteen coat, the thin brown waves of hair pushed back from his white forehead, his lean sunburnt cheeks furrowed by a smile that lifted the tips of a self-confident moustache, I felt to what a degree he had the same quality as his pictures--the quality of looking cleverer than he was. + +His wife glanced at him deprecatingly, but his eyes travelled past her to the portrait. + +"Mr. Rickham wanted to see it," she began, as if excusing herself. He shrugged his shoulders, still smiling. + +"Oh, Rickham found me out long ago," he said lightly; then, passing his arm through mine: "Come and see the rest of the house." + +He showed it to me with a kind of naive suburban pride: the bath-rooms, the speaking-tubes, the dress-closets, the trouser-presses--all the complex simplifications of the millionaire's domestic economy. And whenever my wonder paid the expected tribute he said, throwing out his chest a little: "Yes, I really don't see how people manage to live without that." + +Well--it was just the end one might have foreseen for him. Only he was, through it all and in spite of it all--as he had been through, and in spite of, his pictures--so handsome, so charming, so disarming, that one longed to cry out: "Be dissatisfied with your leisure!" as once one had longed to say: "Be dissatisfied with your work!" + +But, with the cry on my lips, my diagnosis suffered an unexpected check. + +"This is my own lair," he said, leading me into a dark plain room at the end of the florid vista. It was square and brown and leathery: no "effects"; no bric-a-brac, none of the air of posing for reproduction in a picture weekly--above all, no least sign of ever having been used as a studio. + +The fact brought home to me the absolute finality of Jack's break with his old life. + +"Don't you ever dabble with paint any more?" I asked, still looking about for a trace of such activity. + +"Never," he said briefly. + +"Or water-colour--or etching?" + +His confident eyes grew dim, and his cheeks paled a little under their handsome sunburn. + +"Never think of it, my dear fellow--any more than if I'd never touched a brush." + +And his tone told me in a flash that he never thought of anything else. + +I moved away, instinctively embarrassed by my unexpected discovery; and as I turned, my eye fell on a small picture above the mantel-piece--the only object breaking the plain oak panelling of the room. + +"Oh, by Jove!" I said. + +It was a sketch of a donkey--an old tired donkey, standing in the rain under a wall. + +"By Jove--a Stroud!" I cried. + +He was silent; but I felt him close behind me, breathing a little quickly. + +"What a wonder! Made with a dozen lines--but on everlasting foundations. You lucky chap, where did you get it?" + +He answered slowly: "Mrs. Stroud gave it to me." + +"Ah--I didn't know you even knew the Strouds. He was such an inflexible hermit." + +"I didn't--till after. . . . She sent for me to paint him when he was dead." + +"When he was dead? You?" + +I must have let a little too much amazement escape through my surprise, for he answered with a deprecating laugh: "Yes--she's an awful simpleton, you know, Mrs. Stroud. Her only idea was to have him done by a fashionable painter--ah, poor Stroud! She thought it the surest way of proclaiming his greatness--of forcing it on a purblind public. And at the moment I was _the_ fashionable painter." + +"Ah, poor Stroud--as you say. Was _that_ his history?" + +"That was his history. She believed in him, gloried in him--or thought she did. But she couldn't bear not to have all the drawing-rooms with her. She couldn't bear the fact that, on varnishing days, one could always get near enough to see his pictures. Poor woman! She's just a fragment groping for other fragments. Stroud is the only whole I ever knew." + +"You ever knew? But you just said--" + +Gisburn had a curious smile in his eyes. + +"Oh, I knew him, and he knew me--only it happened after he was dead." + +I dropped my voice instinctively. "When she sent for you?" + +"Yes--quite insensible to the irony. She wanted him vindicated--and by me!" + +He laughed again, and threw back his head to look up at the sketch of the donkey. "There were days when I couldn't look at that thing--couldn't face it. But I forced myself to put it here; and now it's cured me--cured me. That's the reason why I don't dabble any more, my dear Rickham; or rather Stroud himself is the reason." + +For the first time my idle curiosity about my companion turned into a serious desire to understand him better. + +"I wish you'd tell me how it happened," I said. + +He stood looking up at the sketch, and twirling between his fingers a cigarette he had forgotten to light. Suddenly he turned toward me. + +"I'd rather like to tell you--because I've always suspected you of loathing my work." + +I made a deprecating gesture, which he negatived with a good-humoured shrug. + +"Oh, I didn't care a straw when I believed in myself--and now it's an added tie between us!" + +He laughed slightly, without bitterness, and pushed one of the deep arm-chairs forward. "There: make yourself comfortable--and here are the cigars you like." + +He placed them at my elbow and continued to wander up and down the room, stopping now and then beneath the picture. + +"How it happened? I can tell you in five minutes--and it didn't take much longer to happen. . . . I can remember now how surprised and pleased I was when I got Mrs. Stroud's note. Of course, deep down, I had always _felt_ there was no one like him--only I had gone with the stream, echoed the usual platitudes about him, till I half got to think he was a failure, one of the kind that are left behind. By Jove, and he _was_ left behind--because he had come to stay! The rest of us had to let ourselves be swept along or go under, but he was high above the current--on everlasting foundations, as you say. + +"Well, I went off to the house in my most egregious mood--rather moved, Lord forgive me, at the pathos of poor Stroud's career of failure being crowned by the glory of my painting him! Of course I meant to do the picture for nothing--I told Mrs. Stroud so when she began to stammer something about her poverty. I remember getting off a prodigious phrase about the honour being _mine_--oh, I was princely, my dear Rickham! I was posing to myself like one of my own sitters. + +"Then I was taken up and left alone with him. I had sent all my traps in advance, and I had only to set up the easel and get to work. He had been dead only twenty-four hours, and he died suddenly, of heart disease, so that there had been no preliminary work of destruction--his face was clear and untouched. I had met him once or twice, years before, and thought him insignificant and dingy. Now I saw that he was superb. + +"I was glad at first, with a merely aesthetic satisfaction: glad to have my hand on such a 'subject.' Then his strange life-likeness began to affect me queerly--as I blocked the head in I felt as if he were watching me do it. The sensation was followed by the thought: if he _were_ watching me, what would he say to my way of working? My strokes began to go a little wild--I felt nervous and uncertain. + +"Once, when I looked up, I seemed to see a smile behind his close grayish beard--as if he had the secret, and were amusing himself by holding it back from me. That exasperated me still more. The secret? Why, I had a secret worth twenty of his! I dashed at the canvas furiously, and tried some of my bravura tricks. But they failed me, they crumbled. I saw that he wasn't watching the showy bits--I couldn't distract his attention; he just kept his eyes on the hard passages between. Those were the ones I had always shirked, or covered up with some lying paint. And how he saw through my lies! + +"I looked up again, and caught sight of that sketch of the donkey hanging on the wall near his bed. His wife told me afterward it was the last thing he had done--just a note taken with a shaking hand, when he was down in Devonshire recovering from a previous heart attack. Just a note! But it tells his whole history. There are years of patient scornful persistence in every line. A man who had swum with the current could never have learned that mighty up-stream stroke. . . . + +"I turned back to my work, and went on groping and muddling; then I looked at the donkey again. I saw that, when Stroud laid in the first stroke, he knew just what the end would be. He had possessed his subject, absorbed it, recreated it. When had I done that with any of my things? They hadn't been born of me--I had just adopted them. . . . + +"Hang it, Rickham, with that face watching me I couldn't do another stroke. The plain truth was, I didn't know where to put it--_I had never known_. Only, with my sitters and my public, a showy splash of colour covered up the fact--I just threw paint into their faces. . . . Well, paint was the one medium those dead eyes could see through--see straight to the tottering foundations underneath. Don't you know how, in talking a foreign language, even fluently, one says half the time not what one wants to but what one can? Well--that was the way I painted; and as he lay there and watched me, the thing they called my 'technique' collapsed like a house of cards. He didn't sneer, you understand, poor Stroud--he just lay there quietly watching, and on his lips, through the gray beard, I seemed to hear the question: 'Are you sure you know where you're coming out?' + +"If I could have painted that face, with that question on it, I should have done a great thing. The next greatest thing was to see that I couldn't--and that grace was given me. But, oh, at that minute, Rickham, was there anything on earth I wouldn't have given to have Stroud alive before me, and to hear him say: 'It's not too late--I'll show you how'? + +"It _was_ too late--it would have been, even if he'd been alive. I packed up my traps, and went down and told Mrs. Stroud. Of course I didn't tell her _that_--it would have been Greek to her. I simply said I couldn't paint him, that I was too moved. She rather liked the idea--she's so romantic! It was that that made her give me the donkey. But she was terribly upset at not getting the portrait--she did so want him 'done' by some one showy! At first I was afraid she wouldn't let me off--and at my wits' end I suggested Grindle. Yes, it was I who started Grindle: I told Mrs. Stroud he was the 'coming' man, and she told somebody else, and so it got to be true. . . . And he painted Stroud without wincing; and she hung the picture among her husband's things. . . ." + +He flung himself down in the arm-chair near mine, laid back his head, and clasping his arms beneath it, looked up at the picture above the chimney-piece. + +"I like to fancy that Stroud himself would have given it to me, if he'd been able to say what he thought that day." + +And, in answer to a question I put half-mechanically--"Begin again?" he flashed out. "When the one thing that brings me anywhere near him is that I knew enough to leave off?" + +He stood up and laid his hand on my shoulder with a laugh. "Only the irony of it is that I _am_ still painting--since Grindle's doing it for me! The Strouds stand alone, and happen once--but there's no exterminating our kind of art." \ No newline at end of file