yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-12-19 03:02:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
LLMs-from-scratch/ch05/07_gpt_to_llama/converting-llama2-to-llama3.ipynb
Sebastian Raschka 398b079efa
RoPE decay plot (#910) 
* RoPE decay plot

* RoPE decay plot
2025-11-17 17:29:49 -06:00

2861 lines
134 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"id": "0_xya1nyDHfY",
"metadata": {
"id": "0_xya1nyDHfY"
},
"source": [
"<table style=\"width:100%\">\n",
"<tr>\n",
"<td style=\"vertical-align:middle; text-align:left;\">\n",
"<font size=\"2\">\n",
"Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
"<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
"</font>\n",
"</td>\n",
"<td style=\"vertical-align:middle; text-align:left;\">\n",
"<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
"</td>\n",
"</tr>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"id": "l62zIRRSBy_R",
"metadata": {
"id": "l62zIRRSBy_R"
},
"source": [
"# Converting Llama 2 to Llama 3.2 From Scratch"
]
},
{
"cell_type": "markdown",
"id": "aFmxTQbwCUMl",
"metadata": {
"id": "aFmxTQbwCUMl"
},
"source": [
"- This is a follow-up notebook to [Converting a From-Scratch GPT Architecture to Llama 2](./converting-gpt-to-llama2.ipynb), converting Meta AI's Llama 2 architecture model step by step to Llama 3, Llama 3.1, and Llama 3.2\n",
"- The explanations are purposefully kept minimal in this notebook so as not to bloat it unnecessarily and focus on the main code\n",
"- For more information about the architectures, please see the Llama 2 and Llama 3 papers\n",
" - [Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)](https://arxiv.org/abs/2307.09288)\n",
" - [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)"
]
},
{
"cell_type": "markdown",
"id": "ohhMKUWvGm9z",
"metadata": {
"id": "ohhMKUWvGm9z"
},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/gpt2-to-llama2-llama3.webp?1\">"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ws0wsUzwLH2k",
"metadata": {
"id": "ws0wsUzwLH2k"
},
"outputs": [],
"source": [
"# pip install -r requirements-extra.txt"
]
},
{
"cell_type": "markdown",
"id": "JBpQwU89ETA1",
"metadata": {
"id": "JBpQwU89ETA1"
},
"source": [
"- Packages that are being used in this notebook:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "34a9a440-84c2-42cc-808b-38677cb6af8a",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "34a9a440-84c2-42cc-808b-38677cb6af8a",
"outputId": "e3d3d4b6-ee63-4e28-d794-e8b0bdd931fd"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"blobfile version: 3.1.0\n",
"huggingface_hub version: 0.34.4\n",
"tiktoken version: 0.11.0\n",
"torch version: 2.8.0\n"
]
}
],
"source": [
"from importlib.metadata import version\n",
"\n",
"pkgs = [\n",
" \"blobfile\", # to download pretrained weights\n",
" \"huggingface_hub\", # to download pretrained weights\n",
" \"matplotlib\", # to visualize RoPE with different base frequencies\n",
" \"tiktoken\", # to implement the tokenizer\n",
" \"torch\", # to implement the model\n",
"]\n",
"for p in pkgs:\n",
" print(f\"{p} version: {version(p)}\")"
]
},
{
"cell_type": "markdown",
"id": "UJJneXpTEg4W",
"metadata": {
"id": "UJJneXpTEg4W"
},
"source": [
"&nbsp;\n",
"# 1. Convert the Llama model implementation step by step"
]
},
{
"cell_type": "markdown",
"id": "v1zpfX2GHBKa",
"metadata": {
"id": "v1zpfX2GHBKa"
},
"source": [
"- If you are new to implementing LLM architectures, I recommend starting with [chapter 4](../../ch04/01_main-chapter-code/ch04.ipynb), which walks you through the implementation of the original GPT architecture step by step\n",
"- The [Converting a From-Scratch GPT Architecture to Llama 2](./converting-gpt-to-llama2.ipynb) then implements the Llama-specific components, such as RMSNorm layers, SiLU and SwiGLU activations, RoPE (rotary position embeddings), and the SentencePiece tokenizer\n",
"- This notebook takes the Llama 2 architecture and transforms it into Llama 3 architecture by\n",
" 1. modifying the rotary embeddings\n",
" 2. implementing grouped-query attention\n",
" 3. and using a customized version of the GPT-4 tokenizer\n",
"- Later, we then load the original Llama 3 weights shared by Meta AI into the architecture"
]
},
{
"cell_type": "markdown",
"id": "c14b9121-abe1-4a46-99b8-acdef71e5b41",
"metadata": {
"id": "c14b9121-abe1-4a46-99b8-acdef71e5b41"
},
"source": [
"&nbsp;\n",
"## 1.1 Reusing Llama 2 components"
]
},
{
"cell_type": "markdown",
"id": "dgDhJGJ6xR4e",
"metadata": {
"id": "dgDhJGJ6xR4e"
},
"source": [
"- Llama 2 is actually quite similar to Llama 3, as mentioned above and illustrated in the figure at the top of this notebook\n",
"- This means that we can import several building blocks from the [Llama 2 notebook](./converting-gpt-to-llama2.ipynb) using the following code"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "a5bc3948-231b-4f1f-8d41-24ad0b7643d0",
"metadata": {
"id": "a5bc3948-231b-4f1f-8d41-24ad0b7643d0"
},
"outputs": [],
"source": [
"import os\n",
"import sys\n",
"import io\n",
"import nbformat\n",
"import types\n",
"\n",
"def import_from_notebook():\n",
" def import_definitions_from_notebook(fullname, names):\n",
" current_dir = os.getcwd()\n",
" path = os.path.join(current_dir, fullname + \".ipynb\")\n",
" path = os.path.normpath(path)\n",
"\n",
" # Load the notebook\n",
" if not os.path.exists(path):\n",
" raise FileNotFoundError(f\"Notebook file not found at: {path}\")\n",
"\n",
" with io.open(path, \"r\", encoding=\"utf-8\") as f:\n",
" nb = nbformat.read(f, as_version=4)\n",
"\n",
" # Create a module to store the imported functions and classes\n",
" mod = types.ModuleType(fullname)\n",
" sys.modules[fullname] = mod\n",
"\n",
" # Go through the notebook cells and only execute function or class definitions\n",
" for cell in nb.cells:\n",
" if cell.cell_type == \"code\":\n",
" cell_code = cell.source\n",
" for name in names:\n",
" # Check for function or class definitions\n",
" if f\"def {name}\" in cell_code or f\"class {name}\" in cell_code:\n",
" exec(cell_code, mod.__dict__)\n",
" return mod\n",
"\n",
" fullname = \"converting-gpt-to-llama2\"\n",
" names = [\"precompute_rope_params\", \"compute_rope\", \"SiLU\", \"FeedForward\", \"RMSNorm\", \"MultiHeadAttention\"]\n",
"\n",
" return import_definitions_from_notebook(fullname, names)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "d546032d-fce4-47cf-8d0e-682b78b21c61",
"metadata": {
"id": "d546032d-fce4-47cf-8d0e-682b78b21c61"
},
"outputs": [],
"source": [
"imported_module = import_from_notebook()\n",
"\n",
"# We need to redefine precompute_rope_params\n",
"# precompute_rope_params = getattr(imported_module, \"precompute_rope_params\", None)\n",
"compute_rope = getattr(imported_module, \"compute_rope\", None)\n",
"SiLU = getattr(imported_module, \"SiLU\", None)\n",
"FeedForward = getattr(imported_module, \"FeedForward\", None)\n",
"RMSNorm = getattr(imported_module, \"RMSNorm\", None)\n",
"\n",
"# MultiHeadAttention only for comparison purposes\n",
"MultiHeadAttention = getattr(imported_module, \"MultiHeadAttention\", None)"
]
},
{
"cell_type": "markdown",
"id": "979c7b6d-1370-4da1-8bfb-a2b27537bf2f",
"metadata": {
"id": "979c7b6d-1370-4da1-8bfb-a2b27537bf2f"
},
"source": [
"&nbsp;\n",
"## 1.2 Modified RoPE"
]
},
{
"cell_type": "markdown",
"id": "m9_oDcHCx8VI",
"metadata": {
"id": "m9_oDcHCx8VI"
},
"source": [
"- Llama 3 uses rotary position embeddings (RoPE) similar to Llama 2 (for a detailed explanation, please see the [RoPE paper](https://arxiv.org/abs/2104.09864))\n",
"- There are some subtle differences in the RoPE settings, though\n",
" - Llama 3 now supports up to 8,192 tokens, twice as many as Llama 2 (4,096)\n",
" - The base value for the so-called RoPE $\\theta$ (see equation below) was increased from 10,000 (Llama 2) to 500,000 (Llama 3) in the following equation (adapted from the [RoPE paper](https://arxiv.org/abs/2104.09864))\n",
"\n",
"$$\\Theta = \\left\\{\\theta_i = \\text{base}^{\\frac{-2(i-1)}{d}}, i \\in \\left[1, 2, ..., d/2\\right]\\right\\}$$\n",
"\n",
"- These $\\theta$ values are a set of predefined parameters that are used to determine the rotational angles in the rotary matrix, where $d$ is the dimensionality of the embedding space\n",
"- Increasing the base from 10,000 to 500,000 makes the frequencies decay faster across the dimensions, which means that the higher-dimensional components are associated with smaller rotation angles than before, so the overall frequency range becomes more compressed rather than expanded"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ec8ded9f-3709-475a-87ff-0faee176207e",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAnYAAAHWCAYAAAD6oMSKAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAcqBJREFUeJzt3Qd4U+XbBvA76d60lC723nuDCCiCqIgD5QMVREVBVBQ3CqioOBEHiqAo+lfBCSigoLJkT2WWvUoHpdCW7ibnu543TUgnHWkzev+u65jk5OTk7Wlqbt6p0zRNAxERERE5Pb29C0BEREREtsFgR0REROQiGOyIiIiIXASDHREREZGLYLAjIiIichEMdkREREQugsGOiIiIyEUw2BERERG5CHd7F8CRGI1GnD17FgEBAdDpdPYuDhERERFkLYnU1FRERUVBry+5To7BzoqEurp169q7GERERESFnD59GnXq1EFJGOysSE2d+cIFBgbauzhERERESElJURVP5pxSEgY7K+bmVwl1DHZERETkSErTTYyDJ4iIiIhcBIMdERERkYtgsCMiIiJyEexjR0RUTRkMBuTk5Ni7GETVnoeHB9zc3GxyLgY7IqJqOCdWXFwcLl68aO+iEFGeGjVqICIiosLz6DLYERFVM+ZQFxYWBl9fX07ITmTnf2ilp6cjISFBPY6MjKzQ+RjsiIiqWfOrOdTVrFnT3sUhIgA+Pj7qVsKd/G1WpFmWgyeIiKoRc586qakjIsdh/pusaL9XBjsiomqIza9Ervk3yWBHRERE5CIY7IiIyOH169cPjz/+uL2LQaWQnZ2NJk2aYOPGjaU6/ssvv1QjQp3Vc889h0cffRSOwiGD3bp16zBkyBBERUWpqsnFixdf8TVr1qxBp06d4OXlpT5Q8kEhIiJyZI899hg6d+6svrs6dOhQ5DH//fcf+vTpA29vb7UQ/FtvvXXF8546dQo33nij6rclnfGffvpp5Obmlvl7c/bs2WjQoIF67+7du2Pr1q1XfO85c+agYcOG6NWrl2Vfab/LHc2aNWswdOhQNVLVz89P/Y6++eabfMc89dRTWLBgAY4dOwZH4JDBLi0tDe3bt1cfqNI4fvy4+gD3798fu3fvVv+qe+CBB/DHH39UelmJiIgq4r777sPw4cOLfC4lJQUDBw5E/fr1sWPHDrz99tt46aWXMHfu3BJHPst3otScSa2ZhA4JbVOnTi3T9+aiRYswadIkTJs2DTt37lTfy4MGDbJMy1Hc1B0fffQR7r//frgCuX7t2rXDTz/9pAL2mDFjMGrUKPz222+WY0JDQ9V1+eSTT+AQNAcnRfzll19KPOaZZ57RWrdunW/f8OHDtUGDBpXpvZKTk9X7yS0RkSvKyMjQ9u/fr26dSd++fbUJEyaoLTAwUKtZs6b24osvakaj0XLMV199pXXu3Fnz9/fXwsPDtREjRmjx8fGW55OSkrSRI0dqoaGhmre3t9akSRNt/vz5ludPnTql3XHHHVpQUJAWHBys3Xzzzdrx48er5OebNm2a1r59+0L7P/74Y1WWrKwsy75nn31Wa968ebHnWr58uabX67W4uDjLvk8++URdN/N5SvO92a1bN3W9zQwGgxYVFaXNmDGj2Pfetm2beu+UlJRSf5d/8cUX6pqbHTlyRF37sLAwzc/PT+vSpYu2atWqfK+pX7++Nn36dO2ee+5Rx9SrV09bsmSJlpCQoF4r+9q2bavKY5aYmKj93//9n/oZfHx8tDZt2mjffvutVlY33HCDNmbMmHz7FixYoNWpU0errL/NsuQTh6yxK6tNmzZhwIAB+fZJepb9JcnKylL/GrLeKtN/Zy5ixvID+N/mk5X6PkREZZocNTvXLpvp+770pObJ3d1dNQe+//77mDlzJj777DPL8zJNxPTp0/Hvv/+qZr8TJ07g3nvvtTw/ZcoU7N+/HytWrMCBAwdUDYvUtphfK98bAQEBWL9+PTZs2AB/f39cf/31quarOHJMSdu4ceNQEfI9dvXVV8PT09OyT8oZHR2NCxcuFPuatm3bIjw8PN9r5Dtu3759pfrelJ9Zagitj9Hr9epxSd+tcu2aNWumrmN5Xbp0CTfccAP++usv7Nq1S/0OpHuWNC9be++999C7d291jNQ+3nPPPao27e6771Y1jI0bN1aPzZ+zzMxM1ey9bNky7N27Fw8++KB6TWmal60lJycjJCQk375u3brhzJkz6jNnb+6uMou69QdYyGP5EGdkZFgm/itoxowZePnll6uolEDMsf3w2vA5ULMG0OPdKntfIqLiZOQY0Gqqfbqt7H9lEHw9S/81JP3L5Mtc+ms1b94ce/bsUY/Hjh1radI0a9SoET744AN07dpVBQUJWRIMOnbsiC5duqhjpO+YdbOj0WhUQdE87cQXX3yhOvVLPytpDi2KNGOWJDAwEBX9fpP+atbM33fyXHBwcKm/E83PleZ7U0KjNOkWdczBgweLLe/JkydV//iKkCZf2cwkrP/yyy9YunQpHnnkEct+CX8PPfSQui/NzBLUu3btijvuuEPte/bZZ9GzZ0/Ex8erpbpq166t+sOZyYAHaXr+/vvvVTArDTl227Zt+PTTT/PtN//M8vNbf67swSWCXXk9//zzqv+AmXyg5X8claWJdwoGe/yIkymyXAiDHRFRWfTo0SPfXF/ypf3uu++qACIz9UsNk/Q/kxo7CSYS1IQEulatWmH8+PG4/fbbVW2OBLVbbrnF0sFfXnPkyJFCNU1Sy3P06NFiyySDDugyCYUy0KIiJIjL71Fq1mJjY9WgDzlvwRo76ftmZg6gbdu2LbRP+gRKsJPPyeuvv67CWUxMjKqVlJa70k7WvXr1atXHbt68eWjdunW+58wVSLI0mL25RLCTX5gkcmvyWP6lVFxtnZCRQLJVlch6pv8BRGiJSE7LRpDf5ap1IiJ78PFwUzVn9npvWw66k6ZE2WTUYq1atVQQkMfmptTBgwerGpXly5dj1apVuPbaazFhwgS88847KkxIM13BEY9CzlUcqQksiTQLyihRW3+/mZ8r7jUFmxcLvuZK35sSlGUr6pji3ldI07bUpFaE1KrJ70d+LxKcpTzDhg0r1CTu4eFhuW8O/B5F7DMHfBl4Ik34s2bNUgFQRrnKoJGSmtrN1q5dq5qDpYZYmncLSkpKuuJnpaq4RLCTf7XJH6o1+VDIfkfiH1oPRujgpcvBwdMn0L5FM3sXiYiqOfnyK0tzqD1t2bIl3+PNmzejadOmKoBI8+D58+fxxhtvWFpetm/fXugc8sU7evRotckUIjINiAQImfZDmmNlapCyNJ9WdlOsfI+98MILqg+gObTI95s0RRfVDGt+zWuvvWZZd9T8GimL1FyW5ntT+vRJ0JV+blKzaQ5I8ti6ObQgaeqWJlHp11belRSkf6P0jbz11lvVYwndtui7tmHDBjV1iYRt889z6NAhyzUpjjTF33TTTXjzzTdVv7yiSJ89+f0UrMmzB4ccPCG/RPljMf/ByLBsuW+uhpUmVOvELJ1TZf6YZ555Rv1xf/zxx6qq9YknnoBDcfdEst70hxh/uviqfSIiKky+A6T7jAwc+O677/Dhhx9i4sSJ6rl69eqpMCL75PtA+mNJ3yxr0g9ryZIlqslVBhHIlBUtW7ZUz911112qtkm++GUAgHzvyBe6zDMnneKLIzVKJW3mYFUcKYt8v0mfN2luNH/3mWuRRo4cqX4umT5EyizhU2qdrLsRFSTNzBJWZGCANDFLP7IXX3xR1U6aW6lK870p7yHNjjJoRQabSFO21IxKc2RxZPoU+Q43D9KwZv4ut97kfAVJWP/555/V81J+uQbmWreKaNq0qQqvMoWJ/DzSP69gjWRRza8yMEM+B9KML78n2cw1dGbymZF/KJTUSlhlNAe0evVqNay34DZ69Gj1vNzK0PeCr+nQoYPm6empNWrUSA2fLquqmO7k9Js9NW1aoPb9V7Mr7T2IiFxxupOHH35YGzdunJq2Q6YAmTx5cr7pTmTqigYNGmheXl5az549taVLl6r/p+/atUs9L9NjtGzZUk11ERISog0dOlQ7duyY5fWxsbHaqFGj1HQocg75Lhk7dmylfifIz1XU9531NCv//vuvdtVVV6ky1a5dW3vjjTeK/M60fs2JEye0wYMHq59Vfp4nn3xSy8nJKfP35ocffqimEpFjZPqTzZs3X/FnuvPOO7Xnnnsu376ifkbZ1q9fX2i6E/k5+vfvr8pet25d7aOPPlLXaeLEifmmO3nvvfdKnFLl+PHj+X7/58+fV79zmQ5HplKR6XLk9y37iiN5o6hyF8wgMv3Md999pznCdCc6+Y+9w6WjkMETQUFBaihzRavPi3Nyzh2oH7cS3wSPx10T36iU9yAiKo4MBpCaExlpWdFO7uQYZPSuDAqQqVys+5jZi0zke91116lBJ1fqg+gKVqxYgSeffFL93DIdT2X8bZYlnzhkU6wr8wipp27dUmPsXRQiInIB0ldOgp0jhDrzaFXpjyYhpTpIS0tT4boioc6WHKMU1YjPVRNwza4WiM2qiaHZBvh42m5UGBERVT8//PADHI31xNCubtiwYXAkrLGrYsFRjXDBpz4yNC8cPXfJ3sUhIiIiF8JgZwdNwkx9DhjsiIiIyJYY7KpabjYe0n7AG+5zcSw2/3BpIiIioopgsKtqbh7od+4b/J/7GpyPtf9iwUREROQ6GOyqmk6HbD/TYsEZiSftXRoiIiJyIQx2dqAPNi13o0uJQY6h4rNpExEREQkGOzvwCjEFu3AtESfPp9u7OEREDq9fv35qwXZyDbLmrCznZjAYXP73n52djQYNGhS5dnFlYLCzA12QKdjV1iXiSEKqvYtDRER2otPpCm0LFy7Md4ysWdupUye1zqusP/vll18WOs/s2bNVeJAVC7p3746tW7eWav67Fi1aqNe0bdtWTXRsTRamkvV1IyMj1RqoAwYMwOHDh/MdI2umyjq7shpCjRo11Jq2slbslcgatbJ+rZubaS5X+Znk9c7ok08+UZMyyzWQrWfPnmo1CjNZ6/epp57Cs88+WyXlYbCzh6A66iZSdx5HEjjlCRFRdSarFsTGxlq2W265xfKcrN4gi9D3798fu3fvVrVWDzzwAP744w/LMYsWLcKkSZMwbdo07Ny5E+3bt8egQYOQkJBQ7Htu3LgRI0aMUEFs165d6j1l27t3r+WYt956Cx988AHmzJmDLVu2wM/PT51Xlr4yk1C3b98+rFq1Cr/99hvWrVuHBx98sMSf959//lHLjd1+++1wBXXq1MEbb7yBHTt2qFq5a665BkOHDlXXxfo6yc9tva/SVGjFWhdTlkV2K+TI35o2LVCLntJSm/jdzsp9LyKiUi407shk0fUJEyaoLTAwUKtZs6ZaxN1oNFqO+eqrr7TOnTurRd7Dw8O1ESNGaPHx8Zbnk5KStJEjR2qhoaGat7e31qRJE23+/PmW50+dOqXdcccdakH64OBg7eabb1YLyVemggvXF/TMM89orVu3zrdv+PDh2qBBgyyPu3Xrpq6LmcFg0KKiorQZM2YUe94777xTu/HGG/Pt6969u/bQQw+p+3JdIyIitLffftvy/MWLFzUvLy/LYvfyOZLyb9u2zXLMihUrNJ1Op8XExBT73lLWYcOG5dv3xRdfqOte0u9/4sSJpf5dr169WpXt999/1zp06KB+3/3791fHLF++XGvRooUWEBCgXpeWlpav/L1791ZlCQkJUdfoyJEjWlnJ5+ezzz7Lt0/eXz6z5fnbLEs+YY2dPdTthvXXr8DN2a/iCCcpJiIqlQULFqj1OKWZ8f3338fMmTPx2WefWZ7PycnB9OnT8e+//2Lx4sU4ceJEvqWtpkyZgv3796tmsgMHDqgmtNDQUMtrpTYqICAA69evx4YNG9QC9tdff73qI1UcOaakbdy4cVf8uSZMmKDK0a1bN8yfP181gZpt2rRJNYFak3LKfiFlk5oi62P0er16bD6mKFc6r9QUxsXF5TtGFqGXZl7zMXIrzaddunSxHCPHy/tLDV9x5Ppav6Y8rvS7NnvppZfw0UcfqRrK06dP484778SsWbPw7bffYtmyZVi5ciU+/PDDfOu+Su2n1LxJP0D5WW699VYYjaUb6Ch9BqUpXc4jTbLW5PcrP3tl41qx9uDph6jG7ZCJCziakAajUYNer7N3qYioOstOK/45nRvg4V3KY/WAh8+Vj/X0K3MR69ati/fee0/1Q2vevDn27NmjHo8dO1Y9f99991mObdSokWpG7Nq1q+rzJSHr1KlT6NixoyVUSJ806+ZM+fKWoCjnNzeRSnCRPm4DBw4sskzSPFoS6XNVkldeeUU13fn6+qqQ8fDDD6vyPvbYY+p5CVfh4eH5XiOPU1JSkJGRgQsXLqgwUdQxBw8eLPZ9izuv7Dc/b95X0jFhYWH5npfgHRISYjmmKCdPnkRUlGnar/K60u/a7NVXX0Xv3r3VfWl2fv7551UzsLzGvM7r6tWrLf3fCjYPS9CuVauW+gdBmzZtUBz5LEqQk2Zqef9ffvkFrVq1yneM/Mzys1c2Bjs7qR/iCw83HTJyDIi5mIG6Ib72LhIRVWevl/BF23QgcJfVQvNvNwFyihnRX/8qYMyyy49ntQXSzxc+7qXkMhexR48eltAl5Iv03XffVcFGOuFLzZXU0EgtjgQecy2LBDr5kh0/frz64pZ+aBLUpE9Zr1691DHymiNHjqgaO2vyRS1BoDgymKEipBbRTEKn1PS8/fbblmDniiSQyoCNirjS79pMBjVYh1IJ0OZQZ95nPdBEBofIgBGpcUxMTMx33pKCnfxDQ0J+cnIyfvzxR4wePRpr167NVxYZgJKeXvkzYbAp1k7cd3+ND3zno4PuCJtjiYgqSAKRNCVKDdk333yDbdu2qVoTYW5KHTx4sKoxeeKJJ3D27Flce+21arSikJqezp07qy9n6+3QoUMYOXJkpTbFWpOmzjNnziArK0s9joiIQHx8fL5j5LH8nBIUpAlXQm1Rx8hri1Pcec2vMd9e6ZiCAzRyc3PVSNmS3lvKLGGsMn/XZh4eHpb78o8C68fmfdbNrEOGDFHlnzdvngp35iblkprjzSNfJeTLZ2jGjBlqAIt0F7Am55Xav8rGGjt7OfQHBueswgZ9HRxNuIT+zfNXZxMRVanJZ0tuirX29JESji1QX/D4HthKwX5bmzdvRtOmTVWwkWbH8+fPq9GJ0mQripo3TL5YpTZFtj59+uDpp5/GO++8o6YTkeZYaVq8UvOpLZtiizpfcHCwmtrEXCtZcBoSGYFq7r8lgULChPQHM4+mlaAijx955JFi30deL8dYzw1nfd6GDRuqcCbHdOjQQe2T5l/5HUjNp/kcFy9eVLVnUgbx999/q/eXgFocqZmUps3yKu3vuqzknNHR0SrUyWdDyEjW8pBrYA7nZjLiWH72ysZgZ+cpT6I45QkROYKy9HmrrGOvQJrDpGP7Qw89pJpTpdO7NMWKevXqqZAj+6SWTL5EpXO9NWlikwDSunVr9aUr03PIJLnm6SikCVSmqZB+bzKFhdTu/fzzz2rONXls66bYX3/9VdWASROzNE1KsHr99dcttYhCfhbp/C9lkH5lEpy+//571fHfTK6JBFXpOygd9GVwgNRqjRkzptj3njhxIvr27auun0ynIh3+JRzNnTvXUpMloU/6qEl4lqAnzcbST8wcIOXayeAS6eMoU6LIgAYJk//3f/9XYh86qW2TgTAFSZN6waAsAdf8OzIrze+6PCRQ16xZU10DmbtPPm/PPffcFV8n/fakNljKlZqaqgZmSL9M6ylphAycsEU5r6jMY3hdWJVNdyL+maWmPPnlxeu12z/eUPnvR0Tk5NOdPPzww9q4cePUdCcyncTkyZPzTXfy7bffag0aNFBTcvTs2VNbunSp+n/6rl271PPTp0/XWrZsqfn4+KipLIYOHaodO3bM8vrY2Fht1KhRajoUOUejRo20sWPHVtp3gkytIVNxyJQdfn5+Wvv27bU5c+ao6UqsydQdcpynp6cqk0wNUtCHH36o1atXTx0j059s3rw53/OjR49W19Da999/rzVr1ky9RqZUWbZsWb7n5dpOmTJFTSci1+Paa6/VoqOj8x1z/vx5NWWI/AzyexkzZoyWmppa4s8tr5HpRw4ePGjZJz+T/K4Kbo0bNy5yupMr/a5X5013cuHChRKnVJk2bZq67marVq1SnxE5b7t27bQ1a9ZccUqa++67T6tfv766jrVq1VLXaeXKlfmO2bhxo1ajRg0tPT290qc70cl/Kj8+OgepZpbh3NL5sazV52W29yfgx/uw1dgcY91exe6p1+XrFExEVBlkMIBMZSE1MBXtwE7OQ2rnZJJjGXDgCKQJXL5zP/30U1QHw4cPV/3uJk+eXK6/zbLkEw6esJe8ZcWkKTY5IweJl0rumElERFQeEgZkZK91E6+9vfDCC6hfv36p54dzZtnZ2WrJNhm0UxXYx87OfewidEnQw6j62dUKMHWWJSIishWp6ZGRto5E5gcsqfbKlXh6eqp1casKa+zsxT8c0LvDHUbUwkVOeUJEREQVxmBnL3o34JFteKvzGsQjRE15QkRERFQRDHb2FNIIDSJM6xRyyhMiIiKqKAY7O2scZlrT7nBCqr2LQkTVCCdEIHLNv0kGO3s6thZtd7yAe9xWIj4lCymZOfYuERG5OPOSSlWxZiURlZ75b7LgsmdlxVGx9nT+CDz/+wYDPLvi64yBqp9dx3rB9i4VEbkwWX5LRiSa1/iURdE5hyaRfWvqJNTJ36T8bcrfaEUw2DnAXHb13ZMs/ewY7IiospkXaC+4gDsR2Y+EOvPfZkUw2NlTUG11E2Y8p2455QkRVQWpoZO1MGXBe1nfk4jsS5pfK1pTZ8Zg5wCTFPsaUuCDTByJZ7AjoqojXyS2+jIhIsfAwRP25B0EeAValhZjjR0RERFVBIOdg9TaSbA7nZSOzByDvUtERERETorBzt4CTf3s6numwKgBxxPT7F0iIiIiclIMdvZ2yyfA5FjsD7tJPeQKFERERFReDHb25l8L8PRFk7wVKBjsiIiIqLwY7BwEgx0RERFVFIOdvV08DSx5BEOOTVcPGeyIiIiovDiPnb1pBmDX14hw8wIwTA2eyDUY4e7GzE1ERERlw/RgbwFRMg88dIYsRHlcQrbBiNMXMuxdKiIiInJCDHb25u4J+Ieru11rpKtbNscSERFReTDYOdAkxW0DUtXt4QTTLREREVFZMNg5ULBr6p2sblljR0REROXBYOdAwa6e23l1Gx3HGjsiIiIqOwY7Bwp2tdxMfewOx19CjsFo50IRERGRs2GwcwSdRqllxXzvnAt/L3c1MvboOTbHEhERUdkw2DkCTz+1rJher0PLyAC160Bsir1LRURERE6Gwc7BtIwMVLf7zzLYERERUdkw2DmKZU8BX92CLjXS1MP9rLEjIiKiMmKwcxTHVquttU+SenggNhWaptm7VEREROREGOwcbGRsXbck6HVAUlo24lOy7F0qIiIiciIMdo4i0BTsPC+dReNa/uo+B1AQERFRWTDYOViNHVLOXB5AwWBHREREZcBg52jBLvkMWkUx2BEREVHZMdg5YLAz19gd4JQnREREVAYMdo4W7HLS0Sov2B0/n4b07Fz7louIiIichru9C0B5QhqpZcVkBYpaAEL9vZB4KQsH41LRqV6wvUtHREREToA1do5C76ZCnZm5nx1HxhIREVFpMdg5KHNzLJcWIyIiotJiU6wj2ToPOLgM6Hg3Wkb2VLtYY0dERESlxRo7R5J42LS0WPxetM5ripU+dgYjlxYjIiKiK2OwcyRBtU23F0+jQU0/eLnrkZ5twMnzafYuGRERETkBBjtHUqOe6fbiSbi76dEiIkA9PBCbat9yERERkVNw6GA3e/ZsNGjQAN7e3ujevTu2bt1a4vGzZs1C8+bN4ePjg7p16+KJJ55AZmYmnEZIY9Nt0jF1c3lpsWR7loqIiIichMMGu0WLFmHSpEmYNm0adu7cifbt22PQoEFISEgo8vhvv/0Wzz33nDr+wIED+Pzzz9U5Jk+eDKeay06knwcyLlpNecIaOyIiInLiYDdz5kyMHTsWY8aMQatWrTBnzhz4+vpi/vz5RR6/ceNG9O7dGyNHjlS1fAMHDsSIESOuWMvnULz8Af9w0/2ko5dr7DjlCRERETlrsMvOzsaOHTswYMAAyz69Xq8eb9q0qcjX9OrVS73GHOSOHTuG5cuX44Ybbij2fbKyspCSkpJvc4jmWN9QVWNn7mMXl5KJpLRse5eMiIiIHJxDzmOXmJgIg8GA8PC82qs88vjgwYNFvkZq6uR1V111FTRNQ25uLsaNG1diU+yMGTPw8ssvw6GMWgy4e6m7Euvq1/TFyfPpaj673k1C7V06IiIicmAOWWNXHmvWrMHrr7+Ojz/+WPXJ+/nnn7Fs2TJMnz692Nc8//zzSE5OtmynT5+G3eWFOrOWEVxajIiIiJy4xi40NBRubm6Ij4/Pt18eR0REFPmaKVOm4J577sEDDzygHrdt2xZpaWl48MEH8cILL6im3IK8vLzU5shkAMXv++LYz46IiIics8bO09MTnTt3xl9//WXZZzQa1eOePU1LbRWUnp5eKLxJOBTSNOs0UmKBr28FPu1bYMoTBjsiIiJywho7IVOdjB49Gl26dEG3bt3UHHVSAyejZMWoUaNQu3Zt1U9ODBkyRI2k7dixo5rz7siRI6oWT/abA55T8AoAjv5tup9xwTLlyZGES8jKNcDL3Yl+FiIiIqpSDhvshg8fjnPnzmHq1KmIi4tDhw4d8Pvvv1sGVJw6dSpfDd2LL74InU6nbmNiYlCrVi0V6l577TU4FfOUJ5fi1UTFUVGdEOjtjpTMXBXuWkcF2buERERE5KB0mlO1U1Yume4kKChIDaQIDDTVlNnF/MHAqY3A7Z8DbYfh/+ZuwuZjSXh7WDvc0aWu/cpFREREDp1PHLKPXbVnXoGi0NJi7GdHRERExWOwc0Q184Ld+aPqplVesOOUJ0RERFQSBjsnqLEzD6CQKU/Yck5ERETFYbBzRLKsmE8I4G0aKNEkzB/uep0aQHE2OdPepSMiIiIH5bCjYqu1iLbAs8ctD2WKEwl3B+NSVa1d7Ro+di0eEREROSbW2Dkina7QLvazIyIioithsHMS1v3siIiIiIrCYOeoNn0MfNARWP9uvilPDsQx2BEREVHRGOwcVW6GaVTsuUP5gt3J8+lIzcyxc+GIiIjIETHYOfyUJ6a57EL8PBER6K3uR8el2rNkRERE5KAY7Bx5yhOruezy9bPjAAoiIiIqAoOdowppaLpNPw9kXMw3MpYDKIiIiKgoDHaOyisA8A/PV2vXprYp2P17JtmeJSMiIiIHxWDnRM2x7evWULfRcSlIz861Z8mIiIjIATHYObLI9kBUJ8DN0/QwyAfhgV4wasAe1toRERFRAVxSzJENfqPQrg51a+CPffH498xFdG9U0y7FIiIiIsfEGjsn06FusLrdfdo0oIKIiIjIjMHOGRgN+WrsxO5TDHZERESUH4OdI8vNBj7qCrwWAWSa+tS1qxMEvQ44m5yJhJRMe5eQiIiIHAiDnSNz9zQFOkM2cN60AoWflzuahQeo+7vYHEtERERWGOyccAWK9nXymmMZ7IiIiMgKg53TrBl7Odh1qMd+dkRERFQYg52zLC1mHezyBlD8d+YiDDKpHRERERGDnROomdcUm9fHTkgfO19PN6RlG3Ak4ZL9ykZEREQOhcHOCZti3fQ6tK0dpO7vPn3BXiUjIiIiB8Ng5wzBLqId0LAPYMgp3M/uNJcWIyIiIhMuKebovAKAcesL7e5onqiYI2OJiIgoD2vsnHxpsei4FKRn59q7OEREROQAGOychdEIZF0eKBER5I2IQG/IoNg9Z9gcS0RERAx2zmH7fNOyYsufyrfbsm4sm2OJiIiIwc5J+AQDhqx8U56I9gx2REREZIXBzqmmPMkf7FhjR0RERNYY7Jwp2KWfBzIuh7h2dYKg1wGxyZmIT8m0X/mIiIjIITDYOcuUJ35hhSYq9vNyV6tQiF1cN5aIiKjaY7BztqXFrIKdYHMsERERmTHYOYuQKwU7Li1GRERU3XHlCWdRrweQlQLUbJJvt3lpMZnLzmDU1DqyREREVD0x2DmLTveYtgKahgXAz9MNadkGHEm4hOYRpj53REREVP2wKdbJSQ1d2zpB6j6bY4mIiKo3BjtnW1YsOQbIzS5y3VgOoCAiIqreGOycyQcdgPdaAQn78u3uUNdUY8cpT4iIiKo3BjtnEhBpui2wtJi5xu5QfCrSsnLtUTIiIiJyAAx2LjCXXUSQNyICvWHUgD0xyfYpGxEREdkdg50zLi1WoMZOcKJiIiIiYrBzJqHNTLfnDhZ6yjyf3W72syMiIqq2GOycSXjry8HOaMj3FGvsiIiIiMHOmQQ3ANx9gNxMIOl4vqfa1g6CLDoRl5KJuORMuxWRiIiI7IfBzpno3YDO9wK9Hwc8vPM95efljmbhplUnWGtHRERUPZU62B09WrjDPtnB4DeA614GguoUeqqjuZ8dgx0REVG1VOq1YseNG4cjR44gIiIC7dq1y7cFBZkmyCX76lgvGN9tPY3tJ5LsXRQiIiJy5Bq7VatW4fjx4xgyZAgSEhIQExODV199FSEhIWjSpEnllpLyu5QAnNpcaHf3hiHq9t8zF5GRnX9wBREREbm+UtfYmX3//ffYvXu35fHKlSvxzTff2LpcVFKoe6cpoNMDk88CHj6Wp+qF+CIyyBuxyZnYeeoCejcJtWtRiYiIyMEHT3h7e2P//v2WxwMHDsTevXttXS4qjl8twLcmoBkLzWen0+kstXZbjp23UwGJiIjIaWrsPv/8cwwfPhz9+vVDhw4dsGfPHhUoqIrItQ5rBZxYD8TvB6I65nu6R6OaWLz7LDYfYz87IiKi6qbUNXbHjpnWJ23dujV27NiBPn364MSJE6hfvz5WrFhRmWWk4iYqTrhcc2rWvVFNy8jYzBz2syMiIqpOSl1j9+KLL6pw16lTJ9X8OnjwYNx5552VWzoqmtTYifh9hZ5qUNMXYQFeSEjNUv3sejVmPzsiIqLqotTB7ttvv4Wmaaq2TgZMfPjhh+px//79VdDr1q0bm2QdoMZOfgfSHLv037PYciyJwY6IiKgaKdPgCQkNXbp0weTJk/HXX39h6dKlaN++Pb766it079698kpJ+dVqYbq9FA+kFR4k0b2RaQDFZg6gICIiqlbKPHjCmr+/P26++Wa1URXy8gf6PgcERgFuHoWelho7sSuvn523h5sdCklEREQOHey2bNmimmQ3btyI2NhY+Pr6omXLlqq/3YgRI7gCRVXq/3yxTzUK9UOovxcSL2Xh39MXLQMqiIiIyLWVuin2pptuwhdffKH600kTrIyI3blzJ15++WVkZWVh2LBhaj/Zn5rPztIcy2lPiIiIqgudJiMgSuHixYuoUaNGhY9xZCkpKarWMTk5GYGBgXBoOZlA7L+mfnatCjeFf735JKYs3otejWvi27E97FJEIiIiqtp8Uuoau5ICmzTRXukYsrELJ4D5A4HF4wGjsdDTPfJWoNhx8gKycjmfHRERUXVQ5iXFinLHHXegMsyePRsNGjRQy5jJqNutW7descZwwoQJiIyMhJeXF5o1a4bly5fDJdVsDOg9gOxLQPKpQk83CfNHTT9PZOUa8d+ZZLsUkYiIiBx08ERxkxFLS25Sku37cS1atAiTJk3CnDlzVKibNWsWBg0ahOjoaISFhRU6Pjs7G9ddd5167scff0Tt2rVx8uRJ161FlNGwtZoD8XuBhANAcIMi+9kt3xOHzUfPo2sDUw0eERERua5SB7s///wTX3/9tZripGCwW7dunc0LNnPmTIwdOxZjxoxRjyXgLVu2DPPnz8dzzz1X6HjZLwFTRux6eJimAJHaPpdfgUKCnaxA0XxwkdOeSLDbcjwJj9qlgEREROSQwa5fv34ICAjA1VdfXei5du3a2bRQUvsmK1w8//zlKT30ej0GDBiATZs2FfkaGZHbs2dP1RS7ZMkS1KpVCyNHjsSzzz4LN7ei53GT0byyWXdOdCrhrYA9Ra9AIbo3rGnpZ5eda4Snu01a3omIiMhBlfqb/ueffy4y1IlVq1bZskxITEyEwWBAeHh4vv3yOC4ursjXyDq20gQrr5N+dVOmTMG7776LV199tdj3mTFjhhplYt7q1q0LpxKWt7RYfNHBrmmYP4J9PZCRY8CemItVWzYiIiKqci5ThWM0GlX/urlz56Jz584YPnw4XnjhBdWEWxypEZShw+bt9OnTcLoaO3H+MJCbXehpvV5nqbXjfHZERESuzyGDXWhoqGo+jY+Pz7dfHkdERBT5GhkJK6NgrZtdZVUMqeGTpt2iyMhZmQ/GenMqgbWBG94B7lksoyWKPITrxhIREVUf5Q52xTWJ2oKnp6eqdfvrr7/y1cjJY+lHV5TevXvjyJEj6jizQ4cOqcAn53NJEua6jQUa9ilyzVjrdWOln12OofB8d0REROQ6yh3sZGmxyiRTncybNw8LFizAgQMHMH78eKSlpVlGyY4aNSrf4Ap5XkbFTpw4UQU6GUH7+uuvq8EU1Vnz8ADU8PVAerb0s+N8dkRERK6s1KNiCyrlSmTlJn3kzp07h6lTp6rawQ4dOuD333+3DKg4deqUGilrJgMf/vjjDzzxxBNqlK7MYychT0bFurRL54AjfwLGHKDTqCL72ckcdqv2x2PLsSR0qhdsl2ISERGRA60VW5CEp//++w+uxKnWijU7uRH4YjAQVBd4Ym+Rh3z+z3FM/20/+jWvhS/HdKvyIhIREZGDrRVLDiqspek2+TSQWXRTa/e8dWO3HU9CLvvZERERuSwGO2fnE2waHStkabEitIwMRKC3O9KyDdh31skmYSYiIqLKD3bFreZAdlpaTMjSYkVw0+vQLa/WjtOeEBERua5yB7tdu3bZtiRU8YmKi1lazHraE1k3loiIiFwTm2JdwRWWFhPmFSikn53BWLkjmomIiMg+GOxcqsZun8xDU+QhraICEeDljtSsXOxnPzsiIqLqHezee+89dbtv3z4YDIbKLBOVVWgz4K4fgfGbij1E+tl1zetnt+U4+9kRERFV6wmKZYJgMXnyZBw8eBA+Pj5o3bo12rZtizZt2uCmm26qzHJSSdy9gKbXXfGwHo1C8PfBBKw/nIgH+jSqkqIRERGRA9bY9e/fX90uWbIE0dHR+Oeff/DYY48hNDQUf/75Z2WWkWykb7Mwy8jYzBzWuhIREVXbYHfdddfhs88+U8t8CX9/f3Tv3h333Xcf7rzzTjz88MP48ssvK7OsVJKEg8DfrwKbPyn2kGbh/ogM8kZWrhGbOO0JERFR9Q12ixcvRlpammpybdiwITp16oRWrVqhcePGmDt3LkaPHo177723cktLxUs6Cqx7G9j9TbGH6HQ6tayYWBttCuhERERUDfvY+fn5YeLEiWrLzs7G+fPn4e3tjeBgLirvUJMUnzsEGHIBN/dim2O/23oaa6ITAORNk0JERETVd7oTT09PREZGMtQ5khr1AQ8/wJBlqr0rRu8mNeHhpsOJ8+k4kZhWpUUkIiIiBwp2GRkZiImJKbRfpkAhO9PrgbAWJS4tJgK8PdClvmnaE1OtHREREVW7YPfjjz+iadOmuPHGG9GuXTts2bLF8tw999xTWeWj8jTHJhwo8TBzP7vV7GdHRERUPYPdq6++ih07dmD37t344osvcP/99+Pbb79Vz2nFrHZAVSy8jek27r8SD+vXnNOeEBERVevBEzk5OQgPD1f3O3fujHXr1uHWW2/FkSNH1GhLcgC1O5luEw+VeJhMexIV5I2zyZlq2pP+eUGPiIiIqkmNXVhYGP7773JNUEhICFatWoUDBw7k2092FNkBmLAVeGRHiYdJEO+bF+Y47QkREVE1DHZff/21CncFR8d+9913WLt2bWWUjcrK3ROo1dw0kOIKzP3sOICCiIioGga7OnXqICIiwvI4Li7Ocr937962LxlVqt5NQi3TnhzntCdERETVdx47MXDgQNuWhGwj8TDw4/3AD2NKPMzfy53TnhAREbmYcgc7joR1UDo9sPdH4OAyIDe7xEP7tzA3x7KfHRERUbUOdhwJ66BCGgE+waYVKOL3lngopz0hIiJyLeUOduSgJHDX7my6H1Py6NimYaZpT7JyjWraEyIiInJuDHauqHYX0+2Z7aWe9mTNQfazIyIiqrbBzs3NzbYlIdupkxfsYkoOdqK/edqTQ+xnR0REVG2D3a5du2xbErIdc1Ps+SNAxoUSD+2VN+3JSU57QkRE5PQq3BS7fv163H333ejZsydiYmIskxn/888/tigflYdvCBDaDIhoC6TGX3Hak64NOO0JERERqnuw++mnnzBo0CD4+PioGrysrCy1Pzk5Ga+//rqtykjlMX4TMO4fIKxFGVahYHMsERFRtQ12r776KubMmYN58+bBw8Mj30oUO3futEX5qLzc3Et9aP+8ARQyMjYjm9OeEBERVctgFx0djauvvrrQ/qCgIFy8eLEipyZbyc2S2aRLPKRJmD9q1/BBdq5RzWlHRERE1TDYydqxR44cKbRf+tc1atSoIqemipIw98UNwIw6QNKxUkx7Ym6OZT87IiKiahnsxo4di4kTJ2LLli0qHJw9exbffPMNnnrqKYwfP952paTyTVQstXWG7CtOVCz6NeO0J0RERM6u9B2xivDcc8/BaDTi2muvRXp6umqW9fLyUsHu0UcftV0pqfzz2clcdjJRcbs7Szy0d4FpTxqG+lVZMYmIiMgBauyklu6FF15AUlIS9u7di82bN+PcuXOYPn26jYpHFVKnq+n2zLYrHurn5Y5uDU3TnqzmKhRERETVd0kxT09PtGrVCt26dYO/v78tTkm2nKg4bg+Qk1nq0bEr98dVdsmIiIjI0YJdRkaGaoI1O3nyJGbNmoU//vjDFmWjigpuAPjWBIw5pnB3Bde3iVC3W48n4VyqaU5CIiIiqibBbujQofjqq6/UfZneRGrs3n33Xdxyyy345JNPbFVGqsgAitqlXze2TrAv2tcJglED/tjHWjsiIqJqFexkEuI+ffqo+z/++KOa/kRq7STsffDBB7YqI1VE0+uAljcDwQ1LdfgNbSPV7Yq9sZVcMCIiInKoYCfNsAEBAer+ypUrcdttt0Gv16NHjx4q4JED6DYWGP410Pz6MgW7TUfP4/wlNscSERFVm2DXpEkTLF68GKdPn1b96gYOHKj2JyQkIDAw0FZlpCpUN8QXbWubm2Pj7V0cIiIiqqpgN3XqVDVnXYMGDVT/up49e1pq7zp27FiRU5OtV6FIOg6klK55lc2xRERE1TDYDRs2DKdOncL27dvzjYSVCYvfe+89W5SPbGHZk8AHHYAdX5Tq8BvamkbHbjx6Hklp2ZVcOCIiInKIlSeEDJiQCYqlli47+3IIiIuLQ4sWLSp6erKF8FalnqhY1K/ph9ZRgdh3NgWr9sdheNd6lVs+IiIisn+wO3bsGG699Vbs2bNHrUKhSZNf3ooUwmAw2KaUVDGWKU92AEYjoNeXqjlWgt2yPQx2RERE1aIpduLEiWjYsKEaLOHr64t9+/Zh3bp16NKlC9asWWO7UlLFhLcG3L2BzGQg6WipXjI4b7LijUcScTGdzbFEREQuH+w2bdqEV155BaGhoWqaE9muuuoqzJgxA4899pjtSkkV4+YBRHYw3T9z5YmKRaNa/mgREYBco4aV+zk6loiIyOWDnTS1muexk3B39uxZdb9+/fqIjo62TQnJNuqUfgUKsxvzRscu38PRsURERC4f7Nq0aYN///1X3e/evTveeustbNiwQdXiNWrUyFZlJFuo3blMNXZicF6w23AkEcnpOZVVMiIiInKEYPfiiy/CKJ3xARXmjh8/rpYYW758OZcUczT1egC9HgP6PlPqlzQJ80fz8ADkGDSsOsDmWCIiIken08xDWW1Epj4JDg62jIx1JikpKQgKCkJycjJXzsgz689DmPXnYVzTIgzz7+1q7+IQERFVOyllyCcVqrGzJvlQtpCQEKcMdVRyP7v1h88hJZPNsURERI6swsHu888/V33tvL291Sb3P/vsM9uUjmwrJxM4uhr4d1GpX9I0PABNw/xVc+yfHB1LRETk2mvFylx2Q4YMwQ8//KA2uf/EE0+o58jBxO0Bvr4FWPEMYDSUeRDF8j1xlVg4IiIismsfu1q1aqlBEiNGjMi3/7vvvsOjjz6KxMREOBOX72NnyAXeaghkpQAPrgGiOpbqZdFxqRg0ax083fXY8eIABHh7VHpRiYiIqIr72OXk5KhVJgrq3LkzcnNzK3Jqqgxu7kCDq0z3j5V+ZZBm4f5oVMsP2blG/HUgofLKR0RERBVSoWB3zz334JNPPim0f+7cubjrrrsqcmqqLI36lTnYyWAYTlZMRETk+NzL+oJJkybl+8KXgRIrV65Ejx491L4tW7bg1KlTGDVqlG1LSrbRsK/p9tRm02AKD+9SvWxwm0h8+PcRrDl0DpeycuHvVeaPDhEREVWyMn8779q1q1Czqzh69KhlaTHZ9u3bZ6syki3Vag74RwCX4oDTW4BGeUHvClpGBqBhqB+OJ6bhrwPxGNqhdqUXlYiIiCo52K1evbqsLyFHInMMSpj7bxFwcmOpg53Uzt7QNgKzVx/Fkt1nGeyIiIgckM0mKCYnctUTwLgNQN9ny/Sy2zrVUbdrohMQn5JZSYUjIiKi8mKwq47CWgIRbQB92X79jWv5o0v9YBg14OedMZVWPCIiIiofBjsqkzu6mGrtfth+Wi0hR0RERC4U7M6cOQOj0Vjovi3Mnj0bDRo0UEuVde/eHVu3bi3V6xYuXKj6hN1yyy02K4vLOb0N+PlBYPWMMr3sxnZR8PFww7HENOw4eaHSikdERER2CHatWrXCiRMnCt2vqEWLFqmpVaZNm4adO3eiffv2GDRoEBISSp4gV97/qaeeQp8+fWxSDpclo2JlAMXen8r0Mpnm5Ia8Oe1+2H6mkgpHREREdgl21s1xtmyamzlzJsaOHYsxY8aowDhnzhz4+vpi/vz5xb7GYDCoiZFffvllNGrUyGZlcUmyAoVOD5w/DCSXrb/cnXnNsb/9dxbp2VxhhIiIyFE4ZB+77Oxs7NixAwMGDLDs0+v16vGmTZuKfd0rr7yCsLAw3H///aV6n6ysLLX+mvVWbfgEX14r9vjaMr20W8MQNKjpi7RsA5bviauc8hEREZFrBLvExERV+xYeHp5vvzyOiys6SPzzzz/4/PPPMW/evFK/z4wZM9Siuuatbt26qJarUJRheTEh/ReHdTbV2n2//XRllIyIiIhcJdiVVWpqqlq3VkKdrHpRWs8//zySk5Mt2+nTp6vpurFrpR29TC+9vXMdNdfx1uNJOJGYVjnlIyIiojJxyAU/JZy5ubkhPj4+3355HBERUeh4Wc5MBk0MGTLEss88Otfd3R3R0dFo3Lhxodd5eXmprdqq2x1w9zYNpDgXDYS1KPVLI4N8cHXTWlh76Bx+3HEGTw1qXqlFJSIiIietsfP09FRr0P7111/5gpo87tmzZ6HjW7RogT179mD37t2W7eabb0b//v3V/WrXxFpaHt5A/V5AeFsg40K557STYGeQWYuJiIjIrhyyxk7IVCejR49Gly5d0K1bN8yaNQtpaWlqlKwYNWoUateurfrJyTx3bdq0yff6GjVqqNuC+6mAkd8Dbh7leul1rcJRw9cDcSmZWH/4HPo1D7N58YiIiKgKg93kyZMREhJS6H5FDR8+HOfOncPUqVPVgIkOHTrg999/twyoOHXqlBopSxVUzlAnvNzdcEuH2vhy4wn8sOMMgx0REZGd6TSuC2Uh053I6FgZSBEYGIhqJTsdMOYA3kFletnemGTc9OE/8HTTY8vkaxHs51lpRSQiIqqOUsqQT1jlRaZlxd6sD2yZW+aXtqkdhFaRgcg2GLFkd9kmOiYiIiLbYrAjwL8WYMgu83x2BQdRSHMsERER2Q+DHQGN+ptuz2wFsss+J530s5Om2H1nU7DvbLLty0dERERVG+zee+89dbtv3z61agQ5kZBGQFBdU63dqeKXbCuO9KuTEbLih+2stSMiInL6YCejVs0jY1u1aqUe33XXXXjjjTfw22+/2eptqDLIEhLlXF7MbFhec+zi3THIymWwJyIicupgJ5MBiyVLlqiVHmTt1scee0ytIvHnn3/a6m2oKpYXKwdZhSIi0BsX03Owan/+FUOIiIjIQeexk3VZX3rpJSxfvhyJiYlq+G2zZs3Qu3dv3H777WoVCOHv74/u3burjZxAw6tNt3H/AWnnAb+aZXq5m16HYZ3r4KPVR/DVppO4qV1U5ZSTiIiIbFdjJys+/PDDDxg5ciRee+01PProo/j777/x9ddfo3Xr1mopr5gYTnvhdALCgS73A4NmAOWc+PmuHvXgrtdh6/EkNb8dEREROfgExX5+fqqZtWPHjpZ9AQEB+Pfff+Hm5qbC3rJly9QxDRs2hDOp1hMU28hj3+3C0n/P4raOtTFzuKnfJRERETnoBMWypFd6enqRz9WvXx9z587F+PHjMXHixLKemlzA/VeZwvyv/51FQkqmvYtDRERUrZQ52D3yyCO47777VA1dce6++27VPEtO6NI5YPsXwNnd5Xp5+7o10KV+MHIMmuprR0RERA4c7CZNmoQhQ4agU6dOuP766zFnzhwYjUboZMqMPAsXLlSjYckJ/fUS8NvjwK6vK1xr982Wk8jM4dQnREREVaVcveTfeecdbNy4UfWte/LJJ5GRkYH27dujUaNGqFmzJqZPn463337b9qWlytdyqOn2wG+A0ViuUwxsHYE6wT64kJ6Dn3dyIA0REZHDTndiJtOYyOjY7Oxs7Ny5E4cOHVKd+6Sm7pprrkFYWJhtS0pVo1FfwCsQuBRnWmKsXo8yn0KmPrm3VwO8uuwA5m84jhHd6uar0SUiIiIHC3Zmnp6e6NGjh9rIBbh7Ac2uB/Z8D+xfWq5gJ+7sWhfvrTqEIwmXsO5wIvo2q2XzohIREVElrTxBLqTVzabbA78CZZsNxyLQ20OFO/H5P8dtWToiIiIqBoMdFdb4WsDDF0g+BZzdVe7TjOnVUC1Du+7QORyOT7VpEYmIiKgwBjsqzNMXaHodoHMzLTFWTvVq+uK6luHq/vwNJ2xYQCIiIioKgx0VbcDLwFOHgc73Vug05qlPft55Bklp2TYqHBERERWFwY6KFtIQ8KtZ4dN0axiCNrUDkZVrxLdbOGExERFRZWKwoyvLzSr3S2WaE3OtnaxEkZ1bvrnxiIiI6MoY7Kh456KBzwcB866p0GlubBuFsAAvJKRmYdmeszYrHhEREeXHYEfF8w8DYrYD8XuBxCPlPo2nux6jeta3TH2ilXMKFSIiIioZgx0VzycYaNjXdP/AkgqdamT3+vBy12NvTAq2Hk+yTfmIiIgoHwY7KlnLIaZbWYWiAkL8PHF75zrq/kery1/7R0RERMVjsKOStbgJ0OmB2N3AhYqNah3ftzHc9TqsP5yIbSdYa0dERGRrDHZUMv9aQL1el5cYq4C6Ib64o4tpmbGZKw/ZonRERERkhcGOyrB2bMWaY8Uj1zSBp5sem46dx8ajiRUvGxEREVkw2FHp+tk1Gwx0ua/Cp6pdwwf/181Uazdr1WGOkCUiIrIhBju6ssAoYORCoP3/2eR0D/droqZA2XoiCRuOnLfJOYmIiIjBjuwgIsgbd3Wvp+7PXBXNWjsiIiIbYbCj0ks6Dmz4AEireN+48f0aw9tDj52nLmLtoXM2KR4REVF1x2BHpff9KGDVFODgsgqfKizAG/f0MK1GMXPVIdbaERER2QCDHZVe61tMt/9+Z5PTPdS3MXw83PDfmWT8dSDBJuckIiKqzhjsqPTajwB0bsCpTUDCwQqfLtTfC6N7NVD3WWtHRERUcQx2VLbRsc2uN93fucAmp3zo6kbw83TD/tgU/LEv3ibnJCIiqq4Y7KhsOt97uTk2J7PCpwv288R9VzVU92f9eQhGI2vtiIiIyovBjsqmybVAYB0g40KFlxgze+CqRgjwcsfBuFSs2Btnk3MSERFVRwx2VDZ6N6DTKMDDF7hkmxAW5OuB+/tcrrUzsNaOiIioXBjsqOy6PwQ8GQ30etRmp5Tm2EBvdxxOuIQlu2Nsdl4iIqLqhMGOys6nBuAdaNNTBnp7YFy/xur+m78fRFpWrk3PT0REVB0w2FHFnN0F5GbZ5FT39W6IeiG+iE/Jwkerj9jknERERNUJgx2V33cjgLn9gIO/2eR03h5umHJTK3X/8/XHcTwxzSbnJSIiqi4Y7Kj8ItqZbnd8abNTDmgZhr7NaiHbYMT03/bb7LxERETVAYMdlV/HuwGdHji+Djh/1Can1Ol0mDqkFdz1Ovx9MAF/H+SkxURERKXFYEflV6Mu0OQ6m65EIRrX8rdMWvzKr/uRlWuw2bmJiIhcGYMd2WYlil3fALnZNjvto9c0Qa0AL5w4n475/5yw2XmJiIhcGYMdVUzTgUBAJJCeCEQvs9lpA7w98Nz1LdT9D/8+jLjkii9fRkRE5OoY7Khi3NyBjveY7u9fatNT39qxNjrWq4H0bAPeWHHApucmIiJyRQx2ZJvm2BELgdvm2fS0er0OL9/cGjodsHj3WWw7kWTT8xMREbkaBjuquKDaQPPBpto7G2tXpwaGd6mr7k9bso/ryBIREZWAwY5sy5Br2mzoqUHNEeDtjv2xKfhu6ymbnpuIiMiVMNiR7Wz6GJjVxmYrUZiF+nth0nXN1P13VkbjQprtRt8SERG5EgY7sp3080BqLLDxA0CzbZPp3T3qo1m4Py6m5+DVZRxIQUREVBQGO7Kd7uMAdx8gZgdw9G+bntrDTY/Xb22rBlL8tPMMVu3nihREREQFMdiR7fjXArqMMd1f97bNa+26NAjB2D6N1P3nf96DJDbJEhER5cNgR7bV6zHAzQs4tQk48Y/NTy997ZqE+SPxUhamLNlr8/MTERE5MwY7sq3ASKBT3oTF696y+em9Pdww8872cNPrsOy/WPz671mbvwcREZGzYrAj2+v9OKD3AI6vA84dqpS57Sb0a6zuS61dQiqXGyMiIhIMdmR7NeoCN74DjN8I1DJNU2Jrj1zTFK0iA9Uo2ck/74Fm4/58REREzojBjipvmbHw1pV2ek93PWYObw8PNx3+PJCAn3bGVNp7EREROQsGO6p8aYmVctoWEYF4fICpRvDlpftw9mJGpbwPERGRs2Cwo8ojzaO/TgTebQHE/lcpb/HQ1Y3QoW4NpGbl4tmf/mOTLBERVWsOHexmz56NBg0awNvbG927d8fWrVuLPXbevHno06cPgoOD1TZgwIASj6cqILMJZ10CjDmmee0qgbubHu/e2R5e7nqsP5yIb7ZwLVkiIqq+HDbYLVq0CJMmTcK0adOwc+dOtG/fHoMGDUJCQkKRx69ZswYjRozA6tWrsWnTJtStWxcDBw5ETAz7XtnV1U+Zbg8sBRIqZymwxrX88cz1LdT915cfwInEtEp5HyIiIken0xy07Upq6Lp27YqPPvpIPTYajSqsPfroo3juueeu+HqDwaBq7uT1o0aNKtV7pqSkICgoCMnJyQgMDKzwz0B5Ft1jCnZthgHDPq+UtzAaNYyYtxlbjiehZWQgfh7fCz6ebpXyXkRERFWpLPnEIWvssrOzsWPHDtWcaqbX69VjqY0rjfT0dOTk5CAkJKTYY7KystTFst6oElz9tOl2389A4pFKeQu9XodZ/9cBof6eOBCbghd+4RQoRERU/ThksEtMTFQ1buHh4fn2y+O4uLhSnePZZ59FVFRUvnBY0IwZM1QCNm9SI0iVILId0GwwoBmB9e9W3tsE+eDDEZ3UqhQ/74rB/zafrLT3IiIickQOGewq6o033sDChQvxyy+/qIEXxXn++edVtaZ5O336dJWWs1rpm1drF70cyEqttLfp2bgmnr2+ubr/ym/7sePkhUp7LyIiIkfjkMEuNDQUbm5uiI+Pz7dfHkdERJT42nfeeUcFu5UrV6Jdu3YlHuvl5aXaqq03qiS1OwO3fAI8tgvwCqjUtxrbpxFuaBuBHIOGh7/ZgXOpWZX6fkRERI7CIYOdp6cnOnfujL/++suyTwZPyOOePXsW+7q33noL06dPx++//44uXbpUUWmp1DqMBHyL7/NoKzqdDm8Na48mYf6IT8nCI9/uRK7BWOnvS0REZG8OGeyETHUic9MtWLAABw4cwPjx45GWloYxY8ao52WkqzSlmr355puYMmUK5s+fr+a+k754sl26dMmOPwUVSQY1RK8ADDmV9hb+Xu6Yc3dn+Hm6qZGyb/5+sNLei4iIyFE4bLAbPny4aladOnUqOnTogN27d6uaOPOAilOnTiE2NtZy/CeffKJG0w4bNgyRkZGWTc5BDubnscB3/wdsmVOpbyM1du/c0V7dn7f+OH7772ylvh8REZG9Oew8dvbAeeyqyK7/AUsmAJ7+wCPbgMCoSn27GSsO4NO1x+Dr6YbFE3qjWXjl9vEjIiKyJaefx45cXPuRQJ1uQPYlYOWLlf52Tw9sjp6NaiI924BxX+9ASmblNQETERHZE4MdVT29HrjxHUCnB/b+BBxfV6lvJ+vJfjiyIyKDvHEsMU2Fu6xcQ6W+JxERkT0w2JF9RLYHutxvur/86UodSCFC/b0wb1QXNZhi49HzmLToXxiM7IVARESuhcGO7OeaFwDfUODcwUofSCHa1A7Cp/d0gYebDsv2xOKVX/dx2TEiInIpDHZkPz7BwHUvAzWbAhElTyZtK1c1DcW7d3ZQ9xdsOomP1xytkvclIiKqCu5V8i5EJQ2kaHsn4O5ZZW95c/sonL+UhZd/3Y+3/4hGqL8nhnetV2XvT0REVFlYY0f2H0hhHeqMVbNCxJjeDfFwv8bq/vM/78Gq/fmXryMiInJGDHbkGGTwxIYPgM8HVPpACrOnBzXHHZ3rQMZQyLJj208kVcn7EhERVRYGO3IMMqfdhveBmB3A+plV8paypuyM29ri2hZhyMo14v4F23EoPrVK3puIiKgyMNiR4wykuH6G6f7aN4CTG6vkbWWOu49GdkKnejWQnJGD0fO34nRSepW8NxERka0x2JHjaHcn0H4EoBmBnx4A0qumadTH0w2fj+6q1paNTc7EnZ9uwrFzl6rkvYmIiGyJwY4cyw3vACGNgZQY03qyVTTPXLCfJ755oLtVuNuM6Dg2yxIRkXNhsCPH4uUP3PEF4OYJRC8HtnxaZW8dHuiNRQ/2QKvIQCReysLwuZuw50xylb0/ERFRRTHYkWMuNzbwVUDvUeVvXdPfC9+N7YEOdWvgYnoORs7bzNGyRETkNHQa11SySElJQVBQEJKTkxEYGGjv4lRv8rFMPAzUamaXt7+UlYv7vtyGrceT4OMhffC6oFeTULuUhYiIqreUMuQT1tiRY9Lp8oe6nMwqfXt/L3csGNMNfZqGIiPHgHu/3IbVBxOqtAxERERlxWBHji9uDzDnKmD3t1X6tjJa9rPRXXBdq3Bk5xrx4NfbsWJPbJWWgYiIqCwY7MjxHV4JnD8MLHsSOHeoSt/ay90NH9/VCUPaRyHHoGHCtzvx1aYTVVoGIiKi0mKwI8fX+3GgYV8gJx34cQyQk1Glb+/hpses4R0wvEtdtfzY1CX78MIve5BjqJp1bYmIiEqLwY4cn94NuG0u4BsKxO8FFo8HjFUbqtz0Orxxe1s8P7iF6v73zZZTuOfzLbiQll2l5SAiIioJgx05h4AIYNh80xQo+34Bfn+2yiYvtl5b9qG+jTHvni7w83TD5mNJGDp7Aw5zfVkiInIQDHbkPBr1BW6dY7q/dS6w4wu7FGNAq3D8/HBv1An2wamkdNz68UaOmCUiIofAYEfOpe0w4Po3gHq9gNa32a0YzSMCsGRCb3RrGGKa827BNsxbdwycFpKIiOyJExRb4QTFTsSQC7i527sUahqUqUv2YuG20+rxsM518OotbeDt4WbvohERkYvgBMXk+qxD3Za5wKktdimGp7seM25ri6k3tYJeB/y44wxu/ugfHIxLsUt5iIioemOwI+e2+ztgxdPAt3cCCQftUgQZVHHfVQ2x4L5uCPX3xKH4S7j5ow1YsPEEm2aJiKhKMdiRc2s1FKjTFci8CPzvNiA5xm5F6dO0FlZMvBr9mtdSTbTTlu7DAwu24/ylLLuViYiIqhcGO3Junr7AyO+B0GZASgzwv9uB9CS7FadWgBe+uLcrpg1pBU83Pf46mIDr31+PdYfO2a1MRERUfTDYkfPzDQHu/hkIiALOHQDmDwIumgYz2KtpdkzvhljySG80DfPHudQsjJq/Fa/+th9ZuQa7lYuIiFwfgx25hhp1gVGLgcDaQOIhU7jLumTXIrWMDMTSR67C3T3qqcef/XMct87eiL0xyXYtFxERuS4GO3IdtZoD968CarUEej0KePnbu0Tw8XTDq7e0xbxRXRDs64H9sSlq1Owrv+5X898RERHZEuexs8J57FxETgbg4XP5sSEHcPOAvSWkZqpA99t/sepxZJA3Xrq5NQa1jrB30YiIyIFxHjuq3qxDXcYFYG5/YNtnsLewAG98NLITvhzTFXVDfBCbnImHvt6hRs7GXMywd/GIiMgFMNiR689zF78HWPYk8PdrgANUUPdrHoaVj/fFw/0aw12vw58H4nHdzLVqSbJcg9HexSMiIifGplgrbIp1QfLxXvsmsGaG6XGn0cCNMx1iOTJxKD4VL/yyB9tOXFCPW0QE4IUbW6o58YiIiMqaTxjsrDDYubDt8021dpoRqNsduP0zoIZptKq9GY0afthxGjNWHMTF9By176omoXj2+hZoWyfI3sUjIiI7Y7ArJwY7F3dwOfDLQ0BWCuAdZJrYuF4POIqktGx89PcRfL35BHIMpj/LIe2j8NTAZqhf08/exSMiIjthsCsnBrtq4MIJ4Mf7gNQ4YNw/psmNHczppHTMXHUIi3fHqJZk6Yd3V/d6eOSapmplCyIiql5SGOzKh8GumpDpTy6eAmo2Nj2WP4HUWCAwCo5k39lkvPV7NNbmLUfm6+mGB/o0wn29G6CGr6e9i0dERFWEwa6cGOyqqV3/A5Y/DdzwNtDhLlkTDI5k49FEvLniIP49k2wJeP/XtR4e6NMQUTWspnYhIiKXxGBXTgx21ZB8/BfeBUQvMz1uM8wU8BysiVb+TFfsjcOHfx/BgdgUtU+aaId2qI1xfRuhaXiAvYtIRESVhMGunBjsqimjEdjwXt48dwbAJxgY8BLQcRSgd6ypHuXPdd3hRMxZcxSbjp237B/QMgzj+zVG5/qOFUiJiKjiGOzKicGumju9Dfj1MSBhv+lxVCfg5g+AiLZwRLtPX1QB74/9cZZ5l7vUD8bdPerj+jYR8PZws3cRiYjIBhjsyonBjtTAiq3zTBMaZ6UCY/8GaneCIzt67pJateLnnTHIzlu5ItjXA8M618GIbvXQqJa/vYtIREQVwGBXTgx2ZJEaDxz5E+h41+V9Z3YAUR0AvWPWhCWkZGLhttNYuPUUziZnWvb3alwTI7vXw8BWEfB0d6ymZSIiujIGu3JisKNiJR4GPu4JhLcC+r8INL3O4UbPmhmMGtZEJ+CbLaewOjrB0kwb6u+J2zvXwS0daquly3QOWn4iIsqPwa6cGOyoVKtWiMgOwNVPA81vcLgBFtZiLmZg0dZTqiYvITXLsr9ZuD9ubh+Fm9vXRr2avnYtIxERlYzBrpwY7KhEl84BG98Htn0O5KSb9oW1Bq5+Cmg11GGbaEWOwYi/DiTgl11nsPrgOUtfPNGhbg0M7RCFG9tFIizA267lJCKiwhjsyonBjkol7TyweTawZS6QnQp4BQFP7DGtP+sEkjNy8Me+OCzdfVZNfmzM+z+AXgd0b1gT17UKx4CW4azJIyJyEAx25cRgR2WSccEU7jx9gV6PXp4Tb9s8oPWtgH8YHF1CaiaW/ReLpf+exa5TF/M9J82117Y0hTyp1XOT5EdERFWOwa6cGOyowo7+DXx9K6B3B1rcCHQaDTTq79D98MxOnU/Hyv1x+PNAPLaduKAGYZjV9PPENS3C1NazcU2uVUtEVIUY7MqJwY4q7Pg64K9XgDPbLu+rUR/odA/Q4W4gMBLOIDk9B2sOJeDPAwlqhG1qZq7lORlM2yYqCL2a1ETvxqHo2iAEPp6O27+QiMjZMdiVE4Md2UzcXmDnAuDfRUBWsmmfzg14eBNQqzmciQy82HY8SYW89YfP4XDCpXzPe7rp0al+DRXypDavTe0grnpBRGRDDHblxGBHNpedDuxfYgp56UnAhC2X57/b+KFpwEXzGwG/mnAW8SmZatDFhiPnsfFIYr7JkM1Br03tQHSuH6y2TvWCERbI0bZEROXFYFdODHZUqTJTAO+8z1VuFvB2E9O8eFKT16A30PJmoMVNTtNcK+R/HyfOp+OfI4kq5G07kYTES9mFjqsb4oPO9YLVIIy2dYLQMjIQvp7udikzEZGzYbArJwY7qjKyDu3mOcCBpUDcf/mfq9US6Ho/0G0snI387+R0UgZ2nErC9hMXsOPkBUTHp1pWvzCTAbaNa/mrZtvWUYFoWzsIraICEeDtYa+iExE5LAa7cmKwI7tIOg4c+NUU8s5sl3hkWras79Om56UJd/PHQMO+QN1ugLsXnElqZg52n76ogt6emGTsjUnOtwqGtfo1fdE0LADNI/zRLDxAbY1q+cHLnX32iKj6SmGwKx8GO7I7CXEysjaiLVCzsWnfvl+AH+413XfzAiLbA3W6ALU7m25l1K2TrfuakJKJvWcl5KWosLcvJrlQXz0zmT+vQU1fNI8IQJNa/mhYyw8NQ/3RsKYfgnxZw0dEri+Fwa58GOzIIZ3aYpr0+NhaIC2h8PPDvgDa3Ga6n5YIGLKBgEinC3vnL2UhOi4Vh+JTER1/Sd3KZj3VSkEhfp5oGOpn2aTGr06wL+oG+6jndE52DYiIisJgV04MduTQ5E816ZipuTZmu+k2bg/wyDYgpKHpmPXvmubR864BhLUCwlvl3bYGwlo6zbJnZvK/p7iUTBySoBeXimOJl3DsXBpOnE9DfErRzblmvp5uqBPsg7oS9EIk8Pmgdg0fRAR5I6qGD0L9vbiaBhE5BQa7cmKwI6eTk2nqc2eumfrjBVN/PM1Y9PEPbzYFPHF6G3ApDghpBAQ3NC2N5kQuZeXiRGIajlttp5LSceZC+hVDn5BQFx7ghci8sBcZ6I3wQG+EBXqhlr9X3q03An3cWfNHRHbFYFdODHbkMmEv8RCQsB+I35d3u9/UjDs5FnDPWw5s8QRg9/8uvy4gCghuAATVMW19JgFeAabnDDmAm/P0Z8vMMSDmYgbOXMjA6aR0nL6QjjNJGYhNli1TzcVntWJaiTzd9ZagV9PPC6H+nqqZV7aa/p5qn/l+sK8nJ2cmIrvmE04kReRqPLyByHamreA8euZQJ4LrA1GdgKSjQGYykHrWtJn1e+7y/V8nmkbuSt+9gHDA37yFmW5b33p5tK78W9HONVwSrmQ6FdmKkmswqvn2ziZnIC45U4W92IsZarRuQmomzqnbLNW/LzvXqEKibKV7b70KeEE+Huq2hq9H3mbaF+jtgQBvdwSq++ZbD1UzyNG/RFRRDh3sZs+ejbfffhtxcXFo3749PvzwQ3Tr1q3Y43/44QdMmTIFJ06cQNOmTfHmm2/ihhtuqNIyEzks8+TIZn2fMW3m0bjnjwIXTwIpMabH1tOqJJ82TaYsW2J04XO3zhu8IX4eCxxeCfjWvLz5hAC+IYBPMNDrscsBU6Z6kcmafWqY+v+5e1dJKHR306vmV9muVPNnDnnnUjNVGExKM22Jl7Is98+nZeNCWjZyjRoyc4ymoFjMKN+SyKod/t7u8Pdyh5+XOwK83PM99vdyUxM7+xW89XRX6/X65m0+Hm7qsdzKz0pE1YfDBrtFixZh0qRJmDNnDrp3745Zs2Zh0KBBiI6ORlhYWKHjN27ciBEjRmDGjBm46aab8O233+KWW27Bzp070aZNG7v8DEROQ0KXbHW7Fv38iIVAcgyQGgtcSgAuxZv658n97LT8NYEyMldqAGWTwR4F9X788v2/pwN7f7r8WO8OePqbmoDl9oE/Aa+8Wrdd3wCxuwEPH8DDz9Qn0Pp+s+svh9GUWCAn3fTYXY7xNoVGvVuZa/5k4IVsVyK9WlKzcpGcnoML6dm4mHebnJGDC2mm+1IDmJKZg5SMHKTIfXWbYxn5m20wWsKirXi46SxBT34eb3e51cPLw/xYb7qVfe5u8HLXq+ZndV/tM92XfWpzk1sdPN1M++T8l/frVZBU+9StPNbBQ6+HngNViKqEw/axkzDXtWtXfPTRR+qx0WhE3bp18eijj+K556yaiPIMHz4caWlp+O233yz7evTogQ4dOqhwWBrsY0dkA1Lbl3bOdJt+3rRlyL5EICcDuGnm5WMXPwxELzeFwKIGfEw5D7jl/fvzhzHAvp+Lf99nT5pq/sTSR4GdXxU+RoKjzAX46I7LS7etn2kKl26epiAofQn1HqbHcv+Gt4GACNOxB5cDx9eazqPO5ZH/foe7AL9Q07Gx/5pGLZuf1+lNwVLddwPqdTfVYEoXxotnkJFwDBm5QHqOZtpyNaRna0jLMSBOH4mLBk9kZBuQm3ERuvQLSM8xquPSso1IyzFtcvy5HC8k5+hVi7gHcuGJHGjQwQiduhVG6GUabBjUbdXU6LnrdZaQJ7cqAOp18JAwKLduejWgRfbLY7kvAdFNf/mx9a3e8vjy83qd6T3kVioq5Tm3vPtyvOm++XmdZZ9kTst9OVbtu3weGTxjfk5nec50nFQwm54z7QMuP2d+Xm9+bd5zss98HnmJ+bjL+03HWvaZ71sfK48sz5teZ7q9fLwqTYHX5zuOg4KchtP3scvOzsaOHTvw/PPPW/bp9XoMGDAAmzZtKvI1sl9q+KxJDd/ixYuLfZ+srCy1WV84IrJR7V9p3PKx6VZSiCyzJlv2JSDrkunWHOpEq6GmSZuz0021ceZNPc4APKxq1SSUeQWa9htzLu835po264Eg0swcv7f4Mg589fL9UxuBLSX8Q7HpwMvBTkLg2jeKP/aBv4E6nU3F3fcT/FdNRdE9AgGM/hVoeLXp/tZ5wPKnij/v3d9DazoQWblGGHf+D74rHiv20M2d38XRsOtU83Hts79j0P7n88KfhEBZA0UHLe/Lf37IE1jjda2qVWyTsQ3Ppsy4fIz5VpPQqMOM3BH43tBfva6zLhqfer6njjFFC0Az6CRVqn0f5t6K/xmuU/tb6U7gE893LOUzB1Hz7fzcwZhvGKzuN9DF4iuPN4o8Tiw09Mdsw83qfgTO41vP1wr9/Objlxp64X3D7ep+DaTiR8+Xi7xecvxKY2e8nft/6rE3srDU88VC5zNbb2yLV3PvsTz+3fNZ+bGLtNXYAlNzx1geL/Z8EV6w+uxa2WNshGdyH7r8s3pORxDSijz2kFYHE3MesTz+wuNNhOsuFnnsaYRhQu4kSyic7TYTdXVFzJ0J4ByCMV4zfUdLQHwDH6AJTpt/xfmkwh9j9S9ZnpqqzUFL7ajl+csv0SELHrjfY4Zlz9OGz9DeeLDggRb3er5ruf9I7gJ0NRZYotHKOK8ZyNaZavUfyPkWvQyy0k/RnvCZjks601/kXdk/or9hQ7HHPu89Bdm+4Vj6yFVwBA4Z7BITE2EwGBAeHp5vvzw+eDDvF1yA9MMr6njZXxxptn355aL/gImoCsk3ifQBLNgP0FrrW0xbadz4rmkTRgOQm2kaLSy3hixLTZnS42GgxU2miZ1l9G++22xTH0Gzhv1MoVGel/NKaFT35TbXNH+gmYTQpoMuh0k5XpPXyJZ7uYlZyOtqNsk7xph/k33SjGwmtX7S/FzwODm36WKqL1o1Otej5Nq4Ho1qokfr+qYHe8OA/eaYZiXv4UNXNcBDHXuYHkSnAN8VMZgk74v3zaHN8XrnwcgxaDAc84H/wuL/0TyuZzhuatUDuQYNPuc8EbUyqdhjh7bwR91GrWAwaghI9UC9reeKPbZPhBvSouqrfo9Bme5odKj474L2QTnoH1ILBg0IyHVDk7NWg4gKOOnTAq18A2HUNHga3dFM+qQW44x7PUT4eqtjZWuRe7rYY+N1ofDW6U2jtTWgue4MfHRFN8knF/gnQFPdGdTUpRZ5bJYx/9d8U30M6ugSizzW3ZirrpdZA7ezaKYr+ucL0NKQln05ptb1PIsW+pNFHnteC1BdE8wi1bEnijw2XfNS/VrNQj1i0dztuOlBEe2L1n1Za3jEopn52CLEXkxHBkwtA/4e8SUfm3QJSXm12d7u8WjqXvyxcRdSkZqRN4OAA3DIptizZ8+idu3aqt9cz549LfufeeYZrF27Flu2bCn0Gk9PTyxYsED1szP7+OOPVXCLj48vdY2dNPeyKZaInJb1qGQJhRI8zeFPvhnlefN9CYjm/pESfKVJ3HyMujVevi8DYMxhVPpVploFJcvxeV8n/rUuh2epfb146vK3csFjpZlbRlebz5t4uPCx6j5MzeeBUXnHppum87lciPzXQc5bo97ln+3sruKvmby/eQm/3GzgzLaiLqzpxi8MqNXs8vU9ubHo9xe+oaZJws1k9ZjiyPWyHskuSwtad0/QNPUOkrs07yBokR1MNaSaBt3JDer3LEeYL6v5ymke/jBEdrY853Zmk7oe6jmrY9Wthy+yI039bOV495ht0EmtuDrWaDlePXbzQlbtnuo44Rm7A7psU7hUH7HLxVY17xm1e1keeyXshj5LPmvWZTDd0eCG9DpXWfZ7Jf4Ht8wLRV5eOeZSnastr/c+vw/uGYmFjjU/TI3qrbpCSOzxTjoIj3RTNigqBKVG9IQm/4iTYHfxEDzT8od962uRGt4dOk8fdKpn9Q9GG3P6ptjQ0FC4ubkVCmTyOCIir69LAbK/LMcLLy8vtRERuQzrflOqT18pB4zIABPZSsPT73IQuhIJg9bh5krnjepQymN9ix/sU5D8XPUvVxKUSIJug96lO1aubcM+KLVGfUt/rLnp3Yr8Zov8bTYpfGyxmpehDEFlOG/NMhwbXoZrFlWG5s06pfy9iXql/DyI+t3hTBxyHLzUvnXu3Bl//fWXZZ8MnpDH1jV41mS/9fFi1apVxR5PRERE5GocssZOyECI0aNHo0uXLmruOpnuREa9jhlj6lw6atQo1Vwr/eTExIkT0bdvX7z77ru48cYbsXDhQmzfvh1z5861809CREREVM2DnUxfcu7cOUydOlUNgJBpS37//XfLAIlTp06pkbJmvXr1UnPXvfjii5g8ebKaoFhGxHIOOyIiIqouHHLwhL1wHjsiIiJy5nzikH3siIiIiKjsGOyIiIiIXASDHREREZGLYLAjIiIichEMdkREREQugsGOiIiIyEUw2BERERG5CAY7IiIiIhfBYEdERETkIhjsiIiIiFwEgx0RERGRi3C3dwEciXnZXFmTjYiIiMgRmHOJOaeUhMHOSmpqqrqtW7euvYtCREREVCinBAUFoSQ6rTTxr5owGo04e/YsAgICoNPpyp2qJRiePn0agYGBNi+jK+O1qxhev4rh9asYXr/y47WrmOpw/TRNU6EuKioKen3JvehYY2dFLladOnVsci75cLnqB6yy8dpVDK9fxfD6VQyvX/nx2lWMq1+/oCvU1Jlx8AQRERGRi2CwIyIiInIRDHY25uXlhWnTpqlbKhteu4rh9asYXr+K4fUrP167iuH1y4+DJ4iIiIhcBGvsiIiIiFwEgx0RERGRi2CwIyIiInIRDHY2NHv2bDRo0ADe3t7o3r07tm7dau8iOaR169ZhyJAhaqJFmQh68eLF+Z6Xbp9Tp05FZGQkfHx8MGDAABw+fNhu5XUkM2bMQNeuXdUk2mFhYbjlllsQHR2d75jMzExMmDABNWvWhL+/P26//XbEx8fbrcyO5JNPPkG7du0s81317NkTK1assDzPa1c2b7zxhvobfvzxxy37eA2L99JLL6nrZb21aNHC8jyvXcliYmJw9913q+sj3w1t27bF9u3bLc/zu8OEwc5GFi1ahEmTJqmROTt37kT79u0xaNAgJCQk2LtoDictLU1dHwnCRXnrrbfwwQcfYM6cOdiyZQv8/PzUtZT/6VV3a9euVf/j37x5M1atWoWcnBwMHDhQXVOzJ554Ar/++it++OEHdbyspnLbbbfZtdyOQiYglzCyY8cO9YVwzTXXYOjQodi3b596nteu9LZt24ZPP/1UBWVrvIYla926NWJjYy3bP//8Y3mO1654Fy5cQO/eveHh4aH+MbZ//368++67CA4OthzD7448MiqWKq5bt27ahAkTLI8NBoMWFRWlzZgxw67lcnTyEfzll18sj41GoxYREaG9/fbbln0XL17UvLy8tO+++85OpXRcCQkJ6hquXbvWcq08PDy0H374wXLMgQMH1DGbNm2yY0kdV3BwsPbZZ5/x2pVBamqq1rRpU23VqlVa3759tYkTJ6r9vIYlmzZtmta+ffsin+O1K9mzzz6rXXXVVcU+z++Oy1hjZwPZ2dmqBkCqfa2XJ5PHmzZtsmvZnM3x48cRFxeX71rKMirStM1rWVhycrK6DQkJUbfyOZRaPOvrJ0099erV4/UrwGAwYOHChaq2U5pkee1KT2qNb7zxxnzXSvAaXpk0DUo3lEaNGuGuu+7CqVOn1H5eu5ItXboUXbp0wR133KG6oXTs2BHz5s2zPM/vjssY7GwgMTFRfUmEh4fn2y+P5YNGpWe+XryWV2Y0GlXfJmmeaNOmjdon18jT0xM1atTIdyyv32V79uxR/ZdkMtNx48bhl19+QatWrXjtSknCsHQ3kf6eBfEalkxCxpdffonff/9d9feUMNKnTx+1uDuvXcmOHTumrlnTpk3xxx9/YPz48XjsscewYMEC9Ty/Oy5zt7pPRE5Wa7J37958fXToypo3b47du3er2s4ff/wRo0ePVv2Z6MpOnz6NiRMnqv6dMkiMymbw4MGW+9I3UYJe/fr18f3336vO/lTyP2Slxu71119Xj6XGTv7/J/3p5G+YLmONnQ2EhobCzc2t0OgleRwREWG3cjkj8/XitSzZI488gt9++w2rV69WAwLM5BpJ14CLFy/mO57X7zKpFWnSpAk6d+6sap1kIM/777/Pa1cK0lwoA8I6deoEd3d3tUkolg7rcl9qR3gNS09q55o1a4YjR47w83cFMtJVatattWzZ0tKUze+OyxjsbPRFIV8Sf/31V75/Xchj6btDpdewYUP1R2h9LVNSUtQIJ15L03B+CXXSfPj333+r62VNPocyasz6+sl0KPI/P16/osnfalZWFq9dKVx77bWqKVtqPM2b1KJIXzHzfV7D0rt06RKOHj2qQgs/fyWTLicFp3Y6dOiQqvEU/O6wYjWQgipg4cKFavTNl19+qe3fv1978MEHtRo1amhxcXH2LppDjqjbtWuX2uQjOHPmTHX/5MmT6vk33nhDXbslS5Zo//33nzZ06FCtYcOGWkZGhlbdjR8/XgsKCtLWrFmjxcbGWrb09HTLMePGjdPq1aun/f3339r27du1nj17qo007bnnnlMjiI8fP64+W/JYp9NpK1euVM/z2pWd9ahYwWtYvCeffFL97crnb8OGDdqAAQO00NBQNbpd8NoVb+vWrZq7u7v22muvaYcPH9a++eYbzdfXV/vf//5nOYbfHSYMdjb04Ycfqj9KT09PNf3J5s2b7V0kh7R69WoV6Apuo0ePtgxbnzJlihYeHq7C8rXXXqtFR0fbu9gOoajrJtsXX3xhOUb+J/bwww+raTzkf3y33nqrCn+kaffdd59Wv3599Tdaq1Yt9dkyhzrBa1fxYMdrWLzhw4drkZGR6vNXu3Zt9fjIkSOW53ntSvbrr79qbdq0Ud8LLVq00ObOnZvveX53mOjkP9Y1eERERETknNjHjoiIiMhFMNgRERERuQgGOyIiIiIXwWBHRERE5CIY7IiIiIhcBIMdERERkYtgsCMiIiJyEQx2RERERC6CwY6oGurXrx8ef/xxy+MGDRpg1qxZcGQnTpyATqdTa5JWpnvvvRe33HJLhc8jZV28eDEqy5o1a9R7FFw0viBZO1MWSzcYDHBUtvj8Pffcc3j00UdtViYiZ8VgR0TYtm0bHnzwQTiyunXrIjY2Fm3atKnU93n//ffx5ZdfwtH16tVLXY+goKASj3vmmWfw4osvws3NTT2W14wcORLNmjWDXq/PF/AdjSzs/ueff6oQO3ToUERGRsLPzw8dOnTAN998k+/Yp556CgsWLMCxY8fsVl4iR8BgR0SoVasWfH194cgkmERERMDd3b1S30eCUo0aNeDoPD091fWQWrvi/PPPPzh69Chuv/12y76srCz1+5aw1759eziq//77DxcuXEDfvn2xceNGtGvXDj/99JPaP2bMGIwaNQq//fab5fjQ0FAMGjQIn3zyiV3LTWRvDHZELi4tLU19Cfr7+6saj3ffffeKTWESFj799FPcdNNNKvBJU96mTZtw5MgR1YwrtSZSYyShwdqSJUvQqVMneHt7o1GjRnj55ZeRm5ub77yfffYZbr31VnXepk2bYunSpZbn5Yv8rrvuUsHDx8dHPf/FF18U2xS7du1adOvWDV5eXupnk+Y46/eTsj722GOq1iokJEQFoZdeeqlMTbGlOcfhw4dx9dVXq5+7VatWWLVqVaHznj59GnfeeacKjXIeqYGSn0kcPHhQXY9vv/3Wcvz333+vrsH+/fvL3RS7cOFCXHfddapc1r9rqZWUz8SVavsqQ0JCAoYMGaJ+NqmRK1jzZv1Zuv766+Hh4YHJkydj+vTp6jPXuHFjTJw4UT33888/53uNnFd+ZqLqjMGOyMU9/fTTKgDJF+XKlStVINi5c+cVXydfpPLlL0GqRYsWqvnuoYcewvPPP4/t27dD0zQ88sgjluPXr1+vjpcvXQkjEgylSfO1117Ld14JexJwpOblhhtuUEEuKSlJPTdlyhT12hUrVuDAgQOq9kVqYooSExOjXt+1a1f8+++/6tjPP/8cr776ar7jpHlOguiWLVvw1ltv4ZVXXikyeJWkpHMYjUbcdtttqgZNnp8zZw6effbZfK/PyclRtUkBAQHqOm3YsEEFbQkn2dnZ6vq+8847ePjhh3Hq1CmcOXMG48aNw5tvvqmCYnnJe3Xp0gWORIKzhNzVq1fjxx9/xMcff6zCXkES+CX8Fic5OVkFZGsS8uXamQMzUbWkEZHLSk1N1Tw9PbXvv//esu/8+fOaj4+PNnHiRMu++vXra++9957lsfyv4cUXX7Q83rRpk9r3+eefW/Z99913mre3t+Xxtddeq73++uv53v/rr7/WIiMjiz3vpUuX1L4VK1aox0OGDNHGjBlT5M9y/PhxdeyuXbvU48mTJ2vNmzfXjEaj5ZjZs2dr/v7+msFgUI/79u2rXXXVVfnO07VrV+3ZZ58t9pqNHj1aGzp0qOXxlc7xxx9/aO7u7lpMTIzlefl5pKy//PKL5ToULGtWVpb6PcjrzW688UatT58+6loOHDgw3/EFrV69Wr3HhQsXij0mKChI++qrr4p9Xn42689BZYuOjlZl3rp1q2XfgQMH1D7rz9+ZM2fU57a4n23RokXq+b179+bbn5ycrM61Zs2aSvwpiBxb5XZWISK7kqZSqRHq3r27ZZ/UcjRv3vyKr5U+TWbh4eHqtm3btvn2ZWZmIiUlBYGBgarWTGqirGvoZCSmHJOenm7pw2d9XqkFk9eaa2zGjx+v+oNJjeLAgQNVk6g0vxVFavR69uyZr49Z7969cenSJVVrU69evULvJ6TJtqgaotJei4LnkHLIwI6oqCjL81Iua3JtpBlbauysybWxbs6eP3++ZVDDvn37Suw/VxoZGRn5mmHtTa6V9JHs3LmzZZ/UVhbs0yi1dVdddVWRfR2lpk/62M2bNw+tW7fO95w07wr5vBFVVwx2RFQk6dtkZg4YRe2TpkghgUqaWaVZsiDrcGF9DvN5zOcYPHgwTp48ieXLl6umzmuvvRYTJkxQzZS2+DkKvl9VnUOujYSZovqTSX9C6wAofSIl2MnoVQmQFSHN2NJv0dlIsLv55psL7ZcuBdKP7r333lPN/gWZm/StrylRdcM+dkQuTDqaSyiRvl9m8kV/6NAhm7+XDJqIjo5GkyZNCm0SVEpLvpRHjx6N//3vf2pAx9y5c4s8zjygw9TCayI1hlIrVqdOHVQVKYf0GZMgZrZ58+ZC10YGWISFhRW6NuYBDBJKpP/ZCy+8oG6l76HUuFVEx44dix18YQ9SOyeDW3bs2GHZJ58Z6wEgEoKlVq5g/zrpG3rjjTeqfofFTc2zd+9e9XkvWJNHVJ0w2BG5MOmgf//996sBFH///bf64pPQUJagVVpTp07FV199pWrtpBlRmt1khKJMq1GWc8ggD2m2lHPIdBYSnIoiAw0kUMmktDKqVF43bdo0TJo0qVJ+vuIMGDBANZ9KGJUaNxmwIOHMmoQ0qT2TsCLPHz9+XAUVGW0rzcZCBktIk65cr5kzZ6pmbJmbrSJkwIZMeVKQDIiRTULUuXPn1P0rBUCpPZWBM2Zbt25VQU1uzeR5Oa440gVABozIIBz5x4YEvAceeMDShCp+//13dT1l9K6ZBD0JdXK9pKk+Li5ObeYaOjO5tn369Ml3PqLqhsGOyMW9/fbb6stOmrAkhEjfJes+TrYiIUKCmIy8lZGqPXr0UE1m9evXL/U5ZGSphAPp0ybTh8jcdcVNX1G7dm3VZCvBQuZjk2AkIbYsQdIWJET+8ssvqnZNRmVKUCk4Elj6F65bt071+5OmagmrUlbpYyd9DCUQy8/y9ddfqz5o0vdQaiylH5mMEC4vCZQSkKVWrGBNnmwSrGSKFbkvI4xLIn0BrWslpR+bnNe6P5s8X3AKnIJk+hrpjyjz08m1kNo3qck0k4BesBlWRiXL+8yYMUM1T5u3gs3+8lkZO3bsFa4KkWvTyQgKexeCiIgqh9TWygAXmX7G0UkzrQzKkTArIbks5DVPPvmkmkansiexJnJkrLEjInJh0iwstaZlHTBiD9K0+sQTT6ga37KSQSdSG8hQR9Uda+yIiIiIXARr7IiIiIhcBIMdERERkYtgsCMiIiJyEQx2RERERC6CwY6IiIjIRTDYEREREbkIBjsiIiIiF8FgR0REROQiGOyIiIiIXASDHRERERFcw/8Ds7Bm5oesHD0AAAAASUVORK5CYII=",
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"import torch\n",
"\n",
"d = 128\n",
"i = torch.arange(1, d // 2 + 1)\n",
"\n",
"base_llama2 = 10_000\n",
"base_llama3 = 500_000\n",
"\n",
"theta_llama2 = base_llama2 ** (-2 * (i - 1) / d)\n",
"theta_llama3 = base_llama3 ** (-2 * (i - 1) / d)\n",
"\n",
"plt.plot(i, theta_llama2, label=\"base = 10,000 (Llama 2)\")\n",
"plt.plot(i, theta_llama3, label=\"base = 500,000 (Llama 3)\", ls=\"--\")\n",
"plt.xlabel(\"dimension index i (1 … d/2)\")\n",
"plt.ylabel(r\"$\\theta_i = \\mathrm{base}^{-2(i-1)/d}$\")\n",
"plt.legend()\n",
"plt.tight_layout()"
]
},
{
"cell_type": "markdown",
"id": "17f1d883-72a8-4e01-9aeb-a5a136630a1e",
"metadata": {},
"source": [
"- This changed decay rate is to better support longer context lengths without the rotations becoming too much or too strong at higher dimensions (larger positions)\n",
"- In addition, we introduce a `freq_config` section in the code below that adjusts the frequency; however, we won't be needing it in Llama 3 (only Llama 3.1 and Llama 3.2), so we will revisit this `freq_config` later (it's set to `None` and ignored by default)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "6Upl109OOAcu",
"metadata": {
"id": "6Upl109OOAcu"
},
"outputs": [],
"source": [
"import torch\n",
"\n",
"def precompute_rope_params(head_dim, theta_base=10_000, context_length=4096, freq_config=None):\n",
" assert head_dim % 2 == 0, \"Embedding dimension must be even\"\n",
"\n",
" # Compute the inverse frequencies\n",
" inv_freq = 1.0 / (theta_base ** (torch.arange(0, head_dim, 2)[: (head_dim // 2)].float() / head_dim))\n",
"\n",
" ################################ NEW ###############################################\n",
" # Frequency adjustments\n",
" if freq_config is not None:\n",
" low_freq_wavelen = freq_config[\"original_context_length\"] / freq_config[\"low_freq_factor\"]\n",
" high_freq_wavelen = freq_config[\"original_context_length\"] / freq_config[\"high_freq_factor\"]\n",
"\n",
" wavelen = 2 * torch.pi / inv_freq\n",
"\n",
" inv_freq_llama = torch.where(\n",
" wavelen > low_freq_wavelen, inv_freq / freq_config[\"factor\"], inv_freq\n",
" )\n",
"\n",
" smooth_factor = (freq_config[\"original_context_length\"] / wavelen - freq_config[\"low_freq_factor\"]) / (\n",
" freq_config[\"high_freq_factor\"] - freq_config[\"low_freq_factor\"]\n",
" )\n",
"\n",
" smoothed_inv_freq = (\n",
" (1 - smooth_factor) * (inv_freq / freq_config[\"factor\"]) + smooth_factor * inv_freq\n",
" )\n",
"\n",
" is_medium_freq = (wavelen <= low_freq_wavelen) & (wavelen >= high_freq_wavelen)\n",
" inv_freq_llama = torch.where(is_medium_freq, smoothed_inv_freq, inv_freq_llama)\n",
" inv_freq = inv_freq_llama\n",
" ####################################################################################\n",
"\n",
"\n",
" # Generate position indices\n",
" positions = torch.arange(context_length)\n",
"\n",
" # Compute the angles\n",
" angles = positions.unsqueeze(1) * inv_freq.unsqueeze(0) # Shape: (context_length, head_dim // 2)\n",
"\n",
" # Expand angles to match the head_dim\n",
" angles = torch.cat([angles, angles], dim=1) # Shape: (context_length, head_dim)\n",
"\n",
" # Precompute sine and cosine\n",
" cos = torch.cos(angles)\n",
" sin = torch.sin(angles)\n",
"\n",
" return cos, sin"
]
},
{
"cell_type": "markdown",
"id": "jJBvO0YMJBXR",
"metadata": {
"id": "jJBvO0YMJBXR"
},
"source": [
"- To summarize, what's new so far for Llama 3 compared to Llama 2 are the context length and theta base parameter:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "56c37216-e022-4603-be16-f9d3eaeaf4a1",
"metadata": {
"id": "56c37216-e022-4603-be16-f9d3eaeaf4a1"
},
"outputs": [],
"source": [
"# Instantiate RoPE parameters\n",
"\n",
"llama_2_context_len = 4096\n",
"llama_3_context_len = 8192\n",
"\n",
"llama_2_theta_base = 10_000\n",
"llama_3_theta_base = 500_000"
]
},
{
"cell_type": "markdown",
"id": "_V8v6i7MJItU",
"metadata": {
"id": "_V8v6i7MJItU"
},
"source": [
"- The usage remains the same as before in Llama 2:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "dae70c8a-eb18-40f9-a2e5-a6af2a57628b",
"metadata": {
"id": "dae70c8a-eb18-40f9-a2e5-a6af2a57628b"
},
"outputs": [],
"source": [
"# Settings\n",
"batch_size = 2\n",
"num_heads = 4\n",
"head_dim = 16\n",
"\n",
"# Instantiate RoPE parameters\n",
"cos, sin = precompute_rope_params(\n",
" head_dim=head_dim,\n",
" theta_base=llama_3_theta_base,\n",
" context_length=llama_3_context_len\n",
")\n",
"\n",
"# Dummy query and key tensors\n",
"torch.manual_seed(123)\n",
"queries = torch.randn(batch_size, num_heads, llama_3_context_len, head_dim)\n",
"keys = torch.randn(batch_size, num_heads, llama_3_context_len, head_dim)\n",
"\n",
"# Apply rotary position embeddings\n",
"queries_rot = compute_rope(queries, cos, sin)\n",
"keys_rot = compute_rope(keys, cos, sin)"
]
},
{
"cell_type": "markdown",
"id": "cd19b75c-cf25-47b8-a010-6733fc0e9a8a",
"metadata": {
"id": "cd19b75c-cf25-47b8-a010-6733fc0e9a8a"
},
"source": [
"&nbsp;\n",
"## 1.3 Grouped-query attention"
]
},
{
"cell_type": "markdown",
"id": "111c7d3f-fded-49e8-a617-9fe67b81dddc",
"metadata": {
"id": "111c7d3f-fded-49e8-a617-9fe67b81dddc"
},
"source": [
"- In this section, we replace multi-head attention (MHA) with an alternative mechanism called grouped-query attention (GQA)\n",
"- In short, one can think of GQA as a more compute- and parameter-efficient version of MHA\n",
"- In GQA, we reduce the number of key and value projections by sharing them among multiple attention heads\n",
"- Each attention head still has its unique query, but these queries attend to the same group of keys and values\n",
"- Below is an illustration of GQA with 2 key-value-groups (kv-groups):\n",
"\n",
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/grouped-query-attention.webp\" width=\"500px\">\n"
]
},
{
"cell_type": "markdown",
"id": "perAYa2R_KW2",
"metadata": {
"id": "perAYa2R_KW2"
},
"source": [
"- The main idea behind GQA is to reduce the number of unique query groups that attend to the key-value pairs, reducing the size of some of the matrix multiplications and the number of parameters in MHA without significantly reducing modeling performance\n",
"- The GQA code is very similar to MHA (I highlighted the changes below via the \"NEW\" sections)\n",
"- In short, the main change in GQA is that each query group needs to be repeated to match the number of heads it is associated with, as implemented below"
]
},
{
"cell_type": "markdown",
"id": "842aa71a-4659-424e-8830-392bd6ae86af",
"metadata": {},
"source": [
"- **We also redesign the attention class a bit so it receives the mask through its forward method instead of storing and accessing it as `self.mask`. This lets us build the mask on the fly to reduce memory usage. To foreshadow why: Llama 3.1 can handle sequences of up to 128 k tokens, and precomputing a 128 k × 128 k causal mask would be extremely memory‑intensive, so we avoid it unless absolutely necessary.**"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "9b12e674-ef08-4dd7-8843-615b65b39c91",
"metadata": {
"id": "9b12e674-ef08-4dd7-8843-615b65b39c91"
},
"outputs": [],
"source": [
"import torch.nn as nn\n",
"\n",
"\n",
"class GroupedQueryAttention(nn.Module):\n",
" def __init__(\n",
" self, d_in, d_out, num_heads,\n",
" num_kv_groups, # NEW\n",
" dtype=None\n",
" ):\n",
" super().__init__()\n",
" assert d_out % num_heads == 0, \"d_out must be divisible by num_heads\"\n",
" assert num_heads % num_kv_groups == 0, \"num_heads must be divisible by num_kv_groups\" # NEW\n",
"\n",
" self.d_out = d_out\n",
" self.num_heads = num_heads\n",
" self.head_dim = d_out // num_heads\n",
"\n",
" ############################# NEW #############################\n",
" # self.W_key = nn.Linear(d_in, d_out, bias=False, dtype=dtype)\n",
" # self.W_value = nn.Linear(d_in, d_out, bias=False, dtype=dtype)\n",
" self.W_key = nn.Linear(d_in, num_kv_groups * self.head_dim, bias=False, dtype=dtype)\n",
" self.W_value = nn.Linear(d_in, num_kv_groups * self.head_dim, bias=False, dtype=dtype)\n",
" self.num_kv_groups = num_kv_groups\n",
" self.group_size = num_heads // num_kv_groups\n",
" ################################################################\n",
"\n",
" self.W_query = nn.Linear(d_in, d_out, bias=False, dtype=dtype)\n",
" self.out_proj = nn.Linear(d_out, d_out, bias=False, dtype=dtype)\n",
"\n",
"\n",
" def forward(self, x, mask=None, cos=None, sin=None):\n",
" ##################### NEW #####################\n",
" # The forward method now accepts `mask` instead of accessing it via self.mask.\n",
" # Also, we now have cos and sin as input for RoPE\n",
" ################################################ \n",
" b, num_tokens, d_in = x.shape\n",
"\n",
" queries = self.W_query(x) # Shape: (b, num_tokens, d_out)\n",
" keys = self.W_key(x) # Shape: (b, num_tokens, num_kv_groups * head_dim)\n",
" values = self.W_value(x) # Shape: (b, num_tokens, num_kv_groups * head_dim)\n",
"\n",
" # Reshape queries, keys, and values\n",
" queries = queries.view(b, num_tokens, self.num_heads, self.head_dim)\n",
"\n",
" ##################### NEW #####################\n",
" # keys = keys.view(b, num_tokens, self.num_heads, self.head_dim)\n",
" # values = values.view(b, num_tokens, self.num_heads, self.head_dim)\n",
" keys = keys.view(b, num_tokens, self.num_kv_groups, self.head_dim)\n",
" values = values.view(b, num_tokens, self.num_kv_groups, self.head_dim)\n",
" ################################################\n",
"\n",
" # Transpose keys, values, and queries\n",
" keys = keys.transpose(1, 2) # Shape: (b, num_kv_groups, num_tokens, head_dim)\n",
" values = values.transpose(1, 2) # Shape: (b, num_kv_groups, num_tokens, head_dim)\n",
" queries = queries.transpose(1, 2) # Shape: (b, num_heads, num_tokens, head_dim)\n",
"\n",
" ##################### NEW #####################\n",
" # Apply RoPE\n",
" if cos is not None:\n",
" keys = compute_rope(keys, cos, sin)\n",
" queries = compute_rope(queries, cos, sin)\n",
" ################################################\n",
"\n",
" ##################### NEW #####################\n",
" # Expand keys and values to match the number of heads\n",
" # Shape: (b, num_heads, num_tokens, head_dim)\n",
"\n",
" keys = keys.repeat_interleave(self.group_size, dim=1) # Shape: (b, num_heads, num_tokens, head_dim)\n",
" values = values.repeat_interleave(self.group_size, dim=1) # Shape: (b, num_heads, num_tokens, head_dim)\n",
" # For example, before repeat_interleave along dim=1 (query groups):\n",
" # [K1, K2]\n",
" # After repeat_interleave (each query group is repeated group_size times):\n",
" # [K1, K1, K2, K2]\n",
" # If we used regular repeat instead of repeat_interleave, we'd get:\n",
" # [K1, K2, K1, K2]\n",
" ################################################\n",
"\n",
" # Compute scaled dot-product attention (aka self-attention) with a causal mask\n",
" # Shape: (b, num_heads, num_tokens, num_tokens)\n",
" attn_scores = queries @ keys.transpose(2, 3) # Dot product for each head\n",
"\n",
" ##################### NEW #####################\n",
" # Create mask on the fly\n",
" if mask is None:\n",
" mask = torch.triu(torch.ones(num_tokens, num_tokens, device=x.device, dtype=torch.bool), diagonal=1)\n",
" ################################################\n",
" \n",
" # Use the mask to fill attention scores\n",
" attn_scores.masked_fill_(mask, -torch.inf)\n",
"\n",
" attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)\n",
" assert keys.shape[-1] == self.head_dim\n",
"\n",
" # Shape: (b, num_tokens, num_heads, head_dim)\n",
" context_vec = (attn_weights @ values).transpose(1, 2)\n",
"\n",
" # Combine heads, where self.d_out = self.num_heads * self.head_dim\n",
" context_vec = context_vec.reshape(b, num_tokens, self.d_out)\n",
" context_vec = self.out_proj(context_vec) # optional projection\n",
"\n",
" return context_vec"
]
},
{
"cell_type": "markdown",
"id": "roAXSwJs9hR8",
"metadata": {
"id": "roAXSwJs9hR8"
},
"source": [
"- To illustrate the parameter savings in GQA over MHA, consider the following multi-head attention example from the GPT and Llama 2 code:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b4b8f085-349e-4674-a3f0-78fde0664fac",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "b4b8f085-349e-4674-a3f0-78fde0664fac",
"outputId": "9da09d72-43b1-45af-d46f-6928ea4af33a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"W_key: torch.Size([4096, 4096])\n",
"W_value: torch.Size([4096, 4096])\n",
"W_query: torch.Size([4096, 4096])\n"
]
}
],
"source": [
"# Settings\n",
"batch_size = 1\n",
"context_len = 3000\n",
"max_context_len = 8192\n",
"embed_dim = 4096\n",
"num_heads = 32\n",
"\n",
"\n",
"example_batch = torch.randn((batch_size, context_len, embed_dim))\n",
"\n",
"mha = MultiHeadAttention(\n",
" d_in=embed_dim,\n",
" d_out=embed_dim,\n",
" context_length=max_context_len,\n",
" num_heads=num_heads\n",
")\n",
"\n",
"mha(example_batch)\n",
"\n",
"print(\"W_key:\", mha.W_key.weight.shape)\n",
"print(\"W_value:\", mha.W_value.weight.shape)\n",
"print(\"W_query:\", mha.W_query.weight.shape)"
]
},
{
"cell_type": "markdown",
"id": "IMQtFkcQ9sXC",
"metadata": {
"id": "IMQtFkcQ9sXC"
},
"source": [
"- Now, if we use grouped-query attention instead, with 8 kv-groups (that's how many Llama 3 8B uses), we can see that the number of rows of the key and value matrices are reduced by a factor of 4 (because 32 attention heads divided by 8 kv-groups is 4)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "15e65d3c-7b42-4ed3-bfee-bb09578657bb",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "15e65d3c-7b42-4ed3-bfee-bb09578657bb",
"outputId": "69709a78-2aaa-4597-8142-2f44eb59753f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"W_key: torch.Size([1024, 4096])\n",
"W_value: torch.Size([1024, 4096])\n",
"W_query: torch.Size([4096, 4096])\n"
]
}
],
"source": [
"gqa = GroupedQueryAttention(\n",
" d_in=embed_dim,\n",
" d_out=embed_dim,\n",
" num_heads=num_heads,\n",
" num_kv_groups=8,\n",
")\n",
"\n",
"gqa(example_batch)\n",
"\n",
"print(\"W_key:\", gqa.W_key.weight.shape)\n",
"print(\"W_value:\", gqa.W_value.weight.shape)\n",
"print(\"W_query:\", gqa.W_query.weight.shape)"
]
},
{
"cell_type": "markdown",
"id": "1a5d4c88-c66a-483b-b4e2-419ff9fd60d5",
"metadata": {
"id": "1a5d4c88-c66a-483b-b4e2-419ff9fd60d5"
},
"source": [
"- As a side note, to make the GroupedQueryAttention equivalent to standard multi-head attention, you can set the number of query groups (`num_kv_groups`) equal to the number of heads (`num_heads`)\n",
"- Lastly, let's compare the number of parameters below:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "58f713aa-ac00-4e2f-8247-94609aa01350",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "58f713aa-ac00-4e2f-8247-94609aa01350",
"outputId": "486dfd9c-9f3a-4b9e-f9a2-35fb43b9a5fb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of parameters:\n",
"MHA: 67,108,864\n",
"GQA: 41,943,040\n"
]
}
],
"source": [
"print(\"Total number of parameters:\")\n",
"\n",
"mha_total_params = sum(p.numel() for p in mha.parameters())\n",
"print(f\"MHA: {mha_total_params:,}\")\n",
"\n",
"gqa_total_params = sum(p.numel() for p in gqa.parameters())\n",
"print(f\"GQA: {gqa_total_params:,}\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "78b60dfd-6c0f-41f7-8f0c-8e57116f07f5",
"metadata": {
"id": "78b60dfd-6c0f-41f7-8f0c-8e57116f07f5"
},
"outputs": [],
"source": [
"# Free up memory:\n",
"del mha\n",
"del gqa"
]
},
{
"cell_type": "markdown",
"id": "8fcd8802-2859-45a2-905a-f4fe96629dd9",
"metadata": {
"id": "8fcd8802-2859-45a2-905a-f4fe96629dd9"
},
"source": [
"&nbsp;\n",
"## 1.4 Update the TransformerBlock module"
]
},
{
"cell_type": "markdown",
"id": "KABNccft_YnR",
"metadata": {
"id": "KABNccft_YnR"
},
"source": [
"- Next, we update the `TransformerBlock`\n",
"- Here, we simply swap `MultiHeadAttention` with `GroupedQueryAttention` and add the new RoPE settings\n",
"- In addition, we also modify the `forward` method so that it receives `mask`, `cos`, and `sin`; since the values for those are the same for each transformer block, we only have to compute them once and then can reuse them"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "f9fa8eb4-7196-4dee-aec6-0dcbc70921c4",
"metadata": {
"id": "f9fa8eb4-7196-4dee-aec6-0dcbc70921c4"
},
"outputs": [],
"source": [
"class TransformerBlock(nn.Module):\n",
" def __init__(self, cfg):\n",
" super().__init__()\n",
" self.att = GroupedQueryAttention( # MultiHeadAttention(\n",
" d_in=cfg[\"emb_dim\"],\n",
" d_out=cfg[\"emb_dim\"],\n",
" num_heads=cfg[\"n_heads\"],\n",
" num_kv_groups=cfg[\"n_kv_groups\"], # NEW\n",
" dtype=cfg[\"dtype\"]\n",
" )\n",
" self.ff = FeedForward(cfg)\n",
" self.norm1 = RMSNorm(cfg[\"emb_dim\"], eps=1e-5)\n",
" self.norm2 = RMSNorm(cfg[\"emb_dim\"], eps=1e-5)\n",
"\n",
" def forward(self, x, mask=None, cos=None, sin=None):\n",
" ##################### NEW #####################\n",
" # The forward method now accepts `mask` instead of accessing it via self.mask.\n",
" # Also, we now have cos and sin as input for RoPE\n",
" ################################################\n",
" # Shortcut connection for attention block\n",
" shortcut = x\n",
" x = self.norm1(x)\n",
" x = self.att(x.to(torch.bfloat16), mask, cos, sin) # Shape [batch_size, num_tokens, emb_size]\n",
" x = x + shortcut # Add the original input back\n",
"\n",
" # Shortcut connection for feed-forward block\n",
" shortcut = x\n",
" x = self.norm2(x)\n",
" x = self.ff(x.to(torch.bfloat16))\n",
" x = x + shortcut # Add the original input back\n",
"\n",
" return x"
]
},
{
"cell_type": "markdown",
"id": "fd921ab5-c48c-4c52-bf41-b847b3b822b9",
"metadata": {
"id": "fd921ab5-c48c-4c52-bf41-b847b3b822b9"
},
"source": [
"&nbsp;\n",
"## 1.5 Defining the model class"
]
},
{
"cell_type": "markdown",
"id": "M_tLAq_r_llN",
"metadata": {
"id": "M_tLAq_r_llN"
},
"source": [
"- When setting up the model class, we technically don't have to do much; we just update the name to `Llama3Model`\n",
"- However, since we now pass the `mask`, `cos`, and `sin` to the transformer blocks, we also have to add them here"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "475755d6-01f7-4e6e-ad9a-cec6f031ebf6",
"metadata": {
"id": "475755d6-01f7-4e6e-ad9a-cec6f031ebf6"
},
"outputs": [],
"source": [
"# class Llama2Model(nn.Module):\n",
"class Llama3Model(nn.Module):\n",
" def __init__(self, cfg):\n",
" super().__init__()\n",
" self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"], dtype=cfg[\"dtype\"])\n",
"\n",
" self.trf_blocks = nn.Sequential(\n",
" *[TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n",
"\n",
" self.final_norm = RMSNorm(cfg[\"emb_dim\"], eps=1e-5)\n",
" self.out_head = nn.Linear(cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False, dtype=cfg[\"dtype\"])\n",
"\n",
" #################### NEW #####################\n",
" cos, sin = precompute_rope_params(\n",
" head_dim=cfg[\"emb_dim\"] // cfg[\"n_heads\"],\n",
" theta_base=cfg[\"rope_base\"],\n",
" context_length=cfg[\"context_length\"],\n",
" freq_config=cfg[\"rope_freq\"]\n",
" )\n",
" \n",
" self.register_buffer(\"cos\", cos, persistent=False)\n",
" self.register_buffer(\"sin\", sin, persistent=False)\n",
" ##############################################\n",
"\n",
" self.cfg = cfg\n",
"\n",
" def forward(self, in_idx):\n",
" tok_embeds = self.tok_emb(in_idx)\n",
" x = tok_embeds\n",
"\n",
" #################### NEW #####################\n",
" num_tokens = x.shape[1]\n",
" mask = torch.triu(torch.ones(num_tokens, num_tokens, device=x.device, dtype=torch.bool), diagonal=1)\n",
" ##############################################\n",
" \n",
" for block in self.trf_blocks:\n",
" x = block(x, mask, self.cos, self.sin)\n",
" x = self.final_norm(x)\n",
" logits = self.out_head(x.to(self.cfg[\"dtype\"]))\n",
" return logits"
]
},
{
"cell_type": "markdown",
"id": "4bc94940-aaeb-45b9-9399-3a69b8043e60",
"metadata": {
"id": "4bc94940-aaeb-45b9-9399-3a69b8043e60"
},
"source": [
"&nbsp;\n",
"## 2. Initialize model"
]
},
{
"cell_type": "markdown",
"id": "HoGGRAGykQTE",
"metadata": {
"id": "HoGGRAGykQTE"
},
"source": [
"- Now we can define a Llama 3 config file (the Llama 2 config file is shown for comparison)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "e0564727-2d35-4f0c-b0fc-cde1e9134a18",
"metadata": {
"id": "e0564727-2d35-4f0c-b0fc-cde1e9134a18"
},
"outputs": [],
"source": [
"LLAMA2_CONFIG_7B = {\n",
" \"vocab_size\": 32_000, # Vocabulary size\n",
" \"context_length\": 4096, # Context length\n",
" \"emb_dim\": 4096, # Embedding dimension\n",
" \"n_heads\": 32, # Number of attention heads\n",
" \"n_layers\": 32, # Number of layers\n",
" \"hidden_dim\": 11_008, # Size of the intermediate dimension in FeedForward\n",
" \"dtype\": torch.bfloat16 # Lower-precision dtype to reduce memory usage\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "2ad90f82-15c7-4806-b509-e45b56f57db5",
"metadata": {
"id": "2ad90f82-15c7-4806-b509-e45b56f57db5"
},
"outputs": [],
"source": [
"LLAMA3_CONFIG_8B = {\n",
" \"vocab_size\": 128_256, # NEW: Larger vocabulary size\n",
" \"context_length\": 8192, # NEW: Larger context length\n",
" \"emb_dim\": 4096, # Embedding dimension\n",
" \"n_heads\": 32, # Number of attention heads\n",
" \"n_layers\": 32, # Number of layers\n",
" \"hidden_dim\": 14_336, # NEW: Larger size of the intermediate dimension in FeedForward\n",
" \"n_kv_groups\": 8, # NEW: Key-Value groups for grouped-query attention\n",
" \"rope_base\": 500_000.0, # NEW: The base in RoPE's \"theta\" was increased to 500_000\n",
" \"rope_freq\": None, # NEW: Additional configuration for adjusting the RoPE frequencies\n",
" \"dtype\": torch.bfloat16 # Lower-precision dtype to reduce memory usage\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "FAP7fiBzkaBz",
"metadata": {
"id": "FAP7fiBzkaBz"
},
"source": [
"- Using these settings, we can now initialize a Llama 3 8B model\n",
"- Note that this requires ~34 GB of memory (for comparison, Llama 2 7B required ~26 GB of memory)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "7004d785-ac9a-4df5-8760-6807fc604686",
"metadata": {
"id": "7004d785-ac9a-4df5-8760-6807fc604686"
},
"outputs": [],
"source": [
"model = Llama3Model(LLAMA3_CONFIG_8B)"
]
},
{
"cell_type": "markdown",
"id": "8056a521-91a6-440f-8473-591409c3177b",
"metadata": {},
"source": [
"- Let's now compute the number of trainable parameters:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "6079f747-8f20-4c6b-8d38-7156f1101729",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "6079f747-8f20-4c6b-8d38-7156f1101729",
"outputId": "0a8cd23b-d9fa-4c2d-ca63-3fc79bc4de0d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of parameters: 8,030,261,248\n"
]
}
],
"source": [
"total_params = sum(p.numel() for p in model.parameters())\n",
"print(f\"Total number of parameters: {total_params:,}\")"
]
},
{
"cell_type": "markdown",
"id": "Bx14NtzWk2wj",
"metadata": {
"id": "Bx14NtzWk2wj"
},
"source": [
"- As shown above, the model contains 8 billion parameters\n",
"- Additionally, we can calculate the memory requirements for this model using the code below:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "0df1c79e-27a7-4b0f-ba4e-167fe107125a",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0df1c79e-27a7-4b0f-ba4e-167fe107125a",
"outputId": "3425e9ce-d8c0-4b37-bded-a2c60b66a41a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"float32 (PyTorch default): 59.84 GB\n",
"bfloat16: 29.92 GB\n"
]
}
],
"source": [
"def model_memory_size(model, input_dtype=torch.float32):\n",
" total_params = 0\n",
" total_grads = 0\n",
" for param in model.parameters():\n",
" # Calculate total number of elements per parameter\n",
" param_size = param.numel()\n",
" total_params += param_size\n",
" # Check if gradients are stored for this parameter\n",
" if param.requires_grad:\n",
" total_grads += param_size\n",
"\n",
" # Calculate buffer size (non-parameters that require memory)\n",
" total_buffers = sum(buf.numel() for buf in model.buffers())\n",
"\n",
" # Size in bytes = (Number of elements) * (Size of each element in bytes)\n",
" # We assume parameters and gradients are stored in the same type as input dtype\n",
" element_size = torch.tensor(0, dtype=input_dtype).element_size()\n",
" total_memory_bytes = (total_params + total_grads + total_buffers) * element_size\n",
"\n",
" # Convert bytes to gigabytes\n",
" total_memory_gb = total_memory_bytes / (1024**3)\n",
"\n",
" return total_memory_gb\n",
"\n",
"print(f\"float32 (PyTorch default): {model_memory_size(model, input_dtype=torch.float32):.2f} GB\")\n",
"print(f\"bfloat16: {model_memory_size(model, input_dtype=torch.bfloat16):.2f} GB\")"
]
},
{
"cell_type": "markdown",
"id": "zudd-5PulKFL",
"metadata": {
"id": "zudd-5PulKFL"
},
"source": [
"- Lastly, we can also transfer the model to an NVIDIA or Apple Silicon GPU if applicable:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "a4c50e19-1402-45b6-8ccd-9077b2ba836d",
"metadata": {
"id": "a4c50e19-1402-45b6-8ccd-9077b2ba836d"
},
"outputs": [],
"source": [
"if torch.cuda.is_available():\n",
" device = torch.device(\"cuda\")\n",
"elif torch.backends.mps.is_available():\n",
" device = torch.device(\"mps\")\n",
"else:\n",
" device = torch.device(\"cpu\")\n",
"\n",
"model.to(device);"
]
},
{
"cell_type": "markdown",
"id": "5dc64a06-27dc-46ec-9e6d-1700a8227d34",
"metadata": {
"id": "5dc64a06-27dc-46ec-9e6d-1700a8227d34"
},
"source": [
"&nbsp;\n",
"## 3. Load tokenizer"
]
},
{
"cell_type": "markdown",
"id": "0eb30f0c-6144-4bed-87d9-6b2bac377005",
"metadata": {
"id": "0eb30f0c-6144-4bed-87d9-6b2bac377005"
},
"source": [
"- In this section, we are going to load the tokenizer for the model\n",
"- Llama 2 used Google's [SentencePiece](https://github.com/google/sentencepiece) tokenizer instead of OpenAI's BPE tokenizer based on the [Tiktoken](https://github.com/openai/tiktoken) library\n",
"- Llama 3, however, reverted back to using the BPE tokenizer from Tiktoken; specifically, it uses the GPT-4 tokenizer with an extended vocabulary\n",
"- You can find the original Tiktoken-adaptation by Meta AI [here](https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py) in their official Llama 3 repository\n",
"- Below, I rewrote the tokenizer code to make it more readable and minimal for this notebook (but the behavior should be similar)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "5f390cbf-8f92-46dc-afe3-d90b5affae10",
"metadata": {
"id": "5f390cbf-8f92-46dc-afe3-d90b5affae10"
},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"import tiktoken\n",
"from tiktoken.load import load_tiktoken_bpe\n",
"\n",
"\n",
"class Tokenizer:\n",
" \"\"\"Thin wrapper around tiktoken that keeps track of Llama-3 special IDs.\"\"\"\n",
" def __init__(self, model_path):\n",
" if not os.path.isfile(model_path):\n",
" raise FileNotFoundError(model_path)\n",
"\n",
" mergeable = load_tiktoken_bpe(model_path)\n",
"\n",
" # hard-coded from Meta's tokenizer.json\n",
" self.special = {\n",
" \"<|begin_of_text|>\": 128000,\n",
" \"<|end_of_text|>\": 128001,\n",
" \"<|start_header_id|>\": 128006,\n",
" \"<|end_header_id|>\": 128007,\n",
" \"<|eot_id|>\": 128009,\n",
" }\n",
" self.special.update({f\"<|reserved_{i}|>\": 128002 + i\n",
" for i in range(256)\n",
" if 128002 + i not in self.special.values()})\n",
"\n",
" self.model = tiktoken.Encoding(\n",
" name=Path(model_path).name,\n",
" pat_str=r\"(?i:'s|'t|'re|'ve|'m|'ll|'d)\"\n",
" r\"|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+\"\n",
" r\"|\\p{N}{1,3}\"\n",
" r\"| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*\"\n",
" r\"|\\s*[\\r\\n]+\"\n",
" r\"|\\s+(?!\\S)\"\n",
" r\"|\\s+\",\n",
" mergeable_ranks=mergeable,\n",
" special_tokens=self.special,\n",
" )\n",
"\n",
" def encode(self, text, bos=False, eos=False):\n",
" ids = ([self.special[\"<|begin_of_text|>\"]] if bos else []) \\\n",
" + self.model.encode(text)\n",
" if eos:\n",
" ids.append(self.special[\"<|end_of_text|>\"])\n",
" return ids\n",
"\n",
" def decode(self, ids):\n",
" return self.model.decode(ids)"
]
},
{
"cell_type": "markdown",
"id": "0a1509f8-8778-4fec-ba32-14d95c646167",
"metadata": {
"id": "0a1509f8-8778-4fec-ba32-14d95c646167"
},
"source": [
"- Meta AI shared the original Llama 3 model weights and tokenizer vocabulary on the Hugging Face Hub\n",
"- We will first download the tokenizer vocabulary from the Hub and load it into the code above"
]
},
{
"cell_type": "markdown",
"id": "KbnlzsbYmJU6",
"metadata": {
"id": "KbnlzsbYmJU6"
},
"source": [
"- Please note that Meta AI requires that you accept the Llama 3 licensing terms before you can download the files; to do this, you have to create a Hugging Face Hub account and visit the [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) repository to accept the terms\n",
"- Next, you will need to create an access token; to generate an access token with READ permissions, click on the profile picture in the upper right and click on \"Settings\"\n",
"\n",
"\n",
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/settings.webp?1\" width=\"300px\">\n",
"\n",
"- Then, create and copy the access token so you can copy & paste it into the next code cell\n",
"\n",
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/access-token.webp?1\" width=\"600px\">"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "3357a230-b678-4691-a238-257ee4e80185",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3357a230-b678-4691-a238-257ee4e80185",
"outputId": "a3652def-ea7f-46fb-f293-2a59affb71a0"
},
"outputs": [],
"source": [
"from huggingface_hub import login\n",
"import json\n",
"\n",
"with open(\"config.json\", \"r\") as config_file:\n",
" config = json.load(config_file)\n",
" access_token = config[\"HF_ACCESS_TOKEN\"]\n",
"\n",
"login(token=access_token)"
]
},
{
"cell_type": "markdown",
"id": "IxGh6ZYQo0VN",
"metadata": {
"id": "IxGh6ZYQo0VN"
},
"source": [
"- After login via the access token, which is necessary to verify that we accepted the Llama 3 licensing terms, we can now download the tokenizer vocabulary:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "69714ea8-b9b8-4687-8392-f3abb8f93a32",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "69714ea8-b9b8-4687-8392-f3abb8f93a32",
"outputId": "c9836ba8-5176-4dd5-b618-6cc36fdbe1f0"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "685326b4fd014ff689e928f4200f5182",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"original/tokenizer.model: 0%| | 0.00/2.18M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from huggingface_hub import hf_hub_download\n",
"\n",
"tokenizer_file_path = hf_hub_download(\n",
" repo_id=\"meta-llama/Meta-Llama-3-8B\",\n",
" filename=\"original/tokenizer.model\",\n",
" local_dir=\"Llama-3-8B\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "F8BH1Nk0AYCS",
"metadata": {
"id": "F8BH1Nk0AYCS"
},
"source": [
"- Note that for using Llama 3 files, we may need the `blobfile` package, which is used when handling datasets or models stored in cloud storage solutions like Google Cloud Storage (GCS), Azure Blob Storage, or Amazon S3\n",
"- You can install this dependency by uncommenting and executing the `pip` command below\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "5dm6Oz7uAytV",
"metadata": {
"id": "5dm6Oz7uAytV"
},
"outputs": [],
"source": [
"# pip install blobfile"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "8b8c0ce6-a6fb-4b8a-8de2-ee7bb7646fd0",
"metadata": {
"id": "8b8c0ce6-a6fb-4b8a-8de2-ee7bb7646fd0"
},
"outputs": [],
"source": [
"tokenizer = Tokenizer(tokenizer_file_path)"
]
},
{
"cell_type": "markdown",
"id": "NVhmFeX3pT_M",
"metadata": {
"id": "NVhmFeX3pT_M"
},
"source": [
"- We can now use the `generate` function to have the Llama 3 model generate new text:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "e0a2b5cd-6cba-4d72-b8ff-04d8315d483e",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "e0a2b5cd-6cba-4d72-b8ff-04d8315d483e",
"outputId": "990d7b74-cb35-476b-d8bd-d544006e00f4",
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output text:\n",
" Every effort_dead aeros Ingredients başında.extensionégor clangmissions güc như submodule.and report官方%，.Reader(\",\");\n",
"ामल ندار Parliamentary !!! HigginsDynamicZhamincus_beam cyc......\n",
"\n",
" haciendo\n"
]
}
],
"source": [
"from previous_chapters import generate, text_to_token_ids, token_ids_to_text\n",
"# If the `previous_chapters.py` file is not available locally,\n",
"# you can import it from the `llms-from-scratch` PyPI package.\n",
"# For details, see: https://github.com/rasbt/LLMs-from-scratch/tree/main/pkg\n",
"# E.g.,\n",
"# from llms_from_scratch.ch05 import generate, text_to_token_ids, token_ids_to_text\n",
"\n",
"\n",
"torch.manual_seed(123)\n",
"\n",
"token_ids = generate(\n",
" model=model,\n",
" idx=text_to_token_ids(\"Every effort\", tokenizer).to(device),\n",
" max_new_tokens=30,\n",
" context_size=LLAMA3_CONFIG_8B[\"context_length\"],\n",
" top_k=1,\n",
" temperature=0.\n",
")\n",
"\n",
"print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
]
},
{
"cell_type": "markdown",
"id": "93WTtAA5paYV",
"metadata": {
"id": "93WTtAA5paYV"
},
"source": [
"- Of course, as we can see above, the text is nonsensical since we haven't trained the Llama 3 model yet\n",
"- In the next section, instead of training it ourselves, which would cost tens to hundreds of thousands of dollars, we load the pretrained weights from Meta AI"
]
},
{
"cell_type": "markdown",
"id": "f63cc248-1d27-4eb6-aa50-173b436652f8",
"metadata": {
"id": "f63cc248-1d27-4eb6-aa50-173b436652f8"
},
"source": [
"&nbsp;\n",
"## 4. Load pretrained weights"
]
},
{
"cell_type": "markdown",
"id": "aKeN7rUfqZMI",
"metadata": {
"id": "aKeN7rUfqZMI"
},
"source": [
"- We are loading the [\"meta-llama/Meta-Llama-3-8B\"](https://huggingface.co/meta-llama/Meta-Llama-3-8B) base model below, which is a simple text completion model before finetuning\n",
"- Alternatively, you can load the instruction-finetuned and aligned [\"meta-llama/Meta-Llama-3-8B-Instruct\"](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) model by modifying the string in the next code cell accordingly\n",
"- Combined, the weight files are about 16 GB large"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "5fa9c06c-7a53-4b4d-9ce4-acc027322ee4",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 145,
"referenced_widgets": [
"f3788acce34f4956b0727b58d0cf38c6",
"6022a9426683420690d9b41a0ca4f870",
"e9aba3d53b4d45c485a7aad649c7b465",
"f1a12d7929db4309b9881853135359fc",
"58c9dec75a3346b1b787f88dd510d254",
"9492edc02dee456f840325d913fa4e4f",
"66dc94b23556499f985f8accbb1f89cb",
"7c6658cfff1a4d27af3de148184f77d9",
"7266a729edfb4a44b5b1c67dc79be146",
"76dbab4873f342019c5d7624ae2c9775",
"3cea4b431147441a8d9bd872811d5974",
"8ae98969541849efa356cf912ac39b1e",
"f9373112649945e3b446c3e1ec274dc1",
"d49791082a304ade95c185c79fae1f41",
"616e383bb3d442bcb6edb2721a8180b6",
"87f474861e54432e9d533e0a89bb77da",
"e805bb6dfee34dab8870f4618d8bffdb",
"be3e9bf271f04eb0b119659e1af3a0ea",
"00148825ce0248b7a23eb28e3eca6749",
"f1a9b0c2431640298a6c1b258298b12d",
"8ba9f009e92a46fcbcbb401dc444f12e",
"d74186bb74d142dfb683fa347b6990f7",
"9bb60a5a3710463ebe3a17f8d2a446be",
"0a08fb81165748748ccb080e6df0600f",
"603690f543114a7fb6aebd433c80bdc3",
"773b802daed942f5a11f3eab3b83be08",
"7989003a613e45f780d3f800e121543a",
"9d49589118f5432cac49650251046429",
"f114549fe8ce49638a791ca2fecb2d89",
"0aa155b794a8426aa265f4a7670f43ad",
"a06fbde549cc47fdaddfbdb82d35d823",
"172c0c6955e1428b999dcb2d133704cd",
"1bf7108774c34016a2193e2cd7639b7d",
"ed28e180d94a4b7aa548581612e31232",
"ff4338faded5494da1ccb660e1c441ed",
"b46a08cf4929422eb0f76d8d9af11249",
"f049eb4a50f54c34912ca959d2eaf353",
"80dfd3e80ceb444a83ec1fd65f9af80e",
"519147a10b984befbd0f255f78c1f66a",
"562e82438dbe41b793ff488b8447c5bf",
"1da83719e47c4196b06f3aa32056b560",
"c4a2c88326d14fbca87cfde073755a2e",
"f0ab5a46cbb0444c88ed137d8a95002b",
"f8f28ac0e149428f9fef42373c6a87d0"
]
},
"id": "5fa9c06c-7a53-4b4d-9ce4-acc027322ee4",
"outputId": "c05118ce-9f81-41c8-a1f2-72caa932ae86"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3af9f77314b14682bbdd1c4921cd193e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00001-of-00004.safetensors: 0%| | 0.00/4.98G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7aeb092ad0a14b5e9aaf33bea4751490",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00002-of-00004.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "20adbc86984344a39a55f012b8c18d68",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00003-of-00004.safetensors: 0%| | 0.00/4.92G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e6bb24f8ca4344dfb3870fca8c90e4fb",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00004-of-00004.safetensors: 0%| | 0.00/1.17G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from safetensors.torch import load_file\n",
"\n",
"combined_weights = {}\n",
"\n",
"for i in range(1, 5):\n",
" weights_file = hf_hub_download(\n",
" repo_id=\"meta-llama/Meta-Llama-3-8B\",\n",
" filename=f\"model-0000{i}-of-00004.safetensors\",\n",
" local_dir=\"Llama-3-8B\"\n",
" )\n",
" current_weights = load_file(weights_file)\n",
" combined_weights.update(current_weights)"
]
},
{
"cell_type": "markdown",
"id": "-15SJ7btq2zE",
"metadata": {
"id": "-15SJ7btq2zE"
},
"source": [
"- The `weights` contains the following tensors (only the first 15 are shown for simplicity):"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "ee26bd0b-fea9-4924-97f7-409c14f28e49",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ee26bd0b-fea9-4924-97f7-409c14f28e49",
"outputId": "2fbc2786-677f-4fea-9472-5fb8542ff14b"
},
"outputs": [
{
"data": {
"text/plain": [
"['model.embed_tokens.weight',\n",
" 'model.layers.0.input_layernorm.weight',\n",
" 'model.layers.0.mlp.down_proj.weight',\n",
" 'model.layers.0.mlp.gate_proj.weight',\n",
" 'model.layers.0.mlp.up_proj.weight',\n",
" 'model.layers.0.post_attention_layernorm.weight',\n",
" 'model.layers.0.self_attn.k_proj.weight',\n",
" 'model.layers.0.self_attn.o_proj.weight',\n",
" 'model.layers.0.self_attn.q_proj.weight',\n",
" 'model.layers.0.self_attn.v_proj.weight',\n",
" 'model.layers.1.input_layernorm.weight',\n",
" 'model.layers.1.mlp.down_proj.weight',\n",
" 'model.layers.1.mlp.gate_proj.weight',\n",
" 'model.layers.1.mlp.up_proj.weight',\n",
" 'model.layers.1.post_attention_layernorm.weight']"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(combined_weights.keys())[:15]"
]
},
{
"cell_type": "markdown",
"id": "UeeSpnunrDFB",
"metadata": {
"id": "UeeSpnunrDFB"
},
"source": [
"- The following function, modeled after the `load_weights_into_gpt` function in [chapter 5](../01_main-chapter-code/ch05.ipynb), loads the pretrained weights into our Llama 3 model:"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "3820e2a7-4f26-41bc-953b-f3879b0aff65",
"metadata": {
"id": "3820e2a7-4f26-41bc-953b-f3879b0aff65"
},
"outputs": [],
"source": [
"def assign(left, right, tensor_name=\"unknown\"):\n",
" if left.shape != right.shape:\n",
" raise ValueError(f\"Shape mismatch in tensor '{tensor_name}'. Left: {left.shape}, Right: {right.shape}\")\n",
" \n",
" with torch.no_grad():\n",
" if isinstance(right, torch.Tensor):\n",
" left.copy_(right)\n",
" else:\n",
" left.copy_(torch.as_tensor(right, dtype=left.dtype, device=left.device))\n",
"\n",
" return left \n",
"\n",
"\n",
"def load_weights_into_llama(model, param_config, params):\n",
" model.tok_emb.weight = assign(model.tok_emb.weight, params[\"model.embed_tokens.weight\"], \"model.embed_tokens.weight\")\n",
"\n",
" for l in range(param_config[\"n_layers\"]):\n",
"\n",
" # Load attention weights\n",
" model.trf_blocks[l].att.W_query.weight = assign(\n",
" model.trf_blocks[l].att.W_query.weight,\n",
" params[f\"model.layers.{l}.self_attn.q_proj.weight\"],\n",
" f\"model.layers.{l}.self_attn.q_proj.weight\"\n",
" )\n",
" model.trf_blocks[l].att.W_key.weight = assign(\n",
" model.trf_blocks[l].att.W_key.weight,\n",
" params[f\"model.layers.{l}.self_attn.k_proj.weight\"],\n",
" f\"model.layers.{l}.self_attn.k_proj.weight\"\n",
" )\n",
" model.trf_blocks[l].att.W_value.weight = assign(\n",
" model.trf_blocks[l].att.W_value.weight,\n",
" params[f\"model.layers.{l}.self_attn.v_proj.weight\"],\n",
" f\"model.layers.{l}.self_attn.v_proj.weight\"\n",
" )\n",
" model.trf_blocks[l].att.out_proj.weight = assign(\n",
" model.trf_blocks[l].att.out_proj.weight,\n",
" params[f\"model.layers.{l}.self_attn.o_proj.weight\"],\n",
" f\"model.layers.{l}.self_attn.o_proj.weight\"\n",
" )\n",
" model.trf_blocks[l].norm1.weight = assign(\n",
" model.trf_blocks[l].norm1.weight,\n",
" params[f\"model.layers.{l}.input_layernorm.weight\"],\n",
" f\"model.layers.{l}.input_layernorm.weight\"\n",
" )\n",
"\n",
" # Load FeedForward weights\n",
" model.trf_blocks[l].ff.fc1.weight = assign(\n",
" model.trf_blocks[l].ff.fc1.weight,\n",
" params[f\"model.layers.{l}.mlp.gate_proj.weight\"],\n",
" f\"model.layers.{l}.mlp.gate_proj.weight\"\n",
" )\n",
" model.trf_blocks[l].ff.fc2.weight = assign(\n",
" model.trf_blocks[l].ff.fc2.weight,\n",
" params[f\"model.layers.{l}.mlp.up_proj.weight\"],\n",
" f\"model.layers.{l}.mlp.up_proj.weight\"\n",
" )\n",
" model.trf_blocks[l].ff.fc3.weight = assign(\n",
" model.trf_blocks[l].ff.fc3.weight,\n",
" params[f\"model.layers.{l}.mlp.down_proj.weight\"],\n",
" f\"model.layers.{l}.mlp.down_proj.weight\"\n",
" )\n",
" model.trf_blocks[l].norm2.weight = assign(\n",
" model.trf_blocks[l].norm2.weight,\n",
" params[f\"model.layers.{l}.post_attention_layernorm.weight\"],\n",
" f\"model.layers.{l}.post_attention_layernorm.weight\"\n",
" )\n",
"\n",
" # Load output layer weights\n",
" model.final_norm.weight = assign(model.final_norm.weight, params[\"model.norm.weight\"], \"model.norm.weight\")\n",
"\n",
" if \"lm_head.weight\" in params.keys():\n",
" model.out_head.weight = assign(model.out_head.weight, params[\"lm_head.weight\"], \"lm_head.weight\")\n",
" else:\n",
" model.out_head.weight = model.tok_emb.weight\n",
" print(\"Model uses weight tying.\")\n",
"\n",
"\n",
"load_weights_into_llama(model, LLAMA3_CONFIG_8B, combined_weights)\n",
"model.to(device);\n",
"del combined_weights # free up memory"
]
},
{
"cell_type": "markdown",
"id": "TDuv_Us2rNvk",
"metadata": {
"id": "TDuv_Us2rNvk"
},
"source": [
"- Next, we are ready to use the model for text generation"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "240987e8-a023-462e-9376-9edfb27559ec",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "240987e8-a023-462e-9376-9edfb27559ec",
"outputId": "6dab0e56-40a8-45db-a096-ab2b9ee97a69"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output text:\n",
" Every effort has been made to trace copyright holders and to obtain their permission for the use of copyright material. The publisher apologizes for any\n"
]
}
],
"source": [
"torch.manual_seed(123)\n",
"\n",
"token_ids = generate(\n",
" model=model,\n",
" idx=text_to_token_ids(\"Every effort\", tokenizer).to(device),\n",
" max_new_tokens=25,\n",
" context_size=LLAMA3_CONFIG_8B[\"context_length\"],\n",
" top_k=1,\n",
" temperature=0.\n",
")\n",
"\n",
"print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
]
},
{
"cell_type": "markdown",
"id": "1203041e-4794-4157-a978-3ce80909da44",
"metadata": {
"id": "1203041e-4794-4157-a978-3ce80909da44"
},
"source": [
"&nbsp;\n",
"## 5. Using the instruction-finetuned model"
]
},
{
"cell_type": "markdown",
"id": "akyo7WNyF_YL",
"metadata": {
"id": "akyo7WNyF_YL"
},
"source": [
"- Above, we used the pretrained base model; if you want to use a model capable of following instructions, use the `\"meta-llama/Llama-3-8B-Instruct\"` model instead, as shown below"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "hdA-xjjdS26J",
"metadata": {
"id": "hdA-xjjdS26J"
},
"outputs": [],
"source": [
"# to free up memory\n",
"\n",
"import gc\n",
"\n",
"del model\n",
"\n",
"gc.collect() # Run Python garbage collector\n",
"\n",
"if torch.cuda.is_available():\n",
" torch.cuda.empty_cache()"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "nbvAV7vaz6yc",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 145,
"referenced_widgets": [
"409470784b6346a981920350de4f6f28",
"9ba6a11ffd194bf9a0900f52a7ed4d4f",
"acae8bbbb4a84ed49be72fecd11fb052",
"e8a4b441281b4038bb0204d093411f68",
"bdf8b693821344fc97918e6cbc31c8bf",
"97e8877869cd4be68ff38ce745be5045",
"cc3da88e93c4499993b7bbb7d3064326",
"0d51fdc2c416474da04079db6579890f",
"c4598300a77b4667b1117f9499f5ccb7",
"77606cd2fe1b4d33a91ede944bb1dec0",
"f1ba439c26d64c90af2f162c74348405",
"d598f094c3ce4daeab19fac8094cba7e",
"0afc2d23514b45c9890b5d2ee4e6fa0b",
"3da5d38bf3314d3eaa7cedebae41c076",
"55e6b727a4594078beb3853cc1891308",
"f17fa78263414ef8b414c7bf3ac03192",
"e8b187b40ec14db3af17a380830a35bf",
"e94ca32eaa9f4714a3b05a5fdf24d02b",
"3edd464991204b8690eae02f10b4cc00",
"ac1e34f4bd6c420bb6cc2fdde5f3ed4d",
"1cd5e07cad35450182004952de32c8e7",
"a63351a6715643378491ba831b3fb05d",
"98b4680141ee423bb5e43c47613d8440",
"b02ffefca3f34252914e76f4a8a467dc",
"31d27bf34a74432f8e0dbfe9ecb76130",
"a3137f3669b54e84be91010c9654d985",
"5a2886564d3f40ceaa30b743dbe81f45",
"15ea8fcfe097471e8fc9502a162f5904",
"c779e80c50ba4434bfa1d326c5cc9b0f",
"eb94612785e64552aea8674dc8647a93",
"279cffe683fe4e7383062162e07ed9ed",
"6176990205cc499f8995c71fc6b9d4df",
"66c23ae98bcc45f18fc5c91e0e73c3e4",
"05b502e1e3a9436297dafbb1ce7af722",
"25977b0d89084703ad787fe9208b5aad",
"71a84ee5fc964ec89ff2832c84735cc2",
"6aed783eccb942318e6384e253ad4924",
"84c34bfecda64391a609e19f131d51d4",
"20ecac7c646b45938ed393cb20977c37",
"ebe04aeaaac042aaaa0885992e45793d",
"ca81071ab07446df96795a482ce0c630",
"e0550cab24c7492787af40dc4b8576bf",
"7015bf6f85954036aaf8cc4f1c44ea0f",
"2a2ba3d065634484a932b8d3c212af56"
]
},
"id": "nbvAV7vaz6yc",
"outputId": "9e1badc9-a6c4-48b7-9125-e0810655528b"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "bdcebc6a21ae41e3bb78834b4f244fae",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00001-of-00004.safetensors: 0%| | 0.00/4.98G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "89949427bf5142c29c54978c4f0ae52a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00002-of-00004.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a88b441b15714e138db6fa813dd82a47",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00003-of-00004.safetensors: 0%| | 0.00/4.92G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "1c4f8df93db246d18494820bb8ec37be",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00004-of-00004.safetensors: 0%| | 0.00/1.17G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"combined_weights = {}\n",
"\n",
"for i in range(1, 5):\n",
" weights_file = hf_hub_download(\n",
" repo_id=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
" filename=f\"model-0000{i}-of-00004.safetensors\",\n",
" local_dir=\"Llama-3-8B-Instruct\"\n",
" )\n",
" current_weights = load_file(weights_file)\n",
" combined_weights.update(current_weights)\n",
"\n",
"\n",
"model = Llama3Model(LLAMA3_CONFIG_8B)\n",
"load_weights_into_llama(model, LLAMA3_CONFIG_8B, combined_weights)\n",
"model.to(device)\n",
"del combined_weights # free up memory"
]
},
{
"cell_type": "markdown",
"id": "VlH7qYVdDKQr",
"metadata": {
"id": "VlH7qYVdDKQr"
},
"source": [
"- Note that the Llama 3 model should ideally be used with the correct prompt template that was used during finetuning (as discussed in chapter 7)\n",
"- Below is a wrapper class around the tokenizer based on Meta AI's Llama 3-specific [ChatFormat code](https://github.com/meta-llama/llama3/blob/11817d47e1ba7a4959b025eb1ca308572e0e3963/llama/tokenizer.py#L202) that constructs the prompt template"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "4be5b481-1110-46e8-a931-3988d890cf8c",
"metadata": {
"id": "4be5b481-1110-46e8-a931-3988d890cf8c"
},
"outputs": [],
"source": [
"class ChatFormat:\n",
"\n",
" def __init__(self, tokenizer: Tokenizer, *,\n",
" default_system=\"You are a helpful assistant.\"):\n",
" self.tok = tokenizer\n",
" self.default_system = default_system\n",
"\n",
" def _header(self, role):\n",
" \"\"\"Encode <|start_header_id|>role<|end_header_id|>\\n\\n\"\"\"\n",
" return (\n",
" [self.tok.special[\"<|start_header_id|>\"]]\n",
" + self.tok.encode(role)\n",
" + [self.tok.special[\"<|end_header_id|>\"]]\n",
" + self.tok.encode(\"\\n\\n\")\n",
" )\n",
"\n",
" def encode(self, user_message, system_message=None):\n",
" sys_msg = system_message if system_message is not None else self.default_system\n",
"\n",
" ids = [self.tok.special[\"<|begin_of_text|>\"]]\n",
"\n",
" # system\n",
" ids += self._header(\"system\")\n",
" ids += self.tok.encode(sys_msg)\n",
" ids += [self.tok.special[\"<|eot_id|>\"]]\n",
"\n",
" # user\n",
" ids += self._header(\"user\")\n",
" ids += self.tok.encode(user_message)\n",
" ids += [self.tok.special[\"<|eot_id|>\"]]\n",
"\n",
" # assistant header (no content yet)\n",
" ids += self._header(\"assistant\")\n",
"\n",
" return ids"
]
},
{
"cell_type": "markdown",
"id": "M-dkSNvwDttN",
"metadata": {
"id": "M-dkSNvwDttN"
},
"source": [
"- The usage is as follows:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "nwBrTGTsUNhn",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "nwBrTGTsUNhn",
"outputId": "72a495b4-b872-429a-88ef-49a9b4577f0f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 18328, 13, 128009, 128006, 882, 128007, 271, 9906, 4435, 0, 128009, 128006, 78191, 128007, 271]\n"
]
}
],
"source": [
"tokenizer = Tokenizer(tokenizer_file_path)\n",
"chat_tokenizer = ChatFormat(tokenizer)\n",
"\n",
"token_ids = chat_tokenizer.encode(\"Hello World!\")\n",
"print(token_ids)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "0fpmpVgYVTRZ",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 36
},
"id": "0fpmpVgYVTRZ",
"outputId": "bb3e819a-112a-466c-ac51-5d14a9c3475b"
},
"outputs": [
{
"data": {
"text/plain": [
"'<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\nHello World!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n'"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer.decode(token_ids)"
]
},
{
"cell_type": "markdown",
"id": "Wo-aUGeKDvqq",
"metadata": {
"id": "Wo-aUGeKDvqq"
},
"source": [
"- Let's now see the Llama 3 instruction model in action:"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "ozGOBu6XOkEW",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ozGOBu6XOkEW",
"outputId": "4f689c70-bed9-46f3-a52a-aea47b641283"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output text:\n",
" Llamas are herbivores, which means they primarily eat plants and plant-based foods. Their diet typically consists of:\n",
"\n",
"1. Grasses: Llamas love to graze on grasses, including tall grasses, short grasses, and even weeds.\n",
"2. Hay: Hay is a staple in a llama's diet. They enjoy a variety of hays, such as timothy hay, alfalfa hay, and oat hay.\n",
"3. Grains: Llamas may be fed grains like oats, corn, and barley as a supplement to their diet.\n",
"4. Fruits and vegetables: Llamas enjoy fruits and vegetables like apples, carrots, and sweet potatoes as treats or additions to their meals.\n",
"5. Minerals:\n"
]
}
],
"source": [
"torch.manual_seed(123)\n",
"\n",
"token_ids = generate(\n",
" model=model,\n",
" idx=text_to_token_ids(\"What do llamas eat?\", chat_tokenizer).to(device),\n",
" max_new_tokens=150,\n",
" context_size=LLAMA3_CONFIG_8B[\"context_length\"],\n",
" top_k=1,\n",
" temperature=0.\n",
")\n",
"\n",
"output_text = token_ids_to_text(token_ids, tokenizer)\n",
"\n",
"\n",
"def clean_text(text, header_end=\"assistant<|end_header_id|>\\n\\n\"):\n",
" # Find the index of the first occurrence of \"<|end_header_id|>\"\n",
" index = text.find(header_end)\n",
"\n",
" if index != -1:\n",
" # Return the substring starting after \"<|end_header_id|>\"\n",
" return text[index + len(header_end):].strip() # Strip removes leading/trailing whitespace\n",
" else:\n",
" # If the token is not found, return the original text\n",
" return text\n",
"\n",
"print(\"Output text:\\n\", clean_text(output_text))"
]
},
{
"cell_type": "markdown",
"id": "2r5JKrO-ZOHK",
"metadata": {
"id": "2r5JKrO-ZOHK"
},
"source": [
"&nbsp;\n",
"# Llama 3.1 8B"
]
},
{
"cell_type": "markdown",
"id": "QiQxX0XnP_iC",
"metadata": {
"id": "QiQxX0XnP_iC"
},
"source": [
"- A few months after the initial Llama 3 release, Meta AI followed up with their Llama 3.1 suite of models (see the official [Introducing Llama 3.1: Our most capable models to date](https://ai.meta.com/blog/meta-llama-3-1/) announcement blog post for details)\n",
"- Conveniently, we can reuse our previous Llama 3 code from above to implement Llama 3.1 8B\n",
"\n",
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/llama3-to-llama31.webp\" width=\"700px\">\n",
"\n",
"- The architecture is identical, with the only change being a rescaling of the RoPE frequencies as indicated in the configuration file below\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "X5Fg8XUHMv4M",
"metadata": {
"id": "X5Fg8XUHMv4M"
},
"outputs": [],
"source": [
"LLAMA3_CONFIG_8B = {\n",
" \"vocab_size\": 128_256, # Vocabulary size\n",
" \"context_length\": 8192, # Context length\n",
" \"emb_dim\": 4096, # Embedding dimension\n",
" \"n_heads\": 32, # Number of attention heads\n",
" \"n_layers\": 32, # Number of layers\n",
" \"hidden_dim\": 14_336, # Size of the intermediate dimension in FeedForward\n",
" \"n_kv_groups\": 8, # Key-Value groups for grouped-query attention\n",
" \"rope_base\": 500_000.0, # The base in RoPE's \"theta\"\n",
" \"rope_freq\": None, # Additional configuration for adjusting the RoPE frequencies\n",
" \"dtype\": torch.bfloat16 # Lower-precision dtype to reduce memory usage\n",
"}\n",
"\n",
"LLAMA31_CONFIG_8B = {\n",
" \"vocab_size\": 128_256, # Vocabulary size\n",
" \"context_length\": 131_072, # NEW: Larger supported context length\n",
" \"emb_dim\": 4096, # Embedding dimension\n",
" \"n_heads\": 32, # Number of attention heads\n",
" \"n_layers\": 32, # Number of layers\n",
" \"hidden_dim\": 14_336, # Size of the intermediate dimension in FeedForward\n",
" \"n_kv_groups\": 8, # Key-Value groups for grouped-query attention\n",
" \"rope_base\": 500_000.0, # The base in RoPE's \"theta\"\n",
" \"dtype\": torch.bfloat16, # Lower-precision dtype to reduce memory usage\n",
" \"rope_freq\": { # NEW: RoPE frequency scaling\n",
" \"factor\": 8.0,\n",
" \"low_freq_factor\": 1.0,\n",
" \"high_freq_factor\": 4.0,\n",
" \"original_context_length\": 8192,\n",
" }\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "xa3bpMDtTdBs",
"metadata": {
"id": "xa3bpMDtTdBs"
},
"source": [
"- As we've seen in the code earlier, the RoPE method uses sinusoidal functions (sine and cosine) to embed positional information directly into the attention mechanism\n",
"- In Llama 3.1, via the additional configuration, we introduce additional adjustments to the inverse frequency calculations\n",
"- These adjustments influence how different frequency components contribute to the positional embeddings (a detailed explanation is a topic for another time)\n",
"- Let's try out the Llama 3.1 model in practice; first, we clear out the old model to free up some GPU memory"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "7dUtYnNUOqhL",
"metadata": {
"id": "7dUtYnNUOqhL"
},
"outputs": [],
"source": [
"# free up memory\n",
"del model\n",
"\n",
"gc.collect() # Run Python garbage collector\n",
"\n",
"if torch.cuda.is_available():\n",
" torch.cuda.empty_cache()"
]
},
{
"cell_type": "markdown",
"id": "DbbVsll6TYWR",
"metadata": {
"id": "DbbVsll6TYWR"
},
"source": [
"- Next, we download the tokenizer\n",
"- Note that since the Llama 3.1 family is distinct from the Llama 3 family, you'd have to go to the [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B) repository and acknowledge the license terms for your Hugging Face access token to work for the download\n",
"- Tip: For simplicity, we only load the base model below, but there's also an instruction-finetuned version you can use by replacing `\"meta-llama/Llama-3.1-8B\"` with `\"meta-llama/Llama-3.1-8B-Instruct\"`"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "8xDk4chtPNU4",
"metadata": {
"id": "8xDk4chtPNU4"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ac808a4fe89d4ca89597a90f6ab83a30",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"original/tokenizer.model: 0%| | 0.00/2.18M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"tokenizer_file_path = hf_hub_download(\n",
" repo_id=\"meta-llama/Llama-3.1-8B\",\n",
" filename=\"original/tokenizer.model\",\n",
" local_dir=\"Llama-3.1-8B\"\n",
")\n",
"\n",
"tokenizer = Tokenizer(tokenizer_file_path)"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "a7l21VE4Otcs",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "a7l21VE4Otcs",
"outputId": "3dd5cfba-bf3f-44d2-9be1-7cd42bfe4ba9"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of parameters: 8,030,261,248\n"
]
}
],
"source": [
"model = Llama3Model(LLAMA31_CONFIG_8B)\n",
"\n",
"total_params = sum(p.numel() for p in model.parameters())\n",
"print(f\"Total number of parameters: {total_params:,}\")"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "u4J7IxOvOyPM",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 145,
"referenced_widgets": [
"5bbaa046d8934c8fae0a12c3d7bd991b",
"e1e4125eac004bae92dc1f22f673bf0e",
"d5b4bb4891ec4e44be46e9815c7e10dc",
"4f6595a392b244bd8e887935defc06f0",
"100c1b15cc4046cea1147f657eb2d8d0",
"81458e7953a349cfafccaa213b370406",
"a3dc9dfadae642b4a873705596739468",
"f55b59efcefa4ad5955d082f4bf7c637",
"1b02e0c7d1604b1c87a327c4c4f8b0e7",
"02ad170019454fd096b37347de5c481d",
"c52e0f34892b4daa84c1bf61500ac399",
"af985cf6fa26475eb2c4dd81e0c79ff4",
"8659c3eddb014c3bb5931fd9e6fadad8",
"f5fa00d96c4c49e48e1806d23a5b8570",
"080c484114f64f5591fa1287a35b46c9",
"14dc6a3717484c55a116612e28447dbb",
"00d3286c9c1d4161bb777b7b65ae744d",
"66f27fb11edf453b8144c2dfcdc66baa",
"5798e5118430439fb1f6bf29e1bafe58",
"357f367cf74146b8825be371acd51d06",
"94073be250cd42d5b82e196e30cbf22e",
"0cd0724f825e480389a82f0c49f91e6d",
"dffa208978f34e6a9aae94ecda92fe67",
"b8a98f163ebd4ac89af08a49c0881c23",
"f0d9febe1a634a0ba7e8e50fa104dcc2",
"e23870f0c7ff40cc8fa6a1e862a4af99",
"87da9905a0534c26ad0712ad426ca930",
"b953419300604b8e86fc0ad003fdfd2f",
"f1865ed0fbcc40eeabdca90a43d00069",
"ea0128909a9d4801ba312a876b0cf183",
"d160986df978416c9ad91d1e10fc90fc",
"5e97f7c2e8f5453dafcdad0552060e60",
"4b3e7b8774df4b458bb6c6146fe3226d",
"2ffd8dbed00e46d2887b9a2590cad297",
"a06dcb3bdfc84905a7222066c32fe500",
"e7602abc26714ee890a0cf5c0c7b67e1",
"dc5d555099f64a998514ebde90eeb6df",
"ef93a2f58cc54373941f43658bb808cf",
"fea1e2327d2944859af3d91c216b9008",
"320c00a5d18c45ccae634d166f1bd810",
"6c857e69d5204cd3b7c3bf426993ad1f",
"2145e47428f1446fba3e62b3cde0a7f5",
"3d519ce3562c4e249bf392c7f43d04c0",
"cc20ffcf0c1a4656945959bf457dfd84"
]
},
"id": "u4J7IxOvOyPM",
"outputId": "925348d7-fc69-4d1b-90f1-7029426bcfcf"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4864b6a5f55340809e1e392cbeb5ca3c",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00001-of-00004.safetensors: 0%| | 0.00/4.98G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a7c77ab5f83a4319b66856b75cf04e1e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00002-of-00004.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e69661497025474b9523f5035634f788",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00003-of-00004.safetensors: 0%| | 0.00/4.92G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f4ca91e917af4a37868e416717c9e762",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model-00004-of-00004.safetensors: 0%| | 0.00/1.17G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"combined_weights = {}\n",
"\n",
"for i in range(1, 5):\n",
" weights_file = hf_hub_download(\n",
" repo_id=\"meta-llama/Llama-3.1-8B\",\n",
" filename=f\"model-0000{i}-of-00004.safetensors\",\n",
" local_dir=\"Llama-3.1-8B\"\n",
" )\n",
" current_weights = load_file(weights_file)\n",
" combined_weights.update(current_weights)\n",
"\n",
"load_weights_into_llama(model, LLAMA31_CONFIG_8B, combined_weights)\n",
"model.to(device);\n",
"del combined_weights # free up memory"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "wJFnF8ATPbtD",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "wJFnF8ATPbtD",
"outputId": "67d5cb66-3588-4fd4-ac75-39bfe3aa82d8"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output text:\n",
" Every effort has been made to trace copyright holders and to obtain their permission for the use of copyright material. The publisher apologizes for any\n"
]
}
],
"source": [
"torch.manual_seed(123)\n",
"\n",
"token_ids = generate(\n",
" model=model,\n",
" idx=text_to_token_ids(\"Every effort\", tokenizer).to(device),\n",
" max_new_tokens=25,\n",
" context_size=LLAMA31_CONFIG_8B[\"context_length\"],\n",
" top_k=1,\n",
" temperature=0.\n",
")\n",
"\n",
"print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
]
},
{
"cell_type": "markdown",
"id": "DR9NBDUjPrDp",
"metadata": {
"id": "DR9NBDUjPrDp"
},
"source": [
"&nbsp;\n",
"# Llama 3.2 1B"
]
},
{
"cell_type": "markdown",
"id": "imoxFiDzJcxk",
"metadata": {
"id": "imoxFiDzJcxk"
},
"source": [
"- As of this writing, Meta AI's latest models are the Llama 3.2 models announced [here](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)\n",
"- The code for the Llama 3.2 text model is similar to that of Llama 3.1, except that the model has shrunk in size (there is a 1B and 3B version)\n",
"- The other efficiency tweak was that they added back weight tying (a concept that was original used in the GPT-2 architecture); here, they reuse the same weight parameter values in the input (token) embedding layer and output layer\n",
"- The small model size of Llama 3.2 1B is quite convenient, since it can even run on many mobile devices\n",
"- The architectural differences between Llama 3.1 8B and Llama 3.2 1B are illustrated in the figure below"
]
},
{
"cell_type": "markdown",
"id": "OL1EoXQ6TPb7",
"metadata": {
"id": "OL1EoXQ6TPb7"
},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/llama31-to-llama32.webp?1\" width=\"700px\">"
]
},
{
"cell_type": "markdown",
"id": "K0KgjwCCJ9Fb",
"metadata": {
"id": "K0KgjwCCJ9Fb"
},
"source": [
"- As we can see based on the figure above, the main difference between the Llama 3.1 8B and Llama 3.2 1B architectures are the respective sizes\n",
"- A small additional change is an increased RoPE rescaling factor, which is reflected in the configuration file below"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "Yv_yF3NCQTBx",
"metadata": {
"id": "Yv_yF3NCQTBx"
},
"outputs": [],
"source": [
"LLAMA31_CONFIG_8B = {\n",
" \"vocab_size\": 128_256, # Vocabulary size\n",
" \"context_length\": 131_072, # NEW: Larger supported context length\n",
" \"emb_dim\": 4096, # Embedding dimension\n",
" \"n_heads\": 32, # Number of attention heads\n",
" \"n_layers\": 32, # Number of layers\n",
" \"hidden_dim\": 14_336, # Size of the intermediate dimension in FeedForward\n",
" \"n_kv_groups\": 8, # Key-Value groups for grouped-query attention\n",
" \"rope_base\": 500_000.0, # The base in RoPE's \"theta\"\n",
" \"dtype\": torch.bfloat16, # Lower-precision dtype to reduce memory usagey\n",
" \"rope_freq\": { # NEW: RoPE frequency scaling\n",
" \"factor\": 8.0,\n",
" \"low_freq_factor\": 1.0,\n",
" \"high_freq_factor\": 4.0,\n",
" \"original_context_length\": 8192,\n",
" }\n",
"}\n",
"\n",
"\n",
"LLAMA32_CONFIG_1B = {\n",
" \"vocab_size\": 128_256, # Vocabulary size\n",
" \"context_length\": 131_072, # Context length\n",
" \"emb_dim\": 2048, # NEW: Half the embedding dimension\n",
" \"n_heads\": 32, # Number of attention heads\n",
" \"n_layers\": 16, # NEW: Half the number of layers\n",
" \"hidden_dim\": 8192, # NEW: Almost half the size of the intermediate dimension in FeedForward\n",
" \"n_kv_groups\": 8, # Key-Value groups for grouped-query attention\n",
" \"rope_base\": 500_000.0, # The base in RoPE's \"theta\"\n",
" \"dtype\": torch.bfloat16, # Lower-precision dtype to reduce memory usage\n",
" \"rope_freq\": { # RoPE frequency scaling\n",
" \"factor\": 32.0, # NEW: Adjustment of the rescaling factor\n",
" \"low_freq_factor\": 1.0,\n",
" \"high_freq_factor\": 4.0,\n",
" \"original_context_length\": 8192,\n",
" }\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "Dl4_0EoJKKYv",
"metadata": {
"id": "Dl4_0EoJKKYv"
},
"source": [
"- Below, we can reuse the code from the Llama 3.1 8B section to load the Llama 3.2 1B model\n",
"- Again, since the Llama 3.2 family is distinct from the Llama 3.1 family, you'd have to go to the [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B) repository and acknowledge the license terms for your Hugging Face access token to work for the download\n",
"- Tip: For simplicity, we only load the base model below, but there's also an instruction-finetuned version you can use by replacing `\"meta-llama/Llama-3.2-1B\"` with `\"meta-llama/Llama-3.2-1B-Instruct\"`"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "tCstHgyRRD2x",
"metadata": {
"id": "tCstHgyRRD2x"
},
"outputs": [],
"source": [
"# free up memory\n",
"del model\n",
"\n",
"\n",
"gc.collect() # Run Python garbage collector\n",
"\n",
"if torch.cuda.is_available():\n",
" torch.cuda.empty_cache()"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "jt8BKAHXRCPI",
"metadata": {
"id": "jt8BKAHXRCPI"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7658da8b2a5e4273b45c35411bdba8a0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"original/tokenizer.model: 0%| | 0.00/2.18M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"tokenizer_file_path = hf_hub_download(\n",
" repo_id=\"meta-llama/Llama-3.2-1B\",\n",
" filename=\"original/tokenizer.model\",\n",
" local_dir=\"Llama-3.2-1B\"\n",
")\n",
"\n",
"tokenizer = Tokenizer(tokenizer_file_path)"
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "uf8KjasmRFSt",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "uf8KjasmRFSt",
"outputId": "4e718852-2aa1-4b5a-bec3-3d5f866a4038"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of parameters: 1,498,482,688\n",
"\n",
"Total number of unique parameters: 1,235,814,400\n"
]
}
],
"source": [
"model = Llama3Model(LLAMA32_CONFIG_1B)\n",
"\n",
"total_params = sum(p.numel() for p in model.parameters())\n",
"print(f\"Total number of parameters: {total_params:,}\")\n",
"\n",
"# Account for weight tying\n",
"total_params_normalized = total_params - model.tok_emb.weight.numel()\n",
"print(f\"\\nTotal number of unique parameters: {total_params_normalized:,}\")"
]
},
{
"cell_type": "markdown",
"id": "cc004791-9e28-4872-9ae9-fb51c6c83d7c",
"metadata": {},
"source": [
"- Alternatively, we can use more robust function that factors in the weight tying based on shared data pointers in memory as suggested in [#822](https://github.com/rasbt/LLMs-from-scratch/issues/822):"
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "7aaeb28e-62ab-4711-9f07-1b32ac9dbeba",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Total number of unique parameters: 1,498,482,688\n"
]
}
],
"source": [
"def count_unique_parameters(model):\n",
" unique_params = set()\n",
" total_unique_params = 0\n",
" \n",
" for param in model.parameters():\n",
" if param.data_ptr() not in unique_params:\n",
" total_unique_params += param.numel()\n",
" unique_params.add(param.data_ptr())\n",
" \n",
" return total_unique_params\n",
"\n",
"total_params_uniq = count_unique_parameters(model)\n",
"print(f\"\\nTotal number of unique parameters: {total_params_uniq:,}\")"
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "9FbCIYW7RIOe",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "9FbCIYW7RIOe",
"outputId": "35588405-e2e1-4871-a1db-1d4bcb852e49"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ebf98f844b6b49669d51601cbceea91e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model.safetensors: 0%| | 0.00/2.47G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model uses weight tying.\n"
]
}
],
"source": [
"weights_file = hf_hub_download(\n",
" repo_id=\"meta-llama/Llama-3.2-1B\",\n",
" filename=\"model.safetensors\",\n",
" local_dir=\"Llama-3.2-1B\"\n",
")\n",
"current_weights = load_file(weights_file)\n",
"\n",
"load_weights_into_llama(model, LLAMA32_CONFIG_1B, current_weights)\n",
"model.to(device);\n",
"del current_weights # free up memory"
]
},
{
"cell_type": "code",
"execution_count": 49,
"id": "pPp5yjir6FYJ",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "pPp5yjir6FYJ",
"outputId": "6c8e79d2-0769-43a7-93b3-f04c030e1aac"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Weight tying: True\n"
]
}
],
"source": [
"# Checks that the weight values are the same\n",
"print(\"Weight tying:\", torch.equal(model.tok_emb.weight, model.out_head.weight))"
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "b2bdebe0-d2b0-4d33-8b7e-1b4f9a02ca12",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Weight tying: True\n"
]
}
],
"source": [
"# Furthermore, check if PyTorch uses the same underlying memory\n",
"print(\"Weight tying:\", model.tok_emb.weight.data_ptr() == model.out_head.weight.data_ptr())"
]
},
{
"cell_type": "code",
"execution_count": 49,
"id": "3kh7yrw2W4qr",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3kh7yrw2W4qr",
"outputId": "b7e66a17-57ec-4b0e-c4ff-8d9a6b8e6ea5"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output text:\n",
" Every effort is made to ensure that the information on this website is accurate and up to date. However, the information is provided without any\n"
]
}
],
"source": [
"torch.manual_seed(123)\n",
"\n",
"token_ids = generate(\n",
" model=model,\n",
" idx=text_to_token_ids(\"Every effort\", tokenizer).to(device),\n",
" max_new_tokens=25,\n",
" context_size=LLAMA32_CONFIG_1B[\"context_length\"],\n",
" top_k=1,\n",
" temperature=0.\n",
")\n",
"\n",
"print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
]
},
{
"cell_type": "markdown",
"id": "VO4Qf0zyW1ZC",
"metadata": {
"id": "VO4Qf0zyW1ZC"
},
"source": [
"&nbsp;\n",
"# What's next?"
]
},
{
"cell_type": "markdown",
"id": "CjCewpo2XPAd",
"metadata": {
"id": "CjCewpo2XPAd"
},
"source": [
"- This notebook concludes the conversion from GPT to Llama 3.2\n",
"- If you are interested in a more compact, standalone notebook, which only contains the Llama 3.2 code, check out the [standalone-llama32.ipynb](standalone-llama32.ipynb) notebook"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"gpuType": "A100",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.5"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink