yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-11-17 02:25:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
LLMs-from-scratch/ch06/01_main-chapter-code/ch06.ipynb
Sebastian Raschka d6c3990c57
Training on MPS in PyTorch 2.9 (#900) 
* Training on MPS in PyTorch 2.9

* update
2025-11-01 16:55:09 -05:00

2606 lines
144 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"id": "c024bfa4-1a7a-4751-b5a1-827225a3478b",
"metadata": {
"id": "c024bfa4-1a7a-4751-b5a1-827225a3478b"
},
"source": [
"<table style=\"width:100%\">\n",
"<tr>\n",
"<td style=\"vertical-align:middle; text-align:left;\">\n",
"<font size=\"2\">\n",
"Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
"<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
"</font>\n",
"</td>\n",
"<td style=\"vertical-align:middle; text-align:left;\">\n",
"<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
"</td>\n",
"</tr>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"id": "bfabadb8-5935-45ff-b39c-db7a29012129",
"metadata": {
"id": "bfabadb8-5935-45ff-b39c-db7a29012129"
},
"source": [
"# Chapter 6: Finetuning for Text Classification"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90",
"outputId": "9495f150-9d79-4910-d6e7-6c0d9aae4a41"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"matplotlib version: 3.10.7\n",
"numpy version: 2.3.4\n",
"tiktoken version: 0.12.0\n",
"torch version: 2.9.0\n",
"tensorflow version: 2.20.0\n",
"pandas version: 2.3.3\n"
]
}
],
"source": [
"from importlib.metadata import version\n",
"\n",
"pkgs = [\"matplotlib\", # Plotting library\n",
" \"numpy\", # PyTorch & TensorFlow dependency\n",
" \"tiktoken\", # Tokenizer\n",
" \"torch\", # Deep learning library\n",
" \"tensorflow\", # For OpenAI's pretrained weights\n",
" \"pandas\" # Dataset loading\n",
" ]\n",
"for p in pkgs:\n",
" print(f\"{p} version: {version(p)}\")"
]
},
{
"cell_type": "markdown",
"id": "a445828a-ff10-4efa-9f60-a2e2aed4c87d",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/01.webp\" width=500px>"
]
},
{
"cell_type": "markdown",
"id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c",
"metadata": {
"id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c"
},
"source": [
"## 6.1 Different categories of finetuning"
]
},
{
"cell_type": "markdown",
"id": "ede3d731-5123-4f02-accd-c670ce50a5a3",
"metadata": {
"id": "ede3d731-5123-4f02-accd-c670ce50a5a3"
},
"source": [
"- No code in this section"
]
},
{
"cell_type": "markdown",
"id": "ac45579d-d485-47dc-829e-43be7f4db57b",
"metadata": {},
"source": [
"- The most common ways to finetune language models are instruction-finetuning and classification finetuning\n",
"- Instruction-finetuning, depicted below, is the topic of the next chapter"
]
},
{
"cell_type": "markdown",
"id": "6c29ef42-46d9-43d4-8bb4-94974e1665e4",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/02.webp\" width=500px>"
]
},
{
"cell_type": "markdown",
"id": "a7f60321-95b8-46a9-97bf-1d07fda2c3dd",
"metadata": {},
"source": [
"- Classification finetuning, the topic of this chapter, is a procedure you may already be familiar with if you have a background in machine learning -- it's similar to training a convolutional network to classify handwritten digits, for example\n",
"- In classification finetuning, we have a specific number of class labels (for example, \"spam\" and \"not spam\") that the model can output\n",
"- A classification finetuned model can only predict classes it has seen during training (for example, \"spam\" or \"not spam\"), whereas an instruction-finetuned model can usually perform many tasks\n",
"- We can think of a classification-finetuned model as a very specialized model; in practice, it is much easier to create a specialized model than a generalist model that performs well on many different tasks"
]
},
{
"cell_type": "markdown",
"id": "0b37a0c4-0bb1-4061-b1fe-eaa4416d52c3",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/03.webp\" width=400px>"
]
},
{
"cell_type": "markdown",
"id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf",
"metadata": {
"id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf"
},
"source": [
"## 6.2 Preparing the dataset"
]
},
{
"cell_type": "markdown",
"id": "5f628975-d2e8-4f7f-ab38-92bb868b7067",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/04.webp\" width=500px>"
]
},
{
"cell_type": "markdown",
"id": "9fbd459f-63fa-4d8c-8499-e23103156c7d",
"metadata": {
"id": "9fbd459f-63fa-4d8c-8499-e23103156c7d"
},
"source": [
"- This section prepares the dataset we use for classification finetuning\n",
"- We use a dataset consisting of spam and non-spam text messages to finetune the LLM to classify them\n",
"- First, we download and unzip the dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
"outputId": "424e4423-f623-443c-ab9e-656f9e867559"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"sms_spam_collection/SMSSpamCollection.tsv already exists. Skipping download and extraction.\n"
]
},
{
"data": {
"text/plain": [
"'\\nimport urllib.request\\nimport zipfile\\nimport os\\nfrom pathlib import Path\\n\\nurl = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\\nzip_path = \"sms_spam_collection.zip\"\\nextracted_path = \"sms_spam_collection\"\\ndata_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\\n\\ndef download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\\n if data_file_path.exists():\\n print(f\"{data_file_path} already exists. Skipping download and extraction.\")\\n return\\n\\n # Downloading the file\\n with urllib.request.urlopen(url) as response:\\n with open(zip_path, \"wb\") as out_file:\\n out_file.write(response.read())\\n\\n # Unzipping the file\\n with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\\n zip_ref.extractall(extracted_path)\\n\\n # Add .tsv file extension\\n original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\\n os.rename(original_file_path, data_file_path)\\n print(f\"File downloaded and saved as {data_file_path}\")\\n\\ntry:\\n download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\\nexcept (urllib.error.HTTPError, urllib.error.URLError, TimeoutError) as e:\\n print(f\"Primary URL failed: {e}. Trying backup URL...\")\\n url = \"https://f001.backblazeb2.com/file/LLMs-from-scratch/sms%2Bspam%2Bcollection.zip\"\\n download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\\n'"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import requests\n",
"import zipfile\n",
"import os\n",
"from pathlib import Path\n",
"\n",
"url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
"zip_path = \"sms_spam_collection.zip\"\n",
"extracted_path = \"sms_spam_collection\"\n",
"data_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\n",
"\n",
"\n",
"def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\n",
" if data_file_path.exists():\n",
" print(f\"{data_file_path} already exists. Skipping download and extraction.\")\n",
" return\n",
"\n",
" # Downloading the file\n",
" response = requests.get(url, stream=True, timeout=60)\n",
" response.raise_for_status()\n",
" with open(zip_path, \"wb\") as out_file:\n",
" for chunk in response.iter_content(chunk_size=8192):\n",
" if chunk:\n",
" out_file.write(chunk)\n",
"\n",
" # Unzipping the file\n",
" with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n",
" zip_ref.extractall(extracted_path)\n",
"\n",
" # Add .tsv file extension\n",
" original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\n",
" os.rename(original_file_path, data_file_path)\n",
" print(f\"File downloaded and saved as {data_file_path}\")\n",
"\n",
"\n",
"try:\n",
" download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
"except (requests.exceptions.RequestException, TimeoutError) as e:\n",
" print(f\"Primary URL failed: {e}. Trying backup URL...\")\n",
" url = \"https://f001.backblazeb2.com/file/LLMs-from-scratch/sms%2Bspam%2Bcollection.zip\"\n",
" download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
"\n",
"\n",
"\n",
"# The book originally used the following code below\n",
"# However, urllib uses older protocol settings that\n",
"# can cause problems for some readers using a VPN.\n",
"# The `requests` version above is more robust\n",
"# in that regard.\n",
"\n",
"\"\"\"\n",
"import urllib.request\n",
"import zipfile\n",
"import os\n",
"from pathlib import Path\n",
"\n",
"url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
"zip_path = \"sms_spam_collection.zip\"\n",
"extracted_path = \"sms_spam_collection\"\n",
"data_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\n",
"\n",
"def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\n",
" if data_file_path.exists():\n",
" print(f\"{data_file_path} already exists. Skipping download and extraction.\")\n",
" return\n",
"\n",
" # Downloading the file\n",
" with urllib.request.urlopen(url) as response:\n",
" with open(zip_path, \"wb\") as out_file:\n",
" out_file.write(response.read())\n",
"\n",
" # Unzipping the file\n",
" with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n",
" zip_ref.extractall(extracted_path)\n",
"\n",
" # Add .tsv file extension\n",
" original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\n",
" os.rename(original_file_path, data_file_path)\n",
" print(f\"File downloaded and saved as {data_file_path}\")\n",
"\n",
"try:\n",
" download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
"except (urllib.error.HTTPError, urllib.error.URLError, TimeoutError) as e:\n",
" print(f\"Primary URL failed: {e}. Trying backup URL...\")\n",
" url = \"https://f001.backblazeb2.com/file/LLMs-from-scratch/sms%2Bspam%2Bcollection.zip\"\n",
" download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1",
"metadata": {
"id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1"
},
"source": [
"- The dataset is saved as a tab-separated text file, which we can load into a pandas DataFrame"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 423
},
"id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
"outputId": "a16c5cde-d341-4887-a93f-baa9bec542ab"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Label</th>\n",
" <th>Text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>ham</td>\n",
" <td>Go until jurong point, crazy.. Available only ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>ham</td>\n",
" <td>Ok lar... Joking wif u oni...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>spam</td>\n",
" <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>ham</td>\n",
" <td>U dun say so early hor... U c already then say...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>ham</td>\n",
" <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5567</th>\n",
" <td>spam</td>\n",
" <td>This is the 2nd time we have tried 2 contact u...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5568</th>\n",
" <td>ham</td>\n",
" <td>Will ü b going to esplanade fr home?</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5569</th>\n",
" <td>ham</td>\n",
" <td>Pity, * was in mood for that. So...any other s...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5570</th>\n",
" <td>ham</td>\n",
" <td>The guy did some bitching but I acted like i'd...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5571</th>\n",
" <td>ham</td>\n",
" <td>Rofl. Its true to its name</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5572 rows × 2 columns</p>\n",
"</div>"
],
"text/plain": [
" Label Text\n",
"0 ham Go until jurong point, crazy.. Available only ...\n",
"1 ham Ok lar... Joking wif u oni...\n",
"2 spam Free entry in 2 a wkly comp to win FA Cup fina...\n",
"3 ham U dun say so early hor... U c already then say...\n",
"4 ham Nah I don't think he goes to usf, he lives aro...\n",
"... ... ...\n",
"5567 spam This is the 2nd time we have tried 2 contact u...\n",
"5568 ham Will ü b going to esplanade fr home?\n",
"5569 ham Pity, * was in mood for that. So...any other s...\n",
"5570 ham The guy did some bitching but I acted like i'd...\n",
"5571 ham Rofl. Its true to its name\n",
"\n",
"[5572 rows x 2 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"df = pd.read_csv(data_file_path, sep=\"\\t\", header=None, names=[\"Label\", \"Text\"])\n",
"df"
]
},
{
"cell_type": "markdown",
"id": "e7b6e631-4f0b-4aab-82b9-8898e6663109",
"metadata": {
"id": "e7b6e631-4f0b-4aab-82b9-8898e6663109"
},
"source": [
"- When we check the class distribution, we see that the data contains \"ham\" (i.e., \"not spam\") much more frequently than \"spam\""
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "495a5280-9d7c-41d4-9719-64ab99056d4c",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "495a5280-9d7c-41d4-9719-64ab99056d4c",
"outputId": "761e0482-43ba-4f46-f4b7-6774dae51b38"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Label\n",
"ham 4825\n",
"spam 747\n",
"Name: count, dtype: int64\n"
]
}
],
"source": [
"print(df[\"Label\"].value_counts())"
]
},
{
"cell_type": "markdown",
"id": "f773f054-0bdc-4aad-bbf6-397621bf63db",
"metadata": {
"id": "f773f054-0bdc-4aad-bbf6-397621bf63db"
},
"source": [
"- For simplicity, and because we prefer a small dataset for educational purposes anyway (it will make it possible to finetune the LLM faster), we subsample (undersample) the dataset so that it contains 747 instances from each class\n",
"- (Next to undersampling, there are several other ways to deal with class balances, but they are out of the scope of a book on LLMs; you can find examples and more information in the [`imbalanced-learn` user guide](https://imbalanced-learn.org/stable/user_guide.html))"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "7be4a0a2-9704-4a96-b38f-240339818688",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7be4a0a2-9704-4a96-b38f-240339818688",
"outputId": "396dc415-cb71-4a88-e85d-d88201c6d73f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Label\n",
"ham 747\n",
"spam 747\n",
"Name: count, dtype: int64\n"
]
}
],
"source": [
"def create_balanced_dataset(df):\n",
" \n",
" # Count the instances of \"spam\"\n",
" num_spam = df[df[\"Label\"] == \"spam\"].shape[0]\n",
" \n",
" # Randomly sample \"ham\" instances to match the number of \"spam\" instances\n",
" ham_subset = df[df[\"Label\"] == \"ham\"].sample(num_spam, random_state=123)\n",
" \n",
" # Combine ham \"subset\" with \"spam\"\n",
" balanced_df = pd.concat([ham_subset, df[df[\"Label\"] == \"spam\"]])\n",
"\n",
" return balanced_df\n",
"\n",
"\n",
"balanced_df = create_balanced_dataset(df)\n",
"print(balanced_df[\"Label\"].value_counts())"
]
},
{
"cell_type": "markdown",
"id": "d3fd2f5a-06d8-4d30-a2e3-230b86c559d6",
"metadata": {
"id": "d3fd2f5a-06d8-4d30-a2e3-230b86c559d6"
},
"source": [
"- Next, we change the string class labels \"ham\" and \"spam\" into integer class labels 0 and 1:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "c1b10c3d-5d57-42d0-8de8-cf80a06f5ffd",
"metadata": {
"id": "c1b10c3d-5d57-42d0-8de8-cf80a06f5ffd"
},
"outputs": [],
"source": [
"balanced_df[\"Label\"] = balanced_df[\"Label\"].map({\"ham\": 0, \"spam\": 1}) "
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "e6f7f062-ef4e-4020-8275-71990cab4414",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Label</th>\n",
" <th>Text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>4307</th>\n",
" <td>0</td>\n",
" <td>Awww dat is sweet! We can think of something t...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4138</th>\n",
" <td>0</td>\n",
" <td>Just got to &amp;lt;#&amp;gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4831</th>\n",
" <td>0</td>\n",
" <td>The word \"Checkmate\" in chess comes from the P...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4461</th>\n",
" <td>0</td>\n",
" <td>This is wishing you a great day. Moji told me ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5440</th>\n",
" <td>0</td>\n",
" <td>Thank you. do you generally date the brothas?</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5537</th>\n",
" <td>1</td>\n",
" <td>Want explicit SEX in 30 secs? Ring 02073162414...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5540</th>\n",
" <td>1</td>\n",
" <td>ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5547</th>\n",
" <td>1</td>\n",
" <td>Had your contract mobile 11 Mnths? Latest Moto...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5566</th>\n",
" <td>1</td>\n",
" <td>REMINDER FROM O2: To get 2.50 pounds free call...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5567</th>\n",
" <td>1</td>\n",
" <td>This is the 2nd time we have tried 2 contact u...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>1494 rows × 2 columns</p>\n",
"</div>"
],
"text/plain": [
" Label Text\n",
"4307 0 Awww dat is sweet! We can think of something t...\n",
"4138 0 Just got to &lt;#&gt;\n",
"4831 0 The word \"Checkmate\" in chess comes from the P...\n",
"4461 0 This is wishing you a great day. Moji told me ...\n",
"5440 0 Thank you. do you generally date the brothas?\n",
"... ... ...\n",
"5537 1 Want explicit SEX in 30 secs? Ring 02073162414...\n",
"5540 1 ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...\n",
"5547 1 Had your contract mobile 11 Mnths? Latest Moto...\n",
"5566 1 REMINDER FROM O2: To get 2.50 pounds free call...\n",
"5567 1 This is the 2nd time we have tried 2 contact u...\n",
"\n",
"[1494 rows x 2 columns]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"balanced_df"
]
},
{
"cell_type": "markdown",
"id": "5715e685-35b4-4b45-a86c-8a8694de9d6f",
"metadata": {
"id": "5715e685-35b4-4b45-a86c-8a8694de9d6f"
},
"source": [
"- Let's now define a function that randomly divides the dataset into training, validation, and test subsets"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "uQl0Psdmx15D",
"metadata": {
"id": "uQl0Psdmx15D"
},
"outputs": [],
"source": [
"def random_split(df, train_frac, validation_frac):\n",
" # Shuffle the entire DataFrame\n",
" df = df.sample(frac=1, random_state=123).reset_index(drop=True)\n",
"\n",
" # Calculate split indices\n",
" train_end = int(len(df) * train_frac)\n",
" validation_end = train_end + int(len(df) * validation_frac)\n",
"\n",
" # Split the DataFrame\n",
" train_df = df[:train_end]\n",
" validation_df = df[train_end:validation_end]\n",
" test_df = df[validation_end:]\n",
"\n",
" return train_df, validation_df, test_df\n",
"\n",
"train_df, validation_df, test_df = random_split(balanced_df, 0.7, 0.1)\n",
"# Test size is implied to be 0.2 as the remainder\n",
"\n",
"train_df.to_csv(\"train.csv\", index=None)\n",
"validation_df.to_csv(\"validation.csv\", index=None)\n",
"test_df.to_csv(\"test.csv\", index=None)"
]
},
{
"cell_type": "markdown",
"id": "a8d7a0c5-1d5f-458a-b685-3f49520b0094",
"metadata": {},
"source": [
"## 6.3 Creating data loaders"
]
},
{
"cell_type": "markdown",
"id": "7126108a-75e7-4862-b0fb-cbf59a18bb6c",
"metadata": {
"id": "7126108a-75e7-4862-b0fb-cbf59a18bb6c"
},
"source": [
"- Note that the text messages have different lengths; if we want to combine multiple training examples in a batch, we have to either\n",
" 1. truncate all messages to the length of the shortest message in the dataset or batch\n",
" 2. pad all messages to the length of the longest message in the dataset or batch\n",
"\n",
"- We choose option 2 and pad all messages to the longest message in the dataset\n",
"- For that, we use `<|endoftext|>` as a padding token, as discussed in chapter 2"
]
},
{
"cell_type": "markdown",
"id": "0829f33f-1428-4f22-9886-7fee633b3666",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/06.webp\" width=500px>"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
"outputId": "b5b48439-32c8-4b37-cca2-c9dc8fa86563"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[50256]\n"
]
}
],
"source": [
"import tiktoken\n",
"\n",
"tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
"print(tokenizer.encode(\"<|endoftext|>\", allowed_special={\"<|endoftext|>\"}))"
]
},
{
"cell_type": "markdown",
"id": "04f582ff-68bf-450e-bd87-5fb61afe431c",
"metadata": {
"id": "04f582ff-68bf-450e-bd87-5fb61afe431c"
},
"source": [
"- The `SpamDataset` class below identifies the longest sequence in the training dataset and adds the padding token to the others to match that sequence length"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "d7791b52-af18-4ac4-afa9-b921068e383e",
"metadata": {
"id": "d7791b52-af18-4ac4-afa9-b921068e383e"
},
"outputs": [],
"source": [
"import torch\n",
"from torch.utils.data import Dataset\n",
"\n",
"\n",
"class SpamDataset(Dataset):\n",
" def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):\n",
" self.data = pd.read_csv(csv_file)\n",
"\n",
" # Pre-tokenize texts\n",
" self.encoded_texts = [\n",
" tokenizer.encode(text) for text in self.data[\"Text\"]\n",
" ]\n",
"\n",
" if max_length is None:\n",
" self.max_length = self._longest_encoded_length()\n",
" else:\n",
" self.max_length = max_length\n",
" # Truncate sequences if they are longer than max_length\n",
" self.encoded_texts = [\n",
" encoded_text[:self.max_length]\n",
" for encoded_text in self.encoded_texts\n",
" ]\n",
"\n",
" # Pad sequences to the longest sequence\n",
" self.encoded_texts = [\n",
" encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))\n",
" for encoded_text in self.encoded_texts\n",
" ]\n",
"\n",
" def __getitem__(self, index):\n",
" encoded = self.encoded_texts[index]\n",
" label = self.data.iloc[index][\"Label\"]\n",
" return (\n",
" torch.tensor(encoded, dtype=torch.long),\n",
" torch.tensor(label, dtype=torch.long)\n",
" )\n",
"\n",
" def __len__(self):\n",
" return len(self.data)\n",
"\n",
" def _longest_encoded_length(self):\n",
" max_length = 0\n",
" for encoded_text in self.encoded_texts:\n",
" encoded_length = len(encoded_text)\n",
" if encoded_length > max_length:\n",
" max_length = encoded_length\n",
" return max_length\n",
" # Note: A more pythonic version to implement this method\n",
" # is the following, which is also used in the next chapter:\n",
" # return max(len(encoded_text) for encoded_text in self.encoded_texts)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "uzj85f8ou82h",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "uzj85f8ou82h",
"outputId": "d08f1cf0-c24d-445f-a3f8-793532c3716f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"120\n"
]
}
],
"source": [
"train_dataset = SpamDataset(\n",
" csv_file=\"train.csv\",\n",
" max_length=None,\n",
" tokenizer=tokenizer\n",
")\n",
"\n",
"print(train_dataset.max_length)"
]
},
{
"cell_type": "markdown",
"id": "15bdd932-97eb-4b88-9cf9-d766ea4c3a60",
"metadata": {},
"source": [
"- We also pad the validation and test set to the longest training sequence\n",
"- Note that validation and test set samples that are longer than the longest training example are being truncated via `encoded_text[:self.max_length]` in the `SpamDataset` code\n",
"- This behavior is entirely optional, and it would also work well if we set `max_length=None` in both the validation and test set cases"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e",
"metadata": {
"id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e"
},
"outputs": [],
"source": [
"val_dataset = SpamDataset(\n",
" csv_file=\"validation.csv\",\n",
" max_length=train_dataset.max_length,\n",
" tokenizer=tokenizer\n",
")\n",
"test_dataset = SpamDataset(\n",
" csv_file=\"test.csv\",\n",
" max_length=train_dataset.max_length,\n",
" tokenizer=tokenizer\n",
")"
]
},
{
"cell_type": "markdown",
"id": "20170d89-85a0-4844-9887-832f5d23432a",
"metadata": {},
"source": [
"- Next, we use the dataset to instantiate the data loaders, which is similar to creating the data loaders in previous chapters"
]
},
{
"cell_type": "markdown",
"id": "64bcc349-205f-48f8-9655-95ff21f5e72f",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/07.webp\" width=500px>"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
"outputId": "3266c410-4fdb-4a8c-a142-7f707e2525ab"
},
"outputs": [],
"source": [
"from torch.utils.data import DataLoader\n",
"\n",
"num_workers = 0\n",
"batch_size = 8\n",
"\n",
"torch.manual_seed(123)\n",
"\n",
"train_loader = DataLoader(\n",
" dataset=train_dataset,\n",
" batch_size=batch_size,\n",
" shuffle=True,\n",
" num_workers=num_workers,\n",
" drop_last=True,\n",
")\n",
"\n",
"val_loader = DataLoader(\n",
" dataset=val_dataset,\n",
" batch_size=batch_size,\n",
" num_workers=num_workers,\n",
" drop_last=False,\n",
")\n",
"\n",
"test_loader = DataLoader(\n",
" dataset=test_dataset,\n",
" batch_size=batch_size,\n",
" num_workers=num_workers,\n",
" drop_last=False,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "ab7335db-e0bb-4e27-80c5-eea11e593a57",
"metadata": {},
"source": [
"- As a verification step, we iterate through the data loaders and ensure that the batches contain 8 training examples each, where each training example consists of 120 tokens"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "4dee6882-4c3a-4964-af15-fa31f86ad047",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train loader:\n",
"Input batch dimensions: torch.Size([8, 120])\n",
"Label batch dimensions torch.Size([8])\n"
]
}
],
"source": [
"print(\"Train loader:\")\n",
"for input_batch, target_batch in train_loader:\n",
" pass\n",
"\n",
"print(\"Input batch dimensions:\", input_batch.shape)\n",
"print(\"Label batch dimensions\", target_batch.shape)"
]
},
{
"cell_type": "markdown",
"id": "5cdd7947-7039-49bf-8a5e-c0a2f4281ca1",
"metadata": {},
"source": [
"- Lastly, let's print the total number of batches in each dataset"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "IZfw-TYD2zTj",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "IZfw-TYD2zTj",
"outputId": "6934bbf2-9797-4fbe-d26b-1a246e18c2fb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"130 training batches\n",
"19 validation batches\n",
"38 test batches\n"
]
}
],
"source": [
"print(f\"{len(train_loader)} training batches\")\n",
"print(f\"{len(val_loader)} validation batches\")\n",
"print(f\"{len(test_loader)} test batches\")"
]
},
{
"cell_type": "markdown",
"id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c",
"metadata": {
"id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c"
},
"source": [
"## 6.4 Initializing a model with pretrained weights"
]
},
{
"cell_type": "markdown",
"id": "97e1af8b-8bd1-4b44-8b8b-dc031496e208",
"metadata": {},
"source": [
"- In this section, we initialize the pretrained model we worked with in the previous chapter\n",
"\n",
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/08.webp\" width=500px>"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "2992d779-f9fb-4812-a117-553eb790a5a9",
"metadata": {
"id": "2992d779-f9fb-4812-a117-553eb790a5a9"
},
"outputs": [],
"source": [
"CHOOSE_MODEL = \"gpt2-small (124M)\"\n",
"INPUT_PROMPT = \"Every effort moves\"\n",
"\n",
"BASE_CONFIG = {\n",
" \"vocab_size\": 50257, # Vocabulary size\n",
" \"context_length\": 1024, # Context length\n",
" \"drop_rate\": 0.0, # Dropout rate\n",
" \"qkv_bias\": True # Query-key-value bias\n",
"}\n",
"\n",
"model_configs = {\n",
" \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
" \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
" \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
" \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
"}\n",
"\n",
"BASE_CONFIG.update(model_configs[CHOOSE_MODEL])\n",
"\n",
"assert train_dataset.max_length <= BASE_CONFIG[\"context_length\"], (\n",
" f\"Dataset length {train_dataset.max_length} exceeds model's context \"\n",
" f\"length {BASE_CONFIG['context_length']}. Reinitialize data sets with \"\n",
" f\"`max_length={BASE_CONFIG['context_length']}`\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "022a649a-44f5-466c-8a8e-326c063384f5",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "022a649a-44f5-466c-8a8e-326c063384f5",
"outputId": "7091e401-8442-4f47-a1d9-ecb42a1ef930"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"File already exists and is up-to-date: gpt2/124M/checkpoint\n",
"File already exists and is up-to-date: gpt2/124M/encoder.json\n",
"File already exists and is up-to-date: gpt2/124M/hparams.json\n",
"File already exists and is up-to-date: gpt2/124M/model.ckpt.data-00000-of-00001\n",
"File already exists and is up-to-date: gpt2/124M/model.ckpt.index\n",
"File already exists and is up-to-date: gpt2/124M/model.ckpt.meta\n",
"File already exists and is up-to-date: gpt2/124M/vocab.bpe\n"
]
}
],
"source": [
"from gpt_download import download_and_load_gpt2\n",
"from previous_chapters import GPTModel, load_weights_into_gpt\n",
"# If the `previous_chapters.py` file is not available locally,\n",
"# you can import it from the `llms-from-scratch` PyPI package.\n",
"# For details, see: https://github.com/rasbt/LLMs-from-scratch/tree/main/pkg\n",
"# E.g.,\n",
"# from llms_from_scratch.ch04 import GPTModel\n",
"# from llms_from_scratch.ch05 import download_and_load_gpt2, load_weights_into_gpt\n",
"\n",
"model_size = CHOOSE_MODEL.split(\" \")[-1].lstrip(\"(\").rstrip(\")\")\n",
"settings, params = download_and_load_gpt2(model_size=model_size, models_dir=\"gpt2\")\n",
"\n",
"model = GPTModel(BASE_CONFIG)\n",
"load_weights_into_gpt(model, params)\n",
"model.eval();"
]
},
{
"cell_type": "markdown",
"id": "ab8e056c-abe0-415f-b34d-df686204259e",
"metadata": {},
"source": [
"- To ensure that the model was loaded correctly, let's double-check that it generates coherent text"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "d8ac25ff-74b1-4149-8dc5-4c429d464330",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Every effort moves you forward.\n",
"\n",
"The first step is to understand the importance of your work\n"
]
}
],
"source": [
"from previous_chapters import (\n",
" generate_text_simple,\n",
" text_to_token_ids,\n",
" token_ids_to_text\n",
")\n",
"\n",
"# Alternatively:\n",
"# from llms_from_scratch.ch05 import (\n",
"# generate_text_simple,\n",
"# text_to_token_ids,\n",
"# token_ids_to_text\n",
"# )\n",
"\n",
"\n",
"text_1 = \"Every effort moves you\"\n",
"\n",
"token_ids = generate_text_simple(\n",
" model=model,\n",
" idx=text_to_token_ids(text_1, tokenizer),\n",
" max_new_tokens=15,\n",
" context_size=BASE_CONFIG[\"context_length\"]\n",
")\n",
"\n",
"print(token_ids_to_text(token_ids, tokenizer))"
]
},
{
"cell_type": "markdown",
"id": "69162550-6a02-4ece-8db1-06c71d61946f",
"metadata": {},
"source": [
"- Before we finetune the model as a classifier, let's see if the model can perhaps already classify spam messages via prompting"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "94224aa9-c95a-4f8a-a420-76d01e3a800c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Is the following text 'spam'? Answer with 'yes' or 'no': 'You are a winner you have been specially selected to receive $1000 cash or a $2000 award.'\n",
"\n",
"The following text 'spam'? Answer with 'yes' or 'no': 'You are a winner\n"
]
}
],
"source": [
"text_2 = (\n",
" \"Is the following text 'spam'? Answer with 'yes' or 'no':\"\n",
" \" 'You are a winner you have been specially\"\n",
" \" selected to receive $1000 cash or a $2000 award.'\"\n",
")\n",
"\n",
"token_ids = generate_text_simple(\n",
" model=model,\n",
" idx=text_to_token_ids(text_2, tokenizer),\n",
" max_new_tokens=23,\n",
" context_size=BASE_CONFIG[\"context_length\"]\n",
")\n",
"\n",
"print(token_ids_to_text(token_ids, tokenizer))"
]
},
{
"cell_type": "markdown",
"id": "1ce39ed0-2c77-410d-8392-dd15d4b22016",
"metadata": {},
"source": [
"- As we can see, the model is not very good at following instructions\n",
"- This is expected, since it has only been pretrained and not instruction-finetuned (instruction finetuning will be covered in the next chapter)"
]
},
{
"cell_type": "markdown",
"id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522",
"metadata": {
"id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522"
},
"source": [
"## 6.5 Adding a classification head"
]
},
{
"cell_type": "markdown",
"id": "d6e9d66f-76b2-40fc-9ec5-3f972a8db9c0",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/09.webp\" width=500px>"
]
},
{
"cell_type": "markdown",
"id": "217bac05-78df-4412-bd80-612f8061c01d",
"metadata": {},
"source": [
"- In this section, we are modifying the pretrained LLM to make it ready for classification finetuning\n",
"- Let's take a look at the model architecture first"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "b23aff91-6bd0-48da-88f6-353657e6c981",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1d8f7a01-b7c0-48d4-b1e7-8c12cc7ad932",
"outputId": "b6a5b9b5-a92f-498f-d7cb-b58dd99e4497"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GPTModel(\n",
" (tok_emb): Embedding(50257, 768)\n",
" (pos_emb): Embedding(1024, 768)\n",
" (drop_emb): Dropout(p=0.0, inplace=False)\n",
" (trf_blocks): Sequential(\n",
" (0): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (1): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (2): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (3): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (4): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (5): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (6): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (7): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (8): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (9): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (10): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (11): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" )\n",
" (final_norm): LayerNorm()\n",
" (out_head): Linear(in_features=768, out_features=50257, bias=False)\n",
")\n"
]
}
],
"source": [
"print(model)"
]
},
{
"cell_type": "markdown",
"id": "3f640a76-dd00-4769-9bc8-1aed0cec330d",
"metadata": {},
"source": [
"- Above, we can see the architecture we implemented in chapter 4 neatly laid out\n",
"- The goal is to replace and finetune the output layer\n",
"- To achieve this, we first freeze the model, meaning that we make all layers non-trainable"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "fkMWFl-0etea",
"metadata": {
"id": "fkMWFl-0etea"
},
"outputs": [],
"source": [
"for param in model.parameters():\n",
" param.requires_grad = False"
]
},
{
"cell_type": "markdown",
"id": "72155f83-87d9-476a-a978-a15aa2d44147",
"metadata": {},
"source": [
"- Then, we replace the output layer (`model.out_head`), which originally maps the layer inputs to 50,257 dimensions (the size of the vocabulary)\n",
"- Since we finetune the model for binary classification (predicting 2 classes, \"spam\" and \"not spam\"), we can replace the output layer as shown below, which will be trainable by default\n",
"- Note that we use `BASE_CONFIG[\"emb_dim\"]` (which is equal to 768 in the `\"gpt2-small (124M)\"` model) to keep the code below more general"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "7e759fa0-0f69-41be-b576-17e5f20e04cb",
"metadata": {},
"outputs": [],
"source": [
"torch.manual_seed(123)\n",
"\n",
"num_classes = 2\n",
"model.out_head = torch.nn.Linear(in_features=BASE_CONFIG[\"emb_dim\"], out_features=num_classes)"
]
},
{
"cell_type": "markdown",
"id": "30be5475-ae77-4f97-8f3e-dec462b1339f",
"metadata": {},
"source": [
"- Technically, it's sufficient to only train the output layer\n",
"- However, as I found in [Finetuning Large Language Models](https://magazine.sebastianraschka.com/p/finetuning-large-language-models), experiments show that finetuning additional layers can noticeably improve the performance\n",
"- So, we are also making the last transformer block and the final `LayerNorm` module connecting the last transformer block to the output layer trainable"
]
},
{
"cell_type": "markdown",
"id": "0be7c1eb-c46c-4065-8525-eea1b8c66d10",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/10.webp\" width=500px>"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7",
"metadata": {
"id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7"
},
"outputs": [],
"source": [
"for param in model.trf_blocks[-1].parameters():\n",
" param.requires_grad = True\n",
"\n",
"for param in model.final_norm.parameters():\n",
" param.requires_grad = True"
]
},
{
"cell_type": "markdown",
"id": "f012b899-8284-4d3a-97c0-8a48eb33ba2e",
"metadata": {},
"source": [
"- We can still use this model similar to before in previous chapters\n",
"- For example, let's feed it some text input"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
"outputId": "27e041b1-d731-48a1-cf60-f22d4565304e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Inputs: tensor([[5211, 345, 423, 640]])\n",
"Inputs dimensions: torch.Size([1, 4])\n"
]
}
],
"source": [
"inputs = tokenizer.encode(\"Do you have time\")\n",
"inputs = torch.tensor(inputs).unsqueeze(0)\n",
"print(\"Inputs:\", inputs)\n",
"print(\"Inputs dimensions:\", inputs.shape) # shape: (batch_size, num_tokens)"
]
},
{
"cell_type": "markdown",
"id": "fbbf8481-772d-467b-851c-a62b86d0cb1b",
"metadata": {},
"source": [
"- What's different compared to previous chapters is that it now has two output dimensions instead of 50,257"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
"outputId": "9cae7448-253d-4776-973e-0af190b06354"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Outputs:\n",
" tensor([[[-1.5854, 0.9904],\n",
" [-3.7235, 7.4548],\n",
" [-2.2661, 6.6049],\n",
" [-3.5983, 3.9902]]])\n",
"Outputs dimensions: torch.Size([1, 4, 2])\n"
]
}
],
"source": [
"with torch.no_grad():\n",
" outputs = model(inputs)\n",
"\n",
"print(\"Outputs:\\n\", outputs)\n",
"print(\"Outputs dimensions:\", outputs.shape) # shape: (batch_size, num_tokens, num_classes)"
]
},
{
"cell_type": "markdown",
"id": "75430a01-ef9c-426a-aca0-664689c4f461",
"metadata": {},
"source": [
"- As discussed in previous chapters, for each input token, there's one output vector\n",
"- Since we fed the model a text sample with 4 input tokens, the output consists of 4 2-dimensional output vectors above"
]
},
{
"cell_type": "markdown",
"id": "7df9144f-6817-4be4-8d4b-5d4dadfe4a9b",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/11.webp\" width=500px>"
]
},
{
"cell_type": "markdown",
"id": "e3bb8616-c791-4f5c-bac0-5302f663e46a",
"metadata": {},
"source": [
"- In chapter 3, we discussed the attention mechanism, which connects each input token to each other input token\n",
"- In chapter 3, we then also introduced the causal attention mask that is used in GPT-like models; this causal mask lets a current token only attend to the current and previous token positions\n",
"- Based on this causal attention mechanism, the 4th (last) token contains the most information among all tokens because it's the only token that includes information about all other tokens\n",
"- Hence, we are particularly interested in this last token, which we will finetune for the spam classification task"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
"outputId": "e79eb155-fa1f-46ed-ff8c-d828c3a3fabd"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Last output token: tensor([[-3.5983, 3.9902]])\n"
]
}
],
"source": [
"print(\"Last output token:\", outputs[:, -1, :])"
]
},
{
"cell_type": "markdown",
"id": "8df08ae0-e664-4670-b7c5-8a2280d9b41b",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/12.webp\" width=200px>"
]
},
{
"cell_type": "markdown",
"id": "32aa4aef-e1e9-491b-9adf-5aa973e59b8c",
"metadata": {},
"source": [
"## 6.6 Calculating the classification loss and accuracy"
]
},
{
"cell_type": "markdown",
"id": "669e1fd1-ace8-44b4-b438-185ed0ba8b33",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/13.webp\" width=300px>"
]
},
{
"cell_type": "markdown",
"id": "7a7df4ee-0a34-4a4d-896d-affbbf81e0b3",
"metadata": {},
"source": [
"- Before explaining the loss calculation, let's have a brief look at how the model outputs are turned into class labels"
]
},
{
"cell_type": "markdown",
"id": "557996dd-4c6b-49c4-ab83-f60ef7e1d69e",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/14.webp\" width=600px>"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "c77faab1-3461-4118-866a-6171f2b89aa0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Last output token: tensor([[-3.5983, 3.9902]])\n"
]
}
],
"source": [
"print(\"Last output token:\", outputs[:, -1, :])"
]
},
{
"cell_type": "markdown",
"id": "7edd71fa-628a-4d00-b81d-6d8bcb2c341d",
"metadata": {},
"source": [
"- Similar to chapter 5, we convert the outputs (logits) into probability scores via the `softmax` function and then obtain the index position of the largest probability value via the `argmax` function"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "b81efa92-9be1-4b9e-8790-ce1fc7b17f01",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Class label: 1\n"
]
}
],
"source": [
"probas = torch.softmax(outputs[:, -1, :], dim=-1)\n",
"label = torch.argmax(probas)\n",
"print(\"Class label:\", label.item())"
]
},
{
"cell_type": "markdown",
"id": "414a6f02-307e-4147-a416-14d115bf8179",
"metadata": {},
"source": [
"- Note that the softmax function is optional here, as explained in chapter 5, because the largest outputs correspond to the largest probability scores"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "f9f9ad66-4969-4501-8239-3ccdb37e71a2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Class label: 1\n"
]
}
],
"source": [
"logits = outputs[:, -1, :]\n",
"label = torch.argmax(logits)\n",
"print(\"Class label:\", label.item())"
]
},
{
"cell_type": "markdown",
"id": "dcb20d3a-cbba-4ab1-8584-d94e16589505",
"metadata": {},
"source": [
"- We can apply this concept to calculate the so-called classification accuracy, which computes the percentage of correct predictions in a given dataset\n",
"- To calculate the classification accuracy, we can apply the preceding `argmax`-based prediction code to all examples in a dataset and calculate the fraction of correct predictions as follows:"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "3ecf9572-aed0-4a21-9c3b-7f9f2aec5f23",
"metadata": {},
"outputs": [],
"source": [
"def calc_accuracy_loader(data_loader, model, device, num_batches=None):\n",
" model.eval()\n",
" correct_predictions, num_examples = 0, 0\n",
"\n",
" if num_batches is None:\n",
" num_batches = len(data_loader)\n",
" else:\n",
" num_batches = min(num_batches, len(data_loader))\n",
" for i, (input_batch, target_batch) in enumerate(data_loader):\n",
" if i < num_batches:\n",
" input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
"\n",
" with torch.no_grad():\n",
" logits = model(input_batch)[:, -1, :] # Logits of last output token\n",
" predicted_labels = torch.argmax(logits, dim=-1)\n",
"\n",
" num_examples += predicted_labels.shape[0]\n",
" correct_predictions += (predicted_labels == target_batch).sum().item()\n",
" else:\n",
" break\n",
" return correct_predictions / num_examples"
]
},
{
"cell_type": "markdown",
"id": "7165fe46-a284-410b-957f-7524877d1a1a",
"metadata": {},
"source": [
"- Let's apply the function to calculate the classification accuracies for the different datasets:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "390e5255-8427-488c-adef-e1c10ab4fb26",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Device: mps\n",
"Training accuracy: 46.25%\n",
"Validation accuracy: 45.00%\n",
"Test accuracy: 48.75%\n"
]
}
],
"source": [
"if torch.cuda.is_available():\n",
" device = torch.device(\"cuda\")\n",
"elif torch.backends.mps.is_available():\n",
" # Use PyTorch 2.9 or newer for stable mps results\n",
" major, minor = map(int, torch.__version__.split(\".\")[:2])\n",
" if (major, minor) >= (2, 9):\n",
" device = torch.device(\"mps\")\n",
"else:\n",
" device = torch.device(\"cpu\")\n",
"\n",
"print(\"Device:\", device)\n",
"\n",
"model.to(device) # no assignment model = model.to(device) necessary for nn.Module classes\n",
"\n",
"torch.manual_seed(123) # For reproducibility due to the shuffling in the training data loader\n",
"\n",
"train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=10)\n",
"val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=10)\n",
"test_accuracy = calc_accuracy_loader(test_loader, model, device, num_batches=10)\n",
"\n",
"print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
"print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
"print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
]
},
{
"cell_type": "markdown",
"id": "30345e2a-afed-4d22-9486-f4010f90a871",
"metadata": {},
"source": [
"- As we can see, the prediction accuracies are not very good, since we haven't finetuned the model, yet"
]
},
{
"cell_type": "markdown",
"id": "4f4a9d15-8fc7-48a2-8734-d92a2f265328",
"metadata": {},
"source": [
"- Before we can start finetuning (/training), we first have to define the loss function we want to optimize during training\n",
"- The goal is to maximize the spam classification accuracy of the model; however, classification accuracy is not a differentiable function\n",
"- Hence, instead, we minimize the cross-entropy loss as a proxy for maximizing the classification accuracy (you can learn more about this topic in lecture 8 of my freely available [Introduction to Deep Learning](https://sebastianraschka.com/blog/2021/dl-course.html#l08-multinomial-logistic-regression--softmax-regression) class)\n",
"\n",
"- The `calc_loss_batch` function is the same here as in chapter 5, except that we are only interested in optimizing the last token `model(input_batch)[:, -1, :]` instead of all tokens `model(input_batch)`"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "2f1e9547-806c-41a9-8aba-3b2822baabe4",
"metadata": {
"id": "2f1e9547-806c-41a9-8aba-3b2822baabe4"
},
"outputs": [],
"source": [
"def calc_loss_batch(input_batch, target_batch, model, device):\n",
" input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
" logits = model(input_batch)[:, -1, :] # Logits of last output token\n",
" loss = torch.nn.functional.cross_entropy(logits, target_batch)\n",
" return loss"
]
},
{
"cell_type": "markdown",
"id": "a013aab9-f854-4866-ad55-5b8350adb50a",
"metadata": {},
"source": [
"The `calc_loss_loader` is exactly the same as in chapter 5"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "b7b83e10-5720-45e7-ac5e-369417ca846b",
"metadata": {},
"outputs": [],
"source": [
"# Same as in chapter 5\n",
"def calc_loss_loader(data_loader, model, device, num_batches=None):\n",
" total_loss = 0.\n",
" if len(data_loader) == 0:\n",
" return float(\"nan\")\n",
" elif num_batches is None:\n",
" num_batches = len(data_loader)\n",
" else:\n",
" # Reduce the number of batches to match the total number of batches in the data loader\n",
" # if num_batches exceeds the number of batches in the data loader\n",
" num_batches = min(num_batches, len(data_loader))\n",
" for i, (input_batch, target_batch) in enumerate(data_loader):\n",
" if i < num_batches:\n",
" loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
" total_loss += loss.item()\n",
" else:\n",
" break\n",
" return total_loss / num_batches"
]
},
{
"cell_type": "markdown",
"id": "56826ecd-6e74-40e6-b772-d3541e585067",
"metadata": {},
"source": [
"- Using the `calc_closs_loader`, we compute the initial training, validation, and test set losses before we start training"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
"outputId": "49df8648-9e38-4314-854d-9faacd1b2e89"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training loss: 2.453\n",
"Validation loss: 2.583\n",
"Test loss: 2.322\n"
]
}
],
"source": [
"with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet\n",
" train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)\n",
" val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)\n",
" test_loss = calc_loss_loader(test_loader, model, device, num_batches=5)\n",
"\n",
"print(f\"Training loss: {train_loss:.3f}\")\n",
"print(f\"Validation loss: {val_loss:.3f}\")\n",
"print(f\"Test loss: {test_loss:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "e04b980b-e583-4f62-84a0-4edafaf99d5d",
"metadata": {},
"source": [
"- In the next section, we train the model to improve the loss values and consequently the classification accuracy"
]
},
{
"cell_type": "markdown",
"id": "456ae0fd-6261-42b4-ab6a-d24289953083",
"metadata": {
"id": "456ae0fd-6261-42b4-ab6a-d24289953083"
},
"source": [
"## 6.7 Finetuning the model on supervised data"
]
},
{
"cell_type": "markdown",
"id": "6a9b099b-0829-4f72-8a2b-4363e3497026",
"metadata": {},
"source": [
"- In this section, we define and use the training function to improve the classification accuracy of the model\n",
"- The `train_classifier_simple` function below is practically the same as the `train_model_simple` function we used for pretraining the model in chapter 5\n",
"- The only two differences are that we now \n",
" 1. track the number of training examples seen (`examples_seen`) instead of the number of tokens seen\n",
" 2. calculate the accuracy after each epoch instead of printing a sample text after each epoch"
]
},
{
"cell_type": "markdown",
"id": "979b6222-1dc2-4530-9d01-b6b04fe3de12",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/15.webp\" width=500px>"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "Csbr60to50FL",
"metadata": {
"id": "Csbr60to50FL"
},
"outputs": [],
"source": [
"# Overall the same as `train_model_simple` in chapter 5\n",
"def train_classifier_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n",
" eval_freq, eval_iter):\n",
" # Initialize lists to track losses and examples seen\n",
" train_losses, val_losses, train_accs, val_accs = [], [], [], []\n",
" examples_seen, global_step = 0, -1\n",
"\n",
" # Main training loop\n",
" for epoch in range(num_epochs):\n",
" model.train() # Set model to training mode\n",
"\n",
" for input_batch, target_batch in train_loader:\n",
" optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n",
" loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
" loss.backward() # Calculate loss gradients\n",
" optimizer.step() # Update model weights using loss gradients\n",
" examples_seen += input_batch.shape[0] # New: track examples instead of tokens\n",
" global_step += 1\n",
"\n",
" # Optional evaluation step\n",
" if global_step % eval_freq == 0:\n",
" train_loss, val_loss = evaluate_model(\n",
" model, train_loader, val_loader, device, eval_iter)\n",
" train_losses.append(train_loss)\n",
" val_losses.append(val_loss)\n",
" print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n",
" f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
"\n",
" # Calculate accuracy after each epoch\n",
" train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=eval_iter)\n",
" val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=eval_iter)\n",
" print(f\"Training accuracy: {train_accuracy*100:.2f}% | \", end=\"\")\n",
" print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
" train_accs.append(train_accuracy)\n",
" val_accs.append(val_accuracy)\n",
"\n",
" return train_losses, val_losses, train_accs, val_accs, examples_seen"
]
},
{
"cell_type": "markdown",
"id": "9624cb30-3e3a-45be-b006-c00475b58ae8",
"metadata": {},
"source": [
"- The `evaluate_model` function used in the `train_classifier_simple` is the same as the one we used in chapter 5"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "bcc7bc04-6aa6-4516-a147-460e2f466eab",
"metadata": {},
"outputs": [],
"source": [
"# Same as chapter 5\n",
"def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n",
" model.eval()\n",
" with torch.no_grad():\n",
" train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n",
" val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n",
" model.train()\n",
" return train_loss, val_loss"
]
},
{
"cell_type": "markdown",
"id": "e807bfe9-364d-46b2-9e25-3b000c3ef6f9",
"metadata": {},
"source": [
"- The training takes about 5 minutes on a M3 MacBook Air laptop computer and less than half a minute on a V100 or A100 GPU"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "X7kU3aAj7vTJ",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "X7kU3aAj7vTJ",
"outputId": "504a033e-2bf8-41b5-a037-468309845513"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Ep 1 (Step 000000): Train loss 2.153, Val loss 2.392\n",
"Ep 1 (Step 000050): Train loss 0.617, Val loss 0.637\n",
"Ep 1 (Step 000100): Train loss 0.523, Val loss 0.557\n",
"Training accuracy: 70.00% | Validation accuracy: 72.50%\n",
"Ep 2 (Step 000150): Train loss 0.561, Val loss 0.489\n",
"Ep 2 (Step 000200): Train loss 0.419, Val loss 0.397\n",
"Ep 2 (Step 000250): Train loss 0.409, Val loss 0.353\n",
"Training accuracy: 82.50% | Validation accuracy: 85.00%\n",
"Ep 3 (Step 000300): Train loss 0.333, Val loss 0.320\n",
"Ep 3 (Step 000350): Train loss 0.340, Val loss 0.306\n",
"Training accuracy: 90.00% | Validation accuracy: 90.00%\n",
"Ep 4 (Step 000400): Train loss 0.136, Val loss 0.200\n",
"Ep 4 (Step 000450): Train loss 0.153, Val loss 0.132\n",
"Ep 4 (Step 000500): Train loss 0.222, Val loss 0.137\n",
"Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
"Ep 5 (Step 000550): Train loss 0.207, Val loss 0.143\n",
"Ep 5 (Step 000600): Train loss 0.083, Val loss 0.074\n",
"Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
"Training completed in 1.07 minutes.\n"
]
}
],
"source": [
"import time\n",
"\n",
"start_time = time.time()\n",
"\n",
"torch.manual_seed(123)\n",
"\n",
"optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)\n",
"\n",
"num_epochs = 5\n",
"train_losses, val_losses, train_accs, val_accs, examples_seen = train_classifier_simple(\n",
" model, train_loader, val_loader, optimizer, device,\n",
" num_epochs=num_epochs, eval_freq=50, eval_iter=5,\n",
")\n",
"\n",
"end_time = time.time()\n",
"execution_time_minutes = (end_time - start_time) / 60\n",
"print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
]
},
{
"cell_type": "markdown",
"id": "1261bf90-3ce7-4591-895a-044a05538f30",
"metadata": {},
"source": [
"- Similar to chapter 5, we use matplotlib to plot the loss function for the training and validation set"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "cURgnDqdCeka",
"metadata": {
"id": "cURgnDqdCeka"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_values(epochs_seen, examples_seen, train_values, val_values, label=\"loss\"):\n",
" fig, ax1 = plt.subplots(figsize=(5, 3))\n",
"\n",
" # Plot training and validation loss against epochs\n",
" ax1.plot(epochs_seen, train_values, label=f\"Training {label}\")\n",
" ax1.plot(epochs_seen, val_values, linestyle=\"-.\", label=f\"Validation {label}\")\n",
" ax1.set_xlabel(\"Epochs\")\n",
" ax1.set_ylabel(label.capitalize())\n",
" ax1.legend()\n",
"\n",
" # Create a second x-axis for examples seen\n",
" ax2 = ax1.twiny() # Create a second x-axis that shares the same y-axis\n",
" ax2.plot(examples_seen, train_values, alpha=0) # Invisible plot for aligning ticks\n",
" ax2.set_xlabel(\"Examples seen\")\n",
"\n",
" fig.tight_layout() # Adjust layout to make room\n",
" plt.savefig(f\"{label}-plot.pdf\")\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "OIqRt466DiGk",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 307
},
"id": "OIqRt466DiGk",
"outputId": "b16987cf-0001-4652-ddaf-02f7cffc34db"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAToJJREFUeJzt3Qd4U/X6B/Bv0z1pS3cpFGgpe+8hCMhQUdwXvYJcxxXRi6LXKw4Q+StuUEEQvYobEAW8Ciiy95BhGWVTWqALSume5/+8vzRpUlroTtJ+P89znuScnCQnp2ne85uvnaZpGoiIiMgq6Sx9AERERFQ+BmoiIiIrxkBNRERkxRioiYiIrBgDNRERkRVjoCYiIrJiDNRERERWjIGaiIjIijFQExERWTEGaiKqkEGDBuHpp5/m2SKqYwzURHXkoYcegp2d3VXLiBEj+DcgonI5lP8QEdU0CcpffPGF2TZnZ2eeaCIqF0vURHVIgnJQUJDZ4uPjox7bsGEDnJycsHnzZuP+b7/9NgICApCYmKjWV69ejf79+8Pb2xuNGzfGrbfeipMnTxr3P3PmjCqlL1myBAMGDICrqyt69OiBY8eOYffu3ejevTs8PDwwcuRIJCcnm5X2R48ejenTp8Pf3x9eXl54/PHHkZeXV+5nyc3NxXPPPYfQ0FC4u7ujV69e6jMYxMbGYtSoUerzyePt2rXDypUry329jz/+GJGRkXBxcUFgYCDuvvtu42NFRUWYOXMmmjdvrj5Tp06dsHTpUrPnHzx4UH0u+Xzy/AcffBApKSlmVff/+te/8Pzzz8PX11ed+1dffbVCfzciS2KgJrKyNmAJMGlpadi3bx9eeeUVfPbZZyrwiMzMTEyePBl79uzB2rVrodPpcMcdd6hAZmratGl4+eWXsXfvXjg4OOD+++9XAeqDDz5QFwInTpzA1KlTzZ4jr3fkyBEVbL///nv89NNPKnCX58knn8T27duxaNEi/PXXX7jnnntUjcHx48fV4xMnTlTBfNOmTYiOjsZbb72lgmhZ5PNIEH3ttddw9OhRdUFyww03GB+XIP3VV19h/vz5OHToEJ555hn8/e9/x8aNG9Xjly9fxuDBg9GlSxf1WvJ8ubi59957zd7nyy+/VBcNO3fuVBdB8n5r1qyp9N+KqE5Jmksiqn3jxo3T7O3tNXd3d7Pl9ddfN+6Tm5urde7cWbv33nu1tm3bao8++ug1XzM5OVnS1GrR0dFq/fTp02r9s88+M+7z/fffq21r1641bps5c6YWFRVldmy+vr5aZmamcdu8efM0Dw8PrbCwUK0PHDhQmzRpkrofGxurPsu5c+fMjmfIkCHalClT1P0OHTpor776aoXOzY8//qh5eXlpV65cueqxnJwczc3NTdu2bZvZ9ocfflgbM2aMuj9jxgxt2LBhZo/HxcWpz3306FHj8ffv399snx49emj/+c9/KnSMRJbCNmqiOnTjjTdi3rx5ZtukGtZAqr6//fZbdOzYEc2aNcOsWbPM9pXSqpSEpUQo1bqGkvTZs2fRvn17437yfANDabxDhw5m25KSksxeW6qT3dzcjOt9+vRBRkYG4uLi1LGYkhJyYWEhWrVqZbZdStBSJS+khDxhwgT8/vvvGDp0KO666y6z4zJ10003qfdo0aKFKpXLIjUFcjxS+s/KylL7mJJqeSlBiwMHDmD9+vVlltilacBwnKXfPzg4+KrzQGRtGKiJ6pBUu0ZERFxzn23btqnbS5cuqUWeYyBtvhLQPv30U4SEhKhALQG6dFuyo6Oj8b60WZe1rXR1eWVIALe3t8eff/6pbk0ZguUjjzyC4cOH49dff1XBWqqv33vvPTz11FNXvZ6np6eqppdqd9lXLkak/Vja1eW9hLyOtIeX1RFP9pFzI9XrpUkwLuu81MR5IKoLDNREVkRKf9L+KoF48eLFGDduHP744w/VFn3x4kXVfiuPSUcxsWXLlhp7bymVZmdnq85aYseOHSrohoWFXbWvlGSlRC2lUcOxlEWeK53SZJkyZYo69rICtZC2dCl5yyJt7NJhbt26daokLQFZag0GDhxY5nO7du2KH3/8EeHh4ep1iOoTfqOJ6pBUDSckJJj/Ezo4wM/PTwU+6SAlpdDx48er6l+prpZS6L///W/Ve1qqlRcsWKBKiRK4XnjhhRo7NimVP/zww6oTmvQel2ApHcbkIqE0qUp+4IEHMHbsWHV8ErilF7l0SJPq5VtuuUV1jJNe2LJvamqqqppu06ZNme/9yy+/4NSpU6oDmXxO6R0uJd2oqChV2pbe5XIBI9uk17t0ttu6davqnS4XM9JxTS4CxowZY+zVLVXm0tFNOuOVLvUT2RIGaqI6JL2RTatihQSjmJgYvP7662pIkwQtIftJUJbgM2zYMNWGLIFH2n6lulue9+GHH6re4jVhyJAhaniUBEu5oJD3vdbwJRkP/n//93949tlnce7cOXWx0bt3bzVkTMiFhwTQ+Ph4FVDlwqN0m7uBlJ6ll7m8X05OjjoO6XkuQ7rEjBkz1LAxqT6XgC77Syn6xRdfVI9LM4AE7v/85z/qXMnxSxOBvGdZFxpEtsROepRZ+iCIyLJkHLUMcVq+fDn/FERWhpeaREREVoyBmoiIyIqx6puIiMiKsURNRERkxRioiYiIrBgDNRERkRVjoK6GuXPnqpmQJC2fpPjbtWsX6ivJgCRTNMp4VZl2sfQwHhnlJ9M+ythfmdlKZpcyZFEykOkwZZIMGVMr42Blcg3D9JAGkoVJZrqScyqzWkmGI1sg43slnaRMziFpKSVlpMwiZkrGB8u4Ypm0RGb8krmvDekrDWQSE5ksROa4lteRiU4KCgrM9pFpNmUMsczWJdORLly4ELZA5jiXyVDk7y+LzCW+atUq4+MN/fyU5c0331T/bzJ5jAHPE9R4ezkvpkvr1q3r7zmyWDoQG7do0SLNyclJ+/zzz7VDhw6pLEfe3t5aYmKiVh+tXLlSe+mll7SffvpJZSRatmyZ2eNvvvmm1qhRI2358uXagQMHtNtuu01r3ry5lp2dbdxnxIgRWqdOnbQdO3Zomzdv1iIiIozZj0RaWpoWGBioPfDAA9rBgwdV1idXV1ftk08+0azd8OHDtS+++EId9/79+7Wbb75Za9q0qZaRkWHc5/HHH9fCwsJUFqs9e/ZovXv31vr27Wt8vKCgQGvfvr02dOhQbd++feqc+/n5GbNRiVOnTqlMUpMnT9YOHz6sffTRRyqL1erVqzVr9/PPP2u//vqrduzYMZXR6sUXX9QcHR3VORMN/fyUtmvXLi08PFzr2LGjMWuZ4HnStGnTpmnt2rXTLly4YFwkk1x9PUcM1FXUs2dPbeLEicZ1SQUYEhKi0gfWd6UDdVFRkRYUFKS98847xm2XL1/WnJ2dVbAV8kWX5+3evdu4z6pVqzQ7OztjqsSPP/5Y8/HxUakeDSQFoWk6RluRlJSkPu/GjRuN50OC0g8//GDc58iRI2qf7du3q3X5sdDpdFpCQoJZqklJ/2g4J88//7z6gTJ13333qQsFWyR/b0nJyfNjLj09XYuMjNTWrFljll6U56kkUMtFf1nq4zli1XcV50SWrEFSvWsg0xTK+vbt29HQnD59Ws1fbXo+GjVqpJoDDOdDbqW6u3v37sZ9ZH85b5Ky0bCPTF8pqR4NZN5rqUKWuaJticxFbZrCUr4v+fn5ZudIquqaNm1qdo5kbm9DWkrD579y5QoOHTpk3Mf0NQz72Nr3TqYXlelQMzMzVRU4z485qbaVatnSf2uepxLStCZNcZIaVZrUpCq7vp4jBuoqkDzA8kNj+kcWsl464UJDYPjM1zofcivtQKWTUUggM92nrNcwfQ9bIIkjpE2xX79+xhzRcvxyASIXK9c6R9f7/OXtIz8wkvnK2kkea2kzlDY/yai1bNkytG3blufHhFzASMpP6fdQGr9HelIIkPZimTtf+j5IYUH6tqSnp9fLc8SkHES1UBo6ePBgjaagrC8kkcj+/ftVjcPSpUtV5quNGzda+rCsRlxcHCZNmoQ1a9aoDpVUNsnKZiAdFCVwSxKWJUuWGNO01icsUVeBZAmStHmlexHKelBQEBoaw2e+1vmQW8ldbEp6WEpPcNN9ynoN0/ewdpIWUrJfSUrHJk2aGLfL8UuTiSS+uNY5ut7nL28f6UVtCz9QUtKR3rPdunVTJUbJCPbBBx/w/BSTalv5P5GexlLjJItcyEiWNLkvJTp+j64mpWdJpyqpTevj/xoDdRV/bOSHRnLvmlZ3yrq0tzU0zZs3V19q0/Mh1UPS9mw4H3Ir/zjyQ2Swbt06dd7katiwjwwDk/YlAylZSClMchRbM+ljJ0FaqnLlc8k5MSXfF0dHR7NzJG3v0q5meo6katj0gkY+v/wwSPWwYR/T1zDsY6vfO/n7S0pKnp+SVKPyHZBaB8Mi/TqkDdZwn9+jq8kwz5MnT6rhofXyu1Tn3dfq0fAs6dW8cOFC1aP5scceU8OzTHsR1ifSC1WGMcgiX5v3339f3Y+NjTUOz5LPv2LFCu2vv/7Sbr/99jKHZ3Xp0kXbuXOntmXLFtWr1XR4lvTWlOFZDz74oBqyI+dYhkfYwvCsCRMmqOFpGzZsMBsykpWVZTZkRIZsrVu3Tg0Z6dOnj1pKDxkZNmyYGuIlw0D8/f3LHDLy73//W/VknTt3rs0MP3rhhRdUL/jTp0+r74isS6//33//XT3e0M9PeUx7fQueJ0179tln1f+afJe2bt2qhlnJ8CoZbVEfzxEDdTXIuDr5Msh4ahmuJeOD66v169erAF16GTdunHGI1iuvvKICrVzADBkyRI2VNXXx4kUVmD08PNQwiPHjx6sLAFMyBrt///7qNUJDQ9UFgC0o69zIImOrDeSi5YknnlBDkuQH4I477lDB3NSZM2e0kSNHqvHj8sMjP0j5+flX/S06d+6svnctWrQwew9r9o9//ENr1qyZOm75UZTviCFIi4Z+fioaqHmeNDVMKjg4WP2N5XdC1k+cOFFvzxGzZxEREVkxtlETERFZMQZqIiIiK8ZATUREZMUYqImIiKwYAzUREZEVY6AmIiKyYgzU1SAzKkkCc7klnid+l2oX/994jhrq98ii46hlrt+ffvoJMTExau7Uvn374q233lJTRpZHMqaMHz/ebJtk4snJyUFdk2kyJZ2jJBiQqeeI54nfJf6/WRJ/k+rnObJoiVomm5dMQzt27FBzqMocz8OGDVM5aq9FTu6FCxeMS2xsbJ0dMxERUYNJcym5REuXliVnsSRuuOGGG8p9np2dnc1kUyIiIqo3+ailKkL4+vpeN1OK5B6VzDuSDu6NN95Au3btKvQeklpx3759Kl2cTle9CgVJUi7OnTunqlOI54nfpdrD/zeeo/r0PZL4JWkzu3TpolKYXovVzPUtB33bbbepVIhbtmwpd7/t27fj+PHjKlm4BPZ3331XpUY8dOiQWf5fA+kwYNppQErrgwcPrrXPQUREVFG7du1Cjx49bCNQT5gwAatWrVJBuqyAWx5p127Tpg3GjBmDGTNmXPW49O6bPn16mSdHcpcSERHVNelf1bNnT9XHqmnTptYfqJ988kmsWLFClYybN29e6effc889qurg+++/v26JWqo7JDF4XFxcpS4IiIiIakp8fDzCwsIqFIss2utbrhEkSC9btgzr1q2rUpAuLCxEdHR0uaVjGbolvcQNi6enZw0cORERUQPoTCZDs7777jtVmpYAmpCQoLbLGDcZVy3Gjh2L0NBQNeZavPbaa+jduzciIiJUe/Y777yjqg4eeeQRS34UIiKi+heo582bp24HDRpktv2LL77AQw89pO6fPXvWrHd2amoqHn30URXUfXx80K1bN2zbtk1VZxMREdU3VtFGba3tAkTU8EhzmnRSJaoOR0dH2Nvb10gssqpx1EREliJlFqmpkyY1oprg7e2tJueSSbqqg4G6OrIvA2d3AI2aAEHtq/VSRGRZhiAtsyO6ublV+8eVGvZFX1ZWFpKSktR6dYcCM1BXx7r/A3Z/CvR6HBj5VrVeiogsW91tCNKNGzfmn4KqzdAhWoK1fK+uVQ1+PUxzWR3h/fS3Z7ZW62WIyLIMbdJSkiaqKYbvU3X7PDBQV0ez4kCdeBDIulStlyIiy2N1N1nj94mBujo8AgC/VtIiAZzdXiN/ECIiIlMM1NUV3l9/y+pvIqonwsPDMXv27Arvv2HDBlV6rO0e8wsXLlQ9qRsaBuqaqv4+s7n6fw0iokqQ4HitRZISVcXu3bvx2GOPVXj/vn37qiQTMqsk1Tz2+q6pEnVCtH64lmvDu9ojIsuQ4GiwePFiTJ06FUePHjVu8/DwMBsyJL3br5f7WPj7+1fqOJycnNR4YaodLFFXl2cQ0DiiuJ16R438UYiIKkKCo2GR0qyUog3rMTExKoeCpA+WqZYlQZGkET558iRuv/12BAYGqkAuuZD/+OOPa1Z9y+t+9tlnuOOOO1RP5sjISPz888/lVn0bqqh/++03lYZY3mfEiBFmFxYFBQX417/+pfaTIXH/+c9/MG7cOIwePbrSU1G3bNlSXSxERUXh66+/Nrs4kVoFSSMpnz8kJES9p8HHH3+sPouLi4s6H3fffbdVfvEYqGsCq7+J6uekFXkFFllqcmbnF154AW+++SaOHDmCjh07IiMjAzfffDPWrl2Lffv2qQA6atQolVfhWqZPn457770Xf/31l3r+Aw88gEuXyh/tIhN+vPvuuypwSgpjef3nnnvO+Phbb72Fb7/9VuV22Lp1K65cuYLly5dX6rMtW7YMkyZNwrPPPouDBw/in//8J8aPH4/169erx3/88UfMmjULn3zyCY4fP65ev0OHDuqxPXv2qKAtiZ6kFmL16tW44YYbYI1Y9V1T1d97vwRiOZ6aqL7Izi9E26m/WeS9D782HG5ONfPzLIHopptuMq77+vqiU6dOxvUZM2aogCclZEk7XB5JlDRmzBh1/4033sCHH36IXbt2qUBfFhk7PH/+fFXaFfLaciwGH330EaZMmaJK6WLOnDlYuXJlpT7bu+++q47riSeeUOuTJ0/Gjh071PYbb7xRXRxI7cLQoUPV3NtSsu7Zs6faVx5zd3fHrbfeqmoemjVrhi5dusAasURdkyXqCweAnLQaeUkioprQvXt3s3UpUUvJVqqkpdpZqqWltH29ErWUxg0kwHl5eRmnyCyLVJEbgrRhGk3D/mlpaUhMTDQGTSEzd0kVfWUcOXIE/foV//4Wk3XZLu655x5kZ2ejRYsWKuuiXJBIlbuQixcJzvLYgw8+qEr3UgtgjViirgmNQgGf5kDqaeDsTqDVsBp5WSKyHFdHe1WytdR71xQJqqYkSK9Zs0aVOiMiItRUl9I2m5eXd83XkRKpKWmTLioqqtT+dZ2sMSwsTFVrSxu8fGYpeb/zzjvYuHGjKkXv3btXta///vvvqiOetGdLj3drGwLGEnVNiboZaDUCcDL/pyAi2ySBRaqfLbHU5gxp0h4s1cVS5SzttVI1fObMGdQl6fgmnbckKBpIj3QJnJXRpk0b9XlMyXrbtm2N63IhIm3wUlUvQXn79u2Ijo5Wj0kPeKkWf/vtt1Xbu5yHdevWwdqwRF1TRrxRYy9FRFRbpJfzTz/9pIKXXBC88sor1ywZ15annnoKM2fOVKX61q1bqzbr1NTUSl2k/Pvf/1Yd3KRtWQLu//73P/XZDL3Ypfe5XAD06tVLVcV/8803KnBLlfcvv/yCU6dOqQ5kPj4+qn1czoP0HLc2DNRERA3I+++/j3/84x9qkhI/Pz81LEp6XNc1eV9JLTp27FjVPi0TrAwfPrxSWaZGjx6NDz74QFXjS+/v5s2bq17kgwYNUo9LFbb0eJdOZhKwpQZBgrkMB5PHJKhLdXdOTo66gPn+++/Rrl07WBs7ra4bDSwsPj5etVvExcWhSZMm1X69gsIi2Ov0swApl+MAnQPgVb38o0RUd+SH+vTp0+qHXsbUUt2T0qxUZUsJWXqi1/fvVXwlYhHbqKvh+aUH0HXGGhw8V3w1uvpFYHZ7YNeC6rwsEVG9Fxsbi08//RTHjh1TbcYTJkxQQe3++++39KFZHQbqakjNyseVnAJsPFY8RCGwHWBnD2RdrKE/DxFR/aTT6VQbssyMJkOqJFhL27KUqskc26irYWArf6w5nIiNx5Lx5OBIoN1ooO1tgLNndV6WiKjek2rf0j22qWwM1NUM1GLv2ctIy85HI1cOzSIioprFqu9qCPN1Q0t/dxQWadh6IsX8QQsMdyAiovqHgbqaBrYKULcbjybrN5z7E/h0MPDVbdX+4xARETFQV9PAKH31t7RTq5FuLt76YB23E8jP5jeMiIiqhYG6mno194Wzgw4JV3JwNDEd8G0BeAYDhXlAfMn0eERERDYXqGX6OOmaL5OjBwQEqFlmZAL16/nhhx/UlHMygFxmmqlsarSa5OJojz4tG5dUf8vEJ5L2Upxhj0YiIrLhQC0ZTCZOnKjyh0pmE8lfOmzYMGRmZpb7nG3btqmcqA8//LBKei7BXRZJGm7p3t9S/W2W9vLMFosdExFRRcmUm08//bRxPTw8HLNnz77mc2Q2xuXLl1f7JNfU61yLTBPauXNn2CqLBurVq1erLC4yt6okMpfB75IT9c8//yz3OTKvqyQql8nYZWC8TDXXtWtXlXTc0oF695lLyMwtKClRS9V3fo7FjouI6jdJrCG/h2XZvHmzCoKSFaqyJKuVzL1dF8HywoULGDlyZI2+V31jVW3Ukkxc+Pr6lruPpCiTLCmmZCJ32V6W3NxcNeG8YUlPT6/howaa+7mjqa8b8gs1bDt5EWgcAXgEAoW5+o5lRES1QGoWpTZS5o0uTZJTdO/eHR07dqz06/r7+6tsU3VB0mw6OzvXyXvZKp01TcguVS8ylVz79u3L3U+yrUgeU1OyLtvLaweX3KeGxTRPaU2Rq9aS6u8kfTs1q7+JqJbdeuutKqhKbaSpjIwM1ZdHAvnFixdVc2FoaKgKvtKvR7JEXUvpqu/jx4+rdJDSL0h+Q+XioKxsWK1atVLv0aJFC5U+U5ozhRzf9OnTceDAAfV7KYvhmEtXfctUooMHD1bpKCXL1WOPPaY+j4HUwkpzp2TMCg4OVvtIE6rhvSoab1577TWVDEMuEqSkLzW8Bnl5eXjyySfV68tnlrSYEkuEjO6R2oGmTZuq54aEhOBf//oXGkSglhMt7cyLFi2q0dedMmWKKqkblsOHD6M2GAL1hqPFw7TCi9upY9lOTWTT8jIrvxQWlDxf7su20sM1y3tuJTg4OKg0kRL0TBMhSpCWtI4SoCWDU7du3fDrr7+q31gJfA8++CB27dpV4aB25513wsnJCTt37sT8+fNVUC5NOgXLcchvrDRRSsKNWbNmqcfuu+8+PPvss6qZU6q6ZZFtpUn/JKkhlfzQUv0un+OPP/5QQdPU+vXrcfLkSXX75ZdfqvctfbFyLXJ87733ngr20jQg73nbbbepCxLx4Ycf4ueff8aSJUtUB+dvv/1WXbyIH3/8UX2uTz75RO0vFxly8VPvpxCVP4Ik8d60adN1031JNUliYqLZNlmX7WWRKx7TapXayrsqPb+d7HWIT83GqZRMtGxW3E4dtxsoyAUcWLVDZJPeCKn8c+5ZCLS7Q38/5n/ADw8B8psw/teSfWZ3KDuBz6v6JsCKktzS77zzjuqca8jDLNXed911l7Em8bnnnjPu/9RTT+G3335TQahnz57XfX0JlDExMeo5UnoUb7zxxlXtyi+//LLxvgQ1eU8peD3//POqdOzh4aEuLMr7rRbfffedurD46quv4O6un5J5zpw5qi3+rbfeMtamSiCX7ZK7WkYA3XLLLVi7di0effTRCp0zCdBysfG3v/1NrctrS9CXWoS5c+eqvlKSn7p///6qxC8lagN5TD6DNME6OjqqknVFzqPNlqjlClCC9LJly7Bu3TqVs/N6+vTpo/4gpqQaRrZbkruzA3o09ykZpuUfBbj5AQXZwLm9Fj02Iqq/JFD17dsXn3/+uVo/ceKE6kgm1d5CStbS6VZKfdL/RwKmBF0JOBVx5MgRlUDDEKRFWb+3ixcvVk2XEsTkPSRwV/Q9TN9LOhYbgrTo16+fKtWbDt2VkrkEaQOpok5KKs5ieB1SWDt//rx6XVOyLu9vqF7fv38/oqKiVLX277//btzvnnvuQXZ2tqrelwsDiV8FBSY1KPWtRC3V3XIFtWLFClVtYmhnlitAuQITUq0jbSuG9oFJkyZh4MCBqtpCrqLkim3Pnj1YsMDyOaCl+nvriYtqmNY/+jfXV38fXqGv/m5m2QsJIqqiF89X/jn2JjVorUfpX8OuVLno6ega+5NIUJaSspQGpTTdsmVL9TsppLQtVb1SWpRgLUFQ+gNJO2xNkc68DzzwgGqHlmpk+Q2X32b5na4Njo6OZutS6pVgXlNkJJHkxl61apWqUbj33ntVCXrp0qXqokUuGmS7FBKfeOIJY41G6eOqFyXqefPmqXZjqa6RKyLDIldmBnJFJu0ZBnLlKMFdArNcecmJkzaCa3VAqyuDovTzfu84dRE5+YX6qi5D9TcR2SYn98ov9iZlILkv2xxdK/a6VSCBRPI7y2+jVBtLdbgELyGpJG+//Xb8/e9/V7+ZUhI8duxYhV9bhsHGxcWZ/Q7L3Bel57eQ6uGXXnpJ9TSXauPY2Fjzj+vkpEr313sv6XBmOpfG1q1b1WeT0m1N8PLyUrUDpVNsyrppZ2PZT9rRpa1dYpK0TV+6dEk9JgVJqY6XtuwNGzaoCxXpBFcvS9SmnR/KIyehNKl6kMXaRAZ4ILiRCy6k5ahgPajt7UBoNyC4k6UPjYjqMalqlqAinWelaleqbg0kaEqBRoKptO2+//77ql9PRUfASElSenOPGzdOlRzl9SUgm5L3kEKVlKJltknpuCZVwqak3VpKqVKlLH2RpBa19LAsKZVPmzZNvZf0rE5OTlY1BdL5rfRon+qQeTjkfaTmQXp8Sy2EHJd0GhNyjqTQ2KVLF3WRIJ3apErf29tbdVqTC45evXqpHu7ffPONCtym7dj1ttd3fWA+TCsZ8AwEmnQzv7omIqoFUv2dmpqqqp5N25OlrViqcmW71F5KwJHhTRUlgUqCrrTLSqepRx55BK+//rrZPtJj+plnnlF9jiTwyUWBDM8yJZ3bZHKWG2+8UQ0pK2uImAQ+aT+XkqsE/LvvvhtDhgyp8QmtpN158uTJqie6NAfI0Czp5S0XHEIuIt5++21VOyDHcebMGTVVtZwLCdZSypY2bRmjLlXg//vf/9Qwsdpip1WkWFuPyMQA0sYgVTnX62FeFauiL2DCt3vRwt8d657V98AkIusmPY2ltCcdWmXcLFFtf68qE4tY1Kth/SL9YK+zw6nkTMRdykJYYTyw/SPAzh4Yde25c4mIiEpj1XcN83JxRLem+mFaG6T6W6YR3fsVEP2D+SQIREREFcBAXQsGRvmXjKcOaAf0nwzcLWMcG1QrAxER1QAG6lpg6FC27WQKcos0YOg0oNVwwL52xtgREVH9xUBdC9oGe8HPwxlZeYX480xqbbwFERE1EAzUtXFSdXa4oZVfyTCtokLgxFpg3ev6+0RklWpydiuiohr6PrHXdy3OUvbT3nMqUE8Z0Qr4YTyQmwa0vhkI6VJbb0tEVSCzZskYWZkDWsb4yrphZi+iypJRzzJFq0zYIt8r+T5VBwN1LRkQ4afSUsckpONCeh6CZa7vY6uBM1sZqImsjPyYylhXmSZTgjVRTZAJXCS7lny/qoOBupb4uDuhUxNv7I+7jE3HknFfs37FgXoL0Nc8tyoRWZ6UeuRHVTIhXW9OaqLrkexektazJmpmGKhrufe3BGqp/r5vUHFKtbPb9O3UupIUbURkHeRHVTIg1VYWJKKqYGeyWjSoeDz15uMpKAjoADh5AjlpQOKh2nxbIiKqRxioa1HHJt7wdnNEek4B9p3LAJr21j8g1d9EREQVwEBdi2TO7wGRJrOUhRdXf8ea50ElIiIqDwN1LRtUPEvZhmNJQLP+JYGa4zWJiKgCGKhr2YDiiU8OnruCZM82gKM7kJ0KJB2u7bcmIqJ6gIG6lgV4uqBdiJe6v/nUZaBpL/0DrP4mIqIKYKCuw97fajpRGU8t2KGMiIgqgIG6DgxsFaBuZeKTQtN2ao1pL4mI6No44Ukd6NLUG57ODkjNysdBrQU6RQ7XV4EX5AKOLnVxCEREZKMYqOuAo70O/SP9sOpgAjacSEOnB5bUxdsSEVE9wKrvOpxO1DhMi4iIqIIYqOvIDcWB+kDcZaRm5gHpicCh5WynJiKia2KgriMh3q5oFeiBIg3Yeuw88EFH4IdxwMUTdXUIRERkgywaqDdt2oRRo0YhJCREZa1Zvnz5NfffsGGD2q/0kpCQAFswKErf+1vaqRHWCwjqCGRdsvRhERGRFbNooM7MzESnTp0wd+7cSj3v6NGjKsG7YQkI0AdAW2mnlvHURQ/8CDy+uWQCFCIiImvr9T1y5Ei1VJYEZm9vb9ia7uE+cHOyR3J6Lo4kZaFdSCNLHxIREVk5m2yj7ty5M4KDg3HTTTdh61bbyUTl7GCPvi0bl8xSJvKzgbwsyx4YERFZLZsK1BKc58+fjx9//FEtYWFhGDRoEPbu3Vvuc3Jzc3HlyhXjkp6eDqsYpiVpL1c+D7zZFIj+waLHRERE1sumJjyJiopSi0Hfvn1x8uRJzJo1C19//XWZz5k5cyamT58O65pO9BD2xqYit7kHnAvz9NOJdhtn6UMjIiIrZFMl6rL07NkTJ06UP8RpypQpSEtLMy6HD1s2vWTTxm5o4eeOgiINB+w7lCTo4LzfRERUHwP1/v37VZV4eZydneHl5WVcPD09YS2Tn/yS2gTQOQJXzgGpZyx9WEREZIUsGqgzMjJUoJVFnD59Wt0/e/assTQ8duxY4/6zZ8/GihUrVAn64MGDePrpp7Fu3TpMnDgRtmRgcdrLP45fgRbaVb+RaS+JiMja2qj37NmDG2+80bg+efJkdTtu3DgsXLhQjZE2BG2Rl5eHZ599FufOnYObmxs6duyIP/74w+w1bEGfFo3h7KDD+bQcpLbrCd+4nfp26q4PWvrQiIjIythpWsNqHI2Pj1e9xePi4tCkSROLHcfYz3ep/NTzel/GyP1PAI2aAs9EW+x4iIjIOmORzbdR2yrDMK2lSaGAnT2QdhZIjbX0YRERkZVhoLZwoN4cm43CkC76jVL9TUREVN1ALUV1KbYb7Nq1S3XsWrBgQVVerkFq6e+OJj6uyCssQrxXcaA+w0BNREQ1EKjvv/9+rF+/Xt2XzFUylacE65deegmvvfZaVV6ywZGsX4ZS9abc4klczmy27EEREVH9CNQyNEomGhFLlixB+/btsW3bNnz77beqtzZVjCFQf5cQom+nvhwLpJXUVBAREVUpUOfn56uJRIQMj7rtttvU/datW6shVVQxfSP84GhvhyOXgFz/DoCDC5B8lKePiIiqF6jbtWunkmNs3rwZa9aswYgRI9T28+fPo3FjfXYouj4PZwd0b+ar7v8vaibwwlkgYghPHRERVS9Qv/XWW/jkk09U5qoxY8agU6dOavvPP/9srBKnys1S9utZB8BBX0tBRERUrZnJJECnpKSotJE+Pj7G7Y899piaMYwqcS6j/PHmqhhsP3UROfmFcHG01yfosLPjaSQioqqVqLOzs1WeZ0OQjo2NVfNwHz16FAEBksaRKioq0BOBXs7IyS/C+ZVvA3N7Awd/5AkkIqKqB+rbb78dX331lbp/+fJl9OrVC++99x5Gjx6NefPmVeUlGyzTYVqJ52OB5CNM0EFERNUL1Hv37sWAAQPU/aVLlyIwMFCVqiV4f/jhh1V5yQZtUJS+FuLzjN7AvV8Dg1+x9CEREZEtB+qsrCxjXufff/8dd955J3Q6HXr37q0CNlVOvwg/2OvssOaiP+KDhwLu7DlPRETVCNQRERFYvny5mkr0t99+w7Bhw9T2pKQkeHl5VeUlG7RGro7oEuat7m86lmLpwyEiIlsP1FOnTsVzzz2H8PBwNRyrT58+xtJ1ly7F81ZTpRjaqQ9H/wlseBPY+QnPIBERVS1Q33333Th79iz27NmjStQGQ4YMwaxZs3haq9FOnREXDWyYCez5nOeRiIiqNo5aBAUFqcWQRUsSX3Oyk6prF+KFxu5O2JgZCbgASI4BMlMAdz9+TYmIGrAqlaiLiopUlqxGjRqhWbNmavH29saMGTPUY1SFP4TODje08kcqvJDk2lK/kfmpiYgavCoFaklnOWfOHLz55pvYt2+fWt544w189NFHeOUVDi2qzixlYkdRG/2GM1sa/BeUiKihq1LV95dffonPPvvMmDVLdOzYEaGhoXjiiSfw+uuv1+QxNhj9I/zUzKGr0lviNicJ1FstfUhERGSLJepLly6plJalyTZ5jKqmsYczOoY2wq6i4nObdAjI4vkkImrIqhSoJVuWVH2XJtukZE1VNzAqABfRCBecmuk3xG7j6SQiasCqVPX99ttv45ZbbsEff/xhHEO9fft2NQHKypUra/oYG9x46g/XHsemvCjch1h9O3WbWy19WEREZEsl6oEDB+LYsWO44447VFIOWWQa0UOHDuHrr7+u+aNsQDo1aaRmKtucF6XfEMsOZUREDVmVx1GHhIRc1WnswIED+O9//4sFCxbUxLE1SA72OvSP9MPOv4p7ficcBLJTAdeSvN9ERNRwVKlETbVrUCt/JMMb8fZNAGhA7HaeciKiBsqigXrTpk0YNWqUKp1LXmZJ9HE9GzZsQNeuXeHs7KySgyxcuBD1dd7vTXmt9Bs48QkRUYNl0UCdmZmpepDPnTu3QvufPn1adWK78cYbsX//fjz99NN45JFHzOYbrw8CvFzQJtgL/yvsgyNRE4H2d1n6kIiIyBbaqKXD2LVIp7LKGDlypFoqav78+WjevDnee+89td6mTRts2bJFJQIZPnw46tssZfMutMMCXShmhXa29OEQEZEtlKhlbu9rLTLn99ixY2vtYGUI2NChQ822SYCW7fW2+vtYMoqKNEsfDhER2UKJ+osvvoAlJSQkIDAw0GybrF+5cgXZ2dlwdXW96jm5ublqMUhPT4ct6NbMBx7ODsjPvIS4bUvQzM8TaH2zpQ+LiIjqWL3v9T1z5kyzUn/btm1hCxztdegX0RiDdfvR7I/HgM3vWvqQiIjIAmwqUEv+68TERLNtsu7l5VVmaVpMmTIFaWlpxuXw4cOwFQNbBWBnURvE2YcBod0BjVXgREQNjU0FapmudO3atWbb1qxZY5zGtCwyjEsCuWHx9PSErRgY5Y8LaIyBWW8hbdDrUKm1iIioQbFooM7IyFDDrGQxDL+S+2fPnjWWhk07pz3++OM4deoUnn/+ecTExODjjz/GkiVL8Mwzz6A+CvV2RWSAB6Qv2ZYTKZY+HCIiamiBes+ePejSpYtaxOTJk9X9qVOnqvULFy4Yg7aQoVm//vqrKkXL+GsZpiV5sevb0Kyyen9viTkHJERb+nCIiKiO2Wlaw2r4jI+PR1hYmMr01aSJTNFp3TYfT8bT/12DrS6T4Kwrgt0LZwEnd0sfFhERVUNlYpFNtVE3RD3CfZHl6IsUzQt2RQVA3E5LHxIREdUhBmor5+Jojz4tG2NnUWv9BslPTUREDQYDtY20U+8oKh7/fWarpQ+HiIjqEAO1jQRqGU8ttHN/AnlZlj4kIiKqIwzUNiDczx06n3Bc0HxhV5QPxO+29CEREVEdYaC2EQOjArCjuFTNdmoiooaDgdqGZikzVn/HskMZEVFDwUBtI3q3aIy9dvoOZVr8n0B+jqUPiYiI6gADtY1wc3JAYHg7JGre0BXmAuf2WPqQiIioDjBQ21g7taH6m+3UREQNAwO1DRlk0k5deJrt1EREDQEDtQ1p6e+BU+5dkKfZIy23iPmpiYgaAAZqG2JnZ4fwqM7omPsZPgx5h/mpiYgaAAZqG2ynzoEzNh1LtvShEBFRHWCgtjH9IhrDQWeHUymZiEtIsfThEBFRLWOgtjGeLo4Y1MQOvzi9iKBPOwAFeZY+JCIiqkUM1Daoa5sIBNtdhGNhFpB40NKHQ0REtYiB2gYNigrEP/OewcCiecgN7GTpwyEiolrEQG2D2gR7ItajE2LzGmHPmVRLHw4REdUih9p8caq9YVqSo3rpn/HI3fAesCUaCGwPBMnSAfBvDTg48/QTEdUDDNQ2PEuZBGrXCzuBwj+BM5tLHtQ5AH6tSoK3uu0AeARY8pCJiKgKGKhtVP8IPzVMa1rWveio6462urPo5nwOkdoZuBVeAZIO65foJSVPcg/QB+6OfwM63WfJwyciogpioLZR3m5O+HBMFyzfF4CNcRFYmp4L5MsjGoJwCW11sejiFI+ebucRWXQGPjlxsMtMAk6uA5r2LXmh1Fhg8d+B0G7AqNkW/ERERFQWBmobdnOHYLVomobzaTnYf/Yy9p1Nxf44X2w95491OV2B4rTVrshBlF08+nteQNGZFgh0PIMuTb3RJu0vOCb8pQK8me+KS9zG6vMOgG9zQGdf9x+UiKgBY6CuJ53LQr1d1XJLx2C1Lb+wCDEX0rE/LhX74i6rIL4/xQX7r0QAVwAcOaT2C3TIwp2NX0Zzd3e4HjivgneopwPspORdmAccW13yRo5uQEBb83Zv6bjm6l3rn1EuRtJzC3AxIw8XM3KRkpGH7PwC9Aj3RRMft1p/fyIiS7HT5BewAYmPj0dYWBji4uLQpEkTNCSXs/KwX4J28bLv7GWkZav6cjOB7g64M/A8ertdQGuchl/mcdgnxwAF2WW/sEegvvPa0OlAk276bYX5+k5tdnblHk9uQSEuZUrgzUNKRq4+CGfqb1OK7xu3Z+Qhr7CozNfp1KQRRrQPxsj2QQj3c6/i2SEiss5YZBWBeu7cuXjnnXeQkJCATp064aOPPkLPnj3L3HfhwoUYP3682TZnZ2fk5BTX8V5HQw7Upcmf/szFrOLqcn3wPnz+CgqKzL8SEmtb+7thaGAGertfQGu7WPimH4OdzIqWft64X9Ej65Hm014FWPvdnyJs3zs4GnoXfmvyL1UKvpieC6e0kzic0xiJmYVIzymo9DF7ODugsYcTGrs7qcp6OWbTb3CbYC/c3D4IIzsEISLAs3oniIiollQmFlm86nvx4sWYPHky5s+fj169emH27NkYPnw4jh49ioCAsocTeXl5qcdNq36p8uS8NfdzV8udXfVflJz8Qhw6n6ZK24Yq83OXs3EkKQtHknT4CKEAQuHuNAAdmjSCp2c2XK+cgk/2Gfz48RlkFF1Qr/OawzaMdcjCppOX8eHR42qbP1Kx22Ui8jV7xGqBOOkYglMIRaJTU6S6NUeWVwt4ePmoINzYw1kFZD8VlJ3h5+mstrs4mreRJ6fn4vfDCVgVnYDtpy7iyIUranlvzTFEBnioUvbIDsFoHeTJ7wkR2SSLl6glOPfo0QNz5sxR60VFReoq46mnnsILL7xQZon66aefxuXLl6v0fixRV15SenFHteLA/Vf8ZWTmFZa7fyNXRwS626GdyyW4unvC3qepCrqtCo9j2K5H4CBzlJfHMwTwb6WvSjcsYb0AR5frHmdqZh7WHE7EyoMXsPVECvILS77a4Y3dVMCWwN0htBGDNhFZlM1Ufefl5cHNzQ1Lly7F6NGjjdvHjRunAvGKFSvKDNSPPPIIQkNDVVDv2rUr3njjDbRr167M98jNzVWLwblz59C2bVtWfVdDYZGG40npiI5Pg4O9nSrx6ku/zvBxc4KTwzVmpi0q0leXJx8FUo4DKcW3si7Dx8ry3PGSyVoO/gRcPgtE3gQElv03F9L2vvZIIlYdTMDGY8nIKyhp35ZOd4aSdpcwb+h0rJEhorplM1XfKSkpKCwsRGBgoNl2WY+JiSnzOVFRUfj888/RsWNHpKWl4d1330Xfvn1x6NChMj/szJkzMX369Fr7DA2Rvc4OrYO81FJpOh3QqIl+iRhi/lh2anHwPlYcyI8B6QmAu3/JPn8t1vdEd3IvCdSXTgP7vtGPBZfFM1CV6qU6X5aM3AKsj0nCqoMXsD4mWVXlf7bltFqCvFwwon2QWqQHuXw2IiJrYtES9fnz51XJeNu2bejTp49x+/PPP4+NGzdi586d132N/Px8tGnTBmPGjMGMGTOuepwl6npm16fA2R1Anyf0QVns/Rr4+cmSfRqFAaFdSwJ3cGfA2UM9lJ1XiI3HJGgnYO2RJBXEDaQ9fFi7INzcPhi9WvjC0Z45a4iogZeo/fz8YG9vj8TERLPtsh4UFFSh13B0dESXLl1w4sSJMh+XHuGyGFy5IoOIyWb1fFS/mGrcEujyd+DcXiDpCJAWp18OFzed2OkA/zYqeLuGdsMIWe7pgJwiO9WWvTI6AWsOJ6ghYd/tPKsWbzdHDGsbiJHtg9Evwu/a1flERLXIooHayckJ3bp1w9q1a41t1NLuLOtPPmlSQroGqTqPjo7GzTffXMtHS1arWV/9InLTgQsHgPg9wLk/9cH7SjyQdEi/7Ptav19IF7g8tgFD2gSqJe9yALYn2mP1oQT8dihRje9esideLZ4uDhjaRoJ2EG5o5X9Vz3Miotpk8eFZMjRLOo91795djZ2W4VmZmZnGsdJjx45V1ePS1ixee+019O7dGxEREarDmYy/jo2NVR3MiODsCYT31y8G0s6tgrZh2WveEa0wH05zOmOgkwcGPr4FM25vj12nL+G36HisPJyihoAt23dOLW5O9hjcOkCVtCUxipuzvUqOwiGCRFRvA/V9992H5ORkTJ06VU140rlzZ6xevdrYwezs2bPQSQekYqmpqXj00UfVvj4+PqpELm3c0pObqEyeQUDrW/SLoed5fmbJ49IZragQKMpXs6w56HToG+GHvvufx6ue+3EprD125TfHjwmB2JwejF/+uqAWU072Ojja28HRQW51JevqVqe2O5muyz4Oduq9DPfNHjPsa3y9q1/L39MZrQI94eniyD88UT1m8XHUdY3jqKlM+Tn6YV8yhttgdkfgcqzZbkU6RyS6RmB7bjPsym6CBM0HSZoPEjUfXIInNNR9W7YMN4sK8tQvgfrbFv7ucHZgFT2RtbKZcdSWwEBNFZZ1CTi/T19Vfm6Pvt07K6Xc3TWdA5L7z0BK67+rpCh2l8/C+8RPyPAIx/nQkWqbzFeeX1CE/CJNrcukLPmGbepxw/bi9YJS6/J4gf51zqVmI+FK2VPnSnW8zDjXKsgTrQM99bdBngjzceO4cSIrYDO9vomsmpuvfqy3Yby3XNNKb3Jp55agLWO9MxKA9EQgMxl2RQUI8A9AQEjx+PLMbcCBWUBIV7S96aGS153TA8jL0lfJmy6+wYCHYT1Y//7XmR5XEq0cS8zA0YQrOJqYjqMJ6YhJSFfzqB9PylDLryippnd1tEerQA9V6pZqcxkL3yrIA/4ezmxnJ7JSDNREFSVB07upfml3h/ljki0sIwlwMZkExjMQ6PKgfn8DCfaX4/SZyKQ3+rXoHEuC+IBngaiR+u2ZKcD5/YB3GLz9o9Czua9aSt5CUyVtCdrGJTFdBe3s/EIciE9TiylfdycVwFXgLq4+l0WSoBCRZfG/kKgm2DsCjSRhiQnDhCulPbVH3xNdLReAjET9rVovvi9V7NK5zTAmPN9kfnSZ8GXxA0CTnsAja0q2fzoYKMiFnZsvgl19Eezmi0FujYGmvkBrXxS6+OBCvgdOpDvh0GVHRCcX4VhSBs5czFTD0XacuqQWs4/g7aqqzA1V5xLEW/p7VHlcuVxEyBS0kqHN/LZIf1tY9nbDYtguQ+Q4/Ss1FAzURHVdKjdMoXotBXn6uc8NAV1mWjPQ2QMB7YDGEebPSTxcfs5wuZYA0KR4GaRexwG4dTZyOtyPE0kZOH98H/wP/RdH8oPxYdZwVSqX6VYbpR3B6aNOWKR5IA0e0Ons0ayxmwqWZQVRY9CV9ULz7aUyqFZLS393/HNgS4zuHMoJaaheY2cyovpAqtSl41v2JSArFci6WHz/Uqn7l/T3DSX0uz8H2t+lv3/4Z2DJg/psZQ//rtq/pdq8/eLecM/VJ0wpgh2uaG5I1TyQDRfkwhE5mpP+FvrbXM0Ry4r6Y3uRfqx6MC7iVvvtSNK8saKoZHx7D7sYONgVqv1z4YQCnWFxQYGdEwp1zqqXvb29Ts3BLh3k9Lc6nL+cjfTi6V+DG7ngkQEt8LceYXBnVT3ZCHYmI2qIJXXTUvf15Gfrg7ZLo5JtklL0xpeNmcq83ZzQq0VjwMsXuJIL5KZBBw3edplquZYbbhiJjPYDVXB1i9+MgOXfocCvDaY+9KoKtPb2dnBbMA26i/pc5WWShGdF0pnOBbBzBnSuQL9JQO8JSM/Jx6LtJxGz5Sf8kdYSM37JwUfrjmNcn3A81DccPu5OFT8XRFaOVd9EDZGj69Vt6gGt9UtpE3eWdJiTDGfGUnk2UJBTvOQWr+eq9aCIfkCAPhEKCpoAHe+Dg2cwGnuUzLsP3xb6anyT5xkXI01fnS9LzmXjYzLJy6ORGcDGt5Dr6Y3hjl/gzKVsfLD2OL7edBi394zEowNaIMTbtebPHVEdY6Amoop3mJPStiE3eEUFtQfuXHD19geWlF+NX5hXKoDLbbaaOc4oJw3wi4KzXyTW3nsjVh9MwMfrj+OTS+ORs9sJG3e1QVHTvug7ZBSat4jiX5lsFtuoici2SUlfLiIkxqedg92sq6cTTnYIhq55PzRuOxgI7wd4N7vuGHWi2sQ2aiJqOIqDtLCT6vznT6shbEnRa5F1YjPCco7Bv+ACcHypfpGA7hUKu2b99FnXJIGL9KBn4CYrxapvIqpfZEa31jcjoLU+9e3J+PNY//v/UHB6C3rYHUFHu1NwvHIOiF6iX8Qzh0qGzKlOdt6ASTIgIktioCaieq1lkxC0/Mc/cf7yWHy2+TQe2XUcbQpj0EsXg4FOR9HcNRsu7sEwdnNb9jgQvwu4bQ7Q5lY0dBm5BTiZlKHG2p9IzkDcpSzVm1/G0ZcsOjU9reG+6WOuJtvkvrPJfckGR9fHQE1EDYL0AJ86qi2eGhyBL7e3wRfbzmBWVj7ssorg/9Z6PNy/Oe7vGQbPhGh973bTXvEHfwIOfK+vKpcq8+DOgEP9GQImM8alZOQZg7EhMJ9MzsCFtLITv9QEGRfv4qCDq5O9yvamAr6TPVzUffPA71p839fdGf0iGqN9SKMGk2CGncmIqEHKzC3Aot1x+GzzKWMw8nRxwEO9QvFwyzR4t+wF2BeXZVZMBPZ9U/JkmdVNhpfJ2HOzJcJ8bLqVKSrSEJ+ajRPJ6fpAnJSpArPcT8vOL/d5fh7OiAhwR0SAB8Ibu6tt2XmFyCkoRE5+kZpDPie/ELkm92XJzi9CrvG+fl95Tk3kbGzs7oQbWvljUJQ/BkT6q/nqbQnTXNbQySGi+i+voAgr9p/D/I0ncTJZP5GLs4MO93YPw2M3tECYrxuQdAQ4uR6I3apfpMRdHsmA5hcJtL5FTc5iJNGpjjqs5RYU4nRKpj4QF5eS5fZUcgZyC2QmmavJoUkaVAnGMj2r3KrF3xON3Eo67NVE6V2OIbc4aJsF/OL7uaaBPb/kvmyXz7Xt5EVVJW967J2aeKugPbCVPzo28ValdWvGQF1DJ4eIGg4pba45koiPN5zEgbjLapv82I/qGIzHB7VUmcWKdwTSz+vTnKYcB1KOFS/H9WlPDbr/A7h1lv5+Xibwbit9KfwfvwFObvrtkoRFSuCOLlU65is5+Wbtx4b7Zy9llTuvupO9Di383VVylZbGYOyhtkkVs61cXP0Zm4oNx5Kw8WiySu1qysfN0VjaviHS33yiHSvBQF1DJ4eIGh4p8W0/dRHzNpzE5uMpxu2DWwdgwqCW6BFeklL0KjIJS8oJfeD2bQ407a3ffuEA8MkNgGQze/4U8gv1VcTOi+6D05l1yPcKQ7ZXS2R6tsAVj+ZIdQtHinMzpNl5IadAX9KU/dWSV6gCsQTkpPTccg/F09mhJBAXB2O5lRoCay9tVlZCWg42HkvChqPJ2HI8xTgPvKG03SG0EQa18sfAqAB0DrOO0jYDdQ2dHCJq2A6eS8O8jSexMvqCsV21ezMf3NoxWGUFyy0VRHNKBVRDtW1+Xj5888/DI/8Stua3Us8VvzpNQTtdbLnvL8lPTmohOFkUghNyq4Uguqg5kuGjHndGHqI8shHm54XGweHGgBzlmAhf5yLYaUWALFILoO4XAkWFJfdNH2vcUr8Iqdo/uQ6wdzbv+X50lT4Nq7yGWgpMlnLWpQNeu9ElQ99WPifhE7j7vyWvu+514Oz2a7xmfsm6vZN+3HvkTUCvf151zuQiaG9sKjYeS1aB+/CFK2aPN3J1xIBIPwyKClDV5P6eliltM1DX0MkhIhLSLrpg00n8+Oc55BWW3cZbFXZ2Gpo4ZqC1QwIidRfQUnce4do5hBXFw68wSSVBKW1b+EScaz9BBeVWmXvgvvhuILA9MGFryU4fdgUunazcwUhCloH/1t+Xnu/z++unbH3uWMk+/x0GxBXP/V5RPf8J3Py2/r6kbH0vCrCzB6aZ5D5f9AAQ80vlXrfzA8Doj0vSwn7QSX+h8bfvAJfiZoq8TCRl67DheIoK3JuPJeNKTklpW7QP9cKgVgEYGOWvcpw71NGQMc5MRkRUg5r7uWPmnR3x9NBW+HLbGRxPylDDhdQiw4mM90vGEJf9eMl2GU8sndbsyutglpelD7aG9u/itvC+fW4AosL0+5x2ARxczGZnU9z9gLwMwE6nD4pyKxO4mK3LrSx2+vumc7g7eQDhAwBXfcndSErHbn76/aXnu7yv3BrWjYvJepMeJc939gJGvKnfbqrPRKD9neW8hqP5NvlcqmmhRcnzL53S9xvITQecPUu2L/snAk5txL1+rXCvfxQKh0TiFEKx8aIvfj7rgL/OZ+LguStqmbP+BLxcHFQPcgnaUlUe4FW1vgM1jcOziIjIthXkAokHgYxkIGpEyfa5vYHkI2U/x94ZBb4tccGxGaJzA7Hhkg8O5ATitBaMPOgvfNoEe6kOaRK0uzbzqdEJWlj1XUMnh4iIbDyAX5RaiaNA8rHi2+Le+oVld8Tb0eQfmJlzF/46l4ZGWjoG6/bhqBaGs06R6B/pp9q1b+0UAg/n6s0XxqpvIiIiB2cgsK1+MSWd0i7HmgTvY0ByjKpS792rH1Z06I+LGbmI2fIT+u2Yr6rLB+e8g1UHE/DboQQMbxckPfnqDKcQJSKihkVnr2/jlsW0qly69ksPeJn5zMMZ/VqFAAkDEO7bEsu79MOGo0lIvJIDnzqeBc0qZkSfO3cuwsPD4eLigl69emHXrl3X3P+HH35A69at1f4dOnTAypUr6+xYiYionrIr7lhn0GIg8NAv0N32gRp/LZ0JpVNhXbN4oF68eDEmT56MadOmYe/evejUqROGDx+OpKSkMvfftm0bxowZg4cffhj79u3D6NGj1XLw4ME6P3YiIqJ63+tbStA9evTAnDlz1HpRUZHq7PXUU0/hhRdeuGr/++67D5mZmfjll5Ixd71790bnzp0xf/78674fO5MREZGlVSYWWbREnZeXhz///BNDhw4tOSCdTq1v3769zOfIdtP9hZTAy9ufiIjIllm0M1lKSgoKCwsRGBhotl3WY2JiynxOQkJCmfvL9rLk5uaqxSA93XzydiIiImtm8Tbq2jZz5kw0atTIuLRtW6qbPhERkRWzaKD28/ODvb09EhMTzbbLelBQUJnPke2V2X/KlClIS0szLocPH67BT0BERFSPq76dnJzQrVs3rF27VvXcNnQmk/Unn3yyzOf06dNHPf70008bt61Zs0ZtL4uzs7NaDC5f1ueZvXDhQg1/GiIioooxxCCJedelWdiiRYs0Z2dnbeHChdrhw4e1xx57TPP29tYSEhLU4w8++KD2wgsvGPffunWr5uDgoL377rvakSNHtGnTpmmOjo5adHR0hd5v165d0sudC88BvwP8DvA7wO+AZulzIDHpeiw+M5kMt0pOTsbUqVNVhzAZZrV69Wpjh7GzZ8+qnuAGffv2xXfffYeXX34ZL774IiIjI7F8+XK0b9++Qu/XpUsXNaGKvL7p61aFdEyTNm+pTvf0NMnYQjxfNYTfMZ6v2sTvl+XOl5SkpdlWYpLVj6O2ZVeuXFEd1KTt28urOP8p8XzxO2Yx/J/k+aqP36963+ubiIjIljFQExERWTEG6mqQ3uQyR7lpr3Li+apJ/I7xfNUmfr9s43yxjZqIiMiKsURNRERkxRioiYiIrBgDNRERkRVjoK6GuXPnIjw8HC4uLiqvtkykQmXbtGkTRo0ahZCQENjZ2alJaqj8RDKSo10mVAgICFDT6x49epSnqxzz5s1Dx44d1bhWWWQ64VWrVvF8VdCbb76p/idNp2Umc6+++qo6R6ZL69atUVcYqKto8eLFmDx5suoBuHfvXnTq1EnlxU5KSqrZv1A9kZmZqc6RXNzQtW3cuBETJ07Ejh071Dz2+fn5GDZsmDqHdLUmTZqoYCO57ffs2YPBgwfj9ttvx6FDh3i6rmP37t345JNP1IUOXVu7du3U/NyGZcuWLagzVZ+lu2Hr2bOnNnHiRON6YWGhFhISos2cOdOix2UL5Gu3bNkySx+GzUhKSlLnbOPGjZY+FJvh4+OjffbZZ5Y+DKuWnp6uRUZGamvWrNEGDhyoTZo0ydKHZLWmTZumderUyWLvzxJ1FeTl5amr96FDhxq3ybzhsr59+/aavI4iUtMVCl9fX56N6ygsLMSiRYtU7UN5GfVIT2ptbrnlFrPfMSrf8ePHVdNdixYt8MADD6g8FHXF4kk5bFFKSor6QTAkDjGQ9ZiYGIsdF9U/MnG/tB3269evwolnGqLo6GgVmHNycuDh4YFly5ap5AlUNrmYkSY7qfqm65M+SAsXLkRUVJSq9p4+fToGDBiAgwcP1klCJgZqIisv9ciPQZ22h9kg+QHdv3+/qn1YunQpxo0bp9r6GayvFhcXh0mTJqn+D9IRlq5v5MiRxvvSni+Bu1mzZliyZAkefvhh1DYG6irw8/ODvb29SlFmStaDgoJq6m9DDdyTTz6JX375RfWYlw5TVD4nJydERESo+926dVMlxQ8++EB1lCJz0mwnnV67du1q3CY1hPI9mzNnDnJzc9XvG5XP29sbrVq1wokTJ1AX2EZdxR8F+TFYu3atWRWlrLNdjKpL+ttJkJbq23Xr1qF58+Y8qZUk/48ScOhqQ4YMUU0FUgNhWLp3767aXeU+g/T1ZWRk4OTJkwgODkZdYIm6imRollSvyRe8Z8+emD17turAMn78+Jr9C9WjL7bp1efp06fVj4J0kGratKlFj80aq7u/++47rFixQrV/JSQkqO2SB9fV1dXSh2d1pkyZoqom5XuUnp6uzt2GDRvw22+/WfrQrJJ8p0r3d3B3d0fjxo3ZD6Iczz33nJoHQqq7z58/r4blygXNmDFjUBcYqKvovvvuQ3JyMqZOnap+SDt37ozVq1df1cGM9GR864033mh2oSPkYkc6aZD5BB5i0KBBZqfliy++wEMPPcRTVYpU444dO1Z18pGLGWlDlCB900038VxRjYiPj1dB+eLFi/D390f//v3VPAdyvy4wexYREZEVYxs1ERGRFWOgJiIismIM1ERERFaMgZqIiMiKMVATERFZMQZqIiIiK8ZATUREZMUYqImIiKwYAzUR1Ro7OzssX76cZ5ioGhioieopmW5UAmXpZcSIEZY+NCKqBM71TVSPSVCWOcJNOTs7W+x4iKjyWKImqsckKEuOdNPFx8dHPSala0kAIpmnJCtXixYtsHTpUrPnSzrEwYMHq8clu9Jjjz2mMqGZ+vzzz9GuXTv1XpL2T1J0mkpJScEdd9wBNzc3REZG4ueffzY+lpqaqtIrSnIDeQ95vPSFBVFDx0BN1IC98soruOuuu3DgwAEVMP/2t7/hyJEj6jFJ2zp8+HAV2Hfv3o0ffvgBf/zxh1kglkAvaTklgEtQlyAcERFh9h7Tp0/Hvffei7/++gs333yzep9Lly4Z3//w4cNYtWqVel95PT8/vzo+C0RWTiOiemncuHGavb295u7ubra8/vrr6nH593/88cfNntOrVy9twoQJ6v6CBQs0Hx8fLSMjw/j4r7/+qul0Oi0hIUGth4SEaC+99FK5xyDv8fLLLxvX5bVk26pVq9T6qFGjtPHjx9fwJyeqX9hGTVSPSQ5wQ35rA19fX+P9Pn36mD0m6/v371f3pYTbqVMnuLu7Gx/v168fioqKcPToUVV1fv78eQwZMuSaxyD5oQ3ktby8vFQOaTFhwgRVot+7dy+GDRuG0aNHo2/fvtX81ET1CwM1UT0mgbF0VXRNkTblinB0dDRblwAvwV5I+3hsbCxWrlyJNWvWqKAvVenvvvturRwzkS1iGzVRA7Zjx46r1tu0aaPuy620XUtbtcHWrVuh0+kQFRUFT09PhIeHY+3atdU6BulINm7cOHzzzTeYPXs2FixYUK3XI6pvWKImqsdyc3ORkJBgts3BwcHYYUs6iHXv3h39+/fHt99+i127duG///2vekw6fU2bNk0F0VdffRXJycl46qmn8OCDDyIwMFDtI9sff/xxBAQEqNJxenq6CuayX0VMnToV3bp1U73G5Vh/+eUX44UCEekxUBPVY6tXr1ZDpkxJaTgmJsbYI3vRokV44okn1H7ff/892rZtqx6T4VS//fYbJk2ahB49eqh1aU9+//33ja8lQTwnJwezZs3Cc889py4A7r777gofn5OTE6ZMmYIzZ86oqvQBAwao4yGiEnbSo8xknYgaCGkrXrZsmerARUTWi23UREREVoyBmoiIyIqxjZqogWKrF5FtYImaiIjIijFQExERWTEGaiIiIivGQE1ERGTFGKiJiIisGAM1ERGRFWOgJiIismIM1ERERFaMgZqIiAjW6/8BZX/3EzbaTIMAAAAASUVORK5CYII=",
"text/plain": [
"<Figure size 500x300 with 2 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n",
"examples_seen_tensor = torch.linspace(0, examples_seen, len(train_losses))\n",
"\n",
"plot_values(epochs_tensor, examples_seen_tensor, train_losses, val_losses)"
]
},
{
"cell_type": "markdown",
"id": "dbd28174-1836-44ba-b6c0-7e0be774fadc",
"metadata": {},
"source": [
"- Above, based on the downward slope, we see that the model learns well\n",
"- Furthermore, the fact that the training and validation loss are very close indicates that the model does not tend to overfit the training data\n",
"- Similarly, we can plot the accuracy below"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "yz8BIsaF0TUo",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 307
},
"id": "yz8BIsaF0TUo",
"outputId": "3a7ed967-1f2a-4c6d-f4a3-0cc8cc9d6c5f"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAEiCAYAAADONmoUAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAU6JJREFUeJztnQdYk9cXxl/ZIjhw40TFrbj33qNWrbtaqVr9a11ttbVaV22tdmmts9qqXe7dukq17r03bsGB4gSRTf7PuTEhICCBQCB5f4/fY+6XL993c0nyfufcc8/JotFoNCCEEEJIumOT/pckhBBCiEARJoQQQswERZgQQggxExRhQgghxExQhAkhhBAzQREmhBBCzARFmBBCCDETFGFCCCHETFCECSGEEDNBESaEJEiTJk3wwQcfcHQISUMowoSkEe+++y6yZMnyytamTRuOOSFEYaf9jxCSFojgLlmyJM4+R0dHDjYhREFLmJA0RAS3QIECcbZcuXKp53bt2gUHBwfs3btXf/w333yDfPny4f79+6q9bds2NGjQADlz5kTu3Lnxxhtv4Nq1a/rjb968qazrVatWoWHDhsiaNStq1qyJy5cv4+jRo6hRowZcXFzQtm1bBAYGxrHSO3XqhM8//xx58+ZF9uzZMXjwYERERCT6XsLDwzF69GgUKlQI2bJlQ+3atdV70HHr1i106NBBvT95vkKFCtiyZUui55s3bx48PT3h5OSE/Pnzo2vXrvrnYmJiMG3aNHh4eKj35OXlhTVr1sR5/blz59T7kvcnr3/nnXfw8OHDOO70ESNG4JNPPoGbm5sa+8mTJyfr70ZIekERJsTMc64iHs+ePcPJkycxYcIE/Pzzz0pUhJCQEHz00Uc4duwYduzYARsbG3Tu3FmJlCGTJk3C+PHjceLECdjZ2eHtt99W4jNr1iwl8levXsXEiRPjvEbOd/HiRSWky5cvx7p165QoJ8awYcNw8OBBrFixAmfOnEG3bt2UpX/lyhX1/NChQ5VQ79mzB2fPnsXXX3+tBDIh5P2IQE6ZMgW+vr7qZqNRo0b650WAf/vtNyxYsADnz5/Hhx9+iD59+mD37t3q+adPn6JZs2aoWrWqOpe8Xm5cunfvHuc6v/76q7ohOHz4sLrBkev5+PgY/bciJM2QUoaEENPj7e2tsbW11WTLli3ONnXqVP0x4eHhmipVqmi6d++uKV++vGbgwIFJnjMwMFBKj2rOnj2r2jdu3FDtn3/+WX/M8uXL1b4dO3bo902bNk1TpkyZOH1zc3PThISE6PfNnz9f4+LioomOjlbtxo0ba0aOHKke37p1S72XO3fuxOlP8+bNNWPHjlWPK1WqpJk8eXKyxmbt2rWa7Nmza4KCgl55LiwsTOPs7Kw5cOBAnP0DBgzQ9OrVSz3+4osvNK1atYrzvL+/v3rfvr6++v43aNAgzjE1a9bUjBkzJll9JCQ94JwwIWlI06ZNMX/+/Dj7xDWqQ9zRf/75JypXroxixYph5syZcY4VK1MsWLHkxNWqs4D9/PxQsWJF/XHyeh06K7pSpUpx9j148CDOucXF6+zsrG/XrVsXz58/h7+/v+qLIWLZRkdHo3Tp0nH2i+UrbnJBLNshQ4bgn3/+QYsWLdClS5c4/TKkZcuW6holSpRQ1rRsYuFLf8Rqf/HihTrGEHGVi+UrnD59Gv/991+Clra463X9jH/9ggULvjIOhJgTijAhaYi4QkuVKpXkMQcOHFD/P378WG3yGh0yxypitWjRIri7uysRFvGNP3drb2+vfyxzxAnti+/CNgYRZ1tbWxw/flz9b4hOCN977z20bt0amzdvVkIsLuXvv/8ew4cPf+V8rq6uynUurnA5Vm40ZL5W5rHlWoKcR+afEwpqk2NkbMTlHR8R2oTGxRTjQIipoQgTYkbEapP5ThHZlStXwtvbG//++6+a+3306JGaL5XnJOhK2Ldvn8muLdZkaGioCnwSDh06pAS1SJEirxwrFqhYwmJF6vqSEPJaCfCSbezYsarvCYmwIHPXYjHLJnPaEny2c+dOZQGL2Iq137hx4wRfW61aNaxduxbFixdX5yEks8JPLyFpiLhrAwIC4n7p7OyQJ08eJWoSbCTWY79+/ZRLVlzIYj1+/PHHKspYXL0LFy5U1p2I0qeffmqyvok1PWDAABXQJVHWIoQSfCU3APER927v3r3Rt29f1T8RZYm2luAucfm2b99eBZlJtLIc++TJE+UuLleuXILX/vvvv3H9+nUVjCXvU6KoxUItU6aMspIlCltuTmSfRIdL4Nr+/ftVFLfcqEgQmAh8r1699NHP4saWoDEJbItvrROSUaEIE5KGSNSuoXtUEKG5dOkSpk6dqpb1iCAJcpwIrghLq1at1JytiIrMtYoLWl73448/qqhqU9C8eXO1REiEUG4W5LpJLeGR9c5ffvklRo0ahTt37qgbiTp16qhlU4LcVIg43r59W4ml3FTEn+PWIVavRGPL9cLCwlQ/JEJbljUJX3zxhVo6JS5tEWs5XqzfcePGqefFNS+iPGbMGDVW0n9x28s1E7qJICSjkkWis8zdCUJI+iLrhGWZz4YNGzj0hJgR3jISQgghZoIiTAghhJgJuqMJIYQQM0FLmBBCCDETFGFCCCHETFCECSGEEDNBEU4hc+fOVdl6pAyblHQ7cuQILBGpiCPpAWVdpqT8i7+kRVa4ScpBWeMqmZck+5Guqo4OScUoiR5k7ais95QEEbrUhDqkKo9kYpLxlKxLUvEmoyNrWKVsoCSXkPKDUhpQMlwZImtgZe2sJN2QbFSST1lXplCHJOGQZBeSN1nOI4k6oqKi4hwj6R1lnaxkkpI0mEuXLkVGRvJlSxIP+ZvLJnmpt27dCmsfl8SYPn26+n5JwhMd1jxGkydPVuNhuJUtW9YyxyZdykRYGCtWrNA4ODhoFi9erDl//ryqfJMzZ07N/fv3NZbGli1bNJ999plm3bp1qkLN+vXr4zw/ffp0TY4cOTQbNmzQnD59WvPmm29qPDw8NKGhofpj2rRpo/Hy8tIcOnRIs3fvXk2pUqX01XCEZ8+eafLnz6/p3bu35ty5c6oKUNasWTU//fSTJiPTunVrzZIlS1SfT506pWnXrp2maNGimufPn+uPGTx4sKZIkSKqotGxY8c0derU0dSrV0//fFRUlKZixYqaFi1aaE6ePKnGO0+ePPrKRML169dVVaGPPvpIc+HCBc3s2bNVRaNt27ZpMiqbNm3SbN68WXP58mVV1WjcuHEae3t7NVbWPC4JceTIEU3x4sU1lStX1letsvYxmjRpkqZChQqae/fu6TepIGaJY0MRTgG1atXSDB06VN+W0m/u7u6qXJwlE1+EY2JiNAUKFNB8++23+n1Pnz7VODo6KiEV5MMtrzt69Kj+mK1bt2qyZMmiL4s3b948Ta5cuVRZPx1Sbs6w9F5m4MGDB+q97t69Wz8WIjyrV6/WH3Px4kV1zMGDB1VbfhxsbGw0AQEBcUoKSpk/3Xh88skn6gfJkB49eqibgMyE/I2l5CLHJZbg4GCNp6enxsfHJ07pSGsfo0mTJqkb94SwtLGhOzoF+Xalkoy4XXVImjxpS8Fza+LGjRsqL7LhWOTIkUO553VjIf+LC7pGjRr6Y+R4GTMpz6c7RlInSlk/HZJPWVy7koM4syD5jQ1LFcrnJDIyMs74iEutaNGiccZH8kXryg/q3ntQUJAqZq87xvAcumMyy+dN0llK+s2QkBDllua4xCIuVXGZxv/7coygprVkGkzKXcp0lriXLXFsKMJGIjVd5UfF8I8rSDt+on5LR/d+kxoL+V/mY+IXMBChMjwmoXMYXiOjI4UGZD6vfv36+jq/0ne5sZCbkKTG53XvPbFj5AdFqiBlVKQGsczXyXybVFVav349ypcvb/XjokNuTKSco8QWxMfaPzu1a9dW87OSe13iC+SGX2JGgoODLW5sWMCBEBNZNOfOnTNpqcHMjhScOHXqlPIQrFmzRlU/2r17t7m7lSHw9/fHyJEj4ePjo4IRSVykGpcOCfATUZYCHatWrdKX3rQUaAkbiVSOkTJp8SPxpF2gQAFYE7r3m9RYyP9Sg9YQiVCUiGnDYxI6h+E1MjJS/k8qIUnpvsKFC+v3S99l+kIKJSQ1Pq9774kdI1HHGfkHSawViTitXr26svakKtSsWbOsflx0LlX5XkhkrniGZJMbFKmSJY/FIrPmz058xOqVEplSrtLSvlcU4RT8sMiPitRRNXRFSlvmu6wJDw8P9UE2HAtx5chcr24s5H/5ssiPjg4p3C5jJne3umNkKZTM8+gQC0EsKak1m1GRWDURYHGzynuS8TBEPif29vZxxkfmuWVuy3B8xG1reKMi711+CMR1qzvG8By6YzLb503+5lJykOOiLSMpf3fxFOg2iZuQuU/dY352YpEljdeuXVNLIS3u85OuYWAWtERJIoCXLl2qon8HDRqkligZRuJZChK9KSH+ssnHZcaMGerxrVu39EuU5L1v3LhRc+bMGU3Hjh0TXKJUtWpVzeHDhzX79u1T0aCGS5Qk2lGWKL3zzjtqCYuMrywdyOhLlIYMGaKWZ+3atSvOUooXL17EWUohy5Z27typllLUrVtXbfGXUrRq1Uotc5LlEXnz5k1wKcXHH3+sokDnzp2b4ZeZfPrppypK/MaNG+pzIW2JiP/nn3+selySwjA62trHaNSoUep7JZ+f/fv3q6VGssRIViBY2thQhFOIrCmTD4GsF5YlS7IG1hL577//lPjG37y9vfXLlCZMmKBEVG5MmjdvrtaFGvLo0SMlui4uLmqJQL9+/ZS4GyJrjBs0aKDOUahQISXuGZ2ExkU2WTusQ25G3n//fbU8R77wnTt3VkJtyM2bNzVt27ZVa6Plh0Z+gCIjI1/5O1SpUkV93kqUKBHnGhmR/v37a4oVK6b6Kz9+8rnQCbA1j4sxImzNY9SjRw9NwYIFVZ/l90DaV69etcixYRUlQgghxExwTpgQQggxExRhQgghxExQhAkhhBAzQREmhBBCzARFmBBCCDETFGFCCCHETFCEU4Fk/5Hi0/I/4fjws2M6+N3i+FjLZ4frhFOBpGiU0n2SoF7SoRGODz87poHfLY6PtXx2aAkTQgghZoIiTAghhJgJq6snLGX0Tp48qUqF2dik7h5ECkwLd+7cUS4QwvHhZ8c08LvF8cnMnx2pGCZlEatWrapKUyaF1c0JHz16FLVq1TJ3NwghhFg4R44cQc2aNZM8xuosYbGAdYMjtSkJIYQQU3Lv3j1l7On0JimsToR1LmgR4MKFC5u7O4QQQiyU5Ex5MjCLEEIIMRNmFeE9e/agQ4cOcHd3R5YsWbBhw4bXvmbXrl2oVq0aHB0dUapUKSxdujRd+koIIYRYlAiHhITAy8sLc+fOTdbxN27cQPv27dG0aVOcOnUKH3zwAd577z1s3749zftKCCGEmBqzzgm3bdtWbcllwYIF8PDwwPfff6/a5cqVw759+zBz5ky0bt3apH2Ljo5GZGSkSc9JSEbAwcEh1cvzCCGmIVMFZh08eBAtWrSIs0/EVyxiUyErtgICAvD06VOTnZOQjIQIsNzMihiTjElYZDSO3XyCyOgYc3fF6sjr6oiKhXKk2/UylQiLOMYP+Za2LMgODQ1F1qxZX3mNJPE2TOStW8id1DVEgPPlywdnZ2c1V02IpSBJBO7evauWUBQtWpSf7wzIzkv3MWnTefg/DjV3V6ySNyoXxJy3q6Xb9TKVCKeEadOm4fPPP0+2C1onwLlz507zvhFiDvLmzauEWLLH2dvb84+QQbj95AU+/+sCfC7cV+08Lg5wz/mqYUHSlqJuzkhPMpUIFyhQQKUCM0TaUikjIStYGDt2LD766CN9W1KZlS9fPsFjdXPAYgETYqno3NBy00kRNj/hUdH4ee8NzN55BWGRMbCzyYIBDTwworknsjlmqp9okgIy1V+4bt262LJlS5x9Pj4+an9iyFIm2XQkJ5coXdDEkuHnO+Ow/+pDTNh4DtcDQ1S7tocbvuhUEaXzu5q7a8QaRPj58+e4evVqnCVIsvTIzc1NzVeJFSuW62+//aaeHzx4MObMmYNPPvkE/fv3x86dO7Fq1Sps3rzZjO+CEEKM435QGL74+wL+PnNPtfO4OGJ8+3LoWEWbM4FYD2Zdp3Ds2DFVZUI2QdzG8njixImqLcEjfn5++uMlolMEV6xfWV8sS5V+/vlnky9PIlqKFy+OH374IdnDIYlU5AeEkeWEJExUdAx+3nsdzb/frQTYJgvwbr3i2DGqMTpVLUQBtkLMagk3adJELQlKjISyYclrpBQhieV1d86TJk3C5MmTU1RxKlu2bMk+vl69eurGKUeO9AvvJySzcPTmY0zYcA6XArQrNKoWzYkvOlZM1+UwJOORqeaEScKI8OlYuXKl8iT4+vrq97m4uOgfy02PBOS8rsalLorW2IAfCZ6zRiIiIrjuliTIw+fhmLblEtaeuK3auZzt8WnbsuhWvQhsxBQmVg3T5lgAIny6TaxQsYx17UuXLsHV1RVbt25F9erVVZCaZBm7du0aOnbsqNZZi0hLzct///03SXe0nFfc/507d1YR5J6enti0aVOi7mjxZOTMmVOlFZXsZnKdNm3axLlpkGUyI0aMUMfJsrAxY8bA29sbnTp1SvT9Pnr0CL169UKhQoVUPypVqoTly5e/sh72m2++UfnF5T1LjMHUqVP1z9++fVudQ+IPxNqvUaMGDh8+rJ579913X7m+JIQRL4wOeTxs2DC1P0+ePPopkRkzZqj+yDmLFCmC999/X8U+GLJ//371eul7rly51GufPHmiYh9kDAzXtQvSl3feeSfR8SAZk+gYDX4/dAvNvtulF+BetYpg56gm6FGzKAWYKCjCr0EsxxcRUWbZknLVG8unn36K6dOn4+LFi6hcubIShnbt2mHHjh3KvS/iKMU0DOfgE0LWXHfv3h1nzpxRr+/duzceP36c6PEvXrzAd999h99//10V7JDzjx49Wv/8119/jT///BNLlixR4iTR668r5BEWFqZuKCQ+4Ny5cxg0aJASKakRrUOC+uT9TpgwARcuXMCyZcv0iV7kvTdu3FgF/clNxOnTp1Wwnwi3Mfz666/K+pV+S0pVXTaqH3/8EefPn1fPS/CgnFuHBB42b95cLZOTDHByQyTjLt6Jbt26qf8Nb2wePHig3qcEIpLMw2n/p+g8b79yPweFRaGCe3ase78epr1VGbmyMVMZiYXu6NcQGhmN8hPNUyDiwpTWcHYwzZ9oypQpaNmypb4tFqAEt+n44osvsH79eiUAYuElhliJYkEKX331lRIcET8R8cTWXotAlSxZUrXl3NIXHbNnz1aCKda1INHv8ZehxUcsYEMhHz58uLK2JVJeCmlLVrRZs2apc4lVLcj1GzRooB6LIAcGBqo5bxkHQSxmYxFPgFjbhhimUBVPwpdffqmi+ufNm6f2yfFidevaQoUKFfSP3377bXVDIoIs/PHHH8qKN7TCScbl6YsIfLvdF8uO+EHuoV2d7DC6VRn0qVMMtnQ9kwSgCFsJ8sNviFiDEqwlVpa4h8UtLKk/X2cJixWtQ1yukihFrLXEEJerToCFggUL6o9/9uyZSrYiwqnD1tZWWblJWaViLcoNgIiuWLMyHysuXF2SFbH2pS0WZ0KINSpR+DoBTinSz/iIS1+ytMk0gFj1Mq5iuYtHQPon19YJbEIMHDhQTQ3I+5KbDXHpy40Pl61kbGJiNFhz4jamb72ExyERat9bVQthbLtyKhcxIYlBEX4NWe1tlUVqrmubivhRzmJJylIvcRWLFSgZx7p27aoELSniZ1gScUhKMBM6PrVu9m+//VZZujJfrZt/FQtU1/fEsqfpeN3z4lKO38eEKmrFH9ObN2/ijTfewJAhQ9T8s4i8uJsHDBig+iYi/Lpry82BeChkfrhVq1bKrc118BmbC3eDVMKN47eeqHbp/C4q6rl2Caa+Ja+HIvwaRDRM5RLOSMg8plhYOjewWMYiIumJBJHJPK24hRs1aqS3ck+cOIEqVaok2XcJKuvTp49qy03A5cuX9elIxU0sYifz3VJvOiFrXgLMZC47IWtYosJlrtkQsWBfl+Lx+PHjqi+yfl1XKlCs9fjXln4llc9c+iw3GGINS9UwCfAiGY/gsEjM9LmCXw/eVEFYzg62+KCFJ/rV94C9bSrDbeTG9skNIDqBcqo5CgGOLzNqhT4FggMAB2cgZ9HYYwIvAxojKzC55gey5tI+Dn8OPLsN2DkCbh6xxzy6lnCfkiJbXiDbyxuSyFDgyS3Axg7IYzAF9OQmEBlm3Hmlr9JnQfokfZPlmnnLxB7z1B+I0GYjSxZOOYDsBZGeWJ66kGQhQrVu3ToVFCQ3GhLAZGxgkimQ+Vxx34o1XrZsWTVHLJHCSblfpe9r1qzBgQMHVHSxRCSLW1snwk5OTirKWgKiJHCqfv36ag5YrEqxSmVOW9zZEnUs1xYXuQSnubu7qxSozZo1U9a2WKPSlnlZEWVdUpnEkPcgFrO8BxlXw4AtHTL/Lda7RE3LXLH077///lMuaomy1s0Li6di0aJF+mxxJOMgXpJNp+9i6uaLeBCsjWRvX6kgxr9RDgVzpLLgggjR2VXAgTnAw9hlhnHotQIo87IO++VtwPr/ASWbA++siz1mUVMgIm5U/mt5czZQra/2sd8h4M8uQEEv4H97Yo/54y2tYBpD80lAw5f5+wMvAQubANkLAR9diD1mzQDgzjHjzlt3GND65YqH5/eBebUBW0dggsH02JbR2jFKLlX7AB3nIj2hCFspIlwScSsJNuTHX0QrOXm1TY1cV8pH9u3bV80HS6SzLNmRx4kxfvx4XL9+XR0nLl55jQiqzDHrkJsKWQsta6alYpAIrYieIML3zz//YNSoUSrCW+ZtRcDnztV++eS88noRcZnPlXGS/p09ezbJ9yJuZBlXifgWsRXrXkReXqujdOnS6trjxo1Tc+FisdeuXVsf7KbzEHTp0kW5oZNaqkXSn6sPgjFx43kcuPZItT3yZMPnb1ZAo9LGral/hRePgWO/AIcXAiEvRUQExTF2jb8eWwOPjK0D4JwbcMoe95isblor1hjsnAzOa/fyvDletT7Dky4H+wr2BjcmNi/Pq7O4dch1ZL8x2BsU2slio329jJkh4jEw5rwOCYx3GpNFY8p1MJkAWR8q7j1/f38ULlw4znPygyv5qyU9plhTJP0Ra1zWFMsyKInYtlYkqEyipiX63NTwc248smRw9s6rKuVkZLQGjnY2GNa0FAY1LgFHu1TEbohVeXAecPJ3IPKFdl/2wkCdIVqrNL64kkyvM/GhJUzMyq1bt5RlKOt2JaJZlhXJjZC4ZK0RccVL0hPZDJcxEfMgNsr28/dVsYU7T0PVvhbl8mFShwooYoq6s+J2PrpI+zh/JaD+CKBC57jWLrFoKMLErEgAkyzDkTlQ+cGrWLGiWuYj1rA1IvPOIsTi0i5TxiDAhKQ7tx6FYNKm89jlG6jahXJmxeQ3K6Bl+ZfBQMYiMRdXfbTzoQUqavfVfV8bgCXzmyWaaAOLiFVBESZmRVw2EsBEtKR3hDp5lbDIaCzYfQ3zdl1DRFQM7G2z4H+NSmJo01LI6pAK1/POL4B9M4BybwI9ftfucysB9FnLP4MVQxEmhJCX/Of7AJM3ncetR9r52Qal8uDzjhVQMq9LyoKtosJjl7xU7g4c/UUrvBKKQ6uXUIQJIQS4+zQUU/66gG3nA9Rw5M/uiAlvlFdLj4zOVibBVofmAyd+B8p1AN76Sbs/XzlgtG/caGFi9dASJoRYLeJu/mXfDfy444rKEy/5nfvXL46RLUrDxdHIn8c7J4ADs4ELG2ITZcha3+go7ZIfgQJM4kERJoRYJQeuPVRrfq8+0Ca1qFXcDVM6VUDZAtmND7YS8b25N3Z/yWZAvREMtiKvhSJMCLEqHgSFYeqWi9h46q5q53FxwNi25fBWtULJdz3LXO+ZVcDBOdosULpEFBW7AvWGx0Y/E/IaKMKEEKsgKjoGvx28hZk+lxEcHqXiot6pUwyjWpVBjqzJXJcb+gQ4thg4/JM2VaLgmB2o/i5Qe7A2rzMhRpDKLOPEkpCatfHr4UohgaQQy2HDhg2pvrapzkNIQkiFow5z9mPK3xeUAHsVyYlNQxtgSseKyRdgYd0gYMcUrQDLet9WXwIfngNafUEBJimClrAFIMUCpHDAtm2vJirfu3evymF8+vTpOLWAk4NUN4pfri+1SA1jEVupSmSI1DSWYgyEmJJHz8Px9bZLWHXstmqL4I5pUxY9axaBjU0yXM93TwI5igDZtMU1UHMgEHRP63Ku+BYzW5FUQxG2AKQykCT8l3yl8fOULlmyBDVq1DBagHUl/dKLAgUKwBqROsNSUIKYlpgYDZYf9cM323zxLFRbeq9HjSIY07Ys3LIlc7y3fAIc+QloPAZoOk67z7OlduMaX2Ii6I62AKSQvAimpH80RGoEr169Won0o0ePVKWeQoUKqcpDUk5v+fLlSZ43vjv6ypUryqqW4hZSdcjHxyfBqkhSKUiuUaJECVWNSKx0QfondXTFKhf3s2y6Psd3R0vFIikpKFWGcufOrSolyfvRIbWQpcLQd999pyokyTFDhw7VXyshrl27puoQSw1jFxcX1KxZU6XINETyV8t7kExejo6OqjzhL7/8on9eyiHKeGfPnh2urq5o2LChOm9C7nxB+ih9NRxTKUwhlZXkHPK+XjduOv766y/VZxl/qXylqwU9ZcoUle4zPlKTWc5jbZy9/Qyd5x/AZ+vPKQEuVzA71g6pi6+7Vk5agCXYKuJlEQWhWF1tsFVYbHUuJb4UYGJCaAknF2MKQ+uQslq69YGyVjA6XFtyy3CtYGLndUi+G1hK9smPugjaZ599po/wFAGOjo5W4isCVr16dfVjLz/+UibvnXfeQcmSJVVJveRUN3rrrbeUgB0+fFiVDYwvOIIIk/RDavOKkA4cOFDtk7KAPXr0UHV5xW2uEz8p2xefkJAQVU5QavmKS/zBgweq0P2wYcPi3GhIHV4RYPn/6tWr6vwiPHLNhJAxkNKFU6dOVQIrtXrFle/r64uiRbUF0WUcDx48qKoXSWlCKSbx8OFD9dydO3fUTYiI7c6dO9U4SspNKYVoDHLjICUWJ02alKxxE+TvJaIrf1/pt1jQW7ZsUc9JqUW5uZGxEpEWpD7ymTNnVM1oa+HZi0h8948v/jh8SyWkknW+o1qVVsFXdrY2yQu2kupFDT7U7pf0kiPPcK6XpC0aK8Pf319KN6r/4xMaGqq5cOGC+v8VJmU3fju3Lvb18lj2LW4X97xfeyT8WiO5ePGiel///feffl/Dhg01ffr0SfQ17du314waNUrfbty4sWbkyJH6drFixTQzZ85Uj7dv366xs7PT3LlzR//81q1b1TXXr1+f6DW+/fZbTfXq1fXtSZMmaby8vF45zvA8Cxcu1OTKlUvz/Plz/fObN2/W2NjYaAICAlTb29tb9S8qKkp/TLdu3TQ9evTQGEOFChU0s2fPVo99fX1VP3x8fBI8duzYsRoPDw9NREREgs/HHz+hY8eOqq86pM+dOnV6bb/ij1vdunU1vXv3TvT4tm3baoYMGaJvDx8+XNOkSZMEj03yc54JiYmJ0aw55q+pNuUfTbExf6ttxPITmvvPXvP+Ht/UaLaM0Wi+LBj7vfupiZwwvbpOrFBn4kNL2EIoW7Ys6tWrh8WLFytLTSxDCcoSV6UgFvFXX32FVatWKYtOLClxvYr7MzlcvHhRuWjFUtMhlmp8Vq5cqaxIcdGK5SlWoliMxiDXEivUMCisfv36yhoXq1WscUHq7draxibUF6tYrMjEkP5IYJhYlRIIJn0LDQ2Fn5+fel6CxeR8UlYxIeR5cT/b26euzJzM0Rs7bnLtxCx8QZ4Ti3jGjBmqMtWyZcswc+ZMWDqXAoIwccN5HLn5WLVL5XPBlI4VUK/ky0CqxIKtJLnGeclsFa3dl6/CyzKCb9HdTNIVinByGadd2G+0O1pH2Q7ac4g72pAPEhcNY5G53+HDh2Pu3LkqIEtczTpB+fbbbzFr1iw1xyvzwSJw4k4WMTYV4sbt3bu3co2KO1lczStWrMD333+PtCC+GIobXoQ6MaRcosxjiztY5nplvrlr1676MZB2UrzueRE/rVEfS0Jz1PEjzpMzbq+7trjVxcW+fv16Fegl15X3Zqk8D4/CDz6XseTATUTHaJDV3hYjW3iif30PONjZJJLZ6l/gwI9xM1tJ+UDJbCUZrjjXS8wARTi5GDFHmyAyN6ybHzbleQ3o3r07Ro4cqawgmTccMmSIfn5Y5i4lKKlPnz6qLWJ1+fJlFWCVHKS+r7+/v7IgxeIUDh06FOeYAwcOoFixYmreUsetW7fiHCMCIVb5664l86MyN6wTLOm/iFxqauzKOSRIShfQJBanYelAuTmRcdm9ezdatGjxyuslwvzXX39VApeQNSzBcTI+OuR9yhx406ZNk+xXcsZNrr1jxw7069cv0bgAb29vdfMlY9yzZ8/XCndmRG5yNp+9hy/+voD7QeFqX5sKBTChQ3lV7zfBYKuzq7WWry6zVRZboGIX7TKjgsavGiDElDA62oKQiF8JTho7dqwSA8OoXE9PT2UFyg++uHv/97//4f79lxl/koGIkkTvyg+9RDeLq9tQNHTXENeuWHHiVhX3qlhmhkh0sAQ7iXtVAp7EJR4fsQolAliuJSImgVdi4Usgmc4VnRKkfxKoJNeW9/D222/HsZylb3JNcetKpLb0c9euXcqFL0hgWFBQkBK4Y8eOqWjx33//XbnIBYnmFle3bJcuXVI3QU+fPk1Wv143bhLEJdHs8r/8/cTt/vXXX8c5RoLXJGBMAt/kPVga1wKf451fjmDYspNKgIvldsbSfjWx4J3qCQuwsLgNsHGoVoAdXIC6w4CRp4EuiyjAJENAEbYwxCX95MkT5dY0nL8dP348qlWrpvbLnLGsy5XlM8lFrFARBplDlWhq+cGXKGND3nzzTXz44YdKrCRKWQQ//hIZWc/cpk0bZR2K5ZjQMimZp96+fTseP36son3Frdq8eXPMmTMHqUHmSyUhiMydi/tWxkLGxJD58+er673//vtqnl3mWsUiF2QZlIicWNDi5pdo80WLFumtYhE+EXGJsJbnZanR66zg5I6b/M0k2n3Tpk3qGBH8I0eOvCLm8t6k37Vr14alEBoRje+2+6LND3uw7+pD5W7+oIUntn/QCE3K5It78FM/7UoEHRU6Aa4FgZZTgA/PA62nAjmLpPt7ICQxskh0FqwISWghAUbiWo2f2CIsLExZPx4eHsoSIyQzIV9lEWK5gfjoo48SPS4zfc59LtzH5E3ncedpqGo3LZMXk9+sgGK5E5jG2fIxcPQXrZUr7mYhMlTrfrZjQhSSMXQmPpwTJsQCCAwMVO7sgICAROeNMxP+j18o8d1x6YFqi7t5YofyaFU+f2ylI539oGs759ZGO/sfiRVh1u8lGRyKMCEWQL58+VQWrYULF2bqHNzhUdH4afd1zP3vKsKjYmBvmwXvNSyB4c1Kwdnh5c9VVIQ22ErKCDafBJRpo91faxBQpi1Q0Mus74EQY6AIE2IBWMKs0p7LgZi06TxuPNTOwdcrmVtVOZK1v4rQp8DxJdrMVsEvo9CPLooVYWc37UZIJoIiTAgxK/eehaolR1vOBqh2PldHjH+jPDpULqh1PUuw1aEFwIlfgYiX+cMl2Erq90odX0IyMRRhQohZiIyOwZL9N/DDv1fwIiIatjZZ4F23OD5s6QlXJ3vg3mnt+t5z6+JmtlJlBLsw2IpYBBThBEgq6xIhmZ2M4Lo+dP0RJm48h8v3tZZtjWK5lOu5fEFX4OoObWarG7vjZbYaDpRszsxWxKKgCBsgmYZkPezdu3fVGlZp6yMxCbEQAZZIavlcpzYHdkp4EByGaVsuYf3JO6otpQXHti2LLtUKw0as3YVNgHun4mW2GsZgK2KxUIQNEAGWtZOSbUqEmBBLRARY1i4aFr9IayS/8x+HbqmkG8HhUWpV0du1iuLjpoWRM6cumtsOyFceeHRVO9crc75MrEEsHIpwPMT6ldqyUsXmdTmOCcmMiAWcngJ8wu8JJmw4h/N3g1S7UqEc+LJjBXhd+h6YtxTovw0oUFF7cItJQJtpQNac6dY/QswJRTgBdK46c7jrCLEUnoRE4Jvtl7D8iL9qZ3eyw8dtyioLWIKwcMgPiAgGzq2JFWHXAubtNCHpDEWYEGJSYmI0WHXMH19vu4QnL6SUowbjytzDu/gLDp4zARFgofGnQNW+QKnm/AsQq4UiTAgxGefuPMOEjedw0u8p7BGFYW4n8L7DVjjf0laawsG5wBsztI/zl9duhFgxFGFCSKoJCovEjH8u47eDN+GiCcFwh/8wOKsPsr0IBF5IsIULUM0bqDOEo02IARRhQkiqljxtOHUHUzdfgsPzOxhrtw19HHYha8wLIDxeZisGWxHyChRhQkiKuHw/WEU9P795Ap/ZbcabTgdhixjIP7XUSGW26srMVoQkgQ3MzNy5c1G8eHFV11QKkccvVG5IZGQkpkyZgpIlS6rjvby8sG3btnTtLyHWTkh4FKZtuYjus7Zj+O1R2Ow4Dp1t92sF2KMx0HstMOQAUOVtCjAhGdkSXrlypSo+vmDBAiXAP/zwA1q3bg1fX19Vmi0+48ePxx9//IFFixahbNmy2L59Ozp37owDBw6gatWqZnkPhFiT63nr2Xv4YvNF3HsWBsAJhV2joImwRZaKbwF1hwHuVczdTUIyFVk0ZkwkK8Jbs2ZNzJkzR5+zuUiRIhg+fDg+/fTTV453d3fHZ599hqFDh+r3denSBVmzZlXinBxu376truHv76+yBhFCXs+N+09waNmXqPFkK7pETEYOtzz4/M0KaJb9nrZ8YM6iHEZCUqAzRlvC4jru378/3n33XZVZKqVERETg+PHjGDt2bJy0kS1atMDBgwcTfE14eLhyQxsiArxv375EryOvkU1HcHBwivtMiFURFYG7z6OxeN8NFfX8l+02eNrcwaxyF1D37QlwspesW/nN3UtCrGtO+IMPPsC6detQokQJtGzZEitWrIgjcsnl4cOHKi1k/vxxv8TSDgjQ1hWNj7iqZ8yYgStXriir2cfHR/VFcj0nxrRp05AjRw79Vr481yUSkihB94BjSxC8+C2EfVUMb3zzF37edwMR0RpszjcIgc1nomnvsS8FmBBiFhE+deqUCqAqV66cch0XLFgQw4YNw4kTJ5CWzJo1C56enmo+WHI8yzX79eunLOjEEEv72bNn+u3ChQtp2kdCMhUyGxVwFtj9DTQLmwIzygJ/fwBXvx1winmBWjiPuiVyY0m/mvhw6AjkbdgfsHM0d68JsRhSHJhVrVo1tX3//feYN28exowZg/nz56NSpUoYMWKEEsekygDmyZNHJZG/f/9+nP3SLlAg4fyxUl5ww4YNCAsLw6NHj9Qcscwdi1WeGI6OjmrTERSkTSJPiNUSFQ7c3Af4btVuQbfVbt239VRMSeyIqY7wUm0wtHlzVCrCYgqEZDgRluVC69evx5IlS5RbuE6dOhgwYICakB43bhz+/fdfLFu2LNHXiyVbvXp17NixA506dVL7xMUsbbFwk0LmhQsVKqT6sHbtWnTv3j2lb4MQ6+H8eu12dQcQ8Vy/OwwO2BtdCf/GVMNB2+poUdML/eoXRxE3Z7N2lxBrwGgRFpezCO/y5cuVG7hv376YOXOmchHrkGVDEvX8OmR5kre3N2rUqIFatWqpJUohISHKihbk3CK2Mq8rHD58GHfu3EGVKlXU/5MnT1bC/cknnxj7NgixfB5fB9wMvERn1wCX/lYPg+3zYFuEF7ZGVsWBmApwdc2uhHdcrWLI4czqYYRkWBEWcZWALHE9iwWbULk/Dw8P9OzZ87Xn6tGjBwIDAzFx4kQVjCXiKsk3dMFafn5+ceZ7xQ0ta4WvX78OFxcXtGvXDr///jty5qS7jBA90VHAggZA4EVg2HEgTynt96lYF1wKdMP8gNI4FVYcGtjAM58LpjQqgY5V3OFox2ArQjL8OuFbt26hWLFiyKxwnTCxKMKCgKv/Ag8uAs0+i93/65vArQPQvLUIex0aYNHe69h75aH+aQm2GtSoBBqXzgsbXWlBQkjGXyf84MEDZbVKog1DxFUsgVbiWiaEpCFP/QDfbYDvFm2AVUzkSzfVAMBVG9QY0XYmtt2IxLx/H+BSgDYVrK1NFrSrVBADG3qgcmF6jwjJCBgtwpKtSuZg44uwzNF+/fXXSowJISYkJga4exK4/DKa+f65uM/nLgWUaauWG0lJwRVH/LB4300EBElqScDZwRY9ahZB//oeDLYiJLOLsKyzlaVJ8ZHczVyDS4iJiHgB3NitFd3L24DnBkv5stgAResCpdtoxTePJ+4+DcXSfTex7PAZPA+PUofldXXEu/WKo09tBlsRYjEiLGtuZS1v/LW5krXKzo6VEQlJNRKmMaemfv2uwsEVKNUcKNMO8GypzdcsN8V3g7Bo5Sn8dfouomK04R0SbDWQwVaEZAqMVs1WrVqpLFQbN25UaSCFp0+fqrXBEjVNCDEysOrIT8Dt40Cv5YAkuJGteAPg1n6tpStbsQb6soASS7nvSiAW7okbbFWnhBv+16gkg60IsWQR/u6779CoUSMVIa0rHyhpLGVZkSwXIoQkQVSE1sLVrd+1dQD2zgAiXwABZ4CCXtr97b8HHLJpBfklkdExyuIV8b0UoC1EIoHN7Su7M9iKEGsRYUmecebMGfz55584ffq0qmIkyTV69eqV4JphQqyeF4+BKz7awCrJVpXdHRj6MoDR3gloNBpwzg3kKBI7VI4u+ofBYZFYfsQPS/bffFnHl8FWhFgKKZrEzZYtGwYNGmT63hBiKTy8GhvN7HcI0ETHPvfCCUqYX87rouGoBE9x71moEt7lh/0QHC/YqnftosjprHVPE0IyLymOpJJIaMloJXWBDXnzzTdN0S9CMl+WqttHYosiPLoS9/l8FV7O77YD3KtK8exETyXBVj/vvY5NBsFWpfK5YFDDEuhYlZmtCLFqEZaUkZIb+uzZs6pKki7hlq5iktQIJsRqCA8GNo8GrvwDhD6O3W9jrw2uEuGVpUS5ks4yp4Ktrj58Jdiqtocb/te4BJqUzsfMVoRYIEaL8MiRI1VuaKl2JP9LXWEpKzhq1CgVtEWIRfPUH3h0FSjZVNt2cAFu7tUKsFNOoHRrrfCWbA44ZX/t6STY6u8zEmx1AxfvBemDrbSZrUrAi2UECbFojBbhgwcPYufOnaoesBRXkK1Bgwaq0pHUET558mTa9JQQcyPLiH5uBmR1Az6+CtjYaqOX20zXBlYVqQ3YJu8rJcFWK474Y/H+G/pgq6z22sxWAxowsxUh1oLRIizuZldXV/VYhPju3bsoU6aMWrLk6+ubFn0kJH2JDAWu79YGVrkWBJp8qt0vy4ec86gMVQgJ1OdpRvnkx0FIsNXS/ZLZKjbYKo+LoyojyGArQqwPo0W4YsWKammSuKIlf/Q333wDBwcHLFy48JUsWoRkGp4/0KaHlKCqa/8BUaHa/bJsqPEYrcUrVu4HZwEH44vdi6tZKhltOhUbbFUybzZVyahjlUJwsmcZQUKsEaNFWOr5hoSEqMdTpkzBG2+8gYYNGyJ37txYuXJlWvSRENMjAYUPLsRGM985Ljtjn89eODZblRyrS5phhABLsNX+q4+wcO917LkcGCfYSsS3aRkGWxFi7Rgtwq1bt9Y/LlWqFC5duoTHjx8jV65c+ghpQjJstipJBaks3i3akoCGyNIhWUIkwpu/YpxsVcYgwVabz9xTkc4XDIKt2lYqqJYZMdiKEJIiEY6MjFQZsiRNpbildbi5vUw6QEhGLAOoW5P7zB/4vVPsc3ZOgEfj2GVE2Qum6lISbLXyqD8W77uBuwy2IoSYWoQlLWXRokW5FphkfPyPADumANnyAN2WavflLqkthOBWXGvxlmiizc+cSgKehWHJ/hsMtiKEpL07+rPPPlMVk6RYAy1gkiGIiQZuH9Wu2S3w0kNja69dv2ufTeuGflmBCP02m+yylwKClMuZwVaEkHQT4Tlz5uDq1atwd3dXy5Ikj7QhJ06cSHFnCDEqU9W1nYDvNuDKduDFI8DrbaDzfO3zBasA7Wdoa/DqBNgEMNiKEGJWEe7UyWBOjZD05sJG4MRvwI09QLRB3nKnHICjdv26QoKqag4w2WWTCraSzFZVmNmKEJIeIjxp0qSUXIeQ1CfQ2PIxcNKgZnUuj9ho5qJ1tC5oE5NUsFX/+h4omtv4NcOEEJLqKkqEpGtZwNXewP1zYuIC9YYDVfsAeUqneBlRsoKtDrwMtgqLzWz1br1i6F27GHJlYxlBQogZRFhyRSe1HphVlIhJObcO2DQCiAgGsuUFuvysjWpOIyTYatGeG9h0+g4io2MzW4nLuVNVZrYihJhZhNevX//K2mEp2vDrr7/i888/N2XfiLVzaAGwbYz2cbH6QJdfUr2WN7FgqwPXHqn53t0Gma1qSWarhiXQrCwzWxFCMogId+zY8ZV9Xbt2RYUKFVTaygEDTBcMQ6yccm8Ae74BqvUFmo5PdoUiY4KttpzVBludv2sQbFWxIN5r6IGqRXOZ9HqEEBIfk/2q1alTB4MGDTLV6Yi1EugL5C2jfZyjMDDsGOBs2oxsz8OjsOKIH5bsv4k7T0P1wVbdaxRG/wYeKJY79Qk8CCEk3UQ4NDQUP/74IwoVKmSK0xFrRIok+EwEDswGei4DyrbT7jehAN8PClP1e+MGWznAu25x9KnDYCtCSCYQ4fiFGmQ+LTg4GM7Ozvjjjz9M3T9iLchnKkaEUQPcPRkrwibANyBYlRHceCo22KrEy2Crzgy2IoRkJhGeOXNmHBGWaOm8efOq2sIi0IQYRXRU7Fxvi88Bz5ZAyWapHkS5OTx47RF+ih9sVVxbRpDBVoSQTCnC7777btr0hFhfvudd04BbB4G+G7VCLOklUynAumArsXzP3YkNtmpTsYCyfBlsRQjJ1CK8ZMkSuLi4oFu3bnH2r169Gi9evIC3t7cp+0cskeD7wNoB2gILwuWtQLkOJg+2crK3QY8aRRhsRQixHBGeNm0afvrpp1f258uXT0VHU4RJkkjO5zUDgJAH2gpHHWalSoAl2EqE98/DtxhsRQixfBH28/ODh4fHK/ulopI8R0iCxMQA+74H/vsK0MQAecsB3X8D8pZO0YAx2IoQYpUiLBbvmTNnULx48Tj7T58+jdy5c5uyb8RSCHkErBsIXNuhbVfpDbT7DnAwvvjB8VtPMHvnFezyjRtsNbBRCTRnZitCiKWLcK9evTBixAi4urqiUaNGat/u3bsxcuRI9OzZMy36SDIzfoeBNf2AoDuAXVag/Xfa4gtGEhOjwbxdVzHD5zJiNAy2IoRYqQh/8cUXuHnzJpo3bw47O+3LY2Ji0LdvX3z11Vdp0UeSWZNvHJwD/DtZu/43tyfQ/VcgfwWjT/U4JAIfrDyFPS+XGnWq4o4PW5ZmZitCiPWJsIODg8oR/eWXX+LUqVPImjUrKlWqpOaECVGEPgU2vA/4bta2K3bRBmA5uho9QMduPsawZScREBSmop2ndKyI7jWKcKAJIdadttLT01NthLyCjS3w0BewdQDaTAdq9De67q8k2/h57w1M33YJ0TEaleFqXu9qKFsgOwecEGK9ItylSxfUqlULY8a8LDH3km+++QZHjx5V64WJlbqfBRFbsXi7/w5ERwDuVYw+1bMXkRi95jR8LtxX7Te93PHVW5Xg4mjaKkqEEGJubIx9wZ49e9Cu3at5fdu2baueI1ZIWJA2+OrQ/Nh9+cunSIDP3H6K9rP3KgF2sLXBl50qYlbPKhRgQohFYrRp8fz5czUvHB97e3sEBWnTBBIr4+JfwPn1gO82oHJ3IFseo08h7uffD93Cl39fRER0DIq6OSv3c8VCOdKky4QQkiktYQnCksCs+KxYsQLly5c3Vb9IZqLK20DtIYD3phQJcHBYJIYtP4mJG88rAW5dIT/+Gt6AAkwIsXiMtoQnTJiAt956C9euXUOzZtpk+zt27MCyZcuwZs2atOgjyWhEhAC7pgONRgNOObTzwG2np+hUF+4GYeiyE7jxMAR2Nlkwtl059K9fPE6lLkIIsVSMFuEOHTpgw4YNak2wiK4sUfLy8sLOnTvh5ma6AuwkgxLoC6zyBgIvAk/9tGt/U4C4n1cd81fWb3hUDNxzOGFO72qoVpTlMAkh1oPR7mihffv22L9/P0JCQnD9+nV0794do0ePVmJsLHPnzlUpMJ2cnFRN4iNHjiR5/A8//IAyZcoo8S9SpAg+/PBDhIWFpeRtEGM5swpY2FQrwC75gVoDUzSGLyKiMGr1aYxZe1YJcNMyebF5REMKMCHE6kjxmg+JhP7ll1+wdu1auLu7Kxe1CKoxyNzyRx99hAULFigBFoFt3bo1fH19VY7q+IjL+9NPP8XixYtRr149XL58WdU3FtfljBkzUvpWyOuIDAO2jQGOL9W2PRoDXX4GXF79G72Oqw+CMeSPE7jy4Lmq8zu6dRkMblQSNtIghBArwygRDggIwNKlS5X4SiS0WMDh4eHKPZ2SoCwRzoEDB6Jfv36qLWK8efNmJbIitvE5cOAA6tevj7ffflu1xYKWXNaHDx82+tokmTy6Bqz2BgLOyiJgoPEnQOMx2oQcRrLh5B2MW38WLyKikc/VET/2qoo6JVj0gxBivdgYMxcsbmCpoCQW6927dzF79uwUXzgiIgLHjx9HixYtYjtjY6PaBw8eTPA1Yv3Ka3Qua3GFb9myJcF1y8QEXNgILGyiFWDn3ECftUDTcUYLcFhkNMauO6vyP4sA1y+VW7mfKcCEEGsn2Zbw1q1bVfWkIUOGmCRd5cOHDxEdHY38+fPH2S/tS5cuJfgasYDldQ0aNFCBPVFRURg8eDDGjRuX6HXEUpdNR3BwcKr7bvFERQA+E4HDL5NvFK0LdF0MZHc3+lQ3H4bg/T9P4MK9IBVEPaKZJ0Y094Qt3c+EEJJ8S3jfvn1KwKpXr67mb+fMmaMEMT3ZtWuXisqeN28eTpw4gXXr1in3tVR2Soxp06YhR44c+o1rmV+DRDwvaRMrwPVHAt5/pUiAt569hzdm71MCnDubA37rX0tVP6IAE0KIliwaMSmNQCKiJaBK5m3FLSzWrMzt9u/fX9UYNsYd7ezsrJY5derUSb/f29sbT58+xcaNG195TcOGDVGnTh18++23+n1//PEHBg0apDJ5iTv7dZbwnTt3lBD7+/ujcOHCxrx162DlO8DFTYBTTqDzAqBMW6NPEREVg2lbL2LJ/puqXbN4LszuVQ0FcjilQYcJISRjcfv2bbV6Jzk6Y/QSpWzZsinBFcv47NmzGDVqFKZPn66imd98881kn0dSX4pVLYk+dEhdYmnXrVs3wde8ePHiFaG1tdXOTyZ2L+Ho6Ijs2bPrN2NuFKySdt8BZdoB/9uTIgG+/eQFuv10UC/AgxuXxPKBdSjAhBBiqnXCOiRQS6onieovX77c6NfL8qRFixbh119/xcWLF9V8s1jaumjpvn37YuzYsXGCw+bPn69SZN64cQM+Pj4qg5fs14kxMZKge8Dhn2LbrvmBXsuBXMbXh95x8T7a/7gPp/2fIkdWe/ziXQOfti0LO9tUfcwIIcRiMUltOBFAcSkbupWTQ48ePRAYGIiJEyeq5U9VqlTBtm3b9MFafn5+cSzf8ePHqzXB8r+4lfPmzasEeOrUqaZ4G9ZH2DPgp0ZAyANt9HOlrik6TVR0DL79xxc/7b6u2l5FcmLu21VROJeziTtMCCFWPidsTb56q2DHF8Dl7dr0k7lLGv3ygGdhGLH8JI7cfKza/eoXx9i25eBgR+uXEGKd3DZCZ1gl3dp4HghEhQE5i2jbTcZqCzHYZzX6VHuvBOKDFafwKCRC1fv9pmtltKtU0PR9JoQQC4UibE3c3A+s6Q+4FgAG/APYOQK2dtrNCKJjNJi14wpm77wC8aOUL5hd1f4tnidbmnWdEEIsEYqwNRATAxyYpXU9a6K15QdDAoEcxrvjA4PD8cHKk9h/9ZFq96pVFJM6lIeTPQPjCCHEWCjCls6Lx8D6/wFX/tG2K/cE3pgBOBhvtR6+/gjDl5/Eg+BwODvY4qvOldCpaiHT95kQQqwEirAl438UWP0uEHQbsHMC2n4DVOsLlT/SCGJiNFiw5xq+2+6LGA3gmc8F8/tUQ6l8XHNNCCGpgSJsichE7aH5gM8EICYKcCsBdP8NKFDJ6FM9CYnAR6tO4T/fQNV+q2ohfNm5Ipwd+NEhhJDUwl9SSyP0KbBxKHDpb227fCfgzdmAU3ajT3XC7wmG/XkCd5+FwdHOBlM6VkD3GkXUWm1CCCGphyJsSdw9pa39++QmYGMPtP4KqDXQaPezLB1fvP8mpm25iKgYDTzyZMPct6uhvLvxQk4IISRxKMKWwo29wB9dgOhwIEdRoPtSoFB1o0/zLDQSn6w5je3n76t2+0oFMb1LJbg62adBpwkhxLqhCFsKhWsAeTyBHEWATvMAZzejT3HuzjNV+9fv8QvY22bBhDfK4506xeh+JoSQNIIinJl5fB3IWRyQ/NqS8Urq/mbNlSL385+H/TDlrwuIiI5B4VxZlftZckATQghJO5jgN7NyeiUwrx6w9/vYfWL9GinAz8OjMHLFKYzfcE4JcIty+bF5eEMKMCGEpAO0hDMrsvQoKhTwP6zNiBWvznJyuBQQpNzP1wNDYGuTBZ+2KYv3GnrQ/UwIIekERTgzERMN2LxMD1m1t9b1XLp1igR49TF/TNh4DmGRMSiQ3Qlz3q6KGsWNn0cmhBCScuiOziycWwvMqwuEaHM2K8q2ixXlZBIaEY2PV5/Gx2vOKAFuVDovNo9oQAEmhBAzQEs4oxMVDmwfBxz9Wds+NBdoPjFFp7oW+BxD/zyBSwHBsMkCfNSyNN5vUgo20iCEEJLuUIQzMo9vaHM/3zulbTccra3/mwI2nb6LsWvPICQiGnlcHPFjryqoVzKPaftLCCHEKCjCGZWLfwMb3gfCnwFZ3YC3FgKeLY0+TVhkNL7cfAF/HPJT7Tol3PBjr6rI5+qUBp0mhBBiDBThjEZ0JPDvZODgHG27cC2g25IU1f71e/QC7y87jnN3glR7eLNSGNncE3a2DAUghJCMAEU4I/HsNrC6H3D7iLZddxjQYjJga3zKyO3nAzB69WkEh0Uhl7M9ZvaogiZl8pm+z4QQQlIMRTijcMUHWDcICH0MOObQpp4s94bRp4mMjsHXWy/h5303VLt6sVyY3asq3HNmTYNOE0IISQ0U4Yyw9ve/qbGZrwpWAbotBdw8jD7VnaehGLbsBE76PVXtgQ098EmbsrCn+5kQQjIkFGGzkwW4f177sOZ72vKDdo5Gn+U/3wf4cOUpPH0RiexOdviumxdaVShg+u4SQggxGRRhc6HRaPM8S7arTvOBm3uB8h2NPk1UdAxm/nsZc/+7ptqVC+dQxReKuDmnQacJIYSYEopweiN5nsX1/OQm0HGOVoil8EIKBPhBUBiGLz+Jwzceq3bfusXwWftycLQzLosWIYQQ80ARTm/unwV2fQVoYoAqvYDiDVJ0mgNXH2LEipN4+DwC2RxsMb1LZXTwcjd5dwkhhKQdFOH0pqAX0PILbfGFFAhwTIwGc/67qlzQ4tEuW8AV83pXQ4m8LmnSXUIIIWkHRTitEaU8OBfwbAXkLa3dV29Yik716Hk4Plh5CnuvPFTtHjWK4POOFeBkT/czIYRkRijCaUnoE2D9EODyVuDkH8CgXYB9ytJFHr35GMOXnURAUBic7G3wZadK6Frd+CxahBBCMg4U4bTiznFt8YWnfoCtA1B7UIqWHon7edHe6/hmuy+iYzQomTcb5vWujjIFXNOk24QQQtIPinBauJ+PLNKWH4yJBHIVB7r9CrhXMfpUT19EqNST/158oNodq7jjq86VkM2RfzZCCLEE+GtuSsKCgE3DgQsbtO1yHYCOcwGnHEaf6pT/U1X7V7JgOdjZYHKHCuhVqwiyyJImQgghFgFF2FQEnAVWeQOPrwE2dkCrL4Hag7XrgI1Ao9Hg1wM3MXXLRURGa1Ast7NKvlGxkPFCTgghJGNDETaF+/nEb8DWT4CoMCB7YW3u5yI1jT5VUFgkPl17BlvOBqh224oF8HXXysjuZHwVJUIIIRkfinBqiAgB/v4IOLNC25ZlSJ1/0mbAMpLzd58p9/PNRy9gb5sF49qVw7v1itP9TAghFgxFODUcXqAV4Cy2QPMJQL2R2lzQRrqfVxz1x6RN5xERFYNCObNizttVUbVorlR1jRBCSMaHIpwa6g4H7pwA6rwPFK9v9MtDwqMwfsM5rD95R7Wblc2HGd29kNPZIVXdIoQQkjmgCKdq9ByAnn+m6KVX7gdjyJ8ncPXBc9jaZMHHrctgUMMSsLFh9DMhhFgLFGEzsO7EbXy2/hxCI6ORP7sjZveqhloexs8jE0IIydxQhNORsMhofP7XeSw/4q/aDUrlwQ89qyCPi/GZtAghhGR+KMLpxI2HIXj/zxO4eC9ILR0e2dwTw5t5Klc0IYQQ64QinA5sPnMPY9aewfPwKOTO5oBZPauigWee9Lg0IYSQDAxFOA0Jj4rGV5sv4teDt1Rb5n1n96qK/NlTVkmJEEKIZUERTiP8H7/AsGUncPr2M9Ue0qQkRrUsDTtb49YRE0IIsVwowmmAz4X7GLXqFILCopAjqz1m9vBCs7L50+JShBBCMjEUYRMSGR2D77b74qc911W7SpGcKvtV4VzOprwMIYQQCyFD+Ebnzp2L4sWLw8nJCbVr18aRI0cSPbZJkyYqn3L8rX379jAn956FotfCQ3oB7l/fA6v+V5cCTAghJONawitXrsRHH32EBQsWKAH+4Ycf0Lp1a/j6+iJfvnyvHL9u3TpERETo248ePYKXlxe6desGc7HnciA+WHkKj0Mi4Opoh2+7VUabigXN1h9CCCGZA7NbwjNmzMDAgQPRr18/lC9fXomxs7MzFi9enODxbm5uKFCggH7z8fFRx5tDhKNjNJjxjy+8lxxRAlzBPTv+HtGAAkwIISTjW8Ji0R4/fhxjx47V77OxsUGLFi1w8ODBZJ3jl19+Qc+ePZEtW7YEnw8PD1ebjuDgYBP0HHgQHIaRy0/h4PVHqt27dlFMeKM8nOxtTXJ+Qgghlo9ZLeGHDx8iOjoa+fPHjRyWdkCAtrB9Usjc8blz5/Dee+8lesy0adOQI0cO/SbWtinwfxyKozcfw9nBFrN6VsHUzpUowIQQQjKXOzo1iBVcqVIl1KpVK9FjxMp+9uyZfrtw4YJJrl29WC5807UyNg1rgI5VCpnknIQQQqwLs7qj8+TJA1tbW9y/fz/OfmnLfG9ShISEYMWKFZgyZUqSxzk6OqpNR1BQEEzFW9UKm+xchBBCrA+zWsIODg6oXr06duzYod8XExOj2nXr1k3ytatXr1ZzvX369EmHnhJCCCEWuERJlid5e3ujRo0ayq0sS5TEypVoaaFv374oVKiQmtuN74ru1KkTcufObaaeE0IIIZlchHv06IHAwEBMnDhRBWNVqVIF27Zt0wdr+fn5qYhpQ2QN8b59+/DPP/+YqdeEEEJI6smi0Wg0sCJu376NIkWKwN/fH4ULc06XEEKI+XQmU0dHE0IIIZkZs7uj0xsJ/BLu3btn7q4QQgixQHT6otObpLA6EdYth0pqbTEhhBBiCr0pWrRoksdY3ZxwVFQUTp48qQK/4gd8GYukwJQMXJIAxNXV1WR9tDQ4Thwrfq74/bOm36qYmBglwFWrVoWdXdK2rtWJsCmRxB+SClMycWXPnt3c3cmwcJw4Vvxc8fuXGQgyw286A7MIIYQQM0ERJoQQQswERTgVSE7qSZMmxclNTThO/EylD/z+cZws4TPFOWFCCCHETNASJoQQQswERZgQQggxExRhQgghxExQhFPI3LlzUbx4cTg5OaF27do4cuSIaf8yFsKePXvQoUMHuLu7I0uWLNiwYYO5u5QhkVKdNWvWVAkC8uXLp8p0SrUwEpf58+ejcuXKag2nbFJ3fOvWrRym1zB9+nT1/fvggw84VvGYPHmyGhvDrWzZskgvKMIpYOXKlaoOskTRnThxAl5eXmjdujUePHhg+r9QJkdqQ8v4yE0LSZzdu3dj6NChOHToEHx8fBAZGYlWrVqp8SOxSEUaEZTjx4/j2LFjaNasGTp27Ijz589zmBLh6NGj+Omnn9TNC0mYChUqqHzPuk1K5aYbkjGLGEetWrU0Q4cO1bejo6M17u7ummnTpnEok0A+buvXr+cYJYMHDx6o8dq9ezfH6zXkypVL8/PPP3OcEiA4OFjj6emp8fHx0TRu3FgzcuRIjlM8Jk2apPHy8tKYC1rCRhIREaHuwlu0aKHfJzmopX3w4EFT3yMRK0XS5glubm7m7kqGJTo6GitWrFDeAnFLk1cR70r79u3j/F6RV7ly5YqaMitRogR69+4NPz8/pBdWV0UptTx8+FB9+aUAhCHSvnTpktn6RSwHSf4uc3f169dHxYoVzd2dDMfZs2eV6IaFhcHFxQXr169XSfdJXOQGRabLxB1NEkdiepYuXYoyZcooV/Tnn3+Ohg0b4ty5c+lSmIciTEgGtF7kByBd56UyEfJjeerUKeUtWLNmDby9vdWcOoU4Fn9/f4wcOVLFF0jwKEmctm3b6h/LvLmIcrFixbBq1SoMGDAAaQ1F2Ejy5MkDW1tbfV1iHdIuUKCAKf82xAoZNmwY/v77bxVVLkFI5FUcHBxQqlQp9bh69erK0ps1a5YKPiJaZMpMAkWrVaumHxLx4Mnnas6cOQgPD1e/Y+RVcubMidKlS+Pq1atIDzgnnIIfAPni79ixI477UNqclyIpReLWRIDFtbpz5054eHhwMJOJfP9EVEgszZs3V2578Rjotho1aqj5TnlMAU6c58+f49q1ayhYsCDSA1rCKUCWJ4kLTD7UtWrVwg8//KCCQ/r162f6v5AFfKAN7yhv3LihfgQk4Kho0aJm7VtGc0EvW7YMGzduVPNQAQEBar/UNs2aNau5u5dhGDt2rHIfymdHCrDLmO3atQvbt283d9cyFPIZih9PkC1bNuTOnZtxBvEYPXq0ymUgLui7d++qpadyk9KrVy+kBxThFNCjRw8EBgZi4sSJ6seySpUq2LZt2yvBWgRqLWfTpk3j3MAIchMjwRAkNgmF0KRJkzhDsmTJErz77rscppeIi7Vv374qgEZuUGQOTwS4ZcuWHCOSIm7fvq0E99GjR8ibNy8aNGig1uvL4/SAVZQIIYQQM8E5YUIIIcRMUIQJIYQQM0ERJoQQQswERZgQQggxExRhQgghxExQhAkhhBAzQREmhBBCzARFmBBCCDETFGFCiMnIkiULNmzYwBElJJlQhAmxECS9pYhg/K1Nmzbm7hohJBGYO5oQC0IEV/JNG+Lo6Gi2/hBCkoaWMCEWhAiu1LU23HLlyqWeE6tYCkVIFSKpzFSiRAmsWbMmzuul/F2zZs3U81JxZ9CgQaoSliGLFy9GhQoV1LWk3JuUYDTk4cOH6Ny5M5ydneHp6YlNmzbpn3vy5IkqpyfJ8eUa8nz8mwZCrAmKMCFWxIQJE9ClSxecPn1aiWHPnj1x8eJF9ZyU42zdurUS7aNHj2L16tX4999/44isiLiUXRRxFsEWgS1VqlSca3z++efo3r07zpw5g3bt2qnrPH78WH/9CxcuYOvWreq6cr48efKk8ygQkoHQEEIsAm9vb42tra0mW7ZscbapU6eq5+XrPnjw4DivqV27tmbIkCHq8cKFCzW5cuXSPH/+XP/85s2bNTY2NpqAgADVdnd313z22WeJ9kGuMX78eH1bziX7tm7dqtodOnTQ9OvXz8TvnJDMC+eECbEgpHazrjaxDjc3N/3junXrxnlO2qdOnVKPxTL18vJSxd911K9fHzExMfD19VXubCl63rx58yT7IDV+dci5smfPruoAC0OGDFGW+IkTJ9CqVSt06tQJ9erVS+W7JiTzQhEmxIIQ0YvvHjYVMoebHOzt7eO0RbxFyAWZj7516xa2bNkCHx8fJeji3v7uu+/SpM+EZHQ4J0yIFXHo0KFX2uXKlVOP5X+ZK5a5YR379++HjY0NypQpA1dXVxQvXhw7duxIVR8kKMvb2xt//PEHfvjhByxcuDBV5yMkM0NLmBALIjw8HAEBAXH22dnZ6YOfJNiqRo0aaNCgAf78808cOXIEv/zyi3pOAqgmTZqkBHLy5MkIDAzE8OHD8c477yB//vzqGNk/ePBg5MuXT1m1wcHBSqjluOQwceJEVK9eXUVXS1///vtv/U0AIdYIRZgQC2Lbtm1q2ZAhYsVeunRJH7m8YsUKvP/+++q45cuXo3z58uo5WVK0fft2jBw5EjVr1lRtmb+dMWOG/lwi0GFhYZg5cyZGjx6txL1r167J7p+DgwPGjh2LmzdvKvd2w4YNVX8IsVaySHSWuTtBCEl7ZG52/fr1KhiKEJIx4JwwIYQQYiYowoQQQoiZ4JwwIVYCZ54IyXjQEiaEEELMBEWYEEIIMRMUYUIIIcRMUIQJIYQQM0ERJoQQQswERZgQQggxExRhQgghxExQhAkhhBAzQREmhBBCYB7+D6azCLb5uepmAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 500x300 with 2 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"epochs_tensor = torch.linspace(0, num_epochs, len(train_accs))\n",
"examples_seen_tensor = torch.linspace(0, examples_seen, len(train_accs))\n",
"\n",
"plot_values(epochs_tensor, examples_seen_tensor, train_accs, val_accs, label=\"accuracy\")"
]
},
{
"cell_type": "markdown",
"id": "90aba699-21bc-42de-a69c-99f370bb0363",
"metadata": {},
"source": [
"- Based on the accuracy plot above, we can see that the model achieves a relatively high training and validation accuracy after epochs 4 and 5\n",
"- However, we have to keep in mind that we specified `eval_iter=5` in the training function earlier, which means that we only estimated the training and validation set performances\n",
"- We can compute the training, validation, and test set performances over the complete dataset as follows below"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "UHWaJFrjY0zW",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "UHWaJFrjY0zW",
"outputId": "e111e6e6-b147-4159-eb9d-19d4e809ed34"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training accuracy: 97.21%\n",
"Validation accuracy: 97.32%\n",
"Test accuracy: 95.67%\n"
]
}
],
"source": [
"train_accuracy = calc_accuracy_loader(train_loader, model, device)\n",
"val_accuracy = calc_accuracy_loader(val_loader, model, device)\n",
"test_accuracy = calc_accuracy_loader(test_loader, model, device)\n",
"\n",
"print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
"print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
"print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
]
},
{
"cell_type": "markdown",
"id": "6882649f-dc7b-401f-84d2-024ff79c74a1",
"metadata": {},
"source": [
"- We can see that the training and validation set performances are practically identical\n",
"- However, based on the slightly lower test set performance, we can see that the model overfits the training data to a very small degree, as well as the validation data that has been used for tweaking some of the hyperparameters, such as the learning rate\n",
"- This is normal, however, and this gap could potentially be further reduced by increasing the model's dropout rate (`drop_rate`) or the `weight_decay` in the optimizer setting"
]
},
{
"cell_type": "markdown",
"id": "a74d9ad7-3ec1-450e-8c9f-4fc46d3d5bb0",
"metadata": {},
"source": [
"## 6.8 Using the LLM as a spam classifier"
]
},
{
"cell_type": "markdown",
"id": "72ebcfa2-479e-408b-9cf0-7421f6144855",
"metadata": {},
"source": [
"<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/18.webp\" width=500px>"
]
},
{
"cell_type": "markdown",
"id": "fd5408e6-83e4-4e5a-8503-c2fba6073f31",
"metadata": {},
"source": [
"- Finally, let's use the finetuned GPT model in action\n",
"- The `classify_review` function below implements the data preprocessing steps similar to the `SpamDataset` we implemented earlier\n",
"- Then, the function returns the predicted integer class label from the model and returns the corresponding class name"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "aHdn6xvL-IW5",
"metadata": {
"id": "aHdn6xvL-IW5"
},
"outputs": [],
"source": [
"def classify_review(text, model, tokenizer, device, max_length=None, pad_token_id=50256):\n",
" model.eval()\n",
"\n",
" # Prepare inputs to the model\n",
" input_ids = tokenizer.encode(text)\n",
" supported_context_length = model.pos_emb.weight.shape[0]\n",
" # Note: In the book, this was originally written as pos_emb.weight.shape[1] by mistake\n",
" # It didn't break the code but would have caused unnecessary truncation (to 768 instead of 1024)\n",
"\n",
" # Truncate sequences if they too long\n",
" input_ids = input_ids[:min(max_length, supported_context_length)]\n",
" assert max_length is not None, (\n",
" \"max_length must be specified. If you want to use the full model context, \"\n",
" \"pass max_length=model.pos_emb.weight.shape[0].\"\n",
" )\n",
" assert max_length <= supported_context_length, (\n",
" f\"max_length ({max_length}) exceeds model's supported context length ({supported_context_length}).\"\n",
" ) \n",
" # Alternatively, a more robust version is the following one, which handles the max_length=None case better\n",
" # max_len = min(max_length,supported_context_length) if max_length else supported_context_length\n",
" # input_ids = input_ids[:max_len]\n",
" \n",
" # Pad sequences to the longest sequence\n",
" input_ids += [pad_token_id] * (max_length - len(input_ids))\n",
" input_tensor = torch.tensor(input_ids, device=device).unsqueeze(0) # add batch dimension\n",
"\n",
" # Model inference\n",
" with torch.no_grad():\n",
" logits = model(input_tensor)[:, -1, :] # Logits of the last output token\n",
" predicted_label = torch.argmax(logits, dim=-1).item()\n",
"\n",
" # Return the classified result\n",
" return \"spam\" if predicted_label == 1 else \"not spam\""
]
},
{
"cell_type": "markdown",
"id": "f29682d8-a899-4d9b-b973-f8d5ec68172c",
"metadata": {},
"source": [
"- Let's try it out on a few examples below"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "apU_pf51AWSV",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "apU_pf51AWSV",
"outputId": "d0fde0a5-e7a3-4dbe-d9c5-0567dbab7e62"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"spam\n"
]
}
],
"source": [
"text_1 = (\n",
" \"You are a winner you have been specially\"\n",
" \" selected to receive $1000 cash or a $2000 award.\"\n",
")\n",
"\n",
"print(classify_review(\n",
" text_1, model, tokenizer, device, max_length=train_dataset.max_length\n",
"))"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "1g5VTOo_Ajs5",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1g5VTOo_Ajs5",
"outputId": "659b08eb-b6a9-4a8a-9af7-d94c757e93c2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"not spam\n"
]
}
],
"source": [
"text_2 = (\n",
" \"Hey, just wanted to check if we're still on\"\n",
" \" for dinner tonight? Let me know!\"\n",
")\n",
"\n",
"print(classify_review(\n",
" text_2, model, tokenizer, device, max_length=train_dataset.max_length\n",
"))"
]
},
{
"cell_type": "markdown",
"id": "bf736e39-0d47-40c1-8d18-1f716cf7a81e",
"metadata": {},
"source": [
"- Finally, let's save the model in case we want to reuse the model later without having to train it again"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "mYnX-gI1CfQY",
"metadata": {
"id": "mYnX-gI1CfQY"
},
"outputs": [],
"source": [
"torch.save(model.state_dict(), \"review_classifier.pth\")"
]
},
{
"cell_type": "markdown",
"id": "ba78cf7c-6b80-4f71-a50e-3ccc73839af6",
"metadata": {},
"source": [
"- Then, in a new session, we could load the model as follows"
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "cc4e68a5-d492-493b-87ef-45c475f353f5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<All keys matched successfully>"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model_state_dict = torch.load(\"review_classifier.pth\", map_location=device, weights_only=True)\n",
"model.load_state_dict(model_state_dict)"
]
},
{
"cell_type": "markdown",
"id": "5b70ac71-234f-4eeb-b33d-c62726d50cd4",
"metadata": {
"id": "5b70ac71-234f-4eeb-b33d-c62726d50cd4"
},
"source": [
"## Summary and takeaways"
]
},
{
"cell_type": "markdown",
"id": "dafdc910-d616-47ab-aa85-f90c6e7ed80e",
"metadata": {},
"source": [
"- See the [./gpt_class_finetune.py](./gpt_class_finetune.py) script, a self-contained script for classification finetuning\n",
"- You can find the exercise solutions in [./exercise-solutions.ipynb](./exercise-solutions.ipynb)\n",
"- In addition, interested readers can find an introduction to parameter-efficient training with low-rank adaptation (LoRA) in [appendix E](../../appendix-E)"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"gpuType": "V100",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.5"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink