![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/chapter-overview.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c",
"metadata": {
"id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c"
},
"source": [
"## 6.1 Different categories of finetuning"
]
},
{
"cell_type": "markdown",
"id": "ede3d731-5123-4f02-accd-c670ce50a5a3",
"metadata": {
"id": "ede3d731-5123-4f02-accd-c670ce50a5a3"
},
"source": [
"- No code in this section"
]
},
{
"cell_type": "markdown",
"id": "ac45579d-d485-47dc-829e-43be7f4db57b",
"metadata": {},
"source": [
"- The most common ways to finetune language models are instruction-finetuning and classification finetuning\n",
"- Instruction-finetuning, depicted below, is the topic of the next chapter"
]
},
{
"cell_type": "markdown",
"id": "6c29ef42-46d9-43d4-8bb4-94974e1665e4",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/instructions.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "a7f60321-95b8-46a9-97bf-1d07fda2c3dd",
"metadata": {},
"source": [
"- Classification finetuning, the topic of this chapter, is a procedure you may already be familiar with if you have a background in machine learning -- it's similar to training a convolutional network to classify handwritten digits, for example\n",
"- In classification finetuning, we have a specific number of class labels (for example, \"spam\" and \"not spam\") that the model can output\n",
"- A classification finetuned model can only predict classes it has seen during training (for example, \"spam\" or \"not spam\", whereas an instruction-finetuned model can usually perform many tasks\n",
"- We can think of a classification-finetuned model as a very specialized model; in practice, it is much easier to create a specialized model than a generalist model that performs well on many different tasks"
]
},
{
"cell_type": "markdown",
"id": "0b37a0c4-0bb1-4061-b1fe-eaa4416d52c3",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/spam-non-spam.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf",
"metadata": {
"id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf"
},
"source": [
"## 6.2 Preparing the dataset"
]
},
{
"cell_type": "markdown",
"id": "5f628975-d2e8-4f7f-ab38-92bb868b7067",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/overview-1.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "9fbd459f-63fa-4d8c-8499-e23103156c7d",
"metadata": {
"id": "9fbd459f-63fa-4d8c-8499-e23103156c7d"
},
"source": [
"- This section prepares the dataset we use for classification finetuning\n",
"- We use a dataset consisting of spam and non-spam text messages to finetune the LLM to classify them\n",
"- First, we download and unzip the dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
"outputId": "424e4423-f623-443c-ab9e-656f9e867559"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"sms_spam_collection/SMSSpamCollection.tsv already exists. Skipping download and extraction.\n"
]
}
],
"source": [
"import urllib.request\n",
"import zipfile\n",
"import os\n",
"from pathlib import Path\n",
"\n",
"url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
"zip_path = \"sms_spam_collection.zip\"\n",
"extracted_path = \"sms_spam_collection\"\n",
"data_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\n",
"\n",
"def download_and_unzip(url, zip_path, extracted_path, data_file_path):\n",
" if data_file_path.exists():\n",
" print(f\"{data_file_path} already exists. Skipping download and extraction.\")\n",
" return\n",
"\n",
" # Downloading the file\n",
" with urllib.request.urlopen(url) as response:\n",
" with open(zip_path, \"wb\") as out_file:\n",
" out_file.write(response.read())\n",
"\n",
" # Unzipping the file\n",
" with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n",
" zip_ref.extractall(extracted_path)\n",
"\n",
" # Add .tsv file extension\n",
" original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\n",
" os.rename(original_file_path, data_file_path)\n",
" print(f\"File downloaded and saved as {data_file_path}\")\n",
"\n",
"download_and_unzip(url, zip_path, extracted_path, data_file_path)"
]
},
{
"cell_type": "markdown",
"id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1",
"metadata": {
"id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1"
},
"source": [
"- The dataset is saved as a tab-separated text file, which we can load into a pandas DataFrame"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 423
},
"id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
"outputId": "a16c5cde-d341-4887-a93f-baa9bec542ab"
},
"outputs": [
{
"data": {
"text/html": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/pad-input-sequences.webp?123\")
"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
"outputId": "b5b48439-32c8-4b37-cca2-c9dc8fa86563"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[50256]\n"
]
}
],
"source": [
"import tiktoken\n",
"\n",
"tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
"print(tokenizer.encode(\"<|endoftext|>\", allowed_special={\"<|endoftext|>\"}))"
]
},
{
"cell_type": "markdown",
"id": "04f582ff-68bf-450e-bd87-5fb61afe431c",
"metadata": {
"id": "04f582ff-68bf-450e-bd87-5fb61afe431c"
},
"source": [
"- The `SpamDataset` class below identifies the longest sequence in the training dataset and adds the padding token to the others to match that sequence length"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "d7791b52-af18-4ac4-afa9-b921068e383e",
"metadata": {
"id": "d7791b52-af18-4ac4-afa9-b921068e383e"
},
"outputs": [],
"source": [
"import torch\n",
"from torch.utils.data import Dataset\n",
"\n",
"\n",
"class SpamDataset(Dataset):\n",
" def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):\n",
" self.data = pd.read_csv(csv_file)\n",
"\n",
" # Pre-tokenize texts\n",
" self.encoded_texts = [\n",
" tokenizer.encode(text) for text in self.data[\"Text\"]\n",
" ]\n",
"\n",
" if max_length is None:\n",
" self.max_length = self._longest_encoded_length()\n",
" else:\n",
" self.max_length = max_length\n",
" # Truncate sequences if they are longer than max_length\n",
" self.encoded_texts = [\n",
" encoded_text[:self.max_length]\n",
" for encoded_text in self.encoded_texts\n",
" ]\n",
"\n",
" # Pad sequences to the longest sequence\n",
" self.encoded_texts = [\n",
" encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))\n",
" for encoded_text in self.encoded_texts\n",
" ]\n",
"\n",
" def __getitem__(self, index):\n",
" encoded = self.encoded_texts[index]\n",
" label = self.data.iloc[index][\"Label\"]\n",
" return (\n",
" torch.tensor(encoded, dtype=torch.long),\n",
" torch.tensor(label, dtype=torch.long)\n",
" )\n",
"\n",
" def __len__(self):\n",
" return len(self.data)\n",
"\n",
" def _longest_encoded_length(self):\n",
" max_length = 0\n",
" for encoded_text in self.encoded_texts:\n",
" encoded_length = len(encoded_text)\n",
" if encoded_length > max_length:\n",
" max_length = encoded_length\n",
" return max_length"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "uzj85f8ou82h",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "uzj85f8ou82h",
"outputId": "d08f1cf0-c24d-445f-a3f8-793532c3716f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"120\n"
]
}
],
"source": [
"train_dataset = SpamDataset(\n",
" csv_file=\"train.csv\",\n",
" max_length=None,\n",
" tokenizer=tokenizer\n",
")\n",
"\n",
"print(train_dataset.max_length)"
]
},
{
"cell_type": "markdown",
"id": "15bdd932-97eb-4b88-9cf9-d766ea4c3a60",
"metadata": {},
"source": [
"- We also pad the validation and test set to the longest training sequence\n",
"- Note that validation and test set samples that are longer than the longest training example are being truncated via `encoded_text[:self.max_length]` in the `SpamDataset` code\n",
"- This behavior is entirely optional, and it would also work well if we set `max_length=None` in both the validation and test set cases"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e",
"metadata": {
"id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e"
},
"outputs": [],
"source": [
"val_dataset = SpamDataset(\n",
" csv_file=\"validation.csv\",\n",
" max_length=train_dataset.max_length,\n",
" tokenizer=tokenizer\n",
")\n",
"test_dataset = SpamDataset(\n",
" csv_file=\"test.csv\",\n",
" max_length=train_dataset.max_length,\n",
" tokenizer=tokenizer\n",
")"
]
},
{
"cell_type": "markdown",
"id": "20170d89-85a0-4844-9887-832f5d23432a",
"metadata": {},
"source": [
"- Next, we use the dataset to instantiate the data loaders, which is similar to creating the data loaders in previous chapters"
]
},
{
"cell_type": "markdown",
"id": "64bcc349-205f-48f8-9655-95ff21f5e72f",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/batch.webp\")
"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
"outputId": "3266c410-4fdb-4a8c-a142-7f707e2525ab"
},
"outputs": [],
"source": [
"from torch.utils.data import DataLoader\n",
"\n",
"num_workers = 0\n",
"batch_size = 8\n",
"\n",
"torch.manual_seed(123)\n",
"\n",
"train_loader = DataLoader(\n",
" dataset=train_dataset,\n",
" batch_size=batch_size,\n",
" shuffle=True,\n",
" num_workers=num_workers,\n",
" drop_last=True,\n",
")\n",
"\n",
"val_loader = DataLoader(\n",
" dataset=val_dataset,\n",
" batch_size=batch_size,\n",
" num_workers=num_workers,\n",
" drop_last=False,\n",
")\n",
"\n",
"test_loader = DataLoader(\n",
" dataset=test_dataset,\n",
" batch_size=batch_size,\n",
" num_workers=num_workers,\n",
" drop_last=False,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "ab7335db-e0bb-4e27-80c5-eea11e593a57",
"metadata": {},
"source": [
"- As a verification step, we iterate through the data loaders and ensure that the batches contain 8 training examples each, where each training example consists of 120 tokens"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "4dee6882-4c3a-4964-af15-fa31f86ad047",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train loader:\n",
"Input batch dimensions: torch.Size([8, 120])\n",
"Label batch dimensions torch.Size([8])\n"
]
}
],
"source": [
"print(\"Train loader:\")\n",
"for input_batch, target_batch in train_loader:\n",
" pass\n",
"\n",
"print(\"Input batch dimensions:\", input_batch.shape)\n",
"print(\"Label batch dimensions\", target_batch.shape)"
]
},
{
"cell_type": "markdown",
"id": "5cdd7947-7039-49bf-8a5e-c0a2f4281ca1",
"metadata": {},
"source": [
"- Lastly, let's print the total number of batches in each dataset"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "IZfw-TYD2zTj",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "IZfw-TYD2zTj",
"outputId": "6934bbf2-9797-4fbe-d26b-1a246e18c2fb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"130 training batches\n",
"19 validation batches\n",
"38 test batches\n"
]
}
],
"source": [
"print(f\"{len(train_loader)} training batches\")\n",
"print(f\"{len(val_loader)} validation batches\")\n",
"print(f\"{len(test_loader)} test batches\")"
]
},
{
"cell_type": "markdown",
"id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c",
"metadata": {
"id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c"
},
"source": [
"## 6.4 Initializing a model with pretrained weights"
]
},
{
"cell_type": "markdown",
"id": "97e1af8b-8bd1-4b44-8b8b-dc031496e208",
"metadata": {},
"source": [
"- In this section, we initialize the pretrained model we worked with in the previous chapter\n",
"\n",
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/overview-2.webp\")
"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "2992d779-f9fb-4812-a117-553eb790a5a9",
"metadata": {
"id": "2992d779-f9fb-4812-a117-553eb790a5a9"
},
"outputs": [],
"source": [
"CHOOSE_MODEL = \"gpt2-small (124M)\"\n",
"INPUT_PROMPT = \"Every effort moves\"\n",
"\n",
"BASE_CONFIG = {\n",
" \"vocab_size\": 50257, # Vocabulary size\n",
" \"context_length\": 1024, # Context length\n",
" \"drop_rate\": 0.0, # Dropout rate\n",
" \"qkv_bias\": True # Query-key-value bias\n",
"}\n",
"\n",
"model_configs = {\n",
" \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
" \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
" \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
" \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
"}\n",
"\n",
"BASE_CONFIG.update(model_configs[CHOOSE_MODEL])"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "022a649a-44f5-466c-8a8e-326c063384f5",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "022a649a-44f5-466c-8a8e-326c063384f5",
"outputId": "7091e401-8442-4f47-a1d9-ecb42a1ef930"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"File already exists and is up-to-date: gpt2/124M/checkpoint\n",
"File already exists and is up-to-date: gpt2/124M/encoder.json\n",
"File already exists and is up-to-date: gpt2/124M/hparams.json\n",
"File already exists and is up-to-date: gpt2/124M/model.ckpt.data-00000-of-00001\n",
"File already exists and is up-to-date: gpt2/124M/model.ckpt.index\n",
"File already exists and is up-to-date: gpt2/124M/model.ckpt.meta\n",
"File already exists and is up-to-date: gpt2/124M/vocab.bpe\n"
]
}
],
"source": [
"from gpt_download import download_and_load_gpt2\n",
"from previous_chapters import GPTModel, load_weights_into_gpt\n",
"\n",
"model_size = CHOOSE_MODEL.split(\" \")[-1].lstrip(\"(\").rstrip(\")\")\n",
"settings, params = download_and_load_gpt2(model_size=model_size, models_dir=\"gpt2\")\n",
"\n",
"model = GPTModel(BASE_CONFIG)\n",
"load_weights_into_gpt(model, params)\n",
"model.eval();"
]
},
{
"cell_type": "markdown",
"id": "ab8e056c-abe0-415f-b34d-df686204259e",
"metadata": {},
"source": [
"- To ensure that the model was loaded corrected, let's double-check that it generates coherent text"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "d8ac25ff-74b1-4149-8dc5-4c429d464330",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Every effort moves you forward.\n",
"\n",
"The first step is to understand the importance of your work\n"
]
}
],
"source": [
"from previous_chapters import (\n",
" generate_text_simple,\n",
" text_to_token_ids,\n",
" token_ids_to_text\n",
")\n",
"\n",
"\n",
"text_1 = \"Every effort moves you\"\n",
"\n",
"token_ids = generate_text_simple(\n",
" model=model,\n",
" idx=text_to_token_ids(text_1, tokenizer),\n",
" max_new_tokens=15,\n",
" context_size=BASE_CONFIG[\"context_length\"]\n",
")\n",
"\n",
"print(token_ids_to_text(token_ids, tokenizer))"
]
},
{
"cell_type": "markdown",
"id": "69162550-6a02-4ece-8db1-06c71d61946f",
"metadata": {},
"source": [
"- Before we finetune the model as a classifier, let's see if the model can perhaps already classify spam messages via prompting"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "94224aa9-c95a-4f8a-a420-76d01e3a800c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Is the following text 'spam'? Answer with 'yes' or 'no': 'You are a winner you have been specially selected to receive $1000 cash or a $2000 award.' Answer with 'yes' or 'no'. Answer with 'yes' or 'no'. Answer with 'yes' or 'no'. Answer with 'yes'\n"
]
}
],
"source": [
"text_2 = (\n",
" \"Is the following text 'spam'? Answer with 'yes' or 'no':\"\n",
" \" 'You are a winner you have been specially\"\n",
" \" selected to receive $1000 cash or a $2000 award.'\"\n",
" \" Answer with 'yes' or 'no'.\"\n",
")\n",
"\n",
"token_ids = generate_text_simple(\n",
" model=model,\n",
" idx=text_to_token_ids(text_2, tokenizer),\n",
" max_new_tokens=23,\n",
" context_size=BASE_CONFIG[\"context_length\"]\n",
")\n",
"\n",
"print(token_ids_to_text(token_ids, tokenizer))"
]
},
{
"cell_type": "markdown",
"id": "1ce39ed0-2c77-410d-8392-dd15d4b22016",
"metadata": {},
"source": [
"- As we can see, the model is not very good at following instructions\n",
"- This is expected, since it has only been pretrained and not instruction-finetuned (instruction finetuning will be covered in the next chapter)"
]
},
{
"cell_type": "markdown",
"id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522",
"metadata": {
"id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522"
},
"source": [
"## 6.5 Adding a classification head"
]
},
{
"cell_type": "markdown",
"id": "d6e9d66f-76b2-40fc-9ec5-3f972a8db9c0",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/lm-head.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "217bac05-78df-4412-bd80-612f8061c01d",
"metadata": {},
"source": [
"- In this section, we are modifying the pretrained LLM to make it ready for classification finetuning\n",
"- Let's take a look at the model architecture first"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "b23aff91-6bd0-48da-88f6-353657e6c981",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1d8f7a01-b7c0-48d4-b1e7-8c12cc7ad932",
"outputId": "b6a5b9b5-a92f-498f-d7cb-b58dd99e4497"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GPTModel(\n",
" (tok_emb): Embedding(50257, 768)\n",
" (pos_emb): Embedding(1024, 768)\n",
" (drop_emb): Dropout(p=0.0, inplace=False)\n",
" (trf_blocks): Sequential(\n",
" (0): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (1): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (2): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (3): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (4): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (5): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (6): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (7): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (8): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (9): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (10): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (11): TransformerBlock(\n",
" (att): MultiHeadAttention(\n",
" (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
" (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
" (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.0, inplace=False)\n",
" )\n",
" (ff): FeedForward(\n",
" (layers): Sequential(\n",
" (0): Linear(in_features=768, out_features=3072, bias=True)\n",
" (1): GELU()\n",
" (2): Linear(in_features=3072, out_features=768, bias=True)\n",
" )\n",
" )\n",
" (norm1): LayerNorm()\n",
" (norm2): LayerNorm()\n",
" (drop_resid): Dropout(p=0.0, inplace=False)\n",
" )\n",
" )\n",
" (final_norm): LayerNorm()\n",
" (out_head): Linear(in_features=768, out_features=50257, bias=False)\n",
")\n"
]
}
],
"source": [
"print(model)"
]
},
{
"cell_type": "markdown",
"id": "3f640a76-dd00-4769-9bc8-1aed0cec330d",
"metadata": {},
"source": [
"- Above, we can see the architecture we implemented in chapter 4 neatly laid out\n",
"- The goal is to replace and finetune the output layer\n",
"- To achieve this, we first freeze the model, meaning that we make all layers non-trainable"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "fkMWFl-0etea",
"metadata": {
"id": "fkMWFl-0etea"
},
"outputs": [],
"source": [
"for param in model.parameters():\n",
" param.requires_grad = False"
]
},
{
"cell_type": "markdown",
"id": "72155f83-87d9-476a-a978-a15aa2d44147",
"metadata": {},
"source": [
"- Then, we replace the output layer (`model.out_head`), which originally maps the layer inputs to 50,257 dimensions (the size of the vocabulary)\n",
"- Since we finetune the model for binary classification (predicting 2 classes, \"spam\" and \"not spam\"), we can replace the output layer as shown below, which will be trainable by default\n",
"- Note that we use `BASE_CONFIG[\"emb_dim\"]` (which is equal to 768 in the `\"gpt2-small (124M)\"` model) to keep the code below more general"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "7e759fa0-0f69-41be-b576-17e5f20e04cb",
"metadata": {},
"outputs": [],
"source": [
"torch.manual_seed(123)\n",
"\n",
"num_classes = 2\n",
"model.out_head = torch.nn.Linear(in_features=BASE_CONFIG[\"emb_dim\"], out_features=num_classes)"
]
},
{
"cell_type": "markdown",
"id": "30be5475-ae77-4f97-8f3e-dec462b1339f",
"metadata": {},
"source": [
"- Technically, it's sufficient to only train the output layer\n",
"- However, as I found in [experiments finetuning additional layers](https://magazine.sebastianraschka.com/p/finetuning-large-language-models) can noticeably improve the performance\n",
"- So, we are also making the last transformer block and the final `LayerNorm` module connecting the last transformer block to the output layer trainable"
]
},
{
"cell_type": "markdown",
"id": "0be7c1eb-c46c-4065-8525-eea1b8c66d10",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/trainable.webp\")
"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7",
"metadata": {
"id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7"
},
"outputs": [],
"source": [
"for param in model.trf_blocks[-1].parameters():\n",
" param.requires_grad = True\n",
"\n",
"for param in model.final_norm.parameters():\n",
" param.requires_grad = True"
]
},
{
"cell_type": "markdown",
"id": "f012b899-8284-4d3a-97c0-8a48eb33ba2e",
"metadata": {},
"source": [
"- We can still use this model similar to before in previous chapters\n",
"- For example, let's feed it some text input"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
"outputId": "27e041b1-d731-48a1-cf60-f22d4565304e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Inputs: tensor([[5211, 345, 423, 640]])\n",
"Inputs dimensions: torch.Size([1, 4])\n"
]
}
],
"source": [
"inputs = tokenizer.encode(\"Do you have time\")\n",
"inputs = torch.tensor(inputs).unsqueeze(0)\n",
"print(\"Inputs:\", inputs)\n",
"print(\"Inputs dimensions:\", inputs.shape) # shape: (batch_size, num_tokens)"
]
},
{
"cell_type": "markdown",
"id": "fbbf8481-772d-467b-851c-a62b86d0cb1b",
"metadata": {},
"source": [
"- What's different compared to previous chapters is that it now has two output dimensions instead of 50,257"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
"outputId": "9cae7448-253d-4776-973e-0af190b06354"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Outputs:\n",
" tensor([[[-1.5854, 0.9904],\n",
" [-3.7235, 7.4548],\n",
" [-2.2661, 6.6049],\n",
" [-3.5983, 3.9902]]])\n",
"Outputs dimensions: torch.Size([1, 4, 2])\n"
]
}
],
"source": [
"with torch.no_grad():\n",
" outputs = model(inputs)\n",
"\n",
"print(\"Outputs:\\n\", outputs)\n",
"print(\"Outputs dimensions:\", outputs.shape) # shape: (batch_size, num_tokens, num_classes)"
]
},
{
"cell_type": "markdown",
"id": "75430a01-ef9c-426a-aca0-664689c4f461",
"metadata": {},
"source": [
"- As discussed in previous chapters, for each input token, there's one output vector\n",
"- Since we fed the model a text sample with 4 input tokens, the output consists of 4 2-dimensional output vectors above"
]
},
{
"cell_type": "markdown",
"id": "7df9144f-6817-4be4-8d4b-5d4dadfe4a9b",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/input-and-output.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "e3bb8616-c791-4f5c-bac0-5302f663e46a",
"metadata": {},
"source": [
"- In chapter 3, we discussed the attention mechanism, which connects each input token to each other input token\n",
"- In chapter 3, we then also introduced the causal attention mask that is used in GPT-like models; this causal mask lets a current token only attend to the current and previous token positions\n",
"- Based on this causal attention mechanism, the 4th (last) token contains the most information among all tokens because it's the only token that includes information about all other tokens\n",
"- Hence, we are particularly interested in this last token, which we will finetune for the spam classification task"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
"outputId": "e79eb155-fa1f-46ed-ff8c-d828c3a3fabd"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Last output token: tensor([[-3.5983, 3.9902]])\n"
]
}
],
"source": [
"print(\"Last output token:\", outputs[:, -1, :])"
]
},
{
"cell_type": "markdown",
"id": "8df08ae0-e664-4670-b7c5-8a2280d9b41b",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/attention-mask.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "32aa4aef-e1e9-491b-9adf-5aa973e59b8c",
"metadata": {},
"source": [
"## 6.6 Calculating the classification loss and accuracy"
]
},
{
"cell_type": "markdown",
"id": "669e1fd1-ace8-44b4-b438-185ed0ba8b33",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/overview-3.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "7a7df4ee-0a34-4a4d-896d-affbbf81e0b3",
"metadata": {},
"source": [
"- Before explaining the loss calculation, let's have a brief look at how the model outputs are turned into class labels"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "c77faab1-3461-4118-866a-6171f2b89aa0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Last output token: tensor([[-3.5983, 3.9902]])\n"
]
}
],
"source": [
"print(\"Last output token:\", outputs[:, -1, :])"
]
},
{
"cell_type": "markdown",
"id": "7edd71fa-628a-4d00-b81d-6d8bcb2c341d",
"metadata": {},
"source": [
"- Similar to chapter 5, we convert the outputs (logits) into probability scores via the `softmax` function and then obtain the index position of the largest probability value via the `argmax` function:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "b81efa92-9be1-4b9e-8790-ce1fc7b17f01",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Class label: 1\n"
]
}
],
"source": [
"probas = torch.softmax(outputs[:, -1, :], dim=-1)\n",
"label = torch.argmax(probas)\n",
"print(\"Class label:\", label.item())"
]
},
{
"cell_type": "markdown",
"id": "d5241f47-a1e4-4bba-8064-5d06cffa7941",
"metadata": {},
"source": [
"- Note that the softmax function is optional here, as explained in chapter 5, because the largest outputs correspond to the largest probability scores"
]
},
{
"cell_type": "markdown",
"id": "4f4a9d15-8fc7-48a2-8734-d92a2f265328",
"metadata": {},
"source": [
"- Before we can start finetuning (/training), we first have to define the loss function we want to optimize during training\n",
"- The goal is to maximize the spam classification accuracy of the model; however, classification accuracy is not a differentiable function\n",
"- Hence, instead, we minimize the cross entropy loss as a proxy for maximizing the classification accuracy (you can learn more about this topic in lecture 8 of my freely available [Introduction to Deep Learning](https://sebastianraschka.com/blog/2021/dl-course.html#l08-multinomial-logistic-regression--softmax-regression) class.\n",
"\n",
"- Note that in chapter 5, we calculated the cross entropy loss for the next predicted token over the 50,257 token IDs in the vocabulary\n",
"- Here, we calculate the cross entropy in a similar fashion; the only difference is that instead of 50,257 token IDs, we now have only two choices: \"spam\" (label 1) or \"not spam\" (label 0).\n",
"- In other words, the loss calculation training code is practically identical to the one in chapter 5, but we now only have two labels instead of 50,257 labels (token IDs).\n",
"\n",
"\n",
"- Consequently, the `calc_loss_batch` function is the same here as in chapter 5, except that we are only interested in optimizing the last token `model(input_batch)[:, -1, :]` instead of all tokens `model(input_batch)`:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "2f1e9547-806c-41a9-8aba-3b2822baabe4",
"metadata": {
"id": "2f1e9547-806c-41a9-8aba-3b2822baabe4"
},
"outputs": [],
"source": [
"def calc_loss_batch(input_batch, target_batch, model, device):\n",
" input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
" logits = model(input_batch)[:, -1, :] # Logits of last output token\n",
" loss = torch.nn.functional.cross_entropy(logits, target_batch)\n",
" return loss"
]
},
{
"cell_type": "markdown",
"id": "a013aab9-f854-4866-ad55-5b8350adb50a",
"metadata": {},
"source": [
"The `calc_loss_loader` is exactly the same as in chapter 5:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "b7b83e10-5720-45e7-ac5e-369417ca846b",
"metadata": {},
"outputs": [],
"source": [
"# Same as in chapter 5\n",
"def calc_loss_loader(data_loader, model, device, num_batches=None):\n",
" total_loss = 0.\n",
" if len(data_loader) == 0:\n",
" return float(\"nan\")\n",
" elif num_batches is None:\n",
" num_batches = len(data_loader)\n",
" else:\n",
" # Reduce the number of batches to match the total number of batches in the data loader\n",
" # if num_batches exceeds the number of batches in the data loader\n",
" num_batches = min(num_batches, len(data_loader))\n",
" for i, (input_batch, target_batch) in enumerate(data_loader):\n",
" if i < num_batches:\n",
" loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
" total_loss += loss.item()\n",
" else:\n",
" break\n",
" return total_loss / num_batches"
]
},
{
"cell_type": "markdown",
"id": "56826ecd-6e74-40e6-b772-d3541e585067",
"metadata": {},
"source": [
"- Using the `calc_closs_loader`, we compute the initial training, validation, and test set losses before we start training\n",
"- Here, we use `torch.no_grad()` so that no gradients are computed during the forward pass, which reduces memory consumption and speeds up computations since we are not training the model yet\n",
"- Via the `device` setting, the model automatically runs on a GPU if a GPU with Nvidia CUDA support is available and otherwise runs on a CPU"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
"outputId": "49df8648-9e38-4314-854d-9faacd1b2e89"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training loss: 3.095\n",
"Validation loss: 2.583\n",
"Test loss: 2.322\n"
]
}
],
"source": [
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"model.to(device) # no assignment model = model.to(device) necessary for nn.Module classes\n",
"\n",
"torch.manual_seed(123) # For reproducibility due to the shuffling in the training data loader\n",
"\n",
"with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet\n",
" train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)\n",
" val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)\n",
" test_loss = calc_loss_loader(test_loader, model, device, num_batches=5)\n",
"\n",
"print(f\"Training loss: {train_loss:.3f}\")\n",
"print(f\"Validation loss: {val_loss:.3f}\")\n",
"print(f\"Test loss: {test_loss:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "b109556e-ddae-49fd-ad08-e6fa1032ea7a",
"metadata": {},
"source": [
"- Similar to the `calc_loss_loader` function above, we can define a `calc_accuracy_loader` function that calculates the classification accuracy by checking how many predicted class (spam and ham) labels match the given labels in the dataset\n",
"- Note that the classification accuracy is a mathematically non-differentiable function, and we only use it for evaluation; hence, we can disable the gradient calculation permanently to save resources here\n",
"- We can disable the gradient tracking either using the `with torch.no_grad():` inside the function or by using the `@torch.no_grad()` function decorator"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "64ce5b12-84cd-488c-8ea7-4cef5b2d947e",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "64ce5b12-84cd-488c-8ea7-4cef5b2d947e",
"outputId": "239581b4-fd0f-4adf-e67b-364e0f0f96b7"
},
"outputs": [],
"source": [
"@torch.no_grad() # Disable gradient tracking for efficiency\n",
"def calc_accuracy_loader(data_loader, model, device, num_batches=None):\n",
" model.eval()\n",
" correct_predictions, num_examples = 0, 0\n",
"\n",
" if num_batches is None:\n",
" num_batches = len(data_loader)\n",
" else:\n",
" num_batches = min(num_batches, len(data_loader))\n",
" for i, (input_batch, target_batch) in enumerate(data_loader):\n",
" if i < num_batches:\n",
" input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
" logits = model(input_batch)[:, -1, :] # Logits of last output token\n",
" predicted_labels = torch.argmax(logits, dim=-1)\n",
"\n",
" num_examples += predicted_labels.shape[0]\n",
" correct_predictions += (predicted_labels == target_batch).sum().item()\n",
" else:\n",
" break\n",
" return correct_predictions / num_examples"
]
},
{
"cell_type": "markdown",
"id": "90521a9a-639c-4c7f-a5c0-aca8fa5d4c1b",
"metadata": {},
"source": [
"- Let's check the initial classification accuracy before we start training the model"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "2160418f-988b-40f3-bce8-e431021e97dc",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training accuracy: 46.25%\n",
"Validation accuracy: 45.00%\n",
"Test accuracy: 48.75%\n"
]
}
],
"source": [
"torch.manual_seed(123)\n",
"train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=10)\n",
"val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=10)\n",
"test_accuracy = calc_accuracy_loader(test_loader, model, device, num_batches=10)\n",
"\n",
"print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
"print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
"print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
]
},
{
"cell_type": "markdown",
"id": "e04b980b-e583-4f62-84a0-4edafaf99d5d",
"metadata": {},
"source": [
"- As we can see, the model only gets roughly half (50%) of the predictions correctly\n",
"- In the next section, we train the model to improve the classification accuracy"
]
},
{
"cell_type": "markdown",
"id": "456ae0fd-6261-42b4-ab6a-d24289953083",
"metadata": {
"id": "456ae0fd-6261-42b4-ab6a-d24289953083"
},
"source": [
"## 6.7 Finetuning the model on supervised data"
]
},
{
"cell_type": "markdown",
"id": "6a9b099b-0829-4f72-8a2b-4363e3497026",
"metadata": {},
"source": [
"- In this section, we define and use the training function to improve the classification accuracy of the model\n",
"- The `train_classifier_simple` function below is practically the same as the `train_model_simple` function we used for pretraining the model in chapter 5\n",
"- The only two differences are that we now \n",
" 1. track the number of training examples seen (`examples_seen`) instead of the number of tokens seen\n",
" 2. calculate the accuracy after each epoch instead of printing a sample text after each epoch"
]
},
{
"cell_type": "markdown",
"id": "979b6222-1dc2-4530-9d01-b6b04fe3de12",
"metadata": {},
"source": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/training-loop.webp\")
"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "Csbr60to50FL",
"metadata": {
"id": "Csbr60to50FL"
},
"outputs": [],
"source": [
"# Overall the same as `train_model_simple` in chapter 5\n",
"def train_classifier_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n",
" eval_freq, eval_iter, tokenizer):\n",
" # Initialize lists to track losses and tokens seen\n",
" train_losses, val_losses, train_accs, val_accs = [], [], [], []\n",
" examples_seen, global_step = 0, -1\n",
"\n",
" # Main training loop\n",
" for epoch in range(num_epochs):\n",
" model.train() # Set model to training mode\n",
"\n",
" for input_batch, target_batch in train_loader:\n",
" optimizer.zero_grad() # Reset loss gradients from previous epoch\n",
" loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
" loss.backward() # Calculate loss gradients\n",
" optimizer.step() # Update model weights using loss gradients\n",
" examples_seen += input_batch.shape[0] # New: track examples instead of tokens\n",
" global_step += 1\n",
"\n",
" # Optional evaluation step\n",
" if global_step % eval_freq == 0:\n",
" train_loss, val_loss = evaluate_model(\n",
" model, train_loader, val_loader, device, eval_iter)\n",
" train_losses.append(train_loss)\n",
" val_losses.append(val_loss)\n",
" print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n",
" f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
"\n",
" # Calculate accuracy after each epoch\n",
" train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=eval_iter)\n",
" val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=eval_iter)\n",
" print(f\"Training accuracy: {train_accuracy*100:.2f}% | \", end=\"\")\n",
" print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
" train_accs.append(train_accuracy)\n",
" val_accs.append(val_accuracy)\n",
"\n",
" return train_losses, val_losses, train_accs, val_accs, examples_seen"
]
},
{
"cell_type": "markdown",
"id": "9624cb30-3e3a-45be-b006-c00475b58ae8",
"metadata": {},
"source": [
"- The `evaluate_model` function used in the `train_classifier_simple` is the same as the one we used in chapter 5"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "bcc7bc04-6aa6-4516-a147-460e2f466eab",
"metadata": {},
"outputs": [],
"source": [
"# Same as chapter 5\n",
"def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n",
" model.eval()\n",
" with torch.no_grad():\n",
" train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n",
" val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n",
" model.train()\n",
" return train_loss, val_loss"
]
},
{
"cell_type": "markdown",
"id": "e807bfe9-364d-46b2-9e25-3b000c3ef6f9",
"metadata": {},
"source": [
"- The training takes about 5 minutes on a M3 MacBook Air laptop computer and less than half a minute on a V100 or A100 GPU"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "X7kU3aAj7vTJ",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "X7kU3aAj7vTJ",
"outputId": "504a033e-2bf8-41b5-a037-468309845513"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Ep 1 (Step 000000): Train loss 2.153, Val loss 2.392\n",
"Ep 1 (Step 000050): Train loss 0.617, Val loss 0.637\n",
"Ep 1 (Step 000100): Train loss 0.523, Val loss 0.557\n",
"Training accuracy: 70.00% | Validation accuracy: 72.50%\n",
"Ep 2 (Step 000150): Train loss 0.561, Val loss 0.489\n",
"Ep 2 (Step 000200): Train loss 0.419, Val loss 0.397\n",
"Ep 2 (Step 000250): Train loss 0.409, Val loss 0.353\n",
"Training accuracy: 82.50% | Validation accuracy: 85.00%\n",
"Ep 3 (Step 000300): Train loss 0.333, Val loss 0.320\n",
"Ep 3 (Step 000350): Train loss 0.340, Val loss 0.306\n",
"Training accuracy: 90.00% | Validation accuracy: 90.00%\n",
"Ep 4 (Step 000400): Train loss 0.136, Val loss 0.200\n",
"Ep 4 (Step 000450): Train loss 0.153, Val loss 0.132\n",
"Ep 4 (Step 000500): Train loss 0.222, Val loss 0.137\n",
"Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
"Ep 5 (Step 000550): Train loss 0.207, Val loss 0.143\n",
"Ep 5 (Step 000600): Train loss 0.083, Val loss 0.074\n",
"Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
"Training completed in 5.65 minutes.\n"
]
}
],
"source": [
"import time\n",
"\n",
"start_time = time.time()\n",
"\n",
"torch.manual_seed(123)\n",
"\n",
"optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)\n",
"\n",
"num_epochs = 5\n",
"train_losses, val_losses, train_accs, val_accs, examples_seen = train_classifier_simple(\n",
" model, train_loader, val_loader, optimizer, device,\n",
" num_epochs=num_epochs, eval_freq=50, eval_iter=5,\n",
" tokenizer=tokenizer\n",
")\n",
"\n",
"end_time = time.time()\n",
"execution_time_minutes = (end_time - start_time) / 60\n",
"print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
]
},
{
"cell_type": "markdown",
"id": "1261bf90-3ce7-4591-895a-044a05538f30",
"metadata": {},
"source": [
"- Similar to chapter 5, we use matplotlib to plot the loss function for the training and validation set"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "cURgnDqdCeka",
"metadata": {
"id": "cURgnDqdCeka"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_values(epochs_seen, examples_seen, train_values, val_values, label=\"loss\"):\n",
" fig, ax1 = plt.subplots(figsize=(5, 3))\n",
"\n",
" # Plot training and validation loss against epochs\n",
" ax1.plot(epochs_seen, train_values, label=f\"Training {label}\")\n",
" ax1.plot(epochs_seen, val_values, linestyle=\"-.\", label=f\"Validation {label}\")\n",
" ax1.set_xlabel(\"Epochs\")\n",
" ax1.set_ylabel(label.capitalize())\n",
" ax1.legend()\n",
"\n",
" # Create a second x-axis for tokens seen\n",
" ax2 = ax1.twiny() # Create a second x-axis that shares the same y-axis\n",
" ax2.plot(examples_seen, train_values, alpha=0) # Invisible plot for aligning ticks\n",
" ax2.set_xlabel(\"Examples seen\")\n",
"\n",
" fig.tight_layout() # Adjust layout to make room\n",
" plt.savefig(f\"{label}-plot.pdf\")\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "OIqRt466DiGk",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 307
},
"id": "OIqRt466DiGk",
"outputId": "b16987cf-0001-4652-ddaf-02f7cffc34db"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABXi0lEQVR4nO3deVxU9f748dfMwAz7viOCyuIK7uZOSamVZatfr7e0LG+FlZkt3krNfkWL3awsK7vJrVtZWVq3XELc9xUFF9wBlc2FVRhg5vz+GBidxAUEZsD38/E4D+Z8zuec855P5JvzOZ9zPipFURSEEEIIYZPU1g5ACCGEEJcniVoIIYSwYZKohRBCCBsmiVoIIYSwYZKohRBCCBsmiVoIIYSwYZKohRBCCBsmiVoIIYSwYZKohRBCCBsmiVoIcU1iY2OZNGmStcMQ4oYjiVqIJjJu3DhUKtUly7Bhw6wdmhDChtlZOwAhbiTDhg1j/vz5FmU6nc5K0QghmgO5ohaiCel0OgICAiwWT09PAFavXo1Wq2XdunXm+u+++y5+fn7k5uYCsGzZMgYMGICHhwfe3t7ceeedHDlyxFz/+PHjqFQqfvzxRwYOHIijoyO9evXi4MGDbNu2jZ49e+Li4sLw4cPJz8837zdu3DhGjhzJ66+/jq+vL25ubjzxxBNUVFRc9rvo9XqmTJlCcHAwzs7O9OnTh9WrV5u3Z2RkMGLECDw9PXF2dqZTp04sWbLkssf79NNPiYiIwMHBAX9/f+6//37zNqPRSEJCAm3atMHR0ZGYmBgWLlxosX9aWhrDhw/HxcUFf39/HnroIU6fPm3eHhsbyzPPPMOLL76Il5cXAQEBzJgx47LxCGErJFELYSNq7gE/9NBDFBYWsmvXLl577TW+/PJL/P39ASgtLWXy5Mls376d5ORk1Go199xzD0aj0eJY06dP59VXX2Xnzp3Y2dnxt7/9jRdffJEPP/yQdevWcfjwYaZNm2axT3JyMvv372f16tV8//33/PLLL7z++uuXjXfixIls2rSJBQsWsGfPHh544AGGDRvGoUOHAIiPj0ev17N27VpSU1N55513cHFxqfVY27dv55lnnmHmzJmkp6ezbNkyBg0aZN6ekJDA119/zWeffcbevXt57rnn+Pvf/86aNWsAKCgo4JZbbqFbt25s376dZcuWkZuby4MPPmhxnv/85z84OzuzZcsW3n33XWbOnElSUtI1/hcSwkoUIUSTGDt2rKLRaBRnZ2eL5c033zTX0ev1SteuXZUHH3xQ6dixo/L4449f8Zj5+fkKoKSmpiqKoijHjh1TAOXLL7801/n+++8VQElOTjaXJSQkKFFRURaxeXl5KaWlpeayuXPnKi4uLorBYFAURVEGDx6sPPvss4qiKEpGRoai0WiUkydPWsQzZMgQZerUqYqiKEqXLl2UGTNmXFPb/Pzzz4qbm5tSVFR0ybby8nLFyclJ2bhxo0X5+PHjldGjRyuKoihvvPGGctttt1lsz8rKUgAlPT3dHP+AAQMs6vTq1Ut56aWXrilGIaxF7lEL0YRuvvlm5s6da1Hm5eVl/qzVavn222+Jjo4mNDSUDz74wKLuoUOHmDZtGlu2bOH06dPmK+nMzEw6d+5srhcdHW3+XHM13qVLF4uyvLw8i2PHxMTg5ORkXu/bty8lJSVkZWURGhpqUTc1NRWDwUBkZKRFuV6vx9vbG4BnnnmGJ598kj///JO4uDjuu+8+i7guduuttxIaGkrbtm0ZNmwYw4YN45577sHJyYnDhw9z/vx5br31Vot9Kioq6NatGwC7d+9m1apVtV6xHzlyxBznX88fGBh4STsIYWskUQvRhJydnQkPD79inY0bNwJw9uxZzp49i7Ozs3nbiBEjCA0NZd68eQQFBWE0GuncufMl95Lt7e3Nn1UqVa1lf+0ur4uSkhI0Gg07duxAo9FYbKtJlo899hhDhw7ljz/+4M8//yQhIYH333+fp59++pLjubq6snPnTlavXs2ff/7JtGnTmDFjBtu2baOkpASAP/74g+DgYIv9agbilZSUMGLECN55551Ljh0YGGj+fHEbwPW3gxBNQRK1EDbkyJEjPPfcc8ybN48ffviBsWPHsmLFCtRqNWfOnCE9PZ158+YxcOBAANavX99g5969ezdlZWU4OjoCsHnzZlxcXAgJCbmkbrdu3TAYDOTl5ZljqU1ISAhPPPEETzzxBFOnTmXevHm1JmoAOzs74uLiiIuLY/r06Xh4eLBy5UpuvfVWdDodmZmZDB48uNZ9u3fvzs8//0xYWBh2dvLPmmhZ5DdaiCak1+vJycmxKLOzs8PHxweDwcDf//53hg4dyiOPPMKwYcPo0qUL77//Pi+88AKenp54e3vzxRdfEBgYSGZmJi+//HKDxVZRUcH48eN59dVXOX78ONOnT2fixImo1ZeOOY2MjGTMmDE8/PDDvP/++3Tr1o38/HySk5OJjo7mjjvuYNKkSQwfPpzIyEjOnTvHqlWr6NChQ63n/v333zl69CiDBg3C09OTJUuWYDQaiYqKwtXVlSlTpvDcc89hNBoZMGAAhYWFbNiwATc3N8aOHUt8fDzz5s1j9OjR5lHdhw8fZsGCBXz55ZeXXPUL0ZxIohaiCS1btsyiKxYgKiqKAwcO8Oabb5KRkcHvv/8OmLpsv/jiC0aPHs1tt91GTEwMCxYs4JlnnqFz585ERUXx0UcfERsb2yCxDRkyhIiICAYNGoRer2f06NFXfHxp/vz5/L//9/94/vnnOXnyJD4+Ptx0003ceeedABgMBuLj4zlx4gRubm4MGzbsknvuNTw8PPjll1+YMWMG5eXlRERE8P3339OpUycA3njjDXx9fUlISODo0aN4eHjQvXt3/vnPfwIQFBTEhg0beOmll7jtttvQ6/WEhoYybNiwWv/QEKI5USmKolg7CCGEdY0bN46CggIWL15s7VCEEH8hf2oKIYQQNkwStRBCCGHDpOtbCCGEsGFyRS2EEELYMEnUQgghhA2TRC2EEELYMEnU1+GTTz4hLCwMBwcH+vTpw9atW60dUqNZu3YtI0aMICgoCJVKdcljPIqiMG3aNAIDA3F0dCQuLs48i1KNs2fPMmbMGNzc3PDw8GD8+PHm10PW2LNnDwMHDsTBwYGQkBDefffdxv5qDSIhIYFevXrh6uqKn58fI0eOJD093aJOeXk58fHxeHt74+Liwn333WeevrJGZmYmd9xxB05OTvj5+fHCCy9QVVVlUWf16tV0794dnU5HeHg4iYmJjf31GsTcuXOJjo7Gzc0NNzc3+vbty9KlS83bb/T2qc3bb7+NSqVi0qRJ5jJpJ5gxYwYqlcpiad++vXl7i2sjq04J0owtWLBA0Wq1yldffaXs3btXefzxxxUPDw8lNzfX2qE1iiVLliivvPKK8ssvvyiAsmjRIovtb7/9tuLu7q4sXrxY2b17t3LXXXcpbdq0UcrKysx1hg0bpsTExCibN29W1q1bp4SHh5tnP1IURSksLFT8/f2VMWPGKGlpacr333+vODo6Kp9//nlTfc16Gzp0qDJ//nwlLS1NSUlJUW6//XaldevWSklJibnOE088oYSEhCjJycnK9u3blZtuuknp16+feXtVVZXSuXNnJS4uTtm1a5eyZMkSxcfHxzwblaIoytGjRxUnJydl8uTJyr59+5SPP/5Y0Wg0yrJly5r0+9bHb7/9pvzxxx/KwYMHlfT0dOWf//ynYm9vr6SlpSmKIu3zV1u3blXCwsKU6Oho86xliiLtpCiKMn36dKVTp05Kdna2ecnPzzdvb2ltJIm6nnr37q3Ex8eb1w0GgxIUFKQkJCRYMaqm8ddEbTQalYCAAOW9994zlxUUFCg6nU75/vvvFUVRlH379imAsm3bNnOdpUuXKiqVyjxV4qeffqp4enoqer3eXOell16ymI6xucjLy1MAZc2aNYqimNrD3t5e+emnn8x19u/frwDKpk2bFEUx/TGkVquVnJwcc525c+cqbm5u5jZ58cUXlU6dOlmca9SoUcrQoUMb+ys1Ck9PT+XLL7+U9vmL4uJiJSIiQklKSrKYXlTayWT69OlKTExMrdtaYhtJ13c9VFRUsGPHDuLi4sxlarWauLg4Nm3aZMXIrOPYsWPk5ORYtIe7uzt9+vQxt8emTZvw8PCgZ8+e5jpxcXGo1Wq2bNlirjNo0CC0Wq25ztChQ0lPT+fcuXNN9G0aRmFhIXBhCssdO3ZQWVlp0Ubt27endevWFm3UpUsX87SUYPr+RUVF7N2711zn4mPU1Gluv3cGg4EFCxZQWlpK3759pX3+Ij4+njvuuOOS7yLtdMGhQ4cICgqibdu2jBkzhszMTKBltpEk6no4ffo0BoPB4j8ymOb4/euECzeCmu98pfbIycnBz8/PYrudnR1eXl4WdWo7xsXnaA6MRiOTJk2if//+5jmic3Jy0Gq1eHh4WNT9axtd7ftfrk5RURFlZWWN8XUaVGpqKi4uLuh0Op544gkWLVpEx44dpX0usmDBAnbu3ElCQsIl26SdTPr06UNiYiLLli1j7ty5HDt2jIEDB1JcXNwi20gm5RCigcXHx5OWltagU1C2FFFRUaSkpFBYWMjChQsZO3Ysa9assXZYNiMrK4tnn32WpKQkHBwcrB2OzRo+fLj5c3R0NH369CE0NJQff/zRPE1rSyJX1PXg4+ODRqO5ZBRhbm4uAQEBVorKemq+85XaIyAggLy8PIvtVVVVnD171qJObce4+By2buLEifz++++sWrWKVq1amcsDAgKoqKigoKDAov5f2+hq3/9yddzc3JrFP1BarZbw8HB69OhBQkICMTExfPjhh9I+1Xbs2EFeXh7du3fHzs4OOzs71qxZw0cffYSdnR3+/v7STrXw8PAgMjKSw4cPt8jfJUnU9aDVaunRowfJycnmMqPRSHJyMn379rViZNbRpk0bAgICLNqjqKiILVu2mNujb9++FBQUsGPHDnOdlStXYjQa6dOnj7nO2rVrqaysNNdJSkoiKioKT0/PJvo29aMoChMnTmTRokWsXLmSNm3aWGzv0aMH9vb2Fm2Unp5OZmamRRulpqZa/EGTlJSEm5sbHTt2NNe5+Bg1dZrr753RaESv10v7VBsyZAipqamkpKSYl549ezJmzBjzZ2mnS5WUlHDkyBECAwNb5u9Skw9fayEWLFig6HQ6JTExUdm3b58yYcIExcPDw2IUYUtSXFys7Nq1S9m1a5cCKP/617+UXbt2KRkZGYqimB7P8vDwUH799Vdlz549yt13313r41ndunVTtmzZoqxfv16JiIiweDyroKBA8ff3Vx566CElLS1NWbBggeLk5NQsHs968sknFXd3d2X16tUWj4ycP3/eXOeJJ55QWrduraxcuVLZvn270rdvX6Vv377m7TWPjNx2221KSkqKsmzZMsXX17fWR0ZeeOEFZf/+/conn3zSbB6refnll5U1a9Yox44dU/bs2aO8/PLLikqlUv78809FUaR9LufiUd+KIu2kKIry/PPPK6tXr1aOHTumbNiwQYmLi1N8fHyUvLw8RVFaXhtJor4OH3/8sdK6dWtFq9UqvXv3VjZv3mztkBrNqlWrFOCSZezYsYqimB7Reu211xR/f39Fp9MpQ4YMUdLT0y2OcebMGWX06NGKi4uL4ubmpjzyyCNKcXGxRZ3du3crAwYMUHQ6nRIcHKy8/fbbTfUVr0ttbQMo8+fPN9cpKytTnnrqKcXT01NxcnJS7rnnHiU7O9viOMePH1eGDx+uODo6Kj4+Psrzzz+vVFZWWtRZtWqV0rVrV0Wr1Spt27a1OIcte/TRR5XQ0FBFq9Uqvr6+ypAhQ8xJWlGkfS7nr4la2sn0mFRgYKCi1WqV4OBgZdSoUcrhw4fN21taG8nsWUIIIYQNk3vUQgghhA2TRC2EEELYMEnUQgghhA2TRC2EEELYMEnUQgghhA2TRC2EEELYMEnU10Gv1zNjxgz0er21Q7Fp0k5XJ210ddJGVydtdHXNsY2s+hx1QkICv/zyCwcOHMDR0ZF+/frxzjvvEBUVddl9EhMTeeSRRyzKdDod5eXljR3uJYqKinB3d6ewsBA3N7cmP39zIe10ddJGVydtdHXSRlfXHNvIqlfUa9asIT4+ns2bN5OUlERlZSW33XYbpaWlV9zPzc2N7Oxs85KRkdFEEQshhBBNy6rTXC5btsxiPTExET8/P3bs2MGgQYMuu59KpWo2sykJIYQQ18Om5qMuLCwEwMvL64r1SkpKCA0NxWg00r17d9566y06dep0Teeoqqpi165d+Pv7o1ZfX4dCcXExACdPnqSoqOi6jtWSSTtdnbTR1UkbXZ200dXZShsZjUZyc3Pp1q0bdnZXTsU2865vo9HIXXfdRUFBAevXr79svU2bNnHo0CGio6MpLCxk1qxZrF27lr1791rM/1tDr9dbDBrYsWMHt9xyS6N8ByGEEKIutm7dSq9eva5Yx2YS9ZNPPsnSpUtZv359rQn3ciorK+nQoQOjR4/mjTfeuGT7jBkzeP311y8p37p1K4GBgdcVsxBCCFEf2dnZ9O7dm4yMDFq3bn3FujaRqCdOnMivv/7K2rVradOmTZ33f+CBB7Czs+P777+/ZNtfr6hPnjxJx44dycrKqtMfBEIIIURDOXHiBCEhIdeUi6w66ltRFCZOnMiiRYtYuXJlvZK0wWAgNTX1slfHOp0ONzc38+Lq6nq9YQshhBBNxqqDyeLj4/nuu+/49ddfcXV1JScnBwB3d3ccHR0BePjhhwkODiYhIQGAmTNnctNNNxEeHk5BQQHvvfceGRkZPPbYY1b7HkIIIURjsWqinjt3LgCxsbEW5fPnz2fcuHEAZGZmWozOPnfuHI8//jg5OTl4enrSo0cPNm7cSMeOHZsqbCGEEKLJ2MQ96qZUl/sCQogbj8FgoLKy0tphiGbO3t4ejUZz2e11yUU29Ry1EEJYi6Io5OTkUFBQYO1QRAvh4eFBQEAAKpXquo4jifp6lBVA5mZwbwUBna0djRDiOtQkaT8/P5ycnK77H1dx41IUhfPnz5OXlwdw3Y8CS6K+Hiv/H2ybB32egOHvWDsaIUQ9GQwGc5L29va2djiiBagZEJ2Xl4efn98Vu8GvRqa5vB5h/U0/j2+wbhxCiOtSc0/aycnJypGIlqTm9+l6xzxIor4eodWJOjcNzp+1bixCiOsm3d2iITXU75Mk6uvh4gc+kYACmZusHY0QQogWSBL19QobYPop3d9CiBYiLCyM2bNnX3P91atXo1KpGn3EfGJiIh4eHo16Dlskifp61XR/H19n3TiEEDcclUp1xWXGjBn1Ou62bduYMGHCNdfv168f2dnZuLu71+t84spk1Pf1qrmizkk1Pa7l6GHNaIQQN5Ds7Gzz5x9++IFp06aRnp5uLnNxcTF/VhQFg8Fw1bmPAXx9fesUh1arJSAgoE77iGsnV9TXyzUAvMMx3afebO1ohBA3kICAAPPi7u6OSqUyrx84cABXV1eWLl1Kjx490Ol0rF+/niNHjnD33Xfj7++Pi4sLvXr1YsWKFRbH/WvXt0ql4ssvv+See+7BycmJiIgIfvvtN/P2v3Z913RRL1++nA4dOuDi4sKwYcMs/rCoqqrimWeewcPDA29vb1566SXGjh3LyJEj69QGc+fOpV27dmi1WqKiovjmm2/M2xRFYcaMGbRu3RqdTkdQUBDPPPOMefunn35KREQEDg4O+Pv7c//999fp3E1FEnVDkO5vIVocRVE4X1FllaUh3+z88ssv8/bbb7N//36io6MpKSnh9ttvJzk5mV27djFs2DBGjBhBZmbmFY/z+uuv8+CDD7Jnzx5uv/12xowZw9mzl3/a5fz588yaNYtvvvmGtWvXkpmZyZQpU8zb33nnHb799lvmz5/Phg0bKCoqYvHixXX6bosWLeLZZ5/l+eefJy0tjX/84x888sgjrFq1CoCff/6ZDz74gM8//5xDhw6xePFiunTpAsD27dt55plnmDlzJunp6SxbtoxBgwbV6fxNRbq+G0LYANj5H8iQAWVCtBRllQY6TltulXPvmzkUJ23D/PM8c+ZMbr31VvO6l5cXMTEx5vU33niDRYsW8dtvvzFx4sTLHmfcuHGMHj0agLfeeouPPvqIrVu3MmzYsFrrV1ZW8tlnn9GuXTsAJk6cyMyZM83bP/74Y6ZOnco999wDwJw5c1iyZEmdvtusWbMYN24cTz31FACTJ09m8+bNzJo1i5tvvpnMzEwCAgKIi4vD3t6e1q1b07t3b8A04ZOzszN33nknrq6uhIaG0q1btzqdv6nIFXVDqLmizt4N5YXWjUUIIS7Ss2dPi/WSkhKmTJlChw4d8PDwwMXFhf3791/1ijo6Otr82dnZGTc3N/MrMmvj5ORkTtJgeo1mTf3CwkJyc3PNSRNAo9HQo0ePOn23/fv3079/f4uy/v37s3//fgAeeOABysrKaNu2LY8//jiLFi2iqqoKgFtvvZXQ0FDatm3LQw89xLfffsv58+frdP6mIlfUDcE9GDzbwLljkLkFIm+zdkRCiOvkaK9h38yhVjt3Q3F2drZYnzJlCklJScyaNYvw8HAcHR25//77qaiouOJx7O3tLdZVKhVGo7FO9Zt6ssaQkBDS09NZsWIFSUlJPPXUU7z33nusWbMGV1dXdu7cyerVq/nzzz+ZNm0aM2bMYNu2bTb3CJhcUTeUqNshchhona9eVwhh81QqFU5aO6ssjfmGtA0bNjBu3DjuueceunTpQkBAAMePH2+089XG3d0df39/tm3bZi4zGAzs3LmzTsfp0KEDGzZY3nLcsGEDHTt2NK87OjoyYsQIPvroI1avXs2mTZtITU0FwM7Ojri4ON5991327NnD8ePHWbly5XV8s8YhV9QNZdhb1o5ACCGuKiIigl9++YURI0agUql47bXXrnhl3FiefvppEhISCA8Pp3379nz88cecO3euTn+kvPDCCzz44IN069aNuLg4/ve///HLL7+YR7EnJiZiMBjo06cPTk5O/Pe//8XR0ZHQ0FB+//13jh49yqBBg/D09GTJkiUYjUaioqIa6yvXmyRqIYS4gfzrX//i0UcfpV+/fvj4+PDSSy9RVFTU5HG89NJL5OTk8PDDD6PRaJgwYQJDhw6t0yxTI0eO5MMPP2TWrFk8++yztGnThvnz5xMbGwuY5oN+++23mTx5MgaDgS5duvC///0Pb29vPDw8+OWXX5gxYwbl5eVERETw/fff06lTp0b6xvWnUpr6poGVnThxgpCQELKysmjVqtV1H6/KYESjVl34K7AgC9R24HZ9848KIZpOeXk5x44do02bNjg4OFg7nBuS0WikQ4cOPPjgg7zxxhvWDqdBXOn3qi65SO5RX4cXF+6m+xtJpJ2s/mt02T9hdmfY+oV1AxNCCBuXkZHBvHnzOHjwIKmpqTz55JMcO3aMv/3tb9YOzeZIor4O585XUlRexZqD1Y8o+HcClQbOn7FuYEIIYePUajWJiYn06tWL/v37k5qayooVK+jQoYO1Q7M5co/6OgyO9CVpXy5rDuYz8ZYI6DQSOt4FOldrhyaEEDYtJCTkkhHbonaSqK/D4EjTi+t3ZhZQWFaJu6M8miWEEKJhSdf3dQjxcqKdrzMGo8KGw6ctN1rhcQchhBAtjyTq6zQ40g+ANen5poKTO2DeLfD1XVaMSgghREshifo6DY4ydX+vOZhvej2eg4cpWWdtgcoy6wYnhBCi2ZNEfZ36tPFCZ6cmp6ic9Nxi8GoLroFgqIAT265+ACGEEOIKrJqoExIS6NWrF66urvj5+TFy5EjS09Ovut9PP/1E+/btcXBwoEuXLnWeGq0hOdhr6NvOG6ju/lapTNNeAhyXEY1CCCGuj1UT9Zo1a4iPj2fz5s0kJSVRWVnJbbfdRmlp6WX32bhxI6NHj2b8+PHs2rWLkSNHMnLkSNLS0powcks1o7/XHKy+T10z7eXx9VaKSAghrl1sbCyTJk0yr4eFhTF79uwr7qNSqVi8ePF1n7uhjnMlM2bMoGvXro16jsZk1US9bNkyxo0bR6dOnYiJiSExMZHMzEx27Nhx2X0+/PBDhg0bxgsvvECHDh1444036N69O3PmzGnCyC3VJOptx89Sqq+6cEV9YhtUllstLiFEyzZixAiGDRtW67Z169ahUqnYs2dPnY+7bds2JkyYcL3hWbhcsszOzmb48OENeq6WxqbuURcWFgLg5eV12TqbNm0iLi7Oomzo0KFs2rSp1vp6vZ6ioiLzUlxc3HABV2vj40xrLycqDQobj5wB73Bw8QeD3jSwTAghGsH48eNJSkrixIkTl2ybP38+PXv2JDo6us7H9fX1xcnJqSFCvKqAgAB0Ol2TnKu5splEbTQamTRpEv3796dz586XrZeTk4O/v79Fmb+/Pzk5ObXWT0hIwN3d3bxcPE9pQ1GpVBd1f+eZ7lNL97cQopHdeeed+Pr6kpiYaFFeUlLCTz/9xPjx4zlz5gyjR48mODgYJycnunTpwvfff3/F4/616/vQoUMMGjQIBwcHOnbsSFJS0iX7vPTSS0RGRuLk5ETbtm157bXXqKysBEzTTb7++uvs3r0blco0iVFNzH/t+k5NTeWWW27B0dERb29vJkyYQElJiXn7uHHjGDlyJLNmzSIwMBBvb2/i4+PN57oWRqORmTNn0qpVK3Q6HV27dmXZsmXm7RUVFUycOJHAwEAcHBwIDQ0lISEBAEVRmDFjBq1bt0an0xEUFMQzzzxzzeeuD5tJ1PHx8aSlpbFgwYIGPe7UqVMpLCw0L/v27WvQ49eoSdSr06sf0wqrTtQZkqiFaNYqSuu+GKou7G+oMpX99XHNy+1bB3Z2djz88MMkJiZy8USIP/30EwaDgdGjR1NeXk6PHj34448/SEtLY8KECTz00ENs3br1ms5hNBq599570Wq1bNmyhc8++4yXXnrpknqurq4kJiayb98+PvzwQ+bNm8cHH3wAwKhRo3j++efp1KkT2dnZZGdnM2rUqEuOUVpaytChQ/H09GTbtm389NNPrFixgokTJ1rUW7VqFUeOHGHVqlX85z//ITEx8ZI/Vq7kww8/5P3332fWrFns2bOHoUOHctddd3Ho0CEAPvroI3777Td+/PFH0tPT+fbbbwkLCwPg559/5oMPPuDzzz/n0KFDLF68mC5dulzzuevDJl4hOnHiRH7//XfWrl171em+AgICyM3NtSjLzc0lICCg1vo6nc6iW6Wx5l3t284brUbNiXNlHD1dSrvQ6vvUWdugSg920rUjRLP0VlDd93kgETrdY/p84H/w0zgIHQCP/HGhzuwutU/gM6OwTqd69NFHee+991izZo15Hub58+dz3333mXsSp0yZYq7/9NNPs3z5cn788Ud69+591eOvWLGCAwcOsHz5coKCTG3x1ltvXXJf+dVXXzV/DgsLY8qUKSxYsIAXX3wRR0dHXFxcsLOzu+y/1QDfffcd5eXlfP311zg7m17JPGfOHEaMGME777xj7k319PRkzpw5aDQa2rdvzx133EFycjKPP/74NbXZrFmzeOmll/i///s/AN555x1WrVrF7Nmz+eSTT8jMzCQiIoIBAwagUqkIDQ0175uZmUlAQABxcXHY29vTunXra2rH62HVK2pFUZg4cSKLFi1i5cqVtGnT5qr79O3bl+TkZIuypKQk+vbt21hhXhNnnR292ngC1Y9p+UaBkw9UlcHJnVaNTQjRcrVv355+/frx1VdfAXD48GHWrVvH+PHjATAYDLzxxht06dIFLy8vXFxcWL58OZmZmdd0/P379xMSEmJO0kCt/97+8MMP9O/fn4CAAFxcXHj11Vev+RwXnysmJsacpAH69++P0Wi0eHS3U6dOaDQa83pgYCB5eXnXdI6ioiJOnTpF//79Lcr79+/P/v37AVP3ekpKClFRUTzzzDP8+eef5noPPPAAZWVltG3blscff5xFixZRVVVFY7LqFXV8fDzfffcdv/76K66urub7zO7u7jg6OgLw8MMPExwcbL4/8OyzzzJ48GDef/997rjjDhYsWMD27dv54gvrzwE9ONKXDYfPsOZgPo8OaGPq/t73q6n7O9S6f0gIIerpn6fqvo/moh609iNMx1D95bpoUur1xXWR8ePH8/TTT/PJJ58wf/582rVrx+DBgwF47733+PDDD5k9ezZdunTB2dmZSZMmUVFR0WDn37RpE2PGjOH1119n6NChuLu7s2DBAt5///0GO8fF7O3tLdZVKhXGBpxfoXv37hw7doylS5eyYsUKHnzwQeLi4li4cCEhISGkp6ezYsUKkpKSeOqpp8w9Gn+Nq6FY9Yp67ty5FBYWEhsbS2BgoHn54YcfzHUyMzPJzs42r/fr14/vvvuOL774gpiYGBYuXMjixYuvOACtqcRGmd77vfnoGcorDaauLjB1fwshmietc90XzUXXQBo7U5m947Udtx4efPBB1Go13333HV9//TWPPvooKpUKgA0bNnD33Xfz97//nZiYGNq2bcvBgwev+dgdOnQgKyvL4t/hzZs3W9TZuHEjoaGhvPLKK/Ts2ZOIiAgyMjIsv65Wi8FguOq5du/ebfEujQ0bNqBWq4mKirrmmK/Ezc2NoKCgS6bY3LBhg8VgYzc3N0aNGsW8efP44Ycf+Pnnnzl79iwAjo6OjBgxgo8++ojVq1ezadMmUlMb7g+vv7LqFfXFgx8uZ/Xq1ZeUPfDAAzzwwAONENH1ifBzIdDdgezCcjYfPUNsx7shuAcExlg7NCFEC+bi4sKoUaOYOnUqRUVFjBs3zrwtIiKChQsXsnHjRjw9PfnXv/5Fbm7uNT8BExcXR2RkJGPHjuW9996jqKiIV155xaJOREQEmZmZLFiwgF69evHHH3+waNEiizphYWEcO3aMlJQUWrVqhaur6yWPZY0ZM4bp06czduxYZsyYQX5+Pk8//TQPPfTQJU/7XI8XXniB6dOn065dO7p27cr8+fNJSUnh22+/BeBf//oXgYGBdOvWDbVazU8//URAQAAeHh4kJiZiMBjo06cPTk5O/Pe//8XR0dHiPnZDs5lR3y2B5WNa+eDqD616WP51LYQQjWD8+PGcO3eOoUOHWtxPfvXVV+nevTtDhw4lNjaWgIAARo4cec3HVavVLFq0iLKyMnr37s1jjz3Gm2++aVHnrrvu4rnnnmPixIl07dqVjRs38tprr1nUue+++xg2bBg333wzvr6+tT4i5uTkxPLlyzl79iy9evXi/vvvZ8iQIQ3+QqtnnnmGyZMn8/zzz9OlSxeWLVvGb7/9RkREBGAawf7uu+/Ss2dPevXqxfHjx1myZAlqtRoPDw/mzZtH//79iY6OZsWKFfzvf//D29u7QWO8mEq5lsvaFuTEiROEhISQlZV11RHm9bE0NZsnv91JW19nVj4f2+DHF0I0vPLyco4dO0abNm1wcHCwdjiihbjS71VdcpFc6jWw/hE+aNQqjuaXknX2PCGGE7DpY1BpYMRsa4cnhBCimZGu7wbm5mBPj9amx7RWH8w3vUZ059eQ+pPlSxCEEEKIayCJuhEMjqq+T52eD36dYMBkuP8r4Ia6yyCEEKIBSKJuBDUDyjYeOY3eqEDcdIgcCprGecZOCCFEyyWJuhF0DHTDx0XH+QoDO46fs3Y4QgghmjFJ1I1ArVYxKNIHqH5My2iAw8mw8k3TZyGETWrIt1sJ0VC/TzLqu5HERvnxy86TrDmYz9RhkfDTI6AvhPa3Q1A3a4cnhLiIVqtFrVZz6tQpfH190Wq15jd7CVFXiqJQUVFBfn4+arUarVZ7XceTRN1IBob7oFLBgZxisosrCAztCweXwfENkqiFsDFqtZo2bdqQnZ3NqVP1eLe3ELVwcnKidevWqNXX13ktibqReDpriWnlQUpWAWsP5jMqtH91ol4P/SZe/QBCiCal1Wpp3bo1VVVVV30ntRBXo9FosLOza5CeGUnUjWhwpC8pWQWsOZjPqNjqKdUyN5ruU6s1V95ZCNHkVCoV9vb2jTYLkhD1IYPJGlFs9fPU6w6dpsqvC2hdobwQcvdaOTIhhBDNhSTqRhTdygMPJ3uKy6vYdbIEWt9k2nB8vXUDE0II0WxIom5EGrWKgREXvaUsrLr7O2PDFfYSQgghLpBE3chiq99StvpgHoQOMBVmbAB5XlMIIcQ1kETdyAZWv/gk7WQR+a4dwN4Zys5B3j4rRyaEEKI5kETdyPxcHegU5AbAuqMF0LqPaYN0fwshhLgGkqibQM3o7zUH8yG0+j61DCgTQghxDSRRN4HBkX4ArD2Yj+Hi+9SKTHsphBDiyuSFJ02gW2sPXHV2nDtfSZrSlpiIoaYu8Co92DtYOzwhhBA2TBJ1E7DXqBkQ4cPStBxWHy4kZsyP1g5JCCFEMyFd301k8MWPaQkhhBDXSBJ1ExlUnah3ZxVwrrQCinNh72K5Ty2EEOKKJFE3kSAPRyL9XTAqsOHgKfgwGn4aC2cOWzs0IYQQNsyqiXrt2rWMGDGCoKAgVCoVixcvvmL91atXo1KpLllycnKaJuDrFBtlGv29+nAhhPSBgGg4f9bKUQkhhLBlVk3UpaWlxMTE8Mknn9Rpv/T0dLKzs82Ln59fI0XYsGruU685mI9xzM/wxLoLL0ARQgghamHVUd/Dhw9n+PDhdd7Pz88PDw+Phg+okfUM88RJqyG/WM/+vPN0CnK3dkhCCCFsXLO8R921a1cCAwO59dZb2bCh+byKU2enoV87b6D6LWUAlWVQcd6KUQkhhLBlzSpRBwYG8tlnn/Hzzz/z888/ExISQmxsLDt37rzsPnq9nqKiIvNSXFzchBFfyvyYVno+LHkR3m4NqT9ZNSYhhBC2q1m98CQqKoqoqCjzer9+/Thy5AgffPAB33zzTa37JCQk8PrrrzdViFdlep3oXnZmnEPfxgWdocL0OtEeY60dmhBCCBvUrK6oa9O7d28OH778I05Tp06lsLDQvOzbZ93pJVt7O9HWx5kqo8JuTRdT4fH18jy1EEKIWjX7RJ2SkkJgYOBlt+t0Otzc3MyLq6trE0ZXu5qXn/x+rhWo7aHoJJw7bt2ghBBC2CSrJuqSkhJSUlJISUkB4NixY6SkpJCZmQmYroYffvhhc/3Zs2fz66+/cvjwYdLS0pg0aRIrV64kPj7eGuHX2+DqaS9XHCpCCe5uKpRpL4UQQtTCqveot2/fzs0332xenzx5MgBjx44lMTGR7Oxsc9IGqKio4Pnnn+fkyZM4OTkRHR3NihUrLI7RHPRt643OTs2pwnLOdeqNV9YW033q7g9ZOzQhhBA2RqUoN9bN0RMnThASEkJWVhatWrWyWhwPf7WVtQfzmXtTAcNTngL31vBcqtXiEUII0XTqkoua/T3q5qrmMa2FecGg0kBhJpzLsHJUQgghbI0kaiupSdTrMsowBHUzFWY0n5e3CCGEaBr1StRZWVmcOHHCvL5161YmTZrEF1980WCBtXTtfJ1p5elIhcHICbfqRH1cErUQQghL9UrUf/vb31i1ahUAOTk53HrrrWzdupVXXnmFmTNnNmiALZVKpTJfVa/VV7/E5fg6K0YkhBDCFtUrUaelpdG7d28AfvzxRzp37szGjRv59ttvSUxMbMj4WrSaRP1dTpDpPnVBBhSeuMpeQgghbiT1StSVlZXodDoAVqxYwV133QVA+/btyc7ObrjoWrh+4T7Ya1TsPwt63y5g5wD56dYOSwghhA2pV6Lu1KkTn332GevWrSMpKYlhw4YBcOrUKby9vRs0wJbMRWdHz1AvAP4XlQAvZ0L4ECtHJYQQwpbUK1G/8847fP7558TGxjJ69GhiYmIA+O2338xd4uLa1Lyl7I9MO7DTWTkaIYQQtqZebyaLjY3l9OnTFBUV4enpaS6fMGECTk5ODRbcjSA2ype3lx5g09EzlFcacLDXmCboUKmsHZoQQggbUK8r6rKyMvR6vTlJZ2RkMHv2bNLT0/Hz82vQAFu6KH9X/N10lFcaObXkXfjkJkj72dphCSGEsBH1StR33303X3/9NQAFBQX06dOH999/n5EjRzJ37twGDbClu/gxrdxTGZC/XyboEEIIYVavRL1z504GDhwIwMKFC/H39ycjI4Ovv/6ajz76qEEDvBHERpl6Ib4quQke/AZuec3KEQkhhLAV9UrU58+fN8/r/Oeff3LvvfeiVqu56aabyMiQ91XXVf9wHzRqFUlnfDkRGAfOMnJeCCGESb0SdXh4OIsXLyYrK4vly5dz2223AZCXl4ebm1uDBngjcHe0p1uIBwBrD562bjBCCCFsSr0S9bRp05gyZQphYWH07t2bvn37Aqar627dujVogDeKmvvU+1J3wOq3YcvnVo5ICCGELahXor7//vvJzMxk+/btLF++3Fw+ZMgQPvjggwYL7kZSc5+6JCsVVifA9q+sHJEQQghbUK/nqAECAgIICAgwz6LVqlUrednJdegU5Ia3s5Y1pRHgAOQfgNLT4Oxj7dCEEEJYUb2uqI1GIzNnzsTd3Z3Q0FBCQ0Px8PDgjTfewGg0NnSMNwS1WsWgSF/O4UaeYztTocxPLYQQN7x6JepXXnmFOXPm8Pbbb7Nr1y527drFW2+9xccff8xrr8mjRfUVW/060c3GDqYCeZ5aCCFuePXq+v7Pf/7Dl19+aZ41CyA6Oprg4GCeeuop3nzzzQYL8EYyINwHlQqWFrfjLi1wXK6ohRDiRlevK+qzZ8/Svn37S8rbt2/P2bNnrzuoG5W3i47oYHe2GqvbNm8vnJf2FEKIG1m9EnVMTAxz5sy5pHzOnDlER0dfd1A3ssFRfpzBnWxtqKkgY6N1AxJCCGFV9er6fvfdd7njjjtYsWKF+RnqTZs2kZWVxZIlSxo0wBvN4EhfPko+xNqKKEaRYbpP3eFOa4clhBDCSup1RT148GAOHjzIPffcQ0FBAQUFBdx7773s3buXb775pqFjvKHEtHLH3dGedRVRpoIMGVAmhBA3sno/Rx0UFHTJoLHdu3fz73//my+++OK6A7tR2WnUDIjwYcue6pHfOWlQdg4cPa+8oxBCiBapXlfUonHFRvqSjwcnNK0ABTI2WTskIYQQVmLVRL127VpGjBhBUFAQKpWKxYsXX3Wf1atX0717d3Q6HeHh4SQmJjZ6nE2t5r3faysiTQXy4hMhhLhhWTVRl5aWEhMTwyeffHJN9Y8dO8Ydd9zBzTffTEpKCpMmTeKxxx6zeN94S+Dn5kCHQDf+Z+jL/qh46HyftUMSQghhJXW6R33vvfdecXtBQUGdTj58+HCGDx9+zfU/++wz2rRpw/vvvw9Ahw4dWL9+PR988AFDhw6t07ltXWyUL3OzO/GFOpgPgrtaOxwhhBBWUqcrand39ysuoaGhPPzww40VK5s2bSIuLs6ibOjQoWza1PLu4Zq7vw/mYzQqVo5GCCGEtdTpinr+/PmNFcc1ycnJwd/f36LM39+foqIiysrKcHR0vGQfvV6PXq83rxcXFzd6nA2hR6gnLjo7KkvPkrXxR0J9XKH97dYOSwghRBNr8aO+ExISLK76O3bsaO2Qrom9Rk3/cG9uUacQumICrJtl7ZCEEEJYQbNK1AEBAeTm5lqU5ebm4ubmVuvVNMDUqVMpLCw0L/v27WuKUBvE4Eg/thg7kKUJgeCeoEgXuBBC3GiaVaLu27cvycnJFmVJSUnm15jWRqfT4ebmZl5cXV0bO8wGMzjKl2y8GXz+HQpj3wSVytohCSGEaGJWTdQlJSWkpKSQkpICmB6/SklJITMzEzBdDV88OO2JJ57g6NGjvPjiixw4cIBPP/2UH3/8keeee84a4Te6YA9HIvxcMCqw/vBpa4cjhBDCCqyaqLdv3063bt3o1q0bAJMnT6Zbt25MmzYNgOzsbHPSBmjTpg1//PEHSUlJxMTE8P777/Pll1+2uEezLlYz+nv9gZOQk2rlaIQQQjQ1laLcWDc+T5w4QUhICFlZWbRq1cra4VzVukP5TPp3EhscnkWnNqJ6ORO0ztYOSwghxHWoSy5qVveob0S9wrw4b+/FacUNlbEKsrZYOyQhhBBNSBK1jXOw19C3nTdbjO1NBcdl2kshhLiRSKJuBgZH+rLZWP3893GZoEMIIW4kkqibgcGRvmwxmuanVk7ugIrzVo5ICCFEU5FE3QyE+Tij9gwjW/FCZayEE9usHZIQQogmIom6mRgc5cfm6qtquU8thBA3DknUzcTgqIu6vzMkUQshxI1CEnUzcVNbb3aqTAPKlBM7oLLcyhEJIYRoCpKomwknrR3+YZ3IVTxQG/Rwcru1QxJCCNEEJFE3I4Oj/Mzd33KfWgghbgySqJuR2IvuUxuOSaIWQogbgSTqZqSdrwtHnbtRoWgo1BtlfmohhLgBSKJuRlQqFWFRXYnWf8lHQe/J/NRCCHEDkETdzAyO8qMcHWsP5ls7FCGEEE1AEnUz0z/cGzu1iqOnS8nKOW3tcIQQQjQySdTNjKuDPbGtVPyu/ScB87pAVYW1QxJCCNGIJFE3Q907hBOoOoO94Tzkplk7HCGEEI1IEnUzFBvlzz8qnmOwcS56/xhrhyOEEKIRSaJuhjoEupLhEkNGhTvbj5+zdjhCCCEakZ21AxB1p1KpGBzpy8IdJ9Cvfh/Wp4J/ZwjoDAFdwLc92OmsHaYQQogGIIm6mYqNMiVqx+wtYNgBx9dd2Ki2A5/IC8nbvzqBu/hZL2AhhBD1Iom6mRoQ7oOdWsX08w8Sre5JR3UmPXQniVCO42Qogrx9piX1xws7OfuZEnf0/0HMKOsFL4QQ4ppJom6mPJy0fDS6G4t3+bEmK5yFxXqoBFAI4Cwd1Rl0056gt9MpIozH8SzPQlWaB0dWQut+Fw50LgN++DsE94ARs630bYQQQlyOJOpm7PYugdzeJRBFUThVWE5KZgG7Ms+RkuXFhpO+rCzvDtXTVjtSTpTqBANcszEeb4u//XG6tfagQ+Ee7HP2AH95b/h31Vfc5u7zLuDVBtSaJv2OQghxo5NE3QKoVCqCPRwJ9nDkjuhAACoNRg5kF5OSdY5dWQWkZBaQctqBlKJwKAL27wXA3+4893q/ShtnZxx3n6Jbaw+CXe1QHVkJhgo4uOzCieydwK+j5X1v3/bg6NHo31FRFIr1VZwpqeBMiZ7TJRWUVVbRK8yLVp5OjX5+IYSwFpWi3FhTMJ04cYKQkBCysrJo1aqVtcNpUgXnK0jJKjAvuzILKCyrvKSev7Md9/qf4ianbNpzDJ/SQ2jyD0BVWe0HdvE3DV6Lex1a9TCVGSpNg9quMHGIvsrA2dIKzpRUcLpEb0rCpfrqddNnc3lJBRUGY63HiWnlzrDOgQzvHECYj3Od20UIIZpaXXKRTSTqTz75hPfee4+cnBxiYmL4+OOP6d27d611ExMTeeSRRyzKdDod5eXl13SuGzlR/5WiKBw/c766u9yUvPedKqLKaPkroVJBe18n4vxLuMk5m/aqDLyKD6LKTYPiU+Z6xsdWUejZmTOlejTb5hGy6z3Sg+9jeatnOFOi50yxHm3hEfaVe5NbaqC4vKrOMbvo7PB20eLtrEUBUrIKLGb77BDoxu2dAxjeJYBwP9f6No0QQjSquuQiq3d9//DDD0yePJnPPvuMPn36MHv2bIYOHUp6ejp+frU/TuTm5kZ6erp5XSXTPdaLSqWijY8zbXycube76RelvNLA3lOF7MosMHeZnywoY3/eefbnqfmYYCAYZ+1AurRyx9W1DMeio3iWHefnT49TYswGYKbdRh62O8/aIwV8lH4IAF/Osc0hnkpFQ4bizxH7II4STK62Neec2nDerS0ubp54O2vxdtHh7aLFx0WLt7MOH1cd3s5aHOwt75HnF+v5c18OS1Nz2HT0DPuzi9ifXcT7SQeJ8HNheOcAhncJpH2Aq/yeCCGaJatfUffp04devXoxZ84cAIxGIyEhITz99NO8/PLLl9RPTExk0qRJFBQU1Ot8ckVdd3nF1QPVqhP3nhMFlFYYLlvf3dEef2cVnRzO4ujsisazNd4uWiINh7ht62PYGc5f/mSuQeAbaepKr1lC+oC9w1XjPFdaQdK+XJakZbPh8GkqDRd+tcO8nRjexdQ93iXYXZK2EMKqmk3Xd0VFBU5OTixcuJCRI0eay8eOHUtBQQG//vrrJfskJiby2GOPERwcjNFopHv37rz11lt06tSp1nPo9Xr0er15/eTJk3Ts2FES9XUwGBUO5RWTeqIQO40Kb+eaq18dnk5atHZXeDOt0WjqLs9Ph9OH4HT1z/x0KM2rfZ8phy68rCXtFyjIhIhbwb/2/+YAhWWVJO/PZWlaDmsO5lNRdeH+drCHo/lKu1uIB2q1JG0hRNNqNl3fp0+fxmAw4O/vb1Hu7+/PgQMHat0nKiqKr776iujoaAoLC5k1axb9+vVj7969tX7ZhIQEXn/99UaJ/0alUatoH+BG+wC3uu+sVoN7K9MSPsRyW9m56uR9sDqRH4TiHHD2vVBnzw+mkeha5wuJ+uwx2PVf07PgwT3A1R93R3vu7d6Ke7u3okRfxaoDeSxNy2bVgXxOFpTx5fpjfLn+GAFuDgzrHMCwzgH0CvNCI0lbCGFjrHpFferUKYKDg9m4cSN9+/Y1l7/44ousWbOGLVu2XPUYlZWVdOjQgdGjR/PGG29csl2uqFuYrfMgczP0fcqUlAF2fgO/TbxQxz0EgrtfSNyBXUHnAkBZhYE1B/NYmpZD8v48SvQXBrT5uGi5rVMAt3cOpE9bL+w1MmeNEKJxNJsrah8fHzQaDbm5uRblubm5BAQEXNMx7O3t6datG4cPH651u06nQ6e7MEFFUVFR/QMW1tf7cdNyMe920O3vcHIn5O2HwizTsq/61olKDb4dILg7jsE9GBbcg2EPdKHcqGLD4dMsSc0haV8Op0sq+G5LJt9tycTDyZ7bOvozvHMg/cN9rtydL4QQjciqiVqr1dKjRw+Sk5PN96iNRiPJyclMnDjxyjtXMxgMpKamcvvttzdipMKmhfYzLQD6YsjeDSe2w8kdpuRddALy9pqWXd+Y6gV1w2HCaoZ08GdIB38qCvzYlKth2d4clu/N5WxpBT9uP8GP20/g6mBHXAd/hncOYFCk7yUjz4UQojFZ/fGsyZMnM3bsWHr27Env3r2ZPXs2paWl5melH374YYKDg0lISABg5syZ3HTTTYSHh1NQUMB7771HRkYGjz32mDW/hrAVOlcIG2BaahTnVCftHReS98UD0QyVaOd0ZbDWhcFPrOeNuzuz9dhZlqeeYMm+0+QX61m06ySLdp3ESavhlvZ+DO8cyIBwH5x0GuzUKhlFLoRoNFZP1KNGjSI/P59p06aRk5ND165dWbZsmXmAWWZmJmr1hW7Hc+fO8fjjj5OTk4Onpyc9evRg48aNdOzY0VpfQdg61wBof4dpAdPI88rSC9vPHgOjAYyV4OKPnVpNv3Af+qW8yAzXFM6GdGZrZRt+zvFnXXEgv+/J5vc92Ran0GrU2GtU2NupsdeoL6xrTOv2dmq0F69r1GjtVNipL3y22FZT1+4v6xcdy9dVR6S/K64O9k3YmEKIpmb156ibmjxHLWpVWW567Ms38kLZ7GgoyLCoZlTbk+sYziZ9KFvLWpGjeJKneJKreHIWVxSa/l52sIcjUQGupsXf9LOtrzM6O+miF8JWNZvnqK1BErW4ZufPwqldpq7yk9tN973Pn75sdUVtR/6ANzjd/u9UGoyoCjLxOPwLJS5hnAoeTqXBSIXBSGWVkUqjQqXBSKWh+meVsXp7TXn1etVf1g0KlVWm45w8V0ZOUe2vzrVTm946FxngSnt/V9PPAFdCPJ3kuXEhbECzGfUthE1z8jI9613zvLeimEaTn9xhStr56VCSA8W5UJqPyliFn68ffkHVz5eXboTdH0BQdzreOu7Ccef0gorzpi75ixevQHCpWQ80nf8q974LzldwMLeE9Jwi0nOLSc8p5kBOMcXlVRzKK+FQXgl/cKGb3tFeQ6S/C1EBrkT6u9I+wI3IABd8XXRyn10IGyWJWohrpVKBR2vT0ukey22GSijJA4eLXgLj6g/dHjLVr6EoUJBlmoms6MSVz6e2v5DEBz4PUcNN5aWn4VQKeITg4RtF7zZe9G7jddEpFHKKyknPKb6w5BZzKK+EskoDu08UsvtEocWpvJy1RPq7mBJ3dfd5VIArLjr5J0IIa5P/C4VoCBp7cA+2LKt54cpfPb3dNBK9OAeKs6Ek1/SzuPrqvDjb1MVurLzwTHjlRe9Hz9wMP4yBVr3hsaQL5fNugSo9KicvAh29CHTyItbJG1p7QXsvDA6eZFe6cLhYy94Ce1LzjRzMK+H4mVLOllaw+ehZNh89a/kVPBxpH3Ch6zzS35V2vi71fq5cURQMRoUq419/Gk0/DbWXG/5S38FeI69/FTcMSdRCNCWV6sIrVK+kqsL07vOahB7c/cI2tQb8OoF3uOU+ufsuP2c4oAFaVS+xYJov/M7ZlHf5G4fzSjh1aBe+e//N/spAPjo/lJyick4WlOFeuJ9j6VoWKC4U4oJarSHU2wkHe02tSdScdI0KBoNlubEBR8S083XmH4PbMbJrsLyQRrRoMphMiJZAUUwD38rOwvlzcP5M9eezf/l81vS55gr9/q+g832mz/t+gx8fMs1WNv5PCs5XkJ5TTOcfbsJZb5owxYiKIsWJc4oLZTigx55yRWv6iemnXrFnkXEAm4ymZ9UDOcOdmk3kKR78arzwfHsv1QHsVAb0ij16tFSpaxYHqlRaDGodRrU9Go0ajVqFnVpV/VPNqYIyiqtf/xro7sBjA9vyf71CcJauetFMyGAyIW40KpXlVffVVJaZkraD+4Uyn0i4+VXzTGUeTlr6tPUGNy8o0oO+EDUKHqpSPFSllzmwyaBBwynpPBg7tQqnE+vwW/wdVT4dmDZuBnZqNRqNCqcvpqM+c+jyBzECRhXgACodqB2h/7Nw05MUl1eyYNMRDqz/hRWF7Xjj93I+XnmIsX3DGNcvDE9n7bW3hRA2ThK1EDcie8dL76n7tTctfxVfPTmOodI0w5n5qrwMqsqrF331uh6qygkI7w9+polQqGoF0aOwcw3E2+XCe/fxamvqxr9oP/Nippi686vKoLzAvM3VwZ7HI0pgzTvoXT0Yaj+f42fL+DD5EN+s3cfdvSN4fGBbgjwcG6zJhLAWSdRCiGujsTddbdfMDX6tAjrDvV9cWj7mx9rrKwoYKv6SwPWmZO1y0ZS45YXgE4XOJ4LkB29mWVoOn646xOdnH6F8m5Y1WztgbN2PfkNG0KZtVN1iFsKGyD1qIUTzZqg0/REBKIUnUX1w6euE8+0CUbfpj3fHWyCsP3iEXvUZdSEak9yjFkLcODQX3nWucg+GF49B5mbyUpM5f3gdIeUH8a3KhkMLTQuguAWjCu1vmnUtbIBpBL0kbmGjJFELIVoWJy9ofzt+7U1T3x45cYpVf/6PqmPr6aXaT7TqKPZFJyH1R9MC8NzeC4/MlZ0DnTuo5ZEvYRskUQshWrR2rYJo9+g/OFXwMF+uO8ZjWw/RwXCAPuoDDNam08axDAfnQMzD3H75B5zYCnfNgQ53WjN0m1Cir+JIXgmH80o4nF9C1tnz2KlVONhrLlrUOF70+eJtjheVOdpr0F302V4jfwxdC0nUQogbQpCHI9NGdOTpW8L5z6YOzN94nA/OV6I6b8T3nVWMH9CGv/UOwTUn1XRVffGo+LRfYPf3pq7y0P4Q2BXsWs4jYIqicLqkwpyMaxLzkfwSsgtrn/ilIWjUKhzs1DhqNejsqhO+VoODneUfARcnfC9nHf3Dvekc5H7DvJlOBpMJIW5IpfoqFmzL4st1R83JyNXBjnF9ghnfrhCPdn1AU30t82s87PrvhZ3VdqbHy3wi/7KEWz6bbmOMRoUT58o4nF9sSsR5pRzONyXlwrLKy+7n46Ij3M+ZcD8XwrydASirMFBeZaC80khZpYHySgP6iz6XVxooqzSiN3821S2vMtAQWcfbWcugSF9io3wZGOGLVzN7dl6mubwCSdRCiItVVBn5NeUkn605wpF804tcdHZqHuwZwoRBbQnxcoK8/XBkFWRsMC1l5y5/QJcA8ImA9nfATU9eKFeUJhuwpq8ycOx0qSkRV18lH84r4Wh+CfoqY637qFQQ4ulEuJ8L7XxNSTncz4VwX1fcnexr3ac+FEVBX2VEX520LRJ+9Wf9xYn9os/6StP32njkDCXVb6ariT2mlQexUb4MjvQlupUHGhu/2pZEfQWSqIUQtTEaFZL25/Lp6iPszioATF2zI6IDeSK2He0D3GoqQvEp0zSnpw/B6YPVyyHTtKc1ej4Kd35g+lxRCrMiTVfhjy4HrZOpvDjXdAVu71CvmIvKKy3uH9d8zjx7/rLvVddq1LT1daadrwvtzMnYhba+zjjYa+oVR1OrqDKyI+Mcqw/msSY9nwM5xRbbPZ3szVfbgyJ8LV+0YyMkUV+BJGohxJUoisKmo2eYu/oI6w6dNpff0t6PJ2Pb0SvM6/I7lxfC6cOmxO3VBlrfZCrP3g2fDwInH3jxCJUGUxexbsEotMdXUukWQplbO0pd21Lk0oZzTmGc1oVSqHKjvMp0pVlWfWVZVmEg8+x5DueVkFesv2worjq7C4m4OhmH+7kQ4uVk81ebdZVTWM6ag3msTs9n/aHT5vfAg+lqu0uwO7GRvgyO8qNriG1cbUuivgJJ1EKIa5V2spC5a46wJDXbfF+1Z6gnd0YHUmU0deFenETL/5JQa7ptKysq8ao8hUvlWTZURlJVfbn7h3YqndQZlz3/OcWFI0oQR4xBHFaCOKIEkWpsQz6eAOioIMqljBAfN7wDw8wJOco+Fy+dEZViBMVo6gVQjKAYwGi48Pnibd7tTAtAWQEcSQaNznLke/pS0zSsxurjGKsuWi6zHtoPOo007X/+LCyZAqjg/n9fOO7KNyFz0xWOWXlhXaM1PfcecSv0+cclbVZpMLIz4xxrDuazOj2ffdlFFtvdHe0ZGOFDbJQfgyN98XW1ztW2JOorkEQthKirY6dL+WLtEX7ecZIKQ+33eOtDpVJoZV9Ce7scItTZtFOfIkw5SYjxBD6GPNRc+s/zxrB4TnZ+knA/FyJLt+P8w/3g3xme3HCh0kfd4eyRugVz86sw+AXT55xU+GyA6ZWtUw5eqPPv2yBrS92O2/sfcPu7ps/FOfB+FKg0MP2iuc8XjIEDv9ftuF3HwMhPTZ+rKuDDGNMfGv/3HThU36aoKCWvTM3qQ6dZczCfdQfzKSqvsjhM52A3YiP9GBzlS7cQD+ya6JExeTOZEEI0oDY+ziTcG82kuEj+s/E4h/JKcKx+ZMhRe+F5YUet2uL54Uu3XyjX2avR2alRXW6AWcV5U7Ktuf9dfS+8X99BEBViqnPMAewcLN7OBoCzD1SUgEptSooqtekFLhbrmurPKtPni9/hrnWBsIHg6Gl53NB+pu57tcY08l1jb/pZs25eLlpv1evC/jo3GPa2qfxifeOh872XOYa9ZVlFSfWthbYX9j971DRuQF8MOtcL5Yv+gd/RNTzoE8mDvlEYhkRwlGDWnPHit0w79pwqJe1kEWkni5iz6jBuDnYMjPBlcJQvsZG++LnVb+xAQ5MraiGEEM1blR5y06AkH6KGXSj/5CbI31/7PhodVV7tyLYPJVXvz+qznuwu9+eYEkgFpj98OgS6EVudtLuHejboC1qk6/sKJFELIcQNokoPZ47A6XTIP1j9s3q0vqH2gXibWz1KQvl97DlZiLtSzC3qXaQrIWRqIxgQ4cPgSF/ujAnCRXd9HdLS9S2EEELY6cC/o2m5mNEABRkXJe+DkH8ATh/kpj79+bXLAM6U6Dmw/hf6b/6MowRzS/l7LE3LYfneHIZ2CoAmHIMmiVoIIcSNRa0x3eP2amvZVa4ophHwgLeLjv6RQZAzkDCvdizu1p/V6XnkFpXj2cRvQbOJN6J/8sknhIWF4eDgQJ8+fdi6desV6//000+0b98eBwcHunTpwpIlS5ooUiGEEC1WzcC6Gm0Hw7jfUd/1IV1DPJgUF0nCvdFNHpbVE/UPP/zA5MmTmT59Ojt37iQmJoahQ4eSl5dXa/2NGzcyevRoxo8fz65duxg5ciQjR44kLS2tiSMXQgghGp/VB5P16dOHXr16MWfOHACMRiMhISE8/fTTvPzyy5fUHzVqFKWlpfz++4Vn7m666Sa6du3KZ599dtXzyWAyIYQQ1laXXGTVK+qKigp27NhBXFycuUytVhMXF8emTZtq3WfTpk0W9QGGDh162fpCCCFEc2bVwWSnT5/GYDDg7+9vUe7v78+BAwdq3ScnJ6fW+jk5ObXW1+v16PUXhuEXFxfXWk8IIYSwRVa/R93YEhIScHd3Ny8dO3a8+k5CCCGEjbBqovbx8UGj0ZCbm2tRnpubS0BAQK37BAQE1Kn+1KlTKSwsNC/79u1rmOCFEEKIJmDVrm+tVkuPHj1ITk5m5MiRgGkwWXJyMhMnTqx1n759+5KcnMykSZPMZUlJSfTt27fW+jqdDp3uwpPpBQUFAGRnZzfIdxBCCCHqqiYHGY3XMMmLYmULFixQdDqdkpiYqOzbt0+ZMGGC4uHhoeTk5CiKoigPPfSQ8vLLL5vrb9iwQbGzs1NmzZql7N+/X5k+fbpib2+vpKamXtP5tm7dqgCyyCKLLLLIYvVl69atV81bVn8z2ahRo8jPz2fatGnk5OTQtWtXli1bZh4wlpmZiVp9oYe+X79+fPfdd7z66qv885//JCIigsWLF9O5c+drOl+3bt3YunUr/v7+Fsetj+LiYjp27Mi+fftwdXW9+g43OGmvupM2qxtpr7qR9qqbhmwvo9FIbm4u3bp1u2pdqz9H3ZwVFRXh7u5OYWEhbm5u1g7H5kl71Z20Wd1Ie9WNtFfdWKu9WvyobyGEEKI5k0QthBBC2DBJ1NdBp9Mxffp0i1Hl4vKkvepO2qxupL3qRtqrbqzVXnKPWgghhLBhckUthBBC2DBJ1EIIIYQNk0QthBBC2DBJ1Nfhk08+ISwsDAcHB/r06cPWrVutHZLNWrt2LSNGjCAoKAiVSsXixYutHZLNSkhIoFevXri6uuLn58fIkSNJT0+3dlg2a+7cuURHR+Pm5oabmxt9+/Zl6dKl1g6r2Xj77bdRqVQWr2UWlmbMmIFKpbJY2rdv32Tnl0RdTz/88AOTJ09m+vTp7Ny5k5iYGIYOHUpeXp61Q7NJpaWlxMTE8Mknn1g7FJu3Zs0a4uPj2bx5M0lJSVRWVnLbbbdRWlpq7dBsUqtWrXj77bfZsWMH27dv55ZbbuHuu+9m79691g7N5m3bto3PP/+c6Ohoa4di8zp16kR2drZ5Wb9+fdOdvO5v5xaKoii9e/dW4uPjzesGg0EJCgpSEhISrBhV8wAoixYtsnYYzUZeXp4CKGvWrLF2KM2Gp6en8uWXX1o7DJtWXFysREREKElJScrgwYOVZ5991toh2azp06crMTExVju/XFHXQ0VFBTt27CAuLs5cplariYuLY9OmTVaMTLREhYWFAHh5eVk5EttnMBhYsGABpaWll51RT5jEx8dzxx13WPw7Ji7v0KFDBAUF0bZtW8aMGUNmZmaTndvqk3I0R6dPn8ZgMJgnDqnh7+/PgQMHrBSVaImMRiOTJk2if//+1zzxzI0oNTWVvn37Ul5ejouLC4sWLaJjx47WDstmLViwgJ07d7Jt2zZrh9Is9OnTh8TERKKiosjOzub1119n4MCBpKWlNclkJpKohbBh8fHxpKWlNe39sGYoKiqKlJQUCgsLWbhwIWPHjmXNmjWSrGuRlZXFs88+S1JSEg4ODtYOp1kYPny4+XN0dDR9+vQhNDSUH3/8kfHjxzf6+SVR14OPjw8ajYbc3FyL8tzcXAICAqwUlWhpJk6cyO+//87atWtp1aqVtcOxaVqtlvDwcAB69OjBtm3b+PDDD/n888+tHJnt2bFjB3l5eXTv3t1cZjAYWLt2LXPmzEGv16PRaKwYoe3z8PAgMjKSw4cPN8n55B51PWi1Wnr06EFycrK5zGg0kpycLPfFxHVTFIWJEyeyaNEiVq5cSZs2bawdUrNjNBrR6/XWDsMmDRkyhNTUVFJSUsxLz549GTNmDCkpKZKkr0FJSQlHjhwhMDCwSc4nV9T1NHnyZMaOHUvPnj3p3bs3s2fPprS0lEceecTaodmkkpISi78+jx07RkpKCl5eXrRu3dqKkdme+Ph4vvvuO3799VdcXV3JyckBwN3dHUdHRytHZ3umTp3K8OHDad26NcXFxXz33XesXr2a5cuXWzs0m+Tq6nrJeAdnZ2e8vb1lHMRlTJkyhREjRhAaGsqpU6eYPn06Go2G0aNHN8n5JVHX06hRo8jPz2fatGnk5OTQtWtXli1bdskAM2Gyfft2br75ZvP65MmTARg7diyJiYlWiso2zZ07F4DY2FiL8vnz5zNu3LimD8jG5eXl8fDDD5OdnY27uzvR0dEsX76cW2+91dqhiRbixIkTjB49mjNnzuDr68uAAQPYvHkzvr6+TXJ+mT1LCCGEsGFyj1oIIYSwYZKohRBCCBsmiVoIIYSwYZKohRBCCBsmiVoIIYSwYZKohRBCCBsmiVoIIYSwYZKohRBCCBsmiVoI0WhUKhWLFy+2dhhCNGuSqIVoocaNG4dKpbpkGTZsmLVDE0LUgbzrW4gWbNiwYcyfP9+iTKfTWSkaIUR9yBW1EC2YTqcjICDAYvH09ARM3dJz585l+PDhODo60rZtWxYuXGixf2pqKrfccguOjo54e3szYcIESkpKLOp89dVXdOrUCZ1OR2BgIBMnTrTYfvr0ae655x6cnJyIiIjgt99+M287d+4cY8aMwdfXF0dHRyIiIi75w0KIG50kaiFuYK+99hr33Xcfu3fvZsyYMfzf//0f+/fvB6C0tJShQ4fi6enJtm3b+Omnn1ixYoVFIp47dy7x8fFMmDCB1NRUfvvtN8LDwy3O8frrr/Pggw+yZ88ebr/9dsaMGcPZs2fN59+3bx9Lly5l//79zJ07Fx8fn6ZrACGaA0UI0SKNHTtW0Wg0irOzs8Xy5ptvKoqiKIDyxBNPWOzTp08f5cknn1QURVG++OILxdPTUykpKTFv/+OPPxS1Wq3k5OQoiqIoQUFByiuvvHLZGADl1VdfNa+XlJQogLJ06VJFURRlxIgRyiOPPNIwX1iIFkruUQvRgt18883m+a1reHl5mT/37dvXYlvfvn1JSUkBYP/+/cTExODs7Gze3r9/f4xGI+np6ahUKk6dOsWQIUOuGEN0dLT5s7OzM25ubuTl5QHw5JNPct9997Fz505uu+02Ro4cSb9+/er1XYVoqSRRC9GCOTs7X9IV3VAcHR2vqZ69vb3Fukqlwmg0AjB8+HAyMjJYsmQJSUlJDBkyhPj4eGbNmtXg8QrRXMk9aiFuYJs3b75kvUOHDgB06NCB3bt3U1paat6+YcMG1Go1UVFRuLq6EhYWRnJy8nXF4Ovry9ixY/nvf//L7Nmz+eKLL67reEK0NHJFLUQLptfrycnJsSizs7MzD9j66aef6NmzJwMGDODbb79l69at/Pvf/wZgzJgxTJ8+nbFjxzJjxgzy8/N5+umneeihh/D39wdgxowZPPHEE/j5+TF8+HCKi4vZsGEDTz/99DXFN23aNHr06EGnTp3Q6/X8/vvv5j8UhBAmkqiFaMGWLVtGYGCgRVlUVBQHDhwATCOyFyxYwFNPPUVgYCDff/89HTt2BMDJyYnly5fz7LPP0qtXL5ycnLjvvvv417/+ZT7W2LFjKS8v54MPPmDKlCn4+Phw//33X3N8Wq2WqVOncvz4cRwdHRk4cCALFixogG8uRMuhUhRFsXYQQoimp1KpWLRoESNHjrR2KEKIK5B71EIIIYQNk0QthBBC2DC5Ry3EDUruegnRPMgVtRBCCGHDJFELIYQQNkwStRBCCGHDJFELIYQQNkwStRBCCGHDJFELIYQQNkwStRBCCGHDJFELIYQQNkwStRBCCGHD/j8BzPcReWf3bQAAAABJRU5ErkJggg==",
"text/plain": [
"![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/overview-4.webp\")
"
]
},
{
"cell_type": "markdown",
"id": "fd5408e6-83e4-4e5a-8503-c2fba6073f31",
"metadata": {},
"source": [
"- Finally, let's use the finetuned GPT model in action\n",
"- The `classify_review` function below implements the data preprocessing steps similar to the `SpamDataset` we implemented earlier\n",
"- Then, the function returns the predicted integer class label from the model and returns the corresponding class name"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "aHdn6xvL-IW5",
"metadata": {
"id": "aHdn6xvL-IW5"
},
"outputs": [],
"source": [
"def classify_review(text, model, tokenizer, device, max_length=None, pad_token_id=50256):\n",
" model.eval()\n",
"\n",
" # Prepare inputs to the model\n",
" input_ids = tokenizer.encode(text)\n",
" supported_context_length = model.pos_emb.weight.shape[1]\n",
"\n",
" # Truncate sequences if they too long\n",
" input_ids = input_ids[:min(max_length, supported_context_length)]\n",
"\n",
" # Pad sequences to the longest sequence\n",
" input_ids += [pad_token_id] * (max_length - len(input_ids))\n",
" input_tensor = torch.tensor(input_ids, device=device).unsqueeze(0) # add batch dimension\n",
"\n",
" # Model inference\n",
" with torch.no_grad():\n",
" logits = model(input_tensor)[:, -1, :] # Logits of the last output token\n",
" predicted_label = torch.argmax(logits, dim=-1).item()\n",
"\n",
" # Return the classified result\n",
" return \"Positive\" if predicted_label == 1 else \"Negative\""
]
},
{
"cell_type": "markdown",
"id": "f29682d8-a899-4d9b-b973-f8d5ec68172c",
"metadata": {},
"source": [
"- Let's try it out on a few examples below"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "apU_pf51AWSV",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "apU_pf51AWSV",
"outputId": "d0fde0a5-e7a3-4dbe-d9c5-0567dbab7e62"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Positive\n"
]
}
],
"source": [
"text_1 = (\n",
" \"You are a winner you have been specially\"\n",
" \" selected to receive $1000 cash or a $2000 award.\"\n",
")\n",
"\n",
"print(classify_review(\n",
" text_1, model, tokenizer, device, max_length=train_dataset.max_length\n",
"))"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "1g5VTOo_Ajs5",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1g5VTOo_Ajs5",
"outputId": "659b08eb-b6a9-4a8a-9af7-d94c757e93c2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Negative\n"
]
}
],
"source": [
"text_2 = (\n",
" \"Hey, just wanted to check if we're still on\"\n",
" \" for dinner tonight? Let me know!\"\n",
")\n",
"\n",
"print(classify_review(\n",
" text_2, model, tokenizer, device, max_length=train_dataset.max_length\n",
"))"
]
},
{
"cell_type": "markdown",
"id": "bf736e39-0d47-40c1-8d18-1f716cf7a81e",
"metadata": {},
"source": [
"- Finally, let's save the model in case we want to reuse the model later without having to train it again"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "mYnX-gI1CfQY",
"metadata": {
"id": "mYnX-gI1CfQY"
},
"outputs": [],
"source": [
"torch.save(model.state_dict(), \"review_classifier.pth\")"
]
},
{
"cell_type": "markdown",
"id": "ba78cf7c-6b80-4f71-a50e-3ccc73839af6",
"metadata": {},
"source": [
"- Then, in a new session, we could load the model as follows"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "cc4e68a5-d492-493b-87ef-45c475f353f5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"