diff --git a/appendix-E/01_main-chapter-code/appendix-E.ipynb b/appendix-E/01_main-chapter-code/appendix-E.ipynb index d9da9ca..b9f02c9 100644 --- a/appendix-E/01_main-chapter-code/appendix-E.ipynb +++ b/appendix-E/01_main-chapter-code/appendix-E.ipynb @@ -16,7 +16,9 @@ { "cell_type": "markdown", "id": "58b8c870-fb72-490e-8916-d8129bd5d1ff", - "metadata": {}, + "metadata": { + "id": "58b8c870-fb72-490e-8916-d8129bd5d1ff" + }, "source": [ "# Appendix E: Parameter-efficient Finetuning with LoRA" ] @@ -30,7 +32,7 @@ "base_uri": "https://localhost:8080/" }, "id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90", - "outputId": "9495f150-9d79-4910-d6e7-6c0d9aae4a41" + "outputId": "316166b4-027a-4756-e9b4-fe88ae75dd4f" }, "outputs": [ { @@ -63,7 +65,9 @@ { "cell_type": "markdown", "id": "21532056-0ef4-4c98-82c7-e91f61c6485e", - "metadata": {}, + "metadata": { + "id": "21532056-0ef4-4c98-82c7-e91f61c6485e" + }, "source": [ "## E.1 Introduction to LoRA" ] @@ -71,7 +75,9 @@ { "cell_type": "markdown", "id": "66edc999-3d91-4a1c-a157-9d056392e8d8", - "metadata": {}, + "metadata": { + "id": "66edc999-3d91-4a1c-a157-9d056392e8d8" + }, "source": [ "- No code in this section\n", "- Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model to better suit a specific, often smaller, dataset by adjusting only a small, low-rank subset of the model's parameters\n", @@ -81,7 +87,9 @@ { "cell_type": "markdown", "id": "5bb75b5d-d59c-4948-821a-1594a5883dc1", - "metadata": {}, + "metadata": { + "id": "5bb75b5d-d59c-4948-821a-1594a5883dc1" + }, "source": [ "- Suppose we have a large weight matrix $W$ for a given layer\n", "- During backpropagation, we learn a $\\Delta W$ matrix, which contains information on how much we want to update the original weights to minimize the loss function during training\n", @@ -100,7 +108,9 @@ { "cell_type": "markdown", "id": "a8a7419d-cae9-4525-bb44-1641f6ef4f3b", - "metadata": {}, + "metadata": { + "id": "a8a7419d-cae9-4525-bb44-1641f6ef4f3b" + }, "source": [ "" ] @@ -108,7 +118,9 @@ { "cell_type": "markdown", "id": "4edd43c9-8ec5-48e6-b3fc-5fb3c16037cc", - "metadata": {}, + "metadata": { + "id": "4edd43c9-8ec5-48e6-b3fc-5fb3c16037cc" + }, "source": [ "- If you paid close attention, the full finetuning and LoRA depictions in the figure above look slightly different from the formulas I have shown earlier\n", "- That's due to the distributive law of matrix multiplication: we don't have to add the weights with the updated weights but can keep them separate\n", @@ -138,7 +150,9 @@ { "cell_type": "markdown", "id": "669c64df-4431-4d27-834d-2bb38a01fc02", - "metadata": {}, + "metadata": { + "id": "669c64df-4431-4d27-834d-2bb38a01fc02" + }, "source": [ "- This section repeats the code from chapter 6 to load and prepare the dataset\n", "- Instead of repeating this code, one could open and run the chapter 6 notebook and then insert the LoRA code from section E.4 there\n", @@ -155,14 +169,14 @@ "base_uri": "https://localhost:8080/" }, "id": "def7c09b-af9c-4216-90ce-5e67aed1065c", - "outputId": "424e4423-f623-443c-ab9e-656f9e867559" + "outputId": "a67a7afe-b401-4463-c731-87025d20f72d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "sms_spam_collection/SMSSpamCollection.tsv already exists. Skipping download and extraction.\n" + "File downloaded and saved as sms_spam_collection/SMSSpamCollection.tsv\n" ] } ], @@ -198,11 +212,7 @@ "execution_count": 3, "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7", "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7", - "outputId": "b5b48439-32c8-4b37-cca2-c9dc8fa86563" + "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7" }, "outputs": [], "source": [ @@ -223,11 +233,7 @@ "execution_count": 4, "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542", "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542", - "outputId": "3266c410-4fdb-4a8c-a142-7f707e2525ab" + "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542" }, "outputs": [], "source": [ @@ -264,7 +270,9 @@ { "cell_type": "markdown", "id": "ab7335db-e0bb-4e27-80c5-eea11e593a57", - "metadata": {}, + "metadata": { + "id": "ab7335db-e0bb-4e27-80c5-eea11e593a57" + }, "source": [ "- As a verification step, we iterate through the data loaders and check that the batches contain 8 training examples each, where each training example consists of 120 tokens" ] @@ -273,7 +281,13 @@ "cell_type": "code", "execution_count": 5, "id": "4dee6882-4c3a-4964-af15-fa31f86ad047", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "4dee6882-4c3a-4964-af15-fa31f86ad047", + "outputId": "2ae34de1-dd01-4f99-d2c8-ba4dca400754" + }, "outputs": [ { "name": "stdout", @@ -297,7 +311,9 @@ { "cell_type": "markdown", "id": "5cdd7947-7039-49bf-8a5e-c0a2f4281ca1", - "metadata": {}, + "metadata": { + "id": "5cdd7947-7039-49bf-8a5e-c0a2f4281ca1" + }, "source": [ "- Lastly, let's print the total number of batches in each dataset" ] @@ -311,7 +327,7 @@ "base_uri": "https://localhost:8080/" }, "id": "IZfw-TYD2zTj", - "outputId": "6934bbf2-9797-4fbe-d26b-1a246e18c2fb" + "outputId": "4d19ed61-cf7a-4ec4-b822-c847dd1c5d77" }, "outputs": [ { @@ -333,7 +349,9 @@ { "cell_type": "markdown", "id": "dec9aa4a-ffd2-4d9f-a835-cce1059fe604", - "metadata": {}, + "metadata": { + "id": "dec9aa4a-ffd2-4d9f-a835-cce1059fe604" + }, "source": [ "## E.3 Initializing the model" ] @@ -341,7 +359,9 @@ { "cell_type": "markdown", "id": "f36ebdaf-810e-46a2-9ad9-e017a04051b1", - "metadata": {}, + "metadata": { + "id": "f36ebdaf-810e-46a2-9ad9-e017a04051b1" + }, "source": [ "- This section repeats the code from chapter 6 to load and prepare the model" ] @@ -350,19 +370,25 @@ "cell_type": "code", "execution_count": 7, "id": "02b3a506-3879-4258-82b5-93a5b6bafa74", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "02b3a506-3879-4258-82b5-93a5b6bafa74", + "outputId": "b8c9b125-bb52-45d3-8071-fa5054dbf5a9" + }, "outputs": [ { - "name": "stdout", + "name": "stderr", "output_type": "stream", "text": [ - "File already exists and is up-to-date: gpt2/124M/checkpoint\n", - "File already exists and is up-to-date: gpt2/124M/encoder.json\n", - "File already exists and is up-to-date: gpt2/124M/hparams.json\n", - "File already exists and is up-to-date: gpt2/124M/model.ckpt.data-00000-of-00001\n", - "File already exists and is up-to-date: gpt2/124M/model.ckpt.index\n", - "File already exists and is up-to-date: gpt2/124M/model.ckpt.meta\n", - "File already exists and is up-to-date: gpt2/124M/vocab.bpe\n" + "checkpoint: 100%|███████████████████████████| 77.0/77.0 [00:00<00:00, 45.0kiB/s]\n", + "encoder.json: 100%|███████████████████████| 1.04M/1.04M [00:00<00:00, 2.15MiB/s]\n", + "hparams.json: 100%|█████████████████████████| 90.0/90.0 [00:00<00:00, 54.5kiB/s]\n", + "model.ckpt.data-00000-of-00001: 100%|███████| 498M/498M [01:12<00:00, 6.86MiB/s]\n", + "model.ckpt.index: 100%|███████████████████| 5.21k/5.21k [00:00<00:00, 2.99MiB/s]\n", + "model.ckpt.meta: 100%|██████████████████████| 471k/471k [00:00<00:00, 1.32MiB/s]\n", + "vocab.bpe: 100%|████████████████████████████| 456k/456k [00:00<00:00, 1.48MiB/s]\n" ] } ], @@ -401,7 +427,9 @@ { "cell_type": "markdown", "id": "252614cd-7ce6-4908-83e6-3761f519904e", - "metadata": {}, + "metadata": { + "id": "252614cd-7ce6-4908-83e6-3761f519904e" + }, "source": [ "- To ensure that the model was loaded corrected, let's double-check that it generates coherent text" ] @@ -410,7 +438,13 @@ "cell_type": "code", "execution_count": 8, "id": "8b6ce20c-0700-4783-8be0-4cf17c200a7f", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "8b6ce20c-0700-4783-8be0-4cf17c200a7f", + "outputId": "28ccbca5-8de9-41a0-c093-da00fcbaa91c" + }, "outputs": [ { "name": "stdout", @@ -445,7 +479,9 @@ { "cell_type": "markdown", "id": "8174b31b-1ab5-4115-b01c-245369da5af3", - "metadata": {}, + "metadata": { + "id": "8174b31b-1ab5-4115-b01c-245369da5af3" + }, "source": [ "- Then, we prepare the model for classification finetuning similar to chapter 6, where we replace the output layer" ] @@ -454,7 +490,9 @@ "cell_type": "code", "execution_count": 9, "id": "e255ce91-d73a-4854-90a4-95804928eb16", - "metadata": {}, + "metadata": { + "id": "e255ce91-d73a-4854-90a4-95804928eb16" + }, "outputs": [], "source": [ "torch.manual_seed(123)\n", @@ -467,7 +505,9 @@ "cell_type": "code", "execution_count": 10, "id": "02e6f057-1383-4ece-8444-0a88e71ac75d", - "metadata": {}, + "metadata": { + "id": "02e6f057-1383-4ece-8444-0a88e71ac75d" + }, "outputs": [], "source": [ "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", @@ -477,7 +517,9 @@ { "cell_type": "markdown", "id": "8e951cd6-5e42-44d2-b21f-895cb61004fe", - "metadata": {}, + "metadata": { + "id": "8e951cd6-5e42-44d2-b21f-895cb61004fe" + }, "source": [ "- Lastly, let's calculate the initial classification accuracy of the non-finetuned model (we expect this to be around 50%, which means that the model is not able to distinguish between spam and non-spam messages yet reliably)" ] @@ -486,7 +528,13 @@ "cell_type": "code", "execution_count": 11, "id": "fc7dd72c-73a2-4881-ade0-0a9605f1ab8c", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "fc7dd72c-73a2-4881-ade0-0a9605f1ab8c", + "outputId": "74848515-5a49-4125-fecb-9f4bac23f812" + }, "outputs": [ { "name": "stdout", @@ -525,7 +573,9 @@ { "cell_type": "markdown", "id": "652a4a82-61ef-4d0a-9858-8988e844f12c", - "metadata": {}, + "metadata": { + "id": "652a4a82-61ef-4d0a-9858-8988e844f12c" + }, "source": [ "- We begin by initializing a LoRALayer that creates the matrices $A$ and $B$, along with the `alpha` scaling hyperparameter and the `rank` ($r$) hyperparameters\n", "- This layer can accept an input and compute the corresponding output, as illustrated in the figure below\n", @@ -544,11 +594,13 @@ }, "outputs": [], "source": [ + "import math\n", + "\n", "class LoRALayer(torch.nn.Module):\n", " def __init__(self, in_dim, out_dim, rank, alpha):\n", " super().__init__()\n", - " std_dev = 1 / torch.sqrt(torch.tensor(rank).float())\n", - " self.A = torch.nn.Parameter(torch.randn(in_dim, rank) * std_dev)\n", + " self.A = torch.nn.Parameter(torch.empty(in_dim, rank))\n", + " torch.nn.init.kaiming_uniform_(self.A, a=math.sqrt(5)) # similar to standard weight initialization\n", " self.B = torch.nn.Parameter(torch.zeros(rank, out_dim))\n", " self.alpha = alpha\n", "\n", @@ -560,11 +612,13 @@ { "cell_type": "markdown", "id": "ad21faa8-0614-4257-93cd-68952193e14a", - "metadata": {}, + "metadata": { + "id": "ad21faa8-0614-4257-93cd-68952193e14a" + }, "source": [ "- In the code above, `rank` is a hyperparameter that controls the inner dimension of the matrices $A$ and $B$\n", "- In other words, this parameter controls the number of additional parameters introduced by LoRA and is a key factor in determining the balance between model adaptability and parameter efficiency\n", - "- The second hyperparameter, alpha, is a scaling hyperparameter applied to the output of the low-rank adaptation\n", + "- The second hyperparameter, `alpha`, is a scaling hyperparameter applied to the output of the low-rank adaptation\n", "- It essentially controls the extent to which the adapted layer's output is allowed to influence the original output of the layer being adapted\n", "- This can be seen as a way to regulate the impact of the low-rank adaptation on the layer's output\n", "- So far, the `LoRALayer` class we implemented above allows us to transform the layer inputs $x$\n", @@ -576,7 +630,9 @@ { "cell_type": "markdown", "id": "3e6d5da0-dfce-4808-b89b-29ff333f563f", - "metadata": {}, + "metadata": { + "id": "3e6d5da0-dfce-4808-b89b-29ff333f563f" + }, "source": [ "- To incorporate the original `Linear` layer weights as shown in the figure above, we implement a `LinearWithLoRA` layer below that uses the previously implemented LoRALayer and can be used to replace existing `Linear` layers in a neural network, for example, the self-attention module or feed forward modules in an LLM" ] @@ -585,7 +641,9 @@ "cell_type": "code", "execution_count": 13, "id": "127d3a64-8359-4b21-b056-78d58cc75fe8", - "metadata": {}, + "metadata": { + "id": "127d3a64-8359-4b21-b056-78d58cc75fe8" + }, "outputs": [], "source": [ "class LinearWithLoRA(torch.nn.Module):\n", @@ -603,7 +661,9 @@ { "cell_type": "markdown", "id": "e1145a90-35ff-462c-820b-15483fa5b051", - "metadata": {}, + "metadata": { + "id": "e1145a90-35ff-462c-820b-15483fa5b051" + }, "source": [ "- Note that since we initialize the weight matrix $B$ (`self.B` in `LoRALayer`) with zero values in the LoRA layer, the matrix multiplication between $A$ and $B$ results in a matrix consisting of 0's and doesn't affect the original weights (since adding 0 to the original weights does not modify them)" ] @@ -612,7 +672,7 @@ "cell_type": "markdown", "id": "e98a6d36-7bc9-434c-a7f1-533f26aff06d", "metadata": { - "id": "4D21Jk7Vw3nG" + "id": "e98a6d36-7bc9-434c-a7f1-533f26aff06d" }, "source": [ "- To try LoRA on the GPT model we defined earlier, we define a `replace_linear_with_lora` function to replace all `Linear` layers in the model with the new `LinearWithLoRA` layers\n", @@ -642,7 +702,9 @@ { "cell_type": "markdown", "id": "8c172164-cdde-4489-b7d7-aaed9cc2f5f2", - "metadata": {}, + "metadata": { + "id": "8c172164-cdde-4489-b7d7-aaed9cc2f5f2" + }, "source": [ "- We then freeze the original model parameter and use the `replace_linear_with_lora` to replace the said `Linear` layers using the code below\n", "- This will replace the `Linear` layers in the LLM with `LinearWithLoRA` layers" @@ -652,7 +714,13 @@ "cell_type": "code", "execution_count": 15, "id": "dbe15350-4da9-4829-9d23-98bbd3d0b1a1", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "dbe15350-4da9-4829-9d23-98bbd3d0b1a1", + "outputId": "fd4c208f-854a-4701-d9d3-9d73af733364" + }, "outputs": [ { "name": "stdout", @@ -683,19 +751,19 @@ "base_uri": "https://localhost:8080/" }, "id": "mLk_fPq0yz_u", - "outputId": "7ba89607-ca75-4718-e8dc-9cdc44c3e410" + "outputId": "0a93b8fc-05d7-4ace-ee47-e2fc6bdd7d75" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Total trainable LoRA parameters: 1,333,264\n" + "Total trainable LoRA parameters: 2,666,528\n" ] } ], "source": [ - "replace_linear_with_lora(model, rank=8, alpha=8)\n", + "replace_linear_with_lora(model, rank=16, alpha=16)\n", "\n", "total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n", "print(f\"Total trainable LoRA parameters: {total_params:,}\")" @@ -704,17 +772,25 @@ { "cell_type": "markdown", "id": "b8b6819e-ef7a-4f0d-841a-1b467496bef9", - "metadata": {}, + "metadata": { + "id": "b8b6819e-ef7a-4f0d-841a-1b467496bef9" + }, "source": [ - "- As we can see, we reduced the number of trainable parameters by almost 100x when using LoRA\n", + "- As we can see, we reduced the number of trainable parameters by almost 50x when using LoRA\n", "- Let's now double-check whether the layers have been modified as intended by printing the model architecture" ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 17, "id": "1711be61-bb2c-466f-9b5b-24f4aa5ccd9c", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "1711be61-bb2c-466f-9b5b-24f4aa5ccd9c", + "outputId": "acff8eca-3775-45a2-b62d-032a986ef037" + }, "outputs": [ { "name": "stdout", @@ -1189,7 +1265,9 @@ { "cell_type": "markdown", "id": "c4bbc9d7-65ec-4675-bab8-2e56eb0cfb55", - "metadata": {}, + "metadata": { + "id": "c4bbc9d7-65ec-4675-bab8-2e56eb0cfb55" + }, "source": [ "- Based on the model architecture above, we can see that the model now contains our new `LinearWithLoRA` layers\n", "- Also, since we initialized matrix $B$ with 0's, we expect the initial model performance to be unchanged compared to before" @@ -1197,14 +1275,14 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 18, "id": "DAlrb_I00VEU", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "DAlrb_I00VEU", - "outputId": "3dae5ff0-316d-408e-c8dc-2b8c60f9b994" + "outputId": "3da44ac4-230b-4358-d996-30b63f0d962a" }, "outputs": [ { @@ -1231,7 +1309,9 @@ { "cell_type": "markdown", "id": "13735b3e-f0c3-4dba-ae3d-4141b2878101", - "metadata": {}, + "metadata": { + "id": "13735b3e-f0c3-4dba-ae3d-4141b2878101" + }, "source": [ "- Let's now get to the interesting part and finetune the model by reusing the training function from chapter 6\n", "- The training takes about 15 minutes on a M3 MacBook Air laptop computer and less than half a minute on a V100 or A100 GPU" @@ -1239,43 +1319,39 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 19, "id": "wCParRvr0eff", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wCParRvr0eff", - "outputId": "b86fd5f4-1527-4549-e0b0-9dff37836f0a" + "outputId": "ce910a9c-ee89-48bb-bfa6-49c6aee1e450" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Ep 1 (Step 000000): Train loss 2.849, Val loss 2.565\n", - "Ep 1 (Step 000050): Train loss 0.515, Val loss 0.465\n", - "Ep 1 (Step 000100): Train loss 0.191, Val loss 0.423\n", + "Ep 1 (Step 000000): Train loss 3.820, Val loss 3.462\n", + "Ep 1 (Step 000050): Train loss 0.396, Val loss 0.364\n", + "Ep 1 (Step 000100): Train loss 0.111, Val loss 0.229\n", + "Training accuracy: 97.50% | Validation accuracy: 95.00%\n", + "Ep 2 (Step 000150): Train loss 0.135, Val loss 0.073\n", + "Ep 2 (Step 000200): Train loss 0.008, Val loss 0.052\n", + "Ep 2 (Step 000250): Train loss 0.021, Val loss 0.179\n", "Training accuracy: 97.50% | Validation accuracy: 97.50%\n", - "Ep 2 (Step 000150): Train loss 0.170, Val loss 0.072\n", - "Ep 2 (Step 000200): Train loss 0.014, Val loss 0.087\n", - "Ep 2 (Step 000250): Train loss 0.027, Val loss 0.197\n", - "Training accuracy: 100.00% | Validation accuracy: 92.50%\n", - "Ep 3 (Step 000300): Train loss 0.014, Val loss 0.321\n", - "Ep 3 (Step 000350): Train loss 0.015, Val loss 0.146\n", + "Ep 3 (Step 000300): Train loss 0.096, Val loss 0.080\n", + "Ep 3 (Step 000350): Train loss 0.010, Val loss 0.116\n", + "Training accuracy: 97.50% | Validation accuracy: 95.00%\n", + "Ep 4 (Step 000400): Train loss 0.003, Val loss 0.151\n", + "Ep 4 (Step 000450): Train loss 0.008, Val loss 0.077\n", + "Ep 4 (Step 000500): Train loss 0.001, Val loss 0.147\n", "Training accuracy: 100.00% | Validation accuracy: 97.50%\n", - "Ep 4 (Step 000400): Train loss 0.008, Val loss 0.103\n", - "Ep 4 (Step 000450): Train loss 0.010, Val loss 0.178\n", - "Ep 4 (Step 000500): Train loss 0.097, Val loss 0.056\n", + "Ep 5 (Step 000550): Train loss 0.007, Val loss 0.094\n", + "Ep 5 (Step 000600): Train loss 0.000, Val loss 0.056\n", "Training accuracy: 100.00% | Validation accuracy: 97.50%\n", - "Ep 5 (Step 000550): Train loss 0.032, Val loss 0.091\n", - "Ep 5 (Step 000600): Train loss 0.002, Val loss 0.058\n", - "Training accuracy: 100.00% | Validation accuracy: 100.00%\n", - "Ep 6 (Step 000650): Train loss 0.001, Val loss 0.009\n", - "Ep 6 (Step 000700): Train loss 0.001, Val loss 0.039\n", - "Ep 6 (Step 000750): Train loss 0.000, Val loss 0.038\n", - "Training accuracy: 100.00% | Validation accuracy: 95.00%\n", - "Training completed in 13.70 minutes.\n" + "Training completed in 12.10 minutes.\n" ] } ], @@ -1290,7 +1366,7 @@ "\n", "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)\n", "\n", - "num_epochs = 6\n", + "num_epochs = 5\n", "train_losses, val_losses, train_accs, val_accs, examples_seen = train_classifier_simple(\n", " model, train_loader, val_loader, optimizer, device,\n", " num_epochs=num_epochs, eval_freq=50, eval_iter=5,\n", @@ -1305,27 +1381,29 @@ { "cell_type": "markdown", "id": "d0c89e82-3aa8-44c6-b046-0b16200b8e6c", - "metadata": {}, + "metadata": { + "id": "d0c89e82-3aa8-44c6-b046-0b16200b8e6c" + }, "source": [ "- Finally, let's evaluate the model" ] }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 20, "id": "bawWGijA0iF3", "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 307 + "height": 308 }, "id": "bawWGijA0iF3", - "outputId": "4b05b245-ffac-4d36-881b-8306a4da6b75" + "outputId": "af70782a-d605-4376-fa6c-d33b38979cfa" }, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdwAAAEiCAYAAABTO2OcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABPd0lEQVR4nO3dd3hUVfrA8e9Mkpn03ikBJECAEEJdRBAlUlRWsMCyqEFRFw0iIoqsCog/DXYsLAoq6FpiAxcVqVIU6b2E0AKhpICQSurM+f1xk0mGACaQzKS8n+e5T+aWufc9Icw759xzz9EppRRCCCGEqFV6ewcghBBCNAaScIUQQggbkIQrhBBC2IAkXCGEEMIGJOEKIYQQNiAJVwghhLABSbhCCCGEDUjCFUIIIWxAEq4QQghhA5JwhWiE+vXrx4QJE+wdhhCNiiRcIa7C6NGj0el0lZZBgwbZOzQhRB3laO8AhKivBg0axPz58622GY1GO0UjhKjrpIYrxFUyGo0EBwdbLT4+PgCsWbMGg8HAb7/9Zjn+tddeIzAwkPT0dACWLl3KDTfcgLe3N35+ftx+++0cOXLEcvyxY8fQ6XR888039OnTBxcXF7p3787BgwfZsmUL3bp1w93dncGDB3PmzBnL+0aPHs3QoUN58cUXCQgIwNPTk7Fjx1JUVHTZshQWFjJp0iSaNGmCm5sbPXv2ZM2aNZb9x48fZ8iQIfj4+ODm5kaHDh1YsmTJZc/3n//8h/DwcJydnQkKCuLuu++27DObzcTHx9OyZUtcXFyIioriu+++s3r/3r17GTx4MO7u7gQFBXHfffdx9uxZy/5+/foxfvx4nnnmGXx9fQkODmb69OmXjUeIukASrhC1oOwe6X333UdWVhY7duzghRde4KOPPiIoKAiAvLw8Jk6cyNatW1m1ahV6vZ5hw4ZhNputzjVt2jSef/55tm/fjqOjI//85z955plneOedd/jtt984fPgwU6dOtXrPqlWrSExMZM2aNXz11VcsXLiQF1988bLxjhs3jg0bNpCQkMDu3bu55557GDRoEIcOHQIgLi6OwsJC1q1bx549e3j11Vdxd3e/5Lm2bt3K+PHjmTFjBklJSSxdupS+ffta9sfHx/PZZ5/xwQcfsG/fPp588knuvfde1q5dC0BmZiY333wz0dHRbN26laVLl5Kens7w4cOtrvPpp5/i5ubGpk2beO2115gxYwYrVqyo4r+QEHaghBDVFhsbqxwcHJSbm5vV8vLLL1uOKSwsVJ07d1bDhw9X7du3Vw8//PAVz3nmzBkFqD179iillEpOTlaA+uijjyzHfPXVVwpQq1atsmyLj49Xbdu2tYrN19dX5eXlWbbNmTNHubu7K5PJpJRS6sYbb1RPPPGEUkqp48ePKwcHB3Xq1CmrePr376+mTJmilFIqMjJSTZ8+vUq/m++//155enqq7OzsSvsKCgqUq6ur+uOPP6y2jxkzRo0cOVIppdRLL72kBgwYYLX/xIkTClBJSUmW+G+44QarY7p3764mT55cpRiFsAe5hyvEVbrpppuYM2eO1TZfX1/La4PBwBdffEGnTp0ICwvj7bfftjr20KFDTJ06lU2bNnH27FlLzTYlJYWOHTtajuvUqZPldVntODIy0mpbRkaG1bmjoqJwdXW1rPfq1Yvc3FxOnDhBWFiY1bF79uzBZDLRpk0bq+2FhYX4+fkBMH78eB599FGWL19OTEwMd911l1VcFd1yyy2EhYXRqlUrBg0axKBBgxg2bBiurq4cPnyYCxcucMstt1i9p6ioiOjoaAB27drF6tWrL1mDPnLkiCXOi68fEhJS6fcgRF0iCVeIq+Tm5kbr1q2veMwff/wBwLlz5zh37hxubm6WfUOGDCEsLIx58+YRGhqK2WymY8eOle61Ojk5WV7rdLpLbru4Gbo6cnNzcXBwYNu2bTg4OFjtK0t6Dz30EAMHDuTnn39m+fLlxMfH8+abb/L4449XOp+Hhwfbt29nzZo1LF++nKlTpzJ9+nS2bNlCbm4uAD///DNNmjSxel9Zh7Pc3FyGDBnCq6++WuncISEhltcVfwdw7b8HIWqbJFwhasmRI0d48sknmTdvHl9//TWxsbGsXLkSvV7Pn3/+SVJSEvPmzaNPnz4A/P777zV27V27dpGfn4+LiwsAGzduxN3dnWbNmlU6Njo6GpPJREZGhiWWS2nWrBljx45l7NixTJkyhXnz5l0y4QI4OjoSExNDTEwM06ZNw9vbm19//ZVbbrkFo9FISkoKN9544yXf26VLF77//ntatGiBo6N8RImGQ/6ahbhKhYWFpKWlWW1zdHTE398fk8nEvffey8CBA3nggQcYNGgQkZGRvPnmmzz99NP4+Pjg5+fH3LlzCQkJISUlhWeffbbGYisqKmLMmDE8//zzHDt2jGnTpjFu3Dj0+sr9JNu0acOoUaO4//77efPNN4mOjubMmTOsWrWKTp06cdtttzFhwgQGDx5MmzZtOH/+PKtXryYiIuKS1/7pp584evQoffv2xcfHhyVLlmA2m2nbti0eHh5MmjSJJ598ErPZzA033EBWVhbr16/H09OT2NhY4uLimDdvHiNHjrT0Qj58+DAJCQl89NFHlWrhQtQXknCFuEpLly61auIEaNu2LQcOHODll1/m+PHj/PTTT4DWFDp37lxGjhzJgAEDiIqKIiEhgfHjx9OxY0fatm3Lu+++S79+/Woktv79+xMeHk7fvn0pLCxk5MiRV3xsZv78+fzf//0fTz31FKdOncLf35+//e1v3H777QCYTCbi4uI4efIknp6eDBo0qNI96TLe3t4sXLiQ6dOnU1BQQHh4OF999RUdOnQA4KWXXiIgIID4+HiOHj2Kt7c3Xbp04d///jcAoaGhrF+/nsmTJzNgwAAKCwsJCwtj0KBBl/zCIER9oVNKKXsHIYSoOaNHjyYzM5MffvjB3qEIISqQr4tCCCGEDUjCFUIIIWxAmpSFEEIIG5AarhBCCGEDknCFEEIIG5CEK4QQQtiAJNxSs2fPpkWLFjg7O9OzZ082b95s75CqZN26dQwZMoTQ0FB0Ol2lR0GUUkydOpWQkBBcXFyIiYmxzABT5ty5c4waNQpPT0+8vb0ZM2aMZQi+Mrt376ZPnz44OzvTrFkzXnvttdou2mXFx8fTvXt3PDw8CAwMZOjQoSQlJVkdU1BQQFxcHH5+fri7u3PXXXdZpsUrk5KSwm233YarqyuBgYE8/fTTlJSUWB2zZs0aunTpgtFopHXr1ixYsKC2i3dJc+bMoVOnTnh6euLp6UmvXr345ZdfLPsbWnkvNnPmTHQ6HRMmTLBsa4hlnj59Ojqdzmpp166dZX9DLDPAqVOnuPfee/Hz88PFxYXIyEi2bt1q2d9gPsfsOXNCXZGQkKAMBoP65JNP1L59+9TDDz+svL29VXp6ur1D+0tLlixRzz33nFq4cKEC1KJFi6z2z5w5U3l5eakffvhB7dq1S/39739XLVu2VPn5+ZZjBg0apKKiotTGjRvVb7/9plq3bm2ZuUUppbKyslRQUJAaNWqU2rt3r/rqq6+Ui4uL+vDDD21VTCsDBw5U8+fPV3v37lU7d+5Ut956q2revLnKzc21HDN27FjVrFkztWrVKrV161b1t7/9TV1//fWW/SUlJapjx44qJiZG7dixQy1ZskT5+/tbZsdRSqmjR48qV1dXNXHiRLV//3713nvvKQcHB7V06VKbllcppRYvXqx+/vlndfDgQZWUlKT+/e9/KycnJ7V3794GWd6KNm/erFq0aKE6depkmeFIqYZZ5mnTpqkOHTqo1NRUy3LmzBnL/oZY5nPnzqmwsDA1evRotWnTJnX06FG1bNkydfjwYcsxDeVzTBKuUqpHjx4qLi7Osm4ymVRoaKiKj4+3Y1TVd3HCNZvNKjg4WL3++uuWbZmZmcpoNKqvvvpKKaXU/v37FaC2bNliOeaXX35ROp3OMl3bf/7zH+Xj46MKCwstx0yePNlqSjh7ysjIUIBau3atUkoro5OTk/r2228txyQmJipAbdiwQSmlfVHR6/UqLS3NcsycOXOUp6enpZzPPPOM6tChg9W1RowYoQYOHFjbRaoSHx8f9dFHHzXo8ubk5Kjw8HC1YsUKqykFG2qZp02bpqKioi65r6GWefLkyZWmWqyoIX2ONfom5aKiIrZt20ZMTIxlm16vJyYmhg0bNtgxsmuXnJxMWlqaVdm8vLzo2bOnpWwbNmzA29ubbt26WY6JiYlBr9ezadMmyzF9+/bFYDBYjhk4cCBJSUmcP3/eRqW5vKysLKB8arxt27ZRXFxsVe527drRvHlzq3JHRkZaprsDrUzZ2dns27fPckzFc5QdY++/C5PJREJCAnl5efTq1atBlzcuLo7bbrutUlwNucyHDh0iNDSUVq1aMWrUKFJSUoCGW+bFixfTrVs37rnnHgIDA4mOjmbevHmW/Q3pc6zRJ9yzZ89iMpms/kBBm2P04oHp65uy+K9UtrS0NAIDA632Ozo64uvra3XMpc5R8Rr2YjabmTBhAr1797bMIZuWlobBYMDb29vq2IvL/Vdlutwx2dnZ5Ofn10ZxrmjPnj24u7tjNBoZO3YsixYton379g22vAkJCWzfvp34+PhK+xpqmXv27MmCBQtYunQpc+bMITk5mT59+pCTk9Ngy3z06FHmzJlDeHg4y5Yt49FHH2X8+PF8+umnVnE3hM8xmbxA1GtxcXHs3bu3Rqe2q6vatm3Lzp07ycrK4rvvviM2Npa1a9faO6xaceLECZ544glWrFiBs7OzvcOxmcGDB1ted+rUiZ49exIWFsY333xjmWqxoTGbzXTr1o1XXnkF0KaL3Lt3Lx988AGxsbF2jq5mNfoarr+/Pw4ODpV6+qWnpxMcHGynqGpGWfxXKltwcDAZGRlW+0tKSjh37pzVMZc6R8Vr2MO4ceP46aefWL16NU2bNrVsDw4OpqioiMzMTKvjLy73X5Xpcsd4enra5cPPYDDQunVrunbtSnx8PFFRUbzzzjsNsrzbtm0jIyODLl264OjoiKOjI2vXruXdd9/F0dGRoKCgBlfmS/H29qZNmzYcPny4Qf47gzaTVvv27a22RUREWJrSG9LnWKNPuAaDga5du7Jq1SrLNrPZzKpVq+jVq5cdI7t2LVu2JDg42Kps2dnZbNq0yVK2Xr16kZmZybZt2yzH/Prrr5jNZnr27Gk5Zt26dRQXF1uOWbFiBW3btsXHx8dGpSmnlGLcuHEsWrSIX3/9lZYtW1rt79q1K05OTlblTkpKIiUlxarce/bssfpPumLFCjw9PS3/+Xv16mV1jrJj6srfhdlsprCwsEGWt3///uzZs4edO3dalm7dujFq1CjL64ZW5kvJzc3lyJEjhISENMh/Z4DevXtXeqzv4MGDhIWFAQ3sc8xm3bPqsISEBGU0GtWCBQvU/v371SOPPKK8vb2tevrVVTk5OWrHjh1qx44dClBvvfWW2rFjhzp+/LhSSutO7+3trf73v/+p3bt3qzvuuOOS3emjo6PVpk2b1O+//67Cw8OtutNnZmaqoKAgdd9996m9e/eqhIQE5erqarfHgh599FHl5eWl1qxZY/X4xIULFyzHjB07VjVv3lz9+uuvauvWrapXr16qV69elv1lj08MGDBA7dy5Uy1dulQFBARc8vGJp59+WiUmJqrZs2fb7fGJZ599Vq1du1YlJyer3bt3q2effVbpdDq1fPnyBlneS6nYS1mphlnmp556Sq1Zs0YlJyer9evXq5iYGOXv768yMjKUUg2zzJs3b1aOjo7q5ZdfVocOHVJffPGFcnV1VZ9//rnlmIbyOSYJt9R7772nmjdvrgwGg+rRo4fauHGjvUOqktWrVyug0hIbG6uU0rrUv/DCCyooKEgZjUbVv39/lZSUZHWOP//8U40cOVK5u7srT09P9cADD6icnByrY3bt2qVuuOEGZTQaVZMmTdTMmTNtVcRKLlVeQM2fP99yTH5+vnrssceUj4+PcnV1VcOGDVOpqalW5zl27JgaPHiwcnFxUf7+/uqpp55SxcXFVsesXr1ade7cWRkMBtWqVSura9jSgw8+qMLCwpTBYFABAQGqf//+lmSrVMMr76VcnHAbYplHjBihQkJClMFgUE2aNFEjRoyweh61IZZZKaV+/PFH1bFjR2U0GlW7du3U3LlzrfY3lM8xmS1ICCGEsIFGfw9XCCGEsAVJuEIIIYQNSMIVQgghbEASrhBCCGEDknCFEEIIG5CEK4QQQtiAJNxShYWFTJ8+ncLCQnuHYjNS5sZBytw4SJnrPnkOt1R2djZeXl5kZWXh6elp73BsQsosZW6opMxS5rpIarhCCCGEDUjCFUIIIWygXs+HW1JSwo4dOwgKCkKvv7bvDjk5OQCcOnWK7OzsmgivzpMyS5kbKimzlNmWzGYz6enpREdH4+h4+bRar+/hbtmyhR49etg7DCGEEILNmzfTvXv3y+6v1zXcoKAgQCtkSEiInaMRQgjRGKWmptKjRw9LTrqcep1wy5qRQ0JCaNq0qZ2jEUII0Zj91a1N6TQlhBBC2IAkXCGEEMIGJOEKIYQQNlCv7+EKIcSVmEwmiouL7R2GqOecnJxwcHC45vNIwgWUUuxPzeZAag63dQrB2enaf7FCCPtRSpGWlkZmZqa9QxENhLe3N8HBweh0uqs+hyTcUvd+tInzF4ppE+RBZFMve4cjhLgGZck2MDAQV1fXa/qQFI2bUooLFy6QkZEBcE2PoErCBXQ6He2CPdlw9E8SU7Ml4QpRj5lMJkuy9fPzs3c4ogFwcXEBICMjg8DAwKtuXpZOU6UiQrSZJhLTGseQaEI0VGX3bF1dXe0ciWhIyv6erqVPgCTcUhEhHgAkpkrCFaIhkGZkUZNq4u9JEm4pSw03NYd6PLy0EEKIOkoSbqnWge446HVk5ReTmlVg73CEEOKatWjRglmzZlX5+DVr1qDT6Wq9d/eCBQvw9vau1WvURZJwSzk7OXBdgBsAB+Q+rhDChnQ63RWX6dOnX9V5t2zZwiOPPFLl46+//npSU1Px8pKOo7VBeilXEBHiycH0XBJTc7i53ZVnfRBCiJqSmppqef31118zdepUkpKSLNvc3d0tr5VSmEymK867WiYgIKBacRgMBoKDg6v1HlF1UsOtoOw+7n7pOCWEsKHg4GDL4uXlhU6ns6wfOHAADw8PfvnlF7p27YrRaOT333/nyJEj3HHHHQQFBeHu7k737t1ZuXKl1XkvblLW6XR89NFHDBs2DFdXV8LDw1m8eLFl/8VNymVNv8uWLSMiIgJ3d3cGDRpk9QWhpKSE8ePH4+3tjZ+fH5MnTyY2NpahQ4dW63cwZ84crrvuOgwGA23btuW///2vZZ9SiunTp9O8eXOMRiOhoaGMHz/esv8///kP4eHhODs7ExQUxN13312ta9uKJNwKyjtOScIVoiFRSnGhqMTmS012wHz22WeZOXMmiYmJdOrUidzcXG699VZWrVrFjh07GDRoEEOGDCElJeWK53nxxRcZPnw4u3fv5tZbb2XUqFGcO3fussdfuHCBN954g//+97+sW7eOlJQUJk2aZNn/6quv8sUXXzB//nzWr19PdnY2P/zwQ7XKtmjRIp544gmeeuop9u7dy7/+9S8eeOABVq9eDcD333/P22+/zYcffsihQ4f44YcfiIyMBGDr1q2MHz+eGTNmkJSUxNKlS+nbt2+1rm8r0qRcQUSw9mjQsbN55BeZcDHIEI9CNAT5xSbaT11m8+vunzEQV0PNfMzOmDGDW265xbLu6+tLVFSUZf2ll15i0aJFLF68mHHjxl32PKNHj2bkyJEAvPLKK7z77rts3ryZQYMGXfL44uJiPvjgA6677joAxo0bx4wZMyz733vvPaZMmcKwYcMAeP/991myZEm1yvbGG28wevRoHnvsMQAmTpzIxo0beeONN7jppptISUkhODiYmJgYnJycaN68OT169AAgJSUFNzc3br/9djw8PAgLCyM6Orpa17cVqeFWEOBhxM/NgFnBwfQce4cjhBAW3bp1s1rPzc1l0qRJRERE4O3tjbu7O4mJiX9Zw+3UqZPltZubG56enpZhCy/F1dXVkmxBG9qw7PisrCzS09MtyQ/AwcGBrl27VqtsiYmJ9O7d22pb7969SUxMBOCee+4hPz+fVq1a8fDDD7No0SJKSkoAuOWWWwgLC6NVq1bcd999fPHFF1y4cKFa17cVqeFWoNPpiAjx5PfDZ0lMzSaqmbe9QxJC1AAXJwf2zxhol+vWFDc3N6v1SZMmsWLFCt544w1at26Ni4sLd999N0VFRVc8j5OTk9W6TqfDbDZX63hbj1XQrFkzkpKSWLlyJStWrOCxxx7j9ddfZ+3atXh4eLB9+3bWrFnD8uXLmTp1KtOnT2fLli117tEjqeFeREacEqLh0el0uBocbb7U5mhX69evZ/To0QwbNozIyEiCg4M5duxYrV3vUry8vAgKCmLLli2WbSaTie3bt1frPBEREaxfv95q2/r162nfvr1l3cXFhSFDhvDuu++yZs0aNmzYwJ49ewBwdHQkJiaG1157jd27d3Ps2DF+/fXXayhZ7ZAa7kXaBZePOCWEEHVVeHg4CxcuZMiQIeh0Ol544YUr1lRry+OPP058fDytW7emXbt2vPfee5w/f75aXzaefvpphg8fTnR0NDExMfz4448sXLjQ0ut6wYIFmEwmevbsiaurK59//jkuLi6EhYXx008/cfToUfr27YuPjw9LlizBbDbTtm3b2iryVZOEe5GKkxgopWQ8ViFEnfTWW2/x4IMPcv311+Pv78/kyZPJzrZ9y9zkyZNJS0vj/vvvx8HBgUceeYSBAwdWa0adoUOH8s477/DGG2/wxBNP0LJlS+bPn0+/fv0AbS7amTNnMnHiREwmE5GRkfz444/4+fnh7e3NwoULmT59OgUFBYSHh/PVV1/RoUOHWirx1dOpejxw8MmTJ2nWrBknTpygadOmNXLOohIzHaYtpdik+H3yTTT1kRlHhKhPCgoKSE5OpmXLljg7O9s7nEbHbDYTERHB8OHDeemll+wdTo250t9VVXOR1HDLpO+Hk1swtOzDdQHuHEjLITE1RxKuEEJcwfHjx1m+fDk33ngjhYWFvP/++yQnJ/PPf/7T3qHVOdJpqsyKqfDjeDi8ivYyAIYQQlSJXq9nwYIFdO/end69e7Nnzx5WrlxJRESEvUOrc6SGWyY0Gg6vgNM7aRdyM+yQhCuEEH+lWbNmlXoYi0uTGm6Z0NKRSU7vsHScOpAmPZWFEELUDEm4ZcoS7plEIvy1iv+xP/O4UFRix6CEEEI0FJJwy3iGgHswKDP+OQcJ8DCilNRyhRBC1AxJuBVVaFZuFywjTgkhhKg5knArqpBwy3oqH5ARp4QQQtQASbgVXaLjlNRwhRBC1ARJuBWFdtZ+nj1Iez/tV3MgLQezud4OxiWEaET69evHhAkTLOstWrRg1qxZV3yPTqer9oTxtXmeK5k+fTqdO3eu1WvUJkm4FbkHgmdTQNGq5DAGBz25hSWcPJ9v78iEEA3YkCFDLjsB/G+//YZOp2P37t3VPu+WLVt45JFHrjU8K5dLeqmpqQwePLhGr9XQSMK9WGkt1zFtF60D3QFtIgMhhKgtY8aMYcWKFZw8ebLSvvnz59OtWzerieOrKiAgAFdX2wxPGxwcjNFotMm16itJuBeT+7hCCBu7/fbbCQgIYMGCBVbbc3Nz+fbbbxkzZgx//vknI0eOpEmTJri6uhIZGclXX311xfNe3KR86NAh+vbti7OzM+3bt2fFihWV3jN58mTatGmDq6srrVq14oUXXqC4uBjQpsl78cUX2bVrFzqdDp1OZ4n54iblPXv2cPPNN+Pi4oKfnx+PPPIIubm5lv2jR49m6NChvPHGG4SEhODn50dcXJzlWlVhNpuZMWMGTZs2xWg00rlzZ5YuXWrZX1RUxLhx4wgJCcHZ2ZmwsDDi4+MBUEoxffp0mjdvjtFoJDQ0lPHjx1f52ldDhna8WNNuEBwJvq2IcJJHg4RoUIryqv8eByM4lH5UmkrAVAg6PTi5XPm8BrcqX8LR0ZH777+fBQsW8Nxzz1mmBf32228xmUyMHDmS3NxcunbtyuTJk/H09OTnn3/mvvvu47rrrqNHjx5/eQ2z2cydd95JUFAQmzZtIisry+p+bxkPDw8WLFhAaGgoe/bs4eGHH8bDw4NnnnmGESNGsHfvXpYuXWqZq9bLy6vSOfLy8hg4cCC9evViy5YtZGRk8NBDDzFu3DirLxWrV68mJCSE1atXc/jwYUaMGEHnzp15+OGHq/R7e+edd3jzzTf58MMPiY6O5pNPPuHvf/87+/btIzw8nHfffZfFixfzzTff0Lx5c06cOMGJEycA+P7773n77bdJSEigQ4cOpKWlsWvXripd92pJwr1Yq34w9ncA2h8+C8hk9EI0GK+EVv899yyADsO01wd+hG9HQ9gN8MDP5cfMioQLf1q/b3pWtS7z4IMP8vrrr7N27VrLPLDz58/nrrvuwsvLCy8vLyZNmmQ5/vHHH2fZsmV88803VUq4K1eu5MCBAyxbtozQUO338Morr1S67/r8889bXrdo0YJJkyaRkJDAM888g4uLC+7u7jg6OhIcHHzZa3355ZcUFBTw2Wef4eamffF4//33GTJkCK+++ipBQUEA+Pj48P777+Pg4EC7du247bbbWLVqVZUT7htvvMHkyZP5xz/+AcCrr77K6tWrmTVrFrNnzyYlJYXw8HBuuOEGdDodYWFhlvempKQQHBxMTEwMTk5ONG/evEq/x2shTcpX0K60STnl3AVyCqrezCGEENXVrl07rr/+ej755BMADh8+zG+//caYMWMAMJlMvPTSS0RGRuLr64u7uzvLli0jJSWlSudPTEykWbNmlmQL0KtXr0rHff311/Tu3Zvg4GDc3d15/vnnq3yNiteKioqyJFuA3r17YzabSUpKsmzr0KGD1UT1ISEhZGRkVOka2dnZnD59mt69e1tt7927N4mJiYDWbL1z507atm3L+PHjWb58ueW4e+65h/z8fFq1asXDDz/MokWLKCmp3aF8pYZ7OSVF+JJNkKeR9OxCDqbn0DXM195RCSGuxb9PV/89DhU6ArUbop1Dd1FdZcKea4ur1JgxY3j88ceZPXs28+fP57rrruPGG28E4PXXX+edd95h1qxZREZG4ubmxoQJEygqKqqRawNs2LCBUaNG8eKLLzJw4EC8vLxISEjgzTffrLFrVOTk5GS1rtPpMJvNNXb+Ll26kJyczC+//MLKlSsZPnw4MTExfPfddzRr1oykpCRWrlzJihUreOyxxywtDBfHVVOkhnspO7+E+Cbw81OWjlP7pVlZiPrP4Fb9xaFCvcTBUdtW8f7t5c57FYYPH45er+fLL7/ks88+48EHH7Tcz12/fj133HEH9957L1FRUbRq1YqDBw9W+dwRERGcOHGC1NRUy7aNGzdaHfPHH38QFhbGc889R7du3QgPD+f48ePWRTUYMJlMf3mtXbt2kZdXfm97/fr16PV62rZtW+WYr8TT05PQ0NBKUwOuX7+e9u3bWx03YsQI5s2bx9dff83333/PuXPnAHBxcWHIkCG8++67rFmzhg0bNrBnT818eboUqeFeimcTMBXBuSNEtPBkTdIZ6TglhKh17u7ujBgxgilTppCdnc3o0aMt+8LDw/nuu+/4448/8PHx4a233iI9Pd0quVxJTEwMbdq0ITY2ltdff53s7Gyee+45q2PCw8NJSUkhISGB7t278/PPP7No0SKrY1q0aEFycjI7d+6kadOmeHh4VHocaNSoUUybNo3Y2FimT5/OmTNnePzxx7nvvvss929rwtNPP820adO47rrr6Ny5M/Pnz2fnzp188cUXALz11luEhIQQHR2NXq/n22+/JTg4GG9vbxYsWIDJZKJnz564urry+eef4+LiYnWft6ZJDfdSmvWA8TvhX7/Jo0FCCJsaM2YM58+fZ+DAgVb3W59//nm6dOnCwIED6devH8HBwQwdOrTK59Xr9SxatIj8/Hx69OjBQw89xMsvv2x1zN///neefPJJxo0bR+fOnfnjjz944YUXrI656667GDRoEDfddBMBAQGXfDTJ1dWVZcuWce7cObp3787dd99N//79ef/996v3y/gL48ePZ+LEiTz11FNERkaydOlSFi9eTHh4OKD1uH7ttdfo1q0b3bt359ixYyxZsgS9Xo+3tzfz5s2jd+/edOrUiZUrV/Ljjz/i5+dXozFWpFNK1dtxC0+ePEmzZs04ceIETZs2rZVrHErP4Za31+FqcGDv9IHo9bpauY4QomYUFBSQnJxMy5YtcXZ2tnc4ooG40t9VVXOR1HD/Qkt/NwyOei4UmUg5d8He4QghhKinJOFezqlt8M39OC59mrZBMgCGEEKIayMJ93KKC2D//yBpKREhknCFEEJcG0m4lxPSCdBB9kk6+2iDXsijQUIIIa6WJNzLMXqAfxsAop2OAXBAZg0SQghxlSThXknpzEEtC7WHy0+ezydbhngUol6oyRGLhKiJvycZ+OJKQqNhdwLOZ3YT6tWT01kFHEjNoUdLGeJRiLrKYDCg1+s5ffo0AQEBGAwGy2hNQlSXUoqioiLOnDmDXq/HYDBc9bkk4V7JRXPjns4qIDE1WxKuEHWYXq+nZcuWpKamcvr0VYydLMQluLq60rx5c/T6q28YloR7JcGR2iDluWl0bVvAKuQ+rhD1gcFgoHnz5pSUlPzluL9C/BUHBwccHR2vuaXErgk3Pj6ehQsXcuDAAVxcXLj++ut59dVXa2xw62tmcIWACMjYR3en44CP9FQWop7Q6XQ4OTnV2swvQlSXXTtNrV27lri4ODZu3MiKFSsoLi5mwIABVjNM2F1ps/J1JYcASErLxmSut6NhCiGEsBO71nCXLl1qtb5gwQICAwPZtm0bffv2tVNUFwntDDs/xydzH85Of6Og2MyxP/O4LsDd3pEJIYSoR+rUY0FZWVkA+PpeulNSYWEh2dnZliUnxwbNu6FdANCd3kHbQC3JyohTQgghqqvOJFyz2cyECRPo3bs3HTt2vOQx8fHxeHl5WZaqzgN5TYI6gN4RLpzlb/4FAByQ+7hCCCGqqc4k3Li4OPbu3UtCQsJlj5kyZQpZWVmWZf/+/bUfmJMz3PIS3PMpzZtoc1NKDVcIIUR11YnHgsaNG8dPP/3EunXrrjiXoNFoxGg0Wtazs22U+Ho9BkB48jngmCRcIYQQ1WbXGq5SinHjxrFo0SJ+/fVXWrZsac9w/lK70lmDTmcVkHmhyM7RCCGEqE/smnDj4uL4/PPP+fLLL/Hw8CAtLY20tDTy8/PtGVZlpmI48iue2/5DEy9nAA6kyX1cIYQQVWfXhDtnzhyysrLo168fISEhluXrr7+2Z1iVKQVfjoAVU+kToD0jLM3KQgghqsOu93CVqicDSDgaoO1g0DvS1skZDhdLwhVCCFEtdaLTVL0w/DMAgvekwobtJMqjQUIIIaqhzjwWVF+0C/EE4GB6DiUmmW9TCCFE1UjCrQ6zmTB1GjeDjsISbYhHIYQQoiok4VaV2QxvtUM/uxv9/LXmZJk5SAghRFVJwq0qvR68wwDo634SkJ7KQgghqk4SbnWUTtUXqTsKSMIVQghRdZJwq6M04TYrSAJkEgMhhBBVJwm3OkoTrvu5/egxk5ZdwPk8GeJRCCHEX5OEWx3+4eDkhq44j97e5wFpVhZCCFE1knCrQ+8AIVEA3OypdZzaLwlXCCFEFUjCra7QzgB0djgGyCQGQgghqkYSbnWV3sdtUXgQkCZlIYQQVSMJt7pKE65XViIOmDiUnkuxDPEohBDiL0jCrS7f68Dggd5UQJQxjSKTmaNnZIhHIYQQVyYJt7r0est93P5epwBpVhZCCPHXJOFejdKE294lE4DENEm4QgghruyqEu6JEyc4efKkZX3z5s1MmDCBuXPn1lhgdVrvCfBsCqe7TASQuXGFEEL8patKuP/85z9ZvXo1AGlpadxyyy1s3ryZ5557jhkzZtRogHWSmz84exFROjeuNCkLIYT4K1eVcPfu3UuPHj0A+Oabb+jYsSN//PEHX3zxBQsWLKjJ+Oq0tkEe6HRwJqeQs7mF9g5HCCFEHXZVCbe4uBij0QjAypUr+fvf/w5Au3btSE1Nrbno6rJtC3D78g4e8tgEyEQGQgghruyqEm6HDh344IMP+O2331ixYgWDBg0C4PTp0/j5+dVogHXW+WNw/Hf6Oh8GpFlZCCHElTlezZteffVVhg0bxuuvv05sbCxRUdr4wosXL7Y0NTd4HYaBXzhHTwVDRoEkXCGEEFd0VQm3X79+nD17luzsbHx8fCzbH3nkEVxdXWssuDotJApCogg1psPvW2USAyGEEFd0VU3K+fn5FBYWWpLt8ePHmTVrFklJSQQGBtZogHVdu2APAI6cyaWoRIZ4FEIIcWlXlXDvuOMOPvvsMwAyMzPp2bMnb775JkOHDmXOnDk1GmCdlnGApgc/I8Y5kWKT4siZXHtHJIQQoo66qoS7fft2+vTpA8B3331HUFAQx48f57PPPuPdd9+t0QDrtL3foVs6mVEuWk9luY8rhBDicq4q4V64cAEPD60pdfny5dx5553o9Xr+9re/cfz48RoNsE4L7QJAhDoCSMIVQghxeVeVcFu3bs0PP/zAiRMnWLZsGQMGDAAgIyMDT0/PGg2wTiudqi+wIBlnCmWIRyGEEJd1VQl36tSpTJo0iRYtWtCjRw969eoFaLXd6OjoGg2wTvMMAfdg9JhprzvOAZnEQAghxGVcVcK9++67SUlJYevWrSxbtsyyvX///rz99ts1Fly9UFrLjXI4ytncIjJyCuwckBBCiLroqqfnCw4OJjo6mtOnT1tmDurRowft2rWrseDqhdKE28s5BZCZg4QQQlzaVSVcs9nMjBkz8PLyIiwsjLCwMLy9vXnppZcwmxvZs6ilCTdSnwxIxykhhBCXdlUjTT333HN8/PHHzJw5k969ewPw+++/M336dAoKCnj55ZdrNMg6rXQy+uCiFNzI54AkXCGEEJdwVQn3008/5aOPPrLMEgTQqVMnmjRpwmOPPda4Eq57IHg2RZd9kg66YySmNq6RtoQQQlTNVTUpnzt37pL3atu1a8e5c+euOah6p7SWG6k/ypEzuRSWmOwbjxBCiDrnqhJuVFQU77//fqXt77//Pp06dbrmoOqd0vu4XZ2OUWJWHEqXIR6FEEJYu6om5ddee43bbruNlStXWp7B3bBhAydOnGDJkiU1GmC9UJpwox2OAVrHqY5NvOwYkBBCiLrmqmq4N954IwcPHmTYsGFkZmaSmZnJnXfeyb59+/jvf/9b0zHWfaHR0LQ7x32vR4eZA2nyaJAQQghrV1XDBQgNDa3UOWrXrl18/PHHzJ0795oDq1dcfeGhlaRsOYE6sVseDRJCCFHJVQ98ISqLCNHGkU5MzUYpZedohBBC1CWScGtQuI+O6/SnOX+hmPTsQnuHI4QQog6RhFtTTmzG+Y0wPje+DkCiTGQghBCigmrdw73zzjuvuD8zM/NaYqnf/NuAMuPiaMKFAhJTs7mprQyCIYQQQlOthOvldeVHXby8vLj//vuvKaB6y8UbJh3iy63Z5C9NkkkMhBBCWKlWwp0/f35txdEwuAcSEaK9lJ7KQgghKpJ7uDWsfWlP5aNncikoliEehRBCaCTh1qTMEwT+eD8/O7+AWSFDPAohhLCQhFuTXLzRHVpOB47gT5Y0KwshhLCwa8Jdt24dQ4YMITQ0FJ1Oxw8//GDPcK6d0UPrrQx01B9lvyRcIYQQpeyacPPy8oiKimL27Nn2DKNmlU5k0EmXLDVcIYQQFlc9lnJNGDx4MIMHD7ZnCDUvNBp2JxCpP8qnaTkopdDpdPaOSgghhJ3JPdyaVlbD1R8lK7+Y1KwCOwckhBCiLrBrDbe6CgsLKSwsH6M4J6cODi4RHAk6PUFkEsh5ElOzCfV2sXdUQggh7Kxe1XDj4+Px8vKyLO3bt7d3SJUZXCEgAtBquXIfVwghBNSzhDtlyhSysrIsy/79++0d0qWVNitH6o+SKJPRCyGEoJ4lXKPRiKenp2Xx8PCwd0iXFtoZgE46qeEKIYTQ2PUebm5uLocPH7asJycns3PnTnx9fWnevLkdI7tGoV0AiNQnc+xsLvlFJlwMDnYOSgghhD3ZtYa7detWoqOjiY7WmmAnTpxIdHQ0U6dOtWdY1y6oA+gd8ddlE6z+JCldmpWFEKKxs2sNt1+/fiil7BlC7XByhsD2kLZbu4+bmk3nZt72jkoIIYQd1avHguqVm/7N11tPsmGPJ8FyH1cIIRq9etVpql5pOxiniMFk4y6T0QshhJCEW5siSufGTUzLbphN50IIIapMEm4tap31B087fYtbQTonz+fbOxwhhBB2JAm3Fjmte5U4h0V01R/igAyAIYQQjZok3NoUcTubvAaTqnxlAAwhhGjkpJdyberzFLvVUbYvSSRIEq4QQjRqUsOtZZaOU5JwhRCiUZOEW8siAp1prztG5rkM8gpL7B2OEEIIO5GEW8v8Fg5nifHf9NXtkiEehRCiEZOEW9uCOgIyN64QQjR2knBrm2Vu3GRJuEII0YhJwq1tpQm3oy6ZpNOZ9o1FCCGE3UjCrW3+4ZgdXXHTFVKYfhCzWYZ4FEKIxkgSbm3TO0BoFACtiw/KEI9CCNFIScK1AX1oF0C7j7tf7uMKIUSjJAnXFkrv40pPZSGEaLwk4dpCacLtoDvGwdTzdg5GCCGEPUjCtQXfVpQ4ueOsKyb/9H57RyOEEMIOJOHagl6PCtY6TgXk7CenoNjOAQkhhLA1Sbg24tRM6zjVSXeUJJkbVwghGh1JuLYSGo0JPe66fOk4JYQQjZDMh2srbW/j3R6reWfdKULXHKFjEy+im/vYOyohhBA2IjVcW3FyZmTvdrTwc+V0VgHDP9zAJ78no5SMPCWEEI2BJFwbCvZy5sd/deaFFol4m84z46f9PPr5drKlE5UQQjR40qRsS0m/4LH2Ncakbadp77mM26hj6b40mp5czNPGHzD6NAH3IPAILv0ZAh5B4B6s/XT2Bp3O3qUQQghxFSTh2lJIZwhqDwY3Bl7fjW87+xH3xXZcck5iLDwG2ceu/H5H5/KEHNAW/v5e+b5T28HRCL6twMmlFgshhBDiakjCtSXPELhjtmW1sx/8PP4GZiSUcM+h9gTpznNzUzO3t9JjuJABOWmQmw45qVCQBSUFkHlcW4ouWJ970b/g7EG4fzG0utHGBRNCCPFXJOHambergTcfuIW5667jtWVJ/JSimJ3vxpx7u9ImyKP8wOL80uSbpi0OTtYncvXTFo+Q8m0Hl4NnKAR3tE1hhBBCXJZO1eNusidPnqRZs2acOHGCpk2b2juca7bl2DnGfbmd9OxCnJ30/N/QSO7uepXlSt8HH8UAOrjzQ4gYUqOxCiGE0FQ1F0kv5TqkewtflozvQ59wfwqKzUz6dheTv9tNQbGp+ifzCIGm3aE4D76+F9a+BvX3u5UQQtR7knDrGD93Iwse6MHEW9qg08HXW08wdPZ6jp7Jrd6JXH3h3oXQ41/a+uqX4dvRUJRX4zELIYT4a5Jw6yAHvY7x/cP5fExP/N0NHEjLYch7v/PT7tPVPJEj3PoaDHkH9E6w/wf4ZBBknqiVuIUQQlyeJNw6rHdrf5aM70OPlr7kFZkY9+UOpv1vL4Ul1Wxi7joaYheDqz+k7YZ5N0HKplqJWVwk9wyUFJWv55+HP4/YLx4hhN1Iwq3jAj2d+fKhnjzW7zoAPt1wnHs+2MCJcxf+4p0XCbseHlkNQZGQdwYW3AY7Pq+FiIXFVyPhjdZw/Pfybds+hfe6wGdDIfFHMJXYLTwhhG1Jwq0HHB30PDOoHfNHd8fb1YndJ7O47d3fWLE/vXon8m4ODy6FiL+DuRj+FwdL/y0f+teqIAv2LYKfnwKzuXy7q6/2M3V3+bbM44AOjq7WOrPNioQ1MyE71aYhCyFsTx4LqmdOZeYz7svt7EjJBOCRvq14emBbnByq8d3JbIZ1r8GaeG39upvhn99UfrZXXJpScOYAHFquPet8YiOYS7+0PPQrNO2qvT5/XBv9yyPY+v3nj8G2BbD9v3DhrLZN5wDtboPuY6DljTKEpxD1SFVzkSTceqioxMzMXw7wyfpkALqG+fD+P6MJ8armkI77/weLxkK3B2Hgy7UQaQNSdAGO/QYHl8GhFZCVYr3fvw2ED9B+l37XVe2cJYVas/KWjyHlj/Ltfq2180SNLK8lCyHqLEm4jcDSvak8/e1ucgpL8HF1YtY/ormxTUD1TnL2EPi01Ho0A5hNoHeo+WDro/PHtBrsoeVasi0pKN/nYISWfSB8IITfAr4tr+1a6fth68ew62soytG2OTpDx7u0L0MuMneyEHWVJNxG4vifeTz2xXb2nc5Gp4NxN7VmQkwbHPRX0SRZUgSf3wltBkKvcY2vWdNsBn2FpvkP+mi9ust4NdNqseEDoGVfMLjWfAyFubDnG9jyCaTvAc8m8MTu8i9ESjW+fxch6riq5iIZS7meC/Nz4/tHr+eln/bzxaYU3vv1MFuPnWfGHR3wdTPgZnTE6KhHV5UP6X2LtJpc6i6tZuUZWvsFqAuK8rSm9eN/wBO7wOiubW93Gxg9tRpsm4EQ0K72k53RXWtO7voAnNyi9SgvS7amYvjgBu0e701TpNYrRD0jNdwG5H87TzFl4R4uFFk/p+uo1+FqcMDd6IibZXHAzeBo2eZqdMDdyYFuGd9R7BVGdrObcDNq+yu+1706CbwuyjqlfanIPw9/e1TbphS821lrQh6ZAG0Hl2+vS+VMWgpfjQC3AHhyPzga7B1R7SjIhoxE7d8jqIO21KV/ByEuIjXcRuiOzk3oEOrF5O93s+90FgXF2iMqJWZFdkEJ2QVVefwnqvTnDgB66BLJx8ge1cpyhENpAg9wNzKwYzB3RjchvOLMRnVJ7hktwSav037+eVjbbvSE7g9rtUedDga/rs22FBpd/t669iEfPkAbrvPCufJkazbDJwO1+8ldR2uPftUXphLt3yNjn3YPO32f9jrzog5pkffAXR/ZJ0YhapDUcBswk1mRV1RCXmHZYiKvsITcwpLS7abyfUUV9pWue1w4yVtZT2KkkOfMY/m+6G+XvVZkEy/u7NKEIVGh+LsbbVjKi+Sfh2Pry5Nsxn7r/To9hHTW7sH2nQTGOvpFoaoOr4TP79Je6/RaUm4doz2K5BGi/XQPqjuPfO39Xuvlnb4XzhwEU+Glj/MIBe9m2jPMMdPhb2O17VmntLmfG2s/A1EnSacpce0KsuD7h7ReuoD5hqfI6z2ZC8WK3MISDqTmsGjHKdYkZVBi1v6MHPQ6+rUJ4M4uTekfEYizkw16POeegT/e1RJs6i7goj/poEitBtiyrzbilrNX7cdkK6ZiSFqiPVqUvPYyB+nAzb88CXuGwu2zypNV2fPCbgE110PdVAzLn9dqrf/8Ggxu2vafnoStn5QfZ3CHwAgIbA9BHSGovfa67HGo4gLtGeey++rbPoUfx2szYT20svw8x9ZDSKf6/wVK1EuScEXNMJtg1Yuw/h1tve1t2vy6FT7Y/swt5Mddp1m04xS7TmZZtns4O3J7pxDu7NKUbmE+NXPftzgfTmwCZdYG7ADti8GrLbRtoD0T27JvaYK9Adz8rv269cHZQ7DzC+1nTpq25KaVD8pRxi0Qnj5Uvr7gdq1F4K6PIfJubdupbbDji/JaskcIeARpP118AQXnjpY2A5c2B7sFwJBZ5ed9vbXW6evhX6FJ6WAgh1dp5w7qoCVW7zDrnuF/JeskHFiiJeSyWPMz4bVWWg0/rFdpLf8WCGgrNWBhE5JwRc3a9TUsflxrAgxsDyO/Ap8WlQ47nJHDwu2n+GHHKU5nlT+32szXhWHRTbkzugkt/N2qft2SIm0YyrIa0o7PtSEpm/aAh1aUH7f2Ne154pZ9Ko/s1JiZzXDhT8hJLU3CqaBMWk/oMp8M0r7ExP4ILW7Qtm39RKuNXoreSasJV3wuGbTHpp7cW76+5SNwdNGaf938a7ZcFaXuhm/uh/PJF8XTHMJjKjzGVf53dz6viD2nsthzSvuC2NLfjVYBbrTwc6u5VhmltN993lkIbFcz5xR1kiRcUfNOboWEUVqtycUX7pqn1SaLC6AkX7tXWJrszLlnOfLHd2w+nssrJzqSV9pz+iGHn+nmeZ5wH0eaeYBBFWm11uJ87RzFBdavS/JhwMvQ6zEthvPHtU5C1/WHO96XGkxNKRtPu+wRpFPbtFG1sk+X15ZzUsuHogQtmQa2K62tlvYmbnWj7WMv8+cR7fbHoRVw7Her+8NmvYFTXl3Y5NCFhbnt+SPTB6j8t6PTQaiXC60C3LguwN2SiFsFuBPi7oS+4Lz2O8g7q9XeyxJq3hlte9PucP3j2skKsmFmM+31v1PLn9te9hzs/kbrpOfmr9XWXf1KF//Sn74V9vuBUzVHkRM2Jb2URc1r2k2bcSjhn3B6R3lnnTL9p0KfpwDQZ58k/I/JhHuEcufze1m+P43vt5/i1mOb6JJ/GPKrcd1T28pf+4TBxERJtDXN4aKPgiZdy5uBKyopgtx0rdXBO6xujUrmdx257g+zL3A4ic3SKDi0luCMdXQp3EpzztDs/EaasZG7gf2GMOI83qFjU2+MehPOpzeTn3WG7wu6cSozn1OZ+bQ/Op+2DjvxIxtnXTaQC7q/qJ+YTeUJ1+ihjRbm6AwFmeUJNycN8jK05UwVyxY+AEZ9W77+4xPg5Ap9ny6/3515AkxF2mApTs5V/70Jm6kTCXf27Nm8/vrrpKWlERUVxXvvvUePHj3sHZa4FM9QeOAX+HkS7E4AvaP2geLkotV4yrj6ar1l3QJwMThwR+cm3NG5Cdm//4ttR46wI7WAEzmKfAwUKAOORhc6twqhV9tmtA71R+fkop3TyVW7N1iRJFv7cTRovYfrgPwiE/tTs9h9Mos9J7PYfSqLI2dyKW+zCwPuA+7leu/z3Om+n56m7TTJ2k54eAdWj7pJO6wwF+L7A/DvZ5I5mg1Hz+TSbvNXRJ1JrHTd88qdc8qDs3hxTnlwTnmS7+SN3j0Qc2EbzOuO0NLfnVYBbjSffAInp4uelx40E3o/odWOL/ypPeZ14WyF9T8hr8Jrc7F1ZzBTiTb5BVi+4AJax8HNc7XXbgFa4vVqqi2eTcCrCXiWrnsE160vS2WUgqJcrXUABegq/H/XaV8kKg74klfa4uLiW94XoOiC9sVDp6v8/orbdHqts6AN2b1J+euvv+b+++/ngw8+oGfPnsyaNYtvv/2WpKQkAgMDr/heaVK2s2scGGL/6WwW7TjJDztPcyanvPmvVYAbd0Y3YWh0E5r6VH/4xBKTmcKSssVEYXGF1yVmCoorb1MKjI56DI56jI56jE4O2k9HPUZHhwrbtfWyffV2AJB6prDExIHUHHafymL3iUz2nMriUEYuJnPlj69QL2cim3rRqak3kU28iGzihY9bhaRXmKs9Plb2xUEpbRhPoweM+G/5/eYTmyHrBLj6U+Lsy6kSD47mOnHkz0KOnMnj6Jlcks/mkZFzmUeb0HrtN/d1JczPFV83A94uBrxdnfB2dcLLxQkf19J1FwNerk54Ojta/00pBYU5Wse3sppsSSFs+lBL0jdPLW+d+OVZLRGXVKH5SOegdYBrdxvc+lr59oPLwT1A6zFeU4+SlRRqTfwFWdDxzvLtG2ZD8m+Qf0779yhbLu7kV1HEEBhRYR7v6aVPHEw6BO6l+eLnSbBl3l/H5d8Gxm2pfnkuod7cw+3Zsyfdu3fn/fffB8BsNtOsWTMef/xxnn322Su+VxJuw1BiMrP+yJ8s3H6SZfvSLAN2APRo6UsTb5dLJk5tvUISLU2yl/oQri0Gh8qJ2FCWsB3Ktmv7nBx06HQ6yv7LlUVZ9j+wfN16P5b9ynK8usS2MnqdDgcHHU56HQ56PY76i9YddDjqtaXiukPZNgd96bE6HB20Yy5ed9DpMCvt6qriz9LYFGj7VWmUCsxKi1d7XaEsKMxmKp2rsMRMYmoOe05lkpSWQ7Gp8r+rv7uRqKZepQnWi45NvAj0sG1zak5BMcln8zh6Jo+jZ7VEfPRMHsln88gvNv31CSpw0OvwcnHC28UJL1ftp4+rofR1ebL2djXg7eJkSdYezo7o9TrtF5p/XuvNnXUSsk9Z/8w6BTmny5Na1EgY9oH2uqQQ/q80aT19pPyLx+Z5cGp7aQ25idb5rGKCvNTS4U647Q3t/fnntacIAJ4/Uz5oy8JHYPfXl/5F6By0Gqjlj19pryOGwPDPyo+ThFt1RUVFuLq68t133zF06FDL9tjYWDIzM/nf//5ndXxhYSGFheXfJk+dOkX79u0l4TYgOQXFLN2bxsLtp9iY/CfX+td5pYToXKEmC9q0h4UlptKfl68h199uhvWbr5uByCZaYtV+ehPkaayzrQxmsyI9p4CjZ/I4ce4CmfnFZF4oJiu/iPN5xWTmF5Wua9urm5wr0uvA08XJMi+2pRG19IWudItOB3plwo8sgtQZ8nUuJOvDAPBWmbxe9DLeZDPC+QPQ6dCh4/8KZ9LXtLFa8fzq0Jvpxqe1FWXi44KnyNG585zxWXJ1bigFXU27CVHpZOvcyVYe2k+dO9l4UIhBu77Ougw6nVY2na60REppLcSATq9HBzhgQo8ZPTp0pffc9TrQU/ZaodcpQI+HpzefjO5e7d/3xepFp6mzZ89iMpkICgqy2h4UFMSBAwcqHR8fH8+LL75oq/CEHXg4O3FPt2bc060ZpzLzWbk/naISs1VNsWICdbZq4nWodJz+amZNugKlFCVmVVrDLq9VF12m5l0xgReVmCudryxZXPwBab1NZ7XPcshl3mtWWquByazFajIrii9aLzEpSsxmbd2kbb943VS6XmIqfY/ZXHouhVkpy4eeTqfVqss+DNHp0F/0wagv3Vn5+LIP1YvOBej1OloHutOpiVaDbeLtUmeT66Xo9TpCvFyqPE91QbHJknwzLxSRmV9M1oVizpe+LkvWmReKOX+hmKzS7ReKTJgVZF4ornJsp3BFu8cNZT0YT2HkVmaUbip/5Otj/Y1s1IURovuTEN2fOFNEJh5kKjcycSdTuZOFG5lKe52JO2eVJ+fzLljOcQuvaC8ulF/vJOFA+CWiMwMFl9he80Lzsm1ynTJ1otNUVU2ZMoWJEyda1stquKJhauLtQuz1LewdhhWdToeTgw4nBz3uxnr130fUcc5ODjg7ORDkWb0m8cISLVFnXSimxKyueLvhcvsq3s64+FYH9Kp0bHDpApfqxlG+4eJ9Fx9a8QvUxfvKbi2Uvy6NXFnfjim/DVFeLhRW2yrGbiljab8NW7LrJ4a/vz8ODg6kp6dbbU9PTyc4uPLgBUajEaOxvFdZdrZtv50IIURdY3R0INDDweb3rkX12Ta9X8RgMNC1a1dWrVpl2WY2m1m1ahW9evWyY2RCCCFEzbJ7m9jEiROJjY2lW7du9OjRg1mzZpGXl8cDDzxg79CEEEKIGmP3hDtixAjOnDnD1KlTSUtLo3PnzixdurRSRyohhBCiPrN7wgUYN24c48aNs3cYQgghRK2x6z1cIYQQorGoEzXcq2U2a881pqam2jkSIYQQjVVZDirLSZdTrxNu2eNEMtGBEEIIe0tPT6d58+aX3W/3sZSvRUlJCTt27CAoKAi9/tpax3Nycmjfvj379+/Hw8Pjr9/QwDTm8jfmskPjLr+UvXGWHWq2/GazmfT0dKKjo3F0vHw9tl4n3JqUnZ2Nl5cXWVlZeHp62jscm2vM5W/MZYfGXX4pe+MsO9in/NJpSgghhLABSbhCCCGEDUjCLWU0Gpk2bZrVWM2NSWMuf2MuOzTu8kvZG2fZwT7ll3u4QgghhA1IDVcIIYSwAUm4QgghhA1IwhVCCCFsQBJuqdmzZ9OiRQucnZ3p2bMnmzdvtndINrFu3TqGDBlCaGgoOp2OH374wd4h2Ux8fDzdu3fHw8ODwMBAhg4dSlJSkr3Dsok5c+bQqVMnPD098fT0pFevXvzyyy/2DssuZs6ciU6nY8KECfYOxSamT5+OTqezWtq1a2fvsGzm1KlT3Hvvvfj5+eHi4kJkZCRbt261ybUl4QJff/01EydOZNq0aWzfvp2oqCgGDhxIRkaGvUOrdXl5eURFRTF79mx7h2Jza9euJS4ujo0bN7JixQqKi4sZMGAAeXl59g6t1jVt2pSZM2eybds2tm7dys0338wdd9zBvn377B2aTW3ZsoUPP/yQTp062TsUm+rQoQOpqamW5ffff7d3SDZx/vx5evfujZOTE7/88gv79+/nzTffxMfHxzYBKKF69Oih4uLiLOsmk0mFhoaq+Ph4O0Zle4BatGiRvcOwm4yMDAWotWvX2jsUu/Dx8VEfffSRvcOwmZycHBUeHq5WrFihbrzxRvXEE0/YOySbmDZtmoqKirJ3GHYxefJkdcMNN9jt+o2+hltUVMS2bduIiYmxbNPr9cTExLBhwwY7RiZsLSsrCwBfX187R2JbJpOJhIQE8vLy6NWrl73DsZm4uDhuu+02q//7jcWhQ4cIDQ2lVatWjBo1ipSUFHuHZBOLFy+mW7du3HPPPQQGBhIdHc28efNsdv1Gn3DPnj2LyWQiKCjIantQUBBpaWl2ikrYmtlsZsKECfTu3ZuOHTvaOxyb2LNnD+7u7hiNRsaOHcuiRYto3769vcOyiYSEBLZv3058fLy9Q7G5nj17smDBApYuXcqcOXNITk6mT58+5OTk2Du0Wnf06FHmzJlDeHg4y5Yt49FHH2X8+PF8+umnNrl+vZ6eT4iaEhcXx969exvNvSyAtm3bsnPnTrKysvjuu++IjY1l7dq1DT7pnjhxgieeeIIVK1bg7Oxs73BsbvDgwZbXnTp1omfPnoSFhfHNN98wZswYO0ZW+8xmM926deOVV14BIDo6mr179/LBBx8QGxtb69dv9DVcf39/HBwcLHPrlklPTyc4ONhOUQlbGjduHD/99BOrV6+madOm9g7HZgwGA61bt6Zr167Ex8cTFRXFO++8Y++wat22bdvIyMigS5cuODo64ujoyNq1a3n33XdxdHTEZDLZO0Sb8vb2pk2bNhw+fNjeodS6kJCQSl8oIyIibNak3ugTrsFgoGvXrqxatcqyzWw2s2rVqkZ1P6sxUkoxbtw4Fi1axK+//krLli3tHZJdmc1mCgsL7R1Grevfvz979uxh586dlqVbt26MGjWKnTt34uDgYO8QbSo3N5cjR44QEhJi71BqXe/evSs9+nfw4EHCwsJscn1pUgYmTpxIbGws3bp1o0ePHsyaNYu8vDweeOABe4dW63Jzc62+2SYnJ7Nz5058fX1p3ry5HSOrfXFxcXz55Zf873//w8PDw3LP3svLCxcXFztHV7umTJnC4MGDad68OTk5OXz55ZesWbOGZcuW2Tu0Wufh4VHpPr2bmxt+fn6N4v79pEmTGDJkCGFhYZw+fZpp06bh4ODAyJEj7R1arXvyySe5/vrreeWVVxg+fDibN29m7ty5zJ071zYB2K1/dB3z3nvvqebNmyuDwaB69OihNm7caO+QbGL16tUKqLTExsbaO7Rad6lyA2r+/Pn2Dq3WPfjggyosLEwZDAYVEBCg+vfvr5YvX27vsOymMT0WNGLECBUSEqIMBoNq0qSJGjFihDp8+LC9w7KZH3/8UXXs2FEZjUbVrl07NXfuXJtdW2YLEkIIIWyg0d/DFUIIIWxBEq4QQghhA5JwhRBCCBuQhCuEEELYgCRcIYQQwgYk4QohhBA2IAlXCCGEsAFJuEIIIYQNSMIVQlSJTqfjhx9+sHcYQtRbknCFqAdGjx6NTqertAwaNMjeoQkhqkgmLxCinhg0aBDz58+32mY0Gu0UjRCiuqSGK0Q9YTQaCQ4Otlp8fHwArbl3zpw5DB48GBcXF1q1asV3331n9f49e/Zw88034+Ligp+fH4888gi5ublWx3zyySd06NABo9FISEgI48aNs9p/9uxZhg0bhqurK+Hh4SxevNiy7/z584waNYqAgABcXFwIDw+v9AVBiMZMEq4QDcQLL7zAXXfdxa5duxg1ahT/+Mc/SExMBCAvL4+BAwfi4+PDli1b+Pbbb1m5cqVVQp0zZw5xcXE88sgj7Nmzh8WLF9O6dWura7z44osMHz6c3bt3c+uttzJq1CjOnTtnuf7+/fv55ZdfSExMZM6cOfj7+9vuFyBEXWezeYmEEFctNjZWOTg4KDc3N6vl5ZdfVkppUw2OHTvW6j09e/ZUjz76qFJKqblz5yofHx+Vm5tr2f/zzz8rvV6v0tLSlFJKhYaGqueee+6yMQDq+eeft6zn5uYqQP3yyy9KKaWGDBmiHnjggZopsBANkNzDFaKeuOmmm5gzZ47VNl9fX8vrXr16We3r1asXO3fuBCAxMZGoqCjc3Nws+3v37o3ZbCYpKQmdTsfp06fp37//FWPo1KmT5bWbmxuenp5kZGQA8Oijj3LXXXexfft2BgwYwNChQ7n++uuvqqxCNESScIWoJ9zc3Co18dYUFxeXKh3n5ORkta7T6TCbzQAMHjyY48ePs2TJElasWEH//v2Ji4vjjTfeqPF4haiP5B6uEA3Exo0bK61HREQAEBERwa5du8jLy7PsX79+PXq9nrZt2+Lh4UGLFi1YtWrVNcUQEBBAbGwsn3/+ObNmzWLu3LnXdD4hGhKp4QpRTxQWFpKWlma1zdHR0dIx6dtvv6Vbt27ccMMNfPHFF2zevJmPP/4YgFGjRjFt2jRiY2OZPn06Z86c4fHHH+e+++4jKCgIgOnTpzN27FgCAwMZPHgwOTk5rF+/nscff7xK8U2dOpWuXbvSoUMHCgsL+emnnywJXwghCVeIemPp0qWEhIRYbWvbti0HDhwAtB7ECQkJPPbYY4SEhPDVV1/Rvn17AFxdXVm2bBlPPPEE3bt3x9XVlbvuuou33nrLcq7Y2FgKCgp4++23mTRpEv7+/tx9991Vjs9gMDBlyhSOHTuGi4sLffr0ISEhoQZKLkTDoFNKKXsHIYS4NjqdjkWLFjF06FB7hyKEuAy5hyuEEELYgCRcIYQQwgbkHq4QDYDcGRKi7pMarhBCCGEDknCFEEIIG5CEK4QQQtiAJFwhhBDCBiThCiGEEDYgCVcIIYSwAUm4QgghhA1IwhVCCCFsQBKuEEIIYQP/DwJSReFGugSPAAAAAElFTkSuQmCC", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABO1UlEQVR4nO3deVxU5f7A8c/MwAz7IrIq4Ia4grhGLllaYmXprZvX6y0sb/0qzMws81aKdbvaem27VnbT223BNq1b5ppLmSYuKG64pILK4sYqDDDz/P4YGBhBBQRnwO/79TqvmTnnOed8z+PId57znHMejVJKIYQQQgiHpLV3AEIIIYS4OEnUQgghhAOTRC2EEEI4MEnUQgghhAOTRC2EEEI4MEnUQgghhAOTRC2EEEI4MEnUQgghhAOTRC2EEEI4MEnUQggbQ4cOZcqUKfYOQwhRQRK1EI1swoQJaDSaGlNcXJy9QxNCNENO9g5AiJYoLi6OhQsX2swzGAx2ikYI0ZxJi1qIJmAwGAgKCrKZfH19AVi3bh16vZ6ff/7ZWv6VV14hICCA7OxsAJYvX86gQYPw8fHBz8+P22+/ncOHD1vLHz16FI1GwxdffMHgwYNxdXWlX79+HDhwgOTkZPr27YuHhwcjR47k1KlT1vUmTJjA6NGjmT17Nv7+/nh5efHwww9TWlp60WMxGo1MmzaNNm3a4O7uzoABA1i3bp11+bFjxxg1ahS+vr64u7vTvXt3li1bdtHt/etf/yIiIgIXFxcCAwO5++67rcvMZjNz5syhffv2uLq6Eh0dzVdffWWz/u7duxk5ciQeHh4EBgZy7733cvr0aevyoUOHMnnyZJ5++mlatWpFUFAQiYmJF41HCEcniVqIq6yyD/jee+8lLy+PHTt28Pzzz/Phhx8SGBgIQFFREVOnTmXr1q2sWbMGrVbLmDFjMJvNNtuaNWsWzz33HNu3b8fJyYk///nPPP3007z55pv8/PPPHDp0iJkzZ9qss2bNGvbt28e6dev4/PPP+eabb5g9e/ZF4500aRKbNm0iKSmJXbt28cc//pG4uDgOHjwIQEJCAkajkQ0bNpCamsrLL7+Mh4dHrdvaunUrkydP5oUXXiAtLY3ly5czZMgQ6/I5c+bw8ccf895777Fnzx6eeOIJ/vKXv7B+/XoAcnNzuemmm4iJiWHr1q0sX76c7Oxs7rnnHpv9/Oc//8Hd3Z3ffvuNV155hRdeeIFVq1bV8V9ICAejhBCNKj4+Xul0OuXu7m4zvfTSS9YyRqNR9erVS91zzz2qW7du6sEHH7zkNk+dOqUAlZqaqpRS6siRIwpQH374obXM559/rgC1Zs0a67w5c+aoyMhIm9hatWqlioqKrPPmz5+vPDw8lMlkUkopdcMNN6jHH39cKaXUsWPHlE6nUydOnLCJZ9iwYWrGjBlKKaV69uypEhMT61Q3X3/9tfLy8lL5+fk1lpWUlCg3Nzf166+/2syfOHGiGjdunFJKqRdffFHdcsstNsszMjIUoNLS0qzxDxo0yKZMv3791PTp0+sUoxCORvqohWgCN954I/Pnz7eZ16pVK+t7vV7Pp59+SlRUFOHh4fzzn/+0KXvw4EFmzpzJb7/9xunTp60t6fT0dHr06GEtFxUVZX1f2Rrv2bOnzbycnBybbUdHR+Pm5mb9HBsbS2FhIRkZGYSHh9uUTU1NxWQy0blzZ5v5RqMRPz8/ACZPnswjjzzCypUrGT58OHfddZdNXNXdfPPNhIeH06FDB+Li4oiLi2PMmDG4ublx6NAhzp8/z80332yzTmlpKTExMQDs3LmTtWvX1tpiP3z4sDXOC/cfHBxcox6EaC4kUQvRBNzd3enUqdMly/z6668AnD17lrNnz+Lu7m5dNmrUKMLDw1mwYAEhISGYzWZ69OhRoy/Z2dnZ+l6j0dQ678LT5fVRWFiITqdj27Zt6HQ6m2WVyfKvf/0rI0aM4IcffmDlypXMmTOH119/nccee6zG9jw9Pdm+fTvr1q1j5cqVzJw5k8TERJKTkyksLATghx9+oE2bNjbrVV6IV1hYyKhRo3j55ZdrbDs4ONj6vnodwJXXgxD2JIlaCDs4fPgwTzzxBAsWLGDx4sXEx8ezevVqtFotZ86cIS0tjQULFjB48GAAfvnll0bb986dOykuLsbV1RWAzZs34+HhQWhoaI2yMTExmEwmcnJyrLHUJjQ0lIcffpiHH36YGTNmsGDBgloTNYCTkxPDhw9n+PDhzJo1Cx8fH3766SduvvlmDAYD6enp3HDDDbWu27t3b77++mvatWuHk5P8+RLXBvmmC9EEjEYjWVlZNvOcnJxo3bo1JpOJv/zlL4wYMYL777+fuLg4evbsyeuvv85TTz2Fr68vfn5+fPDBBwQHB5Oens4zzzzTaLGVlpYyceJEnnvuOY4ePcqsWbOYNGkSWm3Na0s7d+7M+PHjue+++3j99deJiYnh1KlTrFmzhqioKG677TamTJnCyJEj6dy5M+fOnWPt2rV07dq11n1///33/P777wwZMgRfX1+WLVuG2WwmMjIST09Ppk2bxhNPPIHZbGbQoEHk5eWxceNGvLy8iI+PJyEhgQULFjBu3DjrVd2HDh0iKSmJDz/8sEarX4iWQBK1EE1g+fLlNqdiASIjI9m/fz8vvfQSx44d4/vvvwcsp2w/+OADxo0bxy233EJ0dDRJSUlMnjyZHj16EBkZyVtvvcXQoUMbJbZhw4YRERHBkCFDMBqNjBs37pK3Ly1cuJC///3vPPnkk5w4cYLWrVtz3XXXcfvttwNgMplISEjg+PHjeHl5ERcXV6PPvZKPjw/ffPMNiYmJlJSUEBERweeff0737t0BePHFF/H392fOnDn8/vvv+Pj40Lt3b/72t78BEBISwsaNG5k+fTq33HILRqOR8PBw4uLiav2hIURLoFFKKXsHIYS4OiZMmEBubi5Lly61dyhCiDqSn6BCCCGEA5NELYQQQjgwOfUthBBCODBpUQshhBAOTBK1EEII4cAkUQshhBAOTBJ1hXfffZd27drh4uLCgAED2LJli71DanIbNmxg1KhRhISEoNFoatyyo5Ri5syZBAcH4+rqyvDhw60jJlU6e/Ys48ePx8vLCx8fHyZOnGh9FGSlXbt2MXjwYFxcXAgNDeWVV15p6kNrdHPmzKFfv354enoSEBDA6NGjSUtLsylTUlJCQkICfn5+eHh4cNddd1mHrayUnp7ObbfdhpubGwEBATz11FOUl5fblFm3bh29e/fGYDDQqVMnFi1a1NSH1+jmz59PVFQUXl5eeHl5ERsby48//mhdLnV1cXPnzkWj0TBlyhTrPKmvKomJiWg0GpupS5cu1uUtsq7sOiSIg0hKSlJ6vV599NFHas+ePerBBx9UPj4+Kjs7296hNally5apZ599Vn3zzTcKUEuWLLFZPnfuXOXt7a2WLl2qdu7cqe644w7Vvn17VVxcbC0TFxenoqOj1ebNm9XPP/+sOnXqZB3pSCml8vLyVGBgoBo/frzavXu3+vzzz5Wrq6t6//33r9ZhNooRI0aohQsXqt27d6uUlBR16623qrCwMFVYWGgt8/DDD6vQ0FC1Zs0atXXrVnXdddep66+/3rq8vLxc9ejRQw0fPlzt2LFDLVu2TLVu3do6CpVSSv3+++/Kzc1NTZ06Ve3du1e9/fbbSqfTqeXLl1/V471S3333nfrhhx/UgQMHVFpamvrb3/6mnJ2d1e7du5VSUlcXs2XLFtWuXTsVFRVlHcFMKamv6mbNmqW6d++uMjMzrdOpU6esy1tiXUmiVkr1799fJSQkWD+bTCYVEhKi5syZY8eorq4LE7XZbFZBQUHq1Vdftc7Lzc1VBoNBff7550oppfbu3asAlZycbC3z448/Ko1GYx0W8V//+pfy9fVVRqPRWmb69Ok2Qy82Rzk5OQpQ69evV0pZ6sbZ2Vl9+eWX1jL79u1TgNq0aZNSyvLDSKvVqqysLGuZ+fPnKy8vL2v9PP3006p79+42+xo7dqwaMWJEUx9Sk/P19VUffvih1NVFFBQUqIiICLVq1SqboUalvmzNmjVLRUdH17qspdbVNX/qu7S0lG3btjF8+HDrPK1Wy/Dhw9m0aZMdI7OvI0eOkJWVZVMv3t7eDBgwwFovmzZtwsfHh759+1rLDB8+HK1Wy2+//WYtM2TIEPR6vbXMiBEjSEtL49y5c1fpaBpfXl4eUDV05bZt2ygrK7Opry5duhAWFmZTXz179rQORwmWusjPz2fPnj3WMtW3UVmmOX8XTSYTSUlJFBUVERsbK3V1EQkJCdx22201jknqq6aDBw8SEhJChw4dGD9+POnp6UDLratrPlGfPn0ak8lk848GlnF8LxxU4VpSeeyXqpesrCwCAgJsljs5OdGqVSubMrVto/o+mhuz2cyUKVMYOHCgdWzorKws9Ho9Pj4+NmUvrK/L1cXFyuTn51NcXNwUh9NkUlNT8fDwwGAw8PDDD7NkyRK6desmdVWLpKQktm/fzpw5c2osk/qyNWDAABYtWsTy5cuZP38+R44cYfDgwRQUFLTYupJBOYSop4SEBHbv3t2oQ0+2RJGRkaSkpJCXl8dXX31FfHw869evt3dYDicjI4PHH3+cVatW4eLiYu9wHN7IkSOt76OiohgwYADh4eF88cUX1qFbW5prvkXdunVrdDpdjasCs7OzCQoKslNU9ld57Jeql6CgIHJycmyWl5eXc/bsWZsytW2j+j6ak0mTJvH999+zdu1a2rZta50fFBREaWkpubm5NuUvrK/L1cXFynh5eTW7P0J6vZ5OnTrRp08f5syZQ3R0NG+++abU1QW2bdtGTk4OvXv3xsnJCScnJ9avX89bb72Fk5MTgYGBUl+X4OPjQ+fOnTl06FCL/W5d84lar9fTp08f1qxZY51nNptZs2YNsbGxdozMvtq3b09QUJBNveTn5/Pbb79Z6yU2Npbc3Fy2bdtmLfPTTz9hNpsZMGCAtcyGDRsoKyuzllm1ahWRkZH4+vpepaO5ckopJk2axJIlS/jpp59o3769zfI+ffrg7OxsU19paWmkp6fb1FdqaqrNj5tVq1bh5eVFt27drGWqb6OyTEv4LprNZoxGo9TVBYYNG0ZqaiopKSnWqW/fvowfP976Xurr4goLCzl8+DDBwcEt97tll0vYHExSUpIyGAxq0aJFau/eveqhhx5SPj4+NlcFtkQFBQVqx44daseOHQpQb7zxhtqxY4c6duyYUspye5aPj4/69ttv1a5du9Sdd95Z6+1ZMTEx6rffflO//PKLioiIsLk9Kzc3VwUGBqp7771X7d69WyUlJSk3N7dmd3vWI488ory9vdW6detsbgs5f/68tczDDz+swsLC1E8//aS2bt2qYmNjVWxsrHV55W0ht9xyi0pJSVHLly9X/v7+td4W8tRTT6l9+/apd999t1neQvPMM8+o9evXqyNHjqhdu3apZ555Rmk0GrVy5UqllNTV5VS/6lspqa/qnnzySbVu3Tp15MgRtXHjRjV8+HDVunVrlZOTo5RqmXUlibrC22+/rcLCwpRer1f9+/dXmzdvtndITW7t2rUKqDHFx8crpSy3aD3//PMqMDBQGQwGNWzYMJWWlmazjTNnzqhx48YpDw8P5eXlpe6//35VUFBgU2bnzp1q0KBBymAwqDZt2qi5c+derUNsNLXVE6AWLlxoLVNcXKweffRR5evrq9zc3NSYMWNUZmamzXaOHj2qRo4cqVxdXVXr1q3Vk08+qcrKymzKrF27VvXq1Uvp9XrVoUMHm300Fw888IAKDw9Xer1e+fv7q2HDhlmTtFJSV5dzYaKW+qoyduxYFRwcrPR6vWrTpo0aO3asOnTokHV5S6wrGT1LCCGEcGDXfB+1EEII4cgkUQshhBAOTBK1EEII4cAkUQshhBAOTBK1EEII4cAkUQshhBAOTBJ1NUajkcTERIxGo71DcXhSV/Uj9VV3Ulf1I/VVd821rhzmPuq5c+cyY8YMHn/8cebNm2eXGPLz8/H29iYvLw8vLy+7xNBcSF3Vj9RX3Uld1Y/UV90117pyiBZ1cnIy77//PlFRUfYORQghhHAodk/UhYWFjB8/ngULFjSrQRqEEEKIq8Hu41EnJCRw2223MXz4cP7+97/Xa93y8nJ27NhBYGAgWu2V/+YoKCgA4MSJE+Tn51/x9loyqav6kfqqO6mr+pH6qjtHqiuz2Ux2djYxMTE4OV06Fds1USclJbF9+3aSk5PrVN5oNNpcBLBt2zZuuummRo+rcqgzcXlSV/Uj9VV3Ulf1I/VVd45UV1u2bKFfv36XLGO3RJ2RkcHjjz/OqlWrcHFxqdM6c+bMYfbs2TXmb9myheDg4MYOUQghhGgSmZmZ9O/fn8DAwMuWtdtV30uXLmXMmDHodDrrPJPJhEajQavVYjQabZZBzRb1iRMn6NatGxkZGbRt2/aqxS6EEEJciePHjxMaGlqn/GW3FvWwYcNITU21mXf//ffTpUsXpk+fXiNJAxgMBgwGg/WzvfsYhBBCiKZmt0Tt6elJjx49bOa5u7vj5+dXY74QQghxrbL77VlCCCGEuDi7355V3bp16+wdghDiGmcymSgrK7N3GKKZc3Z2rrULtyEcKlHbU5GxnJ0ZuZSbFUM6+9s7HCHEVaaUIisri9zcXHuHIloIHx8fgoKC0Gg0V7QdSdQV1uzPYfLnO4hq6y2JWohrUGWSDggIwM3N7Yr/uIprl1KK8+fPk5OTA3DFtw9Loq4QE+oDwL7MfErKTLg4N84pCyGE4zOZTNYk7efnZ+9wRAvg6uoKQE5ODgEBAVd0GlwuJqvQ1tcVP3c9ZSbFnpNy25cQ15LKPmk3Nzc7RyJaksrv05Ve8yCJuoJGoyEmzAeAHenn7BuMEMIu5HS3aEyN9X2SRF1Nr4rT3ykZuXaNQwghhKgkibqaXqGWYTYlUQshrmXt2rVj3rx5dS6/bt06NBpNk18xv2jRInx8fJp0H45IEnU1UaHeaDRw/FwxpwuNl19BCCHsSKPRXHJKTExs0HaTk5N56KGH6lz++uuvJzMzE29v7wbtT1yaXPVdjZeLMx39PTiUU0hKei7Du11+VBMhhLCXzMxM6/vFixczc+ZM0tLSrPM8PDys75VSmEymy459DODvX79bVPV6PUFBQfVaR9SdtKgvIP3UQojmIigoyDp5e3uj0Wisn/fv34+npyc//vgjffr0wWAw8Msvv3D48GHuvPNOAgMD8fDwoF+/fqxevdpmuxee+tZoNHz44YeMGTMGNzc3IiIi+O6776zLLzz1XXmKesWKFXTt2hUPDw/i4uJsfliUl5czefJkfHx88PPzY/r06cTHxzN69Oh61cH8+fPp2LEjer2eyMhI/vvf/1qXKaVITEwkLCwMg8FASEgIkydPti7/17/+RUREBC4uLgQGBnL33XfXa99XiyTqC0iiFkJAxUMrSsvtMjXm6MPPPPMMc+fOZd++fURFRVFYWMitt97KmjVr2LFjB3FxcYwaNYr09PRLbmf27Nncc8897Nq1i1tvvZXx48dz9uzZi5Y/f/48r732Gv/973/ZsGED6enpTJs2zbr85Zdf5tNPP2XhwoVs3LiR/Px8li5dWq9jW7JkCY8//jhPPvkku3fv5v/+7/+4//77Wbt2LQBff/01//znP3n//fc5ePAgS5cupWfPngBs3bqVyZMn88ILL5CWlsby5csZMmRIvfZ/tcip7wtU3qK1MyMXs1mh1crtGkJci4rLTHSbucIu+977wgjc9I3z5/mFF17g5ptvtn5u1aoV0dHR1s8vvvgiS5Ys4bvvvmPSpEkX3c6ECRMYN24cAP/4xz9466232LJlC3FxcbWWLysr47333qNjx44ATJo0iRdeeMG6/O2332bGjBmMGTMGgHfeeYdly5bV69hee+01JkyYwKOPPgrA1KlT2bx5M6+99ho33ngj6enpBAUFMXz4cJydnQkLC6N///4ApKen4+7uzu23346npyfh4eHExMTUa/9Xi7SoLxAZ6Imrs44CYzmHTxXaOxwhhLgiffv2tflcWFjItGnT6Nq1Kz4+Pnh4eLBv377LtqijoqKs793d3fHy8rI+IrM2bm5u1iQNlsdoVpbPy8sjOzvbmjQBdDodffr0qdex7du3j4EDB9rMGzhwIPv27QPgj3/8I8XFxXTo0IEHH3yQJUuWUF5eDsDNN99MeHg4HTp04N577+XTTz/l/Pnz9dr/1SIt6gs46bT0bOPNlqNn2ZGRS0Sgp71DEkLYgauzjr0vjLDbvhuLu7u7zedp06axatUqXnvtNTp16oSrqyt33303paWll9yOs7OzzWeNRoPZbK5X+cY8pV8XoaGhpKWlsXr1alatWsWjjz7Kq6++yvr16/H09GT79u2sW7eOlStXMnPmTBITE0lOTna4W8CkRV2LXhWnv6WfWohrl0ajwU3vZJepKZ+QtnHjRiZMmMCYMWPo2bMnQUFBHD16tMn2Vxtvb28CAwNJTk62zjOZTGzfvr1e2+natSsbN260mbdx40a6detm/ezq6sqoUaN46623WLduHZs2bSI1NRUAJycnhg8fziuvvMKuXbs4evQoP/300xUcWdOQFnUtrBeUpefaNQ4hhGhsERERfPPNN4waNQqNRsPzzz9/yZZxU3nssceYM2cOnTp1okuXLrz99tucO3euXj9SnnrqKe655x5iYmIYPnw4//vf//jmm2+sV7EvWrQIk8nEgAEDcHNz45NPPsHV1ZXw8HC+//57fv/9d4YMGYKvry/Lli3DbDYTGRnZVIfcYJKoa1GZqNOyCyguNeGql5G0hBAtwxtvvMEDDzzA9ddfT+vWrZk+fTr5+Vd/IKLp06eTlZXFfffdh06n46GHHmLEiBH1GmVq9OjRvPnmm7z22ms8/vjjtG/fnoULFzJ06FDAMh703LlzmTp1KiaTiZ49e/K///0PPz8/fHx8+Oabb0hMTKSkpISIiAg+//xzunfv3kRH3HAadbU7DRrR8ePHCQ0NJSMjg7Zt217ZxsqNcGwjnD6E6v8gA/6xhpwCI1/8Xyz927dqnICFEA6ppKSEI0eO0L59e1xcXOwdzjXJbDbTtWtX7rnnHl588UV7h9MoLvW9qk/+kj7qSsW58N8x8OPTaEryqt1PLSNpCSFEYzt27BgLFizgwIEDpKam8sgjj3DkyBH+/Oc/2zs0hyOJupJnILTqACjI2EJMmGWAjh3STy2EEI1Oq9WyaNEi+vXrx8CBA0lNTWX16tV07drV3qE5HOmjri7sejj7O6T/Sq/2lvv55MpvIYRofKGhoTWu2Ba1kxZ1deGxltdjm4hq641WA5l5JWTnl9g3LiGEENcsSdTVhVUk6pPbcdeW07niYSdy+lsIIYS9SKKurlUH8AgEUymc2CYDdAghhLA7SdTVaTQQdp3lffqvcuW3EEIIu5NEfaGw6y2vxzZZHyWaejwPk7nZ3m4uhBCiGZNEfaHKC8oythDR2g13vY6iUhMHcwrsG5cQQohrkiTqCwX2AIMXlBagO7WHqLY+gFxQJoRouYYOHcqUKVOsn9u1a8e8efMuuY5Go2Hp0qVXvO/G2s6lJCYm0qtXrybdR1OSRH0hrQ5CK8ZIrXb6WwboEEI4mlGjRhEXF1frsp9//hmNRsOuXbvqvd3k5GQeeuihKw3PxsWSZWZmJiNHjmzUfbU0kqhrU3mbls0FZbl2C0cIIWozceJEVq1axfHjx2ssW7hwIX379iUqKqre2/X398fNza0xQrysoKAgDAbDVdlXcyWJujbtb4AON0L4QGIqEvWBnAIKjeX2jUsIIaq5/fbb8ff3Z9GiRTbzCwsL+fLLL5k4cSJnzpxh3LhxtGnTBjc3N3r27Mnnn39+ye1eeOr74MGDDBkyBBcXF7p168aqVatqrDN9+nQ6d+6Mm5sbHTp04Pnnn6esrAywDDc5e/Zsdu7ciUajQaPRWGO+8NR3amoqN910E66urvj5+fHQQw9RWFhoXT5hwgRGjx7Na6+9RnBwMH5+fiQkJFj3VRdms5kXXniBtm3bYjAY6NWrF8uXL7cuLy0tZdKkSQQHB+Pi4kJ4eDhz5swBQClFYmIiYWFhGAwGQkJCmDx5cp333RDyCNHahPaD+5YCEACEeLtwMq+EXcdzub5ja7uGJoS4ykqL6r+OzgC6ij+vpnIwGUGjBWfXy29X717n3Tg5OXHfffexaNEinn32WetYzl9++SUmk4lx48ZRWFhInz59mD59Ol5eXvzwww/ce++9dOzYkf79+192H2azmT/84Q8EBgby22+/kZeXZ9OfXcnT05NFixYREhJCamoqDz74IJ6enjz99NOMHTuW3bt3s3z5cutY0d7e3jW2UVRUxIgRI4iNjSU5OZmcnBz++te/MmnSJJsfI2vXriU4OJi1a9dy6NAhxo4dS69evXjwwQfrVG9vvvkmr7/+Ou+//z4xMTF89NFH3HHHHezZs4eIiAjeeustvvvuO7744gvCwsLIyMggIyMDgK+//pp//vOfJCUl0b17d7Kysti5c2ed9ttQkqjroFeYDydTs0jJkEQtxDXnHyH1X+ePi6D7GMv7/f+DLydA+CC4/4eqMvN6wvkzNddNzKvXrh544AFeffVV1q9fbx2HeeHChdx11114e3vj7e3NtGnTrOUfe+wxVqxYwRdffFGnRL169Wr279/PihUrCAmx1MU//vGPGv3Kzz33nPV9u3btmDZtGklJSTz99NO4urri4eGBk5MTQUFBF93XZ599RklJCR9//DHu7pYfLO+88w6jRo3i5ZdfJjAwEABfX1/eeecddDodXbp04bbbbmPNmjV1TtSvvfYa06dP509/+hMAL7/8MmvXrmXevHm8++67pKenExERwaBBg9BoNISHh1vXTU9PJygoiOHDh+Ps7ExYWFid6vFKyKnvSynMgRPbq/qp5YIyIYSD6dKlC9dffz0fffQRAIcOHeLnn39m4sSJAJhMJl588UV69uxJq1at8PDwYMWKFaSnp9dp+/v27SM0NNSapAFiY2NrlFu8eDEDBw4kKCgIDw8PnnvuuTrvo/q+oqOjrUkaYODAgZjNZtLS0qzzunfvjk6ns34ODg4mJyenTvvIz8/n5MmTDBw40Gb+wIED2bdvH2A5vZ6SkkJkZCSTJ09m5cqV1nJ//OMfKS4upkOHDjz44IMsWbKE8vKm7Ra1a4t6/vz5zJ8/n6NHjwKWyp85c6ZjXAF4ZAP8ZxT4tifmzp8A2JGRi1LKenpJCHEN+NvJ+q+jq3ZxVJdRlm1oLmgXTUm9sriqmThxIo899hjvvvsuCxcupGPHjtxwww0AvPrqq7z55pvMmzePnj174u7uzpQpUygtLW20/W/atInx48cze/ZsRowYgbe3N0lJSbz++uuNto/qnJ2dbT5rNBrMZnOjbb93794cOXKEH3/8kdWrV3PPPfcwfPhwvvrqK0JDQ0lLS2P16tWsWrWKRx991HpG48K4GotdW9Rt27Zl7ty5bNu2ja1bt3LTTTdx5513smfPHnuGZREcDRod6N3p0doJnVbDqQIjJ/NkJC0hril69/pPumptIJ2TZV71/ulLbbcB7rnnHrRaLZ999hkff/wxDzzwgLVBsXHjRu68807+8pe/EB0dTYcOHThw4ECdt921a1cyMjLIzMy0ztu8ebNNmV9//ZXw8HCeffZZ+vbtS0REBMeOHbM9XL0ek8l02X3t3LmToqKq/vuNGzei1WqJjIysc8yX4uXlRUhISI0hNjdu3Ei3bt1syo0dO5YFCxawePFivv76a86ePQuAq6sro0aN4q233mLdunVs2rSJ1NTG++F1Ibu2qEeNGmXz+aWXXmL+/Pls3ryZ7t272ymqCi7e8MwxMHjiCnQJ8mTPyXxS0nNp4+N62dWFEOJq8fDwYOzYscyYMYP8/HwmTJhgXRYREcFXX33Fr7/+iq+vL2+88QbZ2dk2SelShg8fTufOnYmPj+fVV18lPz+fZ5991qZMREQE6enpJCUl0a9fP3744QeWLFliU6Zdu3YcOXKElJQU2rZti6enZ43bssaPH8+sWbOIj48nMTGRU6dO8dhjj3Hvvfda+6cbw1NPPcWsWbPo2LEjvXr1YuHChaSkpPDpp58C8MYbbxAcHExMTAxarZYvv/ySoKAgfHx8WLRoESaTiQEDBuDm5sYnn3yCq6urTT92Y3OYPmqTyURSUhJFRUW19n8AGI1G8vPzrVNBQRM/1tPgaX0rA3QIIRzZxIkTOXfuHCNGjLDpT37uuefo3bs3I0aMYOjQoQQFBTF69Og6b1er1bJkyRKKi4vp378/f/3rX3nppZdsytxxxx088cQTTJo0iV69evHrr7/y/PPP25S56667iIuL48Ybb8Tf37/WW8Tc3NxYsWIFZ8+epV+/ftx9990MGzaMd955p36VcRmTJ09m6tSpPPnkk/Ts2ZPly5fz3XffERERAViuYH/llVfo27cv/fr14+jRoyxbtgytVouPjw8LFixg4MCBREVFsXr1av73v//h5+fXqDFWp1FK2XW0idTUVGJjYykpKcHDw4PPPvuMW2+9tdayiYmJzJ49u8b8jIwM2rZt23RBmsr4ckcWT321i37tfPny4eubbl9CiKuupKSEI0eO0L59e1xcXOwdjmghLvW9On78OKGhoXXKX3ZvUUdGRpKSksJvv/3GI488Qnx8PHv37q217IwZM8jLy7NOFyvXaMqK4aORMDeMPoGW/p7UE3mUmRrvogUhhBDiUux+H7Ver6dTp04A9OnTh+TkZN58803ef//9GmUNBoNNn0Z+fn7TBufsCgWZUHaeduf34OniREFJOWlZBfRoU/NmfSGEEKKx2b1FfSGz2YzRaLR3GFXCLae5tRmb5LnfQgghrjq7JuoZM2awYcMGjh49SmpqKjNmzGDdunWMHz/enmHZqhyg45gkaiGEEFefXU995+TkcN9995GZmYm3tzdRUVGsWLGCm2++2Z5h2apoUXNyO70HWG7L2pEuV34LIYS4OuyaqP/973/bc/d106oDuAdAUQ69nY4AcPhUEXnFZXi7Ns1TaIQQ9tGYT7cSorG+T3a/mMzhaTQQHgt7v8U7J5nQVr3JOFvMruO5DI7wt3d0QohGoNfr0Wq1nDx5En9/f/R6vTwqWDSYUorS0lJOnTqFVqtFr9df0fYkUddF2PWw91tI30Sv0GFknC0mJV0StRAthVarpX379mRmZnLyZAOe7S1ELdzc3AgLC0OrvbLLwSRR10V4xQVlGVuIGeTJ/3bKBWVCtDR6vZ6wsDDKy8sv+0xqIS5Hp9Ph5OTUKGdmJFHXRWAPMHiBMZ/r3C0Ppk+RkbSEaHE0Gg3Ozs5NNgqSEA3hcPdROyStDkItA4NHlKTirNNwpqiU4+eK7RyYEEKIlk4SdV1V3E/tfHwz3YK9AMv41EIIIURTkkRdV5UPPjmxw/rgE7mfWgghRFOTRF1XbfrA/T/CpGR6hfkAckGZEEKIpicXk9WVs4v1KWW9Qn0B2HMyn9JyM3on+b0jhBCiaUiGaYB2fm74uDlTWm5mX2YTj+AlhBDimiaJuj4KsuGHaWg+H0d0Wx9ATn8LIYRoWpKo68PJAMkfwoEfGRhYDkiiFkII0bSkj7o+XH1g2Exo1YEuhMDP5yRRCyGEaFKSqOtr8FQAos6XAns4crqIc0Wl+Lpf2UPXhRBCiNrIqe8G8nHT0761OwApx3PtG4wQQogWSxJ1fSkFR3+B9a9yXYjlhERKeq59YxJCCNFiSaKuL40Gvp0Ea//OMPejgFxQJoQQoulIom6Iigef9DTvBWDncctIWkIIIURjk0TdEBXP/fY/uw29k5bc82UcPXPezkEJIYRoiSRRN0RFi1p7cjsxwS4ApGTIAB1CCCEanyTqhmjVAdwDwFTKSN9MQC4oE0II0TQkUTeERgPhltPfA5zSABmbWgghRNOQRN1QFf3U7Yp2ArAvM5+SMpM9IxJCCNECSaJuqIpE7ZK1DX83HWUmxZ6TMpKWEEKIxiWJuqGCeoLeE40xn9sDzwJyP7UQQojGJ4m6obQ6CO0PwFDXw4AkaiGEEI1PEvWVqLigrFv5HkBu0RJCCNH4JFFfiTDL/dR+Z7ah0SgyzhZzptBo56CEEEK0JJKor0SbPtD+BrR9JtCldeWDT3LtG5MQQogWRRL1lXB2gfjv4KZn6R7mD8AOefCJEEKIRtSgRJ2RkcHx48etn7ds2cKUKVP44IMPGi2w5qZXqA8gLWohhBCNq0GJ+s9//jNr164FICsri5tvvpktW7bw7LPP8sILLzRqgM3C+bMMwvLgk50ZuZjNMpKWEEKIxtGgRL17927697fcmvTFF1/Qo0cPfv31Vz799FMWLVrUmPE5PmMhvNqJdsvvpa1zPgXGcn4/XWjvqIQQQrQQDUrUZWVlGAwGAFavXs0dd9wBQJcuXcjMzKzzdubMmUO/fv3w9PQkICCA0aNHk5aW1pCQ7MfgAQHdoHVnBgaUAtJPLYQQovE0KFF3796d9957j59//plVq1YRFxcHwMmTJ/Hz86vzdtavX09CQgKbN29m1apVlJWVccstt1BUVNSQsOznr6thUjLeHS1nGaSfWgghRGNxashKL7/8MmPGjOHVV18lPj6e6OhoAL777jvrKfG6WL58uc3nRYsWERAQwLZt2xgyZEhDQrMPZ8utWXJBmRBCiMbWoEQ9dOhQTp8+TX5+Pr6+vtb5Dz30EG5ubg0OJi8vD4BWrVo1eBv21CvEHSfK2Z9VQHGpCVe9zt4hCSGEaOYadOq7uLgYo9FoTdLHjh1j3rx5pKWlERAQ0KBAzGYzU6ZMYeDAgfTo0aPWMkajkfz8fOtUUFDQoH01iaWPEvxeJLe5p2EyK1JP5Nk7IiGEEC1AgxL1nXfeyccffwxAbm4uAwYM4PXXX2f06NHMnz+/QYEkJCSwe/dukpKSLlpmzpw5eHt7W6du3bo1aF9NRVN2nhGevwPy3G8hhBCNo0GJevv27QwePBiAr776isDAQI4dO8bHH3/MW2+9Ve/tTZo0ie+//561a9fStm3bi5abMWMGeXl51mnv3r0NCb9phF0HQC+1D5B+aiGEEI2jQX3U58+fx9PTE4CVK1fyhz/8Aa1Wy3XXXcexY8fqvB2lFI899hhLlixh3bp1tG/f/pLlDQaD9bYwgPz8/IaE3zQqBugIKtiLgVJS5BYtIYQQjaBBLepOnTqxdOlSMjIyWLFiBbfccgsAOTk5eHl51Xk7CQkJfPLJJ3z22Wd4enqSlZVFVlYWxcXFDQnLvvw6grs/WnMp0drfOZlXQk5+ib2jEkII0cw1KFHPnDmTadOm0a5dO/r3709srGVc5pUrVxITE1Pn7cyfP5+8vDyGDh1KcHCwdVq8eHFDwrIvjQbCLPUw0usoADvk9LcQQogr1KBT33fffTeDBg0iMzPTeg81wLBhwxgzZkydt6NUC3smdvj1sO87Yp0OAHGkZOQyonuQvaMSQgjRjDUoUQMEBQURFBRkHUWrbdu29XrYSYtU0aLuULIbLWbppxZCCHHFGnTq22w288ILL+Dt7U14eDjh4eH4+Pjw4osvYjabGzvG5iOoJ+g90ZcX0kWTzq7juZhkJC0hhBBXoEEt6meffZZ///vfzJ07l4EDBwLwyy+/kJiYSElJCS+99FKjBtlsaHUQ2h8Or2Gg8wH2lrbjYE4BXYLqfoGdEEIIUV2DEvV//vMfPvzwQ+uoWQBRUVG0adOGRx999NpN1ADhsXB4DTe5HWZBKaSk50qiFkII0WANOvV99uxZunTpUmN+ly5dOHv27BUH1axV9FP3NO0FlDz4RAghxBVpUKKOjo7mnXfeqTH/nXfeISoq6oqDatba9AGtM27leQRxVhK1EEKIK9KgU9+vvPIKt912G6tXr7beQ71p0yYyMjJYtmxZowbY7Di7wsSVnHYJJ+vVzeRkF1BkLMfd0OAL7IUQQlzDGtSivuGGGzhw4ABjxowhNzeX3Nxc/vCHP7Bnzx7++9//NnaMzU+b3gT4+RHi7YJZwa7jMpKWEEKIhmlwMy8kJKTGRWM7d+7k3//+Nx988MEVB9YS9Arz4WRqFikZucR29LN3OEIIIZqhBrWoxWUoBSueZXZWAv6cY0e6DHkphBCiYSRRNwWNBn5fh3/BPvpqD5CSkdvyHpcqhBDiqpArnJrK4KmUlpWz9UvFqQIjmXklhPi42jsqIYQQzUy9EvUf/vCHSy7Pzc29klhalh53oQf8N/zMqcx8UjJyJVELIYSot3olam9v78suv++++64ooJYmJsyHvRWJ+taewfYORwghRDNTr0S9cOHCpoqjZTqZwljjUnZp/EhJb2XvaIQQQjRDcjFZU/rtfaL2/5MRumR2ncil3HQNjywmhBCiQSRRN6Vwy1PbYp0OUFJmZn9WgZ0DEkII0dxIom5KYdcD0JND6CmT534LIYSoN0nUTcmvI7j7o6eMKM1hSdRCCCHqTRJ1U9JorMNe9temSaIWQghRb5Kom1q45fR3P+1+Dp8qJL+kzM4BCSGEaE4kUTe1ihZ1P91BNMrMrgwZSUsIIUTdSaJuaoE9QO+BB+fpokknJUMG6BBCCFF3kqibms4JQvsD0E/6qYUQQtSTJOqrIayynzqNHekykpYQQoi6k0R9NVQ8+KSfdj9niowcP1ds54CEEEI0F5Kor4Y2fUDrTKAmlzBNDjvk9LcQQog6kkR9NTi7Qu/7+DngL5QpJ1LSc+0dkRBCiGaiXqNniStw+xuc3nGczPSdcuW3EEKIOpMW9VXUK9QXgN0n8yktl5G0hBBCXJ4k6quonXsZt7nuxqU8n/1Z+fYORwghRDMgifoq0vzndt5V/2Cgdo/cTy2EEKJOJFFfTaHXkevSFj1l7JALyoQQQtSBXRP1hg0bGDVqFCEhIWg0GpYuXWrPcJpe3Fx2jFnLt+ZB0qIWQghRJ3ZN1EVFRURHR/Puu+/aM4yrR+dEr7Y+ABw5XUTu+VL7xiOEEMLh2fX2rJEjRzJy5Eh7hnDV+brr6dDKQNbZPFIychkaGWDvkIQQQjgw6aO+2n59m2Ul9/Ko07dy+lsIIcRlNasHnhiNRoxGo/VzQUGBHaNpIBdvXMzn6adNY74kaiGEEJfRrFrUc+bMwdvb2zp169bN3iHVX8VIWr00h9mbfkpG0hJCCHFJzSpRz5gxg7y8POu0d+9ee4dUf34dUe7+GDRlhJXs59iZ8/aOSAghhANrVonaYDDg5eVlnTw9Pe0dUv1pNGjCrgOgvzaNHfLcbyGEEJdg10RdWFhISkoKKSkpABw5coSUlBTS09PtGVbTqzj93U+7X0bSEkIIcUl2TdRbt24lJiaGmJgYAKZOnUpMTAwzZ860Z1hNLzwWgD7ag+xKP2PnYIQQQjgyu171PXTo0GvzYqrAnpid3fEqK6I8ay8lZYNwcdbZOyohhBAOqFn1UbcYOic0YQMAiGEfezNlJC0hhBC1k0RtJ5qKfur+2jTppxZCCHFRkqjtpaKf2nJBmVz5LYQQonaSqO2lTR/MWmcCNbnkpO+3dzRCCCEclCRqe3F2xRwcg0lp8Mo/wJlC4+XXEUIIcc2RRG1HTn94j9Gen7HS3E8G6BBCCFErSdT25NeRzmFtACRRCyGEqJUkajuLCfMBJFELIYSonSRqO7spbwlf62fRKmMlZvM1+PAXIYQQlySJ2s6CyjLooz1Ir/JUfj9dZO9whBBCOBi7PkJUgLbXn3nnoBefZbfDMyOXTgEe9g5JCCGEA5EWtb217UNe5z9yktbskAefCCGEuIAkagfQK9QXkAvKhBBC1CSJ2gH08TrHRN0PROQsp7jUZO9whBBCOBBJ1A4g8Ewyzzt/yjjtanafzLN3OEIIIRyIJGoHoAm3jKTVS3OYXUdz7ByNEEIIRyKJ2hH4deK8cysMmjJyD/1m72iEEEI4EEnUjkCjoTi4HwBuWVvsHIwQQghHIonaQXhEDAEg0ribnPwSO0cjhBDCUUiidhCGjgMB6Ks9wLajp+0cjRBCCEchidpRBPbEqHXFS3Oefy3+H49+uo11aTmY5PnfQghxTZNHiDoKnRPlbfpjyFjPK7p32bJvDcv3hvOJWye69x3C3f3aE9rKzd5RCiGEuMokUTsQ9+jRkLGertoMumozLDPLoPvaf/PWuqMM7NiahLB0+rRxQ9/uOnD3s2u8Qgghmp4kakfS9wEIHwiZOyErFVNWKnnnzhLjHsovh07zy6HTJBx7A71uL0vCn6NL3P/RNdgLzhyGYxshsAcEdAVnV3sfiRBCiEYiidrR+Edapqh70AGtgE+AjLPn+XLbcY5vCmdfeQHvH/Bgf9rPRLX15nn/n+m3b65lfY0W/CIgqAcEdofAnpb3nsGg0djxwIQQQjSEJOpmIrSVG1Nv7oxp2Of8fPAUHbZmcHhvNruO57Ho5HnKnHsQ5ZSBhykPTqdZpt1fV23AtZUlcQf1tLS8g6MtCVwIIYRDk0TdzOi0GoZGBjA0MoAzhUaW7DjB4mQP/pxzHRgVAeQy1CeHMSFniXE+jsvZfXD6IBSfhaM/WyaA0Otg4oqqDW9dCD6hllPvcupcCCEchiTqZszPw8BfB3dg4qD27MjI5YvkDP6304kvcn35IteS1G+MDGDckABu8D2N06m9kLUbsndD235VGyotgu+fABRMO1iVqI9uhPJiaNMHXH3tcYhCCHHNk0TdAmg0GnqH+dI7zJfnb+/GD7syWbw1g23HzrF6Xzar92UT4Gngrj59uaffGNq3drfdgLEQuo+G/EzwCKiav3EeHFxped+qoyVhV05BPcHZ5WodohBCXLM0Sqlm+0SN48ePExoaSkZGBm3btrV3OA7nUE4Bi5Mz+Gb7Cc4UlVrn92/firF9Q7m1ZzCuet3FN7DsKTi0Gs7+XnOZ1tnSx109eftFgFaeodNg589Czj7I2Wt5LcgC7zbg2w5824NfR8uFhkKIZq8++UsS9TWgtNzMT/uzWZycwfoDp6h82JmnwYk7eoUwtl8oPdt4o7nYVeHnz8LJ7XB8G5yomM7X8phTgxfEzYWY8ZbPSsmV5hdTeUtdzv6qxFyYdel1gqPh/zZUfV75vKWbou8D4BnUtPG2RGYzlJ23dP2UFoJWZ/lRVGnnYijJg+ix4OJtmVecC8ps6Qq6Fr7bZjPk7IHCbGjbH1y8LPPLS0HnfG3UQROpT/6SU9/XAL2TlrgewcT1CCYzr5ivth7ni20ZZJwt5tPf0vn0t3S6BHnyp36hjI5pg4+b3nYDbq2g03DLBKAUKvcYZelbURlb0WRuxyl7J1pjPkdL3Dh19CwlZSY8j64icvtsMoJHkBz5JCVlZozlJuursdrnkjITxvKqV29XZzoHetIlyJPOgZ50DHDH4HSJ1r8j27PUcm98v79aWsgAe5bATy/WLOsTBgHdLPfDe7WB/BNw7iicPWJ7lb7ZDL+9DyYjRI+rmv/z67Dri4pWeDtLS9z6Prz5XihoKrMk09Ii0OjAK9gy32yGPd9Y5keNreqO2fUlHP6pah3rVO1zWZHtPtoNhgnfV31e/ozlIsz2Q6oSdfIC+OnvoNODRxB4Blp+JHkEWV4rJ48gyy2Rbq0cL5mVFlnO1hTmWBKwzZQDQVEw7HlLWWWG9wYDCh7fVZWo1/4dfn0HXH0sd5S4+lqO1fre1/JafZl7QNW/m6gXSdTXmGBvVx4bFkHCjZ3Y9PsZFidnsHxPFvuzCkj8317+8eN++rdrBWCTPEsqEmvlPGO5GXAFBgOD0WGis+Y4x741c55NADzptIJeTlnsOHCMZ/fuBsCJcpboZ7LPHM5O1ZFd5o6kqVDKa/kq/rQ/x/pep9XQvrU7kYGWxB0ZZJnCWrmh09r5D6GpHM4dqWoZGwtgxEtVy39+HbJ2WboHKhN1m97QYWhVUg7oZjmtbfCs2z7NZXDjDEsS9w6tmp+zH05VTLXxDK6WuNtZEnlgN8s1BxcoN5lRgLOunt0ZSllaqspcdTzlpfD7Okui7D6mKnnt+gIyfrMkD2NBVTI1VibUinmmqq4buo6CsZ9Y3ms08M1DoEwQcQs4VySCE9tg52d1DFgDeg9wuuCai85xluNwMlTNK8m3vJpKIS/dMl2KTg/tBsG9S6rmbf3I0nXU5TZLAmsMSlUl3sAeVV1QKZ/BgRUVyyqSc2nhpbdVXm30Pp2T5XupdbKNtficpc7Pn7FMdXHhnSaLbgezCUb/C1q1t8zLSLZc7OrWqmaib64/MhuBnPoW5J4v5duUkyQlZ7AvM7/e62s14OKsw8VZh8FJa331ciqjm/kw5XoPctw6Y3DW0aHsEE/8/leb9cu1Bs56dSHXpyf5flGc948m3ezPvuzzHMguYH9WAQUl5bXu28VZS0SAZ1XrO8iTyEBPAr0MFz+V31BKQV6GbT9yzl44dcDSsq2k08PfTlpODQJseM3SMo6515Kgm1LecTh9wNICP3e0YjoCZ49akl4tMgKH8XXEXLLzjeTkFXN/ZiLHyn2ZWzyaQtwI8DRwt8tWujudwM+5FB+nUry0Rtw1JbiqYpzKz6OpbKkaCysSgYJef4HR71p2YiyEORU/Uv6WCfqK59YveaQeCRVLgouMq0rUAJ+NtTzo5/Z5lhYuWH4UZO4EZzdLEta7V0y1vHd2rV+rt9xoSYgF2VCQWfE+84LPWVXdQx1vsk3Uc8OhJBce3Wz5kQbwyzzYsqCihR4MHhWvlZ+dXaHolGUflS3f1p1g0BOW9c0meLG15cfRkweq6mH5DNj8r5rH4Oxm2YdHoOUCUs8gy6tHoOXC0XYDL18H589aEnZxxWutn3OrPocOgHv+Y1lfKfh7oOX/zZRUy5kksHTn/PpW7ft0crH88NN7gMED9J6W14BucPPsqnLb/mP5EdH1DnBvbZlXdNoSi8Gj6t/dzmc6mt2p73fffZdXX32VrKwsoqOjefvtt+nfv7+9w7pm+Ljpib++HffFhrPnZD57T+ajd9Li4qzF4KTDUPHqUu21elJ20moukRSH2X40doKjSVV93Se24VSSR0DuTgJyd8LRamW1TqAzoFz1ZE3aRtoZEweyCwjf9SbtcjfxgfEWvi67ntQTeeSfTGOw01dkK2fScQYnA57u7nh5uOPr5YGftxf+Pl64urpaWkg6veU/foehVUkjP9PSJ+nuX/Uc9VNpsOmdiqS87+KtEWc38O9S1UI2lVUl6iHTGvYPU09KKfKdA8n28Cbb3JMsTQk5zkayXUrIdivmfN5p9PnH8Cg+ThuVTZgmh3BNDj8fb82/jh0EoDV5DHLZSKzSkKjuQQHZ+UaiilcxQre1XvGcOXuGojPnCfJ2Qa93h+Belj+SplKgos673Gq5f9+aOCv/CFdPqtVenfQ1d/TnxTXndRhqmZqCk8GSWCqTy8WUl0JRjuW7UMlshq63WxK5Z7XTwHkZkH/cMtVVu8FViVqrsyRZs8nyHa5M1F1uA5/wqiRcmZD1HleWqJwMltPYV3Iqe/yXlgTuEVg1zz8SIm+tmfTN5ZaWfnmJ5QdLdcYLfoD+9HdLvbftV5Woty20zLfSVPuuVXut/t4nDAZPrVrl8E+WHzG+4Q0/5gaye4t68eLF3Hfffbz33nsMGDCAefPm8eWXX5KWlkZAQMAl15UWdQtgNluuKq+WuMnaZXuqE+D5M5bTcABfPQC7v8Y8Yg5HO93HgewCCtJ+4Y+pf625/cvYN24L7Tt0wsVZBz8+A7/Nh0FTYfgsS4GTO+CDoVUraJ2hdeeK09VdqxKzT3iTXvFeZCwnO7+E7HxjxWvF+4IScqrNt3RJXJ5GA37uBgK9DAR6uRDoZSDA04VQt1J6nl2JDwVohz4DQGZeMbrtC9Hm7OFcuYHTpc7kGJ3ILHYiu8SJQlw4r1wowpUiDBQpV4pwoRg9Ci0aDfh7GAjxcaWNjyshPi6E+LhW++yKr5tz458BaU4KT0HuMUsCr94qL8iynLIuPV+RbAMs/d8eAZbvYbc7qrZhNrfMuy6UsiTj4nMV3SOVXSMFlldXX8uPn0rfTba0oG97veqHxIbXYOOblnVV3f6PENANHt1U9fntvjDwceh9b6McVrO66nvAgAH069ePd955BwCz2UxoaCiPPfYYzzzzzCXXlUTdQpnKLH2BJqPlFJup1Pa2pMxdllPJ/pHQqoNlXt4J2LsUyksoLy3hXEEheQWF5BUUUVRURHHxecrLStBTjoEyDJoy9JQTX/o0BRoP2vm5M93pc24o/JFjkQ+gHfIkBictJuN5vLa9zXnvzpz36UyRZzhlyolyk6LcbLa+lpkUJrOizGSm3Kwsk8myvMxsxmRSlFXOM6sL1qssV325meIyEzkFRnLyjRQaaz/1XxtfN2cCvVwI8HIh0LNaIvZysb5v7WGof99zLUrKTGTllXAyt5gTucWczLW8P5lX+bmYkrLL/2F0cdZWJW5vV9r4WhJ4iLcL7gYntBoNGg1oNRq0WtBpLGdxtJXzKpdrNegq5tdYXrGezbYqXjUV5a+UUgqzqvaKQilLrjErhaLitaLMhfNRWNczVysDlh9XOq3Gejzais8ajaZiftWxVn5uLj9+qteFWVXVkeVzVV1ouPS/o/Zy16tUXj9R2UVjk/gv+OzWCgb8X9W6X9wHfe6Hjjc2yjE3m0RdWlqKm5sbX331FaNHj7bOj4+PJzc3l2+//damvNFoxGis6gs8ceIE3bp1k0Qt6iS/pIyD2QWkZRVW9H3nk5ZVwLnzZZdf2QF4GpwIsLaAXSzvPauSb6CXC/6eBsvZAQehlOJsUSknc0usibsqkVuS+qkC4+U3dBXUSPrVEkD1BFuZWC9MtI54tU/lcegqjqky0V8s6Vf+EKptHbhEIjVX/Tipnlgvl3irL2sstf07Wn7QXPjjzPa4a0v6F5bVajQ8fENHbou68qvXm00f9enTpzGZTAQGBtrMDwwMZP/+mletzpkzh9mzZ9eYL0RdeLk40ye8FX3Cq65eVUpxurCUtKwC0rILOJBVwP7sAn7PKcSsFE46Lc46y390J23Ve2edFiedZZ6TVoOTrmKeVoOuopyTTouztmJd3QXrVsyzrqut3F7VfIOzjoCKFnGApwF3g0NcUlIvGo0GPw8Dfh4Gerb1rrWMsdzSKrdpkefatsgr/6iblLJNCGbLH3lTLQnAZK7fX//KbYL9Mm71MwEaKlv6VfEpZTlzU9dDq6wbkx2P6Wpr6n/Hs+dLL1+okTWr//kzZsxg6tSqzv3KFrUQDaXRaPD3NODvaWBQRGt7h3NNMjjpCPdzJ9zP/fKF68m2lVeR1M21t+rM5pplzRWJsXqLCyyn2DVUP21uSaw1Eq0Wm3LW9at91kCDTr2bK47DdEGsZlVzWeUx2x5/1Q+aylZx1fyayyqPs3orvNbuh2qt8qoytt0Ml9yG9sKWb1U9ATbHa22Rm6v/WLtgebV/b3O1H3rV6+TC5SZzze9B5bYiAj0a7ftZV3ZN1K1bt0an05GdnW0zPzs7m6Cgmk9aMhgMGAxV9zTm59f/ViIhxLVDo9Gg04CO+iXB5kCr1aBF07xaW43gWjteALteIqjX6+nTpw9r1qyxzjObzaxZs4bY2Fg7RiaEEEI4Brv/OJk6dSrx8fH07duX/v37M2/ePIqKirj//vvtHZoQQghhd3ZP1GPHjuXUqVPMnDmTrKwsevXqxfLly2tcYCaEEEJci+yeqAEmTZrEpEmT7B2GEEII4XBa4GNshBBCiJbDIVrUDWU2W554lJmZaedIhBBCiLqrzFuVeexSmnWirrytSwbwEEII0RxlZ2cTFnbpAV7s/qzvK1FeXs6OHTsIDAxE2wgPoy8oKKBbt27s3bsXT886jgsspN6ugNRdw0i9NZzUXcM0dr2ZzWays7OJiYnByenSbeZmnagbW35+Pt7e3uTl5eHl5WXvcJoNqbeGk7prGKm3hpO6axh71ptcTCaEEEI4MEnUQgghhAOTRF2NwWBg1qxZNs8TF5cn9dZwUncNI/XWcFJ3DWPPepM+aiGEEMKBSYtaCCGEcGCSqIUQQggHJolaCCGEcGCSqCu8++67tGvXDhcXFwYMGMCWLVvsHZLD27BhA6NGjSIkJASNRsPSpUvtHVKzMGfOHPr164enpycBAQGMHj2atLQ0e4fVLMyfP5+oqCi8vLzw8vIiNjaWH3/80d5hNTtz585Fo9EwZcoUe4fi8BITE9FoNDZTly5drmoMkqiBxYsXM3XqVGbNmsX27duJjo5mxIgR5OTk2Ds0h1ZUVER0dDTvvvuuvUNpVtavX09CQgKbN29m1apVlJWVccstt1BUVGTv0Bxe27ZtmTt3Ltu2bWPr1q3cdNNN3HnnnezZs8feoTUbycnJvP/++0RFRdk7lGaje/fuZGZmWqdffvnl6gaghOrfv79KSEiwfjaZTCokJETNmTPHjlE1L4BasmSJvcNolnJychSg1q9fb+9QmiVfX1/14Ycf2juMZqGgoEBFRESoVatWqRtuuEE9/vjj9g7J4c2aNUtFR0fbNYZrvkVdWlrKtm3bGD58uHWeVqtl+PDhbNq0yY6RiWtFXl4eAK1atbJzJM2LyWQiKSmJoqIiYmNj7R1Os5CQkMBtt91m8/dOXN7BgwcJCQmhQ4cOjB8/nvT09Ku6/2Y9elZjOH36NCaTicDAQJv5gYGB7N+/305RiWuF2WxmypQpDBw4kB49etg7nGYhNTWV2NhYSkpK8PDwYMmSJXTr1s3eYTm8pKQktm/fTnJysr1DaVYGDBjAokWLiIyMJDMzk9mzZzN48GB279591QY1ueYTtRD2lJCQwO7du69+n1czFhkZSUpKCnl5eXz11VfEx8ezfv16SdaXkJGRweOPP86qVatwcXGxdzjNysiRI63vo6KiGDBgAOHh4XzxxRdMnDjxqsRwzSfq1q1bo9PprGNbV8rOziYoKMhOUYlrwaRJk/j+++/ZsGEDbdu2tXc4zYZer6dTp04A9OnTh+TkZN58803ef/99O0fmuLZt20ZOTg69e/e2zjOZTGzYsIF33nkHo9GITqezY4TNh4+PD507d+bQoUNXbZ/XfB+1Xq+nT58+rFmzxjrPbDazZs0a6fcSTUIpxaRJk1iyZAk//fQT7du3t3dIzZrZbMZoNNo7DIc2bNgwUlNTSUlJsU59+/Zl/PjxpKSkSJKuh8LCQg4fPkxwcPBV2+c136IGmDp1KvHx8fTt25f+/fszb948ioqKuP/+++0dmkMrLCy0+VV55MgRUlJSaNWqFWFhYXaMzLElJCTw2Wef8e233+Lp6UlWVhYA3t7euLq62jk6xzZjxgxGjhxJWFgYBQUFfPbZZ6xbt44VK1bYOzSH5unpWeMaCHd3d/z8/OTaiMuYNm0ao0aNIjw8nJMnTzJr1ix0Oh3jxo27ajFIogbGjh3LqVOnmDlzJllZWfTq1Yvly5fXuMBM2Nq6dSs33nij9fPUqVMBiI+PZ9GiRXaKyvHNnz8fgKFDh9rMX7hwIRMmTLj6ATUjOTk53HfffWRmZuLt7U1UVBQrVqzg5ptvtndoooU6fvw448aN48yZM/j7+zNo0CA2b96Mv7//VYtBRs8SQgghHNg130cthBBCODJJ1EIIIYQDk0QthBBCODBJ1EIIIYQDk0QthBBCODBJ1EIIIYQDk0QthBBCODBJ1EIIIYQDk0QthLhiGo2GpUuX2jsMIVokSdRCNHMTJkxAo9HUmOLi4uwdmhCiEcizvoVoAeLi4li4cKHNPIPBYKdohBCNSVrUQrQABoOBoKAgm8nX1xewnJaeP38+I0eOxNXVlQ4dOvDVV1/ZrJ+amspNN92Eq6srfn5+PPTQQxQWFtqU+eijj+jevTsGg4Hg4GAmTZpks/z06dOMGTMGNzc3IiIi+O6776zLzp07x/jx4/H398fV1ZWIiIgaPyyEELWTRC3ENeD555/nrrvuYufOnYwfP54//elP7Nu3D4CioiJGjBiBr68vycnJfPnll6xevdomEc+fP5+EhAQeeughUlNT+e677+jUqZPNPmbPns0999zDrl27uPXWWxk/fjxnz5617n/v3r38+OOP7Nu3j/nz59O6deurVwFCNGdKCNGsxcfHK51Op9zd3W2ml156SSmlFKAefvhhm3UGDBigHnnkEaWUUh988IHy9fVVhYWF1uU//PCD0mq1KisrSymlVEhIiHr22WcvGgOgnnvuOevnwsJCBagff/xRKaXUqFGj1P333984ByzENUb6qIVoAW688UbrONeVWrVqZX0fGxtrsyw2NpaUlBQA9u3bR3R0NO7u7tblAwcOxGw2k5aWhkaj4eTJkwwbNuySMURFRVnfu7u74+XlRU5ODgCPPPIId911F9u3b+eWW25h9OjRXH/99Q06ViGuNZKohWgB3N3da5yKbiyurq51Kufs7GzzWaPRYDabARg5ciTHjh1j2bJlrFq1imHDhpGQkMBrr73W6PEK0dJIH7UQ14DNmzfX+Ny1a1cAunbtys6dOykqKrIu37hxI1qtlsjISDw9PWnXrh1r1qy5ohj8/f2Jj4/nk08+Yd68eXzwwQdXtD0hrhXSohaiBTAajWRlZdnMc3Jysl6w9eWXX9K3b18GDRrEp59+ypYtW/j3v/8NwPjx45k1axbx8fEkJiZy6tQpHnvsMe69914CAwMBSExM5OGHHyYgIICRI0dSUFDAxo0beeyxx+oU38yZM+nTpw/du3fHaDTy/fffW38oCCEuTRK1EC3A8uXLCQ4OtpkXGRnJ/v37AcsV2UlJSTz66KMEBwfz+eef061bNwDc3NxYsWIFjz/+OP369cPNzY277rqLN954w7qt+Ph4SkpK+Oc//8m0adNo3bo1d999d53j0+v1zJgxg6NHj+Lq6srgwYNJSkpqhCMXouXTKKWUvYMQQjQdjUbDkiVLGD16tL1DEUI0gPRRCyGEEA5MErUQQgjhwKSPWogWTnq3hGjepEUthBBCODBJ1EIIIYQDk0QthBBCODBJ1EIIIYQDk0QthBBCODBJ1EIIIYQDk0QthBBCODBJ1EIIIYQDk0QthBBCOLD/B807PGgIup6kAAAAAElFTkSuQmCC", "text/plain": [ "
" ] @@ -1346,21 +1424,23 @@ { "cell_type": "markdown", "id": "aa074723-e3f7-4f7e-a267-855531a037dc", - "metadata": {}, + "metadata": { + "id": "aa074723-e3f7-4f7e-a267-855531a037dc" + }, "source": [ "- Note that we previously calculated the accuracy values on 5 batches only via the `eval_iter=5` setting; below, we calculate the accuracies on the full dataset" ] }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 21, "id": "1D2awlEq0gZi", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "1D2awlEq0gZi", - "outputId": "b482af19-5ebd-45b9-a9f0-99f621203ef9" + "outputId": "d603eda1-d912-43eb-ec9c-af6a622510a0" }, "outputs": [ { @@ -1369,7 +1449,7 @@ "text": [ "Training accuracy: 100.00%\n", "Validation accuracy: 96.64%\n", - "Test accuracy: 98.00%\n" + "Test accuracy: 97.33%\n" ] } ], @@ -1388,7 +1468,9 @@ { "cell_type": "markdown", "id": "1f87f5e6-339e-4fcf-900b-6d845d3c713d", - "metadata": {}, + "metadata": { + "id": "1f87f5e6-339e-4fcf-900b-6d845d3c713d" + }, "source": [ "- As we can see based on the relatively high accuracy values above, the LoRA finetuning was successful" ] @@ -1415,7 +1497,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.11" + "version": "3.11.4" } }, "nbformat": 4, diff --git a/ch06/02_bonus_additional-experiments/README.md b/ch06/02_bonus_additional-experiments/README.md index 7e011f6..61f6e07 100644 --- a/ch06/02_bonus_additional-experiments/README.md +++ b/ch06/02_bonus_additional-experiments/README.md @@ -19,7 +19,7 @@ For example, | 6 | gpt2-large (774M) | pretrained | last | last_block | longest train ex. (120) | 99.52% | 98.66% | 96.67% | 1.50 min | A100 | | 7 | gpt2-xl (1558M) | pretrained | last | last_block | longest train ex. (120) | 99.81% | 99.33% | 98.33% | 2.83 min | A100 | | 8 | gpt2-small (124M) | random | last | all | longest train ex. (120) | 100% | 96.64% | 93.67% | 0.69 min | A100 | -| 9 | gpt2-small (124M) | pretrained | last | LoRA | longest train ex. (120) | 99.52% | 97.99% | 97.67% | 0.75 min | A100 | +| 9 | gpt2-small (124M) | pretrained | last | LoRA | longest train ex. (120) | 100.00% | 97.32% | 96.67% | 0.75 min | A100 | | 10 | gpt2-small (124M) | pretrained | last | last_block | context length (1024) | 83.08% | 87.92% | 78.33% | 2.46 min | A100 | | 11 | gpt2-small (124M) | pretrained | last | last_block | variable: no padding (batch size 1) | 100.00% | 98.66% | 98.00% | 1.75 min | A100 | | 12 | gpt2-small (124M) | pretrained | last | last_block | variable: no padding (batch size 8) | 99.33% | 98.66% | 98.33% | 1.70 min | A100 | @@ -41,7 +41,7 @@ You can use the following code to reproduce the experiments: - Row 6: `python additional-experiments.py --model_size "gpt2-large (774M)"` - Row 7: `python additional-experiments.py --model_size "gpt2-xl (1558M)"` - Row 8: `python additional-experiments.py --weights random --trainable_layers all` -- Row 9: `python additional-experiments.py --trainable_layers lora --lora_rank 16 --lora_alpha 8` +- Row 9: `python additional-experiments.py --trainable_layers lora --lora_rank 16 --lora_alpha 16` - Row 10: `python additional-experiments.py --context_length "model_context_length"` - Row 11: `python additional-experiments.py --no_padding --batch_size 1` - Row 12: `python additional-experiments.py --no_padding --batch_size 1 --accumulation_steps 8` @@ -59,7 +59,7 @@ I've kept the LLM and dataset small on purpose, so you can run the training on a 3. **Training All Layers vs. Last Transformer Block (Row 1 vs. 4)**: Training all layers shows a modest improvement of ~2% over just training the last transformer block, but it requires almost three times longer in terms of training duration. 4. **Using Larger Pretrained Models (Row 1 vs 5, and Row 1 vs. 6 and 7)**: Employing a 3x larger pretrained model leads to worse results. However, using a 5x larger model improves performance compared to the initial model, as was anticipated. Similarly, the 12x larger model improves the predictive performance even further. (The medium model was perhaps not well pretrained or the particular finetuning configuration works not as well for this model.) 5. **Using a Model with Random Weights vs. Pretrained Weights (Row 1 vs. 8)**: Utilizing a model with random weights yields results that are only slightly worse by 1.3% compared to using pretrained weights. -6. **Using LoRA (Low-Rank Adaptation) vs Training All Layers (Row 9 vs. 4)**: Keeping the model frozen and adding trainable LoRA layers (see [Appendix E](../../appendix-E/01_main-chapter-code/appendix-E.ipynb) for details) is a viable alternative to training all model parameters and even improves the performance by 1% point. As it can be seen by the 1% lower gap between the training and validation accuracy when using LoRA, this is likely due to less overfitting. Moreover, using LoRA is also slightly faster because fewer parameters have to be updated. +6. **Using LoRA (Low-Rank Adaptation) vs Training All Layers (Row 9 vs. 4)**: Keeping the model frozen and adding trainable LoRA layers (see [Appendix E](../../appendix-E/01_main-chapter-code/appendix-E.ipynb) for details) is a viable alternative to training all model parameters and even improves the performance by 1% point. As it can be seen by the ~1% lower gap between the training and validation accuracy when using LoRA, this is likely due to less overfitting. Moreover, using LoRA is also slightly faster because fewer parameters have to be updated. 7. **Padding Input to Full Context Length vs. Longest Training Example (Row 1 vs. 10)**: Padding the input to the full supported context length results is significantly worse. 8. **Padding vs no padding (Row 1 vs. 11 and 12)**: The `--no_padding` option disables the padding in the dataset, which requires training the model with a batch size of 1 since the inputs have variable lengths. This results in a better test accuracy but takes longer to train. In row 12, we additionally enable gradient accumulation with 8 steps to achieve the same batch size as in the other experiments, which helps reduce overfitting and slightly boost the test set accuracy. 9. **Disabling the causal attention mask (Row 1 vs. 13)**: Disables the causal attention mask used in the multi-head attention module. This means all tokens can attend all other tokens. The model accuracy is slightly improved compared to the GPT model with causal mask. diff --git a/ch06/02_bonus_additional-experiments/additional-experiments.py b/ch06/02_bonus_additional-experiments/additional-experiments.py index a3dd719..8228778 100644 --- a/ch06/02_bonus_additional-experiments/additional-experiments.py +++ b/ch06/02_bonus_additional-experiments/additional-experiments.py @@ -4,6 +4,7 @@ # Code: https://github.com/rasbt/LLMs-from-scratch import argparse +import math import os from pathlib import Path import time @@ -23,8 +24,8 @@ from previous_chapters import GPTModel, load_weights_into_gpt class LoRALayer(torch.nn.Module): def __init__(self, in_dim, out_dim, rank, alpha): super().__init__() - std_dev = 1 / torch.sqrt(torch.tensor(rank).float()) - self.A = torch.nn.Parameter(torch.randn(in_dim, rank) * std_dev) + self.A = torch.nn.Parameter(torch.empty(in_dim, rank)) + torch.nn.init.kaiming_uniform_(self.A, a=math.sqrt(5)) self.B = torch.nn.Parameter(torch.zeros(rank, out_dim)) self.alpha = alpha