{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Copyright (c) 2020-2021 Microsoft Corporation. All rights reserved. \n", "\n", "Licensed under the MIT License.\n", "\n", "# AutoVW: ChaCha for Online AutoML with Vowpal Wabbit\n", "\n", "\n", "## 1. Introduction\n", "\n", "\n", "In this notebook, we use one real data example (regression task) to showcase AutoVW, which is an online AutoML solution based on the following work:\n", "\n", "*ChaCha for online AutoML. Qingyun Wu, Chi Wang, John Langford, Paul Mineiro and Marco Rossi. To appear in ICML 2021.*\n", "\n", "AutoVW is implemented in FLAML. FLAML requires `Python>=3.6`. To run this notebook example, please install:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install flaml[notebook,vw];" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 2. Online regression with AutoVW\n", "### Load data from openml and preprocess\n", "\n", "Download [NewFuelCar](https://www.openml.org/d/41506) from OpenML." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "slideshow": { "slide_type": "subslide" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(36203, 17) (36203,)\n" ] } ], "source": [ "import openml\n", "# did = 42183\n", "did = 41506\n", "ds = openml.datasets.get_dataset(did)\n", "target_attribute = ds.default_target_attribute\n", "data = ds.get_data(target=target_attribute, dataset_format='array')\n", "X, y = data[0], data[1]\n", "print(X.shape, y.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Convert the openml dataset into vowpalwabbit examples:\n", "Sequentially group features into up to 10 namespaces and convert the original data examples into vowpal wabbit format." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "openml example: 8.170000076293945 [1.0000e+01 7.0000e+00 3.0000e+00 4.0000e+00 nan 6.3300e+00\n", " 1.3600e-01 7.3300e+00 7.0100e+00 6.9800e+00 3.0000e-03 7.0000e+00\n", " 9.7000e+00 1.2300e+01 1.0217e+03 0.0000e+00 5.8000e+01]\n", "vw example: 8.170000076293945 |a 0:10.000000 1:7.000000|b 2:3.000000 3:4.000000|c 4:nan 5:6.330000|d 6:0.136000 7:7.330000|e 8:7.010000 9:6.980000|f 10:0.003000 11:7.000000|g 12:9.700000 13:12.300000|h 14:1021.700012 15:0.000000|i 16:58.000000\n" ] } ], "source": [ "import numpy as np\n", "import string\n", "NS_LIST = list(string.ascii_lowercase) + list(string.ascii_uppercase)\n", "max_ns_num = 10 # the maximum number of namespaces\n", "orginal_dim = X.shape[1]\n", "max_size_per_group = int(np.ceil(orginal_dim / float(max_ns_num)))\n", "# sequential grouping\n", "group_indexes = []\n", "for i in range(max_ns_num):\n", " indexes = [ind for ind in range(i * max_size_per_group,\n", " min((i + 1) * max_size_per_group, orginal_dim))]\n", " if len(indexes) > 0:\n", " group_indexes.append(indexes)\n", "\n", "vw_examples = []\n", "for i in range(X.shape[0]):\n", " ns_content = []\n", " for zz in range(len(group_indexes)):\n", " ns_features = ' '.join('{}:{:.6f}'.format(ind, X[i][ind]) for ind in group_indexes[zz])\n", " ns_content.append(ns_features)\n", " ns_line = '{} |{}'.format(str(y[i]), '|'.join('{} {}'.format(NS_LIST[j], ns_content[j]) for j in range(len(group_indexes))))\n", " vw_examples.append(ns_line)\n", "print('openml example:', y[0], X[0])\n", "print('vw example:', vw_examples[0])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Set up the online learning loop\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import mean_squared_error\n", "def online_learning_loop(iter_num, vw_examples, vw_alg):\n", " \"\"\"Implements the online learning loop.\n", " \"\"\"\n", " print('Online learning for', iter_num, 'steps...')\n", " loss_list = []\n", " for i in range(iter_num):\n", " vw_x = vw_examples[i]\n", " y_true = float(vw_examples[i].split('|')[0])\n", " # predict step\n", " y_pred = vw_alg.predict(vw_x)\n", " # learn step\n", " vw_alg.learn(vw_x)\n", " # calculate one step loss\n", " loss = mean_squared_error([y_pred], [y_true])\n", " loss_list.append(loss)\n", " return loss_list\n", "\n", "max_iter_num = 10000 # or len(vw_examples)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Vanilla Vowpal Wabbit (VW)\n", "Create and run a vanilla vowpal wabbit learner." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Online learning for 10000 steps...\n", "Final progressive validation loss of vanilla vw: 15.18087237487917\n" ] } ], "source": [ "from vowpalwabbit import pyvw\n", "''' create a vanilla vw instance '''\n", "vanilla_vw = pyvw.vw('--quiet')\n", "\n", "# online learning with vanilla VW\n", "loss_list_vanilla = online_learning_loop(max_iter_num, vw_examples, vanilla_vw)\n", "print('Final progressive validation loss of vanilla vw:', sum(loss_list_vanilla)/len(loss_list_vanilla))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### AutoVW which tunes namespace interactions \n", "Create and run an AutoVW instance which tunes namespace interactions. Each AutoVW instance allows ```max_live_model_num``` of VW models (each associated with its own hyperaparameter configurations that are tuned online) to run concurrently in each step of the online learning loop." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Seed namespaces (singletons and interactions): ['g', 'a', 'h', 'b', 'c', 'i', 'd', 'e', 'f']\n", "Created challengers from champion ||\n", "New challenger size 37, ['|ah|', '|eg|', '|gi|', '|ag|', '|de|', '|ei|', '|eh|', '|fg|', '|cf|', '|hi|', '|bf|', '|cd|', '|ai|', '|ef|', '|cg|', '|ch|', '|ad|', '|bc|', '|gh|', '|bh|', '|ci|', '|fh|', '|bg|', '|be|', '|bd|', '|fi|', '|bi|', '|df|', '|ac|', '|ae|', '|dg|', '|af|', '|di|', '|ce|', '|dh|', '|ab|', '||']\n", "Online learning for 10000 steps...\n", "Seed namespaces (singletons and interactions): ['ce', 'g', 'a', 'h', 'b', 'c', 'i', 'd', 'e', 'f']\n", "Created challengers from champion |ce|\n", "New challenger size 43, ['|be_ce|', '|bce_ce|', '|ce_ei|', '|ce_ceg|', '|ce_fh|', '|ce_gh|', '|ce_cef|', '|cd_ce|', '|ce_cg|', '|cde_ce|', '|ce_cf|', '|bd_ce|', '|ae_ce|', '|ce_gi|', '|ce_ci|', '|ab_ce|', '|ce_fg|', '|ce_di|', '|bi_ce|', '|ce_de|', '|ce_eg|', '|ce_dg|', '|ce_hi|', '|ai_ce|', '|ag_ce|', '|ac_ce|', '|bh_ce|', '|ce_ch|', '|ce|', '|ace_ce|', '|ah_ce|', '|af_ce|', '|bc_ce|', '|ce_dh|', '|ce_ef|', '|ad_ce|', '|ce_df|', '|ce_cei|', '|ce_eh|', '|bg_ce|', '|ce_ceh|', '|bf_ce|', '|ce_fi|']\n", "Final progressive validation loss of autovw: 8.718817421944529\n" ] } ], "source": [ "''' import AutoVW class from flaml package '''\n", "from flaml import AutoVW\n", "\n", "'''create an AutoVW instance for tuning namespace interactions'''\n", "# configure both hyperparamters to tune, e.g., 'interactions', and fixed arguments about the online learner,\n", "# e.g., 'quiet' in the search_space argument.\n", "autovw_ni = AutoVW(max_live_model_num=5, search_space={'interactions': AutoVW.AUTOMATIC, 'quiet': ''})\n", "\n", "# online learning with AutoVW\n", "loss_list_autovw_ni = online_learning_loop(max_iter_num, vw_examples, autovw_ni)\n", "print('Final progressive validation loss of autovw:', sum(loss_list_autovw_ni)/len(loss_list_autovw_ni))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Online performance comparison between vanilla VW and AutoVW" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAFzCAYAAADIY/vqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdeZxcVZ3//9ep6qV637N2VrKRlSyEVdkEghACwgABGQWFrwvCoN8ZcXRUGOcn+kUGcRlBGWHUSWBQgYAMi2yGBEgChKxkXztL7/tadX5/nNud7qzV6a6+Vd3v5+NRj6q6tX36UuRd59xzzzHWWkRERCSxBfwuQERERHpOgS4iItIPKNBFRET6AQW6iIhIP6BAFxER6QcU6CIiIv1Akt8F9ERhYaEdPXq032WIiIj0iVWrVpVZa4uO9lhCB/ro0aNZuXKl32WIiIj0CWPMzmM9pi53ERGRfkCBLiIi0g8o0EVERPqBhD6GLiIi/mhtbWXPnj00NTX5XUq/FAqFKC4uJjk5OerXKNBFRKTb9uzZQ1ZWFqNHj8YY43c5/Yq1lvLycvbs2cOYMWOifp263EVEpNuampooKChQmMeAMYaCgoJu934o0EVE5KQozGPnZPatAl1ERBLOBRdcwEsvvdRl20MPPcSXv/zlbr3Pc889x/333w/A97//fR544AEAPv/5z/P0008f83VPPPEECxcu7LKtrKyMoqIinn32Wa666qqO7T/84Q8ZN25cx/0lS5Zw5ZVXdqvOaCjQRUQk4SxcuJDFixd32bZ48eIjQvZErrzySu65555uf/7VV1/NK6+8QkNDQ8e2p59+mvnz53P22WfzzjvvdGxfvnw52dnZHDx4EIBly5Zx9tlnd/szT0SBLiIiCefaa6/lhRdeoKWlBYAdO3ZQUlLCokWLmDNnDlOmTOF73/tex/NHjx7N9773PWbNmsW0adPYuHEjAI8//jh33HHHcT/rvvvu4/TTT2fq1KncfvvtWGvJzs7mvPPOY8mSJR3Pa/9BUVRURHZ2Nlu2bAFg7969XHPNNSxbtgxwgX7OOef06v4AjXIXEZEeunfJOtaX1PTqe04els335k855uP5+fnMnTuXF198kQULFrB48WKuu+46/vmf/5n8/HzC4TAXXXQRH330EdOnTwegsLCQ999/n1/+8pc88MAD/OY3v4mqljvuuIPvfve7ANx88808//zzzJ8/n4ULF/KHP/yB66+/npKSEjZt2sSFF14IwDnnnMOyZcsIh8OMHz+eM888k5deeokrrriC1atXc/rpp/dwDx1JLXTP1jXvsO7tF/wuQ0REotS52729dfzUU08xa9YsZs6cybp161i/fn3H8z/zmc8AMHv2bHbs2BH157z++uucccYZTJs2jddee41169YBcPnll/P2229TU1PDU089xTXXXEMwGATg7LPPZtmyZSxbtoyzzjqLuXPn8u677/LBBx8wadIkQqFQL+2FQ9RC91S+8gDDaj+Ccy73uxQRkYRyvJZ0LC1YsIC7776b999/n4aGBvLz83nggQdYsWIFeXl5fP7zn+9y6ldqaioAwWCQtra2qD6jqamJr3zlK6xcuZIRI0bw/e9/v+M909LSmDdvHn/+859ZvHgxDz74YMfrzjnnHH72s58RDoe57bbbyMrKoqmpiTfeeCMmx89BLfQO1gQwNuJ3GSIiEqXMzEwuuOACbr31VhYuXEhNTQ0ZGRnk5ORw4MABXnzxxR5/Rnt4FxYWUldXd8TI94ULF/Lggw9y4MABzjrrrI7tp556KiUlJSxdupSZM2cCcNppp/GrX/0qJsfPQYHewZogBut3GSIi0g0LFy5k9erVLFy4kBkzZjBz5kwmTZrEjTfe2CvBmZuby2233cbUqVO59NJLjzj2ffHFF1NSUsL111/f5dxxYwxnnHEGBQUFHdO3nnXWWWzbti1mLXRjbeKG2Jw5c2xvrYf+3k9vZHTlcgZ9f3uvvJ+ISH+2YcMGTj31VL/L6NeOto+NMaustXOO9ny10NuZAAHU5S4iIolJge5Rl7uIiCQyBXo7EyCgQBcRkQSlQO9gCBD2uwgREZGTokD32IC63EVEJHEp0NuZAIEEHvEvIiIDmwK9g9EodxGRBPPMM89gjOlYbOV4HnrooS6rox3NLbfcwiOPPHLEZ1x22WXcfffdPPTQQx3bL730Ur74xS923P/GN77RZbY4cEuypqend6y0Bm5CnKPd7ikFuscGggp0EZEEs2jRIs4991wWLVp0wudGE+jHW5a1fcEVgEgkQllZWce87nDsZVELCwv5yU9+Es2f0yMK9HYa5S4iklDq6upYunQpjz32WEcIv/HGG1xxxRUdz7njjjt4/PHHefjhhykpKeGCCy7gggsuANyPgWnTpjF16lS++c1vAnDRRRexceNG9u3bB0B9fT2vvvoqV111FWeffTbLly8HYN26dUydOpWsrCwqKytpbm5mw4YNzJo164g6b731Vp588kkqKipiuj+0OEs7TSwjInJyXrwH9q/p3fccMg0uu/+4T3n22WeZN28eEyZMoKCggFWrVh3zuXfeeScPPvggr7/+OoWFhZSUlPDNb36TVatWkZeXxyWXXMIzzzzDVVddxTXXXMNTTz3FXXfdxZIlSzj//PPJzs4mOzubpKQkdu3a1bGK2t69e1m+fDk5OTlMmzaNlJQUvvvd7zJnzhyuvPJKwHWr33rrrfz0pz/l3nvv7dXd1Jla6B5jgmqhi4gkkEWLFnHDDTcAcMMNN0TV7d5uxYoVnH/++RQVFZGUlMRNN93EW2+9BRx9WdZ2hy+LetZZZ3Xcb587/r777usI83Z33nknTzzxBLW1tT36m49HLXSPNQECxmIjEUxAv3NERKJ2gpZ0LFRUVPDaa6+xZs0ajDGEw2GMMSxYsIBI5FBva+flU6N19tlns2/fPlavXs2yZcu6HFNvP46+Zs0apk6dyogRI/jJT35CdnY2t9xyyzHfMzc3lxtvvJFf/OIX3a4nWkqudsbtis5fBBERiU9PP/00N998Mzt37mTHjh3s3r2bMWPGEIlEWL9+Pc3NzVRVVfHXv/614zVZWVkdLeS5c+fy5ptvUlZWRjgcZtGiRZx33nmAWynt+uuv53Of+xyXXXYZoVCo4z3OPvtsnn/+efLz8wkGg+Tn51NVVcXy5ctPuIra17/+dR555JGo12LvLgV6u0B7oGu2OBGReLdo0SKuvvrqLtuuueYaFi9ezHXXXcfUqVO57rrrOtYiB7j99tuZN28eF1xwAUOHDuX+++/nggsuYMaMGcyePZsFCxZ0PLfzsqydTZs2jbKyMs4888wu23JycigsLATgu9/9Ls8999wRNRcWFnL11VfT3NzcK/vgcFo+1bP8iX/mrO2/oPmefaSG0nvlPUVE+istnxp7Wj71ZJkgAJGwWugiIpJ4FOgeoy53ERFJYAr0dhoUJyIiCUyB3q490NXlLiISlUQegxXvTmbfKtDbecfQrbrcRUROKBQKUV5erlCPAWst5eXlXU6Xi4YmlvHoGLqISPSKi4vZs2cPpaWlfpfSL4VCIYqLi7v1GgV6O6NAFxGJVnJyMmPGjPG7DOlEXe7tOrrcNShOREQSjwLdoy53ERFJZAp0j+kY5R6bOXZFRERiSYHusV6g24hGbIqISOJRoHtMwDuGbtXlLiIiiUeB7jGaKU5ERBKYAr1dQDPFiYhI4lKge9q73NEodxERSUAKdE9Hl7tVl7uIiCQeBbrHqMtdREQSmAK9nRZnERGRBKZA9xw6bU1d7iIikngU6B4TMIBa6CIikpgU6B5j3MJzmstdREQSkQLd0z4oDk0sIyIiCUiB3s4bFKcWuoiIJCIFuscE2xdnUQtdREQSjwLdY4wWZxERkcSlQPe0H0O3YbXQRUQk8ST5XUA7Y8xVwOVANvCYtfblvvz8QPtc7mqhi4hIAoppC90Y85/GmIPGmLWHbZ9njPnYGLPFGHMPgLX2GWvtbcCXgOtjWddRBXQMXUREElesu9wfB+Z13mDcwepfAJcBk4GFxpjJnZ7yHe/xPhXomCmura8/WkREpMdiGujW2reAisM2zwW2WGu3WWtbgMXAAuP8CHjRWvv+sd7TGHO7MWalMWZlaWlpr9XaMfVrxPbae4qIiPQVPwbFDQd2d7q/x9v2NeBTwLXGmC8d68XW2kettXOstXOKiop6raj25VM19auIiCSiuBkUZ619GHjYr883GhQnIiIJzI8W+l5gRKf7xd42XwWC7V3uGhQnIiKJx49AXwGMN8aMMcakADcAz/lQRxeB9i53LZ8qIiIJKNanrS0ClgMTjTF7jDFfsG4Y+R3AS8AG4Clr7bpY1hGVjkFx6nIXEZHEE9Nj6NbahcfY/hfgL7H87O5q73LXamsiIpKINPWrJxBQl7uIiCQuBbqnY3EWdbmLiEgCUqB7TMC4a7XQRUQkASnQPYGgG06gLncREUlECRnoxpj5xphHq6ure+09O+Zy16A4ERFJQAkZ6NbaJdba23NycnrtPdvXQ9dMcSIikogSMtBj4dB66Gqhi4hI4lGge9TlLiIiiUyB7tHiLCIiksgU6J72meLO3PSAz5WIiIh0nwLd0z5TnIiISCJSink6BsWJiIgkIAW6p2NxFhERkQSkQPeohS4iIolMge5RoIuISCJToHs0KE5ERBJZQqZYLOZyNwp0ERFJYAmZYrGYy73L+2u2OBERSTAJGeix1tbW6ncJIiIi3aJAP4q21ha/SxAREekWBfpRtLQ0+12CiIhItyjQj6KtpcnvEkRERLpFgX4UYR1DFxGRBKNAP4o2dbmLiEiCUaAfRVurutxFRCSxKNCPQl3uIiKSaBToRxFuVZe7iIgkFgV6Jytn/xiANgW6iIgkGAV6J6HcIQBE1OUuIiIJRoHeSSA5FYCIWugiIpJgThjoxpgfG2OyjTHJxpi/GmNKjTGf7YvijlNTr6+2BhBMSgEg3KapX0VEJLFE00K/xFpbA1wB7ADGAf8Yy6JOJFarrQWTXaBbdbmLiEiCiSbQk7zry4H/sdb2brM4jgTbu9zb1OUuIiKJJenET+F5Y8xGoBH4sjGmCOiXM68keS30iLrcRUQkwZywhW6tvQc4G5hjrW0F6oEFsS7MD8HkEAA2rC53ERFJLNEMivs7oNVaGzbGfAf4PTAs5pX5IKnjGLpa6CIikliiOYb+L9baWmPMucCngMeA/4htWf5oD/TJa37scyUiIiLdE02gh73ry4FHrbUvACmxK8k/SSmuyz3TNPpciYiISPdEE+h7jTGPANcDfzHGpEb5uoSTnNwvf6eIiMgAEE0wXwe8BFxqra0C8vH5PPRYSU5J9bsEERGRkxLNKPcGYCtwqTHmDmCQtfblmFfmg2AwmrP4RERE4k80o9zvAv4ADPIuvzfGfC3WhfnBBPrlkQQRERkAommSfgE4w1pbD2CM+RGwHPhZLAvzy+bgOMaHt/hdhoiISLdE0yQ1HBrpjnfbxKYc/5UNOpNmm+x3GSIiIt0STQv9t8C7xpg/e/evwp2L3j8lpZFqWrGRiLrgRUQkYZww0K21Dxpj3gDO9TbdYq39IKZVnYAxZj4wf9y4cb3/5klupHtzcyOhtIzef38REZEYOGYT1BiT337BLZv6e++y09vmm1gtnwpgvPncmxsbev29RUREYuV4LfRVgOXQ8XLrXRvv9tgY1uUbk5wGQGuTAl1ERBLHMQPdWjumLwuJFwGvhd7SrOlfRUQkcWjU12HaA721WS10ERFJHAr0w7QHes2BHf4WIiIi0g0K9MMkpaYDMO31W3yuREREJHpRTV5ujAkCgzs/31q7K1ZF+Snc2ux3CSIiIt12wkD35m3/HnAAiHibLTA9hnX5ZtzcebDU7ypERES6J5oW+l3ARGtteayLiQeZ2Xl+lyAiItJt0RxD3w1Ux7qQeLIi5xL2UeR3GSIiIlGLpoW+DXjDGPMC0HGA2Vr7YMyq8lkkKZ0QOpYuIiKJI5pA3+VdUrxLv2eTMwjZJr/LEBERiVo0i7PcC2CMyfTu18W6KL/Z5HTSTAuRcJhAMOh3OSIiIid0wmPoxpipxpgPgHXAOmPMKmPMlNiX5h+T4s5Fb2yo9bkSERGR6EQzKO5R4OvW2lHW2lHAN4Bfx7Ysf5kUt2xqY70CXUREEkM0gZ5hrX29/Y619g2gXy8UHkh1f15zQ78/uiAiIv1EVKPcjTH/AvzOu/9Z3Mj3fivoBXpLY43PlYiIiEQnmhb6rUAR8CfvUuRt67eCoUwAmhvVQhcRkcQQzSj3SuDOPqglasaY+cD8cePGxeT9k71Ab1Wgi4hIgjhmoBtjHrLW/oMxZglu7vYurLVXxrSy47DWLgGWzJkz57ZYvH9ymgv0tiYFuoiIJIbjtdDbj5k/0BeFxJOUtCwAwgp0ERFJEMcMdGvtKu/madban3Z+zBhzF/BmLAvzU2q6a6FHmut9rkRERCQ60QyK+9xRtn2+l+uIK2nproVeWVXlcyUiIiLROd4x9IXAjcAYY8xznR7KAipiXZifQhku0C/d81PgPn+LERERicLxjqEvA/YBhcBPOm2vBT6KZVF+S0kJ+V2CiIhItxzvGPpOYCdwVt+VEx9MIECZyacsUMQkv4sRERGJQjSLs5xpjFlhjKkzxrQYY8LGmH4/hdr+tHEYG/a7DBERkahEMyju58BCYDOQBnwR+EUsi4oHbcmZhCINfpchIiISlWgCHWvtFiBorQ1ba38LzIttWf6LJGeRbhXoIiKSGKJZnKXBGJMCfGiM+TFuoFxUPwQSmU3JIJ1G9lY1Mjw3ze9yREREjiuaYL4ZCAJ3APXACOCaWBYVD+pJJ8M084n7X/W7FBERkROKZnGWnd7NRuDe2JYTP5oD6QDkUetzJSIiIid2vIll1nCURVnaWWunx6SiOBEKu4H8j6f8CDe/joiISPw6Xgv9Cu/6q951+2Itn+U4Qd9fFGe4P3FaYIe/hYiIiEThmMfQrbU7ve72i621/2StXeNdvglc0ncl+mPM6ZcBEMH4XImIiMiJRTMozhhjzul05+woX5fYJl7GgYyJvB8ZTyTS7zskREQkwUVz2toXgP80xuQABqgEbo1pVXGiIX04ObUfU9fSRnYo2e9yREREjimaUe6rgBleoGOtrY55VXHChvLINXVUN7Qq0EVEJK4db5T7Z621vzfGfP2w7QBYax+McW3+S88nhzoONLQwIj/d72pERESO6Xgt9AzvOqsvColHSRn5pJgwtbXVQK7f5YiIiBzT8ZZPfcS7HjCTyRwuOasQgMaaMmCUv8WIiIgcx/G63B8+3guttXf2fjnRMcbMB+aPGzcupp+Tml0AQEXp/ph+joiISE8d7/SzVSe4+MZau8Rae3tOTk5MPycjpwiA599dF9PPERER6anjdbk/0ZeFxKNUr8t99iBNLiMiIvHthKetGWOKgG8Ck4FQ+3Zr7YUxrCs+pOUBEGodMGfqiYhIgopmxrc/ABuAMbjV1nYAK2JYU/zwAr2q/ADrS2p8LkZEROTYogn0AmvtY0CrtfZNa+2tQP9vnQMkh6gN5lJsSvn0w3/zuxoREZFjimbq11bvep8x5nKgBMiPXUnxpSpYQKFR61xEROJbNIH+A2/a128APwOygbtjWlUcqU/KpcDoGLqIiMS3aAL9XW/+9mrgghjXE3fScocQqtsDQGs4QnKw/y80JyIiiSeadHrbGPOyMeYLxpi8mFcUZ0aOGMGQpFoA/u5Xy32uRkRE5OhOGOjW2gnAd4ApwCpjzPPGmM/GvLI4YZLTCEUaeCblX/hwd5Xf5YiIiBxVVP3H1tr3rLVfB+YCFcDAmXSmyR0/Py2w1edCREREju2EgW6MyTbGfM4Y8yKwDNiHC/aBYdDkjpvZoWiGHIiIiPS9aBJqNfAMcJ+1duAdRJ7zBXj3V9TW1VJT3UZTa5hQctDvqkRERLqIpst9rLX27gEZ5gCBAJw6n/SWcsCy+UCd3xWJiIgcIZpBcbYvColrmYMJ2jaGUsH8ny/1uxoREZEj6KTqaKS5ifGWh77mcyEiIiJHp0CPRuG4LncfenWTT4WIiIgcXTSj3CcYY/5qjFnr3Z9ujPlO7EuLI8NnA1BDBgAPvbrZz2pERESOEE0L/dfAt/AWabHWfgTcEMui4tKcL2ACbnR7MGB8LkZERKSraAI93Vr73mHb2mJRTFzLGkpWpIYUWslM1fnoIiISX6IJ9DJjzCmABTDGXIubXGZgyRoMwHfOK6C6sZX65oH3m0ZEROJXNIH+VeARYJIxZi/wD8CXYlpVPMoaCsDM1vcBeHXDAT+rERER6SKaQN9prf0UUARMstaea63dGeO64k96AQDT3v8uAHct/tDPakRERLqIJtC3G2MeBc4EBu40afljj9i0ZHWJD4WIiIgcKZpAnwS8iut6326M+bkx5tzYlhWH0nJh8lWQnI43nICvLfrA35pEREQ80Uz92mCtfcpa+xlgJpANvBnzyuJR9R5obeBPGT8G4JSiDJ8LEhERcaKaKc4Yc54x5pfAKiAEXBfTquJV9jAAZkbWANDYEvazGhERkQ7RzBS3Azey/W/ANGvtddbaP8a6sLh0/rcAMMNnc/enJlBS3cRL6/ZTWd/ic2EiIjLQRdNCn26tvdpau8haWx/ziqJgjJlvjHm0urq6bz948GSYsRBqSijMSgHg//xuFbf/bmXf1iEiInKYYwa6MeafvJs/MMY8fPilj+o7KmvtEmvt7Tk5OX3/4XljoGYvF43L7ti0Ykclv35rW9/XIiIi4jneHKYbvOtVfVFIwig4BYAhbV0ny/u3v2zgtk8eeWqbiIhIXzhmoFtrl3jXT7RvM8YEgExrbU0f1Baf8se46+fv5o4LHubnr2/peOhbf1rDJ8YX8ulpQ30qTkREBqpoBsX9tzEm2xiTAawF1htj/jH2pcWpAm9t9N3v8I1LJrDxX+dx5Qw3+n3Re7v4yh/e97E4EREZqKIZFDfZa5FfBbwIjAFujmlV8SyUA8WnA2BqSgglBynKSu3ylDk/eJWd5XExflBERAaIaAI92RiTjAv056y1rbRPlTZQ7Vnhrn//GQDuvGh8l4fL6pp5csXuvq5KREQGsGgC/RFgB5ABvGWMGQUM3GPoAKff5q69Fdhy0pIZnpvW5Sm/fGMrY7/1Aj94fj2t4UhfVygiIgOMsbb7jW1jTJK11vcFwefMmWNXrvThHPC2ZvjFGRDKhv/zFgDldc0crG1m5Y4K/uXZdV2efueF4/j6JRP7vk4REelXjDGrrLVzjvZYNIPi7vIGxRljzGPGmPeBC3u9ykSSlArjL4HybeD9ICrITOXUodncfNZotv/w012e/vBrW/jt29vZU9ngR7UiIjIARNPlfqs3KO4SIA83IO7+mFaVCApOgZZaqN13xEPGGH587XTOm1DUse3eJes590evM/qeF3h3WzlLN5dxMr0jIiIiR3O8iWXaGe/608DvrLXrjDHmeC8YEIomuetfnAlfWwWZRV0evm7OCK6bM4LqxlZm3Ptyl8euf/SdjtvjBmVy1WnDGD84ixnFuQzJCcW8dBER6X9OeAzdGPNbYDjudLUZQBB4w1o7O/blHZ9vx9ABIhG4L8/dHv0J+Pzzx3zqX9bs494l6zhQ0xz1208akkXAGL74iTFcPn0oqUnBnlYsIiIJ7njH0KMJ9ABwGrDNWltljCkAhltrP+r9UrvH10AH+P+KXbc7wD273DnqJ7C1tI7fLd/JC2v2UVobfcA/9rk5XHTq4JOtVERE+oGeBroBbgLGWmvvM8aMBIZYa9/r/VK7x/dAX/ZzePnb7nbxXPjiK916eU1TK9bCyh0VjCrIYPOBWh5bup2VOyuP+vykgOGXN83ikilDelq5iIgkoJ4G+n8AEeBCa+2pxpg84GVr7em9X2r3+B7okQg8+xVYvcjd/37vLecaiVgCAUNrOMK/vbCBx5ft6PL44tvPZExhBgUZKSQFA7SFIyQFoxnjKCIiiaqngf6+tXaWMeYDa+1Mb9tqa+2MGNTaLb4HOkBbCzzyCXdu+l0fxuxjGlra+MZTq3lx7f7jPm/GiFzuvHAcZ44tICM1mjGPIiKSKHp0HjrQaowJ4k33aowpwrXYBSApBU69Eiq3w4v3wL7VMfmY9JQk/uOzs9n4r/OYOjz7mM9bvbuKLzyxkinfe4nR97zAi2v2sbeqkUhEp8iJiPRn0bTQbwKuB2YBTwDXAt+x1v5P7Ms7vrhooQOsfw6e6rReTS92vR/PgZomahpb+WB3FdmhJFKTgjz74V6e+bDkqM//P58cy7ypQ5hRnEsgoDMPRUQSzUl3uXsj3M8EKoCLcOek/9VauyEWhXZX3AS6tXBv7qH739oLqZn+1YM7Bv/mplJueXzFEY8FA4Znv3oOU4efeFS+iIjEj54eQ+84dh5v4ibQAR6aBlW7Dt2/ez3kDPevnk6stWzcX8u/vbCBpVvKjvvcKcOyueH0Edx4xiiCasWLiMSVngb6A8By4E82zuYqjatAr9gGJR/A07e6+5/8J7jw2/7WdBTWWp7/aB9fW/TBCZ87tiiDa2YVc+WMYRTnpaEJAkVE/NXTQK/FLZ3aBjThut2ttfbYI7P6SFwFervvd+rG7qNj6SerqTXM/uomyuubaWmzTB6WzbbSOu5/cSPvbq846mt+cNVUFpw2jKxQch9XKyIiPQr0eBaXgV69B/59yqH7d6+DnGL/6jlJreEIf9tcymNLt/Pe9gpaw12/J1mpSSQnBRhVkM6UYdlcMX0Yp4/OJxgw1Da1sr6khjc2lbL1YB0t4QjDc9PYX93E0NwQm/bXceVpw9heVs+cUXnMHZNPKDlIekpQvQAiIsfR0xb6rKNsrgZ2+r0melwGOkDlTvjp9EP3/+9myBzkXz29oDUcoaSqkf9ZuYc/vLuTyobWXv+MjJQg04pzuGDiIGaPymPWyDyNxhcR6aSngf4O7pS1Nd6macBaIAf4srX25WO9NtbiNtAB/uMcOLDW3Z7zBbjiQX/r6WVVDS2s2lnJ5GHZbD5Qx9ItZTz61rYuz7l8+lDOn1DEsNw0ahpbCQYMw3LTWLqljBnFuQVkuxMAACAASURBVDy9ag9bDtbSErZUN7QQsbC/pqnj9cNyQswYkcvU4TksnDuS/IyUvv4zRUTiSk8D/U/Av1hr13n3JwP3Af+EGyh3Wi/XG7W4DvTqvfC3n8DKx9z9fymHYP+fuW3LwTpGFaSTFDAn1X0ejlje217B6x8f5INdlazYcWhe+6nDs7l82jA+M2s4g7O1zKyIDDw9DfS11tqpR9tmjPlQgX4CD06Bmj3u9s3PwCkX+FtPgqluaGXpljJW7qxgyep9lNU1EzDwyQlFnDGmgLlj8pk8NJtQckDH30Wk3+tpoD+Jm1hmsbfpeqAQuBlY6uciLQkR6OE2+NeCQ/eveAjm3OJfPQnMWsvmg3X85m/beG1jKWV1h5afzQol8empQ5k/YxjnjCtQuItIv9TTQE8DvgKc6216G/gl7hS2dGttXS/W2i0JEegAq56AJXceuj/nVrjkB5CS4V9NCa4tHOG97RW8s72C3RUNvLxuP2FraWqNkJuezDmnFDI0J8TMkXl8YkIh2TrNTkT6gR6ftmaMSQEm4hZo+dha2/tDnE9CwgQ6uElnlvwD7Ou0ItvVj8CMG/yrqZ9pbAnz/EclLF6xm80HaqlpOnQSxuiCdM6fOIi/m1PM5KHZasGLSELqaQv9fNyiLDtwk8qMAD5nrX2rd8vsvoQK9HYvfhPe/dWh+yYAX14Ogyb5V1M/VdXQwrKt5azcUcmKHRWsLanGWhiem8YZY/KZNDSL3RWNDMkJMWukOx9e092KSDzraaCvAm601n7s3Z8ALLLWzu71SrspIQMdYMur8Ptrum4bPgeu+TXkj/WnpgHgYE0Tz3y4lxfW7OejPVUc/tUflhNi0tBspg7L5hMTipit8+BFJM70NNA/stZOP9E2PyRsoANEIm7J1Y3Pd90+YR5c/4cBcYqbn8IRy5aDdQzJCdHcGuad7RU8uWIXb28p73hObnoyM0fkMmd0PudNKGLKMHXVi4i/ehrovwXCwO+9TTcBQWvtrb1a5UlI6EBvF26D1/4V3n6o6/ZbX4IRZ4ACpE/VNrXy4e4qDtQ089rGA7yzrYKK+hYAkgKGUQXpzB1TwLypQ5g9Ko/MVP3wEpG+09NATwW+yqFR7n8DfmmtbT72q/pGvwj0zpb9HF4+bIW2r7wDg071px7BWsu2snre+LiU93dV8tqGgzS2hjsenzEil0+OL2RUQQZThmUzcXCWuulFJGZOOtCNMUFgnbU2Lkds9btABzca/tHzu2675jGYdq0v5ciR6prbeHtLGcu3lvP6xwfZWd7Q5fFpw3M4bUQuc0bncdYpBQzK0qx2ItI7etpCfxb4mrV2VyyK64l+Gejtdr0Df/4SVG539z/5j3Dhd/ytSY6qrK6Z9SU17Cyv58Pd1awrqWbzwTrCEff/Vn5GChMHZzFuUCaDs1OZPSqf6cU5ZKi7XkS6qaeB/hYwE3gPqG/fbq29sjeLPBn9OtDbbXwBFt/obp91B1x8HwSC/tYkJ9TQ0saKHZW8vaWMAzVNrNpZyZ7Kxo7HQ8kBzhxbwMwRecwcmcvsUXkKeBE5oZ4G+nlH226tfbMXauuRARHoAFW74aFO0+l/YxNkDfavHjlp+6ub+GBXJW9uKuWNj0u7rC43Ij+N6cNzmTUqj9NH5zEqP4OcdM1wJyKHnFSgG2NCwJeAcbilUx/ze/3zww2YQAeoL4f/1+kc9S+8AiPm+leP9IqDtU2s3FHJh7ur+Hh/LR/tqeqy1vyognTOGVfIqUOyOG/CIEYWpPtYrYj47WQD/UmgFTeq/TJgp7X2rphV2Q3GmPnA/HHjxt22efNmv8vpO+FW+NfCQ/fTC+Cmp2HYTJ3e1o9sL6tnzd5qNh+oZdnWctbsqaYlHAFgTGEGM0fmMnNkHpOHZjOuKFOteJEB5GQDfY21dpp3Owl4z1o7K3Zldt+AaqG3sxZe+ja884uu21Nz3Exz4y9RuPcz1lq2l9Xz1qZS3txUyge7q6jq1IofU5jB+EGZnDIok3FFmYwtyqAgI5XivDSdQifSz5xsoL/fOcAPvx8PBmSgt6svh2e/CptePPKxf1gDuSP7vibpE+3nxq/eXcWO8gY27qth/b6aLoPuwC0pe+rQbGaPyuPccYVMHZaj1rxIgjvZQA9zaFS7AdKABu+2tdZmx6DWbhnQgd6urQXW/Qne+zXs7bQvgqnwleVQcIp/tUmfsdZS19zGttJ6DtQ0UVbXwpq91by/s5LNB2uJWAgYmDM6n0lDshg/KJPxg7OYNCSL3PQUv8sXkSj1ePnUeKVAP0wk4qaQ/eu9h7bNvgUu/4lOdRvAaptaWbWzkve2V/C/a/ezrazj7FOMganDcjhnXCFjizL45PgihuRoIhyReKVAH2jaWmD1IvjLP0LYm6F31ufgsh9Dsv6xHujCEcu20jp2ljewfl8NSzeX8f6uStq8iXCK89KYXpzD7FH5nDYih/GDs8gOqateJB4o0Aeq5jp4835Y9rND2xb8EmbcoBa7dNHYEmZ7WT2vf3yQtXurWbO3uuOYfDBgmFGcw+xReeRnpDJhcCaThmYzLCek1edE+pgCfaBrrITFn4WdS939vDHw1fcgScdO5dgO1DSxencVq/dUsXxrOWv2VtMaPvTvRWFmCtOLc5k5IpfTRuYyIi+dobkhUpP0Y1EkVhTo4hxYD7+5CFobIC0fzvsmnP4FCKo7VU4sErHUNrexdm81W0vrWL27mtV7qthysK7jOUkBw6ShWZxSlMnQnDTGD8pkdGEGOWlJBIwhPyOFnLRktexFTpICXbp67Qfw1v87dH/YLPjk/4VTLtIxdum20tpmPt5fy+7KBnaU1bN6TxVbS+sprT36Csvt68oXZaWSk5ZMcZ67PTw3jVOHZjO2MEPnz4scgwJdjlS6Cf50G+z7sOv2oafB5Q9C8Wx/6pJ+o6GljU0H6jhY00RpXTNBY6hqbKWkqpH91U3sqmigrrmNsrpmmlojHa8LBlxLfkh2iCE5IYrz0hiVn86EwVmcMiiTwsxUggp8GaAU6HJsrY1uRbePnoSKbVC+xW0fOgNu+iNkFvlbn/R71lrqW8JsL61nw/4athyso7K+hdK6ZvZVNbGzor5L4KcEA2SnJVOYmcKYwgyK89IYnpvG4OwQg7JD5KQlMbogg6RgwMe/SiQ2FOgSHWth97uw6AY3kA6gaBJc/wcoHOdvbTJghSOWg7VNbCutZ8vBOkqqGimvb+FgbTN7KhvYW9lIc1uky2tSkwJMGpLF5GHZTBuey8yRuYzMT9cStZLwFOjSfRuWwJOfPXQ/ZyQs+BmMPd+vikSOylpLaV0zWw7UUVbfQktbpGM63PX7arrMe1+QkcKEwVmMLsygKDOFUwZlMrYwkzFFGWQq7CUBKNDl5ITb3LSyS+5yI+MBhs+GKx6CodP9rU0kCtZadpQ3sGZvNbsrGthV3sDGA7XsKq+nurEVby4dAgaGZIeYOSqP4tw0xhZlMDw3nfyMFE4ZlKFT8SRuKNCl53Yug7cegK1/dfdTc+DqX8GgSZA/9vivFYlDzW1htpXWs6uigQ37athaWs/7OyspqW6k8z+LxsCwnDRmj8pj1shcphXnMnloNmkpCnnpewp06T0V2+GPX+y6EMzwOXD+PTD+Yv/qEuklreEI+6ub2FPZyL7qRnaU1bO1tJ73dlR0ORWvKMstUZsSDFCQmUJqUpCirFQKMlLIDCWRkZJEbnoySYEAxrhVraoaW2kNR2hujdDcFqaxNUxuegoZKUkcqGmivrmN5KQAycEAI/LSGJ6XxpjCDMIRS3pKkkb3iwJdYqBiuzuXvXQj7F3ltmUUwY1PQcE4CLdASgYkhbQ+u/QL1lr21zSxZk81a/dWs7OigdLaZmqb2jhY20Q44hbCOXyAXm9KDhpaw5a0ZO/HQ2YKycEAw3JCFGWlMigrRGYoiYIMtz0vI4VhOSGy05IJJatHoT9QoEtsNdfC0ofgbw8c+VggGSKtcO7XYfQ5bhKb9Py+r1GkD1hraWgJU1HfQl1zGzWNrbSGLcGAwRhISQqQEgyQkhQglBQkJSlAS1uEyoYWMlKTyEt3wdvSFmFnRQMlVY1sPVhHxEJDaxt1TW20tEXYXdnAoKwQJVWNtIbdczsP/jtcwEBhZiqZoSRG5KWTkRqkIMPdH57rTvsbmhtidEEGAMnBgHoD4pQCXfrG/rWu1b7xBRfixzPvRzBkqgv4lPS+qU+kH4tEbMfx/8qGFraW1tHQEqa01p3Pb7HUNrWxu7KB3RWNNLWGj+hNMMadvZqWHGTikCxG5qdjgXFFmYwuTKc4L53huWkUZWlyH78o0MVfbc3w0VOw/U3YtxrKNnV9/OyvQeFEmHYtJKf5U6PIAGOtm5u/qr6Vjw/UUt/cxvayeqoaWjDGsGFfDetLaqhtbjvite2z+eWnpzA4J0RhRgp1zW2MyE9nbFEG4wdlMSTbHQbQ4MHepUCX+BIJw54V8PbDsGMpNFe77VlD4dT5MPsWGDzZ3xpFBGstLeEI1sKuigb2VjWy1xssuL+6mbK6ZqoaWiirayEYMByoaTqi1Z8dSmJwdsjN01+UwYi8dEbkpzMiP43BWSHN299NCnSJbxXb4d1HoOQD2P2O21Z8OnziGzD+UghoCk+RRBCJWPZWNbK9rJ591Y2U1bWwv7qJfdWNbNhXe8QpgeCW4Z0wOMvN2V+QQV56CkNzQuSmJ5MVSqI4L10D+jpRoEviqNwJ7z0KH/wOmqohczCc/kWYcytkFPpdnYj0QHNbmL2VjeyubGR3RQM7y+upqG9la2kduysaKK9vOeI1xpv0Jy89hVBygKRAgKSgITuUTHFeGiML0hmcHWJ4bhqFmakUZqb063n8FeiSeFqbYO3TsPTf3YIxJggzboBz7oKiiX5XJyK9zFpLdWMrDS1h9tc0UVnfQk1TKzvKGthT2UhVQwvNbRHaIhFaw5aaxlZ2VzZ0WbgH3A+AosxUhuaEGJrjRu8XZro5A4rz0hmUlUpRVmrCtvoV6JK4rIUDa2Hlb12rPRKGc+6EM78CmYP8rk5EfBSJWMrqmtlf00RJVRNldc0crGlif00T+6q9S1Uj9S3hI15bkJHCiHw3an9Qdiq5aSkMyk5lSE6IIdkhhuaEyElLxsTZPBoKdOkfKrbBq/fC+mfc+e2z/h5mf17zyovIcTW0tLG7opGSqkZK65o7juvvqmigpKqJ0tpm6o4ymj+UHGBIdqgj5IfkpDEkO9Vd57jQL8zs21P4FOjSv+xeAe8/Dh8uAhuG7GKYdo071p470u/qRCQBtYYjHKxtZr83gn9fdSMHvJZ++/XBmmZawl27+IMBw6CsVAZnu9AvzEohJy2ZQVkhBmWlMrYok4lDsnqtTgW69E8V22Ddn91SryUfQFIazPuha7XHWTeZiCS+SMRS0eBG7u+vdl37na8P1Lhu/5qmNsLeUn4XTx7Mr//+qPl7UhTo0v8d3Aj/ew9sex2KToXTboTJCyBvlN+VicgAE4lYyutbOFjbRFIgoBZ6NBTo0kUkAh/8l5tXvnK7Gxk//hLXFX/KhTqfXUQS3vECPamvixGJmUDAdbfP/jyUb4XlP4fVi2HTi5A9HE67yYV71mC/KxUR6XVqoUv/1tLgzmd//3ew5z13nH3Wza5LfthMv6sTEekWdbmLABzcAG/+CNY9A1gYPNWt1z5oEpx6peue12A6EYljCnSRzmr3w9o/wkdPgo24ZV+xMGiKW/Ft1ucgo8DvKkVEjqBAFzmecKsL+KX/DqUb3bYx58Hc21yrPSnV3/pERDwaFCdyPMFkN0/8tL9z57XvWAqbXoInPwvBVBh/sQv3MeepS15E4pYCXaRdIOi63KddC23NsOVVN2nNx3+Bjc9DarY71n7ajTDqbIW7iMQVBbrI0SSlwqTL3aW1Cdb8D2z6X3f94e8h/xSYcjXMvAnyx/pdrYiIjqGLdEtLg+uW//APsPNtwMDoc+HU+TDuU1Bwit8Vikg/pkFxIrFQsw9W/MaNlq/e7bYVjHPd8qdeAcNmqVteRHqVAl0klqyFqp3w8Yuw/lnY/a47HS5nBEy9xs0pP2ymwl1EekyBLtKXakpg88vw0VOwcxlgIWsYjD4HRp7lWvCZRX5XKSIJSIEu4pfqvbD1NdjyCmx+BVob3KIx4z7lTpObeBmkZvpdpYgkCJ2HLuKXnOFu7vhZN0O4Dco2uWPuq34Lm19yc8uPvximX+dWhEvJ8LtiEUlQaqGL+CHc5haLWfM0rF50qOU+4gwY/ymYdAUUTtBxdxHpQl3uIvEs3AY7/uYmstn0EpRvdttzR7kBdVOuhiHTIagONZGBToEukkgqd7hg3/yKO/5uw5AxCCbOg+k3uFa8wl1kQFKgiySq2v1u+tkdS71BdfXudLhJV7hZ7EbM1eIxIgOIAl2kP2iudS33j56EbW9CuBmS07vOVJc9zO8qRSSGNMpdpD9IzTq0eExTNWx9Hba97nXPv+yekz8WTrnITWQz+lzIG+VvzSLSZ9RCF0l01kLJ+7D5Vdi1HHa9A22N7rGiU91sddOuhfwx/tYpIj2mLneRgSTcBqUbXav94xfd6XEAxae7pV/HnOda8jolTiThKNBFBrLqPW7Z14+egoPr3bbcUW4im/EXu+vkNH9rFJGoKNBFxHXN7/8Idr/nRszvWg7NNZCSCRM/DeMvgUmf1mx1InFMg+JExHWxD53hLnNvg3ArbH8L1j/jTo1b85SbinbcRS7gT7kQsof6XbWIREktdBGBSNi12Nc9Ax//BWr2AsatDjfuwkMj53XcXcRX6nIXkei1d81vegnWPwcH1rjt2cNh1DlucN34T7mBdSLSpxToInLyag+45V83veQmtGmudtsLJ7rlXyd+GornQCDob50iA4ACXUR6h7VQvsWdErf5ZTclbaQNQjlQPBdGnuHmmh8+W4PrRGJAg+JEpHcYA4Xj3eWsr7oZ6za/4laL2/UuvPaKe14gyQX7qLNd0A+bCekFEAj4W79IP6ZAF5GTF8o5NB0tQGMl7FkJO9+GLX+Ftx4AOvUCFk2CIdNg6Gkw7DQYPAXS8nwpXaS/UZe7iMROc52binbfh27luPItULbJG0XvyR7ugn3wFBj9CTfwLjnkX80icUzH0EUkvtQegL2rYO9KKN/qgr50ozsen5QGRRNh+CwYPNVdhp2mZWJFUKCLSCJoqnat+a2vQ+kG2Pu+m8kODoX8qHPc8fihM6BgnI7Jy4CjQXEiEv9COTDhUncBiESgYhscXOemq923Glb8GsIt7vFgqhucVzTJnTY36mw3R31arn9/g4iPFOgiEp8CASgc5y6TF7ht4VYo/diF+4F1UPIB7FwGa58+9LqckTDyTK8lP91d6xQ6GQDiJtCNMWOBbwM51tpr/a5HROJQMBmGTHWXziq2uXPi6w7A/rWw/U03Nz2ACcCgKVA82x2PLz7dnUKXXgAp6X3/N4jESEwD3Rjzn8AVwEFr7dRO2+cBPwWCwG+stfdba7cBXzDGPH30dxMROYb8sV2norUWave5cN+7EvasgHV/hlWPH3qOCbjZ7rKGuJAfPBkGTYacYrXoJSHFuoX+OPBz4L/aNxhjgsAvgIuBPcAKY8xz1tr1Ma5FRAYKYyB7mLtMuMRti0SgaqcbeFe9xx2L37sSKra7Fr2NeK8NQP4pbuBdRqH7oTB0BgyZrha9xLWYBrq19i1jzOjDNs8Ftngtcowxi4EFgAJdRGInEID8Me5yuMZKKN0E1buhbLO3bvy7bntLnXuOCbqR9iPmuvPlB02GjCL3AyAQcD8Y0vOjX5GurRnqS93gvswi9yOjahe0NUFTjRskOHgKZA46+mt1Gp8cxo9j6MOB3Z3u7wHOMMYUAP8GzDTGfMta+8OjvdgYcztwO8DIkSNjXauIDARpeW4ees7our29677kA3ca3b7VsPawrvsu75PvfjC0tbiQD3UacZ9TDDUl7nh/QwW01B56LJAMkdZj15ec4X44mMChxXEyitx1eqG7zhvl/o7MwW7kP7gFc0zA9VTkjYFgSvd+dEhCiZtBcdbacuBLUTzvUeBRcOehx7ouERnAOnfdT7rcbQu3uVPpyja7FnZLnWudJ4fcBDnl29xc9hjXwsdCUshNhdtcC4MmuS78UDakZEH2UKjZB0kpLoiT01zwmoAL/30fuR8Ugye7wwKBJPfDIdzsPqtim/uMTRsgZ4Sbke94Pw5Sc9yPgsFTXf3DZ7tegeLTXa+DMd7sfVM1Y1+C8SPQ9wIjOt0v9raJiMS/YJIL5KEz/K7k6MKtULnDnd6XXnBovED5Fhf0LfVuvv1wK7Q0wLY3oLUBtr525HslpUFbowv73FHuh8bwWe7QQ+5Idz9rqFr8ccKPQF8BjDfGjMEF+Q3AjT7UISLS/wSTD62IF61I2IV+6UYo2+J+tBzc4H4UbH3NPb7lFTeb3+r/7vrapJA7ZJCW6z5z8BQX/tnDYNCpbl6AYNx0BvdrsT5tbRFwPlBojNkDfM9a+5gx5g7gJdxpa/9prV0XyzpEROQ4AkEIpLlJeIbNdNumXN31Oda6YK/zFtkp3woH1rpT/NoHFDaUw7uPHJrND1zY5412U/UWjnOHFbKGuLkBMopc6z4Q7LM/tT/TXO4iItJ7wq1ufEFzLZRv9sLfG1tQvsUd++8sKc216odOd4cIMge74/fpBW5cQc4IBX4nmstdRET6RjDZDeAD78yBTsKtbvncuoMu9A+sdQPyakpgzR8PjeDv8n4pbgKgogmQMcj9UEhKhZzhLvwHTXZd+8lpsf/b4pwCXURE+kYw2bXGB0+BUy7o+pi17qyAtiY3w1/ldre9eg8cXO8W6KkpARt25+g3dQp/E3ATABVNcpfCCe70wcLx7lS+AUKBLiIi/jPGnSMPbkDd0bS1uOPzqZlutH7tftfKP7DOXUo/ho9fdKHfLnOwG5VfONFdt4d+RmG/G52vQBcRkcSQlOIu4AbjFZziLu2r8YGbRa9yh5vSt2yTC/nSjbB6cdfJfNLyvYBvD3nvOoFPw1Ogi4hI/5GUeiioJ847tN1a12VfutEL+o0u7Nc9A01Vh56Xmu267DuHfNFEb3BeoO//nm5IyFHuxpj5wPxx48bdtnnzZr/LERGRRGWtmzGvvSXf+br+4KHnJae7Y/Kdg75wojslrw/Psz/eKPeEDPR2Om1NRERipqGia2u+/bqm0+SmwRQoGN+pNT/BjbzPH+sGAfYynbYmIiLSXen5MPJMd+msqcaddle68VDI710F6/4MeI1kE3St91OvgIvv65NyFegiIiLdEcqG4tnu0llLw6EWfdlmN7FOcnqflaVAFxER6Q0p6TDsNHfxQXwP2RMREZGoKNBFRET6AQW6iIhIP6BAFxER6QcU6CIiIv2AAl1ERKQfUKCLiIj0AwkZ6MaY+caYR6urq0/8ZBERkQEgIQPdWrvEWnt7Tk6O36WIiIjEhYQMdBEREelKgS4iItIPKNBFRET6AQW6iIhIP2CstX7XcNKMMaXAzl58y0KgrBffbyDSPuw57cOe0z7sHdqPPdfb+3CUtbboaA8kdKD3NmPMSmvtHL/rSGTahz2nfdhz2oe9Q/ux5/pyH6rLXUREpB9QoIuIiPQDCvSuHvW7gH5A+7DntA97Tvuwd2g/9lyf7UMdQxcREekH1EIXERHpBxTogDFmnjHmY2PMFmPMPX7XE0+MMSOMMa8bY9YbY9YZY+7ytucbY14xxmz2rvO87cYY87C3Lz8yxszq9F6f856/2RjzOb/+Jr8YY4LGmA+MMc9798cYY9719tWTxpgUb3uqd3+L9/joTu/xLW/7x8aYS/35S/xjjMk1xjxtjNlojNlgjDlL38XuMcbc7f2/vNYYs8gYE9J38fiMMf9pjDlojFnbaVuvfe+MMbONMWu81zxsjDEnVai1dkBfgCCwFRgLpACrgcl+1xUvF2AoMMu7nQVsAiYDPwbu8bbfA/zIu/1p4EXAAGcC73rb84Ft3nWedzvP77+vj/fl14H/Bp737j8F3ODd/hXwZe/2V4BfebdvAJ70bk/2vp+pwBjvexv0++/q4334BPBF73YKkKvvYrf233BgO5DW6Tv4eX0XT7jfPgnMAtZ22tZr3zvgPe+5xnvtZSdTp1roMBfYYq3dZq1tARYDC3yuKW5Ya/dZa9/3btcCG3D/KCzA/eOKd32Vd3sB8F/WeQfINcYMBS4FXrHWVlhrK4FXgHl9+Kf4yhhTDFwO/Ma7b4ALgae9pxy+D9v37dPARd7zFwCLrbXN1trtwBbc93dAMMbk4P5hfQzAWttira1C38XuSgLSjDFJQDqwD30Xj8ta+xZQcdjmXvneeY9lW2vfsS7d/6vTe3WLAt2F0+5O9/d42+QwXnfbTOBdYLC1dp/30H5gsHf7WPtzoO/nh4B/AiLe/QKgylrb5t3vvD869pX3eLX3/IG+D8cApcBvvUMXvzHGZKDvYtSstXuBB4BduCCvBlah7+LJ6K3v3XDv9uHbu02BLlExxmQCfwT+wVpb0/kx71elTpc4BmPMFcBBa+0qv2tJcEm4bs//sNbOBOpxXZ0d9F08Pu847wLcj6NhQAYDq3ciJuLle6dAh73AiE73i71t4jHGJOPC/A/W2j95mw94XUV41we97cfanwN5P58DXGmM2YE7pHMh8FNcV1yS95zO+6NjX3mP5wDlDOx9CK7lssda+653/2lcwOu7GL1PAduttaXW2lbgT7jvp76L3ddb37u93u3Dt3ebAh1WAOO9UZ4puIEfz/lcU9zwjpc9Bmyw1j7Y6aHngPZRmp8Dnu20/e+9kZ5nAtVet9RLwCXGmDyvlXCJt63fs9Z+y1pbbK0djft+vWatvQl4HbjWe9rh+7B9317rPd9622/wRh6PAcbjBtMMCNba/cBuY8xEb9NFwHr0XeyOXcCZxph07//t9n2o72L3Ha6QGgAABNlJREFU9cr3znusxhhzpvff5O87vVf3+D16MB4uuFGJm3AjNb/tdz3xdAHOxXUlfQR86F0+jTuO9ldgM/AqkO893wC/8PblGmBOp/e6FTd4Zgtwi99/m0/783wOjXIfi/tHcAvwP0Cqtz3k3d/iPT620+u/7e3bjznJkbCJfAFOA1Z638dncKOF9V3s3j68F9gIrAV+hxupru/i8ffZItyYg9b/v737C62yjuM4/v64Bhn5B3GFBCmFtHZRS9PwoloU2D/oD+KIiuwP4k1E5E0QQXUhFBShZFHEJCkIJEF2kaGOmRou2tbCgrB2WwQSu0hv/Hbx/Z12Om7HZtnZHj4veNhznuf7POe3c8b57vnt2fdLzhQ9/V/+3AE3l/fjJLCDUvRtposrxZmZmVWAp9zNzMwqwAndzMysApzQzczMKsAJ3czMrAKc0M3MzCrACd1slpC0TdIdkh6U9OIMj+0o3bCGJd3aJK5Hpdtbk5huSffO5Pn/b5LGJS1t9TjMZhMndLPZ4xbgK+B2YHCGx94JjEXETRFx+F+Oo5usNWBmc4gTulmLSXpD0rfAGuAY8AywU9LLU8SukHSw9Fk+IOlqSd1kK8cHJI1Imt9wzN3K/uHfAA/XbV8r6Vi5qj8q6bpSLfFVoLecq3equCnGtUzSYDnmu9osgaSdkr5W9t9+pS5+vMxIjJT9qyR9LumkpC0lpqecs1/Zc/tdSed8Zkl6TNLxcq73lH3n2yT1lbGMSXr+gt4cs7mk1RV4vHjxEpDJfDvQDhxpErcPeKKsPwXsLeubgB1TxF9KdnhaSVaw+pTJSnULgUvK+l3AnqnONV1cw/O8QKmyCLQBC8r6krptA8AN5fE4kz233yIrvy0AOoBfyvYe4DRZxayNbDe5oe74pcD15TVpL9vfIUtnriZbVdbGt7jV77EXLxd7qRXjN7PWWgWMAp1kz/nprGPyKvsj8sq8mU6yGcePAJJ2A5vLvkXALkkryfK+7dOc45/EDQEfKhv57I2IkbJ9o6TNZKe0ZUAXmbxhsmfCGHB5REwAE5LOSFpc9h2PiJ/K2D8hSxHX+nZD/qlhNTCUZbCZTzbJ2AdcI2k70A/sb/IamVWCE7pZC5Xp8j6yw9JvwGW5WSPAuoj44yI+/WvAoYh4SNnrfuBC4yJiUNJtwH1An6Q3gcPAVmBNRJyS1EfOGNScKV/P1q3XHtc+mxprUzc+FrArIs65iVDSjcB6YAuwkZzRMKss/w3drIUiYiQiusnmQF3AQWB9RHRPk8yPkh3bAB4lk2YzPwArJF1bHj9St28Rk20aN9VtnyCnv88X9xdJy8mp8veBD8gZh4Vkz/LfJV0J3HOesU5lrbIT4jygF/iyYf8BYIOkK8o4lkhaXu6AnxcRe4CXynjMKs0J3azFJHUApyLiLNAZESeahD8LPFluonsceK7ZuSPiNDnF3l9uivu1bvfrwDZJw/x9tu4Q0FW7Ka5JXL0eYLTE9AJvR8QoMEz+UvExcKTZWKcxRHaf+h74Gfis4fs7QSbs/eU1+YKc2r8KGCgzHbuBGf0boNlc5G5rZjYrSeoBtkbE/a0ei9lc4Ct0MzOzCvAVupmZWQX4Ct3MzKwCnNDNzMwqwAndzMysApzQzczMKsAJ3czMrAKc0M3MzCrgTwX/Tqy41DS7AAAAAElFTkSuQmCC", "image/svg+xml": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "def plot_progressive_loss(obj_list, alias, result_interval=1):\n", " \"\"\"Show real-time progressive validation loss\n", " \"\"\"\n", " avg_list = [sum(obj_list[:i]) / i for i in range(1, len(obj_list))]\n", " total_obs = len(avg_list)\n", " warm_starting_point = 10 #0\n", " plt.plot(range(warm_starting_point, len(avg_list)), avg_list[warm_starting_point:], label = alias)\n", " plt.xlabel('# of data samples',)\n", " plt.ylabel('Progressive validation loss')\n", " plt.yscale('log')\n", " plt.legend(loc='upper right')\n", "plt.figure(figsize=(8, 6))\n", "plot_progressive_loss(loss_list_vanilla, 'VanillaVW')\n", "plot_progressive_loss(loss_list_autovw_ni, 'AutoVW:NI')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### AutoVW which tunes both namespace interactions and learning rate\n", "Create and run an AutoVW instance which tunes both namespace interactions and learning rate." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Seed namespaces (singletons and interactions): ['g', 'a', 'h', 'b', 'c', 'i', 'd', 'e', 'f']\n", "No low-cost partial config given to the search algorithm. For cost-frugal search, consider providing low-cost values for cost-related hps via 'low_cost_partial_config'.\n", "Created challengers from champion ||0.5|\n", "New challenger size 39, ['|gi|0.5|', '|af|0.5|', '|df|0.5|', '|gh|0.5|', '|ae|0.5|', '|di|0.5|', '|be|0.5|', '|ac|0.5|', '|hi|0.5|', '|de|0.5|', '|ef|0.5|', '|bc|0.5|', '|cf|0.5|', '|dg|0.5|', '|fg|0.5|', '|bh|0.5|', '|ei|0.5|', '|ce|0.5|', '|bf|0.5|', '|ah|0.5|', '|ad|0.5|', '|bg|0.5|', '|bd|0.5|', '|ab|0.5|', '|bi|0.5|', '|eg|0.5|', '|ai|0.5|', '|eh|0.5|', '|dh|0.5|', '|cd|0.5|', '|fi|0.5|', '|ci|0.5|', '|ag|0.5|', '|fh|0.5|', '|ch|0.5|', '|cg|0.5|', '||0.05358867312681484|', '||1.0|', '||0.5|']\n", "Online learning for 10000 steps...\n", "Seed namespaces (singletons and interactions): ['g', 'a', 'h', 'b', 'c', 'i', 'd', 'e', 'f']\n", "No low-cost partial config given to the search algorithm. For cost-frugal search, consider providing low-cost values for cost-related hps via 'low_cost_partial_config'.\n", "Created challengers from champion ||1.0|\n", "New challenger size 50, ['|gi|0.5|', '|af|0.5|', '|df|0.5|', '|gh|0.5|', '|ae|0.5|', '|di|0.5|', '|be|0.5|', '|ac|0.5|', '|hi|0.5|', '|de|0.5|', '|ef|0.5|', '|bc|0.5|', '|dh|1.0|', '|ah|1.0|', '|cd|1.0|', '|bh|1.0|', '|bi|1.0|', '|ab|1.0|', '|gi|1.0|', '|bg|1.0|', '|bd|1.0|', '|eh|1.0|', '|af|1.0|', '|hi|1.0|', '|cf|1.0|', '|ei|1.0|', '|ef|1.0|', '|ai|1.0|', '|ch|1.0|', '|gh|1.0|', '|fg|1.0|', '|ad|1.0|', '|ci|1.0|', '|bc|1.0|', '|ag|1.0|', '|df|1.0|', '|dg|1.0|', '|de|1.0|', '|di|1.0|', '|cg|1.0|', '|be|1.0|', '|eg|1.0|', '|ce|1.0|', '|fi|1.0|', '|ae|1.0|', '|bf|1.0|', '|fh|1.0|', '|ac|1.0|', '||0.10717734625362937|', '||0.3273795141019504|']\n", "Final progressive validation loss of autovw_nilr: 7.611900319489723\n" ] } ], "source": [ "from flaml.tune import loguniform\n", "''' create another AutoVW instance for tuning namespace interactions and learning rate'''\n", "# set up the search space and init config\n", "search_space_nilr = {'interactions': AutoVW.AUTOMATIC, 'learning_rate': loguniform(lower=2e-10, upper=1.0), 'quiet': ''}\n", "init_config_nilr = {'interactions': set(), 'learning_rate': 0.5}\n", "# create an AutoVW instance\n", "autovw_nilr = AutoVW(max_live_model_num=5, search_space=search_space_nilr, init_config=init_config_nilr)\n", "\n", "# online learning with AutoVW\n", "loss_list_autovw_nilr = online_learning_loop(max_iter_num, vw_examples, autovw_nilr)\n", "print('Final progressive validation loss of autovw_nilr:', sum(loss_list_autovw_nilr)/len(loss_list_autovw_nilr))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Online performance comparison between vanilla VW and two AutoVW instances\n", "Compare the online progressive validation loss from the vanilla VW and two AutoVW instances." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAFzCAYAAADIY/vqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdd3hc1Z3/8feZURn1LlmWZUvuveNOMSU2HQIBnA4hbLLLkpBkFzYbCMnubze7SViSDdmEkAQ2ydoQJxBq6MYGG3DBveIiW5at3stoNHN+f9yRbGPZHlsaj0b6vJ5HjzR37p35jnjwR+fcU4y1FhEREYlurkgXICIiIj2nQBcREekHFOgiIiL9gAJdRESkH1Cgi4iI9AMKdBERkX4gJtIF9ER2drYtKiqKdBkiIiLnxfr166ustTndPRfVgV5UVMS6desiXYaIiMh5YYwpOdVz6nIXERHpBxToIiIi/YACXUREpB+I6nvoIiISGp/PR2lpKW1tbZEuRULg8XgYMmQIsbGxIV+jQBcRGQBKS0tJSUmhqKgIY0yky5HTsNZSXV1NaWkpxcXFIV+nLncRkQGgra2NrKwshXkUMMaQlZV11r0pCnQRkQFCYR49zuW/lQJdRETCbuHChbzyyisnHHvkkUf46le/elav89xzz/GDH/wAgIceeogf/ehHAHzxi19k+fLlp7zuySefZMmSJSccq6qqIicnh7/85S/ccMMNXcf//d//nZEjR3Y9fv7557nuuuvOqs5IUKCLiEjYLVmyhGXLlp1wbNmyZSeF7Jlcd9113H///Wf9/jfeeCOvvfYaLS0tXceWL1/Otddey7x583jvvfe6jq9Zs4bU1FQqKioAWL16NfPmzTvr9zzfFOgiIhJ2N998My+++CLt7e0AHDhwgLKyMpYuXcrMmTOZMGEC3/3ud7vOLyoq4rvf/S7Tp09n0qRJ7Ny5E4AnnniCu++++7Tv9f3vf58LLriAiRMnctddd2GtJTU1lYsvvpjnn3++67zOPyhycnJITU3lo48+AuDw4cPcdNNNrF69GnACff78+b36+wgHjXIXERlgvvf8NraXNfTqa44fnMp3r51wyuczMzOZNWsWL7/8Mtdffz3Lli3jlltu4dvf/jaZmZn4/X4uu+wyNm/ezOTJkwHIzs5mw4YN/PznP+dHP/oRjz/+eEi13H333Tz44IMAfO5zn+OFF17g2muvZcmSJfzhD3/g1ltvpaysjN27d3PppZcCMH/+fFavXo3f72fUqFHMmTOHV155hWuuuYZNmzZxwQUX9PA3FH5qoQe9/v4fefq1n0a6DBGRfuv4bvfO1vHTTz/N9OnTmTZtGtu2bWP79u1d53/yk58EYMaMGRw4cCDk93nrrbeYPXs2kyZN4s0332Tbtm0AXH311bz77rs0NDTw9NNPc9NNN+F2uwGYN28eq1evZvXq1cydO5dZs2bx/vvv8+GHHzJ27Fg8Hk8v/RbCRy30oKc3PcJ+Vz23cE+kSxERCavTtaTD6frrr+fee+9lw4YNtLS0kJmZyY9+9CPWrl1LRkYGX/ziF0+YqhUfHw+A2+2mo6MjpPdoa2vjb//2b1m3bh2FhYU89NBDXa+ZkJDA4sWLeeaZZ1i2bBkPP/xw13Xz58/nv//7v/H7/Xz5y18mJSWFtrY2VqxYERX3z0Et9C4uDBYb6TJERPqt5ORkFi5cyB133MGSJUtoaGggKSmJtLQ0ysvLefnll3v8Hp3hnZ2dTVNT00kj35csWcLDDz9MeXk5c+fO7To+btw4ysrKeOedd5g2bRoAU6dO5Re/+EVU3D8HBfpxDAFN0RQRCaslS5awadMmlixZwpQpU5g2bRpjx47l05/+dK8EZ3p6Ol/+8peZOHEiixYtOune9xVXXEFZWRm33nrrCXO9jTHMnj2brKysruVW586dy759+6KmhW6sjd5W6cyZM21v7Yf+t49dxHZ3NSu+tK1XXk9EpC/ZsWMH48aNi3QZcha6+29mjFlvrZ3Z3flqoQcZXOpwFxGRqKVADzJGXe4iIhK9FOhBzqA4ERGR6KRADzK4CES6CBERkXOkQA9yGZe63EVEJGop0IMMRi10ERGJWgr0ICfQ1UQXEQmnZ599FmNM12Yrp/PII4+csDtad26//XZ++ctfnvQeV155Jffeey+PPPJI1/FFixZx5513dj3+5je/ecJqceBsyZqYmNi10xo4C+J093Nfo0APchlNWxMRCbelS5eyYMECli5desZzQwn0023L2rnhCkAgEKCqqqprXXc49bao2dnZ/PjHPw7l4/QpCvQgo5XiRETCqqmpiXfeeYdf//rXXSG8YsUKrrnmmq5z7r77bp544gl++tOfUlZWxsKFC1m4cCHg/DEwadIkJk6cyH333QfAZZddxs6dOzly5AgAzc3NvP7669xwww3MmzePNWvWALBt2zYmTpxISkoKtbW1eL1eduzYwfTp00+q84477uCpp56ipqYmrL+P3qbNWYJcxq176CIyMLx8Pxzd0ruvOWgSXPmD057yl7/8hcWLFzN69GiysrJYv379Kc+95557ePjhh3nrrbfIzs6mrKyM++67j/Xr15ORkcEnPvEJnn32WW644QZuuukmnn76ab72ta/x/PPPc8kll5CamkpqaioxMTEcPHiwaxe1w4cPs2bNGtLS0pg0aRJxcXE8+OCDzJw5k+uuuw5wutXvuOMOfvKTn/C9732vV39N4aQWepCzsIwh4PdHuhQRkX5p6dKl3HbbbQDcdtttIXW7d1q7di2XXHIJOTk5xMTE8JnPfIaVK1cC3W/L2unj26LOnTu363Hn2vHf//73u8K80z333MOTTz5JY2Njjz7z+aQWepAr+LdNwAZw4Y5wNSIiYXSGlnQ41NTU8Oabb7JlyxaMMfj9fowxXH/99QQCx/pHj98+NVTz5s3jyJEjbNq0idWrV59wT73zPvqWLVuYOHEihYWF/PjHPyY1NZXbb7/9lK+Znp7Opz/9aR599NGzridS1EIPMsFfRXtHe4QrERHpf5YvX87nPvc5SkpKOHDgAIcOHaK4uJhAIMD27dvxer3U1dXxxhtvdF2TkpLS1UKeNWsWb7/9NlVVVfj9fpYuXcrFF18MOD2st956K1/4whe48sor8Xg8Xa8xb948XnjhBTIzM3G73WRmZlJXV8eaNWvOuIvaN77xDX75y1+GvBd7pCnQg4xxfhV+vy/ClYiI9D9Lly7lxhtvPOHYTTfdxLJly7jllluYOHEit9xyS9de5AB33XUXixcvZuHCheTn5/ODH/yAhQsXMmXKFGbMmMH111/fde7x27Ieb9KkSVRVVTFnzpwTjqWlpZGdnQ3Agw8+yHPPPXdSzdnZ2dx44414vd5e+R2Em7ZPDfruk7fyZ7az6pNvkZ6S3SuvKSLSV2j71Oij7VPPkSu40b1PXe4iIhKFFOhBxjgD4fwa5S4iIlFIgR7UOcrd16F76CIiEn0U6EGdLfRAIDpGM4qIiBxPgR6ke+giIhLNFOhBndPWAlYtdBERiT4K9CCXBsWJiITdQNo+9ZJLLuHjU6tXrFhBWloaU6dOZezYsXzrW98K+fXORIEe5Aq20H1aWEZEJGz64/apTzzxBA899FDI51944YVs3LiRDz/8kBdeeIF333035GtPR4Ee5NKgOBGRsNL2qSdKSEhg6tSpHD58uFdeT5uzBHW20KNlzV4RkXP1Hx/8BztrztzlfTbGZo7lvln3nfYcbZ96otraWvbs2cNFF13UK6+nFnqQWugiIuHVn7ZPra6uZurUqUydOpUHH3yQX/ziF12Pt2w5/V7zq1atYsqUKRQUFLBo0SIGDRoU8u/hdNRCD+oc5d4R0D10EenfztSSDof+tn1qVlYWGzduBJx76AcOHAj5PvqFF17ICy+8wP79+5kzZw633HILU6dODf0Dn4Ja6EEuV2cLXaPcRUR6m7ZPPVlxcTH3338///Ef/9Err6dAD3IZp7PCr0AXEel1A3X71KuvvpohQ4YwZMgQPvWpT530/Fe+8hVWrlzJgQMHzvk9Omn71KBfPvttflb/PD8Z910unXVzr7ymiEhfoe1To4+2Tz1Hbpda6CIiEr0U6EFdK8VplLuIiEQhBXqQyxVcy10tdBERiUIK9KDOFrqmrYlIfxXNY6YGmnP5b6VAD+oMdKsWuoj0Qx6Ph+rqaoV6FLDWUl1dfcL0u1BoYZkgV3BQXMAGznCmiEj0GTJkCKWlpVRWVka6FAmBx+NhyJAhZ3WNAj2oay13vwbFiUj/ExsbS3FxcaTLkDBSl3uQ2xULQMAq0EVEJPoo0IM6R7lrHrqIiESjPtPlboy5AbgaSAV+ba199Xy+f0zwHrq1CnQREYk+YW2hG2N+Y4ypMMZs/djxxcaYXcaYj4wx9wNYa5+11n4Z+Apwazjr6rZWlxaWERGR6BXuLvcngMXHHzDGuIFHgSuB8cASY8z44075TvD588rduduaWugiIhKFwhro1tqVQM3HDs8CPrLW7rPWtgPLgOuN4z+Al621G071msaYu4wx64wx63pz+kXXoLiA5miKiEj0icSguALg0HGPS4PH/h64HLjZGPOVU11srX3MWjvTWjszJyen14rqnLamUe4iIhKN+sygOGvtT4GfRur9Y9yd09bU5S4iItEnEi30w0DhcY+HBI9FlKtrUJxWihMRkegTiUBfC4wyxhQbY+KA24DnIlDHCTpb6Jq2JiIi0Sjc09aWAmuAMcaYUmPMl6y1HcDdwCvADuBpa+22cNYRCpemrYmISBQL6z10a+2SUxx/CXgpnO99tjoD3WpzFhERiUJa+jUoxhUHaLc1ERGJTgr0IJdbC8uIiEj0UqAHHRsUpxa6iIhEHwV6kDu4OYtfLXQREYlCURnoxphrjTGP1dfX99prxsZ07ramFrqIiESfqAx0a+3z1tq70tLSeu01XcYJdN1DFxGRaBSVgR4ObndnoGtzFhERiT4K9KAYtzNtTV3uIiISjRToQe7gtLVV3lPu3CoiItJnKdCDYmOcaWtlsSbClYiIiJw9BXpQjCs20iWIiIicMwV6UEyMAl1ERKKXAj3I7Vagi4hI9FKgB8W4wrrxnIiISFgp0INi3Ap0ERGJXlEZ6OFY+rVztzUREZFoFJWBHo6lX48X8Gv5VxERiS5RGejh1uptiXQJIiIiZ0WB3o0Wb2OkSxARETkrCvRutLU1R7oEERGRs6JA70ZruwJdRESiiwK9G226hy4iIlFGgd4NBbqIiEQbBXo3vD4FuoiIRBcFejfa2lsjXYKIiMhZUaAfZ77XWaimvUOBLiIi0UWBfpyLht0IQLuvLcKViIiInB0F+nHiYjwAtHco0EVEJLpEZaCHY3MWgLhYJ9B9Hd5efV0REZFwi8pAD9fmLPGxCQC0K9BFRCTKRGWgh0tnoPv86nIXEZHookA/TnxcZ6C3R7gSERGRs3PGQDfG/KcxJtUYE2uMecMYU2mM+ez5KO58i49LAqDDry53ERGJLqG00D9hrW0ArgEOACOBfwhnUZFyrMtdLXQREYkuoQR6TPD71cAfrbW9O7S8D0kIdrn/b/vqCFciIiJydmLOfAovGGN2Aq3AV40xOUC/HDXm8SRHugQREZFzcsYWurX2fmAeMNNa6wOagevDXVgkJHqSIl2CiIjIOQllUNynAJ+11m+M+Q7we2Bw2CuLgMT4lEiXICIick5CuYf+gLW20RizALgc+DXwP+EtKzLi4uIjXYKIiMg5CSXQ/cHvVwOPWWtfBOLCV5KIiIicrVAC/bAx5pfArcBLxpj4EK+LSgvbc0gMBCJdhoiIyFkJJZhvAV4BFllr64BM+uk8dACPK4F2YyJdhoiIyFkJZZR7C7AXWGSMuRvItda+GvbKIiTWFU+HMbR5WyJdioiISMhCGeX+NeAPQG7w6/fGmL8Pd2FnqCks26cCxLmdgXH1TbW9/toiIiLhEkqX+5eA2dbaB621DwJzgC+Ht6zTC9f2qXAs0Jtb6nr9tUVERMIllEA3HBvpTvDnfnuTOc7tLP/a1NpvV7gVEZF+KJSlX38LvG+MeSb4+Aacuej9UlxMAngV6CIiEl3OGOjW2oeNMSuABcFDt1trPwxrVRHUueNaS1tThCsREREJ3Sm73I0xmZ1fONum/j74VRI81i95Yp313P9l60ORLUREROQsnK6Fvh6wHLtfboPfTfDn4WGsK2KsdT5mTUy/XTtHRET6oVMGurW2+HwW0ldMG30ZVC2LdBkiIiJnRc3Qj5k8ak6kSxARETlrCvRuzPWmMshnz3yiiIhIH6FA70aciaPNpUAXEZHoEco8dIwxbiDv+POttQfDVVSkxZt42vrt0jkiItIfnTHQg+u2fxcoBzr3FbXA5DDWFVFxbg9tLhc+Xzuxsdr6XURE+r5QWuhfA8ZYa6vDXUxf4XEngIXaxkpyMwsiXY6IiMgZhXIP/RAwoNZBjY9JBKCusTLClYiIiIQmlBb6PmCFMeZFwNt50Fr7cNiqijBPbDL4oK5hwHRKiIhIlAsl0A8Gv+KCX/1eYlwKtEBDc1WkSxEREQlJKJuzfA/AGJMcfNzvdy1JjE8FoLlVe6KLiEh0OOM9dGPMRGPMh8A2YJsxZr0xZkL4SzttTdcaYx6rrw/Prf1kTzoAjW0KdBERiQ6hDIp7DPiGtXaYtXYY8E3gV+Et6/Sstc9ba+9KS0sLy+snJTiB3tLeEJbXFxER6W2hBHqStfatzgfW2hVAUtgq6gPSk7MAaPX2+7sLIiLST4Q0yt0Y8wDwu+Djz+KMfO+3UjsD3adAFxGR6BBKC/0OIAf4c/ArJ3is38pIyQWgrnxnhCsREREJTSij3GuBe85DLX1GRko2AJWJFRGuREREJDSnDHRjzCPW2q8bY57HWbv9BNba68JaWQR1rt8eE9BmdCIiEh1O10LvvGf+o/NRSF8zqj2GRpcCXUREosMpA91auz7441Rr7U+Of84Y8zXg7XAWFmkJxNLqaol0GSIiIiEJpQn6hW6OfbGX6+hzPCaONnPSnQYREZE+6XT30JcAnwaKjTHPHfdUClAT7sIiLcF4aDGWNp8fT6w70uWIiIic1unuoa8GjgDZwI+PO94IbA5nUX1BrImn2RjGPfAS+39wbaTLEREROa3T3UMvAUqAueevnL7DTSJtLhcpNEa6FBERkTMKZXOWOcaYtcaYJmNMuzHGb4zp94ucJwYCAIwo+s8IVyIiInJmoQyK+xmwBNgDJAB3Ao+Gs6i+wMY6I9w/SvBHuBIREZEzC2mitbX2I8BtrfVba38LLA5vWZGXkj880iWIiIiELJRAbzHGxAEbjTH/aYy5N8TrotrtCx4CIL0DAgFNXxMRkb4tlGD+HOAG7gaagULgpnAW1RfkJOZwhS8Ng6WxrSPS5YiIiJxWKJuzlAR/bAW+F95y+paUmCSaqKO2xUtaYmykyxERETml0y0ss4VuNmXpZK2dHJaK+pCU2FR8HWVUNDZQlJ0c6XJERERO6XQt9GuC3/8u+L1zs5bPcpqg70/SPOnQBBW1B6F4cKTLEREROaVT3kO31pYEu9uvsNb+o7V2S/DrPuAT56/EkxljrjXGPFZfXx/W90lPdPZFr6k9GNb3ERER6alQBsUZY8z84x7MC/G6sLHWPm+tvSstLS2s75OdMgiATSUfhfV9REREeuqMg+KALwG/McakAQaoBe4Ia1V9RG7GEACO1hyOcCUiIiKnF8oo9/XAlGCgY60Nbz93H5KZVgBAdnJrhCsRERE5vdONcv+stfb3xphvfOw4ANbah8NcW8SlpQ0FYKdbXe4iItK3ne5eeFLwe8opvvq9xCTnHrrpaOdTv1gd4WpERERO7XTbp/4y+H1ALSZzApeL8V6LFx9rD9RGuhoREZFTOl2X+09Pd6G19p7eL6fvSbax1Lu19KuIiPRtpxsUt/68VdGHpboSWBfbAC5vpEsRERE5pdN1uT95Pgvpq4amZBHwNpI0/GEeXzWVOy/UtqoiItL3nHGBGGNMjjHmR8aYl4wxb3Z+nY/i+gJ/bBwArth6/vXFHRGuRkREpHuhrPj2B2AHUIyz29oBYG0Ya+pTCqyJdAkiIiJnFEqgZ1lrfw34rLVvW2vvAC4Nc119xs25sxnrbccTCJAQG9EVb0VERE4plITyBb8fMcZcbYyZBmSGsaY+JXbe17i6qYU2l4tWfwveDn+kSxIRETlJKIH+r8FlX78JfAt4HLg3rFX1JTFx5Iy4HICEIf/LnvKmCBckIiJyslAC/X1rbb21dqu1dqG1doa19rmwV9aH5CQ7e6HHJO3jmv9+J8LViIiInCyUQH/XGPOqMeZLxpiMsFfUB41ILT7hsbU2QpWIiIh074yBbq0dDXwHmACsN8a8YIz5bNgr60Oyhs4jxR/oevzNP26KYDUiIiInC2nYtrX2A2vtN4BZQA0wsBadyRvP7a3BwXDGx583aH90ERHpW0JZWCbVGPMFY8zLwGrgCE6wDyg5+dMBMDENxLo1N11ERPqWUFrom4CpwPettaOttfdZawfcOu+5wYFxiakf4olx6z66iIj0KafbnKXTcKv0IiUY6K7c12msvpzq5nayk+MjXJWIiIgjlEFxAz7MAUblTgYgwTq/skde3x3JckRERE6gtUxD5EnJ59LmFlpNAHfCAX7/3kFa27VqnIiI9A0K9FDljGOjx+liTyz6BQC3PrYmkhWJiIh0CWWU+2hjzBvGmK3Bx5ONMd8Jf2l9jMvFzLhs58cOJ9gPVDVHsiIREZEuobTQfwX8E8FNWqy1m4HbwllUX/VPtQ0AFFpnPffMpLhIliMiItIllEBPtNZ+8LFjHeEopq/LXvAtbq9r4HBMLLddUMDRhjYCAY0ZFBGRyAsl0KuMMSMAC2CMuRlncZmIMcZca4x5rL6+/vy+8ZRPU5g8hA4DQ3LaafMFGP7tl3hs5d7zW4eIiMjHhBLofwf8EhhrjDkMfB34SlirOgNr7fPW2rvS0tLO7xu7XAzLnQjA9pY/dx3+t5d2sq3sPP9xISIicpxQAr3EWns5kAOMtdYusNaWhLmuPmt4zhQAVpW/eMLxGx59NxLliIiIAKEF+n5jzGPAHKApzPX0edn5U8nvcIYQzBzl7Tru81ueWnuQrYfVUhcRkfMvlEAfC7yO0/W+3xjzM2PMgvCW1YdljezaSjUx/zmevGMWeanONLb7/rSFa/77nUhWJyIiA1QoS7+2WGufttZ+EpgGpAJvh72yvio5l0VeJ9Ctbefi0TlcPi7vhFPueGItFY1tkahOREQGqJBWijPGXGyM+TmwHvAAt4S1qj7uzuoK5ra2Ulu5HYB7Lht1wvNv7qxg2QeHIlGaiIgMUKGsFHcAZ2T7KmCStfYWa+2fwl1YX+aa+hkmetspxY8v4CMv1UNaQuwJ5zz82m6m/8trPPL6bjqCXfQiIiLhEkoLfbK19kZr7VJrrdY6BbjqhxQaD34Drx14jYAN8NevX8ifvjqP+68c23VaTXM7j7y+h5+v0Dx1EREJL3Oq3VGNMf9orf1PY8xPu3veWntPWCsLwcyZM+26desi8t4lz/0d19SuBOD2ibfzjRnf6HrOWkvxP710wvlfvrCYG6YVMGHweZ47LyIi/YYxZr21dmZ3z52uhb4j+H39Kb4GtGGDpnb9/Nutvz3hOWMMP7ltKtdOGdx17Fer9nP1T9+h6P4XeWrtQZavL9WysSIi0mtO2ULv9mRjXECytbYhfCWFLpItdA6+zxde/iwbPB4Avjnjm3xx4hdPOq213c+4B/962peaPzKLqYXpXDw6l+lD0+kIWDyx7nBULSIiUex0LfQzBrox5v9wlnr1A2txpq39xFr7w94u9GxFNNCtpez/ZbGosKDr0JYvbOn21LUHavjVyn0crGlh59HGs3qbm6YP4WuXjWJoVmKPyhURkeh3ukCPCeH68dbaBmPMZ4CXgftxutwjHugRZQyDU4ay7HAptxXk4zanvntxQVEmFxRlAlDf4uPdvVVsLq3nF2+febDcnzaU8qcNpdwxv5hvfmI0SfGh/CcTEZGBJpQW+jZgKvB/wM+stW8bYzZZa6ecjwJPJ6ItdICdL8GyJfwkI43H09NYfu1yClMKSYw9u9b0oZoWclLiOVrfxps7K/jhK7to9fm5cFQ2q/ZUnXDuVZMG8cA148lPS+jNTyIiIlGgp13u9wD3AZuAq4GhwO+ttRf2dqFnK+KBDrD2cf6w6rv8IMtpgS8oWMD/XP4/vfoW60tquOl/1pxwbFJBGnOGZ3LR6BxG5iZT3uAlMc7NsKxE4mN0/11EpD/qUaCf4gVjrLUdPa6sh/pEoAO+v9zN9Lpjq+Ge6l56T63cXckXfvsBZ/pPlp0cx6dmFvK5OcMYnK6WvIhIf9HTFvrXgN8CjcDjOOu532+tfbW3Cz1bfSXQWflDJu3/3xMOvXPbO6TFh2fO+e7yRv7rtd28tr2cjjNMfUv1xDCrOIslswq5aHQOse6QVvsVEZE+qKeBvslaO8UYswj4G+AB4HfW2um9X+rZ6TOBXlvCrp/P4JXkRH6V7oT430z+G+6edvd5eXufP0CMy2CMoa6lnec3lfHsxjLWl9SedG5mUhzL7prD6LyU81KbiIj0np4G+mZr7WRjzE+AFdbaZ4wxH1prp4Wj2LPRZwId4CEnyCcVDwXAYNj0+U0YYyJZFTuONLD0g4P875qSE47HuV0smVXIyNxkphZmsLu8kb2VTVw2LpcZwzIjVK2IiJxOTwP9t0ABUAxMAdw4wT6jtws9W30q0D/4Fbz0LXbExfKF/DxaXU7X9jPXPcPIjJERLs7R7O3gnY+q+JvfnXmhv9F5yfzz1eO5aFR2xP8oERERR08D3YUzbW2ftbbOGJMFFFhrN/d+qWenTwV6p4fSqHC7uWzosQVn+lKoA3T4A+w82shLW46csHHMRaNz2FPeyJH6E/dyT4xzs2BkNldOGsTVkwYTF6P78CIikdDTQDfAZ4Dh1trvG2OGAoOstR/0fqlnp08G+pbl8Kcv8fXcbN5Icuajzx40m8cXPR7hwkLX5vOzZl819y3fTEWj96Tn89M8XDM5n5lFmVQ1eYZZcO0AACAASURBVJlVlMmwrKQTgt7b4ae+xcfRhjY6Apac5Hiqm9vJSYnH77ekJsTQ7g+QmRhHjAbqiYiEpKeB/j9AALjUWjvOGJMBvGqtvaD3Sz07fTLQAVY9jPfN7/FOQgJfz8sBYMnYJdx3wX24XdE1R9xaS12Lj1UfVfHmjnK2ljXwUUXTSee5DEwsSOPCUdksX19KecPJfwicSkp8DInxbuYMz2JaYTpzRmQxJi9FXf0iIh/T00DfYK2dfvxAOK0UF4LgILnvZ2Xwx9RjI8rfXfIuqXGpkaqqV1Q0tvH2rkpe3HKEzMQ4mts72FPRxL7K5pPOzU6OoyA9gVi3C7+1DE5LYOXuSi4ek8MLm4+c8j1S4mO4bFwuM4symVqYzoTBqQp4ERnwehro7wPzgLXBYM/BaaFrlPvp7H0LfncDAIuHDOZw7LE12PtDqHfnw4O1rD1Qw4icZKYUppOdHB/ytYGApaalnT3lTazYXcF7e6vZVd5Imy8AQGFmAgtG5rB44iAWjMzG7VK4i8jA09NA/wxwKzAdeBK4GfiOtfaPvV3o2erTgQ5gLXwvHa+BX6andc1RB7hz0p3cPfXuqOuCP59a2jtYtaeKFbsq2F3e1DWvPjs5jpumD2HBqGwmDE4jMykuwpWKiJwf5xzowRHuc4Aa4DLAAG9Ya3eEo9Cz1ecDHaCtHn7gzE2vdxkWDCvsempyzmSeXPwkMS7toBaKykYvq/dW8Yf3D7K+pBZ/cJU8T6yLCYPTmD8ii6sm5zN2UP/r/RARgZ630PvEIjLdiYpAB1j5Q3jzXwGocrtYOHTICU8/e/2zjEgfEYnKolZ1k5c3dlSwrqSGgzUt1Ld2sOtoAwELxdlJzBmeyaDUBKYUpjFneBaeWPWEiEj062mg/whYA/zZnstOLmEUNYEOsH8lLP8SNFcA8ERqCj/Oyuh6en7BfB6c8yCDkwdHqsKoV9HQxrMbD7N8fSn7KptPWOd+dF4yF47K4YapBUws0AA7EYlOPQ30RiAJ6ADacLrdrbU24v2aURXonV78Jqx15qQ/n5TIt3OzT3j6hxf9kMXFiyNRWb8SCFjK6lt5b18Nmw7Vsb6klu1HGgBnHv30YRmMzUvhYE0LuanxzBiWwbwR2WrJi0if1uvbp/YVURno4IyAf+qz0O7M5340PY1fZBwbMDc+azz/d9X/acBcL6toaOO5TWW8ur2cTYfq8HYEiI9x4e1wRtJnJcUxqziT7OR4Lhqdw0Wjs7W3vIj0KT1toXe3q1o9UBLpPdGjNtDBGQH/6ndgzc+6Dv06LYVHMo91w//XJf/F5cMuj0R1/Z61ltLaVrKS4+gIWNYfqOUP75fw+o6KE84bk5fC9GEZLBiZzezhmWQmxuHSlDkRiZCeBvp7OFPWtgQPTQK2AmnAVyO5L3pUB/rx1j4O7/wE6g/y16RE/uG4bvhbRt/CA3MfiGBxA4u1ln1VzRysaWHV7irWldSw80gj7X6nFe92GWYMy+CysbksGJXNuEGpCngROW96Guh/Bh6w1m4LPh4PfB/4R5yBclN7ud6Q9ZtA77T3TfjdjTQbwycL8ikLLkYzKmMUT13zFLGu2AgXODA1ezv4YH8NHx6qY3NpHSXVLeyvclbFi4txMbs4k6GZiYzNT2XcoBTGD04lMU5TEUWk9/U00Ldaayd2d8wYs1GB3suaKuCFe2HnC5TGuLmy0Nm1bXDSYJ65/hkSYxMjXKAA7Kt0FrpZuaeKDSW1HK5rPeH5ETlJTC3MYMawDMblpzA6L4WkeIW8iPRMTwP9KZyFZZYFD90KZAOfA96J5CYt/TLQO7XWwpbl+F76FtOLh3YdfnLxk0zP625Yg0SSt8PPvspmtpTWs/1IA7uONvLhodqupWvdLsOwzEQGpycwsSCNKUPSWDAqmxSPel1EJHQ9DfQE4G+BBcFD7wI/x5nClmitPXnrrfOkXwd6p/Zm7A9H8Y/pHv6anATAA3Me4JYxt0S4MDkTb4efrYcbOFDVzKbSOo7Ut1FW18ru8kZ8fkuMyzA8J4nx+anMHp7F6LwURuYmk5agkBeR7vV42poxJg4YA1hgl7XW17slnpsBEegA/g7s72/gqy07eTcxAYAbRt7AA3MeIM6tdcyjTWu7n3UlNazZW822sgbWHaihud3f9fyo3GSmD82gMDOBSUPSmV2cqfnxIgL0vIV+Cc6mLAdwFpUpBL5grV3Zu2WevQET6ACBAPzhJv5S/j7fycnqOvzt2d/mxpE34onxRLA46YlAwBlZ/1FFE9vL6ll7wFkEp77V+bvZGCjKSmLByGwWjMpm+tAMspI0fU5kIOppoK8HPm2t3RV8PBpYaq2d0euVhsgYcy1w7ciRI7+8Z8+eSJURGaXrqPvNFXyicDCtLtcJTz1y8cNcOuxyLWvaT1Q0tPHe/hq2ldWz+2gj7++voSXYks9P8zBmUApFWUlMHpLG6LwUxgxKIdbtOsOrikg062mgb7bWTj7TsUgYUC3047W3YP/vUxwqfY8783M5EnPi6OnXb36dvKS8CBUn4dLeEWDDwVo2HqpjS2k9Ww7Xc6S+FZ//2P/DOSnxDM9OIiMxjnH5qYzNT2FqYTp5qerBEekPehrovwX8wO+Dhz4DuK21d/RqledgwAZ6J18rrH+Sqlf/iV+np/L7tGPL69+WN5d/uPQR4uI0za0/a+8IsL+qmZ1HG9h4qI6qpnaO1LVS3dzOgepmOv/3HpGTxMSCNBaOyWVkbjJF2UkkaxqdSNTpaaDHA3/HsVHuq4CfW2u9vVrlORjwgX68QAC7/S88tOIb/DklGYAYa3ly3r8zefS1ES5OIqGxzdfVkv9gfw0bD9VR3dwOONPoBqV6GDsohbkjspgzPIuhWYmkahqdSJ92zoFujHED26y1Y8NVXE8o0Lvh76Bu61P82zsP8nKy0zof7OvgN5f+jILhl0W4OImkDn+ADw/Vsbu8kZLqFg7VtLDraCP7gqveGQOTh6QzY2gGY/NTmDwkjTF5KRqTIdKH9LSF/hfg7621B8NRXE8o0E9v7/Y/seSD79Ia/Af5cX8Ws5c8AwkZZ7hSBpIj9a28v6+GnUcbeX9/NdsON3StXZ/iiWF0nnMfflJBGiNykhmRm6SlbUUipKeBvhKYBnwANHcet9Ze15tFngsF+pnVNh7lb178NDu8lQAMb/fxaNY8hnzyNxGuTPoqf8Cyv6qZ9SU1bD3cwObD9ew80tC1zSzA4DQPEwvSmFiQxtTCdCYWpJGRGKvWvEiY9TTQL+7uuLX27V6orUcU6KHbV7Wdr79yJ/s7GgF4mgLGXf5vUKBlZOXMvB1+9pQ3sbeyidLaVnYdbWTr4fqu7nqApDg3+ekJjMlzNqgZnO6hODuZYZmJpCXEYnHu3YvIuTunQDfGeICvACNxtk79daT3P/84BfrZsdbyXskb3PX2vQAsaGnl9voGZo79FK7LHoCUQRGuUKJNQ5uP9SW1fFTeRGltCwdrWthyuIGqpu7HzKYlxJKVHEdRVhLZyXFkJceTkxxPXqqHQWnxjMpL0cA8kdM410B/CvDhjGq/Eiix1n4tbFWeAwX6uSlpKOGfX/97NjXu7zr26fpGrmtqYkLeDLj1d5CcG8EKJdo1tvn4qKKJA9XNHK33UtPsxeUyNLT6OFrfxtEGL2V1rV2r4R0vLzWezKR4clPiyU/zMDg9gaGZiYzMTaY4O0m71smAdq6BvsVaOyn4cwzwgbW2T/XPKtDPnbWWdUfXcserXzrh+PB2H4+WVzCkcD4seQo0j13CxFqLtyNAdXM7tc3tHKxpYV9lE3srm6lraXfm1Ne3Ud3s5fh/ptISYklPjCUjMY7h2UkUZCRQkJ5ATko82cnxpCbEMjQzUd370i+da6BvOD7AP/64L1Cg946ShhJWlq7klf1/ZVPVZgBur2vga7V1uIfMgmt/AnnjI1ylDFRN3g4O1bRQUt3M3spmjtS3Utvso7LRS2ltC0cb2gh87J+xhFg3Y/NTGJ+fyvjBqYwdlEJhRiI5KfEauCdR7VwD3c+xUe0GSABagj9ba21qtxeeRwr03rerZhffevtbHGg4gMvCP1fXcEtjE8SlwLWPwKSbI12iyAl8/gBH69vYV+W07L2+ADuONrC9rIHtRxpobDs29CfFE8P4/FSyU5x798XZSRRnJzEqL5lBqR6FvfR5Pd4+ta9SoIeHP+Dn0Y2P8qstvwLAWMv/q6zmmuYWTM5YuOJfYORl4NKWntK3WWsprW1lT0Ujh2qcvei3lTVQ0dBGQ1sHTd5jYZ+RGMvI3GQK0hMYnJ7A6LwU0hJiKcpOYkhGgja+kT5BgS7npKq1irvfuJtt1dsAcGP4YWU1VzQ1OSfMvRuGzoVRV0BMfAQrFTl71loqm7zsq2xmd3kj28sa2FvZRFldG+UNbXQc14/vdhkmDE5lYoGzet6ovGRG5iSTnRyvbWzlvFKgS480tDfwX+v/i+W7lwNQ6ErgW0cOcmlLq3OCJw0uuBPmfx08Eb8TI9JjPn+AA1XNVDZ5Ka1tZX9VMxtKatl5tPGEkfkuAymeWDKT4nC7DFlJcWQkxpGRFEtCbAxJ8W5i3S7iYpzWfV2Ljw5/AG9HAG+HnzZfgPTEWBLjYqhobKPZ20FCrJu4GBcZiXHkpDhT+nKD9/4T49zEB18rIymOzMQ4jEG3CgYQBbr0iraONn615Vf8dutv8QWcf9SuTBzG3x4poagmuDLwgnth0CRoa4DETCiYCUk5EBMXwcpFeoe1lspGL7vKGzlQ1eyMwm9qp7LJiwEa2zqoaWmnstFLa7u/awnd47kMxLpdeGLddP7729DmBHmyJ4b4GBcNrT7afIFur/+4GJdhUJqHrGRnql9yfAwZiXFdr1WYmUiqJ4aRucnEx7jJSopTr0IUU6BLrzrSdIQfrvshr5W81nVscvJQ/mXnewz3nWLtofwpkD4Uhl8Ck2+D+OTzUqtIJHX4A3QELIHgv7PxMW5c3bSoO/8dPv64tZaG1g6ONrRxtKGNFm8HLe1+fP4AvoDF7w9wpL4Nn99ysKaF+lZnml+bz0+z10+rz99tTTEuQ6zbRW5qPCmeGAalJpCf5mFQmofi7CTaOwJkJccxOi+lq2dA+g4FuoTF0eajBGyA32z9DU/vehqLZZQnh8+7cxnV4WeYdZPc4YW9b558ccEMp+U+/QswejG4NOBIpLd0/jHgdht2HW2ksc3HgapmfH5LbUs7h+taaWn3U9HQRl2rj5Lqlm5fJynOTUKcm/y0BLKT48hOjmd4TjKD0z1kJcWTmeTcFkhNiKHDb0mMc+sPgDBToEvYlTSU8M0V32RX7a4TjucmOivOfXvmPzI/NhNPyRpY/wQ0V4G33jkpMQvSh8GCr8OoRRDrOc/ViwxsgYCl1ednf1Uz9a0+rIV9VU3sOtrIwZoWjDHUNrdT3tBGRWP3y/oCpCfGUpydxLj8VDISY8lMiicvNZ6hmYkMzUwkPVG33npKgS7nzebKzTy39zlWlq7kSPORk56fmTeT0Rmj+fy4z1Lg98PuV2Hrcjj0vnOCccG0z8HUz0DhLGeTbhHpM+pbfVQEg72y0Utjm4/6Vh/+ABxtaGNfZRM7jjTQ6O3g4/GSnRzP6DxndsDg4Op+mUnOyn4F6YnkpmjWwJko0CWidtXs4oltT7CzZieHmw7T2uGMjk+MSeT6kddz56Q7yY1Jhp0vwtY/wZ5XwfqhcA5MuQ2mLFGrXSTKBAKW+lYfRxvaOBhc6W9PeRN7KpqobvZyNHj//3hxbheD0z0MyUgkxRMT7O73UJiRyKA0D+mJcXT4AxgD6YlxZCXFkZYwsLbtVaBLn9Hub+el/S+xs2Ynz+99nob2BgAKUwq5dcytXD/ietLbGuHD3ztfDaXOhTO+CLO/CrljI1e8iPSaQMBZB6C+1cfhulZKa1sprW3hcK3zc2Wjl4C1VDR68X98bd/jJMS6KchIYEhGgtPyT/OQn55AqieW3FSnJyArKQ5PbP9YCEuBLn3W9urtPL3raZ7b+9yxqXDFV/L58Z9nQvpozJ5XYeMf4KPXwd/u3GO/6j8hoyiyhYvIedHhDzgj/evbqGtx/o2IcRvqWnxUNTkt/UO1LRyua6W6ybnP313+ZybFkZ0cx+D0BAaleijMTGRYViK5KR4yEmNJT4zrWk+gL1OgS5/nD/hZW76WxzY/xpbKLbT525iSM4XPjPsMlw29jLi2Bnjnv+CDxyDQAeNvgPn3wOBpkS5dRPqQ9o4AVU1eGtp8lAe36a1u8lJW30ZVo5fDda2UN7RR1dR+0rVulyEn2RnIl5fqCX4d/7OHQakeUhNiItbNr0CXqFLRUsEL+17g6V1Pc7jpMAD3zriX28bcRmJLDaz4AWxa6gR75ggYexVMvEnhLiIh69zFr6rJS22Lj9pmZ0Gg8uC8/4oGL+WNx3oFjueJdTkBn+IhNzWeQcGwz0yKIz0xlpyUeHJTPGQlx/X6HgAKdIlKHYEOVpet5h/e/gdaOlrI9GRy3Yjr+NToTzHU5YENT8K2v0D5FueCghlw/c91n11Eek2bz98V7kfrnXX+nS9v189HG9po83W/qt/Vk/N59NO9t/O4Al2imrWWdeXr+N3237GydCUBG2B2/mw+N/5zXFhwIaahDLY/C6t+DO3NzmI1Ez8JQy7QjnAiEnbWWhraOqhtbqeu1UdlcEpfRWMbhRmJ3DRjSK+9lwJd+o2jzUf5w44/8Nze56hpq2FoylBuGXMLlxZeSqErHl75Z9j2Z6c7PiUfpn/emdeeXhjp0kVEekyBLv1Oa0crL+9/mT/t/hObqzYDMDt/NteNuI6rcy/AvW8lfPBLOLzeWaymaIEzn33CJzWnXUSilgJd+rVt1dt4o+QNlu5cSpOviYLkAhYWLmRR0SKmEu/MZ9+0DFqqIDkPZt3lDKLLLI506SIiZ0WBLgOCtZY3Dr7BU7ueYn35enwBHyPTR3LbmNu4qvhKUkreg3cehoNrwLhh6Bxwx0H+ZBh5ORRdqKVmRaRPU6DLgFPvref5vc/z9O6n2V+/n+TYZBYXL+az4z7LCK/XWWJ22zNOgFd/5Fw0aDLMvB3GXQdJ2ZH9ACIi3VCgy4BlrWVDxQae2vkUrx18DX/AzyWFl3Db2NuYmz/XWRzC2whb/wxrHoWq4G5xQ+c6y82OuQo8qRH9DCIinRToIkBVaxVPbnuSp3Y9RWtHK0WpRczIm8GUnClcNOQisuLTYd9b8NEbzkYxdSXOhUPnwbTPOAPq4hIj+yFEZEBToIscp7Wjlb/u/yvLdi1jb91evH4vca44rh95PV+e9GXyk/PB3+Hca9/1Eux4AeoPOhcXXQiTb4UJN0J8cmQ/iIgMOAp0kVMI2ACbKjfx3N7neO6j52gPtDMlZwrXDr+W60ZeR0JMAgT8cGAVbFkO+9+GuoPgioGRVzjbu45erKlwInJeKNBFQnCk6Qi/3fZb3jz4JuUt5STEJHBJ4SUsKlrExUMuJsYVA9bCwfdg5wtOwDcdBU+aM0p+2DwYew2kDIr0RxGRfkqBLnIWOgfSPb/3ed44+AZ13jryEvO4bOhlfHLUJxmTOcY5MeCHfStg81Ow720n3DFQMN0ZTDfmKsgbH8mPIiL9jAJd5Bz5/D5Wlq7kj7v/yLtl7wIwLnMcN4++mcXFi0mNC46ADwTg6CbY+ZLTeq/Y7hzPHgNjr3a65YfOjtCnEJH+QoEu0gvqvfU8s+cZnvnoGfbV78Pj9nDFsCu4ZsQ1zB40G/fxG8E0lDld8tufhbIPwQYgPg2KL4Tii2DSpyAxM3IfRkSikgJdpBdZa9lWvY3lu5fz6oFXafQ1khCTwFXFV3Hp0EuZN3iec7+9U0uNs4jN4Q3OoLr6Q85KdcUXwvjrYfwNCncRCYkCXSRMvH4vbx96mzcPvckbJW/Q5m8j05PJRUMu4saRNzItd5qzeM3xjm5xFrLZ/heo2QsYKJwFwxfCiEude/Du2Ih8HhHp2xToIudBU3sT75a9yysHXuHtQ2/THmgnPymf2fmzWVS0iLn5c0/slgenO373q8589yObAOt0zY9Y6IycH70IknMj8nlEpO9RoIucZ03tTbxW8horDq1g1eFV+AI+XMbFjLwZXDb0Mq4qvooMT8aJFzVXw/4VsOd12PMKtFSDKxYKZ8OIS5xR87njtYGMyACmQBeJoGZfM6tKV7GufB0rS1dypPkIMa4YFhQs4Jrh13BJ4SXEu+NPvMhap8W+dbmzUl3tfud4RrGzr/vYqyFvgsJdZIBRoIv0ITuqd/Dc3ud4cd+L1HprSYlN4dKhl7JgyALm5s8lLT7t5IvqD8Puvzqj5vevdI6lD3O2gB1xKYz6hAbWiQwACnSRPsgX8LGmbA0v73+Zt0vfprG9EbdxMy13GgsLF7KoaBF5SXknX9h41Nk8Zt9bcOBdaK1xRs3nT3FWqyu+CPKnQko314pIVFOgi/RxHYEOtlZtZWXpSt44+Ab76vcBcMGgC7hg0AUsGLyA8VnjTx5U5/dB6TrnnvveN+HIZiD4/3TmCGdBm3HXQOEccLnO74cSkV6nQBeJMvvq9/HKgVf46/6/doV7XmIeVxZfyTXDr2F0xuiTp8MB+Frh0AfO6Pn9K51NZfztkFoA466DyZ+CwdN1710kSinQRaLYoYZDrDmyhjcPvsl7R97Db/0UphSysHAhc/LnMCt/1smD6jp5m2DXy7D1T04L3u+F2CQYNtdpvRdfBNmjFfAiUUKBLtJP1LbV8lrJa7x56E0+OPJB13S46bnTGZs5lqm5U5mRN4PshOyTL25rgG1/dlrv+1ZA7QHneEYxTLjBCfghs9Q1L9KHKdBF+qGm9iY2VGzg3cPv8v6R99lbv7frueFpw5meN525+XOZXzCfpNikEy+21gn0vW84G8rsWwHWD6lDnHvuwy9xWu9xH7tORCJKgS4yAHj9XjZWbOTDig/ZWLmRjRUbafY1E+OKYWHhQj4x7BMsHLqw++75lhr46A3Y8kdnvfmONnDHOaPmR14OxRfDoEnqmheJMAW6yADk8/vYVLmJ1w++zkv7Xuqa8z45ZzIzB81kZt5MJmRPINb1sXXjO7xwcA3seQ32vApVu53jaYXO1Lihc50WvBa2ETnvFOgiA1zABvjg6Af8ec+f2Vy5mcNNhwFIjk1mfsF8Li28lIsLLz65ax6g4YgT7HtedUbQN1c4x1MGw5jFzqI2RRdCfPJ5/EQiA5MCXUROUNtWy9qja3nn8DusOryKqtYq4t3xXFJ4CZcPvZz5BfNJiUvp/uK6g7D3LfjoNfjoTfA1AwayRzmD6govcOa9Z4/WADuRXqZAF5FTCtgAH1Z8yMv7X+aVA69Q563DZVxMzp7MgoIFXFx4MaMzRuMy3YRzhxdK3oWD78PB1c7WsK21znMJGU73/LD5kDMGUgY5IR9ziil2InJGCnQRCYk/4GdT5SZWl61mZelKdtbsxGK79nifmDWRaXnTGJE24uRV68AZPV+9Fw6959yH378K6kqOOyHYks+bCPmTne85Y5z787ofL3JGCnQROSeVLZWsObKGVaWreLv0bVo7WgHI9GRywaALuHjIxVw05KLuN5TpVF/qtNybyqFmH1R95DyuP3jsnIQMp/U+bL6zF3zeRG02I9INBbqI9Ji1lj11e9hYsZENFRt4/8j7VLVW4TZuJmRPYErOFCZkTaAotYihqUNPfQ++U2O5s8hN7QGo3AlHN0PZRmc+PEDaUMgbDzljnV3l8iZC2hC15GVAU6CLSK8L2ADbqrbx1qG3+ODoB2yt2oo/GMYGw8iMkUzLmcbMQTMpTismIz6D3MTc7teg7+RthJI1TsAfXu+05Gv3gw04z6cWODvJ5U859pUySCEvA4YCXUTCzh/ws6NmB+vL19Pka2Jz5WY2V26mydfUdU6mJ5Oi1CLGZ41nQvYERqWPYkT6CGJcMad+YV+r05I/utW5L1++Far20LWrXFKuE+w5YyA5F3InwJCZkJAe3g8sEgEKdBGJCH/Az7bqbeyp3UNDewP76vdxoP4AO2t20uZvA5zWfH5SPmMyxzAjbwbzBs9jWOow4txxp35hb5MT7Ec2Hfuq2HGsux4go8jpph88DQqmO98TMsL7gUXCLCoC3RgzHPhnIM1ae3Mo1yjQRaKTL+Bjd81udtbs5EjzEQ40HGBjxUbKW8oBcBs3I9JHMDZzLEWpRYzJHMPk7Mmke07T6rYW2uqcbvpD7zv34yt2QM2xNe5JGey05uOSnMDPnwzJg5xV77QwjkSBiAW6MeY3wDVAhbV24nHHFwM/AdzA49baHxz33HIFusjAY63laPNR3i17l7KmMrZVb2NnzU5q2mq6zhmeNpxpudMYlzmOEekjKEorIiM+o/spdJ1aa51wL9vgdNuXrnVG3hvXsRa9cUHOuGArPh1yxzmD8eKSIXM4xJymt0DkPIpkoF8ENAH/2xnoxhg3sBu4AigF1gJLrLXbg88r0EWkS21bLbtqd7G1aisbyjewsXIjje2NXc8nxyYzPms8U3OnUpRaxPC04YzKGHXqLvvOf/N8rU63fW0JVO+B0nXOQLz2Zgj4jp3vioXMYiiY6ayCN3i6s1HN6f6IEAmTiHa5G2OKgBeOC/S5wEPW2kXBx/8EYK399+Dj0wa6MeYu4C6AoUOHzigpKTnVqSLSD1lrKW8p56O6j9hXt4+DjQf5sOJDdtfu7jon1hXL2MyxTMmZwsxBM5mQNYG8xLzTj7DvFAg48+UrtjnhXrnL2aDm0AfQUuWcE5cCOaOde/KBDojxOMezRjpd+f52p9XvSccZvGcgJc953f/f3r0Hx3mVdxz/PpJWV+u+si35KlvyRZKN7VxIQktdYEgIpKGdlKSlEKCUoZ3p0LRMB4ZOGdphmNIO5dZCaaDhUgI0UApkOpBCmECA3IhjyZItyZJtRb5pJXklW9bF1ukfhfsxnQAAEuBJREFU5+xq7diKZcva1er3mXln3/e877579vi1nj3nPe85Y0NwJuZbCUZe9OewHJ82ecbPdDdxGgrLfEtBSQ1EiiEnz/fmH4/7HyPlq6Gg1M+KV1jmz1NY7jsJVjdAbgSmz6t1IctkWkC/B7jDOffusP024JXAh4GP4mvuDyYC/GxUQxeRhPhEnBNjJ+iN97Ivto/WWCutsVYmzk8AUJpfSmNFI42VjWwo30BDRQNbq7e+/PPyCYk55F981o+EF+uCkX7/SN30eR/UTx32AflK5ESgJArLVvgAPdgNa272zfyW4wPyUK9/hC+1sx/4Y85P+uWSjOQPCRyU1vr8RxuheiOMj8DyJn87om4n5OZBZb1/UiAn4lsf9ChgRpotoM/yrMjCcs4NAu9Ndz5EZHEqLyinvKCcTZWbuH397QBMnp9k3+A+DgwdoGu4i65TXTza8+gFj9IV5RWxLbqNLVVb2LF8B+vK1lFfVk8k96JpZc1803tVPWz//Utn4vw5GD3mg3xuxNemnfPj148eg3g/rLvVB9hEjfvlnAtBOycXMP8eM/8jYvSY/yFxsh3OT0GkCEaO+h8HA/thcsy/b/SYD+JDvX4An/E47PvO5T+zuNoH/Iq1vkVh+dYw2U6uHwegukET72SgjGtynwvV0EVkrpxznBw7SfepbtoH2+k/3U/7YDs98Z5kbT6SE6GxspHKwko2VWyiKdrEjpodrCxZmebczwPnYGIELNe3OLjz0P9r33wf64LRozA17vsVxLohpb/CDPOtCwVlULHG1+qLKn0HwuoG/wOgsBzK6tTXYJ5lWg39GaDRzOqBfuA+4A/TkA8RWYLMjBUlK1hRsoJXrXpVMn3q/BStsVb6T/fTOdxJ53AnsbEYTx97mqnQSa6upI5dK3Zx44obWV++PjkC3hXdm88UZj7YAqwMDx/VvuLSxzrnZ9Qz8zX+WJffHurxI/idPunv9w92+0cGk039QW6+H91v2XJYud037yf6BWgY33l3vXu5PwzsBqLACeDDzrkvmtmdwCfxj619yTn30as5v2roInK9TZ2fonO4kz0De3juxHM8d+K5Cx6ly8/JpyRSAkBDZQMNFQ1UF1ZTlFfEsvxlFOQWUFFQwdqytURyIsQn4uRaLjk5OZybPkdtSe2V38fPVIk4cn7SB/djL/h+AcOH/I+A0yd8x8LJmVsdFJT5VoFIsa/VV2/0HQor1vl7/fklvlOhOvVdYFEMLHM1FNBFZKE55zg8cpgjo0foG+3jxJkTxM763u+HRw7TE++54B79lSjMLUy2ApRESqgtqWV16Wrqy+vZXLWZsvwyKgsqqSqsojS/lEhO5KX3+DPd9Hkf7E+0+c54J/f7bTfte/gPHXxph0LL8TX5qo2+Ob9y3Uy/hNJa/wMg2ujv9S+Rpn0FdBGRBTQ1PcXY1BjxiThnz51ldHKUvtE+4hNxygrKKMgtYGh8iEhOhNHJUY6eOcqyyDJOTZxi2k0Tn4jTN9rHkZEjnHPnLvkZq5atYkP5BpYXL6e+vJ66ZXVsqtxESaSE6sLqxXUbAPzjgqeP+1r9UI/v0DcW8+uDB33AH4/7Yy33wp7/eYW+lh9thOjm8LrJp+UXp+XrXC8K6CIii9D4uXE6hzsZGBvgzLkzDJ4dZOzcGJ1DnRTkFtA+1M7JsZPJeeoTKgoqks/iF+YVsqNmBzesuIGygjJWFK+YfZz8TDY+4p8eyCv09++HenxTfqzTN+3HOv2jg4nZ+cBPw1uzyQf4RKCPbvL38Rfbjx4U0EVEspZzjqNnjnLizAl64j3EJ+IcHjnMnoE9HD19NNlzP8EwaopqWFmyktWlq2moaGBN2ZrkKHuLNtgnTI372nxqkE+sT43NHFdYMRPcE4G+ZrO/h5+bMU90v4QCuojIEjU1PcXQ2SH2DOxhbGqM42eO03+6nyOjR9gX28fk9MzgNHmWR3lBOatLV9MSbaG5upnivGLWlq2lvrx+9mluM930tB8I6CWBvtN32kvIifgOetFNfjz/5VtmBt3JL0lf/oOsC+hmdhdwV0NDw590dXWlOzsiIotWfCKebNY/MHwgWcvfP7T/gqb8wtxCokVRGiobaKpqYkPFBmqKalhbtnZx3rNPdfaU76AX6wxD/Xb5x/SGey9svi9bNfOsfc1mH/CjjX4WvwUaaCfrAnqCaugiItfHuelz9MR76BvtY2xqjI6hDvpG+jg0cojDI4dxKc+bF+UVUVlQyabKTbREW2iJtrC+fD11JXWLO9BPjftAP9TjA3yig17swEwHPbiwU17NFr9esdZvF1XOa5YU0EVEZN6MTY1xZPQIg2cHOTRyiBdHXyR2NkbncCc98Z7kcZUFlTRFm2ip9s33LdEWaopr0pjzeeKc75Q3sN/frx88ONN8P3yYCwbX2XoX3Pu1eftoBXQREVkQo5OjtA+20xvvpX2wnbbBNg6eOsh0aLpeXryclmpfi2+qbmJt2VpWL1u9uGvyqRID6gwf9gG+tPbyY/9fBQV0ERFJm7GpseSc9m2xNvYN7uPwyMzU15UFlbREW9gW3ZZ8rSisSGOOM1emjeUuIiJLSHGkmJ3Ld7Jz+c5kWnwiTvtgO32jfbTF2miNtfLz/p8n782vKV1DS7SF7dHttERbks/Uy+Wphi4iIhnhzNQZ2gfbaY21JoP88TPHAf9IXWNlIxsrNvphccvqaY42s7Z0bfY0118BNbmLiMiiNDA2kAzwe2N76Y33EjsbS96TL8svSzbVb6/ZTnN1M9VF1WnO9fWjgC4iIllj8vwkvfHeZC1+b2zvJTveba/ZngzyxZHsGNNdAV1ERLJa4ln5tlib710fa+PI6BEAciyHjRUbaaluYXPVZpqrmxftPXkFdBERWXKGx4dpjbX6ZaCVjqGO5Fz2eZbH5qrNbK/ZzrboNpqqm1hXti7jh7fNuoCuoV9FRGSunHMMnB3w9+MH9rI3tpe2WFtyiNvC3EK2Vm9lW3Qb22q2sT26ndqS2ozqdJd1AT1BNXQREbkWiSFuDwwdSDbVdwx1JGepK42UsrlqM03VTTRXN7Mtuo3VpekbCEcBXURE5ApNTU/ROdxJ60Ar3ae66Rjs4MDwgWSQryiouGAgnIXsWa+BZURERK5QJCdCc3UzzdXNybSp6SkOnjqYvB/fGmvlyf4nkwPh1JXU0RxtTg5pu7VqK+UF5Quab9XQRURErkJiIJx9sX20DfphbftP9yf315XUcUf9HTxwwwPz9pmqoYuIiMyzkkgJN628iZtW3pRMGx4fpmOog/1D++kY7KAwd+EejVNAFxERmSeVhZXcVncbt9XdtuCfnbPgnygiIiLzTgFdREQkCyigi4iIZAEFdBERkSyggC4iIpIFFNBFRESywKIM6GZ2l5l9IR6PpzsrIiIiGWFRBnTn3Pedc+8pL1/YYfVEREQy1aIM6CIiInIhBXQREZEsoIAuIiKSBRTQRUREsoACuoiISBZY1POhm9kAcHgeTxkFYvN4vqVIZXjtVIbXTmU4P1SO126+y3Cdc67mUjsWdUCfb2b27OUmjpcrozK8dirDa6cynB8qx2u3kGWoJncREZEsoIAuIiKSBRTQL/SFdGcgC6gMr53K8NqpDOeHyvHaLVgZ6h66iIhIFlANXUREJAsooANmdoeZHTCzbjP7QLrzk0nMbI2ZPW5m7Wa2z8zeF9KrzOwxM+sKr5Uh3czs06Es95rZrpRz3R+O7zKz+9P1ndLFzHLN7Hkz+0HYrjezp0JZfdPM8kN6QdjuDvvXp5zjgyH9gJndnp5vkj5mVmFmj5jZfjPrMLNbdS3OjZk9EP4vt5nZw2ZWqGtxdmb2JTM7aWZtKWnzdt2Z2Q1m1hre82kzs6vKqHNuSS9ALnAQ2ADkAy8ATenOV6YsQC2wK6yXAp1AE/Bx4AMh/QPAP4T1O4H/BQy4BXgqpFcBPeG1MqxXpvv7LXBZ/iXwdeAHYftbwH1h/fPAn4b1PwM+H9bvA74Z1pvC9VkA1IfrNjfd32uBy/DLwLvDej5QoWtxTuW3CugFilKuwXfoWnzZcns1sAtoS0mbt+sOeDoca+G9b7iafKqGDjcD3c65HufcJPAN4O405yljOOeOOed+HdZHgQ78H4W78X9cCa9vDut3A19x3q+ACjOrBW4HHnPODTnnhoHHgDsW8KuklZmtBt4IPBi2DXgN8Eg45OIyTJTtI8Brw/F3A99wzk0453qBbvz1uySYWTn+D+sXAZxzk865U+hanKs8oMjM8oBi4Bi6FmflnHsCGLooeV6uu7CvzDn3K+ej+1dSzjUnCug+OPWlbL8Y0uQiobltJ/AUsMI5dyzsOg6sCOuXK8+lXs6fBP4amA7b1cAp59y5sJ1aHsmyCvvj4filXob1wADwH+HWxYNmVoKuxSvmnOsH/gk4gg/kceA5dC1ejfm67laF9YvT50wBXa6ImS0Dvg38hXNuJHVf+FWpxyUuw8zeBJx0zj2X7rwscnn4Zs/POed2AmfwTZ1JuhZnF+7z3o3/cVQHlLC0Wieui0y57hTQoR9Yk7K9OqRJYGYRfDD/T+fcd0LyidBURHg9GdIvV55LuZxfBfyOmR3C39J5DfApfFNcXjgmtTySZRX2lwODLO0yBF9zedE591TYfgQf4HUtXrnXAb3OuQHn3BTwHfz1qWtx7ubruusP6xenz5kCOjwDNIZenvn4jh/fS3OeMka4X/ZFoMM594mUXd8DEr007wf+JyX97aGn5y1APDRL/RB4vZlVhlrC60Na1nPOfdA5t9o5tx5/ff3EOfdW4HHgnnDYxWWYKNt7wvEupN8Xeh7XA434zjRLgnPuONBnZptD0muBdnQtzsUR4BYzKw7/txNlqGtx7ublugv7RszslvBv8vaUc81NunsPZsKC75XYie+p+aF05yeTFuA38E1Je4E9YbkTfx/tx0AX8H9AVTjegH8JZdkK3JhyrnfhO890A+9M93dLU3nuZqaX+wb8H8Fu4L+AgpBeGLa7w/4NKe//UCjbA1xlT9jFvAA7gGfD9fhdfG9hXYtzK8OPAPuBNuCr+J7quhZnL7OH8X0OpvAtRX88n9cdcGP49zgIfJYw6NtcF40UJyIikgXU5C4iIpIFFNBFRESygAK6iIhIFlBAFxERyQIK6CIiIllAAV0kQ5jZx8zst83szWb2wTm+tybMhvW8mf3mLMfttjDb2yzH7DCzO+fy+QvNzA6ZWTTd+RDJJAroIpnjlcCvgN8Cnpjje18LtDrndjrnfnaN+diBH2tARBYRBXSRNDOzfzSzvcBNwC+BdwOfM7O/vcSx683sJ2Ge5R+b2Voz24GfyvFuM9tjZkUXvecO8/OH/xr4vZT0m83sl6FW/wsz2xxGS/w74N5wrnsvddwl8lVrZk+E97QlWgnM7HNm9qz5+bc/knL8odAisSfs32VmPzSzg2b23nDM7nDOR83Puf15M3vJ3ywz+yMzezqc69/Mzzufa2YPhby0mtkDV/WPI7KYpHsEHi1atDjwwfwzQAR4cpbjvg/cH9bfBXw3rL8D+Owlji/Ez/DUiB/B6lvMjFRXBuSF9dcB377UuS533EWf81eEURaBXKA0rFelpP0U2B62DzEz5/Y/40d+KwVqgBMhfTcwjh/FLBc/3eQ9Ke+PAltDmURC+r/ih868AT9VZSJ/Fen+N9ai5XovicH4RSS9dgEvAFvwc85fzq3M1LK/iq+Zz2YLfjKOLgAz+xrwnrCvHPiymTXih/eNXOYcV3LcM8CXzE/k813n3J6Q/hYzew9+prRaoAkfvGFmzoRWYJlzbhQYNbMJM6sI+552zvWEvD+MH4o4MW83+FsNNwDP+GGwKcJPkvF9YIOZfQZ4FPjRLGUkkhUU0EXSKDSXP4SfYSkGFPtk2wPc6pw7ex0//u+Bx51zv2t+rvufXu1xzrknzOzVwBuBh8zsE8DPgPcDNznnhs3sIXyLQcJEeJ1OWU9sJ/42XTw29cXbBnzZOfeSToRm9grgduC9wFvwLRoiWUv30EXSyDm3xzm3Az85UBPwE+B259yOywTzX+BnbAN4Kz5ozmY/sN7MNobtP0jZV87MNI3vSEkfxTd/v9xxSWa2Dt9U/u/Ag/gWhzL8nOVxM1sBvOFl8nopN5ufCTEHuBf4+UX7fwzcY2bLQz6qzGxd6AGf45z7NvA3IT8iWU0BXSTNzKwGGHbOTQNbnHPtsxz+58A7Qye6twHvm+3czrlxfBP7o6FT3MmU3R8HPmZmz3Nha93jQFOiU9wsx6XaDbwQjrkX+JRz7gXgefyPiq8DT86W18t4Bj/7VAfQC/z3Rd+vHR+wfxTK5DF80/4q4KehpeNrwJweAxRZjDTbmohkJDPbDbzfOfemdOdFZDFQDV1ERCQLqIYuIiKSBVRDFxERyQIK6CIiIllAAV1ERCQLKKCLiIhkAQV0ERGRLKCALiIikgX+H/lJboJZbH5nAAAAAElFTkSuQmCC", "image/svg+xml": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(8, 6))\n", "plot_progressive_loss(loss_list_vanilla, 'VanillaVW')\n", "plot_progressive_loss(loss_list_autovw_ni, 'AutoVW:NI')\n", "plot_progressive_loss(loss_list_autovw_nilr, 'AutoVW:NI+LR')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### AutoVW based on customized VW arguments\n", "You can easily create an AutoVW instance based on customized VW arguments (For now only arguments that are compatible with supervised regression task are well supported). The customized arguments can be passed to AutoVW through init_config and search space." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Seed namespaces (singletons and interactions): ['g', 'a', 'h', 'b', 'c', 'i', 'd', 'e', 'f']\n", "Created challengers from champion |supervised||classic|\n", "New challenger size 37, ['|supervised|fg|classic|', '|supervised|dh|classic|', '|supervised|ef|classic|', '|supervised|ei|classic|', '|supervised|di|classic|', '|supervised|ch|classic|', '|supervised|bh|classic|', '|supervised|cf|classic|', '|supervised|ae|classic|', '|supervised|bc|classic|', '|supervised|ci|classic|', '|supervised|eg|classic|', '|supervised|ag|classic|', '|supervised|be|classic|', '|supervised|bd|classic|', '|supervised|ce|classic|', '|supervised|af|classic|', '|supervised|ad|classic|', '|supervised|ab|classic|', '|supervised|dg|classic|', '|supervised|gh|classic|', '|supervised|bg|classic|', '|supervised|fh|classic|', '|supervised|gi|classic|', '|supervised|cg|classic|', '|supervised|cd|classic|', '|supervised|ai|classic|', '|supervised|ac|classic|', '|supervised|bi|classic|', '|supervised|eh|classic|', '|supervised|fi|classic|', '|supervised|de|classic|', '|supervised|hi|classic|', '|supervised|bf|classic|', '|supervised|df|classic|', '|supervised|ah|classic|', '|supervised||classic|']\n", "Online learning for 10000 steps...\n", "Seed namespaces (singletons and interactions): ['df', 'g', 'a', 'h', 'b', 'c', 'i', 'd', 'e', 'f']\n", "Created challengers from champion |supervised|df|classic|\n", "New challenger size 43, ['|supervised|ce_df|classic|', '|supervised|df_gi|classic|', '|supervised|df_fi|classic|', '|supervised|bd_df|classic|', '|supervised|ab_df|classic|', '|supervised|bi_df|classic|', '|supervised|df_ei|classic|', '|supervised|bh_df|classic|', '|supervised|cd_df|classic|', '|supervised|df_dfg|classic|', '|supervised|def_df|classic|', '|supervised|bdf_df|classic|', '|supervised|ag_df|classic|', '|supervised|cg_df|classic|', '|supervised|df_dg|classic|', '|supervised|af_df|classic|', '|supervised|ci_df|classic|', '|supervised|df_dh|classic|', '|supervised|ah_df|classic|', '|supervised|df|classic|', '|supervised|df_di|classic|', '|supervised|ad_df|classic|', '|supervised|df_ef|classic|', '|supervised|ae_df|classic|', '|supervised|ai_df|classic|', '|supervised|be_df|classic|', '|supervised|df_eg|classic|', '|supervised|ch_df|classic|', '|supervised|ac_df|classic|', '|supervised|df_gh|classic|', '|supervised|df_fg|classic|', '|supervised|bc_df|classic|', '|supervised|df_dfh|classic|', '|supervised|df_fh|classic|', '|supervised|df_dfi|classic|', '|supervised|de_df|classic|', '|supervised|bf_df|classic|', '|supervised|bg_df|classic|', '|supervised|df_hi|classic|', '|supervised|cdf_df|classic|', '|supervised|df_eh|classic|', '|supervised|cf_df|classic|', '|supervised|adf_df|classic|']\n", "Average final loss of the AutoVW (tuning namespaces) based on customized vw arguments: 8.828759490602918\n" ] } ], "source": [ "''' create an AutoVW instance with ustomized VW arguments'''\n", "# parse the customized VW arguments\n", "fixed_vw_hp_config = {'alg': 'supervised', 'loss_function': 'classic', 'quiet': ''}\n", "search_space = fixed_vw_hp_config.copy()\n", "search_space.update({'interactions': AutoVW.AUTOMATIC,})\n", "\n", "autovw_custom = AutoVW(max_live_model_num=5, search_space=search_space) \n", "loss_list_custom = online_learning_loop(max_iter_num, vw_examples, autovw_custom)\n", "print('Average final loss of the AutoVW (tuning namespaces) based on customized vw arguments:', sum(loss_list_custom)/len(loss_list_custom))\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "interpreter": { "hash": "4502d015faca2560a557f35a41b6dd402f7fdfc08e843ae17a9c41947939f10c" }, "kernelspec": { "display_name": "Python 3.8.10 64-bit ('py38': conda)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 2 }