haystack/tutorials/Tutorial5_Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "MGSXn0USOhtu",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Evaluation of a QA System\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial5_Evaluation.ipynb)\n",
    "\n",
    "To be able to make a statement about the performance of a question-answering system, it is important to evalute it. Furthermore, evaluation allows to determine which parts of the system can be improved."
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Prepare environment\n",
    "\n",
    "#### Colab: Enable the GPU runtime\n",
    "Make sure you enable the GPU runtime to experience decent speed in this tutorial.\n",
    "**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/deepset-ai/haystack/master/docs/_src/img/colab_gpu_runtime.jpg\">"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Make sure you have a GPU running\n",
    "!nvidia-smi"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E6H_7lAmOht8"
   },
   "source": [
    "## Start an Elasticsearch server\n",
    "You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in your environment (eg., in Colab notebooks), then you can manually download and execute Elasticsearch from source."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vgmFOp82Oht_",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Install the latest release of Haystack in your own environment \n",
    "#! pip install farm-haystack\n",
    "\n",
    "# Install the latest master of Haystack\n",
    "!pip install grpcio-tools==1.34.1\n",
    "!pip install git+https://github.com/deepset-ai/haystack.git\n",
    "\n",
    "# If you run this notebook on Google Colab, you might need to\n",
    "# restart the runtime after installing haystack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tNoaWcDKOhuL",
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# In Colab / No Docker environments: Start Elasticsearch from source\n",
    "! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q\n",
    "! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz\n",
    "! chown -R daemon:daemon elasticsearch-7.9.2\n",
    "\n",
    "import os\n",
    "from subprocess import Popen, PIPE, STDOUT\n",
    "es_server = Popen(['elasticsearch-7.9.2/bin/elasticsearch'],\n",
    "                   stdout=PIPE, stderr=STDOUT,\n",
    "                   preexec_fn=lambda: os.setuid(1)  # as daemon\n",
    "                  )\n",
    "# wait until ES has started\n",
    "! sleep 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "w0MHgxrYOhur",
    "outputId": "9e530bf3-44b1-4ea1-86e2-8be0bb9163ad",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from haystack.modeling.utils import initialize_device_settings\n",
    "\n",
    "device, n_gpu = initialize_device_settings(use_cuda=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 87
    },
    "colab_type": "code",
    "id": "tTXxr6TAOhuz",
    "outputId": "99a4e32b-e0ec-4c94-dab3-1a09c53d4dc1",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from haystack.preprocessor.utils import fetch_archive_from_http\n",
    "\n",
    "# Download evaluation data, which is a subset of Natural Questions development set containing 50 documents\n",
    "doc_dir = \"../data/nq\"\n",
    "s3_url = \"https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/nq_dev_subset_v2.json.zip\"\n",
    "fetch_archive_from_http(url=s3_url, output_dir=doc_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make sure these indices do not collide with existing ones, the indices will be wiped clean before data is inserted\n",
    "doc_index = \"tutorial5_docs\"\n",
    "label_index = \"tutorial5_labels\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B_NEtezLOhu5",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Connect to Elasticsearch\n",
    "from haystack.document_store.elasticsearch import ElasticsearchDocumentStore\n",
    "\n",
    "# Connect to Elasticsearch\n",
    "document_store = ElasticsearchDocumentStore(host=\"localhost\", username=\"\", password=\"\", index=\"document\",\n",
    "                                            create_index=False, embedding_field=\"emb\",\n",
    "                                            embedding_dim=768, excluded_meta_data=[\"emb\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "bRFsQUAJOhu_",
    "outputId": "56b84800-c524-4418-9664-e2720b66a1af",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from haystack.preprocessor import PreProcessor\n",
    "\n",
    "# Add evaluation data to Elasticsearch Document Store\n",
    "# We first delete the custom tutorial indices to not have duplicate elements\n",
    "# and also split our documents into shorter passages using the PreProcessor\n",
    "preprocessor = PreProcessor(\n",
    "    split_length=500,\n",
    "    split_overlap=0,\n",
    "    split_respect_sentence_boundary=False,\n",
    "    clean_empty_lines=False,\n",
    "    clean_whitespace=False\n",
    ")\n",
    "document_store.delete_documents(index=doc_index)\n",
    "document_store.delete_documents(index=label_index)\n",
    "document_store.add_eval_data(\n",
    "    filename=\"../data/nq/nq_dev_subset_v2.json\",\n",
    "    doc_index=doc_index,\n",
    "    label_index=label_index,\n",
    "    preprocessor=preprocessor\n",
    ")\n",
    "\n",
    "# Let's prepare the labels that we need for the retriever and the reader\n",
    "labels = document_store.get_all_labels_aggregated(index=label_index, drop_negative_labels=True, drop_no_answers=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gy8YwmSYOhvE",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Initialize components of QA-System"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JkhaPMIJOhvF",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Initialize Retriever\n",
    "from haystack.retriever.sparse import ElasticsearchRetriever\n",
    "retriever = ElasticsearchRetriever(document_store=document_store)\n",
    "# Alternative: Evaluate DensePassageRetriever\n",
    "# Note, that DPR works best when you index short passages < 512 tokens as only those tokens will be used for the embedding.\n",
    "# Here, for nq_dev_subset_v2.json we have avg. num of tokens = 5220(!).\n",
    "# DPR still outperforms Elastic's BM25 by a small margin here.\n",
    "# from haystack.retriever.dense import DensePassageRetriever\n",
    "# retriever = DensePassageRetriever(document_store=document_store,\n",
    "#                                  query_embedding_model=\"facebook/dpr-question_encoder-single-nq-base\",\n",
    "#                                  passage_embedding_model=\"facebook/dpr-ctx_encoder-single-nq-base\",\n",
    "#                                  use_gpu=True,\n",
    "#                                  embed_title=True,\n",
    "#                                  max_seq_len=256,\n",
    "#                                  batch_size=16,\n",
    "#                                  remove_sep_tok_from_untitled_passages=True)\n",
    "#document_store.update_embeddings(retriever, index=doc_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 725,
     "referenced_widgets": [
      "398e8dc496594a5f8e65daecc3ddad4a",
      "ee436c1e0fb24802b5d0706299ce7e81",
      "9f5b31e32c5c4398a6eede32431ad55e",
      "479f015ad2e8412d96a34ddb59a054d1",
      "eb25be6e7fcc49cfbf9815d69f849637",
      "f7ebc8ee77094382a2b26f879cdf8613",
      "93e868a4b6384840b3d245391ee2915a",
      "e4a33236e1954bbf83e57ff15c40ff7f",
      "18250ec6840147658d1e038138f8aba0",
      "a8ba7f1398f2401b8d0139aabdf6ce1f",
      "49daa700f016433e88b492490b1a8d89",
      "08adb4accbe649fd9d21da2246c00d63",
      "c840113abcc44f84a88e6120fe198fba",
      "47e0a33131714f31bb8b4a456b54d060",
      "75ed8c01b4ed4c37a024a6c24036e35a",
      "7675cb36a32e49b1b14d301bf29fa668",
      "ec3c00e615164fb488ebfb51d8ac9d9e",
      "9da0f76f26294b27982e080b8af6e28b",
      "ef78454fe89347ea8f6f3148e23440db",
      "bbbeec7c73c24059bffd602af5dcbd62",
      "cb53b988f7df44d29ff7efcb0a236fce",
      "8b9a20a51500438c9d6e84d9f71e8cbd",
      "082e03f9f75c4546bd5b2326b33e8017",
      "9c78a3ef3d804f89b4102a5029415e35",
      "9597bad322c34d02a0c10dd66b21813e",
      "62661d74e0ea462cb6b8e574aa17dc2f",
      "71416028306340ee8987d703b506245b",
      "100352c2499749e48ed10f3dee4f569d",
      "dd518bb2a8d141f58118e2cf7e140ffb",
      "5957d73fcef34606ac8eac654b9584d4",
      "5c8d0662ec12422a83525292cc6a51ac",
      "45469f8ddb3b4df8aab3e7ca3fbe6921",
      "6c5fb91a498840a1a2cae914660f8684",
      "1314e2ff61264f81acaf0042efd0cf5a",
      "fce574414e154590a4a642362db98421",
      "7937f15a8f874591bb693ce1546c7c5b",
      "706c079c7d484975ad69a91b3f95e9f3",
      "4473fa7bf766425bb704bb9772c021fd",
      "ef088eda6fe04abe958059620da9b5d0",
      "e1819597e4d840b19cfa8aba00ef3ef1",
      "19173b1d00314580a4f40efa0df4b174",
      "d7ce404f2f784b0f93abf8f3ba592e18",
      "9da1585dcdd440fa8b2e8351ffeba348",
      "8880e647ca4a4796b571f5b983b541b4",
      "01d9cd6656494aef83a3947945c0acdd",
      "b01601a11a324141b2d88190a87ef950",
      "c7db7b055dbe4361965ca4ab8c028536",
      "c1852f69951e4d80aecb5a6c942b2ca6"
     ]
    },
    "colab_type": "code",
    "id": "cW3Ypn_gOhvK",
    "outputId": "89ad5598-1017-499f-c986-72bba2a3a6cb",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Initialize Reader\n",
    "from haystack.reader.farm import FARMReader\n",
    "\n",
    "reader = FARMReader(\"deepset/roberta-base-squad2\", top_k=4, return_no_answer=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from haystack.eval import EvalAnswers, EvalDocuments\n",
    "\n",
    "# Here we initialize the nodes that perform evaluation\n",
    "eval_retriever = EvalDocuments()\n",
    "eval_reader = EvalAnswers(sas_model=\"sentence-transformers/paraphrase-multilingual-mpnet-base-v2\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qwkBgzh5OhvR",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Evaluation of Retriever\n",
    "Here we evaluate only the retriever, based on whether the gold_label document is retrieved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "YzvLhnx3OhvS",
    "outputId": "1d45f072-0ae0-4864-8ccc-aa12303a8d04",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "## Evaluate Retriever on its own\n",
    "retriever_eval_results = retriever.eval(top_k=20, label_index=label_index, doc_index=doc_index)\n",
    "## Retriever Recall is the proportion of questions for which the correct document containing the answer is\n",
    "## among the correct documents\n",
    "print(\"Retriever Recall:\", retriever_eval_results[\"recall\"])\n",
    "## Retriever Mean Avg Precision rewards retrievers that give relevant documents a higher rank\n",
    "print(\"Retriever Mean Avg Precision:\", retriever_eval_results[\"map\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fjZRnB6bOhvW",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Evaluation of Reader\n",
    "Here we evaluate only the reader in a closed domain fashion i.e. the reader is given one query\n",
    "and one document and metrics are calculated on whether the right position in this text is selected by\n",
    "the model as the answer span (i.e. SQuAD style)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 203
    },
    "colab_type": "code",
    "id": "Lgsgf4KaOhvY",
    "outputId": "24d3755e-bf2e-4396-f1a2-59c925cc54d3",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Evaluate Reader on its own\n",
    "reader_eval_results = reader.eval(document_store=document_store, device=device, label_index=label_index, doc_index=doc_index)\n",
    "# Evaluation of Reader can also be done directly on a SQuAD-formatted file without passing the data to Elasticsearch\n",
    "#reader_eval_results = reader.eval_on_file(\"../data/nq\", \"nq_dev_subset_v2.json\", device=device)\n",
    "\n",
    "## Reader Top-N-Accuracy is the proportion of predicted answers that match with their corresponding correct answer\n",
    "print(\"Reader Top-N-Accuracy:\", reader_eval_results[\"top_n_accuracy\"])\n",
    "## Reader Exact Match is the proportion of questions where the predicted answer is exactly the same as the correct answer\n",
    "print(\"Reader Exact Match:\", reader_eval_results[\"EM\"])\n",
    "## Reader F1-Score is the average overlap between the predicted answers and the correct answers\n",
    "print(\"Reader F1-Score:\", reader_eval_results[\"f1\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7i84KXONOhvc",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Evaluation of Retriever and Reader (Open Domain)\n",
    "Here we evaluate retriever and reader in open domain fashion i.e. a document is considered\n",
    "correctly retrieved if it contains the answer string within it. The reader is evaluated based purely on the\n",
    "predicted string, regardless of which document this came from and the position of the extracted span."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "yLpMHAexOhvd",
    "outputId": "fd74be7d-5c8e-4eb9-a653-062427b74347",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from haystack import Pipeline\n",
    "\n",
    "# Here is the pipeline definition\n",
    "p = Pipeline()\n",
    "p.add_node(component=retriever, name=\"ESRetriever\", inputs=[\"Query\"])\n",
    "p.add_node(component=eval_retriever, name=\"EvalRetriever\", inputs=[\"ESRetriever\"])\n",
    "p.add_node(component=reader, name=\"QAReader\", inputs=[\"EvalRetriever\"])\n",
    "p.add_node(component=eval_reader, name=\"EvalReader\", inputs=[\"QAReader\"])\n",
    "results = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-1-ae565b9a84f2>, line 10)",
     "output_type": "error",
     "traceback": [
      "\u001B[0;36m  File \u001B[0;32m\"<ipython-input-1-ae565b9a84f2>\"\u001B[0;36m, line \u001B[0;32m10\u001B[0m\n\u001B[0;31m    index=doc_index,\u001B[0m\n\u001B[0m        ^\u001B[0m\n\u001B[0;31mSyntaxError\u001B[0m\u001B[0;31m:\u001B[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# This is how to run the pipeline\n",
    "for l in labels:\n",
    "    res = p.run(\n",
    "        query=l.query,\n",
    "        labels=l,\n",
    "        params={\"index\": doc_index, \"Retriever\": {\"top_k\": 10}, \"Reader\": {\"top_k\": 5}},\n",
    "    )\n",
    "    results.append(res)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# When we have run evaluation using the pipeline, we can print the results\n",
    "n_queries = len(labels)\n",
    "eval_retriever.print()\n",
    "print()\n",
    "retriever.print_time()\n",
    "print()\n",
    "eval_reader.print(mode=\"reader\")\n",
    "print()\n",
    "reader.print_time()\n",
    "print()\n",
    "eval_reader.print(mode=\"pipeline\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## About us\n",
    "\n",
    "This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany\n",
    "\n",
    "We bring NLP to the industry via open source!  \n",
    "Our focus: Industry specific language models & large scale QA systems.  \n",
    "  \n",
    "Some of our other work: \n",
    "- [German BERT](https://deepset.ai/german-bert)\n",
    "- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)\n",
    "- [FARM](https://github.com/deepset-ai/FARM)\n",
    "\n",
    "Get in touch:\n",
    "[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)\n",
    "\n",
    "By the way: [we're hiring!](https://apply.workable.com/deepset/) "
   ],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Tutorial5_Evaluation.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "01d9cd6656494aef83a3947945c0acdd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "082e03f9f75c4546bd5b2326b33e8017": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "08adb4accbe649fd9d21da2246c00d63": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7675cb36a32e49b1b14d301bf29fa668",
      "placeholder": "​",
      "style": "IPY_MODEL_75ed8c01b4ed4c37a024a6c24036e35a",
      "value": " 499M/499M [00:41&lt;00:00, 11.9MB/s]"
     }
    },
    "100352c2499749e48ed10f3dee4f569d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_45469f8ddb3b4df8aab3e7ca3fbe6921",
      "placeholder": "​",
      "style": "IPY_MODEL_5c8d0662ec12422a83525292cc6a51ac",
      "value": " 456k/456k [00:04&lt;00:00, 92.6kB/s]"
     }
    },
    "1314e2ff61264f81acaf0042efd0cf5a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "18250ec6840147658d1e038138f8aba0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_49daa700f016433e88b492490b1a8d89",
       "IPY_MODEL_08adb4accbe649fd9d21da2246c00d63"
      ],
      "layout": "IPY_MODEL_a8ba7f1398f2401b8d0139aabdf6ce1f"
     }
    },
    "19173b1d00314580a4f40efa0df4b174": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_9da1585dcdd440fa8b2e8351ffeba348",
       "IPY_MODEL_8880e647ca4a4796b571f5b983b541b4"
      ],
      "layout": "IPY_MODEL_d7ce404f2f784b0f93abf8f3ba592e18"
     }
    },
    "398e8dc496594a5f8e65daecc3ddad4a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_9f5b31e32c5c4398a6eede32431ad55e",
       "IPY_MODEL_479f015ad2e8412d96a34ddb59a054d1"
      ],
      "layout": "IPY_MODEL_ee436c1e0fb24802b5d0706299ce7e81"
     }
    },
    "4473fa7bf766425bb704bb9772c021fd": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "45469f8ddb3b4df8aab3e7ca3fbe6921": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "479f015ad2e8412d96a34ddb59a054d1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e4a33236e1954bbf83e57ff15c40ff7f",
      "placeholder": "​",
      "style": "IPY_MODEL_93e868a4b6384840b3d245391ee2915a",
      "value": " 559/559 [00:54&lt;00:00, 10.3B/s]"
     }
    },
    "47e0a33131714f31bb8b4a456b54d060": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "49daa700f016433e88b492490b1a8d89": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_47e0a33131714f31bb8b4a456b54d060",
      "max": 498637366,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c840113abcc44f84a88e6120fe198fba",
      "value": 498637366
     }
    },
    "5957d73fcef34606ac8eac654b9584d4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5c8d0662ec12422a83525292cc6a51ac": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "62661d74e0ea462cb6b8e574aa17dc2f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c5fb91a498840a1a2cae914660f8684": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_fce574414e154590a4a642362db98421",
       "IPY_MODEL_7937f15a8f874591bb693ce1546c7c5b"
      ],
      "layout": "IPY_MODEL_1314e2ff61264f81acaf0042efd0cf5a"
     }
    },
    "706c079c7d484975ad69a91b3f95e9f3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "71416028306340ee8987d703b506245b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5957d73fcef34606ac8eac654b9584d4",
      "max": 456318,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_dd518bb2a8d141f58118e2cf7e140ffb",
      "value": 456318
     }
    },
    "75ed8c01b4ed4c37a024a6c24036e35a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7675cb36a32e49b1b14d301bf29fa668": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7937f15a8f874591bb693ce1546c7c5b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e1819597e4d840b19cfa8aba00ef3ef1",
      "placeholder": "​",
      "style": "IPY_MODEL_ef088eda6fe04abe958059620da9b5d0",
      "value": " 150/150 [00:01&lt;00:00, 89.2B/s]"
     }
    },
    "8880e647ca4a4796b571f5b983b541b4": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c1852f69951e4d80aecb5a6c942b2ca6",
      "placeholder": "​",
      "style": "IPY_MODEL_c7db7b055dbe4361965ca4ab8c028536",
      "value": " 189/189 [00:00&lt;00:00, 1.94kB/s]"
     }
    },
    "8b9a20a51500438c9d6e84d9f71e8cbd": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "93e868a4b6384840b3d245391ee2915a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9597bad322c34d02a0c10dd66b21813e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_71416028306340ee8987d703b506245b",
       "IPY_MODEL_100352c2499749e48ed10f3dee4f569d"
      ],
      "layout": "IPY_MODEL_62661d74e0ea462cb6b8e574aa17dc2f"
     }
    },
    "9c78a3ef3d804f89b4102a5029415e35": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9da0f76f26294b27982e080b8af6e28b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9da1585dcdd440fa8b2e8351ffeba348": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b01601a11a324141b2d88190a87ef950",
      "max": 189,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_01d9cd6656494aef83a3947945c0acdd",
      "value": 189
     }
    },
    "9f5b31e32c5c4398a6eede32431ad55e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f7ebc8ee77094382a2b26f879cdf8613",
      "max": 559,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_eb25be6e7fcc49cfbf9815d69f849637",
      "value": 559
     }
    },
    "a8ba7f1398f2401b8d0139aabdf6ce1f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b01601a11a324141b2d88190a87ef950": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bbbeec7c73c24059bffd602af5dcbd62": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9c78a3ef3d804f89b4102a5029415e35",
      "placeholder": "​",
      "style": "IPY_MODEL_082e03f9f75c4546bd5b2326b33e8017",
      "value": " 899k/899k [00:02&lt;00:00, 334kB/s]"
     }
    },
    "c1852f69951e4d80aecb5a6c942b2ca6": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c7db7b055dbe4361965ca4ab8c028536": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c840113abcc44f84a88e6120fe198fba": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "cb53b988f7df44d29ff7efcb0a236fce": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "d7ce404f2f784b0f93abf8f3ba592e18": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dd518bb2a8d141f58118e2cf7e140ffb": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "e1819597e4d840b19cfa8aba00ef3ef1": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e4a33236e1954bbf83e57ff15c40ff7f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "eb25be6e7fcc49cfbf9815d69f849637": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "ec3c00e615164fb488ebfb51d8ac9d9e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ef78454fe89347ea8f6f3148e23440db",
       "IPY_MODEL_bbbeec7c73c24059bffd602af5dcbd62"
      ],
      "layout": "IPY_MODEL_9da0f76f26294b27982e080b8af6e28b"
     }
    },
    "ee436c1e0fb24802b5d0706299ce7e81": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ef088eda6fe04abe958059620da9b5d0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ef78454fe89347ea8f6f3148e23440db": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8b9a20a51500438c9d6e84d9f71e8cbd",
      "max": 898822,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cb53b988f7df44d29ff7efcb0a236fce",
      "value": 898822
     }
    },
    "f7ebc8ee77094382a2b26f879cdf8613": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fce574414e154590a4a642362db98421": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4473fa7bf766425bb704bb9772c021fd",
      "max": 150,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_706c079c7d484975ad69a91b3f95e9f3",
      "value": 150
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}