{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"# Evaluation of a Pipeline and its Components\n",
"\n",
"[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial5_Evaluation.ipynb)\n",
"\n",
"To be able to make a statement about the quality of results a question-answering pipeline or any other pipeline in haystack produces, it is important to evaluate it. Furthermore, evaluation allows determining which components of the pipeline can be improved.\n",
"The results of the evaluation can be saved as CSV files, which contain all the information to calculate additional metrics later on or inspect individual predictions."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lEKOjCS5U7so",
"pycharm": {
"is_executing": true
}
},
"source": [
"### Prepare environment\n",
"\n",
"#### Colab: Enable the GPU runtime\n",
"Make sure you enable the GPU runtime to experience decent speed in this tutorial.\n",
"**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**\n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "xhFIMX_7U7ss",
"outputId": "285b2491-01e5-4bfd-cba9-c2279d4417c4",
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# Make sure you have a GPU running\n",
"!nvidia-smi"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"id": "vgmFOp82Oht_",
"outputId": "5bbcbb42-3a90-43a9-ebfd-598a98fa7143",
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# Install the latest release of Haystack in your own environment\n",
"#! pip install farm-haystack\n",
"\n",
"# Install the latest master of Haystack\n",
"!pip install --upgrade pip\n",
"!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Start an Elasticsearch server\n",
"You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in your environment (eg., in Colab notebooks), then you can manually download and execute Elasticsearch from source."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tNoaWcDKOhuL",
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# If Docker is available: Start Elasticsearch as docker container\n",
"# from haystack.utils import launch_es\n",
"# launch_es()\n",
"\n",
"# Alternative in Colab / No Docker environments: Start Elasticsearch from source\n",
"! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q\n",
"! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz\n",
"! chown -R daemon:daemon elasticsearch-7.9.2\n",
"\n",
"import os\n",
"from subprocess import Popen, PIPE, STDOUT\n",
"\n",
"es_server = Popen(\n",
" [\"elasticsearch-7.9.2/bin/elasticsearch\"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1) # as daemon\n",
")\n",
"# wait until ES has started\n",
"! sleep 30"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fetch, Store And Preprocess the Evaluation Dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "tTXxr6TAOhuz",
"outputId": "586d4775-4354-4ed9-a72c-c30bedcdfbee",
"pycharm": {
"is_executing": true,
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from haystack.utils import fetch_archive_from_http\n",
"\n",
"# Download evaluation data, which is a subset of Natural Questions development set containing 50 documents with one question per document and multiple annotated answers\n",
"doc_dir = \"data/tutorial5\"\n",
"s3_url = \"https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/nq_dev_subset_v2.json.zip\"\n",
"fetch_archive_from_http(url=s3_url, output_dir=doc_dir)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "T-G7Ay2jU7s_"
},
"outputs": [],
"source": [
"# make sure these indices do not collide with existing ones, the indices will be wiped clean before data is inserted\n",
"doc_index = \"tutorial5_docs\"\n",
"label_index = \"tutorial5_labels\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "B_NEtezLOhu5",
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# Connect to Elasticsearch\n",
"from haystack.document_stores import ElasticsearchDocumentStore\n",
"\n",
"# Connect to Elasticsearch\n",
"document_store = ElasticsearchDocumentStore(\n",
" host=\"localhost\",\n",
" username=\"\",\n",
" password=\"\",\n",
" index=doc_index,\n",
" label_index=label_index,\n",
" embedding_field=\"emb\",\n",
" embedding_dim=768,\n",
" excluded_meta_data=[\"emb\"],\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "bRFsQUAJOhu_",
"outputId": "477031b9-5c2c-4128-ef5f-54db86259734",
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from haystack.nodes import PreProcessor\n",
"\n",
"# Add evaluation data to Elasticsearch Document Store\n",
"# We first delete the custom tutorial indices to not have duplicate elements\n",
"# and also split our documents into shorter passages using the PreProcessor\n",
"preprocessor = PreProcessor(\n",
" split_by=\"word\",\n",
" split_length=200,\n",
" split_overlap=0,\n",
" split_respect_sentence_boundary=False,\n",
" clean_empty_lines=False,\n",
" clean_whitespace=False,\n",
")\n",
"document_store.delete_documents(index=doc_index)\n",
"document_store.delete_documents(index=label_index)\n",
"\n",
"# The add_eval_data() method converts the given dataset in json format into Haystack document and label objects. Those objects are then indexed in their respective document and label index in the document store. The method can be used with any dataset in SQuAD format.\n",
"document_store.add_eval_data(\n",
" filename=\"data/tutorial5/nq_dev_subset_v2.json\",\n",
" doc_index=doc_index,\n",
" label_index=label_index,\n",
" preprocessor=preprocessor,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gy8YwmSYOhvE",
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"## Initialize the Two Components of an ExtractiveQAPipeline: Retriever and Reader"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "JkhaPMIJOhvF",
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# Initialize Retriever\n",
"from haystack.nodes import BM25Retriever\n",
"\n",
"retriever = BM25Retriever(document_store=document_store)\n",
"\n",
"# Alternative: Evaluate dense retrievers (EmbeddingRetriever or DensePassageRetriever)\n",
"# The EmbeddingRetriever uses a single transformer based encoder model for query and document.\n",
"# In contrast, DensePassageRetriever uses two separate encoders for both.\n",
"\n",
"# Please make sure the \"embedding_dim\" parameter in the DocumentStore above matches the output dimension of your models!\n",
"# Please also take care that the PreProcessor splits your files into chunks that can be completely converted with\n",
"# the max_seq_len limitations of Transformers\n",
"# The SentenceTransformer model \"sentence-transformers/multi-qa-mpnet-base-dot-v1\" generally works well with the EmbeddingRetriever on any kind of English text.\n",
"# For more information and suggestions on different models check out the documentation at: https://www.sbert.net/docs/pretrained_models.html\n",
"\n",
"# from haystack.retriever import EmbeddingRetriever, DensePassageRetriever\n",
"# retriever = EmbeddingRetriever(document_store=document_store,\n",
"# embedding_model=\"sentence-transformers/multi-qa-mpnet-base-dot-v1\")\n",
"# retriever = DensePassageRetriever(document_store=document_store,\n",
"# query_embedding_model=\"facebook/dpr-question_encoder-single-nq-base\",\n",
"# passage_embedding_model=\"facebook/dpr-ctx_encoder-single-nq-base\",\n",
"# use_gpu=True,\n",
"# max_seq_len_passage=256,\n",
"# embed_title=True)\n",
"# document_store.update_embeddings(retriever, index=doc_index)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 313,
"referenced_widgets": [
"118aedffeecd4f109ae04a4561baeb08",
"59efb57b419242a1aba4d20084e29d38",
"ddaf59cedca143c8b4fe005a51077323",
"25d7818d6f7b4b628ab3f83f2c2fa6a3",
"48431fa7696540fc9696799f75166680",
"a21f4958105f4b3ca32f0977bcfd7d48",
"3d211e6614f4451e9d14866cb3a8332d",
"3df3da3d04d5448c810bd00f66bd3a0e",
"8dfc1480100f43e0b4ea2fc2fb9279d3",
"0618a236cc14473f92257aebbc3d930d",
"8e0958b5dc27412e9f0332da3457ffdb",
"6f7ddc1720344ba9b939a8e4ac593d67",
"b480c97c1d9944b9a8dd09ed6e1e9bd3",
"ed683929822b4084ba33e89b23936b16",
"94c88d0fc3f949fbacfb6b4fcd99cc63",
"b262e92ff6484405a0e9364f6ecafb6a",
"bacf1704dbaf4176afbe2cbcc8e036ef",
"fe56b1d86ab84675b82781a1f8edd40a",
"d7e3c8e1e0424cec9dc1b97090b5af87",
"98da94a8d7b94fb4a08adcebea15e114",
"13b75146701145808315dc87d598b3f9",
"2bd3bd51ae644c1894a2ddca09d14e85",
"59039df74ce64c2f9e91663b6498c29c",
"9e130b97f0f4463f85df834d0f99d6ef",
"2c028a3f096344d68071d78387efa117",
"4d922fb301f944fbb0218335a28cf6e5",
"05d82995d5a94b5db39bf639d1cc05c2",
"76f4af76b42f460fa34d5f00a9656dc5",
"73d7fdd3f38349b4882124d8351eace5",
"ea439e2251ed467fb3a775f0c8e0c3bb",
"b6729cc6ba084677af55ac63c819b72f",
"fc011913e8464d439a97fe75ef5f9fa1",
"8a9f9b7bab8e40278430a35720066a61",
"5db857b352964db3a617568ff1dce86d",
"7752437041f745a4af4b9130df3fefa7",
"5f94d400ea884c1cadfc966e44849b3a",
"0d34710578ac4c1db6fe372b5d1215b4",
"994ae85181664e2e87a2ee18a7a237ba",
"368a61e33c3144bfa3cc94af10691146",
"ccfd1a0b6f494d8a9d78e7387261fba8",
"db57281b3d7a448fbd2d63d8f127ea3e",
"978b165c69dd4e14b8479ea7bd8cb1e5",
"6f028c7e888e4ae5ab5c1e42ff142b5f",
"c8ba8c2a210b45f6a9b5257589babac3",
"36f855f41cc1488f9d92ec34bb8d30b0",
"1d5d220bedc54dbdbacb9c43767bf64d",
"0523b10429d04f3d81d7078a13a12168",
"64cd5b6f0c4d4631a1049ee7ee50f063",
"eb11ea5785284bf6a15cc31ad643ed88",
"807e4eee3b2c440c8799afcc6344ff5d",
"6ca6dc2c6b4349fcb39ed8c44f65bdb0",
"4837b34ccb4d4688865dc24dc58a7c1e",
"d4dbfa5e89e7432dbed34606a786fd6f",
"7e058076836e438daf5399428eabac5e",
"a529dbbfdd6b469dbfe80cee993c9a33",
"69750fea2e7149eab8928282ba9bae29",
"08a1d1a6fb884c769d409170d6cda556",
"548ef6c85056414cb0ce79164a086d35",
"cf58b340496b4d62b610451cedbd709a",
"4dad7e58cc47436aafe38230514325a1",
"f36dc87a4af7481cb3c2cba23d57eb5a",
"2a9ef1f2f43d47b28cd0ff7ef4a21ade",
"caa374f7dc5045218c6f71f322d8e6be",
"e567fab4446544f795be2eb0a6705f9c",
"2b17ffac93b8406fac55b695d93d963b",
"3096cae7388e4b988df306be9cc58afd"
]
},
"id": "cW3Ypn_gOhvK",
"outputId": "4b5feff7-ae9f-4cd8-de1e-944f0eb66f66",
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# Initialize Reader\n",
"from haystack.nodes import FARMReader\n",
"\n",
"reader = FARMReader(\"deepset/roberta-base-squad2\", top_k=4, return_no_answer=True)\n",
"\n",
"# Define a pipeline consisting of the initialized retriever and reader\n",
"from haystack.pipelines import ExtractiveQAPipeline\n",
"\n",
"pipeline = ExtractiveQAPipeline(reader=reader, retriever=retriever)\n",
"\n",
"# The evaluation also works with any other pipeline.\n",
"# For example you could use a DocumentSearchPipeline as an alternative:\n",
"\n",
"# from haystack.pipelines import DocumentSearchPipeline\n",
"# pipeline = DocumentSearchPipeline(retriever=retriever)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7i84KXONOhvc",
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"## Evaluation of an ExtractiveQAPipeline\n",
"Here we evaluate retriever and reader in open domain fashion on the full corpus of documents i.e. a document is considered\n",
"correctly retrieved if it contains the gold answer string within it. The reader is evaluated based purely on the\n",
"predicted answer string, regardless of which document this came from and the position of the extracted span.\n",
"\n",
"The generation of predictions is seperated from the calculation of metrics. This allows you to run the computation-heavy model predictions only once and then iterate flexibly on the metrics or reports you want to generate.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from haystack.schema import EvaluationResult, MultiLabel\n",
"\n",
"# We can load evaluation labels from the document store\n",
"# We are also opting to filter out no_answer samples\n",
"eval_labels = document_store.get_all_labels_aggregated(drop_negative_labels=True, drop_no_answers=True)\n",
"\n",
"## Alternative: Define queries and labels directly\n",
"\n",
"# eval_labels = [\n",
"# MultiLabel(\n",
"# labels=[\n",
"# Label(\n",
"# query=\"who is written in the book of life\",\n",
"# answer=Answer(\n",
"# answer=\"every person who is destined for Heaven or the World to Come\",\n",
"# offsets_in_context=[Span(374, 434)]\n",
"# ),\n",
"# document=Document(\n",
"# id='1b090aec7dbd1af6739c4c80f8995877-0',\n",
"# content_type=\"text\",\n",
"# content='Book of Life - wikipedia Book of Life Jump to: navigation, search This article is\n",
"# about the book mentioned in Christian and Jewish religious teachings...'\n",
"# ),\n",
"# is_correct_answer=True,\n",
"# is_correct_document=True,\n",
"# origin=\"gold-label\"\n",
"# )\n",
"# ]\n",
"# )\n",
"# ]\n",
"\n",
"# Similar to pipeline.run() we can execute pipeline.eval()\n",
"eval_result = pipeline.eval(labels=eval_labels, params={\"Retriever\": {\"top_k\": 5}})"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n", " | multilabel_id | \n", "query | \n", "filters | \n", "context | \n", "gold_contexts | \n", "gold_id_match | \n", "context_match | \n", "answer_match | \n", "gold_id_or_answer_match | \n", "gold_id_and_answer_match | \n", "... | \n", "rank | \n", "document_id | \n", "gold_document_ids | \n", "gold_documents_id_match | \n", "gold_contexts_similarity | \n", "gold_answers_match | \n", "type | \n", "node | \n", "eval_mode | \n", "index | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "people considered righteous before God. God has such a book, and to be blott... | \n", "[Book of Life - wikipedia Book of Life Jump to: navigation, search This arti... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "1.0 | \n", "737cb9cd71126bd25296bce035a5956f-1 | \n", "[de2fd2f109e11213af1ea189fd1488a3-0, de2fd2f109e11213af1ea189fd1488a3-0] | \n", "[0.0, 0.0] | \n", "[44.474885844748854, 44.474885844748854] | \n", "[0.0, 0.0] | \n", "document | \n", "Retriever | \n", "integrated | \n", "0 | \n", "
1 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "as adversaries (of God). Also, according to ib. xxxvi. 10, one who contrives... | \n", "[Book of Life - wikipedia Book of Life Jump to: navigation, search This arti... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "2.0 | \n", "fcfe96ddea6611b56ffaeda010d3a111-2 | \n", "[de2fd2f109e11213af1ea189fd1488a3-0, de2fd2f109e11213af1ea189fd1488a3-0] | \n", "[0.0, 0.0] | \n", "[44.5703493862134, 44.5703493862134] | \n", "[0.0, 0.0] | \n", "document | \n", "Retriever | \n", "integrated | \n", "1 | \n", "
2 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "the citizens' registers. The life which the righteous participate in is to b... | \n", "[Book of Life - wikipedia Book of Life Jump to: navigation, search This arti... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "3.0 | \n", "5f3f7a6be74e24b1065a4d770585d210-6 | \n", "[de2fd2f109e11213af1ea189fd1488a3-0, de2fd2f109e11213af1ea189fd1488a3-0] | \n", "[0.0, 0.0] | \n", "[44.374184960316995, 44.374184960316995] | \n", "[0.0, 0.0] | \n", "document | \n", "Retriever | \n", "integrated | \n", "2 | \n", "
3 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "apostles' names are ``written in heaven'' (Luke x. 20), or ``the fellow-work... | \n", "[Book of Life - wikipedia Book of Life Jump to: navigation, search This arti... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "4.0 | \n", "bcc533ac40605a5faa63f4b0d7124404-3 | \n", "[de2fd2f109e11213af1ea189fd1488a3-0, de2fd2f109e11213af1ea189fd1488a3-0] | \n", "[0.0, 0.0] | \n", "[42.68629254829806, 42.68629254829806] | \n", "[0.0, 0.0] | \n", "document | \n", "Retriever | \n", "integrated | \n", "3 | \n", "
4 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "The Absolutely True Diary of a Part-Time Indian - wikipedia The Absolutely T... | \n", "[Book of Life - wikipedia Book of Life Jump to: navigation, search This arti... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "5.0 | \n", "edba8a30690fe62dd0330344f6ad98ba-0 | \n", "[de2fd2f109e11213af1ea189fd1488a3-0, de2fd2f109e11213af1ea189fd1488a3-0] | \n", "[0.0, 0.0] | \n", "[42.55874673629243, 42.55874673629243] | \n", "[0.0, 0.0] | \n", "document | \n", "Retriever | \n", "integrated | \n", "4 | \n", "
5 rows × 24 columns
\n", "\n", " | multilabel_id | \n", "query | \n", "filters | \n", "gold_answers | \n", "answer | \n", "context | \n", "exact_match | \n", "f1 | \n", "exact_match_context_scope | \n", "f1_context_scope | \n", "... | \n", "offsets_in_document | \n", "gold_offsets_in_documents | \n", "gold_answers_exact_match | \n", "gold_answers_f1 | \n", "gold_documents_id_match | \n", "gold_contexts_similarity | \n", "type | \n", "node | \n", "eval_mode | \n", "index | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "[all people considered righteous before God, every person who is destined fo... | \n", "\n", " | None | \n", "0.0 | \n", "0.000000 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "[{'start': 0, 'end': 0}] | \n", "[{'start': 1107, 'end': 1149}, {'start': 374, 'end': 434}] | \n", "[0, 0] | \n", "[0, 0] | \n", "[0.0, 0.0] | \n", "[0.0, 0.0] | \n", "answer | \n", "Reader | \n", "integrated | \n", "0 | \n", "
1 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "[all people considered righteous before God, every person who is destined fo... | \n", "those whose names are written in the Book of Life from the foundation of the... | \n", "ohn of Patmos. As described, only those whose names are written in the Book ... | \n", "0.0 | \n", "0.083333 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "[{'start': 576, 'end': 658}] | \n", "[{'start': 1107, 'end': 1149}, {'start': 374, 'end': 434}] | \n", "[0, 0] | \n", "[0, 0.08333333333333334] | \n", "[0.0, 0.0] | \n", "[50.0, 50.0] | \n", "answer | \n", "Reader | \n", "integrated | \n", "1 | \n", "
2 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "[all people considered righteous before God, every person who is destined fo... | \n", "only the names of the righteous | \n", ". The Psalmist likewise speaks of the Book of Life in which only the names o... | \n", "0.0 | \n", "0.200000 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "[{'start': 498, 'end': 529}] | \n", "[{'start': 1107, 'end': 1149}, {'start': 374, 'end': 434}] | \n", "[0, 0] | \n", "[0.2, 0] | \n", "[0.0, 0.0] | \n", "[52.0, 52.0] | \n", "answer | \n", "Reader | \n", "integrated | \n", "2 | \n", "
3 | \n", "5403753299762339270 | \n", "who is written in the book of life | \n", "b'null' | \n", "[all people considered righteous before God, every person who is destined fo... | \n", "those who are found written in the book and who shall escape the troubles pr... | \n", "those who are found written in the book and who shall escape the troubles pr... | \n", "0.0 | \n", "0.111111 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "[{'start': 135, 'end': 305}] | \n", "[{'start': 1107, 'end': 1149}, {'start': 374, 'end': 434}] | \n", "[0, 0] | \n", "[0, 0.1111111111111111] | \n", "[0.0, 0.0] | \n", "[46.470588235294116, 46.470588235294116] | \n", "answer | \n", "Reader | \n", "integrated | \n", "3 | \n", "
0 | \n", "5186659498043378562 | \n", "who was the girl in the video brenda got a baby | \n", "b'null' | \n", "[Ethel ``Edy'' Proctor] | \n", "her cousin | \n", "ng a story in the newspaper of a 12-year-old girl getting pregnant by her co... | \n", "0.0 | \n", "0.000000 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "[{'start': 423, 'end': 433}] | \n", "[{'start': 181, 'end': 202}] | \n", "[0] | \n", "[0] | \n", "[1.0] | \n", "[100.0] | \n", "answer | \n", "Reader | \n", "integrated | \n", "0 | \n", "
5 rows × 28 columns
\n", "