mirror of
https://github.com/deepset-ai/haystack.git
synced 2025-12-28 15:38:36 +00:00
b752da1cd5
* new docs version * updated directory structure * Add pipelines page * Add Finder deprecation suggestion * header for pipelines file * Document MySQL support * Mention DPR train tutorial coming soon * Mention open distro ES * Update doc strings regarding similarity fn * Add link to API docs * Wrap pipelines docs in box * add api reference for pipelines * copied latest version to v0.6.0 * Remove space * Remove space * Copy to v0.6.0 Co-authored-by: brandenchan <brandenchan@icloud.com>
44 lines
2.0 KiB
JSON
44 lines
2.0 KiB
JSON
{
|
|
"chart_type": "BarChart",
|
|
"title": "Reader Performance",
|
|
"subtitle": "Time and Accuracy Benchmarks",
|
|
"description": "Performance benchmarks of different Readers that can be used off-the-shelf in Haystack. Some models are geared towards speed, while others are more performance-focused. Accuracy is measured as F1 score and speed as passages/sec (with passages of 384 tokens). Each Reader is benchmarked using the SQuAD v2.0 development set, which contains 11866 question answer pairs. When tokenized using the BERT tokenizer and split using a sliding window approach, these become 12350 passages that are passed into the model. We set <i>max_seq_len=384</i> and <i>doc_stride=128</i>. These benchmarking tests are run using an AWS p3.2xlarge instance with a Nvidia V100 GPU with this <a href='https://github.com/deepset-ai/haystack/blob/master/test/benchmarks/reader.py'>script</a>. Please note that we are using the FARMReader class rather than the TransformersReader class. Also, the F1 measure that is reported here is in fact calculated on token level, rather than word level as is done in the official SQuAD script.",
|
|
"bars": "horizontal",
|
|
"columns": [
|
|
"Model",
|
|
"F1",
|
|
"Speed (passages/sec)"
|
|
],
|
|
"data": [
|
|
{
|
|
"F1": 82.62983412843887,
|
|
"Speed": 98.86638639776464,
|
|
"Model": "RoBERTa"
|
|
},
|
|
{
|
|
"F1": 78.90026641413856,
|
|
"Speed": 181.96379531485616,
|
|
"Model": "MiniLM"
|
|
},
|
|
{
|
|
"F1": 74.32668866064459,
|
|
"Speed": 106.04748306200683,
|
|
"Model": "BERT base"
|
|
},
|
|
{
|
|
"F1": 83.29492827667042,
|
|
"Speed": 40.408497243719076,
|
|
"Model": "BERT large"
|
|
},
|
|
{
|
|
"F1": 84.62174414643722,
|
|
"Speed": 40.483264542292716,
|
|
"Model": "XLM-RoBERTa"
|
|
},
|
|
{
|
|
"F1": 42.342513261953935,
|
|
"Speed": 160.41712955027901,
|
|
"Model": "DistilBERT"
|
|
}
|
|
]
|
|
} |