mirror of
https://github.com/deepset-ai/haystack.git
synced 2025-12-29 16:08:38 +00:00
7e79a48540
* quick fix benchmark runs to make them work with current haystack version * fix minor typo * update readme. fix minor things to make benchmarks run again * Update Documentation & Code Style * fix typo in readme * update result files for reader and retriever querying * reduce batch size for update embeddings to prevent xlarge bulk_update requests that exceed elastic's limits (happening in dense 500k runs) * change default memory allocation back to normal. add note to readme * add first indexing results * add memory to docker cmd * full benchmarks results on commit c5a2651fcbbeffca06ffa9036b10e62669bcc1b0 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
3.1 KiB
3.1 KiB
| EM | f1 | top_n_accuracy | top_n | reader_time | seconds_per_query | EM_text_answer | f1_text_answer | top_n_accuracy_text_answer | top_n_EM_text_answer | top_n_f1_text_answer | Total_text_answer | EM_no_answer | f1_no_answer | top_n_accuracy_no_answer | Total_no_answer | passages_per_second | reader | error | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 76.1672 | 78.2154 | 97.4128 | 5 | 104.291 | 0.00878905 | 61.4917 | 65.593 | 94.8194 | 81.0159 | 90.5067 | 5926 | 90.8081 | 90.8081 | 100 | 5940 | 118.419 | deepset/roberta-base-squad2 | |
| 1 | 76.8498 | 79.648 | 97.2189 | 5 | 52.5473 | 0.00442839 | 65.8961 | 71.499 | 94.4313 | 79.9696 | 89.708 | 5926 | 87.7778 | 87.7778 | 100 | 5940 | 235.026 | deepset/minilm-uncased-squad2 | |
| 2 | 70.0826 | 73.4981 | 95.5335 | 5 | 105.557 | 0.00889576 | 56.598 | 63.4371 | 91.0564 | 72.1228 | 84.4785 | 5926 | 83.5354 | 83.5354 | 100 | 5940 | 116.998 | deepset/bert-base-cased-squad2 | |
| 3 | 83.6929 | 87.3504 | 98.0785 | 5 | 284.209 | 0.0239515 | 75.1941 | 82.5177 | 96.1525 | 82.3321 | 91.8921 | 5926 | 92.1717 | 92.1717 | 100 | 5940 | 43.454 | deepset/roberta-large-squad2 | |
| 4 | 79.5045 | 82.2939 | 97.5223 | 5 | 103.885 | 0.00875486 | 71.1104 | 76.6958 | 95.0388 | 81.7246 | 90.891 | 5926 | 87.8788 | 87.8788 | 100 | 5940 | 118.881 | deepset/roberta-base-squad2-distilled | |
| 5 | 38.5387 | 43.3403 | 95.4323 | 5 | 62.2323 | 0.00524459 | 74.5359 | 84.1505 | 90.8539 | 75.0928 | 85.2126 | 5926 | 2.62626 | 2.62626 | 100 | 5940 | 198.45 | distilbert-base-uncased-distilled-squad |