mirror of
https://github.com/deepset-ai/haystack.git
synced 2025-12-28 15:38:36 +00:00
7e79a48540
* quick fix benchmark runs to make them work with current haystack version * fix minor typo * update readme. fix minor things to make benchmarks run again * Update Documentation & Code Style * fix typo in readme * update result files for reader and retriever querying * reduce batch size for update embeddings to prevent xlarge bulk_update requests that exceed elastic's limits (happening in dense 500k runs) * change default memory allocation back to normal. add note to readme * add first indexing results * add memory to docker cmd * full benchmarks results on commit c5a2651fcbbeffca06ffa9036b10e62669bcc1b0 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2.0 KiB
2.0 KiB
| 1 | EM | f1 | top_n_accuracy | top_n | reader_time | seconds_per_query | EM_text_answer | f1_text_answer | top_n_accuracy_text_answer | top_n_EM_text_answer | top_n_f1_text_answer | Total_text_answer | EM_no_answer | f1_no_answer | top_n_accuracy_no_answer | Total_no_answer | passages_per_second | reader | error | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 0 | 76.1672004045171 | 78.21542223321491 | 97.41277599865161 | 5 | 104.290868674987 | 0.0087890501158762 | 61.491731353358084 | 65.59301387433817 | 94.81943975700304 | 81.01586230172123 | 90.50667071721769 | 5926 | 90.8080808080808 | 90.8080808080808 | 100.0 | 5940 | 118.41880460779026 | deepset/roberta-base-squad2 | |
| 3 | 1 | 76.84982302376538 | 79.6479775678566 | 97.21894488454407 | 5 | 52.54729196999688 | 0.004428391367773207 | 65.89605129935876 | 71.49897094501964 | 94.4313196085049 | 79.96962537968275 | 89.70802565150441 | 5926 | 87.77777777777777 | 87.77777777777777 | 100.0 | 5940 | 235.02638360605766 | deepset/minilm-uncased-squad2 | |
| 4 | 2 | 70.0825889094893 | 73.49806294434092 | 95.53345693578291 | 5 | 105.55706145399017 | 0.008895757749367114 | 56.59804252446844 | 63.43705955071704 | 91.05636179547756 | 72.1228484643942 | 84.47853540832811 | 5926 | 83.53535353535354 | 83.53535353535354 | 100.0 | 5940 | 116.99833085428466 | deepset/bert-base-cased-squad2 | |
| 5 | 3 | 83.69290409573573 | 87.35038229973254 | 98.07854373841226 | 5 | 284.2086430560157 | 0.023951512140233923 | 75.19406007424907 | 82.51765716649112 | 96.15254809314884 | 82.33209584880188 | 91.89205932239825 | 5926 | 92.17171717171718 | 92.17171717171718 | 100.0 | 5940 | 43.45399164221017 | deepset/roberta-large-squad2 | |
| 6 | 4 | 79.50446654306421 | 82.29388851439239 | 97.52233271532108 | 5 | 103.88512606802396 | 0.008754856402159445 | 71.110361120486 | 76.69579499017553 | 95.03881201484981 | 81.72460344245697 | 90.89098996208568 | 5926 | 87.87878787878788 | 87.87878787878788 | 100.0 | 5940 | 118.88131118898794 | deepset/roberta-base-squad2-distilled | |
| 7 | 5 | 38.53868194842407 | 43.340283031558755 | 95.43232765885725 | 5 | 62.23228176299017 | 0.005244588046771462 | 74.53594330070874 | 84.1504891077415 | 90.85386432669593 | 75.09281133985826 | 85.21259442024737 | 5926 | 2.6262626262626263 | 2.6262626262626263 | 100.0 | 5940 | 198.4500591997352 | distilbert-base-uncased-distilled-squad |