yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2026-01-04 02:57:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
haystack/docs/_src/benchmarks/retriever_query_results.md
Malte Pietsch 7e79a48540
bug: reactivate benchmarks with quick fixes (#2766) 
* quick fix benchmark runs to make them work with current haystack version

* fix minor typo

* update readme. fix minor things to make benchmarks run again

* Update Documentation & Code Style

* fix typo in readme

* update result files for reader and retriever querying

* reduce batch size for update embeddings to prevent xlarge bulk_update requests that exceed elastic's limits (happening in dense 500k runs)

* change default memory allocation back to normal. add note to readme

* add first indexing results

* add memory to docker cmd

* full benchmarks results on commit  c5a2651fcbbeffca06ffa9036b10e62669bcc1b0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-09-20 10:22:08 +02:00

6.9 KiB
Raw Blame History

retriever doc_store n_docs n_queries retrieve_time queries_per_second seconds_per_query recall map top_k date_time error
31 sentence_transformers elasticsearch 500000 5637 3910.2 1.44161 0.693667 89.6399 76.4956 10 2022-09-18 08:25:36.627368
28 dpr elasticsearch 500000 5637 3839.53 1.46815 0.68113 93.0814 80.8614 10 2022-09-18 03:20:53.663119
7 sentence_transformers elasticsearch 1000 1064 22.0857 48.1761 0.0207572 98.7782 90.0664 10 2022-09-17 21:28:29.461918
15 sentence_transformers elasticsearch 10000 5637 187.414 30.0779 0.0332471 96.523 87.1126 10 2022-09-17 22:03:32.200427
20 dpr elasticsearch 100000 5637 903.509 6.23901 0.160282 95.7956 86.5456 10 2022-09-17 22:58:20.213034
0 elastic elasticsearch 1000 1064 4.12203 258.125 0.00387409 89.0977 74.2044 10 2022-09-17 21:19:21.768721
24 elastic elasticsearch 500000 5637 83.5613 67.4595 0.0148237 62.7461 45.604 10 2022-09-18 00:29:07.528766
16 elastic elasticsearch 100000 5637 42.6961 132.026 0.00757426 71.9354 56.2611 10 2022-09-17 22:08:40.215954
8 elastic elasticsearch 10000 5637 28.6855 196.51 0.00508879 81.107 66.3294 10 2022-09-17 21:30:14.745163
4 dpr elasticsearch 1000 1064 29.5037 36.0633 0.0277291 99.1541 92.9511 10 2022-09-17 21:24:50.768859
12 dpr elasticsearch 10000 5637 224.793 25.0764 0.0398782 97.4987 89.871 10 2022-09-17 21:48:36.651933
23 sentence_transformers elasticsearch 100000 5637 886.585 6.35811 0.15728 93.791 82.7469 10 2022-09-18 00:09:04.840841
29 dpr faiss_flat 500000 5637 3683.52 1.53033 0.653453 93.0814 80.8614 10 2022-09-18 05:09:24.992364
13 dpr faiss_flat 10000 5637 181.465 31.0639 0.0321917 97.4987 89.871 10 2022-09-17 21:53:56.820064
5 dpr faiss_flat 1000 1064 29.98 35.4904 0.0281767 99.1541 92.9511 10 2022-09-17 21:26:11.004926
21 dpr faiss_flat 100000 5637 841.228 6.70092 0.149233 95.7956 86.5461 10 2022-09-17 23:23:47.065373
22 dpr faiss_hnsw 100000 5637 138.644 40.6581 0.0245954 93.1879 84.3342 10 2022-09-17 23:39:25.396237
14 dpr faiss_hnsw 10000 5637 112.711 50.0131 0.0199948 96.9842 89.5134 10 2022-09-17 21:58:05.961829
6 dpr faiss_hnsw 1000 1064 25.615 41.5382 0.0240742 99.1541 92.9511 10 2022-09-17 21:27:21.948745
30 dpr faiss_hnsw 500000 5637 124.996 45.0975 0.0221742 86.9079 75.7332 10 2022-09-18 06:09:37.007518
9 elastic opensearch_flat 10000 5637 32.4295 173.823 0.00575297 81.107 66.3294 10 2022-09-17 21:32:03.707902
10 dpr opensearch_flat 10000 5637 182.627 30.8662 0.0323979 97.4987 89.871 10 2022-09-17 21:37:08.044354
17 elastic opensearch_flat 100000 5637 46.926 120.125 0.00832464 71.9354 56.2611 10 2022-09-17 22:13:58.492431
1 elastic opensearch_flat 1000 1064 6.10185 174.373 0.00573482 89.0977 74.2044 10 2022-09-17 21:20:04.003708
18 dpr opensearch_flat 100000 5637 320.205 17.6044 0.0568041 95.7602 86.5195 10 2022-09-17 22:25:11.379134
25 elastic opensearch_flat 500000 5637 83.3962 67.593 0.0147944 62.7461 45.5951 10 2022-09-18 00:49:52.903674
26 dpr opensearch_flat 500000 5637 466.157 12.0925 0.0826959 93.0814 80.8614 10 2022-09-18 01:20:53.923798
2 dpr opensearch_flat 1000 1064 30.0942 35.3556 0.028284 99.1541 92.9511 10 2022-09-17 21:21:42.413924
11 dpr opensearch_hnsw 10000 5637 181.917 30.9866 0.032272 97.4987 89.871 10 2022-09-17 21:42:16.601719
3 dpr opensearch_hnsw 1000 1064 29.8264 35.673 0.0280324 99.1541 92.9511 10 2022-09-17 21:23:12.487039
19 dpr opensearch_hnsw 100000 5637 327.224 17.2268 0.0580493 95.7779 86.5372 10 2022-09-17 22:36:46.502520
27 dpr opensearch_hnsw 500000 5637 477.335 11.8093 0.0846789 93.0814 80.8614 10 2022-09-18 01:53:29.735540