yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2026-01-06 12:07:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
haystack/docs/_src/benchmarks/retriever_index_results.md
Malte Pietsch 7e79a48540
bug: reactivate benchmarks with quick fixes (#2766) 
* quick fix benchmark runs to make them work with current haystack version

* fix minor typo

* update readme. fix minor things to make benchmarks run again

* Update Documentation & Code Style

* fix typo in readme

* update result files for reader and retriever querying

* reduce batch size for update embeddings to prevent xlarge bulk_update requests that exceed elastic's limits (happening in dense 500k runs)

* change default memory allocation back to normal. add note to readme

* add first indexing results

* add memory to docker cmd

* full benchmarks results on commit  c5a2651fcbbeffca06ffa9036b10e62669bcc1b0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-09-20 10:22:08 +02:00

4.5 KiB
Raw Blame History

retriever doc_store n_docs indexing_time docs_per_second date_time error
31 sentence_transformers elasticsearch 500000 4165.33 120.039 2022-09-17 21:16:26.676611
28 dpr elasticsearch 500000 6180.35 80.9016 2022-09-17 16:45:36.329915
7 sentence_transformers elasticsearch 1000 9.84432 101.581 2022-09-17 08:26:44.750140
15 sentence_transformers elasticsearch 10000 84.8816 117.811 2022-09-17 08:43:51.615496
20 dpr elasticsearch 100000 1230.76 81.2505 2022-09-17 09:57:09.456572
0 elastic elasticsearch 1000 1.79888 555.901 2022-09-17 08:21:11.001120
24 elastic elasticsearch 500000 1031.72 484.627 2022-09-17 11:09:14.536450
16 elastic elasticsearch 100000 206.838 483.47 2022-09-17 08:47:59.723168
8 elastic elasticsearch 10000 20.8192 480.325 2022-09-17 08:27:43.460016
4 dpr elasticsearch 1000 12.1146 82.545 2022-09-17 08:24:34.582262
12 dpr elasticsearch 10000 123.219 81.1565 2022-09-17 08:37:02.872786
23 sentence_transformers elasticsearch 100000 831.661 120.241 2022-09-17 10:50:54.888143
29 dpr faiss_flat 500000 5485.97 91.1415 2022-09-17 18:19:50.024476
13 dpr faiss_flat 10000 105.085 95.1615 2022-09-17 08:39:25.198922
5 dpr faiss_flat 1000 10.1512 98.5103 2022-09-17 08:25:18.819509
21 dpr faiss_flat 100000 1068.48 93.5905 2022-09-17 10:15:58.178598
22 dpr faiss_hnsw 100000 1195.9 83.6191 2022-09-17 10:36:24.783587
14 dpr faiss_hnsw 10000 114.679 87.2002 2022-09-17 08:41:49.541725
6 dpr faiss_hnsw 1000 10.7109 93.3628 2022-09-17 08:25:58.505307
30 dpr faiss_hnsw 500000 6349.9 78.7414 2022-09-17 20:06:17.374842
9 elastic opensearch_flat 10000 21.9185 456.235 2022-09-17 08:28:40.051299
10 dpr opensearch_flat 10000 123.36 81.0633 2022-09-17 08:31:26.201876
17 elastic opensearch_flat 100000 208.893 478.713 2022-09-17 08:52:13.178937
1 elastic opensearch_flat 1000 2.23007 448.417 2022-09-17 08:21:45.610495
18 dpr opensearch_flat 100000 1245.01 80.3208 2022-09-17 09:13:46.979550
25 elastic opensearch_flat 500000 1045.89 478.062 2022-09-17 11:28:08.906406
26 dpr opensearch_flat 500000 6235.61 80.1847 2022-09-17 13:13:24.174105
2 dpr opensearch_flat 1000 12.92 77.3995 2022-09-17 08:22:43.256934
11 dpr opensearch_hnsw 10000 121.491 82.3109 2022-09-17 08:34:11.508717
3 dpr opensearch_hnsw 1000 12.356 80.9326 2022-09-17 08:23:36.852546
19 dpr opensearch_hnsw 100000 1242.96 80.4531 2022-09-17 09:35:31.609351
27 dpr opensearch_hnsw 500000 6237.19 80.1643 2022-09-17 14:59:57.763007