mirror of
https://github.com/deepset-ai/haystack.git
synced 2025-11-01 18:29:32 +00:00
7e79a48540
* quick fix benchmark runs to make them work with current haystack version * fix minor typo * update readme. fix minor things to make benchmarks run again * Update Documentation & Code Style * fix typo in readme * update result files for reader and retriever querying * reduce batch size for update embeddings to prevent xlarge bulk_update requests that exceed elastic's limits (happening in dense 500k runs) * change default memory allocation back to normal. add note to readme * add first indexing results * add memory to docker cmd * full benchmarks results on commit c5a2651fcbbeffca06ffa9036b10e62669bcc1b0 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
3.0 KiB
3.0 KiB
| 1 | retriever | doc_store | n_docs | indexing_time | docs_per_second | date_time | error | |
|---|---|---|---|---|---|---|---|---|
| 2 | 31 | sentence_transformers | elasticsearch | 500000 | 4165.329931593005 | 120.03851032486591 | 2022-09-17 21:16:26.676611 | |
| 3 | 28 | dpr | elasticsearch | 500000 | 6180.3500814459985 | 80.90156599721556 | 2022-09-17 16:45:36.329915 | |
| 4 | 7 | sentence_transformers | elasticsearch | 1000 | 9.844315000998904 | 101.5814711230319 | 2022-09-17 08:26:44.750140 | |
| 5 | 15 | sentence_transformers | elasticsearch | 10000 | 84.88162445399212 | 117.8111289024663 | 2022-09-17 08:43:51.615496 | |
| 6 | 20 | dpr | elasticsearch | 100000 | 1230.761674939 | 81.2504988059173 | 2022-09-17 09:57:09.456572 | |
| 7 | 0 | elastic | elasticsearch | 1000 | 1.7988810280075995 | 555.9011321096525 | 2022-09-17 08:21:11.001120 | |
| 8 | 24 | elastic | elasticsearch | 500000 | 1031.7205082759901 | 484.6273733915616 | 2022-09-17 11:09:14.536450 | |
| 9 | 16 | elastic | elasticsearch | 100000 | 206.83811076599522 | 483.4698964792531 | 2022-09-17 08:47:59.723168 | |
| 10 | 8 | elastic | elasticsearch | 10000 | 20.81922616600059 | 480.3252493760203 | 2022-09-17 08:27:43.460016 | |
| 11 | 4 | dpr | elasticsearch | 1000 | 12.114610354998149 | 82.54495775734364 | 2022-09-17 08:24:34.582262 | |
| 12 | 12 | dpr | elasticsearch | 10000 | 123.21874477900565 | 81.15648327642944 | 2022-09-17 08:37:02.872786 | |
| 13 | 23 | sentence_transformers | elasticsearch | 100000 | 831.661083732004 | 120.24128813537725 | 2022-09-17 10:50:54.888143 | |
| 14 | 29 | dpr | faiss_flat | 500000 | 5485.974153257004 | 91.1415158059306 | 2022-09-17 18:19:50.024476 | |
| 15 | 13 | dpr | faiss_flat | 10000 | 105.0845138200093 | 95.16150036272884 | 2022-09-17 08:39:25.198922 | |
| 16 | 5 | dpr | faiss_flat | 1000 | 10.151226033005514 | 98.51026829159531 | 2022-09-17 08:25:18.819509 | |
| 17 | 21 | dpr | faiss_flat | 100000 | 1068.483952116003 | 93.59054930302146 | 2022-09-17 10:15:58.178598 | |
| 18 | 22 | dpr | faiss_hnsw | 100000 | 1195.8989856069966 | 83.61910261947709 | 2022-09-17 10:36:24.783587 | |
| 19 | 14 | dpr | faiss_hnsw | 10000 | 114.6786555860017 | 87.20018515128682 | 2022-09-17 08:41:49.541725 | |
| 20 | 6 | dpr | faiss_hnsw | 1000 | 10.710903742990922 | 93.36280336329109 | 2022-09-17 08:25:58.505307 | |
| 21 | 30 | dpr | faiss_hnsw | 500000 | 6349.903588575005 | 78.7413530025274 | 2022-09-17 20:06:17.374842 | |
| 22 | 9 | elastic | opensearch_flat | 10000 | 21.9185139649926 | 456.2353093814486 | 2022-09-17 08:28:40.051299 | |
| 23 | 10 | dpr | opensearch_flat | 10000 | 123.36043137000524 | 81.06327036102982 | 2022-09-17 08:31:26.201876 | |
| 24 | 17 | elastic | opensearch_flat | 100000 | 208.89336037798785 | 478.7131568904452 | 2022-09-17 08:52:13.178937 | |
| 25 | 1 | elastic | opensearch_flat | 1000 | 2.2300678319879808 | 448.4168533602657 | 2022-09-17 08:21:45.610495 | |
| 26 | 18 | dpr | opensearch_flat | 100000 | 1245.0073385680007 | 80.32081169498916 | 2022-09-17 09:13:46.979550 | |
| 27 | 25 | elastic | opensearch_flat | 500000 | 1045.8889810429973 | 478.06221220667453 | 2022-09-17 11:28:08.906406 | |
| 28 | 26 | dpr | opensearch_flat | 500000 | 6235.606822678994 | 80.18465791997863 | 2022-09-17 13:13:24.174105 | |
| 29 | 2 | dpr | opensearch_flat | 1000 | 12.919973790005315 | 77.3995378205476 | 2022-09-17 08:22:43.256934 | |
| 30 | 11 | dpr | opensearch_hnsw | 10000 | 121.49061479599914 | 82.3108848102505 | 2022-09-17 08:34:11.508717 | |
| 31 | 3 | dpr | opensearch_hnsw | 1000 | 12.355953149002744 | 80.93264744053441 | 2022-09-17 08:23:36.852546 | |
| 32 | 19 | dpr | opensearch_hnsw | 100000 | 1242.960266391994 | 80.45309468360983 | 2022-09-17 09:35:31.609351 | |
| 33 | 27 | dpr | opensearch_hnsw | 500000 | 6237.192302857991 | 80.16427516125985 | 2022-09-17 14:59:57.763007 |