haystack/retriever_query_results.csv at 159cd5a666581a593f99aaab220207b41349b48f

mirror of https://github.com/deepset-ai/haystack.git synced 2025-08-31 20:03:38 +00:00

bug: reactivate benchmarks with quick fixes (#2766 )

* quick fix benchmark runs to make them work with current haystack version

* fix minor typo

* update readme. fix minor things to make benchmarks run again

* Update Documentation & Code Style

* fix typo in readme

* update result files for reader and retriever querying

* reduce batch size for update embeddings to prevent xlarge bulk_update requests that exceed elastic's limits (happening in dense 500k runs)

* change default memory allocation back to normal. add note to readme

* add first indexing results

* add memory to docker cmd

* full benchmarks results on commit  c5a2651fcbbeffca06ffa9036b10e62669bcc1b0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

2022-09-20 10:22:08 +02:00

5.1 KiB

Raw Blame History

1		retriever	doc_store	n_docs	n_queries	retrieve_time	queries_per_second	seconds_per_query	recall	map	top_k	date_time
2	31	sentence_transformers	elasticsearch	500000	5637	3910.198875275557	1.4416146543448518	0.6936666445406345	89.6398793684584	76.49564526892908	10	2022-09-18 08:25:36.627368
3	28	dpr	elasticsearch	500000	5637	3839.528848723392	1.4681488854743858	0.6811298294701778	93.0814262905801	80.86137228234092	10	2022-09-18 03:20:53.663119
4	7	sentence_transformers	elasticsearch	1000	1064	22.08566212869482	48.176051675516526	0.020757201248773328	98.7781954887218	90.06638620360428	10	2022-09-17 21:28:29.461918
5	15	sentence_transformers	elasticsearch	10000	5637	187.41362470001332	30.0778559137467	0.03324705068298977	96.52297321270179	87.11255142468552	10	2022-09-17 22:03:32.200427
6	20	dpr	elasticsearch	100000	5637	903.5089274104685	6.239008635095737	0.16028187465149343	95.7956359765833	86.5456409043424	10	2022-09-17 22:58:20.213034
7	0	elastic	elasticsearch	1000	1064	4.122031082282774	258.12517634164914	0.00387408936304772	89.09774436090225	74.2044471297291	10	2022-09-17 21:19:21.768721
8	24	elastic	elasticsearch	500000	5637	83.56125180030358	67.45949681882962	0.014823709739276845	62.746141564662054	45.603960228760656	10	2022-09-18 00:29:07.528766
9	16	elastic	elasticsearch	100000	5637	42.696119271276984	132.02605052193084	0.007574262776525986	71.93542664537875	56.26106985872818	10	2022-09-17 22:08:40.215954
10	8	elastic	elasticsearch	10000	5637	28.68551177883637	196.51035140878582	0.005088790452161854	81.10697179350718	66.32941083712781	10	2022-09-17 21:30:14.745163
11	4	dpr	elasticsearch	1000	1064	29.50371559121413	36.06325436233689	0.0277290560067802	99.15413533834587	92.9510532283089	10	2022-09-17 21:24:50.768859
12	12	dpr	elasticsearch	10000	5637	224.79332777825766	25.076367059971158	0.03987818481076063	97.49866950505589	89.87097014904354	10	2022-09-17 21:48:36.651933
13	23	sentence_transformers	elasticsearch	100000	5637	886.5846678019152	6.358106794217024	0.15727952240587462	93.79102359411034	82.74686664920834	10	2022-09-18 00:09:04.840841
14	29	dpr	faiss_flat	500000	5637	3683.516663164017	1.5303310709494677	0.6534533729224795	93.0814262905801	80.86137228234091	10	2022-09-18 05:09:24.992364
15	13	dpr	faiss_flat	10000	5637	181.46477086472441	31.06388073640026	0.032191728022835625	97.49866950505589	89.87097014904354	10	2022-09-17 21:53:56.820064
16	5	dpr	faiss_flat	1000	1064	29.979966158716707	35.49036694594937	0.028176659923605928	99.15413533834587	92.9510532283089	10	2022-09-17 21:26:11.004926
17	21	dpr	faiss_flat	100000	5637	841.2282260610373	6.700916380795566	0.14923332021661118	95.7956359765833	86.54606328368972	10	2022-09-17 23:23:47.065373
18	22	dpr	faiss_hnsw	100000	5637	138.64402735611657	40.6580803190388	0.024595356990618513	93.18786588610963	84.33419639513305	10	2022-09-17 23:39:25.396237
19	14	dpr	faiss_hnsw	10000	5637	112.71057137736352	50.01305495228924	0.019994779382182637	96.98421145999644	89.51337675393017	10	2022-09-17 21:58:05.961829
20	6	dpr	faiss_hnsw	1000	1064	25.61499380285386	41.538171283159	0.02407424229591528	99.15413533834587	92.9510532283089	10	2022-09-17 21:27:21.948745
21	30	dpr	faiss_hnsw	500000	5637	124.99580949536175	45.097511850660666	0.022174172342622273	86.90792974986695	75.73315903145605	10	2022-09-18 06:09:37.007518
22	9	elastic	opensearch_flat	10000	5637	32.42948475538287	173.8232982275283	0.005752968734323731	81.10697179350718	66.32941083712774	10	2022-09-17 21:32:03.707902
23	10	dpr	opensearch_flat	10000	5637	182.6267182478914	30.86623936563611	0.03239785670532046	97.49866950505589	89.87097014904354	10	2022-09-17 21:37:08.044354
24	17	elastic	opensearch_flat	100000	5637	46.926014778233366	120.12526583899732	0.008324643388013725	71.93542664537875	56.26106985872818	10	2022-09-17 22:13:58.492431
25	1	elastic	opensearch_flat	1000	1064	6.101849522849079	174.3733594241761	0.005734820980121315	89.09774436090225	74.2044471297291	10	2022-09-17 21:20:04.003708
26	18	dpr	opensearch_flat	100000	5637	320.2046442734136	17.604366772353018	0.056804088038569026	95.76015611140679	86.51945338480733	10	2022-09-17 22:25:11.379134
27	25	elastic	opensearch_flat	500000	5637	83.39615098570357	67.59304756122785	0.014794420966064143	62.746141564662054	45.59509026246651	10	2022-09-18 00:49:52.903674
28	26	dpr	opensearch_flat	500000	5637	466.1570336711302	12.092491570076481	0.08269594352867309	93.0814262905801	80.861372282341	10	2022-09-18 01:20:53.923798
29	2	dpr	opensearch_flat	1000	1064	30.094220374303404	35.35562598951788	0.02828404170517237	99.15413533834587	92.9510532283089	10	2022-09-17 21:21:42.413924
30	11	dpr	opensearch_hnsw	10000	5637	181.91717168685864	30.986629506879073	0.032271983623710955	97.49866950505589	89.87097014904357	10	2022-09-17 21:42:16.601719
31	3	dpr	opensearch_hnsw	1000	1064	29.82644817981054	35.67303735213834	0.028032376108844494	99.15413533834587	92.9510532283089	10	2022-09-17 21:23:12.487039
32	19	dpr	opensearch_hnsw	100000	5637	327.2236303977261	17.226750993345046	0.058049251445401115	95.77789604399504	86.53719331739565	10	2022-09-17 22:36:46.502520
33	27	dpr	opensearch_hnsw	500000	5637	477.3347775057773	11.809321812785313	0.08467886774982744	93.0814262905801	80.861372282341	10	2022-09-18 01:53:29.735540

5.1 KiB Raw Blame History

5.1 KiB

Raw Blame History