haystack/reader_results.csv at 4c2804e38eada91a16ffbec0b2d3b36234facdb8

mirror of https://github.com/deepset-ai/haystack.git synced 2025-12-31 09:10:15 +00:00

* add latest benchmark run

* update templates and fix small json errors

* Change scale

Co-authored-by: brandenchan <brandenchan@icloud.com>

2020-12-10 16:25:51 +01:00

1		EM	f1	top_n_accuracy	top_n	reader_time	seconds_per_query	passages_per_second	reader
2	0	0.7836676217765043	0.8262983412843887	0.9742963087813922	5	124.91606550999859	0.01052722615118815	98.86638639776463	deepset/roberta-base-squad2
3	1	0.7439743805831789	0.7890026641413856	0.9720209000505646	5	67.87064415001078	0.005719757639475036	181.96379531485616	deepset/minilm-uncased-squad2
4	2	0.6947581324793528	0.7432668866064459	0.9557559413450194	5	116.45726653200109	0.009814365964267747	106.04748306200683	deepset/bert-base-cased-squad2
5	3	0.7900724759817968	0.8329492827667042	0.976908815101972	5	305.62878707199707	0.02575668187021718	40.40849724371908	deepset/bert-large-uncased-whole-word-masking-squad2
6	4	0.803472105174448	0.846217441464372	0.9742120343839542	5	305.06433064700104	0.025709112645120602	40.48326454229272	deepset/xlm-roberta-large-squad2
7	5	0.3730827574582842	0.42342513261953935	0.9539019046013821	5	76.98679084099422	0.006488015408814615	160.417129550279	distilbert-base-uncased-distilled-squad