haystack/retriever_query_results.csv at efc754b1663215f7f8a555f4d2c8fd9282d362c8

mirror of https://github.com/deepset-ai/haystack.git synced 2025-12-31 17:17:31 +00:00

* add time and perf benchmark for es

* Add retriever benchmarking

* Add Reader benchmarking

* add nq to squad conversion

* add conversion stats

* clean benchmarks

* Add link to dataset

* Update imports

* add first support for neg psgs

* Refactor test

* set max_seq_len

* cleanup benchmark

* begin retriever speed benchmarking

* Add support for retriever query index benchmarking

* improve reader eval, retriever speed benchmarking

* improve retriever speed benchmarking

* Add retriever accuracy benchmark

* Add neg doc shuffling

* Add top_n

* 3x speedup of SQL. add postgres docker run. make shuffle neg a param. add more logging

* Add models to sweep

* add option for faiss index type

* remove unneeded line

* change faiss to faiss_flat

* begin automatic benchmark script

* remove existing postgres docker for benchmarking

* Add data processing scripts

* Remove shuffle in script bc data already shuffled

* switch hnsw setup from 256 to 128

* change es similarity to dot product by default

* Error includes stack trace

* Change ES default timeout

* remove delete_docs() from timing for indexing

* Add support for website export

* update website on push to benchmarks

* add complete benchmarks results

* new json format

* removed NaN as is not a valid json token

* versioning for docs

* unsaved changes

* cleaning

* cleaning

* Edit format of benchmarks data

* update also jsons in v0.4.0

Co-authored-by: brandenchan <brandenchan@icloud.com>
Co-authored-by: deepset <deepset@Crenolape.localdomain>
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>

2020-10-19 11:46:51 +02:00

1.8 KiB

Raw Blame History

1	retriever	doc_store	n_docs	n_queries	retrieve_time	queries_per_second	seconds_per_query	recall	map	top_k	date_time	name	note
2	dpr	elasticsearch	1000	1085	26.592	40.802	0.025	0.991	0.929	10	2020-10-07 15:06:57	dpr-elasticsearch
3	dpr	elasticsearch	10000	5791	214.425	27.007	0.037	0.975	0.898	10	2020-10-07 15:11:35	dpr-elasticsearch
4	dpr	elasticsearch	100000	5791	886.045	6.536	0.153	0.958	0.863	10	2020-10-07 15:30:52	dpr-elasticsearch
5	dpr	elasticsearch	500000	5791	3824.624	1.514	0.660	0.930	0.805	10	2020-10-07 17:44:02	dpr-elasticsearch
6	dpr	faiss_flat	1000	1085	27.092	40.048	0.025	0.991	0.929	10	2020-10-07 13:06:35	dpr-faiss_flat
7	dpr	faiss_flat	10000	5791	241.524	23.977	0.042	0.975	0.898	10	2020-10-07 13:17:21	dpr-faiss_flat
8	dpr	faiss_flat	100000	5791	1148.181	5.044	0.198	0.958	0.863	10	2020-10-07 14:04:51	dpr-faiss_flat
9	dpr	faiss_flat	500000	5791	5308.016	1.091	0.917	0.930	0.805	10	2020-10-08 10:01:32	dpr-faiss_flat
10	elastic	elasticsearch	1000	1085	4.657	232.978	0.004	0.891	0.748	10	2020-10-07 13:04:47	elastic-elasticsearch
11	elastic	elasticsearch	10000	5791	34.509	167.810	0.006	0.811	0.661	10	2020-10-07 13:07:52	elastic-elasticsearch
12	elastic	elasticsearch	100000	5791	35.529	162.996	0.006	0.717	0.560	10	2020-10-07 13:21:48	elastic-elasticsearch
13	elastic	elasticsearch	500000	5791	60.645	95.491	0.010	0.624	0.452	10	2020-10-07 16:14:52	elastic-elasticsearch
14	dpr	faiss_hnsw	1000	1085	28.640	37.884	0.026	0.991	0.929	10	2020-10-09 07:19:29	dpr-faiss_hnsw	128,20,80
15	dpr	faiss_hnsw	10000	5791	173.272	33.421	0.030	0.972	0.896	10	2020-10-09 07:23:28	dpr-faiss_hnsw	128,20,80
16	dpr	faiss_hnsw	100000	5791	451.884	12.815	0.078	0.940	0.849	10	2020-10-09 07:37:56	dpr-faiss_hnsw	128,20,80
17	dpr	faiss_hnsw	500000	5791	1777.023	3.259	0.307	0.882	0.766	10	2020-10-09	dpr-faiss_hnsw	128,20,80

1.8 KiB Raw Blame History

1.8 KiB

Raw Blame History