mirror of
https://github.com/deepset-ai/haystack.git
synced 2026-01-07 12:37:27 +00:00
2531c8e061
* add time and perf benchmark for es * Add retriever benchmarking * Add Reader benchmarking * add nq to squad conversion * add conversion stats * clean benchmarks * Add link to dataset * Update imports * add first support for neg psgs * Refactor test * set max_seq_len * cleanup benchmark * begin retriever speed benchmarking * Add support for retriever query index benchmarking * improve reader eval, retriever speed benchmarking * improve retriever speed benchmarking * Add retriever accuracy benchmark * Add neg doc shuffling * Add top_n * 3x speedup of SQL. add postgres docker run. make shuffle neg a param. add more logging * Add models to sweep * add option for faiss index type * remove unneeded line * change faiss to faiss_flat * begin automatic benchmark script * remove existing postgres docker for benchmarking * Add data processing scripts * Remove shuffle in script bc data already shuffled * switch hnsw setup from 256 to 128 * change es similarity to dot product by default * Error includes stack trace * Change ES default timeout * remove delete_docs() from timing for indexing * Add support for website export * update website on push to benchmarks * add complete benchmarks results * new json format * removed NaN as is not a valid json token * versioning for docs * unsaved changes * cleaning * cleaning * Edit format of benchmarks data * update also jsons in v0.4.0 Co-authored-by: brandenchan <brandenchan@icloud.com> Co-authored-by: deepset <deepset@Crenolape.localdomain> Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
906 B
906 B
| 1 | retriever | doc_store | n_docs | indexing_time | docs_per_second | date_time | Notes |
|---|---|---|---|---|---|---|---|
| 2 | dpr | elasticsearch | 1000 | 14.16526405 | 70.59522482 | 2020-10-08 10:30:56 | |
| 3 | elastic | elasticsearch | 1000 | 5.805040058 | 172.2640998 | 2020-10-08 10:30:25 | |
| 4 | elastic | elasticsearch | 10000 | 22.56448254 | 443.1743553 | 2020-10-08 13:01:09 | |
| 5 | dpr | elasticsearch | 10000 | 126.2442168 | 79.21154929 | 2020-10-08 13:03:32 | |
| 6 | dpr | elasticsearch | 100000 | 1257.202958 | 79.54165185 | 2020-10-08 13:28:16 | |
| 7 | elastic | elasticsearch | 100000 | 209.681252 | 476.9143596 | 2020-10-08 13:07:05 | |
| 8 | dpr | faiss_flat | 1000 | 8.223732258 | 121.5992895 | 44112.24392 | |
| 9 | dpr | faiss_flat | 10000 | 89.72649358 | 111.4498026 | 44112.24663 | |
| 10 | dpr | faiss_flat | 100000 | 927.0740565 | 107.8662479 | 44112.56656 | |
| 11 | dpr | faiss_hnsw | 1000 | 8.86507699 | 112.8021788 | 44113.37262 | hnsw 128,20,80 |
| 12 | dpr | faiss_hnsw | 10000 | 100.1804832 | 99.81984193 | 44113.37413 | hnsw 128,20,80 |
| 13 | dpr | faiss_hnsw | 100000 | 1084.063917 | 92.24548333 | 44113.38721 | hnsw 128,20,80 |