mirror of https://github.com/deepset-ai/haystack.git synced 2025-09-18 20:53:25 +00:00

History

Integrate sentence transformers into benchmarks (#843 )

* Integrate sentence transformers into benchmarks

* Add doc store asserts

* switch data downloads from s3 client to https. add license info

* Fix mypy, revert config

Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

2021-04-09 17:24:16 +02:00

data_scripts

Create time and performance benchmarks for all readers and retrievers (#339 )

2020-10-12 13:34:42 +02:00

config.json

Integrate sentence transformers into benchmarks (#843 )

2021-04-09 17:24:16 +02:00

nq_to_squad.py

Create time and performance benchmarks for all readers and retrievers (#339 )

2020-10-12 13:34:42 +02:00

reader_results.csv

Choose correct similarity fns during benchmark runs & re-run benchmarks (#773 )

2021-02-03 11:45:18 +01:00

reader.py

Choose correct similarity fns during benchmark runs & re-run benchmarks (#773 )

2021-02-03 11:45:18 +01:00

README.md

add readme

2020-10-22 15:32:56 +02:00

results_to_json.py

add automatic json update

2020-10-21 17:59:44 +02:00

retriever_index_results.csv

Choose correct similarity fns during benchmark runs & re-run benchmarks (#773 )

2021-02-03 11:45:18 +01:00

retriever_query_results.csv

Choose correct similarity fns during benchmark runs & re-run benchmarks (#773 )

2021-02-03 11:45:18 +01:00

retriever_query_results.md

Choose correct similarity fns during benchmark runs & re-run benchmarks (#773 )

2021-02-03 11:45:18 +01:00

retriever.py

Integrate sentence transformers into benchmarks (#843 )

2021-04-09 17:24:16 +02:00

run.py

Integrate sentence transformers into benchmarks (#843 )

2021-04-09 17:24:16 +02:00

templates.py

Choose correct similarity fns during benchmark runs & re-run benchmarks (#773 )

2021-02-03 11:45:18 +01:00

utils.py

Integrate sentence transformers into benchmarks (#843 )

2021-04-09 17:24:16 +02:00

README.md

Benchmarks

Run the benchmarks with the following command:

python run.py [--reader] [--retriever_index] [--retriever_query] [--ci] [--update-json]

You can specify which components and processes to benchmark with the following flags.

--reader will trigger the speed and accuracy benchmarks for the reader. Here we simply use the SQuAD dev set.

--retriever_index will trigger indexing benchmarks

--retriever_query will trigger querying benchmarks (embeddings will be loaded from file instead of being computed on the fly)

--ci will cause the the benchmarks to run on a smaller slice of each dataset and a smaller subset of Retriever / Reader / DocStores.

--update-json will cause the script to update the json files in docs/_src/benchmarks so that the website benchmarks will be updated.