mirror of
https://github.com/deepset-ai/haystack.git
synced 2025-08-28 18:36:36 +00:00
84147edcca
* initial commit * Add latest docstring and tutorial changes * added comments and fixed bug * fixed bugs, added benchmark and added documentation * Add latest docstring and tutorial changes * fix type: ignore comment * fix logging in benchmark * fixed distillation config * Add latest docstring and tutorial changes * added type annotations * fixed distillation loss calculation * added type annotations * fixed distillation mse loss * improved model distillation benchmark config loading * added temperature for model distillation * removed uncessary imports, added comments, added named parameter calls * Add latest docstring and tutorial changes * added some more comments * added distillation test * fixed distillation test * removed unnecessary import * fix softmax dimension * add grid search * improved model distillation benchmark config * fixed model distillation hyperparameter search * added doc strings and type hints for model distillation * Add latest docstring and tutorial changes * fixed type hints * fixed type hints * fixed type hints * wrote out params instead of kwargs in DistillationDataSilo initializer * fixed type hints * fixed typo * fixed typo Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Benchmarks
Run the benchmarks with the following command:
python run.py [--reader] [--retriever_index] [--retriever_query] [--ci] [--update-json]
You can specify which components and processes to benchmark with the following flags.
--reader will trigger the speed and accuracy benchmarks for the reader. Here we simply use the SQuAD dev set.
--retriever_index will trigger indexing benchmarks
--retriever_query will trigger querying benchmarks (embeddings will be loaded from file instead of being computed on the fly)
--ci will cause the the benchmarks to run on a smaller slice of each dataset and a smaller subset of Retriever / Reader / DocStores.
--update-json will cause the script to update the json files in docs/_src/benchmarks so that the website benchmarks will be updated.