b8ff1052d4
* Generate eval result in separate method * Adapt benchmarking utils * Adapt running retriever benchmarks * Adapt error message * Adapt running reader benchmarks * Adapt retriever reader benchmark script * Adapt running benchmarks script * Adapt README.md * Raise error if file doesn't exist * Raise error if path doesn't exist or is a directory * minor readme update * Create separate methods for checking if pipeline contains reader or retriever * Fix reader pipeline case --------- Co-authored-by: Darja Fokina <daria.f93@gmail.com>
Benchmarks
The tooling provided in this directory allows running benchmarks on reader pipelines, retriever pipelines, and retriever-reader pipelines.
Defining configuration
To run a benchmark, you need to create a configuration file first. This file should be a Pipeline YAML file that contains both the querying and, optionally, the indexing pipeline, in case the querying pipeline includes a retriever.
The configuration file should also have a benchmark_config section that includes the following information:
labels_file: The path to a SQuAD-formatted JSON or CSV file that contains the labels to be benchmarked on.documents_directory: The path to a directory containing files intended to be indexed into the document store. This is only necessary for retriever and retriever-reader pipelines.data_url: This is optional. If provided, the benchmarking script will download data from this URL and save it in thedata/directory.
Here is an example of how a configuration file for a retriever-reader pipeline might look like:
components:
- name: DocumentStore
type: ElasticsearchDocumentStore
- name: TextConverter
type: TextConverter
- name: Reader
type: FARMReader
params:
model_name_or_path: deepset/roberta-base-squad2-distilled
- name: Retriever
type: BM25Retriever
params:
document_store: DocumentStore
top_k: 10
pipelines:
- name: indexing
nodes:
- name: TextConverter
inputs: [File]
- name: Retriever
inputs: [TextConverter]
- name: DocumentStore
inputs: [Retriever]
- name: querying
nodes:
- name: Retriever
inputs: [Query]
- name: Reader
inputs: [Retriever]
benchmark_config:
data_url: http://example.com/data.tar.gz
documents_directory: /path/to/documents
labels_file: /path/to/labels.csv
Running benchmarks
Once you have your configuration file, you can run benchmarks by using the run.py script.
python run.py [--output OUTPUT] config
The script takes the following arguments:
config: This is the path to your configuration file.--output: This is an optional path where benchmark results should be saved. If not provided, the script will create a JSON file with the same name as the specified config file.
Metrics
The benchmarks yield the following metrics:
- Reader pipelines:
- Exact match score
- F1 score
- Total querying time
- Seconds/query
- Retriever pipelines:
- Recall
- Mean-average precision
- Total querying time
- Seconds/query
- Queries/second
- Total indexing time
- Number of indexed Documents/second
- Retriever-Reader pipelines:
- Exact match score
- F1 score
- Total querying time
- Seconds/query
- Total indexing time
- Number of indexed Documents/second
You can find more details about the performance metrics in our evaluation guide.