haystack/test/benchmarks

Benchmarks

The tooling provided in this directory allows running benchmarks on reader pipelines, retriever pipelines, and retriever-reader pipelines.

Defining configuration

To run a benchmark, you need to create a configuration file first. This file should be a Pipeline YAML file that contains both the querying and, optionally, the indexing pipeline, in case the querying pipeline includes a retriever.

The configuration file should also have a benchmark_config section that includes the following information:

• labels_file: The path to a SQuAD-formatted JSON or CSV file that contains the labels to be benchmarked on.
• documents_directory: The path to a directory containing files intended to be indexed into the document store. This is only necessary for retriever and retriever-reader pipelines.
• data_url: This is optional. If provided, the benchmarking script will download data from this URL and save it in the data/ directory.

Here is an example of how a configuration file for a retriever-reader pipeline might look like:

components:
  - name: DocumentStore
    type: ElasticsearchDocumentStore
  - name: TextConverter
    type: TextConverter
  - name: Reader
    type: FARMReader
    params:
      model_name_or_path: deepset/roberta-base-squad2-distilled
  - name: Retriever
    type: BM25Retriever
    params:
      document_store: DocumentStore
      top_k: 10

pipelines:
  - name: indexing
    nodes:
      - name: TextConverter
        inputs: [File]
      - name: Retriever
        inputs: [TextConverter]
      - name: DocumentStore
        inputs: [Retriever]
  - name: querying
    nodes:
      - name: Retriever
        inputs: [Query]
      - name: Reader
        inputs: [Retriever]

benchmark_config:
  data_url: http://example.com/data.tar.gz
  documents_directory: /path/to/documents
  labels_file: /path/to/labels.csv

Running benchmarks

Once you have your configuration file, you can run benchmarks by using the run.py script.

python run.py [--output OUTPUT] config

The script takes the following arguments:

• config: This is the path to your configuration file.
• --output: This is an optional path where benchmark results should be saved. If not provided, the script will create a JSON file with the same name as the specified config file.

Metrics

The benchmarks yield the following metrics:

• Reader pipelines:
  • Exact match score
  • F1 score
  • Total querying time
  • Seconds/query
• Retriever pipelines:
  • Recall
  • Mean-average precision
  • Total querying time
  • Seconds/query
  • Queries/second
  • Total indexing time
  • Number of indexed Documents/second
• Retriever-Reader pipelines:
  • Exact match score
  • F1 score
  • Total querying time
  • Seconds/query
  • Total indexing time
  • Number of indexed Documents/second

You can find more details about the performance metrics in our evaluation guide.