mirror of
https://github.com/deepset-ai/haystack.git
synced 2025-11-30 08:56:16 +00:00
38747ff7a3
* wip * initial import * adding tests * adding params * adding safeguards for nan in evaluators * adding docstrings * fixing tests * removing unused imports * adding tests to context and faithfullness evaluators * fixing docstrings * nit * removing unused imports * adding release notes * attending PR comments * fixing tests * fixing tests * adding types * removing unused imports * Update haystack/components/evaluators/context_relevance.py Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com> * Update haystack/components/evaluators/faithfulness.py Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com> * attending PR comments --------- Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
6 lines
370 B
YAML
6 lines
370 B
YAML
---
|
|
enhancements:
|
|
- |
|
|
If an LLM-based evaluator (e.g., `Faithfulness` or `ContextRelevance`) is initialised with `raise_on_failure=False`, and if a call to an LLM fails or an LLM outputs an invalid JSON, the score of the sample is set to `NaN` instead of raising an exception.
|
|
The user is notified with a warning indicating the number of requests that failed.
|