Julian Risch
e974a23fa3
docs: Fix eval metric examples in docstrings ( #7505 )
...
* fix eval metric docstrings, change type of individual scores
* change import order
* change exactmatch docstring to single ground truth answer
* change exactmatch comment to single ground truth answer
* reverted changing docs to single ground truth
* add warm up in SASEvaluator example
* fix FaithfulnessEvaluator docstring example
* extend FaithfulnessEvaluator docstring example
* Update FaithfulnessEvaluator init docstring
* Remove outdated default from LLMEvaluator docstring
* Add examples param to LLMEvaluator docstring example
* Add import and print to LLMEvaluator docstring example
2024-04-10 11:00:20 +02:00
David S. Batista
aae2b31359
fix: typo in sas_evaluator arg ( #7486 )
...
* fixing typo on SAS arg
* fixing tests
* fixing tests
2024-04-08 10:21:37 +02:00
Julian Risch
9d02dc607a
feat: Add FaithfulnessEvaluator component ( #7424 )
...
* draft FaithfulnessEvaluator
* reno
* calculate score per statement and aggregate
* Update release note
* update default values in tests and fix import path
* remove instructions, inputs, outputs params
* remove unused imports
* add expected format example to docstring
* remove name 'llm' from tests and docstring
2024-04-04 16:33:59 +00:00
Julian Risch
8ef6062748
refactor: Remove name 'llm' from LLMEvaluator output ( #7479 )
2024-04-04 15:19:30 +00:00
Silvano Cerza
8b8a93bc0d
refactor: Rename DocumentMeanAveragePrecision and DocumentMeanReciprocalRank ( #7470 )
...
* Rename DocumentMeanAveragePrecision and DocumentMeanReciprocalRank
* Update releasenotes
* Simplify names
2024-04-04 17:04:59 +02:00
Silvano Cerza
bdc25ca2a0
feat: Add DocumentMeanReciprocalRank ( #7468 )
...
* Add DocumentMeanReciprocalRank
* Fix float precision error
2024-04-04 14:55:37 +02:00
Silvano Cerza
7799909069
feat: Add DocumentMeanAveragePrecision ( #7461 )
...
* Add DocumentMeanAveragePrecision
* Remove questions input
* Update docstrings
* Update haystack/components/evaluators/document_map.py
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
---------
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
2024-04-04 14:15:45 +02:00
Silvano Cerza
dc87f51759
refactor: Remove questions inputs from evaluators ( #7466 )
...
* Remove questions input from AnswerExactMatchEvaluator
* Remove questions input from DocumentRecallEvaluator
2024-04-04 14:14:18 +02:00
Silvano Cerza
12acb3f12e
feat: Add SASEvaluator ( #7428 )
...
* Add SASEvaluator
* Add release notes
* Apply suggestions from code review
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* Simplify similarity calculation with bi-encoders models
* Fix linting
* Update docstrings
* Move tensor to CPU after calculating cosine similarity
* Fix CI failing
---------
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
2024-04-04 10:10:41 +02:00
Silvano Cerza
685343d13f
feat: Add DocumentRecallEvaluator ( #7399 )
...
* Add DocumentRecallEvaluator
* Fix mypy error
* Simplify recall logic and change output for single hit mode
* Remove unused import
* Add comment for RecallMode fields
* Reword RecallMode comments
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
---------
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
2024-03-26 16:15:03 +01:00
Silvano Cerza
f398b29e7f
feat: Change outputs of AnswerExactMatchEvaluator ( #7390 )
...
* Change outputs of AnswerExactMatchEvaluator
* Changes scores to return the number of matches per question
* Revert "Changes scores to return the number of matches per question"
This reverts commit e4358720793d4584b0b961402d4557c50c4c2381.
* Change output names
2024-03-26 10:57:59 +01:00
Julian Risch
bfd0d3eacd
feat: Add new LLMEvaluator component ( #7401 )
...
* draft llm evaluator
* docstrings
* flexible inputs; validate inputs and outputs
* add tests
* add release note
* remove example
* docstrings
* make outputs parameter optional. default:
* validate init parameters
* linting
* remove mention of binary scores from template
* make examples and outputs params non-optional
* removed leftover from optional outputs param
* simplify building examples section for template
* validate inputs and outputs in examples are dict with str as key
* fix pylint too-many-boolean-expressions
* increase test coverage
2024-03-25 07:05:27 +01:00
Silvano Cerza
610ad6f6b2
Add AnswerExactMatchEvaluator ( #7381 )
...
* Add AnswerExactMatchEvaluator
* Add release notes
* Fix linting
* Update docstrings
* Update docstrings
* Remove to_dict and from_dict
* Fix linting
2024-03-19 16:58:01 +01:00
Silvano Cerza
0a7dfc1b32
Revert "Add AnswerExactMatchEvaluator ( #7050 )" ( #7075 )
...
This reverts commit b4011af8e9bc4ae2f72e51db254bfda69e20b651.
2024-02-23 14:05:57 +01:00
Silvano Cerza
b4011af8e9
Add AnswerExactMatchEvaluator ( #7050 )
...
* Add AnswerExactMatchEvaluator
* Add release notes
* Fix linting
* Update docstrings
2024-02-23 10:37:18 +01:00
Silvano Cerza
8ca4bf405b
Remove all evaluator components ( #7053 )
2024-02-21 18:24:14 +01:00
Ashwin Mathur
327c2d260d
feat: Add Mean Reciprocal Rank (MRR) metric to StatisticalEvaluator ( #7042 )
...
* Add MRR Metric
* Add release notes
* Update logic
2024-02-20 13:58:48 +01:00
Silvano Cerza
9215882779
Add Recall Multi Hit and Single Hit metric ( #7038 )
2024-02-19 18:00:39 +01:00
Silvano Cerza
6fe1d3b595
refactor: Clean eval components ( #7005 )
...
* Remove preprocess.py
* Rename eval components to evaluators
2024-02-15 17:17:59 +01:00
