4180 Commits

Author SHA1 Message Date
Stefano Fiorucci
bcaef53cbc
test: export HF_TOKEN env var in e2e environment (#9551)
* try to fix e2e tests for private NER models

* explanatory comment

* extend skipif condition
2025-06-25 15:00:28 +02:00
Haystack Bot
85e8493f4f
Update unstable version to 2.16.0-rc0 (#9554)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-06-25 14:57:16 +02:00
Amna Mubashar
1cd0a128d0
feat: enable parallel tool execution in ToolInvoker (#9530)
* Enable parallel tool execution in ToolInvoker

* Update handling of errors

* Small fixes

* Small fixes

* Adapt number of executors

* Add release notes

* Add parallel tool calling to sync run

* Deprecate async_executor

* Deprecate async_executor

* Add thread lock

* extract methods

* Update release notes

* Update release notes

* Updates

* Add new tests

* Add test for async

* PR comments
v2.16.0-rc0
2025-06-25 13:32:11 +02:00
Vladimir Blagojevic
91094e1038
feat: Add finish_reason field to StreamingChunk (#9536)
* Initial commit

* Update deprecation version

* Improve comment

* Minor simplification

* Add reno note

* Remove deprecation warning

* Remove fallback in haystack/components/generators/utils.py

* FinishReason alphabetical import

* Add tool_call_results finish reason, adapt codebase

* Define finish_reason to be Optional[FinishReason]

* Add StreamingChunk finish_reason in HF generators

* Update reno note

* Repair merge issue

* Update tests for finish_reason

* Resolve mypy issues

* Lint issue

* Enhance HF finish_reason translation

* Remove irrlevant test

* PR comments

---------

Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
2025-06-25 09:06:01 +00:00
Julian Risch
1d1c13a8bc
chore: add DocusaurusRenderer and use description, title, id (#9538) 2025-06-25 09:56:26 +02:00
Stefano Fiorucci
0d0a66b4f5
feat: add LLMMessagesRouter, a component to route Chat Messages using LLMs (#9540)
* llmmessagesrouter - draft

* serde methods

* refinements, tests and release note

* Apply suggestions from code review

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2025-06-24 14:54:20 +02:00
Michele Pangrazzi
3207a76d50
chore: Update pydoc-markdown.sh (#9547)
* Make config path a $1 param ; Add usage in comment ; Add echo log

* Update sync command
2025-06-24 14:01:51 +02:00
Amna Mubashar
9ed0b9b0bc
fix: Update the de/serialization with schema utils (#9526)
* Update the util methods

* Update tests

* fix tests

* schema fix

* Add json schema for tuples and sets

* Add proper conversion for sets and tuples

* Adjust typing

* PR comments

* Linting

* Optimize deserialization

* remove TODO

* PR comments

* PR comments

* Update tests and deserialization error

* Support legacy deserialization

* Update deprecating warning

* Update test
2025-06-24 13:10:12 +02:00
Stefano Fiorucci
d14f5dca0e
feat: add trust_remote_code parameter to SentenceTransformersSimilarityRanker (#9546) 2025-06-24 11:39:59 +02:00
Stefano Fiorucci
556dcc9e46
chore: update transformers test dependency (#9537) 2025-06-23 10:26:11 +02:00
Sebastian Husch Lee
ec371387f0
refactor: Update to StreamingChunk, better index setting and change tool_call to tool_calls (#9525)
* Fixes to setting StreamingChunk.index properly and refactoring tests for conversion

* Make _convert_chat_completion_chunk_to_streaming_chunk a member of OpenAIChatGenerator so we can overwrite it in integrations that inherit from it

* Fixes

* Modify streaming chunk to accept a list of tool call deltas.

* Fix tests

* Fix mypy and update original reno

* Undo change

* Update conversion to return a single streaming chunk

* update to print streaming chunk

* Fix types

* PR comments
2025-06-23 08:14:25 +00:00
Ahmad Zidan
f911459647
feat: add resource name for Haystack Component Datadog spans (#9337)
* feat: add resource name for Haystack Component Datadog spans

* fest: format resource name

Signed-off-by: Ahmad Zidan <ahmad.zidan@traveloka.com>

* feat: add release notes

Signed-off-by: Ahmad Zidan <ahmad.zidan@traveloka.com>

---------

Signed-off-by: Ahmad Zidan <ahmad.zidan@traveloka.com>
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
2025-06-18 09:15:15 +00:00
Sebastian Husch Lee
3784889e5d
fix: Fix Tool and ComponentTool serialization when specifying outputs_to_string (#9524)
* Fix serialization of outputs_to_string in Tool and ComponentTool

* Add reno

* Fix mypy, simplify logic

* fix pylint

* Fix test

---------

Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-06-18 11:00:46 +02:00
Stefano Fiorucci
a16ee96003
fix: fix SuperComponent class serialization/deserialization for async Pipelines (#9527)
* draft

* better test + release note

* improve test
2025-06-18 08:17:52 +00:00
Amna Mubashar
67a8f1249b
chore: update linter configuration for compatibility with latest ruff release (#9528)
* Fix linting

* Fix linting

* Update error suppression

* Update pre commit

* Update pyproject.toml
2025-06-18 09:53:19 +02:00
Sriniketh J
6198f0cba9
feat: adding support for torch xpu device (#9470)
* feat: add support for torch xpu device support

* test: xpu based tests ci/cd

* test: add xpu code device support

---------

Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-06-17 14:15:19 +02:00
baki gul
7dbac5b3c9
Fixes incorrect ID generation for identical chunks in RecursiveDocumentSplitter (#9517)
* fix(preprocessor): ensure RecursiveDocumentSplitter generates unique chunk IDs

* fix: update meta handling in RecursiveDocumentSplitter to ensure correct overlap information

---------

Co-authored-by: Michele Pangrazzi <xmikex83@gmail.com>
2025-06-16 21:49:00 +02:00
Stefano Fiorucci
7570f6b769
fix: re-export symbols in __init__.py files (#9521)
* chore: re-export symbols in __init__.py files

* release note
2025-06-16 16:29:08 +02:00
Sebastian Husch Lee
a1484cb91c
Add unit test (#9519) 2025-06-16 13:14:02 +02:00
Sebastian Husch Lee
ba6f5eeb9a
feat: Make PipelineBase().validate_input public (#9520)
* Make validate_input public

* Add reno
2025-06-16 11:58:28 +02:00
Sebastian Husch Lee
c5027d711c
refactor: Refactor HuggingFaceLocalChatGenerator (#9455)
* Refactoring to better align run and run_async and reduce duplicate code

* Docstrings and align run and run_async

* More changes

* add missing type

* Refactor async part a bit

* Fix import error

* Fix mypy
2025-06-13 15:38:00 +02:00
Sebastian Husch Lee
379df4ab84
feat: Warn users if Agent is called with only system messages (#9514)
* Add warning message and raise error in agent run method

* Add tests

* Add reno

* Updates

* Updates
2025-06-13 14:58:50 +02:00
Stefano Fiorucci
580683b79d
chore: improve select_streaming_callback type hints (#9513) 2025-06-13 14:24:18 +02:00
Mohammed Abdul Razak Wahab
a28b2851d9
feat: Add async streaming support in HuggingFaceLocalChatGenerator (#9405)
* feat: Add async streaming support in hugging face generator

* enforce streamingcallback to be async

* refactor

* fix: schedule and await async task in Event Loop

* unenforce typecheck

* add integration test

* After merge fixes:
- fix breaking tests
- added component_info to AsyncHFTokenStreamingHandler

* fix integration test

* refactor: improve async handling in HuggingFaceLocalChatGenerator and update tests

* fix typo

* address review comments

* refactors

* typo

* refactor
2025-06-11 14:50:25 +00:00
Stefano Fiorucci
f8155e1b77
chore: clean up (#9504) 2025-06-11 11:05:05 +02:00
Sebastian Husch Lee
54c5057e0b
feat: (and fix) Add enable_streaming_passthrough to ToolInvoker and add missing params to to_dict (#9498)
* Fixes and tests

* Add reno

* Change variable name

* Add test and fix for passing streaming_callback to a component tool

* Add unit test

* Remove unused import

* Fix reno
2025-06-06 14:16:05 +02:00
Amna Mubashar
1d6a9f652a
fix: serialization of nested ChatMessage in GeneratedAnswerdataclass (#9497)
* Fix serialization

* small fix

* fix the erros

* Fix tests

* PR comments
2025-06-06 11:46:24 +02:00
Stefano Fiorucci
12665ade14
chore: simplify Haystack Hatch scripts (#9491)
* try unifying hatch scripts

* formatting

* simplify

* improve contributing guidelines

* fmt-check
2025-06-06 10:43:02 +02:00
Sebastian Husch Lee
b61886b138
feat: Update streaming chunk (#9424)
* Start expanding StreamingChunk

* First pass at expanding Streaming Chunk

* Working version!

* Some tweaks and also make ToolInvoker stream a chunk with a finish reason

* Properly update test

* Change to tool_name, remove kw_only since its python 3.10 only and update HuggingFaceAPIChatGenerator to start following new StreamingChunk

* Add reno

* Some cleanup

* Fix unit tests

* Fix mypy and integration test

* Fix pylint

* Start refactoring huggingface local api

* Refactor openai generator and chat generator to reuse util methods

* Did some reorg

* Reusue utility method in HuggingFaceAPI

* Get rid of unneeded default values in tests

* Update conversion of streaming chunks to chat message to not rely on openai dataclass anymore

* Fix tests and loosen check in StreamingChunk post_init

* Fixes

* Fix license header

* Add start and index to HFAPIGenerator

* Fix mypy

* Clean up

* Update haystack/components/generators/utils.py

Co-authored-by: Julian Risch <julian.risch@deepset.ai>

* Update haystack/components/generators/utils.py

Co-authored-by: Julian Risch <julian.risch@deepset.ai>

* Change StreamingChunk.start to only a bool

* PR comments

* Fix unit test

* PR comment

* Fix test

---------

Co-authored-by: Julian Risch <julian.risch@deepset.ai>
2025-06-06 08:17:02 +00:00
Stefano Fiorucci
f85ce19a32
test: replace tool calling model in tests with Qwen2.5-72B-Instruct (#9500) 2025-06-06 08:42:46 +02:00
Sebastian Husch Lee
8e21c501df
fix: Fix serialization and deserialization of ConditionalRouter with multiple outputs (#9490)
* Fix sede of ConditionalRouter with multiple outputs

* Add reno
2025-06-05 15:57:24 +02:00
David S. Batista
715a9f9347
chore: fixing release notes (#9496) 2025-06-05 12:36:40 +02:00
David S. Batista
9c2bc666f9
fixing UID colllision on release notes files (#9495) 2025-06-05 12:25:10 +02:00
David S. Batista
529a7f5b6a
docs: fixing typo docstring (#9493) 2025-06-05 11:43:17 +02:00
Vladimir Blagojevic
b69d261280
chore: Make docstring-parser core dep (#9477)
* Make docstring-parser core dep

* Add reno note
2025-06-05 11:28:18 +02:00
Vladimir Blagojevic
853a32f8da
feat: Improve ChatMessage _deserialize_content ValueError - make it more LLM friendly (#9484)
* Improve ChatMessage _deserialize_content ValueError - make it more LLM friendly

* Add unit test

* Add reno note

* Add descriptive ValueError for missing role

* Update haystack/dataclasses/chat_message.py

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>

* Update releasenotes/notes/improve-chatmessage-error-messages-llm-agents-a1b2c3d4e5f6g7h8.yaml

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>

* Add role check in ChatMessage

* fixes + refinements

---------

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
2025-06-04 15:14:05 +00:00
Sebastian Husch Lee
db359cff40
Add state to agent pydocs (#9486) 2025-06-04 14:01:58 +02:00
Sebastian Husch Lee
ff56363db1
fix: In set_output_types check that the decorator @component.output_types is not present on the run_async method (#9485)
* Fix

* Add reno
2025-06-04 12:17:47 +02:00
Stefano Fiorucci
1e2214a1a0
feat: ChatMessage.to_openai_dict_format - add require_tool_call_ids parameter (#9481) 2025-06-03 16:55:13 +02:00
Sebastian Husch Lee
ce0917e586
feat: Add raise_on_failure boolean parameter to OpenAIDocumentEmbedder and AzureOpenAIDocumentEmbedder (#9474)
* Add raise_on_failure to OpenAIDocumentEmbedder

* Add reno

* Add parameter to Azure Doc embedder as well

* Fix bug

* Update reno

* PR comments

* update reno
2025-06-03 10:22:34 +00:00
Sebastian Husch Lee
5fcd7c4732
feat: Allow passing of additional parameters to HF Inference clients in HuggingFaceAPIChatGenerator and HuggingFaceAPIGenerator (#9457)
* Fix tests by allowing passing of provider

* Add reno

* Fix mypy

* Update release note
2025-06-03 10:21:51 +00:00
Sebastian Husch Lee
12e3de364a
Fix test (#9475) 2025-06-03 08:00:10 +00:00
David S. Batista
b85c8e3382
feat: adding deserialize_component_inplace() (#9459)
* adding tests

* adding release notes

* deserialize_chatgenerator_inplace uses deserialize_component_inplace

* removing tests
2025-06-02 09:40:35 +02:00
Sebastian Husch Lee
25c8d7ef9a
fix: In State schema validation use != instead of is not for checking the type of messages (#9454)
* Use != instead of is not

* Add reno

* Use more == instead of is

* Fix mypy
2025-05-30 10:07:37 +02:00
Stefano Fiorucci
2616d4d55b
test: speed up some tests + minor refactorings (#9451)
* this is an integration test

* more improvements

* rm redundant comments
2025-05-29 09:49:11 +02:00
Sebastian Husch Lee
81c0cefa41
refactor: Refactor hf api chat generator (#9449)
* Refactor HFAPI Chat Generator

* Add component info to generators

* Fix type hint

* Add reno

* Fix unit tests

* Remove incorrect dev comment

* Move _convert_streaming_chunks_to_chat_message to utils file
2025-05-27 15:55:06 +02:00
atopx
3deaa20cb6
feat: Add HuggingFace API (text-embeddings-inference for rerank model) for component.rankers (#9414)
* feat(component.rankers): Add HuggingFace API (text-embeddings-inference for rerank) ranker component

* update test flow & doc loaders

* Support run_async for HuggingFaceAPIRanker

* Add release note for HuggingFace API support in component.rankers

* Add release note for HuggingFace API support in component.rankers

* Add release note for HuggingFace API support in component.rankers

* Add release note for HuggingFace API support in component.rankers

* fix:
1. `hugging_face_api.HuggingFaceAPIRanker` rename to `hugging_face_tei.HuggingFaceAPIRanker`
2. HuggingFaceAPIRanker: use our Secret API for token
3. add the missing modules for `docs/pydoc/config/rankers_api.yml`
4. added function `async_request_with_retry` for `haystack/utils/requests_utils.py` and added unittest on `test/utils/test_requests_utils.py`
4. HuggingFaceAPIRanker: refactor the retry function to support configuration based on attempts and status code.
5. HuggingFaceAPIRanker: refactor the test into unit tests using mocks

* fix(HuggingFaceTEIRanker): change the token check logic to use the resolve_value method.

* fix(format): run `hatch run format`

* fix:
- Force keyword-only arguments in __init__ method by adding *,
- Clarify token docstring that it's not always required
- Copy documents to avoid modifying original objects
- Remove test file from slow workflow
- Add monkeypatch eånvironment variable cleanup in tests
- Fix missing module in rankers_api.yml and sort modules alphabetically
- Remove unnecessary test info from release notes

* fix HuggingFaceTEIRanker:
- "None" of "Optional[Secret]" has no attribute "resolve_value"
- run/run_async: too many parameters

* fix(HuggingFaceTEIRanker) :Revise the docstring of the HuggingFaceTEIRanker, improve the parameter descriptions, ensure consistency and clarity. Add error handling information to enhance the readability of the API response.

* fix:unit test for HuggingFaceTEIRanker raise message

* fix fmt

* minor refinements

* refine release note

---------

Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
2025-05-27 12:44:54 +02:00
Sebastian Husch Lee
db3d95b12a
refactor: Refactor openai generator (#9445)
* Refactor openai generator and chat generator to reusue same util methods

* Start fixing tests

* More fixes

* Fix mypy

* Fix
2025-05-27 12:44:17 +02:00
Amna Mubashar
64def6d41b
feat: add component name and type to StreamingChunk (#9426)
* Stream component name in openai

* Fix type

* PR comments

* Update huggingface gen

* Typing fix

* Update huggingfacelocal gen

* Fix errors

* Remove model changes

* Fix minor errors

* Update releasenotes/notes/add-component-info-dataclass-be115dee2fa50abd.yaml

Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>

* PR comments

* update annotation

* Update hf files

* Fix linting

* Add a from_component method

* use add_component

---------

Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
2025-05-27 12:23:40 +02:00
Stefano Fiorucci
085c3add41
ci: prevent DocumentWriter tests from blocking CI (#9448) 2025-05-27 12:10:21 +02:00