3874 Commits

Author SHA1 Message Date
Ulises M
bfdad40a80
feat: Add ONNX & OpenVINO backend support, and torch dtype kwargs in Sentence Transformers Components (#8813)
* initial rough draft

* expose backend instead of extracting from model_kwargs

* explictly set backend model path

* add reno

* expose backend for ST diversity backend

* add dtype tests and expose kwargs to ST ranker for backend parameters

* skip dtype tests as torch isnt compiled with cuda

* add new openvino dependency release, unskip tests

* resolve suggestion

* mock calls, turn integrations into unit tests

* remove unnecessary test dependencies
2025-02-13 12:04:14 +01:00
Sebastian Husch Lee
71416c81bc
feat: Add store_full_path to converter (#8849)
* Add missing store_full_path to converter

* Add release note

* Fix pylint
2025-02-12 17:11:59 +01:00
Bilge Yücel
043b88f181
Remove run_inner code snippet from sync run (#8850) 2025-02-12 15:06:52 +01:00
Stefano Fiorucci
7b5b84d377
stop collecting transformers and torch in telemetry (#8847) 2025-02-12 11:36:29 +00:00
mathislucka
88ed301712
fix: AsyncPipeline logging name can't be overwritten (#8845)
* fix: component_name not name in logging statement

* fix: component_name not name in logging statement

* fix: don't use fstrings in logging

---------

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
2025-02-12 11:21:07 +00:00
David S. Batista
cee52435bf
adding to pydocs (#8846) 2025-02-12 12:04:50 +01:00
Amna Mubashar
dcefb48fba
fix: Improve docstrings of AsyncPipeline (#8843)
* Update the docstrings

* Add an example for run_async_generator in docstrings

* fixing docstring code

* fixing docstring code

---------

Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-02-12 10:23:00 +00:00
Haystack Bot
0d4720ef79
Update unstable version to 2.11.0-rc0 (#8842)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-02-11 13:54:52 +01:00
Vladimir Blagojevic
a7c1661f13
fix: Look through all streaming chunks for tools calls (#8829)
* Look through all streaming chunks for tools calls

* Add reno note

* mypy fixes

* Improve robustness

* Don't concatenate, use the last value

* typing

* Update releasenotes/notes/improve-tool-call-chunk-search-986474e814af17a7.yaml

Co-authored-by: David S. Batista <dsbatista@gmail.com>

* Small refactoring

* isort

---------

Co-authored-by: David S. Batista <dsbatista@gmail.com>
v2.11.0-rc0
2025-02-11 13:25:39 +01:00
Stefano Fiorucci
b5d2854b93
pin posthog<3.12.0 (#8841) 2025-02-11 10:30:57 +00:00
Sebastian Husch Lee
2c0a72844f
Fix splitter when table is only one row wide (#8839) 2025-02-11 09:55:35 +00:00
David S. Batista
f189a1c349
fix: LLMMetadataExtractor removing from_dict/to_dict AWS tests (#8840)
* removint from_dict/to_dict AWS tests

* removing boto3 import from tests
2025-02-11 09:40:58 +00:00
TripleChecker
ad90e106ac
chore: Fix typos in proposals (#8837) 2025-02-11 09:33:22 +01:00
Sebastian Husch Lee
f9e6e481a1
feat: Add new component CSVDocumentSplitter to recursively split CSV documents (#8815)
* CSV Document Splitter

* Add license header

* Add newline

* Add to docs

* Add lineterminator

* Updated csv splitter to allow user to specify to split by row, column or both

* Adding more tests

* Column tests

* Some refactoring to remove incorrect dropna call

* Fix

* More complicated test

* Adding more relevant metadata to match whats provided in our other splitters

* value error tests

* Fix mypy

* Docstring updates

* Add skip_blank_lines=False

* Add to dict test

* More from and to dict tests

* Fixes

* Move dict creation outside of for loop
2025-02-10 18:10:18 +01:00
David S. Batista
f798a9e935
feat: adding LLMMetadataExtractor (#8833)
* fixing linting

* adding release notes

* updating tests

* adding to pydocs

* fixing typing due to Optional

* fixing docstring
2025-02-10 16:54:25 +00:00
Vladimir Blagojevic
b6ebd3cd77
fix: Update OpenAPIServiceConnector to new ChatMessage (#8817)
* Update OpenAPIServiceConnector to new ChatMessage, bypass model response validation

* Add reno

* Lint fixes

* Add serde pipeline test

* Update haystack/components/connectors/openapi_service.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/connectors/openapi_service.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/connectors/openapi_service.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/connectors/openapi_service.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/connectors/openapi_service.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Add edge case unit test

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2025-02-10 17:36:53 +01:00
Vladimir Blagojevic
fd5040108a
feat: Add OpenAPIConnector component, improve OpenAPI integration (#8808)
* Initial OpenAPIConnector

* Add reno note

* Format

* Add headers

* Add test dep

* Use haystack logger

* Fix test

* Minor fix, spin CI

* Update reno release note format

* Add to docs, pydocs improvements
2025-02-10 10:34:37 +01:00
Vladimir Blagojevic
73bfc08b71
feat: HuggingFaceLocalChatGenerator unified support for tools (#8827)
* Add tools to HuggingFaceLocalChatGenerator

* Add reno

* Fix types

* Small post merge fix

* Add unit tests

* Add tools serde and tests

* PR feedback

* PR feedback
2025-02-10 09:44:51 +01:00
mathislucka
e5b9bdeb66
feat: AsyncPipeline that can schedule components to run concurrently (#8812)
* add component checks

* pipeline should run deterministically

* add FIFOQueue

* add agent tests

* add order dependent tests

* run new tests

* remove code that is not needed

* test: intermediate from cycle outputs are available outside cycle

* add tests for component checks (Claude)

* adapt tests for component checks (o1 review)

* chore: format

* remove tests that aren't needed anymore

* add _calculate_priority tests

* revert accidental change in pyproject.toml

* test format conversion

* adapt to naming convention

* chore: proper docstrings and type hints for PQ

* format

* add more unit tests

* rm unneeded comments

* test input consumption

* lint

* fix: docstrings

* lint

* format

* format

* fix license header

* fix license header

* add component run tests

* fix: pass correct input format to tracing

* fix types

* format

* format

* types

* add defaults from Socket instead of signature

- otherwise components with dynamic inputs would fail

* fix test names

* still wait for optional inputs on greedy variadic sockets

- mirrors previous behavior

* fix format

* wip: warn for ambiguous running order

* wip: alternative warning

* fix license header

* make code more readable

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>

* Introduce content tracing to a behavioral test

* Fixing linting

* Remove debug print statements

* Fix tracer tests

* remove print

* test: test for component inputs

* test: remove testing for run order

* chore: update component checks from experimental

* chore: update pipeline and base from experimental

* refactor: remove unused method

* refactor: remove unused method

* refactor: outdated comment

* refactor: inputs state is updated as side effect

- to prepare for AsyncPipeline implementation

* format

* test: add file conversion test

* format

* fix: original implementation deepcopies outputs

* lint

* fix: from_dict was updated

* fix: format

* fix: test

* test: add test for thread safety

* remove unused imports

* format

* test: FIFOPriorityQueue

* chore: add release note

* feat: add AsyncPipeline

* chore: Add release notes

* fix: format

* debug: switch run order to debug ubuntu and windows tests

* fix: consider priorities of other components while waiting for DEFER

* refactor: simplify code

* fix: resolve merge conflict with mermaid changes

* fix: format

* fix: remove unused import

* refactor: rename to avoid accidental conflicts

* fix: track pipeline type

* fix: and extend test

* fix: format

* style: sort alphabetically

* Update test/core/pipeline/features/conftest.py

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>

* Update test/core/pipeline/features/conftest.py

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>

* Update releasenotes/notes/feat-async-pipeline-338856a142e1318c.yaml

* fix: indentation, do not close loop

* fix: use asyncio.run

* fix: format

---------

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-02-07 16:37:29 +01:00
Sebastian Husch Lee
35788a2d06
feat: Update csv cleaner (#8828)
* More refactoring

* Add more new options and more tests

* Improve docstrings

* Add release notes

* Fix pylint
2025-02-07 14:29:53 +01:00
Sebastian Husch Lee
1785ea622e
feat: Add component CSVDocumentCleaner for removing empty rows and columns (#8816)
* Initial commit for csv cleaner

* Add release notes

* Update lineterminator

* Update releasenotes/notes/csv-document-cleaner-8eca67e884684c56.yaml

Co-authored-by: David S. Batista <dsbatista@gmail.com>

* alphabetize

* Use lazy import

* Some refactoring

* Some refactoring

---------

Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-02-06 17:56:38 +01:00
Stefano Fiorucci
1f257944a6
chore: fix Hugging Face components for mypy 1.15.0 (#8822)
* chore: fix Hugging Face components for mypy 1.15.0

* small fixes

* fix test

* rm print

* use cast and be more permissive
2025-02-06 16:25:59 +00:00
David S. Batista
e7c6d14431
docs: removing undefined param from docstring (#8826) 2025-02-06 16:04:57 +01:00
mathislucka
eec91824bc
fix: pipeline run bugs in cyclic and acyclic pipelines (#8707)
* add component checks

* pipeline should run deterministically

* add FIFOQueue

* add agent tests

* add order dependent tests

* run new tests

* remove code that is not needed

* test: intermediate from cycle outputs are available outside cycle

* add tests for component checks (Claude)

* adapt tests for component checks (o1 review)

* chore: format

* remove tests that aren't needed anymore

* add _calculate_priority tests

* revert accidental change in pyproject.toml

* test format conversion

* adapt to naming convention

* chore: proper docstrings and type hints for PQ

* format

* add more unit tests

* rm unneeded comments

* test input consumption

* lint

* fix: docstrings

* lint

* format

* format

* fix license header

* fix license header

* add component run tests

* fix: pass correct input format to tracing

* fix types

* format

* format

* types

* add defaults from Socket instead of signature

- otherwise components with dynamic inputs would fail

* fix test names

* still wait for optional inputs on greedy variadic sockets

- mirrors previous behavior

* fix format

* wip: warn for ambiguous running order

* wip: alternative warning

* fix license header

* make code more readable

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>

* Introduce content tracing to a behavioral test

* Fixing linting

* Remove debug print statements

* Fix tracer tests

* remove print

* test: test for component inputs

* test: remove testing for run order

* chore: update component checks from experimental

* chore: update pipeline and base from experimental

* refactor: remove unused method

* refactor: remove unused method

* refactor: outdated comment

* refactor: inputs state is updated as side effect

- to prepare for AsyncPipeline implementation

* format

* test: add file conversion test

* format

* fix: original implementation deepcopies outputs

* lint

* fix: from_dict was updated

* fix: format

* fix: test

* test: add test for thread safety

* remove unused imports

* format

* test: FIFOPriorityQueue

* chore: add release note

* fix: resolve merge conflict with mermaid changes

* fix: format

* fix: remove unused import

* refactor: rename to avoid accidental conflicts

* chore: remove unused inputs, add missing license header

* chore: extend release notes

* Update releasenotes/notes/fix-pipeline-run-2fefeafc705a6d91.yaml

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>

* fix: format

* fix: format

* Update release note

---------

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-02-06 14:19:47 +00:00
Stefano Fiorucci
05300490a6
docs: add ListJoiner to pydoc configuration (#8821)
* docs: add ListJoiner to pydoc configuration

* Update docs/pydoc/config/joiners_api.yml

Co-authored-by: David S. Batista <dsbatista@gmail.com>

---------

Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-02-06 08:52:24 +00:00
Amna Mubashar
b0809b75f5
feat: Add a ListJoiner component (#8810)
* Add a ListJoiner

* Add tests and release notes
2025-02-05 23:19:14 +01:00
György Orosz
d2348ad462
feat: SentenceTransformersDocumentEmbedder and SentenceTransformersTextEmbedder can accept and pass any arguments to SentenceTransformer.encode (#8806)
* feat: SentenceTransformersDocumentEmbedder and SentenceTransformersTextEmbedder can accept and pass any arguments to SentenceTransformer.encode

* refactor: encode_kwargs parameter of SentenceTransformersDocumentEmbedder and SentenceTransformersTextEmbedder mae to be the last positional parameter for backward compatibility reasons

* docs: added explanation for encode_kwargs in SentenceTransformersTextEmbedder and SentenceTransformersDocumentEmbedder

* test: added tests for encode_kwargs in SentenceTransformersTextEmbedder and SentenceTransformersDocumentEmbedder

* doc: removed empty lines from docstrings of SentenceTransformersTextEmbedder and SentenceTransformersDocumentEmbedder

* refactor: encode_kwargs parameter of SentenceTransformersDocumentEmbedder and SentenceTransformersTextEmbedder mae to be the last positional parameter for backward compatibility (part II.)
2025-02-05 16:09:35 +00:00
Stefano Fiorucci
2828d9e4ae
refactor!: DOCXToDocument converter - store DOCX metadata as a dict (#8804)
* DOCXToDocument - store DOCX metadata as a dict

* do not export DOCXMetadata to converters package
2025-02-05 14:43:19 +01:00
Stefano Fiorucci
5ae94886b2
fix: fix test failures with Transformers models in PRs from forks (#8809)
* trigger

* try pinning sentence transformers

* make integr tests run right away

* pin transformers instead

* older transformers version

* rm transformers pin

* try ignoring cache

* change ubuntu version

* try removing token

* try again

* more HF_API_TOKEN local deletions

* restore test priority

* rm leftover

* more deletions

* moreee

* more

* deletions

* restore jobs order
2025-02-04 19:08:37 +01:00
dependabot[bot]
f1679f1dca
build(deps): bump fossas/fossa-action from 1.4.0 to 1.5.0 (#8771)
Bumps [fossas/fossa-action](https://github.com/fossas/fossa-action) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/fossas/fossa-action/releases)
- [Commits](https://github.com/fossas/fossa-action/compare/v1.4.0...v1.5.0)

---
updated-dependencies:
- dependency-name: fossas/fossa-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-04 15:01:54 +01:00
Sebastian Husch Lee
1ee86b5041
fix: Fix filters to handle date times with timezones (loading and comparison) (#8800)
* Fix on date time parsing with timezones. And comparing naive and aware date times.

* Add release note

* Add more filter tests
2025-02-04 14:51:06 +01:00
Stefano Fiorucci
ad5d29d92f
chore: ToolInvoker - remove warning (#8803) 2025-02-04 09:39:17 +00:00
Stefano Fiorucci
877f826da0
refactor: HF API Embedders - use InferenceClient.feature_extraction instead of InferenceClient.post (#8794)
* HF API Embedders: refactoring

* rename variables

* rm leftovers

* rm pin

* rm unused import

* relnote

* warning with truncate/normalize and serverless inference API

* test that warnings are raised
2025-02-03 15:11:16 +00:00
David S. Batista
f1652121ac
feat: Add support for custom (or offline) Mermaid.ink server and support all parameters (#8799)
* compress graph data to support pako endpoint

* support mermaid.ink parameters and custom servers

* dont try to resolve conflicts with the github web ui...

* avoid double graph copy

* fixing typing, improving docstrings and release notes

* reverting type

* nit - force type checker no cache

* nit - force type checker no cache

---------

Co-authored-by: Ulises M <ulises@lbux.org>
Co-authored-by: Ulises M <30765968+lbux@users.noreply.github.com>
2025-02-03 15:55:29 +01:00
David S. Batista
503d275ade
chore: remove DocumentSplitter warning related to split_by='sentence' 2025-02-03 12:47:14 +01:00
mathislucka
1a91365cc8
fix: callables can be deserialized from fully qualified import path (#8788)
* fix: callables can be deserialized from fully qualified import path

* fix: license header

* fix: format

* fix: types

* fix? types

* test: extend test case

* format

* add release notes
2025-02-03 12:35:37 +01:00
Amna Mubashar
379711f63e
fix: Pin nltk version for sentence tokenizer (#8786)
* Pin nltk version for sentence tokenizer

* Update pyproject.toml

* Update haystack/components/preprocessors/sentence_tokenizer.py

---------

Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-01-31 17:01:00 +01:00
Stefano Fiorucci
80575a7e9c
deprecate dataframe and ExtractedTableAnswer (#8789) 2025-01-31 15:03:15 +01:00
Stefano Fiorucci
3ef609a3e8
temporarily pin huggingface_hub<0.28.0 (#8790) 2025-01-31 10:35:15 +01:00
Ulises M
d939321505
fix: compress pipeline graphs before sending to mermaid (#8767)
* compress graph data to support pako endpoint

* Update haystack/core/pipeline/draw.py

Co-authored-by: David S. Batista <dsbatista@gmail.com>

* Update haystack/core/pipeline/draw.py

Co-authored-by: David S. Batista <dsbatista@gmail.com>

---------

Co-authored-by: David S. Batista <dsbatista@gmail.com>
2025-01-28 12:18:54 +01:00
Sebastian Husch Lee
bba84e5517
fix: Fix JSONConverter to properly skip files that are not utf-8 encoded (#8775)
* Small fix

* Add reno

* Trying out license header fix here
2025-01-28 10:29:55 +01:00
Sebastian Husch Lee
e3dc164625
Update license-header.txt with breaking changes from hawkeye (#8778) 2025-01-28 10:03:23 +01:00
Per Lunnemann Hansen
0e6d2a4c39
fix: update component registration to use new class reference (#8715)
The pyright language server is now able to resolve the import and provide completions for the component.

Co-authored-by: Michele Pangrazzi <xmikex83@gmail.com>
2025-01-27 14:52:24 +01:00
Stefano Fiorucci
0ac47b0064
pin numba>=0.54.0 (#8773) 2025-01-27 11:55:18 +01:00
Night-Quiet
c989d9c483
fix: skip comment blocks in DOCXToDocument (#8764)
* fix bug #8759

* Apply suggestions from code review

* release note

---------

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
2025-01-24 11:06:09 +00:00
Stefano Fiorucci
223373eced
fix: Document Classifiers - fix error messages (#8765)
* fix: Document Classifiers - fix docstrings + error messages

* grammar

* fix
2025-01-24 11:17:47 +01:00
tstadel
3119ae1ec9
refactor: raise PipelineError when Pipeline.from_dict receives an invalid type (#8711)
* fix: error on invalid type

* add reno

* Update releasenotes/notes/fix-invalid-component-type-error-83ee00d820b63cc5.yaml

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>

* Update test/core/pipeline/test_pipeline.py

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>

* fix reno

* fix reno

* last reno fix

---------

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
2025-01-23 11:40:19 +00:00
tstadel
bf79f04932
feat: support streaming_callback as run param for HF Chat generators (#8763)
* feat: support streaming_callback as run param for HF Chat generators

* add tests
2025-01-23 12:14:32 +01:00
Stefano Fiorucci
c3d0643511
feat: AzureOpenAIChatGenerator - support for tools (#8757)
* feat: AzureOpenAIChatGenerator - support for tools

* release note

* feedback
2025-01-23 09:24:04 +00:00
Stefano Fiorucci
f96839e139
chore: update transformers test dependency (#8752)
* update transformers test dependency

* add pad_token_id to the mock tokenizer

* fix HFLocal test + new test
2025-01-21 14:43:27 +01:00