* Add snapshot and the full file path it was saved to breakpoint exception
* Add checks to test and simplify test
* Make both tests unit tests
* remove unused import
* update reno
* Add docstrings
* Updated reno
* Simplify test
* Uupdate test
* Fix mypy
* Refactor saving pipeline snapshot to handle the try-except inside and to cover more cases (e.g. try-excepts around our serialization logic)
* Add reno
* Fix
* Adding tests
* More tests
* small change
* fix test
* update docstrings
* initial PoC idea running
* removing test code
* cleaning up
* wip
* cleaning up demos
* adding more pipelines to test persistence saving
* wip
* wip
* working example for logging components inputs in run time
* reverting to a simpler solution for intermediate results
* cleaning up
* testing that in a crash components outputs/inputs up to the crash point are returned
* adding tests for state persistance in a RAG pipeline
* updataing tests for state persistance in a RAG pipeline
* removing use cases of agent tests
* adding LICENSE header
* adding LICENSE header
* adding release notes
* updating tests for mocked components only
* updating release notes
* adapting PipelineRuntimeError
* cleaning up tests
* fixing test pipeline crash components inputs/outputs are saved
* fixing tests for state persistance
* removing code
* removing code
* removing code
* updating release notes
* validating parameters
* cleaning
* wip: debugging
* removing persistance tests
* formatting
* formatting
* cleaning up code
* updating release notes
* adding missing docstrings
* typo in release notes
* Update haystack/core/pipeline/pipeline.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* PR comments
* handling potential issues with saving the snapshot file
* updating tests
* Update haystack/core/pipeline/pipeline.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* Update haystack/core/pipeline/pipeline.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* some more improvements
* fixing exxception
* fixing exception error name conflict
---------
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* initial PoC idea running
* removing test code
* cleaning up
* wip
* cleaning up demos
* adding more pipelines to test persistence saving
* wip
* wip
* working example for logging components inputs in run time
* reverting to a simpler solution for intermediate results
* cleaning up
* testing that in a crash components outputs/inputs up to the crash point are returned
* adding tests for state persistance in a RAG pipeline
* updataing tests for state persistance in a RAG pipeline
* removing use cases of agent tests
* adding LICENSE header
* adding LICENSE header
* adding release notes
* updating tests for mocked components only
* updating release notes
* adapting PipelineRuntimeError
* cleaning up tests
* fixing test pipeline crash components inputs/outputs are saved
* fixing tests for state persistance
* isolating changes
* cleaning
* updating release notes
* addding test for regular pipeline
* small improvements and updating release notes
* cleaning imports
* removing code
* improvements/fixes based on PR comments
* raising pipeline_outputs on async version of Pipeline
* fixing async versions + updating tests
* simplifying tests
* Suggested changes pipeline crash (#9744)
* Suggested changes
* Some cleanup
* Small changes
---------
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* fix: more informative error message when two components connect
* chore: releasenote
* fix: (PipelineBase.connect) disable C901 complexity check using noqa
* fix: (PipelineBase:connect) provided two exceptions: when is not defined && is not defined
* test: added tests for the related issues
* Update haystack/core/pipeline/base.py
---------
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* chore(lint): enforce and apply PEP 585 type hinting
* Run fmt fixes
* Fix all typing imports using some regex
* Fix all typing written in string in tests
* undo changes in the e2e tests
* make e2e test use list instead of List
* type fixes
* remove type:ignore
* pylint
* Remove typing from Usage example comments
* Remove typing from most of comments
* try to fix e2e tests on comm PRs
* fix
* Add tests typing.List in to adjust test compatiplity
- test/components/agents/test_state_class.py
- test/components/converters/test_output_adapter.py
- test/components/joiners/test_list_joiner.py
* simplify pyproject
* improve relnote
---------
Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
* Add support for | operator
* Add reno
* Use type(None) instead of NoneType
* Make custom UnionType private
* Add check that test should only run if python is 3.10+
* PR comments
* Update test/utils/test_type_serialization.py
Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
* Fix formatting
---------
Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
* wip: fixing tests
* wip: fixing tests
* wip: fixing tests
* wip: fixing tests
* fixing circular imports
* decoupling resume and initial run() for agent
* adding release notes
* re-raising BreakPointException from pipeline.run()
* fixing imports
* refactor: Refactor suggestions for Pipeline breakpoints (#9614)
* Refactoring
* Start adding debug_path into Breakpoint class
* Fully move debug_path into Breakpoint dataclass
* Simplifications in pipeline run logic
* More simplification
* lint
* More simplification
* Updates
* Rename resume_state to pipeline_snapshot
* PR comments
* Missed renaming of state in a few more places
* feat: Add dataclasses to represent a `PipelineSnapshot` and refactored to use it (#9619)
* Refactor to use dataclasses for PipelineSnapshot and AgentSnapshot
* Fix integration tests
* Mypy
* Fix mypy
* Fix lint
* Refactor AgentSnapshot to only contain needed info
* Fix mypy
* More refactoring
* removing unused import
---------
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* feat: saving include_outputs_from intermediate results to `PipelineState` object (#9629)
* saving intermediate components results in include_outputs_from into the PipelineSnaptshot
* cleaning up
* fixing tests
* fixing tests
* extending tests
* Update haystack/dataclasses/breakpoints.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* Update haystack/dataclasses/breakpoints.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* linting
* moving intermediate results to pipeline state and adding pipeline outputs to state
* moving ordered_component_names and include_outputs_from to PipelineSnapshot
* moving original_input_data to PipelineSnapshot
* simplifying saving the intermediate results
* Update haystack/dataclasses/breakpoints.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* Update haystack/dataclasses/breakpoints.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* Update haystack/dataclasses/breakpoints.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* Update haystack/dataclasses/breakpoints.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
---------
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* linting
* cleaning up
* avoiding creating PipelineSnapshot for every component run
* removing unecessary code
* Update checks in Agent to not unecessarily create AgentSnapshot when not needed.
* Update haystack/components/agents/agent.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* Update haystack/components/agents/agent.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* cleaning up tests
* linting
---------
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
* First pass at alerting a user that the pipeline is blocked
* Change approach and add change to async pipeline
* Fix check in run_async
* Another fix
* Somehow also fixed the max_runs_per_component
* Align sync run and async run component
* Update output type of component outputs to Mapping to align with our protocol
* linting
* ruff
* Make it a core test
* Add reno
* Some refactoring
* Linting
* Fix mypy
* Converting to warning
* Small changes
* Update release note
* More cleanup
* PR comment
* PR comments
* Updating tests
* More tests
* More tests
* More tests
* More tests
* Get bare types to work with builtins
* Add reno
* PR comments
---------
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* Fix types in test_run.py
* Get test_run.py to pass fmt-check
* Add test_run to mypy checks
* Update test folder to pass ruff linting
* Fix merge
* Fix HF tests
* Fix hf test
* Try to fix tests
* Another attempt
* minor fix
* fix SentenceTransformersDiversityRanker
* skip integrations tests due to model unavailable on HF inference
---------
Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
* initial import
* small fixes
* adding tests
* adding tests
* refactoring merge graphs
* updating tests
* docstrings
* adding release notes
* adding SuperComponent name to extended components
* adding colours and legend to different super components
* adding missed docstring parameter
* fixing tests and type checking
* Update haystack/core/pipeline/base.py
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* forcing keyword arguments for draw() and show()
* adding wrapper function and a deprecation warning
* adding pylint disable - this will be removed soon
* wip
* adding a decorator function to test if another function is being called with positional arguments
* adding a decorator function to test if another function is being called with positional arguments
---------
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* First pass at fix for deepcopying inputs and outputs
* Add reno
* Add recursion for dict objects
* Bump recursion depth
* More tests and some improvments
* Fix unit tests
* PR comments
* Agent tracing
* Small changes
* Some changes and refactoring
* Refactoring to reuse code
* Fix
* Add reno
* Fix tests
* Fix tests
* Fix linting
* Refactor and add tracing support to run_async of Agent
* Reduce duplicate code
* Remove finalize_run
* Use break instead of copying code three times
* Adding a test
* Add tracing unit tests
* Make async tracing test actually run async
* Increase test coverage
* Unit test for traces in pipeline
* Add cleanup
* Fix proper indentation
* PR comments
* PR comments and new test
* Update warning message
* Update warning message
---------
Co-authored-by: Vladimir Blagojevic <dovlex@gmail.com>
* Start refactoring PipelineRuntimeError
* Slight change
* Fix test and remove test that had no asserts
* Change back to pipeline runtime error
* PR comments
* Add reno
* PR comments
* Update test
* Starting to refactor type util tests to be more systematic
* refactoring
* Expand tests
* Update to type utils
* Add missing subclass check
* Expand and refactor tests, introduce type_validation Literal
* More test refactoring
* Test refactoring, adding type validation variable to pipeline base
* Update relaxed version of type checking to pass all newly added tests
* trim whitespace
* Add tests
* cleanup
* Updates docstrings
* Add reno
* docs
* Fix mypy and add docstrings
* Changes based on advice from Tobi
* Remove unused imports
* Doc strings
* Add connection type validation to to_dict and from_dict
* Update tests
* Fix test
* Also save connection_type_validation at global pipeline level
* Fix tests
* Remove connection type validation from the connect level, only keep at pipeline level
* Formatting
* Fix tests
* formatting
* fix: only overwrite existing socket inputs when we provide a new value
* chore: add release notes
* Apply suggestions from code review
---------
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
* fix: use is not to compare to sentinel value
* chore: release notes
* Update releasenotes/notes/fix-component-checks-with-ambiguous-truth-values-949c447b3702e427.yaml
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* fix: another sentinel value
* test: also test base class
* add pandas as test dependency
* format
* Trigger CI
* mark test with xfail strict=False
---------
Co-authored-by: Sebastian Husch Lee <sjrl@users.noreply.github.com>
Co-authored-by: David S. Batista <dsbatista@gmail.com>
Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
* add component checks
* pipeline should run deterministically
* add FIFOQueue
* add agent tests
* add order dependent tests
* run new tests
* remove code that is not needed
* test: intermediate from cycle outputs are available outside cycle
* add tests for component checks (Claude)
* adapt tests for component checks (o1 review)
* chore: format
* remove tests that aren't needed anymore
* add _calculate_priority tests
* revert accidental change in pyproject.toml
* test format conversion
* adapt to naming convention
* chore: proper docstrings and type hints for PQ
* format
* add more unit tests
* rm unneeded comments
* test input consumption
* lint
* fix: docstrings
* lint
* format
* format
* fix license header
* fix license header
* add component run tests
* fix: pass correct input format to tracing
* fix types
* format
* format
* types
* add defaults from Socket instead of signature
- otherwise components with dynamic inputs would fail
* fix test names
* still wait for optional inputs on greedy variadic sockets
- mirrors previous behavior
* fix format
* wip: warn for ambiguous running order
* wip: alternative warning
* fix license header
* make code more readable
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
* Introduce content tracing to a behavioral test
* Fixing linting
* Remove debug print statements
* Fix tracer tests
* remove print
* test: test for component inputs
* test: remove testing for run order
* chore: update component checks from experimental
* chore: update pipeline and base from experimental
* refactor: remove unused method
* refactor: remove unused method
* refactor: outdated comment
* refactor: inputs state is updated as side effect
- to prepare for AsyncPipeline implementation
* format
* test: add file conversion test
* format
* fix: original implementation deepcopies outputs
* lint
* fix: from_dict was updated
* fix: format
* fix: test
* test: add test for thread safety
* remove unused imports
* format
* test: FIFOPriorityQueue
* chore: add release note
* feat: add AsyncPipeline
* chore: Add release notes
* fix: format
* debug: switch run order to debug ubuntu and windows tests
* fix: consider priorities of other components while waiting for DEFER
* refactor: simplify code
* fix: resolve merge conflict with mermaid changes
* fix: format
* fix: remove unused import
* refactor: rename to avoid accidental conflicts
* fix: track pipeline type
* fix: and extend test
* fix: format
* style: sort alphabetically
* Update test/core/pipeline/features/conftest.py
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
* Update test/core/pipeline/features/conftest.py
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
* Update releasenotes/notes/feat-async-pipeline-338856a142e1318c.yaml
* fix: indentation, do not close loop
* fix: use asyncio.run
* fix: format
---------
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* add component checks
* pipeline should run deterministically
* add FIFOQueue
* add agent tests
* add order dependent tests
* run new tests
* remove code that is not needed
* test: intermediate from cycle outputs are available outside cycle
* add tests for component checks (Claude)
* adapt tests for component checks (o1 review)
* chore: format
* remove tests that aren't needed anymore
* add _calculate_priority tests
* revert accidental change in pyproject.toml
* test format conversion
* adapt to naming convention
* chore: proper docstrings and type hints for PQ
* format
* add more unit tests
* rm unneeded comments
* test input consumption
* lint
* fix: docstrings
* lint
* format
* format
* fix license header
* fix license header
* add component run tests
* fix: pass correct input format to tracing
* fix types
* format
* format
* types
* add defaults from Socket instead of signature
- otherwise components with dynamic inputs would fail
* fix test names
* still wait for optional inputs on greedy variadic sockets
- mirrors previous behavior
* fix format
* wip: warn for ambiguous running order
* wip: alternative warning
* fix license header
* make code more readable
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
* Introduce content tracing to a behavioral test
* Fixing linting
* Remove debug print statements
* Fix tracer tests
* remove print
* test: test for component inputs
* test: remove testing for run order
* chore: update component checks from experimental
* chore: update pipeline and base from experimental
* refactor: remove unused method
* refactor: remove unused method
* refactor: outdated comment
* refactor: inputs state is updated as side effect
- to prepare for AsyncPipeline implementation
* format
* test: add file conversion test
* format
* fix: original implementation deepcopies outputs
* lint
* fix: from_dict was updated
* fix: format
* fix: test
* test: add test for thread safety
* remove unused imports
* format
* test: FIFOPriorityQueue
* chore: add release note
* fix: resolve merge conflict with mermaid changes
* fix: format
* fix: remove unused import
* refactor: rename to avoid accidental conflicts
* chore: remove unused inputs, add missing license header
* chore: extend release notes
* Update releasenotes/notes/fix-pipeline-run-2fefeafc705a6d91.yaml
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
* fix: format
* fix: format
* Update release note
---------
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* compress graph data to support pako endpoint
* support mermaid.ink parameters and custom servers
* dont try to resolve conflicts with the github web ui...
* avoid double graph copy
* fixing typing, improving docstrings and release notes
* reverting type
* nit - force type checker no cache
* nit - force type checker no cache
---------
Co-authored-by: Ulises M <ulises@lbux.org>
Co-authored-by: Ulises M <30765968+lbux@users.noreply.github.com>
