* add component checks
* pipeline should run deterministically
* add FIFOQueue
* add agent tests
* add order dependent tests
* run new tests
* remove code that is not needed
* test: intermediate from cycle outputs are available outside cycle
* add tests for component checks (Claude)
* adapt tests for component checks (o1 review)
* chore: format
* remove tests that aren't needed anymore
* add _calculate_priority tests
* revert accidental change in pyproject.toml
* test format conversion
* adapt to naming convention
* chore: proper docstrings and type hints for PQ
* format
* add more unit tests
* rm unneeded comments
* test input consumption
* lint
* fix: docstrings
* lint
* format
* format
* fix license header
* fix license header
* add component run tests
* fix: pass correct input format to tracing
* fix types
* format
* format
* types
* add defaults from Socket instead of signature
- otherwise components with dynamic inputs would fail
* fix test names
* still wait for optional inputs on greedy variadic sockets
- mirrors previous behavior
* fix format
* wip: warn for ambiguous running order
* wip: alternative warning
* fix license header
* make code more readable
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
* Introduce content tracing to a behavioral test
* Fixing linting
* Remove debug print statements
* Fix tracer tests
* remove print
* test: test for component inputs
* test: remove testing for run order
* chore: update component checks from experimental
* chore: update pipeline and base from experimental
* refactor: remove unused method
* refactor: remove unused method
* refactor: outdated comment
* refactor: inputs state is updated as side effect
- to prepare for AsyncPipeline implementation
* format
* test: add file conversion test
* format
* fix: original implementation deepcopies outputs
* lint
* fix: from_dict was updated
* fix: format
* fix: test
* test: add test for thread safety
* remove unused imports
* format
* test: FIFOPriorityQueue
* chore: add release note
* fix: resolve merge conflict with mermaid changes
* fix: format
* fix: remove unused import
* refactor: rename to avoid accidental conflicts
* chore: remove unused inputs, add missing license header
* chore: extend release notes
* Update releasenotes/notes/fix-pipeline-run-2fefeafc705a6d91.yaml
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
* fix: format
* fix: format
* Update release note
---------
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* draft
* Enhance
* Almost works
* Simplify some parts and handle intermediate outputs
* Handle connections with default
* Handle cycles with multiple connections from two components
* Update distributed outputs at the correct time
* Remove Component inputs after it runs
* Add agent pipeline test case
* Fix infite loop test
* Handle some corner cases with loops checking and inputs deletion
* Fix tests
* Add new behavioral test
* Remove unused code in behavioural test
* Fix behavioural test
* Fix max run check
* Simplify outputs distribution
* Simplify subgraph run check
* Remove unused _init_run_queue function
* Remove commented code
* Add some missing type hints
* Simplify cycles breaking
* Fix _distribute_output test
* Fix _find_components_that_will_receive_no_input test
* Fix validation test
* Fix tracer losing Component inputs
* Fix some linting issues
* Remove ignore pylint rule
* Rename method that break cycles and make it raise
* Add docstring to _run_subgraph
* Update Pipeline.run() docstring
* Update comment to clarify cycles execution
* Remove SelfLoop sample Component
* Add behavioural test for unsupported cycles
* Rename behavioural test to be more specific
* Add new behavioural test
* Add release notes
* Remove commented out code and random pass
* Use more efficient function to find cycles
* Simplify _break_supported_cycles_in_graph by using defaultdict
* Stop breaking edges as soon as we make the graph acyclic
* Fix docstring and add some more comments
* Fix _distribute_output docstring
* Fix _find_receivers_from docstring
* More detailed release notes
* Minimize calls to networkx.is_directed_acyclic_graph
* Add some more info on edges keys
* Adjust components_in_cycles comment
* Add new Pipeline behavioural test
* Enhance _find_components_that_will_receive_no_input to cover more cases
* Explain why run_queue is reset after running a subgraph cycle
* Rename _init_inputs_state to _normalize_input_data
* Better explain the subgraph output distribution
* Remove for else
* Fix some comments and docstrings
* Fix linting
* Add missing return type
* Fix typo
* Rename _normalize_input_data to _normalize_varidiac_input_data and add more documentation
* Remove unused import
---------
Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
* Added equality check for sender and receiver in connection function of pipeline
* Update base.py
irrelevant changes reverted
* added release note
* altered a walk with cycle test
* added a test to verify that pipeline raises PipelineConnectError when adding a component to itself
* Update release notes
* Remove self connection feature tests
* Tidy up connect unit test
---------
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
* Deprecate max_loops_allowed in favour of new argument max_runs_per_component
* Add missing test file
* Some enhancements
* Add version that will remove deprecate stuff
* Fix bug in Pipeline.run() executing Components in a wrong and unexpected order
* Update haystack/core/pipeline/base.py
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
---------
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* Move utility functions from _enqueue_next_runnable_component (#7895)
* Isolate logic to check if we're stuck in a loop
* Simplify for else
* Add missing return in docstring
* Emit warning when stuck in a loop
* Fix docstring
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* Add utility function to move Components in queues
* Add function to find next Component to run
* Comment update
* Add missing break in loop
* Make _add_missing_input_defaults less error prone and add tests
* Fix tests
* Update docstring
* Simplify enqueue logic
* Remove unused _enqueue_next_runnable_component function
* Add method to find Component with lazy variadic input or all inputs with defaults
* Simplify _find_next_runnable_lazy_variadic_or_default_component
* Remove unnecessary type ignore
* Split _dequeue_components_that_received_no_input into separate functions
* Fix linting
* Simplify variadic check when running Component
* Simplify code
* Reorganize functions used by Pipeline.run
* Rename variables used in Pipeline.run() for clarity
* Add comment clarifying last_waiting_queue and before_last_waiting_queue
* Add functions to easily update waiting_queue
---------
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* ruff settings
enable ruff format and re-format outdated files
feat: `EvaluationRunResult` add parameter to specify columns to keep in the comparative `Dataframe` (#7879)
* adding param to explictily state which cols to keep
* adding param to explictily state which cols to keep
* adding param to explictily state which cols to keep
* updating tests
* adding release notes
* Update haystack/evaluation/eval_run_result.py
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* Update releasenotes/notes/add-keep-columns-to-EvalRunResult-comparative-be3e15ce45de3e0b.yaml
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* updating docstring
---------
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
add format-check
fail on format and linting failures
fix string formatting
reformat long lines
fix tests
fix typing
linter
pull from main
* reformat
* lint -> check
* lint -> check
* Fix formatting
* Isolate logic that finds next runnable component waiting for input
* Explain more lazy variadics
* Enhance logic following review suggestions
* Simplify code to use a single for
* Fix test
* Isolate logic that distributes Component outputs
* Handle variadic reset in correct place
* Move methods to PipelineBase
* Enhance variables and method names
* Add missing return type
* Update comment with correct variable name
* Add comment explaining conditional outputs
* Add variadic list assertion and enhance comment explaining the need of a list
* Rename to_remove_from_res to to_remove_from_component_result and enhance comment
* Split elif
* Enhance code to enqueue greedy variadic components
* Revert "Enhance code to enqueue greedy variadic components"
This reverts commit 052ceb889ec8ea100be6eab810cb06d5febea6fe.
* Enhance variadic greedy enqueue comment
* add remove_component method plus unit tests
* add docstrings
* add reno
* add type annotation to remove_component method
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* solve bug not allowing a component to be reatached to a pipeline after being removed
* Properly remove Component from Pipeline
* Ignore mypy
---------
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
* Fix running Pipeline with conditional branch and Component with default inputs
* Add release notes
* Change arg name of _init_to_run so it's clearer
* Enhance release note
* move graph initialization to the base class
* simplify data normalization
* deepcopy data in base class
* initialize inputs state
* move to_run preparation to the base class
* Test Pipeline._init_to_run()
* Test Pipeline._init_inputs_state()
* Test Pipeline._prepare_component_input_data()
---------
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
* Fix corner case when running Pipeline that causes it to get stuck in a loop
* Update haystack/core/pipeline/pipeline.py
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* move templating code under the core package
* make from_predefined part of the Pipeline API
* add tests
* amend release notes
* import under haystack package
* Apply suggestions from code review
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* from_predefined -> from_template
* remove template inheritance for more readability
---------
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* Enhance Pipeline.draw() to show image directly in Jupyter notebook
* Add util method to check if we're in a Jupyter notebook
* Split Pipeline.draw() in two methods
* Update tests
* Update releasenotes
* Enhance Pipeline.__repr__
* Simplify Pipeline.__repr__
* Update release notes
* Enhance Pipeline.draw() to show image directly in Jupyter notebook
* Add util method to check if we're in a Jupyter notebook
* Split Pipeline.draw() in two methods
* Update tests
* Update releasenotes
* First rough implementation of refactored run
* Further improve run logic
* Properly handle variadic input in run
* Further work
* Enhance names and add more documentation
* Fix issue with output distribution
* This works
* Enhance run comments
* Mark Multiplexer as greedy
* Remove MergeLoop in favour of Multiplexer in tests
* Remove FirstIntSelector in favour of Multiplexer
* Handle corner when waiting for input is stuck
* Remove unused import
* Handle mutable input data in run and misbehaving components
* Handle run input validation
* Test validation
* Fix pylint
* Fix mypy
* Call warm_up in run to fix tests
* track default value in sockets
* remove dead code
* include default value in socket description
* add unit test
* add relnote
* unused import
* clarify
