60 Commits

Author SHA1 Message Date
Madeesh Kannan
d1f8c0dcd6
fix: Prevent component pre-init hook from being called recursively (#7894) 2024-06-21 10:29:37 +02:00
Massimiliano Pippi
3a03fce71c
ci: Add code formatting checks (#7882)
* ruff settings

enable ruff format and re-format outdated files

feat: `EvaluationRunResult` add parameter to specify columns to keep in the comparative `Dataframe`  (#7879)

* adding param to explictily state which cols to keep

* adding param to explictily state which cols to keep

* adding param to explictily state which cols to keep

* updating tests

* adding release notes

* Update haystack/evaluation/eval_run_result.py

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

* Update releasenotes/notes/add-keep-columns-to-EvalRunResult-comparative-be3e15ce45de3e0b.yaml

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

* updating docstring

---------

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

add format-check

fail on format and linting failures

fix string formatting

reformat long lines

fix tests

fix typing

linter

pull from main

* reformat

* lint -> check

* lint -> check
2024-06-18 15:52:46 +00:00
Silvano Cerza
15ee622b3c
refactor: Isolate logic that finds next runnable component waiting for input (#7880)
* Fix formatting

* Isolate logic that finds next runnable component waiting for input

* Explain more lazy variadics

* Enhance logic following review suggestions

* Simplify code to use a single for

* Fix test
2024-06-18 16:43:19 +02:00
Silvano Cerza
1b4bd173b8
refactor: Isolate logic that distributes Components output after run (#7845)
* Isolate logic that distributes Component outputs

* Handle variadic reset in correct place

* Move methods to PipelineBase

* Enhance variables and method names

* Add missing return type

* Update comment with correct variable name

* Add comment explaining conditional outputs

* Add variadic list assertion and enhance comment explaining the need of a list

* Rename to_remove_from_res to to_remove_from_component_result and enhance comment

* Split elif

* Enhance code to enqueue greedy variadic components

* Revert "Enhance code to enqueue greedy variadic components"

This reverts commit 052ceb889ec8ea100be6eab810cb06d5febea6fe.

* Enhance variadic greedy enqueue comment
2024-06-14 15:53:28 +02:00
Silvano Cerza
14c7b02a4c
refactor: Isolate logic to check if a Component can run (#7840)
* Isolate run check

* Update docstrings and remove unnecessary set

* Rename argument
2024-06-11 16:14:04 +02:00
Silvano Cerza
58dd972d1a
refactor: Isolate code that runs single Pipeline Component (#7837)
* Isolate code that runs single Pipeline Component

* Fix mypy
2024-06-10 16:03:14 +00:00
Carlos Fernández
7fe0244258
feat: add methods to remove and replace components in a pipeline (#7820)
* add remove_component method plus unit tests

* add docstrings

* add reno

* add type annotation to remove_component method

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

* solve bug not allowing a component to be reatached to a pipeline after being removed

* Properly remove Component from Pipeline

* Ignore mypy

---------

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
2024-06-10 14:54:07 +02:00
Silvano Cerza
3c8569e12c
fix: Fix running Pipeline with conditional branch and Component with default inputs (#7799)
* Fix running Pipeline with conditional branch and Component with default inputs

* Add release notes

* Change arg name of _init_to_run so it's clearer

* Enhance release note
2024-06-06 13:19:07 +00:00
Silvano Cerza
fd838fc573
Update indexing and rag default templates to use InMemoryDocumentStore (#7782) 2024-06-04 12:57:33 +02:00
Silvano Cerza
74df8ed937
test: Rework Pipeline.run() tests to ease declaration with dataclasses (#7790)
* Rework boilerplate function that run Pipeline in scenarios testing

* Update tests to use new dataclasses

* Update README.md to reflect dataclass changes

* Use absolute import from conftest
2024-06-03 15:59:42 +02:00
Silvano Cerza
07ae45e0c2
test: Migrate Pipeline.run() tests with run arguments (#7777)
* Support Pipeline.run() arguments in tests

* Move intermediate outputs
2024-06-03 12:36:04 +02:00
Silvano Cerza
d81af81fbb
test: Migrate pipeline run tests (#7775)
* Move complex pipeline

* Move pipeline with default

* Move pipeline with distinct loops

* Move pipeline with double loop

* Move pipeline with dynamic inputs

* Move fixed decision pipeline

* Move fixed merging pipeline

* Move fixed decision and merge pipeline

* Remove test_joiners.py

* Move looping and merge pipeline

* Remove test_looping.py

* Move mutable input pipeline

* Move parallel branches pipeline

* Move same input different components pipeline

* Move test_run_with_greedy_variadic_after_component_with_default_input_simple

* Remove test_run_raises_if_max_visits_reached

* Move test_run_with_component_that_does_not_return_dict

* Move test_correct_execution_order_of_components_with_only_defaults

* Move test_pipeline_is_not_stuck_with_components_with_only_defaults

* Move test_pipeline_is_not_stuck_with_components_with_only_defaults_as_first_components

* Move self loop pipeline

* Move variable decision and merge pipeline

* Remove test_variable_decision_pipeline

* Move variable merging pipeline

* Add FakeComponent removed by mistake
2024-05-31 13:00:29 +02:00
Silvano Cerza
a9f989d756
test: Support multiple runs for Pipeline run tests (#7762)
* Support multiple runs for Pipeline run tests

* Apply suggestions from code review

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2024-05-31 11:58:49 +02:00
Silvano Cerza
5c468feecf
test: Update Pipeline.run() tests README.md (#7757)
* Update Pipeline.run() tests README.md

* Add suggestion from review

* Fix typos

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2024-05-29 14:28:42 +02:00
Silvano Cerza
3dcc21fd73
test: Pipeline run tests rework (#7748)
* Rework Pipeline.run() tests

* Remove test_linear_pipeline.py

* Add test for components execution order

* Add new pytest-bdd tests dependency

* Update README.md

* Add function to dinamically add integration marker

* Fix marking tests as integration
2024-05-28 15:42:47 +02:00
Silvano Cerza
22289f590f
Move tests from test_connect.py in test_pipeline.py and test_utils.py (#7742) 2024-05-24 16:41:38 +02:00
Silvano Cerza
da088140ab
Group up Pipeline unit tests in a single class (#7706) 2024-05-21 16:12:28 +02:00
Massimiliano Pippi
cc1d4b1c80
chore: Simplify Pipeline.run method by moving code to the base class (#7680)
* move graph initialization to the base class

* simplify data normalization

* deepcopy data in base class

* initialize inputs state

* move to_run preparation to the base class

* Test Pipeline._init_to_run()

* Test Pipeline._init_inputs_state()

* Test Pipeline._prepare_component_input_data()

---------

Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
2024-05-14 23:25:46 +02:00
Madeesh Kannan
2428bc2a92
fix: Pipeline.run correctly returns all outputs when the include_outputs_from parameter is used (#7697)
* fix: `Pipeline.run` correctly returns all outputs when the `include_outputs_from` parameter is used

* Add release note
2024-05-14 12:29:41 +02:00
Massimiliano Pippi
1d20ac3c5e
chore: extract BasePipeline (#7673)
* extract BasePipeline

* release note

* add missing headers

* move __eq__ to the base class

* proper check type equality, bless the tests
2024-05-10 11:35:15 +02:00
Massimiliano Pippi
10c675d534
chore: add license header to all modules (#7675)
* add license header to modules
* check license header at linting time
2024-05-09 13:40:36 +00:00
Vladimir Blagojevic
8cb3cecf34
feat: Trace pipeline run input/output data (#7590)
* Trace pipeline run

* Add reno note

* Update tracing tests to check input_data and output_data

* empty

---------

Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2024-04-29 17:29:27 +02:00
Madeesh Kannan
ec0e22265a
feat: Expand Pipeline.inputs and Pipeline.outputs to include connected sockets (#7586) 2024-04-24 12:27:18 +02:00
Silvano Cerza
6a8834e43e
fix: Fix corner case when running Pipeline that causes it to get stuck in a loop (#7531)
* Fix corner case when running Pipeline that causes it to get stuck in a loop

* Update haystack/core/pipeline/pipeline.py

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2024-04-11 16:39:38 +02:00
Madeesh Kannan
b1760add56
feat: Add support for pipeline deserialization callbacks (#7518)
* feat: Add support for deserialization callbacks

* Lint

* Fix type hint for older Python versions

* Apply suggestions from code review

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Lint

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2024-04-10 17:47:14 +02:00
Madeesh Kannan
fd84cd5f9a
feat: Add support for returning intermediate outputs of pipeline components (#7504)
* feat: Add support for returning intermediate outputs of pipeline components

The `pipeline.run` method has been extended to accept a set of component
names whose inputs are returned in addition to the outputs of leaf components.

* Add reno

* Lint

---------

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
2024-04-10 17:16:00 +02:00
Silvano Cerza
6e289698e9
fix: Fix Pipeline.run() getting stuck in a loop even though there are components that can run (#7434) 2024-03-28 12:31:36 +01:00
Silvano Cerza
58d91b64dc
Fix: Fix Pipeline.run() running components with only defaults in the wrong order (#7426)
* Fix Pipeline.run() running components with only defaults in the wrong order

* Add release notes
2024-03-26 16:55:31 +01:00
Stefano Fiorucci
6e69d4f188
fix: Pipeline - disable autoshow on Jupyter (#7397)
* try

* fix docstring

* simplify tests

* add release note
2024-03-21 12:55:06 +01:00
Silvano Cerza
de4fca4526
ci: Skip collection of test_json_schema.py to fix CI failures (#7353)
* Skip collection of test_json_schema.py to fix CI failures

* mock chroma instance

* revert

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2024-03-13 16:59:26 +01:00
Stefano Fiorucci
3dbde84a28
test: monkeypatch some env vars in Predefined Pipelines tests (#7321)
* ci: skip some tests if the OPENAI API key is not set

* better idea: monkeypatch the env var
2024-03-07 08:52:25 +01:00
Julian Risch
50ad1fa2c4
fix: Remove pipeline serialization from telemetry code (#7289)
* remove pipeline serialization from telemetry

* simplify getting component instance from pipeline

* reno

* add unit test with non-serializable component

* generate qualified class names

* added pipeline.walk()

* fix imports

* sort Iterator import

* remove bfs

* add test for pipeline.walk() with cycles

* Apply suggestions from code review

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

* raise TypeError if telemetry_data is no dict

* Update haystack/telemetry/_telemetry.py

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

---------

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
2024-03-05 12:45:53 +01:00
Stefano Fiorucci
721691c036
replace flaky with pytest-rerunfailures (#7298) 2024-03-04 12:26:40 +01:00
Silvano Cerza
72d776c390
fix: Fix run order of variadic greedy components in Pipeline.run() (#7258)
* Fix run order of variadic greedy components in Pipeline.run()

* Add release notes
2024-03-01 17:39:13 +01:00
Massimiliano Pippi
34dac5f86f
Update test_pipeline.py (#7284) 2024-03-01 14:20:15 +01:00
Tobias Wochinger
69d5438d6e
tests: make 3.12 compatible (#7265) 2024-02-29 16:11:14 +01:00
Massimiliano Pippi
e7809b6fea
feat: Add from_template class method to Pipeline (#7240)
* move templating code under the core package

* make from_predefined part of the Pipeline API

* add tests

* amend release notes

* import under haystack package

* Apply suggestions from code review

Co-authored-by: David S. Batista <dsbatista@gmail.com>

* from_predefined -> from_template

* remove template inheritance for more readability

---------

Co-authored-by: David S. Batista <dsbatista@gmail.com>
2024-02-29 12:23:32 +01:00
Tobias Wochinger
6fce2dbb11
feat: log trace correlation (#7206)
* feat: implement support for trace-log correlation

* docs: add release notes
2024-02-27 10:06:53 +01:00
Tobias Wochinger
41b93108c0
feat: allow content tracing (#7066)
* feat: implement content tracing capability

* feat: trace component input/output

* chore: extend type serialization to classes with `to_dict`

* docs: add option for content tracing

* style: remove unused import

* chore: add more tags

* style: ignore typing

* fix: fix tagging of types
2024-02-23 11:52:54 +01:00
Tobias Wochinger
6e580e4430
feat: implement pipeline tracing (#7046)
* feat: implement pipeline tracing

* tests: improve test setup for spying tracer

* feat: implement util for type coercion

* fix: trace a after checking pipeline output

* docs: add release notes

* docs: drop unused imports

* refactor: simplify getting raw span

* refactor: implement `ProxyTracer`
2024-02-22 12:52:04 +01:00
Silvano Cerza
5f97e08feb
feat: Reintroduce max_loops_allowed check in Pipeline.run() (#7010)
* Reintroduce max_loops_allowed check in Pipeline.run()

* Add release notes
2024-02-19 10:05:35 +01:00
Stefano Fiorucci
3f85a63468
fix: deepcopy the inputs of components (#6987)
* deepcopy inputs

* reno

* simplify test
2024-02-16 17:07:52 +01:00
Silvano Cerza
06a9349095
Change mermaid integration test to avoid comparing received image (#7002) 2024-02-15 15:01:04 +01:00
Silvano Cerza
f96eb3847f
refactor: Merge Pipelines definition in core package (#6973)
* Move marshalling functions in core Pipeline

* Move telemetry gathering in core Pipeline

* Move run logic in core Pipeline

* Update root Pipeline import

* Add release notes

* Update Pipeline docs path

* Update releasenotes/notes/merge-pipeline-definitions-1da80e9803e2a8bb.yaml

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2024-02-12 18:25:28 +01:00
Silvano Cerza
21488ac6ca
fix: Fix Pipeline.draw() failing cause of missing socket (#6965)
* Fix Pipeline.draw() issue

* Mark draw test as flaky
2024-02-09 16:10:48 +01:00
Silvano Cerza
d2d01f9fe1
feat: Enhance Pipeline.__repr__() (#6963)
* Enhance Pipeline.draw() to show image directly in Jupyter notebook

* Add util method to check if we're in a Jupyter notebook

* Split Pipeline.draw() in two methods

* Update tests

* Update releasenotes

* Enhance Pipeline.__repr__

* Simplify Pipeline.__repr__

* Update release notes
2024-02-09 14:44:34 +01:00
Silvano Cerza
a7f36fdd32
feat: Enhance Pipeline.draw() to show image directly in Jupyter notebook (#6961)
* Enhance Pipeline.draw() to show image directly in Jupyter notebook

* Add util method to check if we're in a Jupyter notebook

* Split Pipeline.draw() in two methods

* Update tests

* Update releasenotes
2024-02-09 14:44:24 +01:00
Silvano Cerza
3d5c329949
Change Pipeline.connect() to return Pipeline instance (#6922) 2024-02-06 11:03:18 +01:00
Silvano Cerza
b9d7a98359
Rework Pipeline.connect() tests and structure (#6917) 2024-02-05 18:16:02 +01:00
Silvano Cerza
0191b1e6e4
feat: Change Component's I/O dunder type (#6916)
* Add Pipeline.get_component_name() method

* Add utility class to ease discoverability of Component I/O

* Move InputOutput in component package

* Rename InputOutput to _InputOutput

* Raise if inputs or outputs field already exist

* Fix tests

* Add release notes

* Move InputSocket and OutputSocket in types package

* Move _InputOutput in socket package

* Rename _InputOutput class to Sockets

* Simplify Sockets class

* Dictch I/O dunder fields in favour of inputs and outputs fields

* Update Sockets docstrings

* Update release notes

* Fix mypy

* Remove unnecessary assignment

* Remove unused logging

* Change SocketsType to SocketsIOType to avoid confusion

* Change sockets type and name

* Change Sockets.__repr__ to return component instance

* Fix linting

* Fix sockets tests

* Revert to dunder fields for Component IO

* Use singular in IO dunder fields

* Delete release notes

* Update haystack/core/component/types.py

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2024-02-05 17:46:45 +01:00