* ruff settings
enable ruff format and re-format outdated files
feat: `EvaluationRunResult` add parameter to specify columns to keep in the comparative `Dataframe` (#7879)
* adding param to explictily state which cols to keep
* adding param to explictily state which cols to keep
* adding param to explictily state which cols to keep
* updating tests
* adding release notes
* Update haystack/evaluation/eval_run_result.py
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* Update releasenotes/notes/add-keep-columns-to-EvalRunResult-comparative-be3e15ce45de3e0b.yaml
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
* updating docstring
---------
Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
add format-check
fail on format and linting failures
fix string formatting
reformat long lines
fix tests
fix typing
linter
pull from main
* reformat
* lint -> check
* lint -> check
* add cache step
* add test commit
* do not use shell to get deps
* force shell on all platforms
* use short-lived cache
* do not use a dedicated build step
* move license header check under format
* merge linting into test workflow
* test commit to trigger CI
* Revert "test commit to trigger CI"
This reverts commit ade5dcc6873cdd29ff5c5e8c9033b137cdef4d81.
* remove unused skipper
* clean up default env and add reno script
* update contributions guidelines
* use test script
* format
* re-add missing dep
* remove black in favour of ruff
* ci: trigger separate workflow
* ci: temporary use current branch
* ci: fix workflow name
* ci: try with same job name
* ci: try with dispatch
* Revert "ci: try with dispatch"
This reverts commit bd66e56c0697ae97fc2599eebaceff417d9be65c.
* Revert "ci: try with same job name"
This reverts commit 9e2ae5b402758c14a9f812c2e06f820bd3ece767.
* ci: try with workflow call in both cases
* ci: introduce change to trigger CI
* Revert "ci: introduce change to trigger CI"
This reverts commit e3ec07c5e26f114364babea69535183253c801b7.
* ci: add name
* Revert "Revert "ci: introduce change to trigger CI""
This reverts commit 6718585fd24069112e0f773e010056e1d96e3eee.
* ci: improve naming
* ci: further improve naming
* Unset reusable workflow version and use relative path
* Remove CI trigger
---------
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
* ci: only only skip tests if no code changes at all
* ci: introduce fake change to test flow
* ci: skip running entire workflow
* Revert "ci: introduce fake change to test flow"
This reverts commit 96eed5d65f0a96e90b5d5a390b1a04386a046dac.
* ci: add missing paths
* Speed up tests.yml by caching dependencies
* Trigger for testing
* Use restore only action to speedup restoring
* Use bash shell to get pip cache dir
* Set shell for caching step
* Cache correct path
* Remove trigger
* add boilerpy3 to the core dependencies
* remove boilerpy3 installation from test workflow
* fix pylint: import order and unused import
* fix import order
* add release note
---------
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
* Ignore some mypy errors
* Fix I/O comparator
* Avoid calling asdict multiple times when comparing dataclasses
* Enhance component tests
* Fix I/O dataclasses comparison
* Use Any instead of type when expecting I/O dataclasses
* Fix mypy
* Change InputSocket taken_by field to sender
* Remove variadics implementation
* Adapt tests
* Enhance docs and simplify run
* Remove useless check on drawing
* Add __canals_optional_inputs__ field in components
* Rework a bit Pipeline._ready_to_run()
* Simplify some logic
* Add __canals_mandatory_inputs__ field in components
* Handle pipeline loops
* Fix tests
* Document component state run logic
* Add double loop pipeline test
* Make component decorator a class
* PR feedback
* Add error logging when registering Component with identical names
* Add 'remove' action that removes current component from Pipeline run input queue
* Simplify run checks and logging
* Better logging
* Apply suggestions from code review
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* Trim whitespace
* Add support for Union in Component's I/O
* Remove dependencies section in marshaled pipelines
* Create Component Protocol
* simpler optional deps
* Simplify component init wrapping and fix issue with save_init_params
* Update canals/pipeline/save_load.py
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* Simplify functions to find I/O sockets
* Fix import
* change import
* testing ci
* testing ci
* Simplify _save_init_params
* testing ci
* testing ci
* use direct pytest call
* trying to force old version for macos
* list macos versions
* list macos versions
* disable on macos
* remove extra
* refactor imports
* re-enable some logs
* some more tests
* small correction
* Remove unused leftover methods
* docs
* update docstring
* mention optionals
* example for dataclass initialization
* missed part
* fix api docs
* improve error reporting and testing
* add tests for Any
* parametrized tests
* fix test for py<3.10
* test type printing
* remove typing. prefix from Any (compat with Py3.11)
* test helpers
* test names
* add type_is_compatible()
* tests pass
* more tests
* add small comment
* handle Unions as anything else
* use sender/receiver for socket pairs
* more sender/receiver renames
* even more renames
* split if statement
* Update __about__.py
* fix logic operator and add tests
* Update __about__.py
* Simplify imports
* Move draw in pipeline module and clearly define public interface
* Format pyproject.toml
* Include only required files in built wheel
* Move sample components out of tests
* stub component class decorator
* update static sample components to new API
* stub
* dynamic output examples
* sum
* add components fixed
* re-add inputsocket and outputsocket creation
* fix component tests
* fixing tests
* Add methods to set I/O dinamically
* fix drawing
* fix some integration tests
* tests green
* pylint
* remove stray files
* Remove default in InputSocket and add is_optional field
* Fix drawing
* Rework sockets string representation
* Add back Component Protocol
* Simplify method to get string representation of types
* Remove sockets __str__
* Remove Component's I/O type checks at run time
* Remove IO check in init wrapper
* Update canals/utils.py
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Split __canals_io__ field in __canals_input__ and __canals_output__
* Order input and output fields
* Add test to verify __canals_component__ is set
* Remove empty line
* Add component class factory
* Fix API docs workflow failure
* fix api docs
* Update __about__.py
* Add component from_dict and to_dict methods
* Add Pipeline to_dict and from_dict
* Fix components tests
* Add some more tests
* Change error messages
* Simplify test_to_dict
* Add max_loops_allowed in test_to_dict
* Test non default max_loops_allowed in test_to_dict
* Rework marshal_pipelines
* Rework unmarshal_pipelines
* Rename some stuff
* allow falsy outputs
* apply falsy fix to validation
* add test for falsy inputs
* Split _cleanup_marshalled_data into two functions
* Use from_dict to deserialise component
* Remove commented out code and update variable name
* Add test to verify difference when unmarshaling Pipeline with duplicate names
* Update marshal_pipelines docstring
* update workflow
* exclude tests from mypy in pre-commit hooks
* add additional falsy tests
* remove unnecessary import
* split test into two
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* remove init_parameters decorator and fix assumptions
* fix accumulate
* stray if
* Bump version to 0.5.0
* Implement generic default_to_dict and default_from_dict
* Update default_to_dict docstring
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Remove all mentions of Component.defaults
* Add Remainder to_dict and from_dict (#91)
* Add Repeat to_dict and from_dict (#92)
* Add Sum to_dict and from_dict (#93)
* Add Greet to_dict and from_dict (#89)
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Rework Accumulate to_dict and from_dict (#86)
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Add to_dict and from_dict for Parity, Subtract, Double, Concatenate (#87)
* Add Concatenate to_dict and from_dict
* Add Double to_dict and from_dict
* Add Subtract to_dict and from_dict
* Add Parity to_dict and from_dict
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Change _to_mermaid_text to use component serialization data (#94)
* Add MergeLoop to_dict and from_dict (#90)
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Add Threshold to_dict and from_dict (#97)
* Add AddFixedValue to_dict and from_dict (#88)
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Remove BaseTestComponent (#99)
* Change @component decorator so it doesn't add default to_dict and from_dict (#98)
* Rename some classes in tests to suppress Pytest warnings (#101)
* Check Component I/O socket names are valid (#100)
* Remove handling of shared component instances on Pipeline serialization (#102)
* Fix docs
* Bump version to 0.6.0
* Revert "Check Component I/O socket names are valid (#100)" (#103)
This reverts commit 4529874b562d12331ee2f4fde926ef5b5e3d24d7.
* Bump canals to 0.7.0
* Downgrade log from ERROR to DEBUG (#104)
* Make to/from_dict optional (#107)
* remove from/to dict from Protocol
* use a default marshaller
* example component with no serializers
* fix linting
* make it smarter
* fix linting
* thank you mypy protector of the dumb programmers
* feat: check returned dictionary (#106)
* better error message if components don't return dictionaries
* add test
* use factory
* needless import
* Update __about__.py
* fix default serialization and adjust sample components accordingly (#109)
* fix default serialization and adjust sample components accordingly
* typo
* fix pylint errors
* fix: `draw` function vs init parameters (#115)
* fix draw
* stray print
* Update version (#118)
* remove extras
* Revert "remove extras"
This reverts commit a096ff8f07bdcb6e54ec8457bcfad5db44d8bf03.
* fix package name, change _parse_connection_name function name, add tests (#126)
* move sockets into components package (#127)
* chore: remove extras (#125)
* remove extras
* workflow
* typo
* fix: Sockets named "text/plain" or containing a "/" fail during pipeline.to_dict (#131)
* don't split sockets by /
* revert hashing edge keys
* docs: remove missing module from docs (#132)
* remove stray print (#123)
* addo sockets docs (#133)
* tidy up utils about types (#129)
* Update canals.md (#134)
* rename module in API docs
* make `__canals_output__` and `__canals_input__` management consistent (#128)
* make __canals_output__ and __canals_input__ management consistent and assign them to the component instance
* make pylint happy
* return the original type instead of the metaclass
* use type checking instead of instance field
* declare the actual returned type
* fix after conflict resolution
* remove check
* Do not use a dict as intermediate format and use `Socket`s directly (#135)
* do not use a dict as intermediate format and use sockets directly to simplify code and remove side effects
* fix leftover from cherry-pick
* move is_optional evaluation for InputSocket to post_init (#136)
* re-introduce variadics to support Joiner node (#122)
* move sockets into components package
make __canals_output__ and __canals_input__ management consistent and assign them to the component instance
do not use a dict as intermediate format and use sockets directly to simplify code and remove side effects
move is_optional evaluation for InputSocket to post_init
re-introduce variadics to support Joiner node
restore connection-time check
use custom type annotation, fix tests
* fix leftovers from rebase
* rename fan-in to joiner
* clean up and fix typing
* let inputs arrive later
* address review comments
* address review comments
* fix docstrings
* try
* try
* fix run input
* linting
* remove comments
* fix pylint
* bumb version to 0.9.0 (#140)
* properly annotate classmethods (#139)
* feat: add `Pipeline.inputs()` (#120)
* add Pipeline.describe_input()
* add tests
* split dict and str outputs and add to error messages
* tests
* accepts/expects
* move methods
* fix tests
* fix module name
* tests
* review feedback
* Add missing typing_extensions dependency (#152)
* feat: use full connection data to route I/O (#148)
* fix sample components
* make sum variadic
* separate queue and buffer
* all works but loops & variadics together
* fix some tests
* fix some tests
* all tests green
* clean up code a bit
* refactor code
* fix tests
* fix self loops
* fix reused sockets bug
* add distinct loops
* add distinct loops test
* break out some code from run()
* docstring
* improve variadics drawing
* black
* document the deepcopy
* re-arrange connection dataclass and add tests
* consumer -> receiver
* fix typing
* move Connection-related code under component package
* clean up connect()
* cosmetics and typing
* fix linter, make Connection a dataclass again
* fix typing
* add test case for #105
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* feat: Add Component inputs/outputs functions (#158)
* Add component inputs/outputs methods
* Different impl approach
* Black fixes
* Rename functions to match naming in pipeline inputs/ouputs
* Fix find_component_inputs, update unit tests (#162)
* Fix API docs (#164)
* make Variadic wrap an iterable (#163)
* Add pipeline outputs method (#150)
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* Update __about__.py (#165)
Update version to 0.10.0
* add CODEOWNERS
* feat: read defaults from `run()` signature (#166)
* Read defaults from run signature
* simplify setting of sockets
* fix test
* Update sample_components/fstring.py
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Update canals/component/component.py
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* dostring
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Use full import path as 'type' in serialization. (#167)
* Use full import path as 'type' in serialization. Try to import the path when deserializing
* fix test data
* add from_dict test
* remove leftover
* Update canals/pipeline/pipeline.py
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* add error message to PipelineError
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* bump version
* fix: copy input values before passing them down pipeline.run (#168)
* copy input values before passing them down pipeline.run
* Update test_mutable_inputs.py
* fix mypy and pyright (#169)
* bump version
* remove data we won't keep
* reformat
* try
* skip tests on transient code
---------
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
Co-authored-by: Michel Bartels <login@michelbartels.com>
Co-authored-by: ZanSara <sarazanzo94@gmail.com>
Co-authored-by: Julian Risch <julianrisch@gmx.de>
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
Co-authored-by: Vladimir Blagojevic <dovlex@gmail.com>
* Upgrade transformers to the latest version 4.34.0 so that Haystack can support the new Mistral, Nougat, and other models.
* update release notes
* updated missing lazy import
* Update .github workflows imports
* bump more versions in .github workflows
* rever import sorting
* Update to catch runtime errors to match haystack_hub changes
* add language parameter value to whisper test
* bump transformers version in linting preview workflow
* bump transformers version in linting preview workflow
* bump version to v4.34.1
* resolve mypy issue with reused variables
* install openai-whisper without dependencies
* remove audio extra, update whisper install instructions
* remove audio extra, update whisper install instructions
* keep audio extra but add version
* keep audio extra with no constraints
* remove audio extra
---------
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
* add preview workflows
* feedback
* feedback
* use preview extra
* remove coverage and add separate e2e
* rename workflow file for consistency
* trigger ci
* undo trigger
* torch import in testing
* add deps to unit tests
* feedback
* run container instead of service
* comment
* add if statement
* fix tika version
* separate out win integration tests
* separate out all CIs
* try installing docker on macos
* exclude tika
* remove tika docker
* copy the deps list over from haystack-ai
* fix lazyimport usage
* keep jinja and openai
* fix ci
* reno
* separate out preview unit tests
* fix import error message for tika
* tika
* add preview to all
* wrap torch
* remove comment
* unwrap openai and jinja
* Add TikaFileToDocument component
* Add tests
* Add tika service to CI
* Add release note
* Change name
* PR feedback
* Fix naming in tests
* Fix tika version in CI
* Update tests
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
