* fix and new test
* make mypy ignore rest_api tests files
* try to improve mypy action
* retry
* fix
* test new action
* ok
* check python files not in root
* really check files!
* enable logging-fstring-interpolation
* remove logging-fstring-interpolation from exclusion list
* remove implicit string interpolations added by black
* remove from rest_api too
* fix % sign
* bug: fix the docs rest api reference url
* revert openapi json changes
* remove last line on json files
* Add explanation about `servers` and remove `servers` parameter from FastAPI
* generate openapi schema without empty end line
* ci: add license compliance check
* ci: run check always for testing purposes
* revamp workflows
* temporary remove path directive
* triggering ci
* check rest api and ui too
* avoid cache to make sure env is clean
* add shield on readme
* ci: trigger CI to get latest scan
Co-authored-by: ZanSara <sarazanzo94@gmail.com>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
* remove json-schemas
* main schema can be removed too
* add .gitignore to schemas folder
* try to explicitly get the new haystack in the rest api tests
* fix workflow again
* fix version string in rest api tests
* add pip freeze
* debug statements in workflow
* -U prevents schema generation
* Use the %s syntax on all debug messages
* Use the %s syntax on some more debug messages
* Use the %s syntax on info messages
* Use the %s syntax on warning messages
* Use the %s syntax on error and exception messages
* mypy
* pylint
* trogger tutorials execution in CI
* trigger tutorials execution on CI
* black
* remove embeddings from repr
* fix Document `__repr__`
* address feedback
* mypy
* feat: add /health endpoint to rest api
* refactor: adjust to new dir structure
* fix: add new rest api dependency
* docs: add new openapi schema
* docs: manual black run
* refactor: remove some sys-wide details
* docs: minor description changes
* docs: minor description changes
* docs: generate openapi schemas
* tests: improved tests
* refactor: add cls method decorator
* fix validation for dynamic outgoing edges
* Update Documentation & Code Style
* use class outgoing_edges as fallback if no instance is provided
* implement classmethod approach
* readd comment
* fix mypy
* fix tests
* set outgoing_edges for all components
* set outgoing_edges for mocks too
* set document store outgoing_edges to 1
* set last missing outgoing_edges
* enforce BaseComponent subclasses to define outgoing_edges
* override _calculate_outgoing_edges for FileTypeClassifier
* remove superfluous test
* set rest_api's custom component's outgoing_edges
* Update docstring
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
* remove unnecessary else
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
* clean up tests and run earlier
* use change detection
* better naming, skip ES
* more cleanup
* fix job name
* dummy commit to trigger the CI
* mock away the PDF converter
* make the test compatible with 3.7
* removed leftover
* always run the api tests, use a matrix for the OS
* refactor all the tests
* remove outdated dependency
* pylint
* new abstract method
* adjust for older python versions
* rename pipeline file
* address PR comments
* Change exception into warning, add strict_version param, and remove compatibility between schemas
* Simplify update_json_schema
* Rename unstable into master
* Prevent validate_config from changing the config to validate
* Fix version validation and add tests
* Rename master into ignore
* Complete parameter rename
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* remove duplicate imports
* fix ungrouped-imports
* Fix wrong-import-position
* Fix unused-import
* pyproject.toml
* Working on wrong-import-order
* Solve wrong-import-order
* fix Pool import
* Move open_search_index_to_document_store and elasticsearch_index_to_document_store in elasticsearch.py
* remove Converter from modeling
* Fix mypy issues on adaptive_model.py
* create es_converter.py
* remove converter import
* change import path in tests
* Restructure REST API to not rely on global vars from search.apy and improve tests
* Fix openapi generator
* Move variable initialization
* Change type of FilterRequest.filters
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* add basic telemetry features
* change pipeline_config to _component_config
* Update Documentation & Code Style
* add super().__init__() calls to error classes
* make posthog mock work with python 3.7
* Update Documentation & Code Style
* update link to docs web page
* log exceptions, send event for raised HaystackErrors, refactor Path(CONFIG_PATH)
* add comment on send_event in BaseComponent.init() and fix mypy
* mock NonPrivateParameters and fix pylint undefined-variable
* Update Documentation & Code Style
* check model path contains multiple /
* add test for writing to file
* add test for en-/disable telemetry
* Update Documentation & Code Style
* merge file deletion methods and ignore pylint global statement
* Update Documentation & Code Style
* set env variable in demo to activate telemetry
* fix mock of HAYSTACK_TELEMETRY_ENABLED
* fix mypy and linter
* add CI as env variable to execution contexts
* remove threading, add test for custom error event
* Update Documentation & Code Style
* simplify config/log file deletion
* add test for final event being sent
* force writing config file in test
* make test compatible with python 3.7
* switch to posthog production server
* Update Documentation & Code Style
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Add BasePipeline.validate_config, BasePipeline.validate_yaml, and some new custom exception classes
* Make error composition work properly
* Clarify typing
* Help mypy a bit more
* Update Documentation & Code Style
* Enable autogenerated docs for Milvus1 and 2 separately
* Revert "Enable autogenerated docs for Milvus1 and 2 separately"
This reverts commit 282be4a78a6e95862a9b4c924fc3dea5ca71e28d.
* Update Documentation & Code Style
* Re-enable 'additionalProperties: False'
* Add pipeline.type to JSON Schema, was somehow forgotten
* Disable additionalProperties on the pipeline properties too
* Fix json-schemas for 1.1.0 and 1.2.0 (should not do it again in the future)
* Cal super in PipelineValidationError
* Improve _read_pipeline_config_from_yaml's error handling
* Fix generate_json_schema.py to include document stores
* Fix json schemas (retro-fix 1.1.0 again)
* Improve custom errors printing, add link to docs
* Add function in BaseComponent to list its subclasses in a module
* Make some document stores base classes abstract
* Add marker 'integration' in pytest flags
* Slighly improve validation of pipelines at load
* Adding tests for YAML loading and validation
* Make custom_query Optional for validation issues
* Fix bug in _read_pipeline_config_from_yaml
* Improve error handling in BasePipeline and Pipeline and add DAG check
* Move json schema generation into haystack/nodes/_json_schema.py (useful for tests)
* Simplify errors slightly
* Add some YAML validation tests
* Remove load_from_config from BasePipeline, it was never used anyway
* Improve tests
* Include json-schemas in package
* Fix conftest imports
* Make BasePipeline abstract
* Improve mocking by making the test independent from the YAML version
* Add exportable_to_yaml decorator to forget about set_config on mock nodes
* Fix mypy errors
* Comment out one monkeypatch
* Fix typing again
* Improve error message for validation
* Add required properties to pipelines
* Fix YAML version for REST API YAMLs to 1.2.0
* Fix load_from_yaml call in load_from_deepset_cloud
* fix HaystackError.__getattr__
* Add super().__init__()in most nodes and docstore, comment set_config
* Remove type from REST API pipelines
* Remove useless init from doc2answers
* Call super in Seq3SeqGenerator
* Typo in deepsetcloud.py
* Fix rest api indexing error mismatch and mock version of JSON schema in all tests
* Working on pipeline tests
* Improve errors printing slightly
* Add back test_pipeline.yaml
* _json_schema.py supports different versions with identical schemas
* Add type to 0.7 schema for backwards compatibility
* Fix small bug in _json_schema.py
* Try alternative to generate json schemas on the CI
* Update Documentation & Code Style
* Make linux CI match autoformat CI
* Fix super-init-not-called
* Accidentally committed file
* Update Documentation & Code Style
* fix test_summarizer_translation.py's import
* Mock YAML in a few suites, split and simplify test_pipeline_debug_and_validation.py::test_invalid_run_args
* Fix json schema for ray tests too
* Update Documentation & Code Style
* Reintroduce validation
* Usa unstable version in tests and rest api
* Make unstable support the latest versions
* Update Documentation & Code Style
* Remove needless fixture
* Make type in pipeline optional in the strings validation
* Fix schemas
* Fix string validation for pipeline type
* Improve validate_config_strings
* Remove type from test p[ipelines
* Update Documentation & Code Style
* Fix test_pipeline
* Removing more type from pipelines
* Temporary CI patc
* Fix issue with exportable_to_yaml never invoking the wrapped init
* rm stray file
* pipeline tests are green again
* Linux CI now needs .[all] to generate the schema
* Bugfixes, pipeline tests seems to be green
* Typo in version after merge
* Implement missing methods in Weaviate
* Trying to avoid FAISS tests from running in the Milvus1 test suite
* Fix some stray test paths and faiss index dumping
* Fix pytest markers list
* Temporarily disable cache to be able to see tests failures
* Fix pyproject.toml syntax
* Use only tmp_path
* Fix preprocessor signature after merge
* Fix faiss bug
* Fix Ray test
* Fix documentation issue by removing quotes from faiss type
* Update Documentation & Code Style
* use document properly in preprocessor tests
* Update Documentation & Code Style
* make preprocessor capable of handling documents
* import document
* Revert support for documents in preprocessor, do later
* Fix bug in _json_schema.py that was breaking validation
* re-enable cache
* Update Documentation & Code Style
* Simplify calling _json_schema.py from the CI
* Remove redundant ABC inheritance
* Ensure exportable_to_yaml works only on implementations
* Rename subclass to class_ in Meta
* Make run() and get_config() abstract in BasePipeline
* Revert unintended change in preprocessor
* Move outgoing_edges_input_node check inside try block
* Rename VALID_CODE_GEN_INPUT_REGEX into VALID_INPUT_REGEX
* Add check for a RecursionError on validate_config_strings
* Address usages of _pipeline_config in data silo and elasticsearch
* Rename _pipeline_config into _init_parameters
* Fix pytest marker and remove unused imports
* Remove most redundant ABCs
* Rename _init_parameters into _component_configuration
* Remove set_config and type from _component_configuration's dict
* Remove last instances of set_config and replace with super().__init__()
* Implement __init_subclass__ approach
* Simplify checks on the existence of _component_configuration
* Fix faiss issue
* Dynamic generation of node schemas & weed out old schemas
* Add debatable test
* Add docstring to debatable test
* Positive diff between schemas implemented
* Improve diff printing
* Rename REST API YAML files to trigger IDE validation
* Fix typing issues
* Fix more typing
* Typo in YAML filename
* Remove needless type:ignore
* Add tests
* Fix tests & validation feedback for accessory classes in custom nodes
* Refactor RAGeneratorType out
* Fix broken import in conftest
* Improve source error handling
* Remove unused import in test_eval.py breaking tests
* Fix changed error message in tests matches too
* Normalize generate_openapi_specs.py and generate_json_schema.py in the actions
* Fix path to generate_openapi_specs.py in autoformat.yml
* Update Documentation & Code Style
* Add test for FAISSDocumentStore-like situations (superclass with init params)
* Update Documentation & Code Style
* Fix indentation
* Remove commented set_config
* Store model_name_or_path in FARMReader to use in DistillationDataSilo
* Rename _component_configuration into _component_config
* Update Documentation & Code Style
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Remove invalid-envvar-default and logging-too-many-args
* Remove import-self, access-member-before-definition and deprecated-argument
* Remove used-before-assignment by restructuring type import
* Remove unneeded-not
* Silence unnecessary-lambda (it's necessary)
* Remove pointless-string-statement
* Update Documentation & Code Style
* Silenced unsupported-membership-test (probably a real bug, can't fix though)
* Remove trailing-newlines
* Remove super-init-not-called and slience invalid-sequence-index (it's valid)
* Remove invalid-envvar-default in ui
* Remove some more warnings from pyproject.toml than actually solrted in code, CI will fail
* Linting all modules together is more readable
* Update Documentation & Code Style
* Typo in pylint disable comment
* Simplify long boolean statement
* Simplify init call in FAISS
* Fix inconsistent-return-statements
* Fix useless-super-delegation
* Fix useless-else-on-loop
* Fix another inconsistent-return-statements
* Move back pylint disable comment moved by black
* Fix consider-using-set-comprehension
* Fix another consider-using-set-comprehension
* Silence non-parent-init-called
* Update pylint exclusion list
* Update Documentation & Code Style
* Resolve unnecessary-else-after-break
* Fix superfluous-parens
* Fix no-else-break
* Remove is_correctly_retrieved along with its pylint issue
* Update exclusions list
* Silence constructor issue in squad_data.py (method is already broken)
* Fix too-many-return-statements
* Fix use-dict-literal
* Fix consider-using-from-import and useless-object-inheritance
* Update exclusion list
* Fix simplifiable-if-statements
* Fix one consider-using-dict-items
* Fix another consider-using-dict-items
* Fix a third consider-using-dict-items
* Fix last consider-using-dict-items
* Fix three use-a-generator
* Silence import errors on numba, tensorboardX and apex, but add comments & logs
* Fix couple of mypy issues
* Fix another typing issue
* Silence mypy, was conflicting with more meaningful pylint issue
* Fix no-else-continue
* Silence unsubscriptable-object and fix an import error with importlib.metadata
* Update Documentation & Code Style
* Fix all no-else-raise
* Update Documentation & Code Style
* Fix inverted parameters in simplified if switch
* Change [test] to [all] in some jobs (for typing and linting)
* Add comment in haystack/schema.py on pydantic's dataclasses
* Move comment from get_documents_by_id into _convert_weaviate_result_to_document in weaviate.py
* Add comment on pylint silencing
* Fix bug introduced rest_api/controller/search.py
* Update Documentation & Code Style
* Add ADR about Pydantic dataclasses
* Update pydantic-dataclasses.md
* Add link to Pydantic docs on Dataclasses
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Make YAML files get the same version as Haystack and throw warning at load in case of mismatch
* Update version of most YAMLs in the codebase (aesthethic chamge, only to avoid the warning).
* Remove quotes from version in tests
* Fix version in generate_json_schema.py
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Add explicit reference to repo name to allow CI to push code back
* Run test matrix only on tested code changes
* Isolate the bot to check if it works
* Clarify situation with a comment
* Simplify autoformat.yml
* Add code and docs check
* Add git pull to make sure to fetch changes if they were created
* Add cache to autoformat.yml too
* Add information on forks in CONTRIBUTING.md
* Add a not about code quality tools in CONTRIBUTING.md
* Add image file types to the CI exclusion list
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Handle no answers and no documents scenarios in '_process_request'
* Fix tests
* Change return type in '_process_request'
* Return to use dicts
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Add type check for meta & add tests
* Improve tests
* Handle properly the ValueError ad an HTTPException
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Add DELETE /feedback for testing and make the ID generate server-side
* Make sure to delete only user generated labels
* Reduce fixture scope, was too broad
* Make test a bit more generic
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
