* Add pre-commit config
* update contributing guidelines
* try failing the workflow
* add pre-commit to the deps
* updating uninstall instructions
* separate jobs in CI
* make tutorials check fail
* make black check fail
* make openapi check fail
* make yaml schema and api docs checks fail
* highlight the instructions
* Update .pre-commit-config.yaml
Co-authored-by: Tobias Wochinger <mail@tobias-wochinger.de>
* Update CONTRIBUTING.md
Co-authored-by: Tobias Wochinger <mail@tobias-wochinger.de>
* Update CONTRIBUTING.md
Co-authored-by: Tobias Wochinger <mail@tobias-wochinger.de>
* Use black --check
* Add images of the CI
* title level
* feedback
Co-authored-by: Tobias Wochinger <mail@tobias-wochinger.de>
* restart tutorials in the loop
* remove container steps in tutorials.yml
* forgotten quotes
* unmatched bracket
* give names to containers
* try to limit the log size
* make the containers restart on the scripts as well
* feedback
* Raise integration tests timeout
* raising limit again
* Remove caching and install audio deps
* Fix `Tutorials` as well
* Run all tutorials even though some fail
* Forgot fi
* fix failure condition
* proper bash string equality
* Enable debug logs
* remove audio files
* Update Documentation & Code Style
* Use the setup action in the Tutorial CI as well
* Try with a file that exists
* Update Documentation & Code Style
* Fix the comments in the tutorials
* Update Documentation & Code Style
* Fix tutorials.sh
* Remove debug logging
* import pprint and try editable install
* Update Documentation & Code Style
* extract no run list
* Add tutorial18 to no run list nightly
* import pprint correctly
* Update Documentation & Code Style
* try making site-packages editable
* Make pythonpath editable every time Tut17 is run on CI
* typo
* fix imports in tut5
* add git clean
* Update Documentation & Code Style
* add comments and remove` -e`
* accidentally deleted a line
* Update .github/utils/tutorials.sh
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Experimental Ci workflow for running tutorials
* Run on every push for now
* Not starting?
* Disabling paths temporarily
* Sort tutorials in natural order
* Install ipython
* remove ipython install
* Try running ipython with sudo
* env.pythonLocation
* Skipping tutorial2 and 9 for speed
* typo
* Use one runner per tutorial, for now
* Typo in dependend job
* Missing quotes broke scripts matrix
* Simplify setup for the tutorials, try to prevent containers conflict
* Remove needless job dependencies
* Try prevent cache issues, fix small Tut10 bug
* Missing deps for running notebook tutorials
* Create three groups of tutorials excluding the longest among them
* remove deps
* use proper bash loop
* Try with a single string
* Fix typo in echo
* Forgot do
* Typo
* Try to make the GraphDB tutorial without launching its own container
* Run notebook and script together
* Whitespace
* separate scrpits and notebooks execution
* Run notebooks first
* Try caching the GoT data before running the scripts
* add note
* fix mkdir
* Fix path
* Update Documentation & Code Style
* missing -r
* Fix folder numbering
* Run notebooks as well
* Typo in notebook command
* complete path in notebook command
* Try with TIKA_LOG_PATH
* Fix folder naming
* Do not use cached data in Tut9
* extracting the number better
* Small tweaks
* Same fix on Tut10 on the notebook
* Exclude GoT cache for tut5 too
* Remove faiss files after tutorial run
* Layout
* fix remove command
* Fix path in tut10 notebook
* Fix typo in node name in tut14
* Third block was too long, rebancing
* Reduce GoT dataset even more, why wasting time after all...
* Fix paths in tut10 again
* do git clean to make sure to cleanup everything (breaks post Python)
* Remove ES file with bad permission at the end of the run
* Split first block, takes >30mins
* take out tut15 for a moment, has an actual bug
* typo
* Forgot rm option
* Simply remove all ES files
* Improve logs of GoT reduction
* Exclude also tut16 from cache to try fix bug
* Replace ll with ls
* Reintroduce 15_TableQA
* Small regrouping
* regrouping to make the min num of runners go for about 30mins
* Add cron schedule and PR paths conditions
* Add some timing information
* Separate tutorials by diff and tutorials by cron
* temp add pull_request to tutorials nightly
* Add badge in README to keep track of the nightly tutorials run
* Remove prefixes from data folder names
* Add fetch depth to get diff with master
* Fix paths again
* typo
* Exclude long-running ones
* Typo
* Fix tutorials.yml as well
* Use head_ref
* Using an action for now
* exclude other files
* Use only the correct command to run the tutorial
* Add long running tutorials in separate runners, just for experiment
* Factor out the complex bash script
* Pass the python path to the bash script
* Fix paths
* adding log statement
* Missing dollarsign
* Resetting variable in loop
* using mini GoT dataset and improving bash script
* change dataset name
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Change exception into warning, add strict_version param, and remove compatibility between schemas
* Simplify update_json_schema
* Rename unstable into master
* Prevent validate_config from changing the config to validate
* Fix version validation and add tests
* Rename master into ignore
* Complete parameter rename
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* remove duplicate imports
* fix ungrouped-imports
* Fix wrong-import-position
* Fix unused-import
* pyproject.toml
* Working on wrong-import-order
* Solve wrong-import-order
* fix Pool import
* Move open_search_index_to_document_store and elasticsearch_index_to_document_store in elasticsearch.py
* remove Converter from modeling
* Fix mypy issues on adaptive_model.py
* create es_converter.py
* remove converter import
* change import path in tests
* Restructure REST API to not rely on global vars from search.apy and improve tests
* Fix openapi generator
* Move variable initialization
* Change type of FilterRequest.filters
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Add BasePipeline.validate_config, BasePipeline.validate_yaml, and some new custom exception classes
* Make error composition work properly
* Clarify typing
* Help mypy a bit more
* Update Documentation & Code Style
* Enable autogenerated docs for Milvus1 and 2 separately
* Revert "Enable autogenerated docs for Milvus1 and 2 separately"
This reverts commit 282be4a78a6e95862a9b4c924fc3dea5ca71e28d.
* Update Documentation & Code Style
* Re-enable 'additionalProperties: False'
* Add pipeline.type to JSON Schema, was somehow forgotten
* Disable additionalProperties on the pipeline properties too
* Fix json-schemas for 1.1.0 and 1.2.0 (should not do it again in the future)
* Cal super in PipelineValidationError
* Improve _read_pipeline_config_from_yaml's error handling
* Fix generate_json_schema.py to include document stores
* Fix json schemas (retro-fix 1.1.0 again)
* Improve custom errors printing, add link to docs
* Add function in BaseComponent to list its subclasses in a module
* Make some document stores base classes abstract
* Add marker 'integration' in pytest flags
* Slighly improve validation of pipelines at load
* Adding tests for YAML loading and validation
* Make custom_query Optional for validation issues
* Fix bug in _read_pipeline_config_from_yaml
* Improve error handling in BasePipeline and Pipeline and add DAG check
* Move json schema generation into haystack/nodes/_json_schema.py (useful for tests)
* Simplify errors slightly
* Add some YAML validation tests
* Remove load_from_config from BasePipeline, it was never used anyway
* Improve tests
* Include json-schemas in package
* Fix conftest imports
* Make BasePipeline abstract
* Improve mocking by making the test independent from the YAML version
* Add exportable_to_yaml decorator to forget about set_config on mock nodes
* Fix mypy errors
* Comment out one monkeypatch
* Fix typing again
* Improve error message for validation
* Add required properties to pipelines
* Fix YAML version for REST API YAMLs to 1.2.0
* Fix load_from_yaml call in load_from_deepset_cloud
* fix HaystackError.__getattr__
* Add super().__init__()in most nodes and docstore, comment set_config
* Remove type from REST API pipelines
* Remove useless init from doc2answers
* Call super in Seq3SeqGenerator
* Typo in deepsetcloud.py
* Fix rest api indexing error mismatch and mock version of JSON schema in all tests
* Working on pipeline tests
* Improve errors printing slightly
* Add back test_pipeline.yaml
* _json_schema.py supports different versions with identical schemas
* Add type to 0.7 schema for backwards compatibility
* Fix small bug in _json_schema.py
* Try alternative to generate json schemas on the CI
* Update Documentation & Code Style
* Make linux CI match autoformat CI
* Fix super-init-not-called
* Accidentally committed file
* Update Documentation & Code Style
* fix test_summarizer_translation.py's import
* Mock YAML in a few suites, split and simplify test_pipeline_debug_and_validation.py::test_invalid_run_args
* Fix json schema for ray tests too
* Update Documentation & Code Style
* Reintroduce validation
* Usa unstable version in tests and rest api
* Make unstable support the latest versions
* Update Documentation & Code Style
* Remove needless fixture
* Make type in pipeline optional in the strings validation
* Fix schemas
* Fix string validation for pipeline type
* Improve validate_config_strings
* Remove type from test p[ipelines
* Update Documentation & Code Style
* Fix test_pipeline
* Removing more type from pipelines
* Temporary CI patc
* Fix issue with exportable_to_yaml never invoking the wrapped init
* rm stray file
* pipeline tests are green again
* Linux CI now needs .[all] to generate the schema
* Bugfixes, pipeline tests seems to be green
* Typo in version after merge
* Implement missing methods in Weaviate
* Trying to avoid FAISS tests from running in the Milvus1 test suite
* Fix some stray test paths and faiss index dumping
* Fix pytest markers list
* Temporarily disable cache to be able to see tests failures
* Fix pyproject.toml syntax
* Use only tmp_path
* Fix preprocessor signature after merge
* Fix faiss bug
* Fix Ray test
* Fix documentation issue by removing quotes from faiss type
* Update Documentation & Code Style
* use document properly in preprocessor tests
* Update Documentation & Code Style
* make preprocessor capable of handling documents
* import document
* Revert support for documents in preprocessor, do later
* Fix bug in _json_schema.py that was breaking validation
* re-enable cache
* Update Documentation & Code Style
* Simplify calling _json_schema.py from the CI
* Remove redundant ABC inheritance
* Ensure exportable_to_yaml works only on implementations
* Rename subclass to class_ in Meta
* Make run() and get_config() abstract in BasePipeline
* Revert unintended change in preprocessor
* Move outgoing_edges_input_node check inside try block
* Rename VALID_CODE_GEN_INPUT_REGEX into VALID_INPUT_REGEX
* Add check for a RecursionError on validate_config_strings
* Address usages of _pipeline_config in data silo and elasticsearch
* Rename _pipeline_config into _init_parameters
* Fix pytest marker and remove unused imports
* Remove most redundant ABCs
* Rename _init_parameters into _component_configuration
* Remove set_config and type from _component_configuration's dict
* Remove last instances of set_config and replace with super().__init__()
* Implement __init_subclass__ approach
* Simplify checks on the existence of _component_configuration
* Fix faiss issue
* Dynamic generation of node schemas & weed out old schemas
* Add debatable test
* Add docstring to debatable test
* Positive diff between schemas implemented
* Improve diff printing
* Rename REST API YAML files to trigger IDE validation
* Fix typing issues
* Fix more typing
* Typo in YAML filename
* Remove needless type:ignore
* Add tests
* Fix tests & validation feedback for accessory classes in custom nodes
* Refactor RAGeneratorType out
* Fix broken import in conftest
* Improve source error handling
* Remove unused import in test_eval.py breaking tests
* Fix changed error message in tests matches too
* Normalize generate_openapi_specs.py and generate_json_schema.py in the actions
* Fix path to generate_openapi_specs.py in autoformat.yml
* Update Documentation & Code Style
* Add test for FAISSDocumentStore-like situations (superclass with init params)
* Update Documentation & Code Style
* Fix indentation
* Remove commented set_config
* Store model_name_or_path in FARMReader to use in DistillationDataSilo
* Rename _component_configuration into _component_config
* Update Documentation & Code Style
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Make YAML files get the same version as Haystack and throw warning at load in case of mismatch
* Update version of most YAMLs in the codebase (aesthethic chamge, only to avoid the warning).
* Remove quotes from version in tests
* Fix version in generate_json_schema.py
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Upgrade pydoc-markdown and fix the YAMLs to work with it
* Pin pydoc-markdown to major version
* Generalize pydoc-markdown workflow
* Make a single Action to perform all tasks that require committing into the local branch
* Merge the code updates and the docs in the Linux CI to prevent the bot from always show the pipeline as green
* Installing Jupyter deps for Black
* Build cache before running generation tasks
* Add check not to run the code generation on master
* Simplify push action
* Add more test deps in setup.cfg and remove from GH Action workflow
* Remove forced upgrades on pip install
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Testing black on ui/
* Applying black on docstores
* Add latest docstring and tutorial changes
* Create a single GH action for Black and docs to reduce commit noise to the minimum, slightly refactor the OpenAPI action too
* Remove comments
* Relax constraints on pydoc-markdown
* Split temporary black from the docs. Pydoc-markdown was obsolete and needs a separate PR to upgrade
* Fix a couple of bugs
* Add a type: ignore that was missing somehow
* Give path to black
* Apply Black
* Apply Black
* Relocate a couple of type: ignore
* Update documentation
* Make Linux CI run after applying Black
* Triggering Black
* Apply Black
* Remove dependency, does not work well
* Remove manually double trailing commas
* Update documentation
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* 🎨 Update type annotations to allow their extraction for JSON Schema
* ✨ Add main script doing all the work to generate the JSON Schema
* ➕ Add GitHub Action dependency to generate JSON Schema
* ✨ Update JSON Schema generation script to allow easily generating the schema without making a PR
* 👷 Add GitHub Action to generate JSON Schema
* 💚 Fix CI GitHub Action
* 💚 Update GitHub Action environment variables
* ✨ Add initial JSON Schema
* Add latest docstring and tutorial changes
* 🐛 Do not allow extra params not defined in each model
* ♻️ Make any additional properties invalid
* ✨ Make other additional properties invalid in all the levels in pipelines
* ♻️ Do not include Base classes as possible nodes
* 🍱 Update JSON Schema
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
