* added meta fields for meta_config to be used during realtime testing of PineconeDocumentStore
* Add documentation on metadata filtering in docstring
* docs
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
* Enable the `JoinDocuments` node to work with documents with `score=None`
This fixes#2983
As of now, the `JoinDocuments` node will error out if any of the documents has `score=None` - which is possible, as some retriever are not able to provide a score, like the `TfidfRetriever` on Elasticsearch or the `BM25Retriever` on Weaviate.
THe reason for the error is that the `JoinDocuments` always sorts the documents by score and cannot sort when `score=None`.
There was a very similar issue for `JoinAnswers` too, which was addressed by this PR: https://github.com/deepset-ai/haystack/pull/2436
This solution applies the same solution to `JoinDocuments` - so both the `JoinAnswers` and `JoinDocuments` now will have the same additional argument to disable sorting when that is requried.
The solution is to add an argument to `JoinDocuments` called `sort_by_score: bool`, which allows the user to turn off the sorting of documents by score, but keeps the current functionality of sorting being performed as the default.
* Fixing test bug
* Addressing PR review comments
- Extending unit tests
- Simplifying logic
* Making the sorting work even with no scores
By making the no score being sorted as -Inf
* Forgot to commit the change in `join_docs.py`
* [EMPTY] Re-trigger CI
* Added am INFO log if the `JoinDocuments` is sorting while some of the docs have `score=None`
* Adjusting the arguments of `any()`
* [EMPTY] Re-trigger CI
* Refactoring the `Raypipeline.run` method - merging it with the `Pipeline.run`
This is to fix#2968
* Bug: variable `i` was already in use
* Removing unused imports
* Removing unused import
* [EMPTY] Re-trigger CI
* Addressing concerns raised pre-review
- Removing the attempt to try to make it without the need for `JoinDocuments` - it is okey to fail without `JoinDocuments` for certain pipelines.
* Refactoring based on reviews
* Adding support for additional distance metrics for Weaviate
Fixes#3000
* Updating the docs
* Fixing error texts
* Fixing issues raised by the review
* Addressing the last issue from the reviews - removing test `test_weaviate.py::test_similarity`
* [EMPTY] Re-trigger CI
* Fixing things based on review
* [EMPTY] Re-trigger CI
* feat: fetch results for DeepsetCloudExperiments
* chore: test DC fetch predicitons for eval run
* chore: switch to dict iteration with .items()
* chore: update DC url to fetch predictions from
* chore: update doc strings for fetching eval run results
* chore: update DeepsetCloudExperiments description, change function names for fetching predictions of an eval run
* chore: test for DeepsetCloudExperiments.get_run_results
* chore: adjust request mock for test_get_eval_run_results
* chore: push first row of dataframe into variable for test checks
* chore: adjust mock data to correct data types
* chore: make documentation more readable with line breaks
* chore: update documentation for eval run result fetching
* use hashlib.md5() instead of (interpreter dependent) hash() funtion to generate MultiLabel id
* add tests to assess constancy of MultiLabel.id
* make test_multilabel_id test ensure that MultiLabel ids are always the same
* Add page number to Documents coming from PDFConverters and PreProcessor
* Fix mypy
* Update API Docs
* Update API Docs
* Remove unused imports
* Generate JSON schema
* Generate JSON schema
* Make test variable shorter
* Make regex a separate function
* Move counting of page breaks to a function
* Generate JSON schema
* Apply suggestions from code review
Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
* Update API Documentation
* Don't create instance for testing staticmethod
* Update haystack/nodes/preprocessor/preprocessor.py
Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
* fix validation for dynamic outgoing edges
* Update Documentation & Code Style
* use class outgoing_edges as fallback if no instance is provided
* implement classmethod approach
* readd comment
* fix mypy
* fix tests
* set outgoing_edges for all components
* set outgoing_edges for mocks too
* set document store outgoing_edges to 1
* set last missing outgoing_edges
* enforce BaseComponent subclasses to define outgoing_edges
* override _calculate_outgoing_edges for FileTypeClassifier
* remove superfluous test
* set rest_api's custom component's outgoing_edges
* Update docstring
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
* remove unnecessary else
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
* enable Opensearch unit tests under Win
* move unit tests into a dedicated job
* skip audio tests on missing dependencies
* avoid failing test collection when soundfile is not available
* Update .github/workflows/tests.yml
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
* fixed tokens in question generation
* simplified assignment
* same behavior also for pad and eos
* use skip_special_tokens in batch_decode
* fixed black error and update docs
* fixed schemas ci error
* JSON schemas
* Add git diff to debug schema issues
* opensearch schema was missing
* Add missing instruction in the workflow error message
* typo
* Ability to run Ray Serve detached
Fixes#2944
Ability to run Ray Serve detached - to allow running multiple instances of the app (HA).
See https://docs.ray.io/en/latest/serve/package-ref.html#core-apis
* Generating the docs
* Re-trigger the CI pipeline
* Retrigger the CI Pipeline
* Typo in docstrings
* Fixing docstring and typing issues
* Regenerating docs
* [EMPTY] Re-trigger CI
* [EMPTY] Re-trigger CI
* Refactoring to allow any number of args for the `serve.start()` method
There seems to be additional arguments of the `serve.start()` method, so we should probably cover all of them at once, instead of only the `detached` option.
* [EMPTY] Re-trigger CI
* Test whether the ServeControllerClient in fact has the supplied `detached` parameter
* Tutorial 06: Replace DPR with EmbeddingRetriever
Closes#2887
* Add updated tutorials/6.md file
Replace `DensePassageRetriever` with `EmbeddingRetriever`
* Update Tutorial 06 based on PR feedback
* Further updates to Tutorial-06 according to review feedback
* [Tutorial 06] Put in review feedback for the py file
* Extending the Ray Serve integration to allow attributes for Serve deployments
This closes#2917
We should be able to set Ray Serve attributes for the nodes of pipelines, like amount of GPU to use, max_concurrent_queries, etc.
Now this is possible from the pipeline yaml file for each node of the pipeline.
* Ran black and regenerated the json schemas
* Fixing the JSON Schema generation
* Trying to fix the schema CI test issue
* Fixing the test and the schemas
Python 3.8 was generating a different schema than Python 3.7 is creating in the CI. You MUST use Python 3.7 to generate the schemas, otherwise the CIs will fail.
* Merge the two Ray pipeline test cases
* Generate the JSON schemas again after `$ pip install .[all]`
* Removing `haystack/json-schemas/haystack-pipeline-1.16.schema.json`
This was generated by the JSON generator, but based on @ZanSara's instructions, I am removing it.
* Making changes based on @ZanSara's request - the newly requested test is failing
* Fixing the JSON schema generation again
* Renaming `replicas` and moving it under `serve_deployment_kwargs`
* add extras validation, untested
* Dcoumentation update
* Black
* [EMPTY] Re-trigger CI
Co-authored-by: Sara Zan <sarazanzo94@gmail.com>
* Update minimum requirement for selenium for using the crawler
* Updating pin of grpcio to match default in google colab
* Adding requests requirement
* Bug fix Weaviate document deletion
If no filters param is passed in, then the original code retrieves *all* documents before then deleting by their IDs. There's no need for that, since we can delete by their IDs directly.
* Edit comment to clarify deletion and recreation
* Write unit tests for bug fix
