* ci: Use ruff in pre-commit to further limit complexity
* Fix invalid escape sequences in Python code
* Delete releasenotes/notes/ruff-4d2504d362035166.yaml
* Refactor codebase so that doc_type metadata is used instead of namespaces for making distinction between documents without embeddings, documents with embeddings and labels
* Fix parameter name in integration test
* Remove code under comment in add_type_metadata_filter method
* Fix mypy and pylint checks
* Add release note
* Apply minimal changes: rename method, update method docs and remove redundant method
* Mypy fixes
* Fix docstrings
* Revert helper methods for fetching documents when the number of documents exceeds Pinecone limit
* Remove unnecessary attributes in PineconeDocumentStore
* Fix unit test
---------
Co-authored-by: Ivana Zeljkovic <ivana.zeljkovic@smartcat.io>
Co-authored-by: DosticJelena <jelena.dostic@smartcat.io>
* Add job for ES8 integration tests
* Add unit test for Elasticsearch 8
* Add tests.yml
* Adapt tests.yml
* Remove added white space
* Adapt tests.yml
* Adapt tests.yml
* Add dependencies to unit test name
* Adapt unit test matrix
* Adapt unit test matrix
* Adapt unit test matrix
* Adapt unit test matrix
* Update tests.yml
* Create separate tests where necessary
* Fix skip
* Adapt tests
* make a package
* Update haystack/document_stores/elasticsearch/es7.py
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
* do not expose ES types from the package
---------
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
* #4653 fix changing scores by returning new document objects from document store queries
* added integration test for InMemoryDocumentStore demonstrating the desired behavior
* Update test/document_stores/test_memory.py
* Include benchmark config in output
* Use queries from aggregated labels
* Introduce batching for querying in ElasticsearchDocStore and OpenSearchDocStore
* Fix mypy
* Use self.batch_size in write_documents
* Use 10_000 as default batch size
* Add unit tests for write documents
* refactor: make the scope param configurable
the scope parameter is used when authenticating using
AuthClientPassword and AuthClientCredentials
* feat: add support for AuthClientCredentials
add support for authenticating using the OIDC Client Credentials
authentication flow
* feat: add support for AuthBearerToken
Add support for authenticating using OIDC and bearer tokens
* Update lg
* refactor how client is built
Signed-off-by: hsm207 <hsm207@users.noreply.github.com>
* unit test the auth methods
Signed-off-by: hsm207 <hsm207@users.noreply.github.com>
* Update test_weaviate.py
* revert formatting change
* Fix type hints
---------
Signed-off-by: hsm207 <hsm207@users.noreply.github.com>
Co-authored-by: John Doe <johndoe@example.com>
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Add support for dicts to Weaviate
* Add support for _split_overlap to Pinecone
* Add tests
* Fix Pylint
* Fix Pylint
* Fix test
* Implement PR feedback
* extract elasticsearch
* update pyproject.toml
* make more import optional
* move MockBaseRetriever in conftest
* install es in the es integration tests
* clean up the ES instance in a more robust way
* do not sleep, refresh the index instead
* remove client warnings
* fix unit tests
* fix opensearch compatibility
* fix unit tests
* update ES version
* bump elasticsearch-py
* adjust docs
* use recreate_index param
* use same fixture strategy for Opensearch
* Update lg
---------
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
* add protection, in case we use IVF* indexing, we need to train the index first
Signed-off-by: Liu,Kaixuan <kaixuan.liu@intel.com>
* fix formatting issue
Signed-off-by: Liu,Kaixuan <kaixuan.liu@intel.com>
* just raising error, instead of silently training the index
* fixed mypy issue
* fixed error msg
---------
Signed-off-by: Liu,Kaixuan <kaixuan.liu@intel.com>
Co-authored-by: Mayank Jobanputra <mayankjobanputra@gmail.com>
* Add IVF and Product Quantization support for OpenSearchDocumentStore
* Remove unused import statement
* Fix mypy
* Adapt doc strings and error messages to account for PQ
* Adapt validation of indices
* Adapt existing tests
* Fix pylint
* Add tests
* Update lg
* Adapt based on PR review comments
* Fix Pylint
* Adapt based on PR review
* Add request_timeout
* Adapt based on PR review
* Adapt based on PR review
* Adapt tests
* Pin tenacity
* Unpin tenacity
* Adapt based on PR comments
* Add match to tests
---------
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
* add e2e tests
* move tests to their own module
* add e2e workflow
* pylint
* remove from job
* fix index field name
* skip test on sql
* removed unused code
* fix embedding tests
* adjust test for pinecone
* adjust assertions to the new documents
* bad copypasta
* test
* fix tests
* fix tests
* fix test
* fix tests
* pylint
* update milvus version
* remove debug
* move graphdb tests under e2e
* bug(OpenSearchDocumentStore): fix authenticate with aws4auth if set.
Rearrange check to authenticate with aws4auth before username
and password, as the username is set to "admin" by default.
* Make username check less restrictive
* Fix test, do not used mocked _init_client function
* Add warning for aws4auth and username to ElasticSearchDocumentStore
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
* Adding condition to `pinecone` object.
While you can assign any values to `PineconeDocumentStore`'s parameter `pinecone_index`, it must have another condition to prevent that from happening.
* Added test, and changed the code to make sure the pinecone idx variable has correct instance
* fixed black error
Co-authored-by: Mayank Jobanputra <mayankjobanputra@gmail.com>
