haystack/releasenotes/notes/optimize-pinecone-document-store.yaml
Ivana Zeljkovic 2326f2f9fe
feat: Pinecone document store optimizations (#5902)
* Optimize methods for deleting documents and getting vector count. Enable warning messages when Pinecone limits are exceeded on Starter index type.

* Fix typo

* Add release note

* Fix mypy errors

* Remove unused import. Fix warning logging message.

* Update release note with description about limits for Starter index type in Pinecone

* Improve code base by:
- Adding new test cases for get_embedding_count method
- Fixing get_embedding_count method
- Improving delete documents
- Fix label retrieval
- Increase default batch size
- Improve get_document_count method

* Remove unused variable

* Fix mypy issues
2023-10-16 19:26:24 +02:00

12 lines
638 B
YAML

---
enhancements:
- |
Optimize particular methods from PineconeDocumentStore (delete_documents and _get_vector_count)
upgrade:
- |
This update enables all Pinecone index types to be used, including Starter.
Previously, Pinecone Starter index type couldn't be used as document store. Due to limitations of this index type
(https://docs.pinecone.io/docs/starter-environment), in current implementation fetching documents is limited to
Pinecone query vector limit (10000 vectors). Accordingly, if the number of documents in the index is above this limit,
some of PineconeDocumentStore functions will be limited.