3174 Commits

Author SHA1 Message Date
Silvano Cerza
76165d024f
Fix corner cases and error handling with filters conversion (#6376) 2023-11-21 18:22:48 +01:00
Stefano Fiorucci
456902235a
feat: make DocumentWriter return the actual number of documents written (#6366)
* make DocumentWriter return the actual number of documents written

* add/improve tests
2023-11-21 15:54:25 +01:00
Silvano Cerza
ec3558021e
Remove Document Store tests with invalid filter (#6375) 2023-11-21 15:08:16 +01:00
Silvano Cerza
0a5b37f3d1
Rework legacy filters embedding tests and remove numpy dependency (#6371) 2023-11-21 14:02:15 +01:00
Daniel Fleischer
0cef17ac13
feat: embedding instructions for dense retrieval (#6372)
* Embedding instructions in EmbeddingRetriever

Query and documents embeddings are prefixed with instructions, useful
for retrievers finetuned on specific tasks, such as Q&A.

* Tests

Checking vectors 0th component vs. reference, using different stores.

* Normalizing vectors

* Release notes
2023-11-21 12:56:40 +01:00
Julian Risch
07cda09aa8
docs: Include TextEmbedder in DocumentJoiner usage example (#6369)
* docs: Include TextEmbedder in DocumentJoiner usage example

* black
2023-11-21 11:27:10 +01:00
Stefano Fiorucci
1fff2bc255
merge lazy import blocks (#6358) 2023-11-21 11:15:37 +01:00
Julian Risch
2943b83b31
fix: Add DocumentJoiner to routers' init (#6368) 2023-11-21 09:45:00 +01:00
Julian Risch
939e443ee8
docs: Add DocumentJoiner to API docs (#6365) 2023-11-20 18:18:06 +01:00
Silvano Cerza
d57760787d
refactor: Rework delete_documents tests (#6363)
* Rework write_documents tests

* Rework delete_documents tests

* Fix linting
2023-11-20 17:54:42 +01:00
Silvano Cerza
9b0e3f5ed4
Rework write_documents tests (#6362) 2023-11-20 17:54:29 +01:00
Silvano Cerza
a7f742fdbd
refactor: Rename docstore fixture to document_store (#6360)
* Prevent pytest_generate_tests from polluting preview tests

* Rename docstore fixture to document_store
2023-11-20 17:41:48 +01:00
Silvano Cerza
365127dc5b
Prevent pytest_generate_tests from polluting preview tests (#6361) 2023-11-20 15:47:06 +01:00
Silvano Cerza
83c245db74
feat: Implement function to convert legacy filters to new style (#6314)
* Implement function to convert legacy filters to new style

* Reduce return statements in conversion to fix linting

* Move convert function in different module

* Fix typos in docstrings

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
2023-11-20 13:00:05 +01:00
Agnieszka Marzec
497299c27a
Docs: Update Rankers docstrings and messages (#6296)
* Update docstrings and messages

* Fix tests

* Fix formatting

* Update haystack/preview/components/rankers/meta_field.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Fix tests

---------

Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-11-20 12:24:01 +01:00
Stefano Fiorucci
0ef06e72ff
fix: InMemoryDocumentStore - recreate Documents in the right way during embedding retrieval (#6354)
* do not recreate docs

* copy Documents

* recreate Document in the right way

* improve naming
2023-11-20 12:10:15 +01:00
ZanSara
9cee2f82c4
feat: extend write_documents to return the number of documents actually written in the document store (#6006)
* add typing and docstring

* reno

* Update releasenotes/notes/extend-write-documents-855ffc315974f03b.yaml

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-11-20 11:54:02 +01:00
Julian Risch
4ef2a680bb
feat: Add DocumentJoiner component 2.0 (#6105)
* draft DocumentJoiner

* implement merge and rrf

* draft end-to-end test with DocumentJoiner in hybrid doc search pipeline

* adjust for variadics Canals PR #122

* fix text_embedder input

* adapt to the new Document class

* adapt to new doc id

* specify documents input as Variadic in run method

* compare doc ids instead of full docs

* rename text_file_converter input to sources

* update docstring

* Update haystack/preview/components/routers/document_joiner.py

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

* Apply suggestions from docstring review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

* capitalize Documents and Retrievers in docstrings

* fix log message in test

---------

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
2023-11-20 10:56:56 +01:00
ZanSara
e905066458
feat: make InMemoryDocumentStore return the number of docs actually written (#6274)
* make InMemoryDocumentStore return the number of documents actually written

* add fixme

* reno

* add missing continue
2023-11-20 10:03:22 +01:00
x110
d03bffab8b
Promptnode timeout (#6282) 2023-11-19 16:32:09 +01:00
Silvano Cerza
9b11462bf8
refactor: Move tests for delete_documents from DocumentStoreBaseTests to separate class (#6336)
* Move tests for delete_documents from DocumentStoreBaseTests to separate class

* Move `filterable_docs` fixture from `DocumentStoreBaseTests` to separate mixin class (#6337)

* Move filterable_docs fixture from DocumentStoreBaseTests to separate mixin class

* refactor: Move generic `filter_documents` tests from `DocumentStoreBaseTests` to separate class (#6338)

* Move generic filter_documents tests from DocumentStoreBaseTests to separate class

* refactor: Move `filter_documents` tests with invalid filters from `DocumentStoreBaseTests` to separate class (#6339)

* Move filter_documents tests with invalid filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with equal filters from `DocumentStoreBaseTests` to separate class (#6340)

* Move filter_documents tests with equal filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with not equal filters from `DocumentStoreBaseTests` to separate class (#6341)

* Move filter_documents tests with not equal filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with in filters from `DocumentStoreBaseTests` to separate class (#6342)

* Move filter_documents tests with in filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with not in filters from `DocumentStoreBaseTests` to separate class (#6343)

* Move filter_documents tests with not in filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with greater than filters from `DocumentStoreBaseTests` to separate class (#6344)

* Move filter_documents tests with greater than filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with greater than equal filters from `DocumentStoreBaseTests` to separate class (#6345)

* Move filter_documents tests with greater than equal filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with less than filters from `DocumentStoreBaseTests` to separate class (#6346)

* Move filter_documents tests with less than filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with less than equal filters from `DocumentStoreBaseTests` to separate class (#6347)

* Move filter_documents tests with less than equal filters from DocumentStoreBaseTests to separate class

* Move `filter_documents` tests with simple logical filters from `DocumentStoreBaseTests` to separate class (#6348)

* Move filter_documents tests with simple logical filters from DocumentStoreBaseTests to separate class

* Move filter_documents tests with nested logical filters from DocumentStoreBaseTests to separate class (#6349)
2023-11-17 19:35:12 +01:00
Silvano Cerza
53b77dda6c
Move tests for write_documents from DocumentStoreBaseTests to separate class (#6334) 2023-11-17 19:25:16 +01:00
Silvano Cerza
326f51df9d
Move tests for count_document from DocumentStoreBaseTests to separate class (#6332) 2023-11-17 18:00:11 +01:00
Stefano Fiorucci
68be0d7f2c
refactor: improve Document representation (#6333)
* new repr

* reno
2023-11-17 17:49:00 +01:00
Silvano Cerza
5184481e50
refactor: Remove unecessary method to compare list of Documents in DocumentStoreBaseTests (#6324)
* Change Document.__eq__ to compare all fields

* Remove unecessary method to compare list of Documents in DocumentStoreBaseTests
2023-11-17 17:03:16 +01:00
ZanSara
e888852aec
Standardize TextFileToDocument (#6232)
* simplify textfiletodocument

* fix error handling and tests

* stray print

* reno

* streams->sources

* reno

* feedback

* test

* fix tests
2023-11-17 15:39:39 +01:00
Silvano Cerza
c26a932423
Change preview tests to run all tests except integration ones (#6325) 2023-11-17 15:33:43 +01:00
ZanSara
dfc1d452bb
feat: upgrade canals to 0.10.1 (#6309)
* upgrade canals

* reno

* trigger preview e2e

* bump canals

* fix decorator

* fix test

* test factory

* tests inmemory

* tests writer

* test audio

* tests builders

* tests caching

* tests embedders

* tests converters

* tests generators

* tests rankers

* tests retrievers

* fix pipeline and telemetry tests

* remove trigger
2023-11-17 14:46:23 +01:00
Vladimir Blagojevic
21bcfe76fb
Convert function call JSON payload to str (#6277) 2023-11-17 14:45:15 +01:00
Stefano Fiorucci
dd6e35d675
build: upgrade to transformers==4.35.2 (#6322)
* upgrade transformers to 4.35.2

* reno
2023-11-17 10:12:34 +01:00
Julian Risch
1c85e44156
test: Add langdetect installation to e2e tests (#6327)
* Add langdetect installation to e2e tests

* compare doc content and id only
2023-11-17 10:12:05 +01:00
Silvano Cerza
6dda6e5b2d
Change Document.__eq__ to compare all fields (#6323) 2023-11-16 17:17:43 +01:00
Massimiliano Pippi
ff3165b8b8
fix: fix un-flattening of metadata (#6318)
* fix un-flattening of metadata

* test should pass

* add relnote

* change policy: raise an error if both meta and keys are passed

* Update document.py

* support python 3.8

* adjust wording in the error message
2023-11-16 17:10:53 +01:00
Julian Risch
34ecff1d19
build: Upgrade openai-whisper and re-introduce audio extra (#6319)
* upgrade openai-whisper and re-introduce audio extra

* add audio extra to
2023-11-16 15:04:50 +01:00
Julian Risch
8b092a90c0
test: Add MetadataRouter to preprocessing pipeline in e2e test (#6321)
* add MetadataRouter to preprocessing pipeline

* replace mimetype check with language check
2023-11-16 11:22:37 +01:00
x110
c4cfe6cb90
fix: Load additional fields from SQUAD-format file to meta field for labels #5978 (#6301)
* Load additional fields from SQUAD-format file to meta field for labels

* added a test function

* rewritten test using pytest

* added release notes

* improve release note

* clean up test

---------

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
2023-11-16 10:44:51 +01:00
Stefano Fiorucci
c691412652
chore: improve the error messages of LazyImport (#6316)
* improve lazy import error messages

* revert changes to adaptive_model
2023-11-16 09:02:12 +01:00
Agnieszka Marzec
414cbcfd92
Update docstrings (#6297) 2023-11-15 17:54:10 +01:00
Stefano Fiorucci
f74f034549
fix pydoc config (#6313) 2023-11-15 15:32:29 +01:00
Vivek Silimkhan
f998bf4a4f
feat: add Amazon Bedrock support (#6226)
* Add Bedrock

* Update supported models for Bedrock

* Fix supports and add extract response in Bedrock

* fix errors imports

* improve and refactor supports

* fix install

* fix mypy

* fix pylint

* fix existing tests

* Added Anthropic Bedrock

* fix tests

* fix sagemaker tests

* add default prompt handler, constructor and supports tests

* more tests

* invoke refactoring

* refactor model_kwargs

* fix mypy

* lstrip responses

* Add streaming support

* bump boto3 version

* add class docstrings, better exception names

* fix layer name

* add tests for anthropic and cohere model adapters

* update cohere params

* update ai21 args and add tests

* support cohere command light model

* add tital tests

* better class names

* support meta llama 2 model

* fix streaming support

* more future-proof model adapter selection

* fix import

* fix mypy

* fix pylint for preview

* add tests for streaming

* add release notes

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

* fix format

* fix tests after msg changes

* fix streaming for cohere

---------

Co-authored-by: tstadel <60758086+tstadel@users.noreply.github.com>
Co-authored-by: tstadel <thomas.stadelmann@deepset.ai>
Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
2023-11-15 13:26:29 +01:00
Julian Risch
08ec492039
refactor!: Remove routing from DocumentLanguageClassifier and rename TextLanguageClassifier (#6307)
* remove routing from DocumentLanguageClassifier

* fix MetadataRouter typo
2023-11-15 13:10:07 +01:00
Julian Risch
5295b40def
docs: Reader returns top_k+1 answers if no_answer is enabled 2023-11-15 10:20:21 +01:00
Julian Risch
807cd6d139
chore: Add MetaFieldRanker to rankers __init__.py 2023-11-14 13:53:58 +01:00
Ashwin Mathur
4e4d5eb3e2
feat!: Remove unused query parameter from MetaFieldRanker (#6300)
* Remove unused query parameter from MetaFieldRanker

* Add release notes
2023-11-14 12:33:38 +01:00
Daria Fokina
34136382c1
docs: 2.0 API reference (#6262)
* docs: 2.0 API reference

* add builders and generators

* classifiers file path
2023-11-14 10:12:28 +01:00
Tuana Çelik
b8fdb880f9
Update docstring in html.py (#6279)
The explanation of 'sources' is inadequate especially because this is probably going to be most used with `LinkContentFetcher` that returns `List[ByteStream]`
2023-11-13 12:32:25 +01:00
Stefano Fiorucci
f708cf6056
refactor!: set scale_score default value to False (#6276)
* set default scale_score to False

* release note
2023-11-13 11:59:18 +01:00
Silvano Cerza
8e7ce208fc
Fix Document init when passing non existing fields (#6286)
* Fix Document init when passing non existing fields

* Update releasenotes/notes/fix-document-init-09c1cbb14202be7d.yaml

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

* Fix linting

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-11-13 11:42:42 +01:00
Tuana Çelik
bf637e9c7e
Update transformers_similarity.py (#6280)
Fixing the Document examples
2023-11-13 09:35:00 +01:00
Stefano Fiorucci
92a8704de4
mypy ignore specific errors (#6278) 2023-11-10 18:10:38 +01:00