1204 Commits

Author SHA1 Message Date
bogdankostic
796340e788
refactor: Adapt reader benchmarks (#5005) 2023-05-26 11:40:35 +02:00
bogdankostic
6e10fdab27
refactor: Adapt retriever benchmarks script (#5004)
* Generate eval result in separate method

* Adapt benchmarking utils

* Adapt running retriever benchmarks

* Adapt error message

* Raise error if file doesn't exist

* Raise error if path doesn't exist or is a directory
2023-05-25 15:39:02 +02:00
bogdankostic
c5f0f820cf
refactor: Adapt benchmarking utils (#5003)
* Adapt benchmarking utils

* Adapt error message

* Adapt doc store launcher registry

* Revert "Adapt doc store launcher registry"

This reverts commit e034936363dde760d393fe00cac998a54a0f5152.
2023-05-25 11:19:46 +02:00
Massimiliano Pippi
929b8d1fb0
ci: run Elasticsearch 8.6 in compatibility mode (#3853)
* bump ES version in CI

disable ssl

wait for service to start

set env vars

do not use choco to install ES

re-enable jobs deps

skip test on windows CI because of OOM

allocate more memory for ES

uniform ES installation and use default heap size

skip tests causing OOM

increase job timeout

restore memory limit for ES8

* Use latest elasticsearch version
2023-05-24 18:53:54 +02:00
Silvano Cerza
56d033e7e7
Add back hardcoded default templates (#4998) 2023-05-24 16:50:11 +02:00
bogdankostic
b85bc44c00
Mock request from prompt hub (#5011) 2023-05-24 12:23:49 +02:00
Silvano Cerza
524d2cba36
Fix CohereInvocationLayer _ensure_token_limit not returning resized (#4978)
prompt
2023-05-23 17:58:01 +02:00
Massimiliano Pippi
68924161df
chore: remove deprecated node PDFToTextOCRConverter (#4982)
* remove deprecated node

* remove related test
2023-05-23 16:55:54 +02:00
ZanSara
949b1b63b3
PromptHub integration in PromptNode (#4879)
* initial integration

* upgrade of prompthub

* fix get_prompt_template

* feedback

* add prompthub-py to dependencies

* tests

* mypy

* stray changes

* review feedback

* missing init

* fix test

* move logic in prompttemplate

* linting

* bugfixes

* fix unit tests

* fix cache

* simplify prompttemplate init

* remove unused function

* removing wrong params

* try remove all instances of prompt names

* more tests

* fix agent tests

* more tests

* fix tests

* pylint

* comma

* black

* fix test

* docstring

* review feedback

* review feedback

* fix mocks

* mypy

* fix mocks

* fix reference to missing templates

* feedback

* remove direct references to default template var

* tests

* Update haystack/nodes/prompt/prompt_node.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-05-23 15:22:58 +02:00
ZanSara
f80ae01174
LocalWhisperTranscriber (v2) (#4909)
* original component

* remove remote parts

* unit tests

* polish docstrings

* fix unit tests

* fix e2e tests

* pylint

* remove check

* review feedback

* add type: ignore

* improve tests

* test stream handling

* upgrade canals and improve tests

* pylint
2023-05-22 18:30:35 +02:00
ZanSara
516db4cb52
RemoteWhisperTranscriber (v2) (#4910)
* original-component

* stub

* fix implementation

* fix tests

* review feedback

* review feedback

* upgrade canals

* upgrade canals

* upgrade canals to fix pipeline test

* remove requests_with_retry

* feedback
2023-05-22 16:02:58 +02:00
Vladimir Blagojevic
068a967e5b
feat: HFInferenceEndpointInvocationLayer streaming support (#4819)
* HFInferenceEndpointInvocationLayer streaming support

* Small fixes

* Add unit test

* PR feedback

* Alphabetically sort params

* Convert PromptNode tests to HFInferenceEndpointInvocationLayer invoke tests

* Rewrite streaming with sseclient

* More PR updates

* Implement and test _ensure_token_limit

* Further optimize DefaultPromptHandler

* Fix CohereInvocationLayer mistypes

* PR feedback

* Break up unit tests, simplify

* Simplify unit tests even further

* PR feedback on unit test simplification

* Proper code identation under patch context manager

* More unit tests, slight adjustments

* Remove unrelated CohereInvocationLayer change

This reverts commit 82337151e8328d982f738e5da9129ff99350ea0c.

* Revert "Further optimize DefaultPromptHandler"

This reverts commit 606a761b6e3333f27df51a304cfbd1906c806e05.

* lg update

mostly full stops at the end of docstrings

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
Co-authored-by: Darja Fokina <daria.f93@gmail.com>
2023-05-22 14:45:53 +02:00
Silvano Cerza
9398183447
Simplify PromptNode generation_kwargs tests (#4975) 2023-05-22 14:28:08 +02:00
Fanli Lin
cd2ea4bc91
feat: enable passing generation_kwargs to the PromptNode in pipeline.run() (#4832)
* add generation_kwargs

* add documentation

* enable max_new_tokens  customization

* add code formatting

* add unit test

* fix formatting

* test with black

* add a new unit test

* remove doc and update tests

* unpack generation_kwargs

* ix comment

* update unit test

* remove generation_kwargs

* not pass `generation_kwargs`

* update tests

* add max_length

* fix formatting

* revert

* reformatting
2023-05-22 11:45:06 +02:00
Massimiliano Pippi
8228081e7a
chore: leftovers from removing knowledge graph support (#4974)
* leftovers from removing knowledge graph support

* more leftovers
2023-05-22 10:03:51 +02:00
Massimiliano Pippi
c6ea542b57
chore: remove BaseKnowledgeGraph (#4953)
* remove BaseKnowledgeGraph

* fix pylint
2023-05-21 10:42:02 +02:00
Massimiliano Pippi
4974bf7ab3
chore: remove deprecated MilvusDocumentStore (#4951)
* remove deprecated MilvusDocumentStore

* remove leftovers

* fix pylint
2023-05-19 16:37:38 +02:00
Massimiliano Pippi
85254fe9f6
leftover from merge conflict (#4962) 2023-05-19 16:10:26 +02:00
Vladimir Blagojevic
eb9d14faeb
fix: Adjust tool pattern to support multi-line inputs (#4801)
* Add support for multi line tool input

* Fix failing agent test, additional test_tools_manager.py tests

* Allow empty tool input, add more tests

* More unit tests

* String formatting

* Small str fix
2023-05-18 16:39:31 +02:00
Massimiliano Pippi
58acef77c4
avoid importing the weaviate client directly (#4945) 2023-05-18 16:08:53 +02:00
ZanSara
123ee55a5c
docstring (#4950) 2023-05-18 16:00:02 +02:00
Vladimir Blagojevic
5d7ee2e5e6
feat: Add max_tokens to BaseGenerator params (#4168)
* Add max_tokens to BaseGenerator params

* Make mypy happy

* Rebase and resolve conflicts

* Fix signature issues

* Update lg

* Add a mocked unit test method

* end-of-file-fixer corrected file

* Convert to unit test

* Mark test as integration

* make the test unit

---------

Co-authored-by: agnieszka-m <amarzec13@gmail.com>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-18 15:19:29 +02:00
Shukri
ad162f2e65
feat: Support authentication using AuthBearerToken and AuthClientCredentials in Weaviate (#4028)
* refactor: make the scope param configurable

the scope parameter is used when authenticating using
AuthClientPassword and AuthClientCredentials

* feat: add support for AuthClientCredentials

add support for authenticating using the OIDC Client Credentials
authentication flow

* feat: add support for AuthBearerToken

Add support for authenticating using OIDC and bearer tokens

* Update lg

* refactor how client is built

Signed-off-by: hsm207 <hsm207@users.noreply.github.com>

* unit test the auth methods

Signed-off-by: hsm207 <hsm207@users.noreply.github.com>

* Update test_weaviate.py

* revert formatting change

* Fix type hints

---------

Signed-off-by: hsm207 <hsm207@users.noreply.github.com>
Co-authored-by: John Doe <johndoe@example.com>
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-18 10:17:11 +02:00
Massimiliano Pippi
3ea784464a
add test case for #4929 (#4936) 2023-05-18 09:12:03 +02:00
Julian Risch
8cfeed095d
build: Remove mmh3 dependency (#4896)
* build: Remove mmh3 dependency

* resolve circular import

* pylint

* make mmh3.py sibling of schema.py

* pylint import order

* pylint

* undo example changes

* increase coverage in modeling module

* increase coverage further

* rename new unit tests
2023-05-17 21:31:08 +02:00
bogdankostic
df46e7fadd
fix: Use AutoTokenizer instead of DPR specific tokenizer (#4898)
* Use AutoTokenizer instead of DPR specific tokenizer

* Adapt TableTextRetriever

* Adapt tests

* Adapt tests
2023-05-17 18:54:34 +02:00
Vladimir Blagojevic
9d52998b25
feat: Add conversational agent (#4931) 2023-05-17 15:19:09 +02:00
tstadel
7625829684
fix: EvaluationResult serialization changes dataframes (#4906)
* fix nan and index values

* add test

* make test for None values after evalresult read explicit
2023-05-16 16:03:09 +02:00
Vladimir Blagojevic
37cadd702a
fix: Make sure summary memory is cumulative (#4932)
* Fix summary memory not being cummulative

* PR feedback - Julian
2023-05-16 13:35:19 +02:00
Stefano Fiorucci
6e0000732d
feat: add BLIP support in TransformersImageToText (#4912)
* add blip support

* fix typo

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-05-16 10:57:41 +02:00
Vladimir Blagojevic
4c9843017c
feat: Add agent memory (#4829) 2023-05-15 18:08:44 +02:00
Ben Heckmann
099d0deb86
fix: Dynamic max_answers for SquadProcessor (fixes IndexError when max_answers is less than the number of answers in the dataset) (#4817)
* #4320 implemented dynamic max_answers for SquadProcessor, fixed IndexError when max_answers is less than the number of answers in the dataset

* #4320 added two unit tests for dataset_from_dicts testing default and manual max_answers

* apply suggestions from code review

Co-authored-by: bogdankostic <bogdankostic@web.de>

* simplify comment, fix mypy & pylint errors, fix old test

* adjust max_answers to each dataset individually

---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-05-15 14:34:23 +02:00
ZanSara
8fbfca9ebb
fix: Document v2 JSON serialization (#4863)
* fix json serialization

* add missing markers

* pylint

* fix decoder bug

* pylint

* add some more tests

* linting & windows

* windows

* windows

* windows paths again
2023-05-15 11:39:04 +02:00
ZanSara
bffe2d8c19
add base test class (#4908) 2023-05-15 10:36:55 +02:00
Farzad E
6eb251d1f0
fix: Support for gpt-4-32k (#4825)
* Add step to loook up tokenizers by prefix in openai_utils

* Updated tiktoken min version + openai_utils test

* Added test case for GPT-4 and Azure model naming

* Broken down tests

* Added default case

---------

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-05-12 19:02:12 +02:00
Vladimir Blagojevic
73380b194a
feat: Add Cohere PromptNode invocation layer (#4827)
* Add CohereInvocationLayer
---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-05-12 17:50:09 +02:00
ZanSara
618699eb52
fix: improve Document comparison (v2) (#4860)
* don't compare on content directly, use id as proxy

* stray change

* add more tests

* fix tests

* pylint

* black

* review feedback

* fix tests
2023-05-11 18:28:56 +02:00
Silvano Cerza
98947e4c3c
feat: Add Anthropic invocation layer (#4818)
* feat: Add Anthropic Claude Invocation Layer

* feat: Add AnthropicClaude Invocation Layer

* fix: Permission changes

* fix: Permission changes

* Move anthropic utils in anthropic invocation layer file

* Rework method to post data

* Simplify invoke

* Simplify supports classmethod

* Remove unnecessary functions

* Use always same tokenizer

* Add module import

* Rename some members and kwargs

* Add tests

* Fix _post not handling HTTPError

* Fix handling of streamed response

* Fix kwargs handling

* Update tests

* Update supports to be generic

* Fix failing test

* Use correct tokenizer and fix tests

* Update lg

* Fix mypy issue

* Move requests-cache from dev to base dependencies

* Fix failing test

* Handle all stop words use cases

---------

Co-authored-by: recrudesce <recrudesce@gmail.com>
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
2023-05-11 10:14:33 +02:00
ZanSara
3a6db68408
feat: allow filtering documents on all fields (v2) (#4773)
* extend tests

* remove stray test

* pylint

* mypy

* review feedback

* fix tests

* fix last tests

* remove comment

* remove print statement

* pylint

* add flatten test

* remove direct acces/ direct write in docstore tests

* fix tests
2023-05-10 16:33:47 +02:00
Sebastian
eff420cce0
test: Update unit tests for schema (#4835)
* Updated text_label tests to match tabel_label tests. Also added answer text as part of the Answer.__eq__ comparison.

* Updated text document unit tests to match ones from table docs

* Converting text answer unit tests to match table answer

* Update some document tests

* Minor update

* Separating unit tests
2023-05-10 16:16:45 +02:00
ZanSara
9cb153d0f4
fix: add unit markers to several v2 tests (#4851)
* add markers

* remove stray marker
2023-05-10 13:46:13 +02:00
Silvano Cerza
f12e5a0127
fix: Fix missing error in openai_request retry strategy (#4802)
* Fix missing error in openai_request retry strategy

* Correctly handle OpenAIUnauthorizedError

Co-authored-by: bogdankostic <bogdankostic@web.de>

---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-05-10 10:31:07 +02:00
ZanSara
c734c58b4b
skip flaky test (#4846) 2023-05-09 20:26:59 +02:00
Sebastian
707f1c3546
Add modeling to unit tests so it we can get coverage for that (#4809)
* Add modeling to unit tests so it we can get coverage for that

* fix unit tests

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-08 19:05:21 +02:00
bogdankostic
5b2ef2afd6
Revert "refactor!: Deprecate name param in PromptTemplate and introduce template_name instead (#4810)" (#4834)
This reverts commit f660f41c0615e6b3064ef3e321f1e5a295fafc1b.
2023-05-08 11:31:04 +02:00
ZanSara
6e982e9283
fix: preserve root_node in JoinNode's output (#4820)
* preserve root_node and add tests

* Added if statement to fix failing tests

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
2023-05-08 10:17:36 +02:00
bogdankostic
f660f41c06
refactor!: Deprecate name param in PromptTemplate and introduce template_name instead (#4810)
* Deprecate name parameter

* Adapt existing tests and uses of PromptTemplate

* Move parameter `name` to end

* Adapt existing tests

* lg update

---------

Co-authored-by: Darja Fokina <daria.f93@gmail.com>
2023-05-08 10:12:29 +02:00
Silvano Cerza
705a2c025f
Update preview Pipelines following Canals changes (#4821) 2023-05-05 19:47:32 +02:00
bogdankostic
43509c88bf
fix: Add support for _split_overlap meta to Pinecone and dict metadata in general to Weaviate (#4805)
* Add support for dicts to Weaviate

* Add support for _split_overlap to Pinecone

* Add tests

* Fix Pylint

* Fix Pylint

* Fix test

* Implement PR feedback
2023-05-05 11:20:21 +02:00
Vladimir Blagojevic
8091ced8d5
refactor: Extract ToolsManager, add it to Agent by composition (#4794)
* Extract ToolsManager, add it to Agent by the composition
* PR feedback Massi
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: Darja Fokina <daria.f93@gmail.com>
2023-05-03 16:45:40 +02:00