3115 Commits

Author SHA1 Message Date
Vladimir Blagojevic
edc328e21a
Remove instruction following warning (#4942) 2023-05-18 15:55:18 +02:00
Vladimir Blagojevic
5d7ee2e5e6
feat: Add max_tokens to BaseGenerator params (#4168)
* Add max_tokens to BaseGenerator params

* Make mypy happy

* Rebase and resolve conflicts

* Fix signature issues

* Update lg

* Add a mocked unit test method

* end-of-file-fixer corrected file

* Convert to unit test

* Mark test as integration

* make the test unit

---------

Co-authored-by: agnieszka-m <amarzec13@gmail.com>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-18 15:19:29 +02:00
Daria Fokina
401520b1d2
web.py docstring update (#4921)
Corrected spelling and added GoogleAPI
2023-05-18 14:20:51 +02:00
Massimiliano Pippi
df55ec5e61
Pin Weaviate client (#4952) 2023-05-18 12:22:16 +02:00
Shukri
ad162f2e65
feat: Support authentication using AuthBearerToken and AuthClientCredentials in Weaviate (#4028)
* refactor: make the scope param configurable

the scope parameter is used when authenticating using
AuthClientPassword and AuthClientCredentials

* feat: add support for AuthClientCredentials

add support for authenticating using the OIDC Client Credentials
authentication flow

* feat: add support for AuthBearerToken

Add support for authenticating using OIDC and bearer tokens

* Update lg

* refactor how client is built

Signed-off-by: hsm207 <hsm207@users.noreply.github.com>

* unit test the auth methods

Signed-off-by: hsm207 <hsm207@users.noreply.github.com>

* Update test_weaviate.py

* revert formatting change

* Fix type hints

---------

Signed-off-by: hsm207 <hsm207@users.noreply.github.com>
Co-authored-by: John Doe <johndoe@example.com>
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-18 10:17:11 +02:00
Massimiliano Pippi
3ea784464a
add test case for #4929 (#4936) 2023-05-18 09:12:03 +02:00
Julian Risch
8cfeed095d
build: Remove mmh3 dependency (#4896)
* build: Remove mmh3 dependency

* resolve circular import

* pylint

* make mmh3.py sibling of schema.py

* pylint import order

* pylint

* undo example changes

* increase coverage in modeling module

* increase coverage further

* rename new unit tests
2023-05-17 21:31:08 +02:00
bogdankostic
df46e7fadd
fix: Use AutoTokenizer instead of DPR specific tokenizer (#4898)
* Use AutoTokenizer instead of DPR specific tokenizer

* Adapt TableTextRetriever

* Adapt tests

* Adapt tests
2023-05-17 18:54:34 +02:00
Sebastian
34b7d1edb0
Small fix to PromptTemplate API docs (#4870) 2023-05-17 18:51:21 +02:00
Vladimir Blagojevic
9d52998b25
feat: Add conversational agent (#4931) 2023-05-17 15:19:09 +02:00
yuanwu2017
ca68601ec7
fix: shaper exception when retriever return 0 docs. (#4929)
* When retriever retrieves 0 documents from the documentStore,
  shaper will raise an exception.

Signed-off-by: root <root@aia-opa-clx-4017.jf.intel.com>
Co-authored-by: root <root@aia-opa-clx-4017.jf.intel.com>
2023-05-17 08:48:11 +02:00
tstadel
7625829684
fix: EvaluationResult serialization changes dataframes (#4906)
* fix nan and index values

* add test

* make test for None values after evalresult read explicit
2023-05-16 16:03:09 +02:00
Vladimir Blagojevic
37cadd702a
fix: Make sure summary memory is cumulative (#4932)
* Fix summary memory not being cummulative

* PR feedback - Julian
2023-05-16 13:35:19 +02:00
Stefano Fiorucci
6e0000732d
feat: add BLIP support in TransformersImageToText (#4912)
* add blip support

* fix typo

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-05-16 10:57:41 +02:00
Vladimir Blagojevic
4c9843017c
feat: Add agent memory (#4829) 2023-05-15 18:08:44 +02:00
Julian Risch
d4bbde2d9d
build: Upgrade transformers to 4.29.1 (#4886)
* Upgrade transformers to 4.29.0

* Upgrade transformers to 4.29.1
2023-05-15 17:11:17 +02:00
Ben Heckmann
099d0deb86
fix: Dynamic max_answers for SquadProcessor (fixes IndexError when max_answers is less than the number of answers in the dataset) (#4817)
* #4320 implemented dynamic max_answers for SquadProcessor, fixed IndexError when max_answers is less than the number of answers in the dataset

* #4320 added two unit tests for dataset_from_dicts testing default and manual max_answers

* apply suggestions from code review

Co-authored-by: bogdankostic <bogdankostic@web.de>

* simplify comment, fix mypy & pylint errors, fix old test

* adjust max_answers to each dataset individually

---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-05-15 14:34:23 +02:00
ZanSara
8fbfca9ebb
fix: Document v2 JSON serialization (#4863)
* fix json serialization

* add missing markers

* pylint

* fix decoder bug

* pylint

* add some more tests

* linting & windows

* windows

* windows

* windows paths again
2023-05-15 11:39:04 +02:00
ZanSara
bffe2d8c19
add base test class (#4908) 2023-05-15 10:36:55 +02:00
Farzad E
6eb251d1f0
fix: Support for gpt-4-32k (#4825)
* Add step to loook up tokenizers by prefix in openai_utils

* Updated tiktoken min version + openai_utils test

* Added test case for GPT-4 and Azure model naming

* Broken down tests

* Added default case

---------

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-05-12 19:02:12 +02:00
bogdankostic
179e9cea08
feat: Send pipeline config hash every 100 runs (#4884)
* Add since_last_run property

* Revert "Add since_last_run property"

This reverts commit c1c907ef58a696a97d964fb9c45fbee0c80365aa.

* Send pipeline config hash for each run

* Send event every 100 runs

* Merge branch 'main' into telemetry_since_last_run

* PR review

* Move constant
2023-05-12 18:51:26 +02:00
Vladimir Blagojevic
73380b194a
feat: Add Cohere PromptNode invocation layer (#4827)
* Add CohereInvocationLayer
---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-05-12 17:50:09 +02:00
Silvano Cerza
7e2b824bea
Add Datadog in Who Uses Haystack README.md section (#4894) 2023-05-12 11:37:22 +02:00
Massimiliano Pippi
428096733d
ci: add a job to vet license of direct dependencies only (#4885)
* add conversion script

* run job in CI

* typo

* invoke python

* install toml

* fix pylint error

* more exclusions

* add toml to dev dependencies

* fix exclusions list

* fix mypy and remove test clause
2023-05-12 11:20:48 +02:00
Massimiliano Pippi
d322beed6c
build: do not install 'dev' extras with 'all' (#4888)
* do not install 'dev' with 'all'

* some fixes around
2023-05-11 19:24:47 +02:00
ZanSara
618699eb52
fix: improve Document comparison (v2) (#4860)
* don't compare on content directly, use id as proxy

* stray change

* add more tests

* fix tests

* pylint

* black

* review feedback

* fix tests
2023-05-11 18:28:56 +02:00
Silvano Cerza
6c84a05d98
Upload coverage only if all unit tests pass (#4874) 2023-05-11 14:29:44 +02:00
Silvano Cerza
98947e4c3c
feat: Add Anthropic invocation layer (#4818)
* feat: Add Anthropic Claude Invocation Layer

* feat: Add AnthropicClaude Invocation Layer

* fix: Permission changes

* fix: Permission changes

* Move anthropic utils in anthropic invocation layer file

* Rework method to post data

* Simplify invoke

* Simplify supports classmethod

* Remove unnecessary functions

* Use always same tokenizer

* Add module import

* Rename some members and kwargs

* Add tests

* Fix _post not handling HTTPError

* Fix handling of streamed response

* Fix kwargs handling

* Update tests

* Update supports to be generic

* Fix failing test

* Use correct tokenizer and fix tests

* Update lg

* Fix mypy issue

* Move requests-cache from dev to base dependencies

* Fix failing test

* Handle all stop words use cases

---------

Co-authored-by: recrudesce <recrudesce@gmail.com>
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
2023-05-11 10:14:33 +02:00
ZanSara
3a6db68408
feat: allow filtering documents on all fields (v2) (#4773)
* extend tests

* remove stray test

* pylint

* mypy

* review feedback

* fix tests

* fix last tests

* remove comment

* remove print statement

* pylint

* add flatten test

* remove direct acces/ direct write in docstore tests

* fix tests
2023-05-10 16:33:47 +02:00
Massimiliano Pippi
c619aa29ec
ci: add new license checker (#4779)
* try

* add exclusions

* fix vanilla distribution

* use different requirements files

* fix comments and file name

* try with a recent version of pip

* use cpu version of torch

* try

* again

* exclude nvidia libraries

* revert old change

* send report to FOSSA

* add gpu section

* display job names

* remove FOSSA check

* send complete report to FOSSA

* removed FIXME
2023-05-10 16:33:08 +02:00
Sebastian
eff420cce0
test: Update unit tests for schema (#4835)
* Updated text_label tests to match tabel_label tests. Also added answer text as part of the Answer.__eq__ comparison.

* Updated text document unit tests to match ones from table docs

* Converting text answer unit tests to match table answer

* Update some document tests

* Minor update

* Separating unit tests
2023-05-10 16:16:45 +02:00
ZanSara
6a7d31fb5b
chore: remove optional imports in v2 (#4855)
* remove optional imports in v2

* unused import
2023-05-10 14:02:18 +02:00
ZanSara
9cb153d0f4
fix: add unit markers to several v2 tests (#4851)
* add markers

* remove stray marker
2023-05-10 13:46:13 +02:00
ZanSara
611b09b6c0
pin canals (#4853) 2023-05-10 13:45:57 +02:00
Silvano Cerza
06193e08b1
Add missing unit tests topics to coverage upload step (#4873) 2023-05-10 12:51:52 +02:00
Daria Fokina
7ef6bd8373
[Docs] Hide api classes in prompt_node (#4869)
* hide api classes in prompt_node

* add jsonconverter to docs
2023-05-10 10:56:46 +02:00
Silvano Cerza
f12e5a0127
fix: Fix missing error in openai_request retry strategy (#4802)
* Fix missing error in openai_request retry strategy

* Correctly handle OpenAIUnauthorizedError

Co-authored-by: bogdankostic <bogdankostic@web.de>

---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-05-10 10:31:07 +02:00
ZanSara
c734c58b4b
skip flaky test (#4846) 2023-05-09 20:26:59 +02:00
Daria Fokina
dbbdc5464a
docs: fix Prompt_Node and Pipelines API reference (#4858)
* fix: Prompt_Node and Pipelines API reference

* deleted invocation layers
2023-05-09 19:04:55 +02:00
ZanSara
28463e38e5
multi-os dep checker (#4845) 2023-05-09 11:46:53 +02:00
Sebastian
707f1c3546
Add modeling to unit tests so it we can get coverage for that (#4809)
* Add modeling to unit tests so it we can get coverage for that

* fix unit tests

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-08 19:05:21 +02:00
ZanSara
28260c5c3f
feat: introduce generalimport (#4662)
* introduce generalimport

* pylint

* fix optional deps typing for schema

* leftover

* typo

* typing with faiss

* make Base generation optional too

* handle sqlalchemy

* (almost) all import are optional

* TO REMOVE hijacking CI for tests

* some deps are actually needed

* get feature branch in CI

* get feature branch in CI

* fix array_equal

* pylint

* pandas also required

* improve imports.yml

* fix SquadData

* fix SquadData again

* generalimport imports list

* Update haystack/utils/openai_utils.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Update haystack/utils/openai_utils.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* review feedback

* remove todos

* reference main release

* pylint

* circular import

* review feedback

* move is_imported in init

* pylint

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-05-08 15:20:10 +02:00
bogdankostic
5b2ef2afd6
Revert "refactor!: Deprecate name param in PromptTemplate and introduce template_name instead (#4810)" (#4834)
This reverts commit f660f41c0615e6b3064ef3e321f1e5a295fafc1b.
2023-05-08 11:31:04 +02:00
ZanSara
6e982e9283
fix: preserve root_node in JoinNode's output (#4820)
* preserve root_node and add tests

* Added if statement to fix failing tests

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
2023-05-08 10:17:36 +02:00
bogdankostic
f660f41c06
refactor!: Deprecate name param in PromptTemplate and introduce template_name instead (#4810)
* Deprecate name parameter

* Adapt existing tests and uses of PromptTemplate

* Move parameter `name` to end

* Adapt existing tests

* lg update

---------

Co-authored-by: Darja Fokina <daria.f93@gmail.com>
2023-05-08 10:12:29 +02:00
Philip May
2ff8b0ddd0
fix: str issues in squad_to_dpr (#4826)
* fix #4754

* fix #4753

* run black formatting

---------

Co-authored-by: Julian Risch <julian.risch@deepset.ai>
2023-05-07 19:50:23 +02:00
Silvano Cerza
705a2c025f
Update preview Pipelines following Canals changes (#4821) 2023-05-05 19:47:32 +02:00
bogdankostic
43509c88bf
fix: Add support for _split_overlap meta to Pinecone and dict metadata in general to Weaviate (#4805)
* Add support for dicts to Weaviate

* Add support for _split_overlap to Pinecone

* Add tests

* Fix Pylint

* Fix Pylint

* Fix test

* Implement PR feedback
2023-05-05 11:20:21 +02:00
Massimiliano Pippi
d8dc0d7403
chore: move custom linter to a separate package (#4790)
* move custom linter to its own package

* install the custom linter

* fix formatting

* drop python 3.7
2023-05-04 15:49:26 +02:00
Vladimir Blagojevic
8091ced8d5
refactor: Extract ToolsManager, add it to Agent by composition (#4794)
* Extract ToolsManager, add it to Agent by the composition
* PR feedback Massi
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: Darja Fokina <daria.f93@gmail.com>
2023-05-03 16:45:40 +02:00