2176 Commits

Author SHA1 Message Date
ZanSara
28463e38e5
multi-os dep checker (#4845) 2023-05-09 11:46:53 +02:00
Sebastian
707f1c3546
Add modeling to unit tests so it we can get coverage for that (#4809)
* Add modeling to unit tests so it we can get coverage for that

* fix unit tests

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-08 19:05:21 +02:00
ZanSara
28260c5c3f
feat: introduce generalimport (#4662)
* introduce generalimport

* pylint

* fix optional deps typing for schema

* leftover

* typo

* typing with faiss

* make Base generation optional too

* handle sqlalchemy

* (almost) all import are optional

* TO REMOVE hijacking CI for tests

* some deps are actually needed

* get feature branch in CI

* get feature branch in CI

* fix array_equal

* pylint

* pandas also required

* improve imports.yml

* fix SquadData

* fix SquadData again

* generalimport imports list

* Update haystack/utils/openai_utils.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Update haystack/utils/openai_utils.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* review feedback

* remove todos

* reference main release

* pylint

* circular import

* review feedback

* move is_imported in init

* pylint

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-05-08 15:20:10 +02:00
bogdankostic
5b2ef2afd6
Revert "refactor!: Deprecate name param in PromptTemplate and introduce template_name instead (#4810)" (#4834)
This reverts commit f660f41c0615e6b3064ef3e321f1e5a295fafc1b.
2023-05-08 11:31:04 +02:00
ZanSara
6e982e9283
fix: preserve root_node in JoinNode's output (#4820)
* preserve root_node and add tests

* Added if statement to fix failing tests

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
2023-05-08 10:17:36 +02:00
bogdankostic
f660f41c06
refactor!: Deprecate name param in PromptTemplate and introduce template_name instead (#4810)
* Deprecate name parameter

* Adapt existing tests and uses of PromptTemplate

* Move parameter `name` to end

* Adapt existing tests

* lg update

---------

Co-authored-by: Darja Fokina <daria.f93@gmail.com>
2023-05-08 10:12:29 +02:00
Philip May
2ff8b0ddd0
fix: str issues in squad_to_dpr (#4826)
* fix #4754

* fix #4753

* run black formatting

---------

Co-authored-by: Julian Risch <julian.risch@deepset.ai>
2023-05-07 19:50:23 +02:00
Silvano Cerza
705a2c025f
Update preview Pipelines following Canals changes (#4821) 2023-05-05 19:47:32 +02:00
bogdankostic
43509c88bf
fix: Add support for _split_overlap meta to Pinecone and dict metadata in general to Weaviate (#4805)
* Add support for dicts to Weaviate

* Add support for _split_overlap to Pinecone

* Add tests

* Fix Pylint

* Fix Pylint

* Fix test

* Implement PR feedback
2023-05-05 11:20:21 +02:00
Massimiliano Pippi
d8dc0d7403
chore: move custom linter to a separate package (#4790)
* move custom linter to its own package

* install the custom linter

* fix formatting

* drop python 3.7
2023-05-04 15:49:26 +02:00
Vladimir Blagojevic
8091ced8d5
refactor: Extract ToolsManager, add it to Agent by composition (#4794)
* Extract ToolsManager, add it to Agent by the composition
* PR feedback Massi
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: Darja Fokina <daria.f93@gmail.com>
2023-05-03 16:45:40 +02:00
Silvano Cerza
4cc69eeb29
Fix release_docs.py to create docs with correct version (#4803) 2023-05-03 15:04:50 +02:00
Silvano Cerza
9b67611169
Add others folder to unit test job (#4800) 2023-05-03 10:47:21 +02:00
Sebastian
a67ca289db
refactor: Update schema objects to handle Dataframes in to_{dict,json} and from_{dict,json} (#4747)
* Adding support for table Documents when serializing Labels in Haystack

* Fix table label equality test

* Add serialization support and __eq__ support for table answers

* Made convenience functions for converting dataframes. Added some TODOs. Epxanded schema tests for table labels. Updated Multilabel to not convert Dataframes into strings.

* get Answer and Label to_json working with DataFrame

* Fix from_dict method of Label

* Use Dict and remove unneccessary if check

* Using pydantic instead of builtins for type detection

* Update haystack/schema.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Update haystack/schema.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Update haystack/schema.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Separated table label equivalency tests and added pytest.mark.unit


* Added unit test for _dict_factory

* Using more descriptive variable names

* Adding json files to test to_json and from_json functions

* Added sample files for tests

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-05-03 09:42:07 +02:00
ZanSara
a9ec954c45
bug: fix filtering in MemoryDocumentStore (v2) (#4768)
* fix filtering bug

* pylint

* improve asserts
2023-05-03 09:33:12 +02:00
Pouyan
75ff768c21
Pouyanpi/feat/search engine/providers/google api (#4722)
* feat: implement google api search engine provider

Signed-off-by: Pouyan <prezakhanipr@gmail.com>

---------

Signed-off-by: Pouyan <prezakhanipr@gmail.com>
2023-05-02 17:09:17 +02:00
yuanwu2017
c88bc19791
fix: load the local finetuning model from pipeline YAML (#4729) (#4760)
If using the local model in pipeline YAML. The PromptModel cannot select
the HFLocalInvocationLayer, because of the get_task cannot support the
offline model.

*Local model usage:
  add the task_name parameter in model_kwargs for local model. for
  example text-generation or text2text-generation.

- name: PModel
  type: PromptModel
  params:
    model_name_or_path: /local_model_path
    model_kwargs:
      task_name: text-generation
- name: Prompter
  params:
    model_name_or_path: PModel
    default_prompt_template: question-answering
  type: PromptNode

Signed-off-by: yuanwu <yuan.wu@intel.com>
2023-05-02 17:04:42 +02:00
duffn
479092e3c1
bug: (rest_api) remove full logging of overwritten env variables (#4791)
* bug: (rest_api) remove logging of overwritten env variables

* Update haystack/pipelines/config.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Update test

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-05-02 16:48:19 +02:00
Vladimir Blagojevic
1e9f4c1d50
feat: Add HF local runtime token streaming support (#4652)
* Add HF local runtime token streaming support

* Add stream and stream_handler as model kwargs

* Improve HF streaming unit tests
2023-05-02 12:50:20 +02:00
Mayank Jobanputra
dcf3ddddff
Added deprecation tests for seq2seq generator and RAG Generator (#4782) 2023-05-02 13:30:22 +05:30
Mayank Jobanputra
896eb6a2ea
chore: fixed reader loading test for hf-hub starting 0.14.0 (#4607)
* fixed test base for hub 0.13.3

* check if test succeed from branch

* 2nd check if test succeed from branch

* removed dependency changes

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-02 08:22:44 +02:00
ZanSara
b60d9a2cbf
test: move several modeling tests in e2e/ (#4308)
* no dpr test seems worth mocking

* move distillation tests

* pylint

* mypy

* pylint

* move feature_extraction tests as well

* move feature_extraction tests as well

* merge feature extractor suites

* get_language_model tests and adaptive model tests

* duplicate test

* moving fixtures

* mypy

* mypy-again

* trigger

* un-mock integration test

* review feedback

* feedback

* pylint
2023-04-28 17:08:41 +02:00
Bilge Yücel
5a17a40685
fix: update ImportError for 'metrics' dependency (#4778) 2023-04-28 13:42:51 +02:00
Silvano Cerza
645a5fe5ba
ci: Add coverage tracking with Coveralls (#4772)
* Format tests.yml properly

* Add pytest-cov dependency

* Add coverage in unit tests

* Ignore cov.info

* Change report format

* Unignore cov.info
2023-04-28 11:59:09 +02:00
Vladimir Blagojevic
dcaf3002f1
fix: SentenceTransformersRanker's predict_batch returns wrong number of documents (#4756)
* Fix SentenceTransformersRanker spredict_batch returning wrong number of documents

* Julian's feedback
2023-04-27 15:24:39 +02:00
Vladimir Blagojevic
c9a415ec8d
refactor: Make agent test more robust (#4767)
* Add more examplars to lower test failure rate

* Easier agent run test, more robust, consistently passing
2023-04-27 14:53:15 +02:00
Vladimir Blagojevic
aebc22d27e
Upgrade transformers to 4.28.1 (#4665)
* Upgrade to transformers 4.28.1

* Commenting out failing piece of test

* trailing-whitespace

* Adjust regex for error match - it changed between releases

* Remove RAG tests failing with transformers update
2023-04-27 12:55:21 +02:00
bogdankostic
c7a20d68d2
fix: Add separate query method for OpenSearchDocumentStore (#4764)
* Add separate query method for OpenSearchDocumentStore

* Convert integration test to unit test + add separate tests for OpenSearch
2023-04-26 21:58:33 +02:00
Vladimir Blagojevic
41b6e33f64
Enhance the error logging in PromptTemplate variable resolution (#4730)
* Enhance the error logging in PromptTemplate variable resolution

* Revert change Daria made

* Silvano PR feedback
2023-04-26 18:09:20 +02:00
Darja Fokina
2f7104704a
fix: README latest and main installation (#4741)
Fixing README to explain how to install from main branch
2023-04-26 15:08:05 +02:00
Vladimir Blagojevic
3fefc475b4
fix: Deprecate Seq2SeqGenerator and RAGenerator (#4745)
* Deprecate Seq2SeqGenerator

* changed the warning to include suggestion

* Added example and msg to API reference docs

* Added RAG deprecation

* renamed name to adapt to naming conven

* update docstrings

---------

Co-authored-by: Mayank Jobanputra <mayankjobanputra@gmail.com>
Co-authored-by: Darja Fokina <daria.f93@gmail.com>
2023-04-26 13:59:35 +02:00
tstadel
9cbe9e0949
fix: recursion of death while loading PromptTemplate from yaml (#4691)
* fix recursion of death when deserializing prompttemplate

* add test

* set api_key

* fix test

* add generic test

* work in feedback on tests

---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-04-26 13:56:51 +02:00
Vladimir Blagojevic
650e1a1a6f
fix: gpt-3.5-turbo is an agent streaming model (#4673)
* gpt-3.5 is also agent streaming model

* Add more streaming capable models

* Add end-of-file-fixer

* List full model names for clarity
2023-04-26 13:56:24 +02:00
s_teja
d033a086d0
fix: loads local HF Models in PromptNode pipeline (#4670)
* bug: fix load local HF Models in PromptNode pipeline

* Update hugging_face.py

remove duplicate validator

* update: black formatted

* update: update doc string, replace pop with get

* test HFLocalInvocationLayer with local model
2023-04-26 13:10:02 +02:00
bogdankostic
0d6fba14fd
Revert "fix: Log 'Observation' on new line (#4704)" (#4751)
This reverts commit 63f24cb1f37455a63dfd4b0c6f6c797fc8077aae.
2023-04-26 12:17:58 +02:00
Massimiliano Pippi
b54fdd3fa0
docs: add deprecation notes to docstrings (#4708) 2023-04-26 12:16:43 +02:00
ZanSara
1b57b96210
refactor!: extract elasticsearch (#4668)
* extract elasticsearch

* update pyproject.toml

* make more import optional

* move MockBaseRetriever in conftest

* install es in the es integration tests
2023-04-26 10:14:20 +02:00
bogdankostic
91b775bf43
Execute pipelines and utils unit tests in CI (#4749) 2023-04-26 10:00:52 +02:00
recrudesce
38768bffdf
fix: Tiktoken does not support Azure gpt-35-turbo (#4739)
* force support for gpt-35-turbo

Cos Tiktoken doesn't support it yet - see https://github.com/openai/tiktoken/pull/72

* Update openai_utils.py

* Appeasing the linting gods

Why hast thou forsaken me ?

* Remove trailing whitespace

* chg: remove redundant elif block
2023-04-25 16:43:24 +02:00
Wang, Yi
2be1a68fce
fix: Allow to set num_beams in HFInvocationLayer (#4731)
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-04-25 16:08:06 +02:00
github-actions[bot]
7fa3591f5f
Update unstable version (#4740)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-04-24 14:07:50 +02:00
bogdankostic
7db025a97b
Update weaviate-client (#4715) 2023-04-20 17:54:55 +02:00
recrudesce
473152eb05
feat: Add AzureChatGPT Capability using new InvocationLayer style (#4675) 2023-04-20 16:27:07 +02:00
Tuana Çelik
4cc416236d
chore: Updating readme (#4714)
Adding another name in our 'who uses' section
2023-04-20 11:28:38 +02:00
Zoltan Fedor
49d548ef10
fix: Fixing the Weaviate BM25 query builder bug (#4703) 2023-04-20 09:56:49 +02:00
Tuana Çelik
63f24cb1f3
fix: Log 'Observation' on new line (#4704) 2023-04-20 09:53:08 +02:00
bogdankostic
3d3b79986f
docs: Adapt Shaper docstrings regarding dropping metadata (#4655) 2023-04-19 13:40:53 +02:00
Sebastian
8d9136bad4
feat: Implementation of Table Cell Proposal (#4616)
* Starting adding support for TableCell

* Update tests to use row and col

* Added schema test to check to_dict and from_dict works for Table documents. Also updated Doc.__eq__ to work for tables.

* Update eval test to use TableCell

* Added more schema tests for table docs, labels and answers.

* Add boolean to toggle between Span and TableCell

* Add deprecation message

* Test that table answers work as responses in the rest API

---------

Co-authored-by: agnieszka-m <amarzec13@gmail.com>
2023-04-19 13:14:49 +02:00
Darja Fokina
ec7fc4aa0b
docs: add web retriever to api docs (#4699) 2023-04-18 17:19:57 +02:00
Silvano Cerza
f13cc751c3
Block requests_cache in unit tests (#4696) 2023-04-18 16:15:26 +02:00