1133 Commits

Author SHA1 Message Date
Branden Chan
e4c979990c
Bring back init defs to api in v1.2 and latest (#2296)
* Bring back init defs to api in v1.2

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-10 15:01:05 +01:00
tstadel
65295bc386
Prevent preprocessor from changing existing documents (#2297) 2022-03-10 14:56:51 +01:00
Sara Zan
18a6545055
Create milvus2 containers outside of haystack/ (#2300) 2022-03-10 14:55:15 +01:00
tstadel
fd46a42130
Allow to deploy and undeploy Pipelines on Deepset Cloud (#2285)
* add deploy_on_deepset_cloud and undeploy_on_deepset_cloud

* increase polling interval to 5 seconds

* Update Documentation & Code Style

* improve logging

* move transitioning logic to PipelineClient

* use enum for Pipeline states

* improve docstrings

* Update Documentation & Code Style

* tests added

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-10 09:49:28 +01:00
Sara Zan
e85b948a4c
Fix PreProcessor test (#2290)
* Adding Document import, missing from recent PR

* Fix mypy signature warning too

* reduce diff to minimum

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-09 13:46:47 +01:00
OmniScience
004e7f33af
Bulk insert in sql document stores (#2264)
* bulk_save_objects

* use bulk_insert_mappings in sql document store

* bug fix : write_document overwrite

* update documentation write_documents

* Update Documentation & Code Style

* small changes for readability

* bug fix: missing quotation marks

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
2022-03-09 12:11:26 +01:00
Tuana Celik
a97a9d2b48
adding quotes for zsh shell issue (#2289)
* adding quotes for zsh shell issue
2022-03-08 17:29:08 +01:00
Dmitry Goryunov
ecec9b4e2c
Remove substrings basic implementation (#2152)
* Remove substrings basic implementation

* Update Documentation & Code Style

* Remove substrings basic tests

* Simplify test
2022-03-08 15:49:56 +01:00
Vladimir Blagojevic
6c0094b5ad
Update LFQA with the latest LFQA seq2seq and retriever models (#2210)
* Register BartEli5Converter for vblagoje/bart_lfqa model

* Update LFQA unit tests

* Update LFQA tutorials
2022-03-08 15:11:41 +01:00
agnieszka-m
255226f9fa
Update other.yml with new node names (#2286)
* Update other.yml with new node names

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-08 12:29:49 +01:00
tstadel
4b46f2047b
save_to_deepset_cloud: automatically convert document stores (#2283)
* automatically convert to DeepsetCloudDocumentStore

* shorten info text.

* fix typo

* the -> this

* add test

* ensure request body has only DeepsetCloudDocumentStores

* mark test as elasticsearch to fix milvus1 ci
2022-03-07 22:35:15 +01:00
thimo72
ccef3effd9
Update reader/base.py to fix UncoundLocalError with add_isolated_node_eval=True (#2275) 2022-03-07 20:20:57 +01:00
tstadel
dde9d59271
fix pip backtracking issue (#2281)
* fix pip backtracking issue

* restrict azure-core version

* Remove the trailing comma

* Add skip_magic_trailing_comma in pyproject.toml for pydoc compatibility

* Pin pydoc-markdown _again_

Co-authored-by: Sara Zan <sarazanzo94@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-07 19:25:33 +01:00
mkkuemmel
5951fc463e
Replace dpr with embeddingretriever tut5 (#2274)
* ipynb: EmbeddingRetriever made more prominent than DPR

* ipynb: EmbeddingRetriever more prominent than DPR

* Update Documentation & Code Style

* indentation fix

* Update Documentation & Code Style

* py: EmbeddingRetriever more prominent than DPR

* indentation fix

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-04 11:29:48 +01:00
tstadel
256450370d
Fix Pipeline.print_eval_report (#2271)
* fix print_eval_report

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-03 16:52:49 +01:00
Sara Zan
130a92aa36
Remove JoinAnswers and RouteDocuments from the 1.2.0 schema (#2270)
* Remove JoinAnswers and RouteDocuments from the 1.2.0 schema

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-03 16:27:44 +01:00
Sara Zan
afe6e0b398
Bump version to 1.2.1rc0 (#2245)
* Bump version to 1.2.1rc0

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-03 16:06:07 +01:00
bogdankostic
447baf77ef
Fix skipping of tests using document stores (#2268)
* Fix skipping document store tests

* Update Documentation & Code Style

* Fix handling of Milvus1 and Milvus2 in tests

* Update Documentation & Code Style

* Fix handling of Milvus1 and Milvus2 in tests

* Update Documentation & Code Style

* Remove SQL from tests requiring embeddings

* Update Documentation & Code Style

* Fix get_embedding_count of Milvus2

* Make sure to start Milvus2 tests with a new collection

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-03 15:19:27 +01:00
tstadel
f7a01624e0
Refactor Pipeline peripherals (#2253)
* move peripheral stuff to utils, add more and better tests

* Update Documentation & Code Style

* move config related peripherals to config module, fix tests

* Update Documentation & Code Style

* remove unnecessary list comprehensions

* apply ZanSara's feedback

* remove classes in pipeline utils

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-03 14:14:42 +01:00
bogdankostic
a5501c72cf
Fix table extraction in ParsrConverter (#2262) 2022-03-02 09:21:54 +01:00
bogdankostic
c5542bd3fb
Add RouteDocuments and JoinAnswers nodes (#2256)
* Add SplitDocumentList and JoinAnswer nodes

* Update Documentation & Code Style

* Add tests + adapt tutorial

* Update Documentation & Code Style

* Remove branch from installation path in Tutorial

* Update Documentation & Code Style

* Fix typing

* Update Documentation & Code Style

* Change name of SplitDocumentList to RouteDocuments

* Update Documentation & Code Style

* Adapt tutorials to new name

* Add test for JoinAnswers

* Update Documentation & Code Style

* Adapt name of test for JoinAnswers node

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-03-01 17:42:11 +01:00
Sara Zan
11eebf8097
Add r to visual.py strings to silence warnings (#2260)
* Add r to visual.py strings to silence warnings

* Fix ASCII workers
2022-03-01 15:54:43 +01:00
Sara Zan
c00cf1c5f7
Fix typo in 0.7 JSON schema naming (#2259) 2022-03-01 12:19:54 +01:00
Branden Chan
b563b6622c
Api pages (#2248)
* Update Readme WIP

* Update Documentation & Code Style

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-25 13:53:46 +01:00
Branden Chan
bb107e5027
Polish Evaluation Tutorial (#2212)
* Polish evaluation tutorial

* Clear notebook output

* Cleanup tutorials

* Fix discrepancy in isolated retriever eval results

* Incorporate reviewer feedback

* Clean notebook output
2022-02-24 17:45:40 +01:00
MichelBartels
2c423ba063
Introduce support for pymilvus>=2.0.0 (#2126)
* update remaining occurences of get_connection

* fix milvus2 import and fix wrong extra references

* change MilvusDocumentStore to Milvus1DocumentStore

* update milvus docstrings to reflect updated dependency management

* enable milvus 2 tests

* fix milvus2 env variable processing

* fix dropping collections for each milvus 2 test

* make Milvus 2 doc store tests work

* allow user to specify consistency level

* Fist attempt at running Milvus2 in the CI

* Install the correct pymilvus

* add batch deletion for milvus2

* change default from milvus 1 to milvus 2

* make milvus2 the default in the docstores extra

* Switch milvus1 and milvus2 in base test run on CI

* Rename docstore flags for pytest: 'milvus'->'milvus1', 'milvus2'->'milvus'

* Rename milvus.py->milvus1.py and milvus2x.py->milvus2.py

* Enable autogenerated docs for Milvus1 and 2 separately

* Partial fix to docstring of Milvus2DocumentStore

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Michel Bartels <kontakt@michelbartels.com>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
2022-02-24 17:43:38 +01:00
Sara Zan
b475a1d4f7
Comment out Milvus cell on Tutorial6 (#2243)
* Comment out Milvus cell on Tutorial6

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-24 15:20:54 +01:00
bogdankostic
b03e9f5872
Fix surrounding context extraction in ParsrConverter (#2162)
* Fix surrounding context extraction

* Update Documentation & Code Style

* Unify Parsr and Azure + add test

* Update Documentation & Code Style

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-24 14:58:36 +01:00
Sara Zan
15c70bdb9f
Generate haystack-pipeline-1.2.0.schema.json (#2239)
* Trigger generation of the json schema for 1.2.0

* Remove path filters for `autoformat.yml`

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-24 11:45:21 +01:00
ju-gu
ca4e66ba16
change old text to content argument for translator examples (#2240)
* Update translator.md

* Update transformers.py

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-23 18:38:32 +01:00
Julian Risch
d21b6a5d4f
update docs for 1.2.0 relase (#2234) v1.2.0 2022-02-23 11:36:04 +01:00
Sara Zan
3e39ce0f4e
Rename openapi-1.2.0rc0.json into openapi-1.2.0.json (#2233) 2022-02-23 11:31:32 +01:00
Julian Risch
842b5ab0f4
update version to 1.2.0 for release (#2232) 2022-02-23 11:20:00 +01:00
tstadel
e20f2e0d54
Generate code from pipeline (pipeline.to_code()) (#2214)
* pipeline.to_code() with jupyter support

* Update Documentation & Code Style

* add imports

* refactoring

* Update Documentation & Code Style

* docstrings added and refactoring

* Update Documentation & Code Style

* improve imports code generation

* add comment param

* Update Documentation & Code Style

* add simple test

* add to_notebook_cell()

* Update Documentation & Code Style

* introduce helper classes for code gen and eval report gen

* add more tests

* Update Documentation & Code Style

* fix Dict typings

* Update Documentation & Code Style

* validate user input before code gen

* enable urls for to_code()

* Update Documentation & Code Style

* remove all chars except colon from validation regex

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-23 11:08:57 +01:00
bogdankostic
4bad21e961
Add Brownfield Support of existing Elasticsearch indices (#2229)
* Add method to transform existing ES index

* Add possibility to regularly add new records

* Fix types

* Restructure import statement

* Add use_system_proxy param

* Update Documentation & Code Style

* Change location and name + add test

* Update Documentation & Code Style

* Add test cases for metadata fields

* Fix linter

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-22 20:58:57 +01:00
tstadel
965cc86b24
Fix ef_search param for hnsw in OpenSearchDocumentStore (#2227)
* fix ef_search param for hnsw

* Update Documentation & Code Style

* adjust ef_search param if index exists

* run black

* Fix label index recreation

* fix merge conflict

* merge source branch 'master' into fix_ef_search_param

* fix pylint issue

* fix test_pipeline_components test

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-22 20:33:21 +01:00
MichelBartels
6918d5b79e
Fix missing embeddings not skipped if filters are used (#2230)
* fix skip embeddings param for elasticsearch when filters are specified

* Update Documentation & Code Style

* add test

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-22 18:40:34 +01:00
MichelBartels
f4efc008f4
Adding extended meta data filtering support for InMemoryDocumenStore (#2120)
* add filter classes

* update filter comments

* Add util classes for converting filters (#2123)

* Apply Black

* reintroduce eval functions to filter ops

* Update documentation

* update to latest pymilvus version

* Apply Black

* fixing type hints

* Apply Black

* update write_documents method of milvus2 doc store

* remove unnecessary method

* update init

* remove changes to milvus 2 as they are part of other PR

* remove changes to milvus 2 as they are part of other PR

* updating doc strings to match elastic search filter doc

* Update Documentation & Code Style

* add support for case where there is no meta data defined for key

* update behaviour in case of field not existing in entry

* Update Documentation & Code Style

* add test for InMemoryDocumentStore extended meta data filtering

* make type hint more precise

Co-authored-by: bogdankostic <bogdankostic@web.de>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sara Zan <sarazanzo94@gmail.com>
2022-02-22 17:44:58 +01:00
Sara Zan
5c119d973d
Versioning openapi.json (#2228)
* Add versions of the the openapi.json file for versioned documentation

* Update Documentation & Code Style

* Use .suffix for extension check

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-22 17:35:00 +01:00
tstadel
fe03ca70de
Fix Pipeline.components (#2215)
* add components property, improve get_document_store()

* Update Documentation & Code Style

* use pipeline.get_document_store() instead of retriever.document_store

* add tests

* Update Documentation & Code Style

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-22 15:01:07 +01:00
Sara Zan
d1b7761504
Generate JSON schema index for Schemastore (#2225)
* Generate JSON schema index

* Add index file

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-22 09:29:44 +01:00
MichelBartels
116fe2db26
Extend meta data support for SQLDocumentStore (#2199)
* update remaining occurences of get_connection

* first commit to add extended metadata filtering support to sql

* fix bugs

* adding sql doc store instead of milvus

* removing updates to milvus2 from other PR

* fixing not operator

* delete left over line

* remove unnecessary import

* Update Documentation & Code Style

* fix circular import

* fix left over merge conflict

* Update Documentation & Code Style

* fix abstract class

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-21 20:40:32 +01:00
Sara Zan
8de1aa3e43
Pylint: solve or silence locally rare warnings (#2170)
* Remove invalid-envvar-default and logging-too-many-args

* Remove import-self, access-member-before-definition and deprecated-argument

* Remove used-before-assignment by restructuring type import

* Remove unneeded-not

* Silence unnecessary-lambda (it's necessary)

* Remove pointless-string-statement

* Update Documentation & Code Style

* Silenced unsupported-membership-test (probably a real bug, can't fix though)

* Remove trailing-newlines

* Remove super-init-not-called and slience invalid-sequence-index (it's valid)

* Remove invalid-envvar-default in ui

* Remove some more warnings from pyproject.toml than actually solrted in code, CI will fail

* Linting all modules together is more readable

* Update Documentation & Code Style

* Typo in pylint disable comment

* Simplify long boolean statement

* Simplify init call in FAISS

* Fix inconsistent-return-statements

* Fix useless-super-delegation

* Fix useless-else-on-loop

* Fix another inconsistent-return-statements

* Move back pylint disable comment moved by black

* Fix consider-using-set-comprehension

* Fix another consider-using-set-comprehension

* Silence non-parent-init-called

* Update pylint exclusion list

* Update Documentation & Code Style

* Resolve unnecessary-else-after-break

* Fix superfluous-parens

* Fix no-else-break

* Remove is_correctly_retrieved along with its pylint issue

* Update exclusions list

* Silence constructor issue in squad_data.py (method is already broken)

* Fix too-many-return-statements

* Fix use-dict-literal

* Fix consider-using-from-import and useless-object-inheritance

* Update exclusion list

* Fix simplifiable-if-statements

* Fix one consider-using-dict-items

* Fix another consider-using-dict-items

* Fix a third consider-using-dict-items

* Fix last consider-using-dict-items

* Fix three use-a-generator

* Silence import errors on numba, tensorboardX and apex, but add comments & logs

* Fix couple of mypy issues

* Fix another typing issue

* Silence mypy, was conflicting with more meaningful pylint issue

* Fix no-else-continue

* Silence unsubscriptable-object and fix an import error with importlib.metadata

* Update Documentation & Code Style

* Fix all no-else-raise

* Update Documentation & Code Style

* Fix inverted parameters in simplified if switch

* Change [test] to [all] in some jobs (for typing and linting)

* Add comment in haystack/schema.py on pydantic's dataclasses

* Move comment from get_documents_by_id into _convert_weaviate_result_to_document in weaviate.py

* Add comment on pylint silencing

* Fix bug introduced rest_api/controller/search.py

* Update Documentation & Code Style

* Add ADR about Pydantic dataclasses

* Update pydantic-dataclasses.md

* Add link to Pydantic docs on Dataclasses

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-21 20:16:14 +01:00
tstadel
b2e6dcc998
Add docker-compose override file for Traffic Monitoring (#2224)
* add docker-compose override file for traffic monitoring

* Update Documentation & Code Style

* remove faulty imports

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-21 16:12:34 +01:00
Kristof Herrmann
9096ddad34
Adding a minimal haystack gpu build (#2185)
* added minimal gpu image

* Update Documentation & Code Style

* removed old installations

* build minimal images in gh action

* quotes to single quotes

* switched repos

* fix

* fix ordering

* move to deepset dockerhub acc

* update nvidia/cuda image to match newest torch+cu

* refactor Dockerfile-GPU-minimal to optimize build

* Remove spurious doc changes from tutorial6

* install ocr dependencies and pdftotext

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sara Zan <sarazanzo94@gmail.com>
Co-authored-by: Ivan Lopez <ivan@askai.net>
2022-02-21 13:34:44 +01:00
Sara Zan
2a840ee248
YAML versioning (#2209)
* Make YAML files get the same version as Haystack and throw warning at load in case of mismatch

* Update version of most YAMLs in the codebase (aesthethic chamge, only to avoid the warning).

* Remove quotes from version in tests

* Fix version in generate_json_schema.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-21 12:22:37 +01:00
bogdankostic
2a674eaff7
Support more data types and extended filters in WeaviateDocStore (#2143)
* Support more data types and extended filters in WeaviateDocStore

* Adapt types to extended filters

* Update Documentation & Code Style

* Fix mypy

* Fix type of filters

* Update Documentation & Code Style

* Add Docstrings for BaseDocStore

* Update Documentation & Code Style

* Add + prettify DocStrings

* Update Documentation & Code Style

* Fix types

* Update Documentation & Code Style

* Remove import of TypedDict

* Fix tests

* Update Documentation & Code Style

* Fix circular import

* Fix inversion of not operation + add test case

* Fix mypy

* Update Documentation & Code Style

* Apply black

* Use convert_date_to_rfc3339 instead of datetime.fromisoformat

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-18 08:55:17 +01:00
Sara Zan
1c61c1edaa
Rename image to fix link in CONTRIBUTING.md (#2211) 2022-02-17 17:45:39 +01:00
Sara Zan
c93b365f98
Improve Label and MultiLabel __str__ and __repr__ (#2202)
* Add proper annotation to MultiLabel and slightly improve __str__ for MultiLabel and Label

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-02-17 16:37:02 +01:00
Julian Risch
542d7b8128
Add release candidate note to version (#2213) 2022-02-17 15:15:07 +01:00