144 Commits

Author SHA1 Message Date
Stefano Fiorucci
ff06da8712
pin fastapi (#6120) 2023-10-19 13:30:50 +02:00
Christian Clauss
bf6d306d68
ci: Simplify Python code with ruff rules SIM (#5833)
* ci: Simplify Python code with ruff rules SIM

* Revert #5828

* ruff --select=I --fix haystack/modeling/infer.py

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-09-20 08:32:44 +02:00
Christian Clauss
1bc03ddc73
ci: Fix all ruff pyflakes errors except unused imports (#5820)
* ci: Fix all ruff pyflakes errors except unused imports

* Delete releasenotes/notes/fix-some-pyflakes-errors-69a1106efa5d0203.yaml
2023-09-15 18:30:33 +02:00
Christian Clauss
6dd52d91b2
ci: Fix typos discovered by codespell (#5778)
* Fix typos discovered by codespell

* pylint: max-args = 38
2023-09-13 16:14:45 +02:00
totto
7c7a486014
fix: in a containerized environment (like AWS ECS) there is a file wr… (#5499)
* fix: in a containerized environment (like AWS ECS) there is a file write permission error: 	PermissionError: [Errno 13] Permission denied: 'feedback_squad_direct.json'. catch this error.
hint: future solution similar to FILE_UPLOAD_PATH to provide a writeable path in a container.

(cherry picked from commit c54ab7ed2d487e4391c0391be7c3e268ae525507)

* fix linter error: dont use f string in logger message

* reformat

* fix: pylint requires using % in logging message
2023-08-25 13:32:29 +02:00
Muhammad Bilal
8920fd6939
feat: add optional index selection for endpoints (#5444)
* add index selection

* reformatting

* updated test script
2023-08-01 10:47:46 +02:00
Massimiliano Pippi
d9fd1ab7bc
feat!: remove original files after indexing (#5459)
* remove original files after indexing

* fix tests
2023-07-31 13:07:16 +02:00
Vladimir Blagojevic
22897c17a2
fix:Improve log warnings in REST API /health endpoint (#5381)
* Improve warning in REST APIs get_health_status method

* Convert log message

* A better solution and documentation

* Add another nested try/except block

* Simplify
2023-07-25 17:06:03 +02:00
Vladimir Blagojevic
395854d823
Add cpu-remote-inference Docker image (#5225)
* Add cpu-remote-inference Docker image

* Add web lfqa pipeline as an example for cpu-remote-inference Docker image

* WebRetriever must have document_store attribute

* Add cpu-remote-inference-latest

* Add image testing in CI

---------

Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
2023-07-07 10:23:14 +02:00
ZanSara
4b380d8fb0
fix: install inference in REST API tests (#5252)
* install inference in restapi tests

* add workflow dispatch to test the REST API CI in PR

* trigger ci

* tablecell

* tablecell

* revert ci trigger

* mypy
2023-07-03 15:10:14 +02:00
Silvano Cerza
ba06bc4805
Unpin typing_extensions and remove all its uses (#5040) 2023-05-29 15:31:34 +02:00
Massimiliano Pippi
4974bf7ab3
chore: remove deprecated MilvusDocumentStore (#4951)
* remove deprecated MilvusDocumentStore

* remove leftovers

* fix pylint
2023-05-19 16:37:38 +02:00
Sebastian
8d9136bad4
feat: Implementation of Table Cell Proposal (#4616)
* Starting adding support for TableCell

* Update tests to use row and col

* Added schema test to check to_dict and from_dict works for Table documents. Also updated Doc.__eq__ to work for tables.

* Update eval test to use TableCell

* Added more schema tests for table docs, labels and answers.

* Add boolean to toggle between Span and TableCell

* Add deprecation message

* Test that table answers work as responses in the rest API

---------

Co-authored-by: agnieszka-m <amarzec13@gmail.com>
2023-04-19 13:14:49 +02:00
ZanSara
c777302fb4
chore: disable posthog in rest api tests (#4507) 2023-03-27 21:12:27 +02:00
ZanSara
6d578ebf3d
refactor: remove telemetry v1 (#4496)
* remove telemetry v1

* more pipeline methods to take out

* send_event_2

* mypy

* pylint

* mypy

* mypy again

* remove test
2023-03-27 17:38:43 +02:00
Mayank Jobanputra
5f72cdc012
fix: stop loading FAISS and InMem doc Store for indexing pipelines (#4396)
* stop loading FAISS and InMem doc Store for indexing pipelines

* pylint fix

* Addressed comments
2023-03-24 19:35:29 +05:30
Jose Pablo Fernandez
dda350088b
feat: add additional params to file upload endpoint (#4445)
* adds additional params to file upload endpoint

* fix mypy

---------

Co-authored-by: Mayank Jobanputra <mayankjobanputra@gmail.com>
2023-03-23 14:18:16 +01:00
bogdankostic
e3503a92c9
build: Use uvicorn instead of gunicorn as server in REST API's Dockerfile (#4304)
* Use uvicorn instead of gunicorn as server

* Added comments and changed service names

* comments improvised

---------

Co-authored-by: Mayank Jobanputra <mayankjobanputra@gmail.com>
2023-03-09 01:46:07 +05:30
ZanSara
13c4ff1b52
refactor: remove direct logging without a logger (#4253)
* remove direct logging without a logger

* add custom pylint checker

* add test

* pylint

* improve checker message

* mypy

* remove test

* add checker for basicConfig

* more logging missed

* ignore basicConfig

* move out logger

* move out statement

* remove logging configuration
2023-02-23 20:42:42 +01:00
oryx1729
8ecadd1cac
fix: query filters in REST API (#4105)
* Remove legacy _format_filters()

* Remove test case
2023-02-09 10:42:31 +01:00
Silvano Cerza
274746db07
style: Update black (#4101)
* Update black version

* Format file with new black style

* Update black pre-commit hook version
2023-02-08 15:34:43 +01:00
tstadel
92c58cfda1
feat: Support multiple document_ids in Answer object (for generative QA) (#4062)
* initial version without shapers

* set document_ids for BaseGenerator

* introduce question-answering-with-references template

* better prompt

* make PromptTemplate control output_variable

* update schema

* fix add_doc_meta_data_to_answer

* Revert "fix add_doc_meta_data_to_answer"

This reverts commit b994db423ad8272c140ce2b785cf359d55383ff9.

* fix add_doc_meta_data_to_answer

* fix eval

* fix pylint

* fix pinecone

* fix other tests

* fix test

* fix flaky test

* Revert "fix flaky test"

This reverts commit 7ab04275ffaaaca96b4477325ba05d5f34d38775.

* adjust docstrings

* make Label loading backward-compatible

* fix Label backward compatibility for pinecone

* fix Label backward compatibility for search engines

* fix Label backward compatibility for deepset Cloud

* fix tests

* fix None issue

* fix test_write_feedback

* add tests for legacy label support

* add document_id test for pinecone

* reduce unnecessary contents

* add comment to pinecone test
2023-02-08 08:37:22 +01:00
tstadel
8002cf92d6
fix: extend schema for prompt node results (#3891)
* extend schema for prompt node results

* extend schema

* update openapi

* fix mypy for test module

* added 1.14 specs

* reverted schema for 1.13

---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
Co-authored-by: Mayank Jobanputra <mayankjobanputra@gmail.com>
Co-authored-by: Sebastian <sjrl@users.noreply.github.com>
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-01-31 16:31:33 +01:00
Stefano Fiorucci
2bbe11b598
fix: overwrite params with environment variables even if there are no params in the pipeline definition; make mypy ignore REST API tests (#3930)
* fix and new test

* make mypy ignore rest_api tests files

* try to improve mypy action

* retry

* fix

* test new action

* ok

* check python files not in root

* really check files!
2023-01-26 16:14:58 +01:00
Sebastian
fac72aa14e
Updated schema in rest_api to make returned table format consistent with how it is returned when using Haystacks' Document.to_dict(). (#3872) 2023-01-25 23:18:12 +05:30
Benjamin BERNARD
eed009eddb
feat: Add CsvTextConverter (#3587)
* feat: Add Csv2Documents, EmbedDocuments nodes and FAQ indexing pipeline

Fixes #3550, allow user to build full FAQ using YAML pipeline description and with CSV import and indexing.

* feat: Add Csv2Documents, EmbedDocuments nodes and FAQ indexing pipeline

Fix linter issues mypy and pylint.

* feat: Add Csv2Documents, EmbedDocuments nodes and FAQ indexing pipeline

Fix linter issues mypy.

* implement proposal's feedback

* tidy up for merge

* use BaseConverter

* use BaseConverter

* pylint

* black

* Revert "black"

This reverts commit e1c45cb1848408bd52a630328750cb67c8eb7110.

* black

* add check for column names

* add check for column names

* add tests

* fix tests

* address lists of paths

* typo

* remove duplicate line

Co-authored-by: ZanSara <sarazanzo94@gmail.com>
2023-01-23 15:56:36 +01:00
Mayank Jobanputra
dad7b12874
fix: Allowing InMemStore and FAISSDocStore for indexing using single worker (#3868)
* Allowing InMemStore and FAISSDocStore for indexing using single worker YAML config

* unified pipeline & doc store loading

* fix pylint warning

* separated tests

* removed unnecessay caplog
2023-01-19 14:06:00 +05:30
ZanSara
3ffdb0a9a3
chore: fix all EOF (#3852)
* fix all eof

* fix test

* fix test

* fix test

* typo

* fix sample

* fix sample

* add logs

* fix page_dynamic_result.txt
2023-01-16 12:34:50 +01:00
ZanSara
d157e41c1f
chore: enable logging-fstring-interpolation and cleanup (#3843)
* enable logging-fstring-interpolation

* remove logging-fstring-interpolation from exclusion list

* remove implicit string interpolations added by black

* remove from rest_api too

* fix % sign
2023-01-12 09:31:21 +01:00
ZanSara
4cbc8550d6
chore: enable trailing-whitespace and cleanup (#3847)
* enable trailing-whitespace

* remove trailing whitespace on rest api too
2023-01-11 20:08:19 +01:00
Agnieszka Marzec
b8fff837b4
docs: Add info where the feedback is stored (#3772)
* Add info where the feedback is stored

* Fix misplaced line breaks

* Generate OpenAPI Specs

* Generate OpenAPI Specs

* Apply black

* Generate OpenAPI specs

* Add missing whitespace

Co-authored-by: bogdankostic <bogdankostic@web.de>
2022-12-28 14:46:26 +01:00
Bilge Yücel
86ade4817e
bug: fix the docs rest api reference url (#3775)
* bug: fix the docs rest api reference url

* revert openapi json changes

* remove last line on json files

* Add explanation about `servers` and remove `servers` parameter from FastAPI

* generate openapi schema without empty end line
2022-12-28 12:30:58 +03:00
Tobias Wochinger
33c480286a
ci: add license compliance check (#3221)
* ci: add license compliance check

* ci: run check always for testing purposes

* revamp workflows

* temporary remove path directive

* triggering ci

* check rest api and ui too

* avoid cache to make sure env is clean

* add shield on readme

* ci: trigger CI to get latest scan

Co-authored-by: ZanSara <sarazanzo94@gmail.com>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
2022-12-22 10:08:26 +01:00
bogdankostic
12c264603e
fix: Fix number of concurrent requests in RequestLimiter (#3705) 2022-12-21 11:40:33 +01:00
tstadel
600dc2d611
refactor: filters type (#3682)
* consolidate filters type

* remove unnecessary optionals

* fix mypy

* fix pylint

* fix pylint

* move FilterType to schema

* remove Optional from FilterType

* move to Dict[str, Any]

* Revert "move to Dict[str, Any]"

This reverts commit e8c561bb7885949e19825697fa4c469945f90ce5.

* fix mypy

* fix pylint

* revert isort changes in elasticsearch

* remove todos in milvus.py

* remove todos in sql.py

* add aggregate_labels tests

* consolidate aggregate_labels tests

* remove superfluous type todos

* remove ALL superfluous #todos
2022-12-12 14:04:29 +01:00
ju-gu
559730649b
fix: [rest_api] support TableQA in the endpoint /documents/get_by_filters (#3551)
* Update schema.py

* Update document.py
2022-11-14 15:09:14 +01:00
Massimiliano Pippi
7af22cd98c
CI: install httpx to run tests (#3565)
* install httpx to run tests

* try
2022-11-14 12:52:04 +01:00
Sara Zan
9539a209ae
refactor: apply pep-484 (#3542)
* apply pep-484

* another implicit optional

* apply pep-484 on rest_api and ui too
2022-11-08 14:30:33 +01:00
Sara Zan
54cc9cd4cf
refactor: remove json-schemas (#3485)
* remove json-schemas

* main schema can be removed too

* add .gitignore to schemas folder

* try to explicitly get the new haystack in the rest api tests

* fix workflow again

* fix version string in rest api tests

* add pip freeze

* debug statements in workflow

* -U prevents schema generation
2022-10-31 11:24:43 +01:00
Massimiliano Pippi
8f76d64f6f
chore: bump release number for unstable version (#3251)
* bump version for unstable

* allow generation of rc schemas

* update schemas
2022-09-21 16:58:06 +02:00
Sara Zan
dcb132ba59
chore: remove f-strings from logs for performance reasons (#3212)
* Use the %s syntax on all debug messages

* Use the %s syntax on some more debug messages

* Use the %s syntax on info messages

* Use the %s syntax on warning messages

* Use the %s syntax on error and exception messages

* mypy

* pylint

* trogger tutorials execution in CI

* trigger tutorials execution on CI

* black

* remove embeddings from repr

* fix Document `__repr__`

* address feedback

* mypy
2022-09-19 18:18:32 +02:00
Daniel Bichuetti
1a6cbca9b6
feat: add health check endpoint to rest api (#3168)
* feat: add /health endpoint to rest api

* refactor: adjust to new dir structure

* fix: add new rest api dependency

* docs: add new openapi schema

* docs: manual black run

* refactor: remove some sys-wide details

* docs: minor description changes

* docs: minor description changes

* docs: generate openapi schemas

* tests: improved tests

* refactor: add cls method decorator
2022-09-08 18:24:16 +02:00
Massimiliano Pippi
6790eaf7d8
refactor: update package strategy in rest_api (#3148)
* update packaging

* fix author metadata

* add newline

* add empty readme

* fix path to pipeline files

* fix pylint job

* fix metadata
2022-09-05 16:58:43 +02:00
Sara Zan
e92ea4fccb
refactor: rename master into main in documentation and links (#3063)
* master->main

* revert master rename

* Revert change to sphinx link and rename master schema
2022-08-24 19:05:12 +02:00
Julian Risch
d61755322f
chore: fix typo in API docs (#3023)
* chore: fix typo in API docs

* fix openapi

Co-authored-by: Thomas Stadelmann <thomas.stadelmann@deepset.ai>
2022-08-15 13:25:20 +02:00
tstadel
b042dd9c82
Fix validation for dynamic outgoing edges (#2850)
* fix validation for dynamic outgoing edges

* Update Documentation & Code Style

* use class outgoing_edges as fallback if no instance is provided

* implement classmethod approach

* readd comment

* fix mypy

* fix tests

* set outgoing_edges for all components

* set outgoing_edges for mocks too

* set document store outgoing_edges to 1

* set last missing outgoing_edges

* enforce BaseComponent subclasses to define outgoing_edges

* override _calculate_outgoing_edges for FileTypeClassifier

* remove superfluous test

* set rest_api's custom component's outgoing_edges

* Update docstring

Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>

* remove unnecessary else

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai>
2022-08-04 10:27:50 +02:00
tstadel
2c56305ed3
Fix serialization of numpy arrays and pandas dataframes in REST API (#2838)
* correct serialization of numpy arrays and pandas dataframes

* Update Documentation & Code Style

* set additional json_encoders globally

* Update Documentation & Code Style

* add tests for non primitive return types

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-08-02 09:49:32 +02:00
Massimiliano Pippi
632cd1c141
Allow values that are not dictionaries in the request params in the /search endpoint (#2720)
* let params contain something else than dictionaries

* rewrite the test same style as the main branch
2022-07-15 13:24:29 +02:00
Massimiliano Pippi
82df677ebf
API tests (#2738)
* clean up tests and run earlier

* use change detection

* better naming, skip ES

* more cleanup

* fix job name

* dummy commit to trigger the CI

* mock away the PDF converter

* make the test compatible with 3.7

* removed leftover

* always run the api tests, use a matrix for the OS

* refactor all the tests

* remove outdated dependency

* pylint

* new abstract method

* adjust for older python versions

* rename pipeline file

* address PR comments
2022-07-14 15:36:28 +02:00
bogdankostic
867695ad0c
Change signature of queries param in batch methods (#2575)
* Change signature of queries param in batch methods

* Update Documentation & Code Style

* Fix mypy

* Remove unused import

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-05-24 12:33:45 +02:00