791 Commits

Author SHA1 Message Date
Julian Risch
d569e66bc7
Update Tutorial1_Basic_QA_Pipeline.ipynb (#1489)
* Update Tutorial1_Basic_QA_Pipeline.ipynb

passing params to pipeline as dict

* Add latest docstring and tutorial changes

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2021-09-22 16:35:20 +02:00
Malte Pietsch
ff1adb64c2
Update README.md 2021-09-21 17:56:40 +02:00
Branden Chan
bddee2def4
Define SAS model in notebook (#1485)
* Define SAS model in notebook

* Add latest docstring and tutorial changes

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2021-09-21 17:05:16 +02:00
Branden Chan
2c4baa7f4e
Regenerate API and Tutorial md files (#1480)
* Change punctuation

* Add latest docstring and tutorial changes

* Change punctuation

* Add documentation for Docs2Answer

* Add latest docstring and tutorial changes

* Generate new API docs

* Replace Finder with Pipeline

* Add latest docstring and tutorial changes

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2021-09-21 14:42:18 +02:00
ju-gu
05da7f71dd
changed delete_all_documents to delete_documents (#1477) 2021-09-20 14:29:33 +02:00
Markus Paff
e0ad6b64bf fixed workflow conflict with introducing new one (#1472) 2021-09-20 12:21:20 +02:00
Malte Pietsch
ab7d5853f2
Bump Version 2021-09-20 08:40:38 +02:00
Sara Zan
21513532e5
Improve save/load of FAISS document store by saving its configuration alongside the index (#1459)
* Saves the FAISSDocumentStore init params to JSON at save() and loads them at load() if they're found. First draft, to be tested.

* Fixing issue with string/Path objects in a few string operations, thanks mypy

* Leverage self.set_config instead of saving the parameters in a separate attribute

* Modify test_faiss_and_milvus:test_faiss_index_save_and_load to test that init params are preserved

* Add assert to verify that the SQL doc count and FAISS vector count is equal. Needs to always specify the name of the SQL db for this to work

* Simplified the implementation a bit, add better comments

* Forgot a return at the end of the file

* Fixing some of the suggestions from the review

* Add a try-catch in the load method and fix the tests

* Typo
2021-09-20 08:32:14 +02:00
mathislucka
9c4e67d9b6
Enable cosine similarity metric in FAISSDocumentStore (#1352)
* feat: normalize embeddings for cosine sim

* WIP add test case for faiss cosine

* input to faiss normalize needs to be an array of vectors

* fix: test should compare correct result embedding to original embedding

* add sanity check for cosine sim

* fix typo

* normalize cosine score

* Update docstring

Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
2021-09-20 07:54:26 +02:00
Markus Paff
5b1b875374
fixed workflow conflict with introducing new one (#1472) 2021-09-17 23:44:45 +02:00
Markus Paff
39845c0624
Automate updates docstrings tutorials (#1461)
* remove not needed githab actions and reactivate docstrings and tutorial generation

* test workflow

* update pydoc version

* update python version

* update watchdog

* move to latest version pydoc-markdown

* remove version check

* Add latest docstring and tutorial changes

* remove test workflow

* test for param docstrings

* pin pydoc-markdown version

* add test workflow

* pin watchdog version

* Add latest docstring and tutorial changes

* update original workflow and delete test

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2021-09-17 13:44:31 +02:00
Timo Moeller
172de1c05f
Merge pull request #1422 from deepset-ai/farm_merging_base
Farm merging base
2021-09-16 11:32:41 +02:00
Malte Pietsch
30dc010171 Bump Haystack version to 0.10.0 v0.10.0 2021-09-16 06:49:44 +02:00
oryx1729
9a1e3fec86
Update DocumentStore env in docker-compose (#1450) 2021-09-14 12:28:30 +02:00
Timo Moeller
d804861fb2 Fix tests 2021-09-13 20:00:22 +02:00
Timo Moeller
ba7178be7f satisfy mypy 2021-09-13 19:29:20 +02:00
Timo Moeller
537204e8c9
Fix tests and adjust folder structure
* Add type annotations in QuestionAnsweringHead

* Fix test by increasing max_seq_len

* Add SampleBasket type annotation

* Remove prediction head param from adaptive model init

* Add type ignore for AdaptiveModel init

* Fix and rename tests

* Adjust folder structure

Co-authored-by: Julian Risch <julian.risch@deepset.ai>
2021-09-13 18:38:14 +02:00
Priyam Mehta
389f6b68fb
Added functionality for Google Colab usecase in Crawler Module (#1436)
* Added functionality for Google Colab usecase

* Corrected typo in installation guide of driver

* Corrected typo in installation guide of driver

* Corrected the copy command
2021-09-13 14:58:36 +02:00
Malte Pietsch
b53ad7af53
quality of life function to access certain nodes in pipeline (#1441) 2021-09-13 13:03:38 +02:00
Ikram Ali
f186d6327d
Add MostSimilarDocumentsPipeline (#1413)
* [pipeline] MostSimilarDocumentsPipeline added

* [pipeline] mypy bug fixed.

* [pipeline] mypy bug fixed.

* [pipeline] test cases added.

* [pipeline] test cases added.

* [pipeline] set return_embedding back to false.

* [pipeline] return a list of Documents

* [pipeline] define the ids

* [pipeline] code refactor.

* [pipeline] code refactor.

* [pipeline] test case improved.

* Update docstring

Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
2021-09-13 12:43:45 +02:00
oryx1729
3deff26b60
Fix Search REST API when filters are None (#1431) 2021-09-10 14:47:34 +02:00
MichelBartels
da2e8da561
Adding multi gpu support for DPR inference (#1414)
* Added support for Multi-GPU inference to DPR including benchmark

* fixed multi gpu

* added batch size to benchmark to better reflect multi gpu capabilities

* remove unnecessary entry in config.json

* fixed typos

* fixed config name

* update benchmark to use DEVICES constant

* changed multi gpu parameters and updated docstring

* adds silent fallback on cpu

* update doc string, warning and config

Co-authored-by: Michel Bartels <kontakt@michelbartels.com>
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
2021-09-10 13:25:02 +02:00
oryx1729
1f859694f1
Add support for Dense Retrievers in REST API Indexing Pipeline (#1430) 2021-09-10 11:53:32 +02:00
oryx1729
9dd7c74f4f
Refactor communication between Pipeline Components (#1321) 2021-09-10 11:41:16 +02:00
Timo Moeller
e8a6427b9e Remove farm mentions from code and docs, reformat code 2021-09-09 15:48:11 +02:00
Julian Risch
4a64c50c7e Merge branch 'farm_merging_base' of github.com:deepset-ai/haystack into farm_merging_base 2021-09-09 13:03:38 +02:00
Julian Risch
ba1fe0ec61 Add fixture distilbert_squad 2021-09-09 13:02:35 +02:00
bogdankostic
2626388961
Fix DPR tests + add Tokenizer tests (#1429)
* Fix DPR tests

* Add Tokenizer tests
2021-09-09 12:56:44 +02:00
oryx1729
3e6def7e03
Add type ignore to resolve mypy errors (#1427) 2021-09-09 12:29:01 +02:00
Julian Risch
23338f1b74 Add tests: prediction head, processor load/save, qa from FARM 2021-09-09 11:54:47 +02:00
bogdankostic
9c409e0012 Remove StreamingDataSilo and fix mypy errors from FARM (#1426)
* Add AdaptiveModel

* Add BiAdaptiveModel

* Add DataSilo

* Remove StreamingDataSilo

* Fix mypy errors
2021-09-09 10:12:35 +02:00
dependabot[bot]
a92f1860f6
Bump pillow from 8.2.0 to 8.3.2 (#1423)
Bumps [pillow](https://github.com/python-pillow/Pillow) from 8.2.0 to 8.3.2.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/8.2.0...8.3.2)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-09-08 17:51:18 +02:00
Timo Moeller
b4fd08a296
Add testdata, add tests for qa processor, add dpr tests (some failing) 2021-09-08 12:02:08 +02:00
Timo Moeller
024b9e0bf8 Merge previous solutions: fix imports, add needed helper functions or remove unused ones 2021-09-08 11:51:41 +02:00
Timo Moeller
a945b43a57
Farm merging base bogdan (#1424)
* Add AdaptiveModel

* Add BiAdaptiveModel

* Add DataSilo

Co-authored-by: Bogdan Kostić <bogdankostic@web.de>
2021-09-08 10:38:28 +02:00
Timo Moeller
c5999c3c8f Add LMand tokenization 2021-09-07 13:37:36 +02:00
Julian Risch
55a8031aeb
Adding prediction head, trainer, evaluator from FARM (#1419) 2021-09-07 13:33:17 +02:00
Timo Moeller
5bc5665c0b Add processor and processing related scripts 2021-09-07 12:33:33 +02:00
Bob van Luijt
c0cc8bc80f
Bump Weaviate version to 1.7.0 (#1412)
* Bump Weaviate

* Bump Weaviate

* Bump Weaviate client

* Bump Weaviate

* Revert client version

There is a change in the client API that needs to be addressed before bumping its version
2021-09-05 09:28:55 +02:00
Malte Pietsch
f3e7074c13
Remove stale bot 2021-09-03 17:39:24 +02:00
Malte Pietsch
f3d1df1664
Enable docker-compose for GPUs & Add public UI image (#1406)
* add docker-compose-gpu file

* Update README.md

* Update docker-compose.yml

* Update docker-compose-gpu.yml

* Update docker-compose.yml

* Update docker-compose-gpu.yml
2021-09-02 17:39:21 +02:00
Malte Pietsch
bb9ec90d3c
Fix tesseract installation in Dockerfile (#1405)
* Fix Dockerfile

* Update Dockerfile-GPU
2021-09-02 11:09:30 +02:00
bogdankostic
38128c6734
Ensure num_hard_negatives is 0 when embedding passages (#1402) 2021-09-02 10:46:02 +02:00
Julian Risch
b552bf9b4d
Add sentence-transformers as mandatory dependency and remove from dev… (#1387)
* Add sentence-transformers as mandatory dependency and remove from dev dependency

* Pin sentence-transformers version
2021-09-02 09:54:13 +02:00
Branden Chan
980d88a0f2
Update faq model (#1401) 2021-09-01 18:39:06 +02:00
Malte Pietsch
e4c3c3d423
Fix CI (introduced by OCR PR #1349) (#1399)
* satisfy mypy

* add import
2021-09-01 17:16:05 +02:00
Malte Pietsch
6093bf9ff6
Fix Github action 2021-09-01 16:50:29 +02:00
Shahrukh Khan
4822536886
Add ImageToTextConverter and PDFToTextOCRConverter that utilize OCR (#1349)
* add image.py converter

* add PDFtoImageConverter

* add init to PDFtoImageConverter and classes to __init__

* update imagetotext pipeline

* update imagetotext pipeline

* update imagetotext pipeline

* update imagetotext pipeline

* update imagetotext pipeline

* update imagetotext pipeline

* update imagetotext pipeline

* revert change in base.py in file_conv

* Update base.py

* Update pdf.py

* add ocr file_converter testcase & update dockerfile

* fix tesseract exception message typo

* fix _image_to_text doctstring

* add tesseract installation to CI

* add tesseract installation to CI

* add content test for PDF OCR converter

* update PDFToTextOCRConverter constructor doctsring

* replace image files with tmp paths for image.py convert

* replace image files with tmp paths for image.py convert

* Update README.md

Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
2021-09-01 16:42:25 +02:00
oryx1729
1d2252e96d
docker-compose always pull REST API Image (#1385) 2021-09-01 16:28:25 +02:00
Ikram Ali
3fc7f3f695
[docs] crawler api docs updated. (#1388) 2021-09-01 12:07:32 +02:00