ZanSara
adf7e49af3
chore: review all extra ( #6029 )
2023-10-12 21:50:53 +02:00
Stefano Fiorucci
2c2549f13d
move embedding backends ( #6033 )
2023-10-12 17:52:28 +02:00
Vladimir Blagojevic
d51be9edac
Add top_k to SimilarityRanker ( #6036 )
2023-10-12 13:52:01 +02:00
Vladimir Blagojevic
3803d23ff6
feat: Update PyPDFToDocument to process ByteStream inputs ( #6021 )
...
* Update PyPDF converter
* Add mixed source unit test
* Update haystack/preview/components/file_converters/pypdf.py
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-10-11 10:52:08 +02:00
Vladimir Blagojevic
1a6a8863e8
feat: Update HTMLToDocument to handle ByteStream inputs ( #6020 )
...
* Update HTML converter
* Add mixed source unit test
* Update haystack/preview/components/file_converters/html.py
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-10-11 10:15:58 +02:00
Vladimir Blagojevic
6a50123b9f
feat: Adjust LinkContentFetcher run method, use ByteStream ( #5972 )
2023-10-10 17:48:31 +02:00
Vladimir Blagojevic
98215aec0d
feat: Rename FileExtensionRouter to FileTypeRouter, handle ByteStream(s) ( #5998 )
...
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2023-10-10 09:14:04 +02:00
DanShatford
07048791aa
feat: allow list of file paths in convert_files_to_docs ( #5961 )
...
* feat: allow list of file paths in `convert_files_to_docs`
* Fix validation
* Fix check errors
2023-10-09 20:19:03 +02:00
Vladimir Blagojevic
40b83d8a47
feat: Add TopPSampler Haystack 2.0 component ( #5924 )
2023-10-09 13:44:01 +02:00
Vladimir Blagojevic
1cdff6427e
feat: Add SimilarityRanker to Haystack 2.0 ( #5923 )
...
* Initial SimilarityRanker
2023-10-06 16:01:34 +02:00
Stefano Fiorucci
ccc9f010bb
fix: fix ChatGPT invocation layer (and add async support) ( #5979 )
...
* ChatGPT async
* release note
* fix tests
2023-10-05 18:43:26 +02:00
Vladimir Blagojevic
282419d82b
feat: Unfreeze Document in Haystack 2.0 ( #5974 )
...
* Unfreeze document
* Remove immutability test
2023-10-05 17:55:07 +02:00
Tobias Wochinger
d5d3a9eef4
chore: adapt deepset cloud sdk endpoint format for saving pipelines ( #5969 )
...
* chore: adapt to new endpoints formats
* docs: add release notes
2023-10-05 08:56:28 +02:00
Massimiliano Pippi
c2ec3f5fde
feat: add File type to preview package ( #5873 )
...
* add Blob type
* review feedback
* fix tests and naming
* Update add-blob-type-2a9476a39841f54d.yaml
* removed unused import
---------
Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
2023-10-04 17:23:12 +02:00
Stefano Fiorucci
cc70b4b613
deprecation ( #5954 )
2023-10-03 12:48:06 +02:00
Massimiliano Pippi
ac408134f4
feat: add support for async openai calls ( #5946 )
...
* add support for async openai calls
* add actual async call
* split the async api
* ask permission
* Update haystack/utils/openai_utils.py
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
* Fix OpenAI content moderation tests
* Fix ChatGPT invocation layer tests
---------
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
2023-10-03 10:42:21 +02:00
Massimiliano Pippi
0947f59545
feat: add async PromptNode run ( #5890 )
...
* add async promptnode
* Remove unecessary calls to dict.keys()
---------
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-09-29 08:40:01 +02:00
Vladimir Blagojevic
e882a7d5c8
feat: Add HTMLToDocument component (v2) ( #5907 )
2023-09-28 17:22:28 +02:00
Stefano Fiorucci
d4aacad5f9
feat: OpenAIDocumentEmbedder ( #5822 )
...
* first draft
* release note
* mypy fix
* fix test
* corrections
* pr feedback
* better secrets handling and new tests
* missing imports in embedders/__init__.py
* better format condition
* address feedback
2023-09-28 15:42:51 +02:00
ZanSara
83724b74e3
feat: Make metadata optional in AnswerBuilder ( #5909 )
...
* optional metadata
* improve docstring
2023-09-28 14:42:19 +02:00
Stefano Fiorucci
9340c572f9
alternative skipif conditions in azure ocr converter test ( #5906 )
2023-09-28 12:09:19 +02:00
Julian Risch
4413675e64
feat: Add TextDocumentSplitter that splits by word, sentence, passage (2.0) ( #5870 )
...
* draft split by word, sentence, passage
* naive way to split sentences without nltk
* reno
* add tests
* make input list of docs, review feedback
* add source_id and more validation
* update docstrings
* add split delimiters back to strings
---------
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2023-09-27 12:26:20 +02:00
bogdankostic
80192589b1
feat: Add AzureOCRDocumentConverter (2.0) ( #5855 )
...
* Add AzureOCRDocumentConverter
* Add tests
* Add release note
* Formatting
* update docstrings
* Apply suggestions from code review
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* PR feedback
* PR feedback
* PR feedback
* Add secrets as environment variables
* Adapt test
* Add azure dependency to CI
* Add azure dependency to CI
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2023-09-26 15:57:55 +02:00
Stefano Fiorucci
6aa471ac5e
chore: make preview integration tests reproducible ( #5871 )
...
* relax extractive reader integration tests
* force reader to CPU
* ensure integration tests reproducibility
* move set_all_seeds to testing package
2023-09-25 18:39:10 +02:00
bogdankostic
9a4373bf8e
feat: Add TikaDocumentConverter (2.0) ( #5847 )
...
* Add TikaFileToDocument component
* Add tests
* Add tika service to CI
* Add release note
* Change name
* PR feedback
* Fix naming in tests
* Fix tika version in CI
* Update tests
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-25 11:47:21 +02:00
MichelBartels
4da43b6b05
Add link output to SerperDevWebSearch ( #5853 )
...
* add link output
* adjust tests
* fix test
* remove print statements
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-25 10:03:01 +02:00
Stefano Fiorucci
c0f22372d4
feat: OpenAITextEmbedder ( #5801 )
...
* first draft
* release notes
* avoid serializing secrets
* fix import order
* simplify serialization
* simplification
* monkeypatch delenv
* Update haystack/preview/components/embedders/openai_text_embedder.py
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
* docstrings updates
* fix test
* Update haystack/preview/components/embedders/openai_text_embedder.py
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* rm comment
---------
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-09-22 21:54:11 +02:00
Massimiliano Pippi
a5a0dc9f87
feat: optionally pass an id to the Document constructor ( #5862 )
...
* revert #5826
* do not use Optional
2023-09-22 11:09:59 +02:00
Silvano Cerza
cc4f95bf51
Remove unnecessary GPT4Generator class ( #5863 )
...
* Remove GPT4Generator class
* Rename GPT35Generator to GPTGenerator
* Fix tests
* Release notes
2023-09-22 11:05:06 +02:00
MichelBartels
f3dc9edd26
feat: initial ExtractiveReader implementation ( #5553 )
...
* initial ExtractiveReader implementation
* initial ExtractiveReader implementation
* fix mypy
* remove unused import
* Use AutoTokenizer
* rename reader to model
* combine no-answer logit
* support document slicing with proper probabilities
* add variable stride
* validate model
* fix typo
* make postprocessing easier to understand
* remove debug code
* set default reader
* add ExtractiveReader to __init__
* remove validation
* use new answer class
* add batching
* use v2 lazy imports
* move reader
* fix type hints
* add doc strings
* add nucleus sampling
* fix types
* fix doc string
* add no_answer parameter
* remove print statement
* fix gpu support
* turn into binary classification task
* change dataclass so document does not need to be provided for no answer
* add simple tests
* add unit tests
* rename reader folder to readers
* add integration tests
* fix type hints
* add release notes
* remove accidentally included test file
* remove unnecessary __init__ file
* revert __init__ file to main
* rename test script by adding test_ prefix
* undo accidentally moving of test script after renaming it
* remove use of bisect
* rename _flatten and _unflatten
* make variable name more intuitive
* remove type: ignore
* fix mypy issue
* refactor long tuple
* add doc strings
* explain HF test
* remove unnecessary top_k check
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-21 12:16:51 +02:00
Vladimir Blagojevic
92a6221927
feat: Add PyPDFToDocument component (2.0) ( #5850 )
...
* Initial PyPDFToDocument implementation
* Remove progress bar
* Add release note
* Minor fix
* import check and dependency
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-21 11:52:26 +02:00
ZanSara
23fdef929e
chore: move GPT35Generator tests in the main test suite ( #5844 )
...
* move tests
* fix no-test-found error from pytest
* missing self
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-09-21 11:42:32 +02:00
ZanSara
28f5c4c780
fix: Whisper integration tests ( #5851 )
...
* fix tests
* add ffmpeg
* apt update for ffmpeg
* not run on windows
2023-09-21 00:14:07 +02:00
bogdankostic
abe2706298
feat: Add MetadataRouter (2.0) ( #5824 )
...
* Move filter utilities
* Add MetadataRouter
* Add tests for MetadataRouter
* Add more tests
* Rename FileExtensionClassifer to FileExtensionRouter
* Add support for dates in filters
* Add tests
* Add release note
* Add release note
* Apply suggestions from code review
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-20 14:49:17 +02:00
ZanSara
c933bcaa69
chore: move Whisper e2e tests in the main tests suite ( #5845 )
...
* move whisper local tests
* remove e2e file
* move remote tests
* remove e2e file
2023-09-20 14:48:09 +02:00
ZanSara
454988672e
feat: UrlCacheChecker ( #5841 )
...
* add UrlCacheChecker
* rename
* add tests
* reno
* pylint
* review feedback
2023-09-20 14:45:50 +02:00
bogdankostic
719c1c040c
feat: Add support for dates in filters (2.0) ( #5823 )
...
* Add support for dates in filters
* Add tests
* Add release note
* Update haystack/preview/utils/filters.py
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-20 12:05:56 +02:00
ZanSara
44f0c468ac
move websearch tests back to main tests suite ( #5842 )
2023-09-20 11:55:18 +02:00
Vladimir Blagojevic
0983fb656a
feat: Add LinkContentFetcher Haystack 2.0 component ( #5724 )
...
* Add LinkContentFetcher
* Add release note
* Small fixes
* Fix pydocs
* PR feedback
* Remove handlers registration
* PR feedback
* adjustments
* improve tests
* initial draft
* tests
* add proposal
* proposal number
* reno
* fix tests and usage of content and content_type
* update branch & fix more tests
* mypy
* use the new document
* add docstring
* fix more tests
* mypy
* fix tests
* add e2e
* review feedback
* improve __str__
* Apply suggestions from code review
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
* Update haystack/preview/dataclasses/document.py
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
* improve __str__
* fix tests
* fix more tests
* fix test
* Fix end-of-file-fixer
* Post merge fixes
* Move e2e tests back into component
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2023-09-20 11:03:52 +02:00
Christian Clauss
bf6d306d68
ci: Simplify Python code with ruff rules SIM ( #5833 )
...
* ci: Simplify Python code with ruff rules SIM
* Revert #5828
* ruff --select=I --fix haystack/modeling/infer.py
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-09-20 08:32:44 +02:00
Stefano Fiorucci
de84a95970
separate classes and tests ( #5819 )
...
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-09-19 19:21:49 +02:00
Malte Pietsch
aa3cc3d5ae
feat: Add support for OpenAI's gpt-3.5-turbo-instruct model ( #5837 )
...
* support gpt-3.5.-turbo-instruct
* add release note
2023-09-19 16:06:43 +02:00
Christian Clauss
91ab90a256
perf: Python performance improvements with ruff C4 and PERF fixes ( #5803 )
...
* Python performance improvements with ruff C4 and PERF
* pre-commit fixes
* Revert changes to examples/basic_qa_pipeline.py
* Revert changes to haystack/preview/testing/document_store.py
* revert releasenotes
* Upgrade to ruff v0.0.290
2023-09-16 16:26:07 +02:00
Christian Clauss
1bc03ddc73
ci: Fix all ruff pyflakes errors except unused imports ( #5820 )
...
* ci: Fix all ruff pyflakes errors except unused imports
* Delete releasenotes/notes/fix-some-pyflakes-errors-69a1106efa5d0203.yaml
2023-09-15 18:30:33 +02:00
Silvano Cerza
5c04cd6ba2
Fix Document constructor accepting unused id parameter ( #5826 )
2023-09-15 17:03:03 +02:00
Christian Clauss
9405eb90ee
ci: Fix invalid escape sequences in Python code ( #5802 )
...
* ci: Use ruff in pre-commit to further limit complexity
* Fix invalid escape sequences in Python code
* Delete releasenotes/notes/ruff-4d2504d362035166.yaml
2023-09-14 16:42:48 +02:00
Stefano Fiorucci
1c69070db6
make MemoryEmbeddingRetriever act in non-batch mode ( #5809 )
2023-09-14 15:37:20 +02:00
Stefano Fiorucci
ad5b615503
make SentenceTransformersTextEmbedder non batch ( #5811 )
2023-09-14 12:38:24 +02:00
Ivana Zeljkovic
4bad202197
feat: Pinecone document store refactoring ( #5725 )
...
* Refactor codebase so that doc_type metadata is used instead of namespaces for making distinction between documents without embeddings, documents with embeddings and labels
* Fix parameter name in integration test
* Remove code under comment in add_type_metadata_filter method
* Fix mypy and pylint checks
* Add release note
* Apply minimal changes: rename method, update method docs and remove redundant method
* Mypy fixes
* Fix docstrings
* Revert helper methods for fetching documents when the number of documents exceeds Pinecone limit
* Remove unnecessary attributes in PineconeDocumentStore
* Fix unit test
---------
Co-authored-by: Ivana Zeljkovic <ivana.zeljkovic@smartcat.io>
Co-authored-by: DosticJelena <jelena.dostic@smartcat.io>
2023-09-14 11:46:47 +02:00
Christian Clauss
6dd52d91b2
ci: Fix typos discovered by codespell ( #5778 )
...
* Fix typos discovered by codespell
* pylint: max-args = 38
2023-09-13 16:14:45 +02:00
