yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2025-10-22 21:39:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,115 Commits 86 Branches 260 Tags
Commit Graph

94 Commits

Author SHA1 Message Date
Madeesh Kannan
e6d6ce1c73
feat: Add NamedEntityExtractorcomponent (#6689) 
* feat: Add `NamedEntityExtractor`component

This component accepts a list of `Document`s which it annotates with named entities. The annotations are stored in the `meta` dictionary of each `Document` under a specific key.

The component currently support two backends for the annotation models: Hugging Face `transformers` and spaCy.

* Address comments

* Expand release note

* Add the `[torch]` extra package specifier to the lazy import

* Remove dead code

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
 2024-01-09 17:56:20 +01:00
ZanSara
abd16ab796
feat: support single metadata dictionary in MarkdownToDocument (#6629) 
* support single metadata dict in markdown2document

* reno

* unwrap list

* direct key access

* typing

* add explicit test
 2024-01-09 14:44:39 +01:00
ZanSara
175b5baf45
feat: support single metadata dictionary in AzureOCRDocumentConverter (#6635) 
* support single metadata dict in azureconverter

* reno

* tests

* Update releasenotes/notes/single-meta-in-azureconverter-ce1cc196a9b161f3.yaml
 2024-01-09 10:49:37 +01:00
ZanSara
974d65f30a
feat: support single metadata dictionary in TikaDocumentConverter (#6698) 
* reno

* converter

* test

* comment
 2024-01-09 09:49:47 +01:00
Massimiliano Pippi
93b2aaee09
chore: move DocumentJoiner to new joiners package (#6692) 
* move DocumentJoiner to new joiners package

* relnote

* leftovers

* fix docstrings generation

* fix unrelated pydoc misconfiguration

* more unrelated work, yay!

* fix assertions
 2024-01-08 22:06:27 +01:00
Silvano Cerza
9445b2d466
Fix skipif with empty env var (#6704) 2024-01-08 19:19:14 +01:00
Silvano Cerza
607e7d1488
Skip integration tests if env var is missing (#6703) 2024-01-08 17:15:10 +01:00
Vladimir Blagojevic
9e0b58784f
feat: Improve UrlCacheChecker, make it more generic (#6699) 
* Rename UrlCacheChecker to CacheChecker, make it field generic

* Add release note
 2024-01-08 16:15:27 +01:00
Sebastian Husch Lee
beade1cef9
feat: Add scaling and thresholding of the similarity ranker scores (#6683) 
* Add scale_score functionality to the TransformersSimilarityRanker

* Updated test to check scores

* Use pytest approx when comparing floats

* Updated how scale score works and added calibration factor. Started to add score threshold.

* Add support for score_threshold

* Add some parameters to the run method

* Add release notes

* Fix mypy

* Be more tolerant on the score values

* Adding unit test for scale_score=False

* Add unit test for score threshold

* Update tests

* Rename test

* Fix typo

* PR comments
 2024-01-08 09:05:24 +01:00
Vladimir Blagojevic
552f0e394b
feat: Add Azure embedders support (#6676) 
* Add Azure embedders
---------

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
 2024-01-05 15:49:25 +01:00
Vladimir Blagojevic
b7159ad7c2
feat: Add AzureOpenAIGenerator and AzureOpenAIChatGenerator (#6648) 
* Add AzureOpenAIGenerator and AzureOpenAIChatGenerator
 2024-01-05 15:48:28 +01:00
Stefano Fiorucci
bb2b1a20f8
refactor: optimize API keys reading (#6655) 
* centralize API keys handling

* fix mypy and pylint

* rm utility function, be more explicit
 2024-01-05 10:40:03 +01:00
Vladimir Blagojevic
1336456b4f
Update prompt builders examples (#6681) 2024-01-04 16:54:26 +01:00
Vladimir Blagojevic
090d66b531
feat: Update OpenAIChatGenerator to handle both tools and functions calling (#6639) 
* Handle tools parameter in OpenAIChatGenerator

* Handle tools/functions parameter in OpenAIChatGenerator streaming mode

* Adjust OpenAPIServiceConnector to handle tools parameter

* We never deal with functions/tools in non-chat generator

* Add release note
 2023-12-28 17:29:47 +01:00
Stefano Fiorucci
c773c30c66
refactor!: rename all remaining metadata to meta (#6650) 
* change metadata to meta

* release note
 2023-12-28 12:18:15 +01:00
Vladimir Blagojevic
ef2f6bd681
feat: Split DynamicPromptBuilder and DynamicChatPromptBuilder (#6557) 
* Split DynamicPromptBuilder

* Add release note

* Julian PR feedback

* dynamicchatbuilder lg upd

* dynamicpromptbuilder lg upd

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2023-12-26 15:27:43 +01:00
Vladimir Blagojevic
506ab81d26
chore: Rename GPT generators, deprecate old names (#6626) 2023-12-22 19:37:29 +01:00
ZanSara
c0f1dab454
feat: support single metadata dictionary in PyPDFToDocument (#6615) 
* support single metadata dict in pypdf2document

* improve tests

* tests

* remove line
 2023-12-22 14:13:11 +01:00
ZanSara
ff55985e2d
feat: support single metadata dictionary in HTMLToDocument (#6613) 
* support single metadata in HTMLToDocument

* reno

* docstring
 2023-12-21 16:45:31 +01:00
Vladimir Blagojevic
4d08be0c2a
feat: Update OpenAI Python Client in Haystack 2.x (#6584) 
* Update openai python client

* Add release note

* Consolidate multiple mock_chat_completion into one

* Ensure all components have api_base_url, organization params

* Update tests

* Enable function calling

* Oversight

* Minor fixes, add streaming test mocks

* Apply suggestions from code review

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* metadata -> meta

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2023-12-21 16:21:24 +01:00
ZanSara
cf79aa1485
feat: add support for single meta dict in TextFileToDocument (#6606) 
* add support for single meta dict

* reno

* reno

* mypy

* extract to function

* docstring

* mypy
 2023-12-21 14:21:17 +01:00
Stefano Fiorucci
7cc6080dfa
chore: replace metadata w meta in tests/examples (#6612) 
* replace metadata w meta in tests/examples

* do not touch already broken e2e tests

* Revert "do not touch already broken e2e tests"

This reverts commit 1f911920d98954b57daacfe8d8ed02fd77d136db.
 2023-12-21 14:09:31 +01:00
sahusiddharth
3d17e6ff76
changed metadata to meta (#6605) 2023-12-21 12:39:58 +01:00
Ashwin Mathur
fc88ef7076
feat: Add HuggingFace TEI Embedders - HuggingFaceTEITextEmbedder and HuggingFaceTEIDocumentEmbedder (#6602) 
* Add TEI Embedders

* Add release notes

* Update release notes with usage examples
 2023-12-21 12:16:36 +01:00
ZanSara
5a0f0ce22f
feat: Multiplexer (#6592) 
* move functions

* tests

* reno

* add component

* reno

* add tests

* mypy

* pylint

* logger

* module name
 2023-12-20 11:03:22 +01:00
Silvano Cerza
f224f991be
Change DocumentWriter default policy from DuplicatePolicy.FAIL to DuplicatePolicy.NONE (#6596) 2023-12-19 17:46:16 +01:00
ZanSara
f877704839
chore: extract type serialization (#6586) 
* move functions

* tests

* reno
 2023-12-19 14:16:20 +01:00
Vladimir Blagojevic
2dd5a94b04
feat: Add RAG based OpenAPI service integration (#6555) 
* Add OpenAPIServiceConnector and OpenAPIServiceToFunctions

* Add release note

* Add test deps

* Better docs on OpenAPI spec reqs, improve tests

* Silvano PR feedback
 2023-12-19 13:27:41 +01:00
Stefano Fiorucci
94cfe5d9ae
feat!: HTMLToDocument - allow choosing the boilerpy3 extractor (#6582) 
* allow extractor customizability

* release note

* typo
 2023-12-19 10:52:12 +01:00
Sebastian Husch Lee
dcf37c5173
feat: Extractive QA answer deduplication (#6459) 
* Add answer deduplication

* Fix test

* Handle None case

* Release notes

* Handle cases where documents or answer spans could be None

* Adding checks for Nones and satisfying mypy

* Add option to turn off deduplication

* Adding unit tests

* Refactored tests to use fixtures

* Added overlap_threshold to run

* Update test

* Fixes related to the merge

* Remove casting, use direct variable names

* Move out if statement and add new test for it

* Update if statement to match comment

* Update how if statements work
 2023-12-18 19:27:04 +01:00
Sebastian Husch Lee
c294b8ac8c
feat: Add auto device checks and model_kwargs to TransformersSimilarityRanker (#6561) 
* Add device checking and model_kwargs like we do in ExtractiveReader

* Add release notes

* Make a utility function for the device checking

* Better warning message and updated ExtractiveReader to use the util function

* Add unit tests for get_device

* Fix pylint
 2023-12-18 15:13:42 +01:00
Sebastian Husch Lee
3e0e81b1e0
feat: Add meta_fields_to_embed to TransformersSimilarityRanker (#6564) 
* Add initial implementation following SentenceTransformersDocumentEmbedder

* Add test for embedding metadata

* Add release notes

* Update name

* Fix tests and to dict

* Fix release notes
 2023-12-18 11:28:16 +01:00
Massimiliano Pippi
0ac1bdc6a0
refactor!: uniform run api for LocalWhisperTranscriber (#6542) 
* uniform run api for LocalWhisperTranscriber

* add relnote

* fix linter
 2023-12-18 10:47:46 +01:00
Stefano Fiorucci
2f034d3c97
refactor!: Converters - standardize inputs (#6540) 
* standardize converters inputs: first draft

* fix precommit

* fix precommit 2

* fix precommit 3

* add default for optional param

* rm leftover

* install boilerpy in linting workflow

* add boilerpy3 to the core dependencies

* add reno

* remove boilerpy3 installation from test workflow

* fix pylint: import order and unused import

* fix import order

* add release note

* better Tika docstring

* rm boilerpy from linting

* leftover

* md link brackets

* feat: Converters - allow passing `meta` in the `run` method (#6554)

* first impl for html

* progressing on other components

* fix test

* add tests - run with meta

* release note

* reintroduce patches wrongly deleted

* add patch in test

* fix tika test

* Update haystack/components/converters/azure.py

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

* Update releasenotes/notes/converters-standardize-inputs-ed2ba9c97b762974.yaml

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* simplify test

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
 2023-12-15 16:41:35 +01:00
Vladimir Blagojevic
c642695ec0
feat: Add FileTypeRouter markdown support (#6551) 
* Add FileTypeRouter markdown support

* Add releae note
 2023-12-14 16:30:57 +01:00
Julian Risch
25a6eaae05
feat!: Rename ExtractiveReader's confidence_threshold to score_threshold (#6532) 
* rename to score_threshold

* Update haystack/components/readers/extractive.py

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>

---------

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
 2023-12-12 15:12:28 +01:00
Silvano Cerza
18dbce25fc
refacotr: Refactor answer dataclasses (#6523) 
* Refactor answer dataclasses

* Add release notes

* Fix tests

* Fix end to end tests

* Enhance ExtractiveReader
 2023-12-11 18:50:49 +01:00
bogdankostic
728383a149
fix: Make TransformersSimilarityRanker run with single document list (#6503) 
* Make `TransformersSimilarityRanker` run with single document list

* Add release note

* Remove unused import in test
 2023-12-08 16:18:46 +01:00
Bijay Gurung
c5342d1110
fix: Prevent invalid answer from being selected in ExtractiveReader (#6460) 
* Fix invalid answer being selected issue on ExtractiveReader

* Rename variables to not shadow arguments
 2023-12-06 09:49:02 +01:00
Stefano Fiorucci
4912f7cb58
refactor!: improve the deserialization logic for components that use a Document Store (#6466) 
* improve deserialization

* rm ds decorator

* improve tests

* fix pylint

* rm decorator from module init

* rm decorator

* rm decorator from factory

* fix tests

* release note

* rm print
 2023-12-04 15:17:28 +01:00
Massimiliano Pippi
a86807b834
move Cohere generator into dedicated integration (#6475) 2023-12-04 11:16:12 +01:00
Massimiliano Pippi
7c05f37a53
remove unit marker (#6450) 2023-11-29 19:24:25 +01:00
Silvano Cerza
e6637f5ec2 Fix all tests 2023-11-24 14:48:43 +01:00
Massimiliano Pippi
8adb8bbab8
Remove preview folder in test/ 
---------

Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
 2023-11-24 11:52:55 +01:00