yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2025-06-26 22:00:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,803 Commits 73 Branches 239 Tags
Commit Graph

3803 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Stefano Fiorucci
aca8f09f7d
fix: DOCXToDocument converter - use forward reference to Paragraph (#8260) 
* docx paragraph forward ref

* fix
 2024-08-21 12:37:43 +02:00
Jon Strutz
471f07c8fe
fix: extract page breaks from .docx files (#8232) 
* fix: extract page breaks from .docx files

Context: Currently, DOCXToDocument does not extract page breaks from
word documents. This makes it impossible to do things like split by page
or get correct page number metadata after using something like
DocumentSplitter. For example, if you split by word, the 'page_number'
metadata field will be 1 for all documents.

Solution: Added a method to DOCXToDocument that extracts page breaks
from word documents as '\f' characters so that they are recognized by
DocumentSplitter.

Caveat: Due to the way the python-docx library is set up, you can only
accurately determine the location of the first page break for a given
paragraph. In the rare case that a paragraph contains more than one page
break (which means it is an extremely long paragraph spanning multiple
pages), the 2nd, 3rd, etc. page break locations are not known. To sort
of fix this, I just appended the page break characters to the end of
the paragraph text to keep the overall page number values for the
document consistent.

* Apply suggestions from code review

---------

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
 2024-08-21 09:48:02 +00:00
Sebastian Husch Lee
7fd0b6a013
feat: Add min_top_k to TopPSampler (#8228) 
* Add feature to Top P Sampler

* Add release notes

* Fix zip call

* Fix mypy

* Restore doc string and make mypy happy hopefully

* Make mypy happy

* PR comment

* Revert change to make mypy happy

* Add back type ignore

* try to fix typing

* Update haystack/components/samplers/top_p.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/samplers/top_p.py

---------

Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2024-08-21 11:29:23 +02:00
Daria Fokina
35b1215b00
clean up docstrings: WhisperTranscribers (#8235) 
* clarify docstrings

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-08-16 11:28:42 +00:00
Daria Fokina
bbe18cfdaf
clean up docstrings: DocumentLanguageClassifier (#8215) 
* doclangclass-strings

* simplify sentence

* simplify sentence 2

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-08-16 12:45:54 +02:00
Daria Fokina
4a058032e7
clean up docstrings: TransformersTextRouter (#8229) 
* Update transformers_text_router.py

* article

* article 2
 2024-08-16 12:44:39 +02:00
Daria Fokina
b51bb6e5a9
Update zero_shot_text_router.py (#8231) 2024-08-16 12:43:13 +02:00
Daria Fokina
b5d0bfa9df
Update cache_checker.py (#8237) 2024-08-16 12:22:09 +02:00
Madeesh Kannan
cf5fd2a821
chore: Remove deprecated ChatMessage.to_openai_format (#8242) 
* chore: Remove deprecated `ChatMessage.to_openai_format`

* lint
 2024-08-16 10:34:44 +02:00
Agnieszka Marzec
9427d7aee6
update docstrings (#8225) 2024-08-14 15:33:21 +02:00
Stefano Fiorucci
bcc4104729
refactor: utility function for docstore deserialization (#8226) 
* refactor docstore deserialization

* more tests

* reno; headers

* expose key
 2024-08-14 13:29:27 +02:00
Stefano Fiorucci
109e98aa44
fix: deserialize Document Stores using specific from_dict class methods (#8207) 
* use from_dict

* unused import

* improve logic

* improve reno
 2024-08-14 10:56:32 +02:00
Daria Fokina
5ac56ebdaf
clean up docstrings: ChatPromptBuilder (#8210) 
* chatbuilder-docstrings

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-08-14 08:44:42 +02:00
Daria Fokina
f0816f67ba
Update metadata_router.py (#8217) 2024-08-14 08:40:18 +02:00
Daria Fokina
b5125c73d3
clean up docstrings: TextLanguageRouter (#8219) 
* Update text_language_router.py

* small upd
 2024-08-14 08:04:07 +02:00
David S. Batista
c4be8370c6
doc: adding some more hatch info to CONTRIBUTING.md (#8201) 
* adding some more hatch info to CONTRIBUTING.md

* nit
 2024-08-13 15:36:41 +02:00
Silvano Cerza
ab7eb25856
Add utility then step in feature testing to draw pipeline to file (#8209) 2024-08-13 14:49:13 +02:00
Vladimir Blagojevic
3318d894c0
Add sede_with_list_output_type_in_pipeline unit test (#8196) 2024-08-13 14:37:24 +02:00
Daria Fokina
1284ca285b
clean up docstrings: AzureOpenAIDocumentEmbedder & AzureOpenAITextEmbedder (#8182) 
* clarify docstrings

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-08-13 12:17:47 +00:00
Daria Fokina
e343f8fbd5
clean up docstrings: HuggingFaceAPIDocumentEmbedder & HuggingFaceAPITextEmbedder (#8184) 
* hf-clarify docstrings

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-08-13 12:06:54 +00:00
Daria Fokina
861c470ef6
clean up docstrings: OpenAIDocumentEmbedder (#8186) 
* docstrings upd

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-08-13 12:05:46 +00:00
Daria Fokina
741dd07227
clean up docstrings: TextCleaner (#8202) 
* update textcleaner strings

* Update haystack/components/preprocessors/text_cleaner.py

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-08-13 12:02:58 +00:00
Amna Mubashar
373de97426
Deprecate SentenceWindowRetrieval (#8206) 2024-08-13 13:49:41 +02:00
Haystack Bot
565d802db9
Update unstable version to 2.5.0-rc0 (#8195) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2024-08-12 13:45:59 +02:00
Vladimir Blagojevic
21c507331c
feat: Implement apply_filter_policy and FilterPolicy.MERGE for the new filters (#8042) v2.5.0-rc0 2024-08-09 12:04:24 +02:00
Nicola Procopio
4c798470b2
added precision parameter to sentence transformers embeddings (#8179) 
* added `precision` parameter to sentence transformers embeddings

* fixed test

* Update haystack/components/embedders/sentence_transformers_document_embedder.py

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>

* Update test/components/embedders/test_sentence_transformers_text_embedder.py

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>

* Update test/components/embedders/test_sentence_transformers_text_embedder.py

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>

* fix format

* Update sentence_transformers_text_embedder.py

---------

Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
 2024-08-09 11:38:47 +02:00
Marie-Luise Klaus
ec02817f14
fix: OutputAdapter from_dict with custom_filters None (#8173) 
Co-authored-by: Marie-Luise Klaus <marieluise.klaus@deepset.ai>
 2024-08-08 14:02:40 +02:00
Stefano Fiorucci
a4eb88e7ea
rm serialize callback handler (#8172) 2024-08-08 11:54:31 +02:00
Corentin Meyer
58517014ec
fix: DocumentCleaner: keep the \f in text (#8078) 
* Keep the \f in Document Cleaner

* Add Reno

* Add Test

* Simplified _remove_empty_lines() code
 2024-08-07 14:50:14 +02:00
Marie-Luise Klaus
031b0bfbd8
fix: ChatPromptBuilder from_dict if template is None (#8165) 
* fix ChatPromptBuilder from dict if template=None

* fix ChatPromptBuilder from dict if template=None

* leave template None

---------

Co-authored-by: Marie-Luise Klaus <marieluise.klaus@deepset.ai>
 2024-08-06 14:48:04 +02:00
Tim Wellbrock
2e2f5f17bb
feat: add unicode normalization & ascii_only mode for DocumentCleaner (#8103) 
* feat: add unicode normalization & ascii_only mode for DocumentCleaner.

* feat: add unicode_normalization parameter valdiation to DocumentCleaner.

* test: fix the unit test to work after code linting.
 2024-08-05 13:00:39 +02:00
Stefano Fiorucci
e17d0c4192
chore: deprecate to_openai_format and create similar utility functions (#8146) 
* deprecate and add new specific functions

* reno
 2024-08-02 16:47:17 +02:00
Agnieszka Marzec
c50a60561b
Docs: Update HFAPIGenerator docstrings (#8154) 
* update docstrings

* Update haystack/components/generators/hugging_face_api.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2024-08-02 12:51:07 +00:00
Tobias Wochinger
5a3ea75196
docs: document Python 3.11 and 3.12 support (#8159) 
* docs: add Python 3.11 and 3.12 to supported versions

* docs: add release notes
 2024-08-02 14:46:20 +02:00
Agnieszka Marzec
153afe77c5
update docstrings (#8156) 2024-08-02 14:30:47 +02:00
Agnieszka Marzec
a75dc89690
update docstrings (#8158) 2024-08-02 14:28:59 +02:00
Agnieszka Marzec
0cda96eedf
Docs: Update AzureOpenAIGenerator docstrings (#8149) 
* update docstrings

* Update haystack/components/generators/azure.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/generators/azure.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/generators/azure.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/generators/azure.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2024-08-02 14:26:37 +02:00
Agnieszka Marzec
ba5d105a78
update docstrings (#8150) 2024-08-02 14:25:56 +02:00
Agnieszka Marzec
d441c2faab
Docs: Update HuggingFaceAPIChatGEnertaor docstrings (#8152) 
* update docstrings

* Update haystack/components/generators/chat/hugging_face_api.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/generators/chat/hugging_face_api.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2024-08-02 09:31:59 +00:00
Sebastian Husch Lee
c90495c2e8
feat: Add model and tokenizer kwargs to TransformersSimilarityRanker, SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder (#8145) 
* Start adding model and tokenizer kwargs support

* Add model and tokenizer kwargs to doc embedder

* Some updates and fixes in tests

* Fix more tests

* Fix tests

* Add release note

* Fix test

* Add from_dict tests
 2024-08-02 10:37:10 +02:00
Agnieszka Marzec
d9a7a7a4db
Docs: Update ConditionalRouter docstrings (#8140) 
* update docstrings

* Update haystack/components/routers/conditional_router.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* add reviewer's comments

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2024-08-02 10:34:01 +02:00
Agnieszka Marzec
c670f0fbee
Docs: update SentenceWindowRetriever docstrings (#8138) 
* update docstrings

* Update haystack/components/retrievers/sentence_window_retriever.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

* Update haystack/components/retrievers/sentence_window_retriever.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2024-08-01 18:05:31 +02:00
Agnieszka Marzec
ffbaed85de
update docstrings (#8142) 2024-08-01 16:27:30 +02:00
Agnieszka Marzec
bec822c361
Docs: Update FilterRetriever docstrings (#8133) 
* update docstrings

* Update haystack/components/retrievers/filter_retriever.py

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2024-08-01 13:16:44 +02:00
Vladimir Blagojevic
25d3520f5a
feat: Add AnswerJoiner new component (#8122) 
* Initial AnswerJoiner

* Initial tests

* Add release note

* Resove mypy warning

* Add custom join function

* Serialize custom join function

* Handle all Answer types, add integration test, improve pydoc

* Make fixes

* Add to API docs

* Add more tests

* Update haystack/components/joiners/answer_joiner.py

Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>

* Update docstrings and release notes

* update docstrings

---------

Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
Co-authored-by: Sebastian Husch Lee <sjrl@users.noreply.github.com>
Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com>
Co-authored-by: Darja Fokina <daria.fokina@deepset.ai>
 2024-08-01 12:51:17 +02:00
Stefano Fiorucci
3d1ad10385
fix html test (#8127) 2024-07-31 10:59:53 +02:00
Daria Fokina
bc153c233c
docs: clean up docstrings of TransformersSimilarityRanker (#8124) 
* update transformerssimilarityranker docstrings

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

* upd device param

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-07-31 09:54:32 +02:00
Daria Fokina
ac51885fe8
docs: clean up docstrings of OpenAITextEmbedder (#8120) 
* update docstrings

* update capitalization

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

---------

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2024-07-31 09:53:25 +02:00
Daria Fokina
28141ec6b9
docs: clean up docstrings of OpenAIChatGenerator (#8125) 
* openaichatgen-docstrings

* link update
 2024-07-31 09:45:14 +02:00
Silvano Cerza
c7e29a83c1
fix: Fix infinite loop when running Pipeline (#8123) 
* Fix infinite loop when running Pipeline

* Simplify if
 2024-07-30 15:00:12 +02:00