haystack/preprocessor-2-0-9828d930562fa3f5.yaml at 35788a2d060d58b901b8d10237f23273399998e5 - haystack - Gitea: Git with a cup of tea

yujunjun/haystack

mirror of https://github.com/deepset-ai/haystack.git synced 2025-07-24 09:20:13 +00:00

Julian Risch 4413675e64

feat: Add TextDocumentSplitter that splits by word, sentence, passage (2.0) (#5870 )

* draft split by word, sentence, passage

* naive way to split sentences without nltk

* reno

* add tests

* make input list of docs, review feedback

* add source_id and more validation

* update docstrings

* add split delimiters back to strings

---------

Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>

2023-09-27 12:26:20 +02:00

5 lines

277 B

YAML

Raw Blame History

 ---
 preview:
   - |
     Add the `TextDocumentSplitter` component for Haystack 2.0 that splits a Document with long text into multiple Documents with shorter texts. Thereby the texts match the maximum length that the language models in Embedders or other components can process.