yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2025-07-30 12:22:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,687 Commits 73 Branches 248 Tags
Commit Graph

2 Commits

Author SHA1 Message Date
Julian Risch
90ddeba579
fix: DocumentSplitter and DocumentCleaner copy id_hash_keys to newly created Documents (#6083) 
* copy id_hash_keys in splitter and cleaner

* reno
 2023-10-17 11:03:48 +02:00
Julian Risch
aaee03aee8
feat: Add DocumentCleaner 2.0 (#5976) 
* remove whitespaces, substrings, regex, empty lines

* remove repeated substrings

* reno

* return empty string as shortest common ngram

* address first half of review feedback

* address second half of review feedback

* mention \f page separator for header/footer removal

* mention \f page separator for header/footer removal

* mark example usage as python code
 2023-10-13 12:39:55 +02:00