Steve Canny eb1b022ff8

feat(chunking): add overlap on chunk-splits (#2305) ...

There are two distinct overlap operations with completely different
implementations. This is "intra-chunk" overlap, applying overlap to
chunks resulting from text-splitting an oversized element.

So if an oversized element had text "abcd efgh ijkl mnop qrst" and was
split at 15 chars with overlap of 5, it would produce "abcd efgh ijkl"
and "ijkl mnop qrst". Any inter-chunk overlap from the prior chunk and
applied at the beginning of the string (before "abcd") is handled in a
separate operation in the next PR.

2023-12-22 20:35:18 +00:00

..

__init__.py

rfctr: extract ChunkingOptions (#2266)

2023-12-15 19:51:02 +00:00

test_base.py

feat(chunking): add overlap on chunk-splits (#2305)

2023-12-22 20:35:18 +00:00

test_title.py

rfctr(chunking): split oversized chunks on word boundary (#2297)

2023-12-21 05:45:36 +00:00