Steve Canny 4379d883a3

chunk: relax table segregation during chunking (#3812) ...

**Summary**
Relax table-segregation rule applied during chunking such that a `Table`
and `Text`-subtype elements can be combined into a single chunk when the
chunking window allows.

**Additional Context**
Until now, `Table` elements have always been segregated during chunking,
i.e. a chunk that contained a table would never contain any other
element. In certain scenarios, especially when a large chunking window
of say 2000 characters is used, this behavior can reduce retrieval
effectiveness by isolating the table from surrounding context.

---------

Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: scanny <scanny@users.noreply.github.com>

2024-12-09 18:57:22 +00:00

..

handbook-1p.docx.json

chunk: relax table segregation during chunking (#3812)

2024-12-09 18:57:22 +00:00