Steve Canny 4096a38371

rfctr(chunking): extract chunking-strategy dispatch (#2545) ...

**Summary**
This is the final step in adding pluggable chunking-strategies. It
introduces the `chunk()` function to replace calls to strategy-specific
chunkers in the `@add_chunking_strategy` decorator. The `chunk()`
function then uses a mapping of chunking-strategy names (e.g.
"by_title", "basic") to chunking functions (chunkers) to dispatch the
chunking call. This allows other chunkers to be added at runtime rather
than requiring a code change, which is what "pluggable" chunkers is.

**Additional Information**
- Move the `@add_chunking_strategy` to the new `chunking.dispatch`
module since it coheres strongly with that operation, but publish it
from `chunking(.__init__)` (as it was before) so users don't couple to
the way we organize the chunking sub-package. Also remove the third
level of nesting as it's unrequired in this case.
- Add unit tests for the `@add_chunking_strategy` decorator which was
previously uncovered by any direct test.

2024-03-05 23:19:29 +00:00

..

__init__.py

rfctr: extract ChunkingOptions (#2266)

2023-12-15 19:51:02 +00:00

test_base.py

rfctr(chunking): extract strategy-specific chunking options (#2556)

2024-02-23 18:22:44 +00:00

test_basic.py

rfctr(chunking): extract strategy-specific chunking options (#2556)

2024-02-23 18:22:44 +00:00

test_dispatch.py

rfctr(chunking): extract chunking-strategy dispatch (#2545)

2024-03-05 23:19:29 +00:00

test_title.py

rfctr(chunking): extract strategy-specific chunking options (#2556)

2024-02-23 18:22:44 +00:00