unstructured

mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-27 10:56:47 +00:00

History

Michał Martyniak 2f25d8f79e

Support for concurrent processing of documents during evaluation (#2973 )

Currently, CCT eval takes a long time for any of the test_metrics CI
runs. Documents in an eval set are evaluated sequentially, and It
appears that a max of 1 cpu core is currently utilized. This implies
there could be a large speedup by running eval across multiple docs
concurrently (probably with multiprocessing).

Things done in this PR:
- [x] concurrent.futures.ProcessPoolExecutor instead of sequential
for-loop
- [x] refactor/reorganization of redundant pieces of code without
changing the inner logic too much. Without that we'd have 3 places where
documents are being processed. Take a look at `BaseMetricsCalculator`
class and classes that inherit from it.
- [x] string paths manipulation is now reworked and relies on
`pathlib.Path()`

2024-05-09 21:25:47 +00:00

__init__.py

rfctr: prepare to add orig_elements serde (#2668 )

2024-03-20 21:27:59 +00:00

test_element_type.py

bug: empty-elements (#1252 )

2023-11-02 10:52:41 -05:00

test_evaluate.py

Support for concurrent processing of documents during evaluation (#2973 )

2024-05-09 21:25:47 +00:00

test_table_formats.py

Save table prediction in cells format (#2892 )