unstructured

mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-23 09:00:40 +00:00

History

Mallori Harrell a5d7ae4611

Feat: Bag of words for testing metric (#1650 )

This PR adds the `bag_of_words` function to count the frequency of words
for evaluation.

**Testing**
```Python
from unstructured.cleaners.core import bag_of_words
string = "The dog loved the cat, but the cat loved the cow."

print(bag_of_words)

---------

Co-authored-by: Mallori Harrell <mallori@Malloris-MacBook-Pro.local>
Co-authored-by: Klaijan <klaijan@unstructured.io>
Co-authored-by: Shreya Nidadavolu <shreyanid9@gmail.com>
Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com>

2023-10-10 18:46:01 +00:00

chunking

chore: adding max_characters to other element type chunking (#1673 )

2023-10-09 19:42:36 +00:00

cleaners

Add clean_ligatures to core cleaners (#1326 )

2023-09-07 21:30:18 +00:00

documents

fix: Fix various cases of HTML text missing after partition (#1587 )

2023-10-03 04:17:51 +00:00

file_utils

Chore: Libmagic detection for "application/octet-stream" when it is not a zip file. (#1347 )