2 Commits

Author SHA1 Message Date
Dayenne Souza
ad5b5120ec
remove unused columns and rename document_attribute_columns (#1672)
* remove unused columns and change property document_attribute_columns to metadata

* format file

* fix 'metadata' column on output

* run check

* fix test on nltk

* remove docs changes
2025-02-03 14:37:06 -03:00
Dayenne Souza
2f2cfa7b70
Test and unify text splitter functionality (#1547)
* add text_splitting unit test

* change folder test text splitting

* fix chunk fn

* test new function

* run formatter

* run spell check

* run semver

* remove tiktoken mocked from tests

* change progress ticker

* fix ruff check
2025-01-13 18:42:44 -03:00