unstructured/typings/nltk/tokenize.pyi

from __future__ import annotations

def sent_tokenize(text: str, language: str = ...) -> list[str]: ...
def word_tokenize(text: str, language: str = ..., preserve_line: bool = ...) -> list[str]: ...
fix(CVE-2024-39705): remove nltk download (#3361) ### Summary Addresses [CVE-2024-39705](https://nvd.nist.gov/vuln/detail/CVE-2024-39705), which highlights the risk of remote code execution when running `nltk.download` . Removes `nltk.download` in favor of a `.tgz` file with the appropriate NLTK data files and checking the SHA256 hash to validate the download. An error now raises if `nltk.download` is invoked. The logic for determining the NLTK download directory is borrowed from `nltk`, so users can still set `NLTK_DATA` as they did previously. ### Testing 1. Create a directory called `~/tmp/nltk_test`. Set `NLTK_DATA=${HOME}/tmp/nltk_test`. 2. From a python interactive session, run: ```python from unstructured.nlp.tokenize import download_nltk_packages download_nltk_packages() ``` 3. Run `ls /tmp/nltk_test/nltk_data`. You should see the downloaded data. --------- Co-authored-by: Steve Canny <stcanny@gmail.com> 2024-07-08 18:55:36 -04:00			`from __future__ import annotations`

			`def sent_tokenize(text: str, language: str = ...) -> list[str]: ...`
			`def word_tokenize(text: str, language: str = ..., preserve_line: bool = ...) -> list[str]: ...`