unstructured/typings/nltk/data.pyi

from __future__ import annotations

from typing import Sequence

path: list[str]

def find(resource_name: str, paths: Sequence[str] | None = None) -> str: ...
fix(CVE-2024-39705): remove nltk download (#3361) ### Summary Addresses [CVE-2024-39705](https://nvd.nist.gov/vuln/detail/CVE-2024-39705), which highlights the risk of remote code execution when running `nltk.download` . Removes `nltk.download` in favor of a `.tgz` file with the appropriate NLTK data files and checking the SHA256 hash to validate the download. An error now raises if `nltk.download` is invoked. The logic for determining the NLTK download directory is borrowed from `nltk`, so users can still set `NLTK_DATA` as they did previously. ### Testing 1. Create a directory called `~/tmp/nltk_test`. Set `NLTK_DATA=${HOME}/tmp/nltk_test`. 2. From a python interactive session, run: ```python from unstructured.nlp.tokenize import download_nltk_packages download_nltk_packages() ``` 3. Run `ls /tmp/nltk_test/nltk_data`. You should see the downloaded data. --------- Co-authored-by: Steve Canny <stcanny@gmail.com> 2024-07-08 18:55:36 -04:00			`from __future__ import annotations`

			`from typing import Sequence`

			`path: list[str]`

			`def find(resource_name: str, paths: Sequence[str] \| None = None) -> str: ...`