John fb2a1d42ce

Jj/1798 languages warning (#1805) ...

### Summary
Closes #1798 
Fixes language detection of elements with empty strings: This resolves a
warning message that was raised by `langdetect` if the language was
attempted to be detected on an empty string. Language detection is now
skipped for empty strings.

### Testing
on the main branch this will log the warning "No features in text", but
it will not log anything on this branch.
```
from unstructured.documents.elements import NarrativeText, PageBreak
from unstructured.partition.lang import apply_lang_metadata

elements = [NarrativeText("Sample text."), PageBreak("")]
elements = list(
        apply_lang_metadata(
            elements=elements,
            languages=["auto"],
            detect_language_per_element=True,
        ),
    )
```

### Other
Also changes imports in test_lang.py so imports are explicit

---------

Co-authored-by: cragwolfe <crag@unstructured.io>

2023-10-20 04:15:28 +00:00

..

csv

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

docx

refactoring linting (#1739)

2023-10-17 12:45:12 +00:00

epub

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

markdown

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

msg

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

odt

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

pdf_image

refactoring linting (#1739)

2023-10-17 12:45:12 +00:00

pptx

fix: default to None for the languages metadata field (#1743)

2023-10-14 22:46:24 +00:00

pypandoc

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

utils

feat: shrink bboxes by top left (#1633)

2023-10-06 05:16:11 +00:00

xlsx

fix: stop csv and tsv dropping the first line of the file (#1530)

2023-10-16 17:59:35 -05:00

__init__.py

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

test_api.py

chore: deprecation warning for file_filename (#1191)

2023-08-24 07:02:47 +00:00

test_auto.py

fix: stop csv and tsv dropping the first line of the file (#1530)

2023-10-16 17:59:35 -05:00

test_common.py

chore: process chipper hierarchy (#1634)

2023-10-13 01:28:46 +00:00

test_constants.py

fix: stop csv and tsv dropping the first line of the file (#1530)

2023-10-16 17:59:35 -05:00

test_email.py

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

test_html_partition.py

serde tests round-trip through JSON (#1681)

2023-10-12 19:47:55 +00:00

test_json.py

fix: update test_json to not use auto partition (#1187)

2023-08-29 16:59:26 -04:00

test_lang.py

Jj/1798 languages warning (#1805)

2023-10-20 04:15:28 +00:00

test_strategies.py

Set default strategy for images to be "hi_res" (#968)

2023-08-02 09:22:20 -07:00

test_text_type.py

fix: zero divide in under_non_alpha_ratio (#1753)

2023-10-13 21:20:01 +00:00

test_text.py

fix: default to None for the languages metadata field (#1743)

2023-10-14 22:46:24 +00:00

test_xml_partition.py

detect document language across all partitioners (#1627)

2023-10-11 01:47:56 +00:00