The `@apply_metadata` decorator already contains logic to detect the
language of the element text (on either a document or element level).
Update pdfs, and later images, to use this decorator to get accurate
element language results outputted.
Test
```
from unstructured.partition.auto import partition
def test_partition_pdf():
pdf_path = "example-docs/language-docs/fr_olap.pdf"
elements = partition(pdf_path) # optionally set `detect_language_per_element=True)`
print(f"Number of elements partitioned: {len(elements)}")
# Check if elements are returned
assert len(elements) > 0, "No elements were partitioned from the PDF."
# check language outputted for each element
for element in elements:
print(element)
print(element.metadata.languages)
print("-------------------------------")
test_partition_pdf()
```
---------
Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: shreyanid <shreyanid@users.noreply.github.com>
The sample docs directory contains the following files:
example-10k.html - A 10-K SEC filing in HTML format
layout-parser-paper.pdf - A PDF copy of the layout parser paper
factbook.xml/factbook.xsl - Example XML/XLS files that you can use to test stylesheets
These documents can be used to test out the parsers in the library. In addition, here are
instructions for pulling in some sample docs that are too big to store in the repo.
XBRL 10-K
You can get an example 10-K in inline XBRL format using the following curl. Note, you need
to have the user agent set in the header or the SEC site will reject your request.
curl -O \
-A '${organization} ${email}'
https://www.sec.gov/Archives/edgar/data/311094/000117184321001344/0001171843-21-001344.txt
You can parse this document using the HTML parser.
344202fa6d