mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-10-31 10:03:07 +00:00
9500d04791
### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
114 lines
7.0 KiB
Plaintext
114 lines
7.0 KiB
Plaintext
MIME-Version: 1.0
|
|
Date: Wed, 4 Oct 2023 09:27:45 -0500
|
|
Message-ID: <CABDvgF2Wpt9+eSO7zgMJZ2fQb=QZ__CS6N_Y+msnGwpKeg1a+A@mail.gmail.com>
|
|
Subject: Test email with multiple languages
|
|
From: John <johnjennings702@gmail.com>
|
|
To: John <johnjennings702@gmail.com>
|
|
Content-Type: multipart/alternative; boundary="000000000000a0666d0606e4cfb7"
|
|
|
|
--000000000000a0666d0606e4cfb7
|
|
Content-Type: text/plain; charset="UTF-8"
|
|
Content-Transfer-Encoding: quoted-printable
|
|
|
|
All human beings are born free and equal in dignity and rights. They are
|
|
endowed with reason and conscience and should act towards one another in a
|
|
spirit of brotherhood. All human beings are born free and equal in dignity
|
|
and rights. They are endowed with reason and conscience and should act
|
|
towards one another in a spirit of brotherhood. All human beings are born
|
|
free and equal in dignity and rights. They are endowed with reason and
|
|
conscience and should act towards one another in a spirit of brotherhood.
|
|
All human beings are born free and equal in dignity and rights. They are
|
|
endowed with reason and conscience and should act towards one another in a
|
|
spirit of brotherhood. All human beings are born free and equal in dignity
|
|
and rights. They are endowed with reason and conscience and should act
|
|
towards one another in a spirit of brotherhood. All human beings are born
|
|
free and equal in dignity and rights. They are endowed with reason and
|
|
conscience and should act towards one another in a spirit of brotherhood.
|
|
|
|
All human beings are born free and equal in dignity and rights. They are
|
|
endowed with reason and conscience and should act towards one another in a
|
|
spirit of brotherhood. "Todos los seres humanos nacen libres e iguales en
|
|
dignidad y derechos y, dotados como est=C3=A1n de raz=C3=B3n y conciencia, =
|
|
deben
|
|
comportarse fraternalmente los unos con los otros. Todos los seres humanos
|
|
nacen libres e iguales en dignidad y derechos y, dotados como est=C3=A1n de
|
|
raz=C3=B3n y conciencia, deben comportarse fraternalmente los unos con los
|
|
otros."
|
|
|
|
All human beings are born free and equal in dignity and rights. They are
|
|
endowed with reason and conscience and should act towards one another in a
|
|
spirit of brotherhood. All human beings are born free and equal in dignity
|
|
and rights. They are endowed with reason and conscience and should act
|
|
towards one another in a spirit of brotherhood. All human beings are born
|
|
free and equal in dignity and rights. They are endowed with reason and
|
|
conscience and should act towards one another in a spirit of brotherhood.
|
|
All human beings are born free and equal in dignity and rights. They are
|
|
endowed with reason and conscience and should act towards one another in a
|
|
spirit of brotherhood.
|
|
|
|
All human beings are born free and equal in dignity and rights. They are
|
|
endowed with reason and conscience and should act towards one another in a
|
|
spirit of brotherhood. All human beings are born free and equal in dignity
|
|
and rights. They are endowed with reason and conscience and should act
|
|
towards one another in a spirit of brotherhood. All human beings are born
|
|
free and equal in dignity and rights. They are endowed with reason and
|
|
conscience and should act towards one another in a spirit of brotherhood.
|
|
|
|
"Todos los seres humanos nacen libres e iguales en dignidad y derechos y,
|
|
dotados como est=C3=A1n de raz=C3=B3n y conciencia, deben comportarse frate=
|
|
rnalmente
|
|
los unos con los otros. Todos los seres humanos nacen libres e iguales en
|
|
dignidad y derechos y, dotados como est=C3=A1n de raz=C3=B3n y conciencia, =
|
|
deben
|
|
comportarse fraternalmente los unos con los otros."
|
|
|
|
--000000000000a0666d0606e4cfb7
|
|
Content-Type: text/html; charset="UTF-8"
|
|
Content-Transfer-Encoding: quoted-printable
|
|
|
|
<div dir=3D"ltr">All human beings are born free and equal in dignity and ri=
|
|
ghts. They are endowed with reason and conscience and should act towards on=
|
|
e another in a spirit of brotherhood. All human beings are born free and eq=
|
|
ual in dignity and rights. They are endowed with reason and conscience and =
|
|
should act towards one another in a spirit of brotherhood. All human beings=
|
|
are born free and equal in dignity and rights. They are endowed with reaso=
|
|
n and conscience and should act towards one another in a spirit of brotherh=
|
|
ood. All human beings are born free and equal in dignity and rights. They a=
|
|
re endowed with reason and conscience and should act towards one another in=
|
|
a spirit of brotherhood. All human beings are born free and equal in digni=
|
|
ty and rights. They are endowed with reason and conscience and should act t=
|
|
owards one another in a spirit of brotherhood. All human beings are born fr=
|
|
ee and equal in dignity and rights. They are endowed with reason and consci=
|
|
ence and should act towards one another in a spirit of brotherhood.<br><br>=
|
|
All human beings are born free and equal in dignity and rights. They are en=
|
|
dowed with reason and conscience and should act towards one another in a sp=
|
|
irit of brotherhood. "Todos los seres humanos nacen libres e iguales e=
|
|
n dignidad y derechos y, dotados como est=C3=A1n de raz=C3=B3n y conciencia=
|
|
, deben comportarse fraternalmente los unos con los otros. Todos los seres =
|
|
humanos nacen libres e iguales en dignidad y derechos y, dotados como est=
|
|
=C3=A1n de raz=C3=B3n y conciencia, deben comportarse fraternalmente los un=
|
|
os con los otros."<br><br>All human beings are born free and equal in =
|
|
dignity and rights. They are endowed with reason and conscience and should =
|
|
act towards one another in a spirit of brotherhood. All human beings are bo=
|
|
rn free and equal in dignity and rights. They are endowed with reason and c=
|
|
onscience and should act towards one another in a spirit of brotherhood. Al=
|
|
l human beings are born free and equal in dignity and rights. They are endo=
|
|
wed with reason and conscience and should act towards one another in a spir=
|
|
it of brotherhood. All human beings are born free and equal in dignity and =
|
|
rights. They are endowed with reason and conscience and should act towards =
|
|
one another in a spirit of brotherhood.<br><br>All human beings are born fr=
|
|
ee and equal in dignity and rights. They are endowed with reason and consci=
|
|
ence and should act towards one another in a spirit of brotherhood. All hum=
|
|
an beings are born free and equal in dignity and rights. They are endowed w=
|
|
ith reason and conscience and should act towards one another in a spirit of=
|
|
brotherhood. All human beings are born free and equal in dignity and right=
|
|
s. They are endowed with reason and conscience and should act towards one a=
|
|
nother in a spirit of brotherhood.<br><br>"Todos los seres humanos nac=
|
|
en libres e iguales en dignidad y derechos y, dotados como est=C3=A1n de ra=
|
|
z=C3=B3n y conciencia, deben comportarse fraternalmente los unos con los ot=
|
|
ros. Todos los seres humanos nacen libres e iguales en dignidad y derechos =
|
|
y, dotados como est=C3=A1n de raz=C3=B3n y conciencia, deben comportarse fr=
|
|
aternalmente los unos con los otros."</div>
|
|
|
|
--000000000000a0666d0606e4cfb7--
|