yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-08-18 13:45:45 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured_ingest/expected-structured-output/confluence-diff/MFS/1605928.json
John 9500d04791
detect document language across all partitioners (#1627) 
### Summary
Closes #1534 and #1535
Detects document language using `langdetect` package. 
Creates new kwargs for user to set the document language (`languages`)
or detect the language at the element level instead of the default
document level (`detect_language_per_element`)

---------

Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com>
Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: Coniferish <Coniferish@users.noreply.github.com>
Co-authored-by: cragwolfe <crag@unstructured.io>
Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-11 01:47:56 +00:00

245 lines
7.1 KiB
JSON
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

[
{
"type": "Title",
"element_id": "35054d4d1455c734e83a868656b4ad16",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": "\\uD83D\\uDDD3 Date"
},
{
"type": "Title",
"element_id": "0126c1353ddd7c8dfdb29f252a64a344",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": "\\uD83D\\uDC65 Participants"
},
{
"type": "ListItem",
"element_id": "e3b0c44298fc1c149afbf4c8996fb924",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": ""
},
{
"type": "ListItem",
"element_id": "e3b0c44298fc1c149afbf4c8996fb924",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": ""
},
{
"type": "Title",
"element_id": "fa64ff027cbc0c6929bc75d3c78c94c3",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": "\\uD83E\\uDD45 Goals"
},
{
"type": "ListItem",
"element_id": "e3b0c44298fc1c149afbf4c8996fb924",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": ""
},
{
"type": "Title",
"element_id": "537ea1b14dcba1742bdbd4a5fbfb488c",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": "\\uD83D\\uDDE3 Discussion topics"
},
{
"type": "Table",
"element_id": "37af06e8e75d96a448a00026754b7942",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1,
"text_as_html": "<table><br><tbody><br><tr><td>Time</td><td>Item</td><td>Presenter</td><td>Notes</td></tr><br><tr><td> </td><td> </td><td> </td><td> </td></tr><br><tr><td> </td><td> </td><td> </td><td> </td></tr><br></tbody><br></table>"
},
"text": "Time Item Presenter Notes"
},
{
"type": "Title",
"element_id": "f158a8eaf72c7e9511d5e8ee03692652",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": "✅ Action items"
},
{
"type": "ListItem",
"element_id": "e3b0c44298fc1c149afbf4c8996fb924",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": ""
},
{
"type": "Title",
"element_id": "addb0aa08f77b69fa754ba55c6600c8a",
"metadata": {
"data_source": {
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605928",
"version": 1,
"record_locator": {
"url": "https://unstructured-ingest-test.atlassian.net",
"page_id": "1605928"
},
"date_created": "2023-07-09T12:54:45.162000",
"date_modified": "2023-07-09T12:54:45.162000"
},
"filetype": "text/html",
"languages": [
"eng"
],
"page_number": 1
},
"text": "⤴ Decisions"
}
]
Reference in New Issue View Git Blame Copy Permalink