yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-13 12:05:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured_ingest/expected-structured-output/local-single-file-with-encoding/example-docs/fake-html-cp1252.html.json

94 lines
2.2 KiB
JSON
Raw Normal View History

chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
[
{
"type": "Title",
"element_id": "0540311f6c077fe8f797080918b8d74b",
"metadata": {
local connector metadata and deserialization fix (#1800) ### Description * Priority of this was to fix deserialization of ingest docs. Currently the source metadata wasn't being persisted * To help debug this, source metadata was added to the local ingest doc as well. * Unit test added to make sure the metadata itself was persisted. * As part of serialization, it was forcing docs to fetch source metadata if it hadn't already to add to the generated dict/json. This shouldn't have happened if the underlying variable `_source_metadata` was `None`. This way the doc can be serialized without any calls being made. * Serialization was moved to the `to_dict` method to make it more universal. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-10-23 11:51:52 -04:00
"data_source": {
"url": "example-docs/fake-html-cp1252.html",
"permissions_data": [
{
"mode": 33188
}
]
},
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
"filetype": "text/html",
detect document language across all partitioners (#1627) ### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-10 20:47:56 -05:00
"languages": [
roman/bugfix fix default language ingest option (#1729) ### Description Set language to None by default. Update ingest test to use local file used in language unit tests to validate. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-10-12 13:31:23 -04:00
"por",
"cat",
"eng",
"vie"
detect document language across all partitioners (#1627) ### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-10 20:47:56 -05:00
],
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
"page_number": 1
},
"text": "My First Heading"
},
{
"type": "Title",
"element_id": "399af454cb1368b8257ed406b430de84",
"metadata": {
local connector metadata and deserialization fix (#1800) ### Description * Priority of this was to fix deserialization of ingest docs. Currently the source metadata wasn't being persisted * To help debug this, source metadata was added to the local ingest doc as well. * Unit test added to make sure the metadata itself was persisted. * As part of serialization, it was forcing docs to fetch source metadata if it hadn't already to add to the generated dict/json. This shouldn't have happened if the underlying variable `_source_metadata` was `None`. This way the doc can be serialized without any calls being made. * Serialization was moved to the `to_dict` method to make it more universal. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-10-23 11:51:52 -04:00
"data_source": {
"url": "example-docs/fake-html-cp1252.html",
"permissions_data": [
{
"mode": 33188
}
]
},
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
"filetype": "text/html",
detect document language across all partitioners (#1627) ### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-10 20:47:56 -05:00
"languages": [
roman/bugfix fix default language ingest option (#1729) ### Description Set language to None by default. Update ingest test to use local file used in language unit tests to validate. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-10-12 13:31:23 -04:00
"por",
"cat",
"eng",
"vie"
detect document language across all partitioners (#1627) ### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-10 20:47:56 -05:00
],
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
"page_number": 1
},
"text": "My first paragraph."
},
{
"type": "Title",
"element_id": "b4cf0d13edfa976816649971bd640a66",
"metadata": {
local connector metadata and deserialization fix (#1800) ### Description * Priority of this was to fix deserialization of ingest docs. Currently the source metadata wasn't being persisted * To help debug this, source metadata was added to the local ingest doc as well. * Unit test added to make sure the metadata itself was persisted. * As part of serialization, it was forcing docs to fetch source metadata if it hadn't already to add to the generated dict/json. This shouldn't have happened if the underlying variable `_source_metadata` was `None`. This way the doc can be serialized without any calls being made. * Serialization was moved to the `to_dict` method to make it more universal. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-10-23 11:51:52 -04:00
"data_source": {
"url": "example-docs/fake-html-cp1252.html",
"permissions_data": [
{
"mode": 33188
}
]
},
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
"filetype": "text/html",
detect document language across all partitioners (#1627) ### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-10 20:47:56 -05:00
"languages": [
roman/bugfix fix default language ingest option (#1729) ### Description Set language to None by default. Update ingest test to use local file used in language unit tests to validate. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-10-12 13:31:23 -04:00
"por",
"cat",
"eng",
"vie"
detect document language across all partitioners (#1627) ### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-10 20:47:56 -05:00
],
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
"page_number": 1
},
"text": "Some CP1252-specific characters:"
},
{
fix: respect `<pre>` tag order in `partition_html` (#1197) ### Summary Closes #1184. Updates `partition_html` to respect the ordering of `<pre>` tags in HTML documents. ### Testing The elements in the following example should be in the correct order. ```python from unstructured.partition.html import partition_html html_text = """ <pre>The Big Brown Bear</pre> <div>The big brown bear is growling.</div> <pre>The big brown bear is sleeping.</pre> <div>The Big Blue Bear</div> """ elements = partition_html(text=html_text) print("\n\n".join([str(el) for el in elements])) ```
2023-08-25 00:14:48 -04:00
"type": "NarrativeText",
"element_id": "24dc2ef4b79651e2c0434d2724b48fc1",
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
"metadata": {
local connector metadata and deserialization fix (#1800) ### Description * Priority of this was to fix deserialization of ingest docs. Currently the source metadata wasn't being persisted * To help debug this, source metadata was added to the local ingest doc as well. * Unit test added to make sure the metadata itself was persisted. * As part of serialization, it was forcing docs to fetch source metadata if it hadn't already to add to the generated dict/json. This shouldn't have happened if the underlying variable `_source_metadata` was `None`. This way the doc can be serialized without any calls being made. * Serialization was moved to the `to_dict` method to make it more universal. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-10-23 11:51:52 -04:00
"data_source": {
"url": "example-docs/fake-html-cp1252.html",
"permissions_data": [
{
"mode": 33188
}
]
},
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
"filetype": "text/html",
detect document language across all partitioners (#1627) ### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-10 20:47:56 -05:00
"languages": [
roman/bugfix fix default language ingest option (#1729) ### Description Set language to None by default. Update ingest test to use local file used in language unit tests to validate. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-10-12 13:31:23 -04:00
"por",
"cat",
"eng",
"vie"
detect document language across all partitioners (#1627) ### Summary Closes #1534 and #1535 Detects document language using `langdetect` package. Creates new kwargs for user to set the document language (`languages`) or detect the language at the element level instead of the default document level (`detect_language_per_element`) --------- Co-authored-by: shreyanid <42684285+shreyanid@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Coniferish <Coniferish@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io>
2023-10-10 20:47:56 -05:00
],
fix: respect `<pre>` tag order in `partition_html` (#1197) ### Summary Closes #1184. Updates `partition_html` to respect the ordering of `<pre>` tags in HTML documents. ### Testing The elements in the following example should be in the correct order. ```python from unstructured.partition.html import partition_html html_text = """ <pre>The Big Brown Bear</pre> <div>The big brown bear is growling.</div> <pre>The big brown bear is sleeping.</pre> <div>The Big Blue Bear</div> """ elements = partition_html(text=html_text) print("\n\n".join([str(el) for el in elements])) ```
2023-08-25 00:14:48 -04:00
"page_number": 1
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
},
fix: respect `<pre>` tag order in `partition_html` (#1197) ### Summary Closes #1184. Updates `partition_html` to respect the ordering of `<pre>` tags in HTML documents. ### Testing The elements in the following example should be in the correct order. ```python from unstructured.partition.html import partition_html html_text = """ <pre>The Big Brown Bear</pre> <div>The big brown bear is growling.</div> <pre>The big brown bear is sleeping.</pre> <div>The Big Blue Bear</div> """ elements = partition_html(text=html_text) print("\n\n".join([str(el) for el in elements])) ```
2023-08-25 00:14:48 -04:00
"text": "¡\t¢\t£\t¤\t¥\t¦\t§\t¨\t©\tª\t«\t¬\tSHY\t®\t¯\n°\t±\t²\t³\t´\tµ\t¶\t·\t¸\t¹\tº\t»\t¼\t½\t¾\t¿\nÀ\tÁ\tÂ\tÃ\tÄ\tÅ\tÆ\tÇ\tÈ\tÉ\tÊ\tË\tÌ\tÍ\tÎ\tÏ\nÐ\tÑ\tÒ\tÓ\tÔ\tÕ\tÖ\t×\tØ\tÙ\tÚ\tÛ\tÜ\tÝ\tÞ\tß\nà\tá\tâ\tã\tä\tå\tæ\tç\tè\té\tê\të\tì\tí\tî\tï\nð\tñ\tò\tó\tô\tõ\tö\t÷\tø\tù\tú\tû\tü\tý\tþ\tÿ"
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
}
]
Reference in New Issue Copy Permalink