yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-11 19:15:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured_ingest/expected-structured-output/confluence-diff/MFS/1605942.json
Yuming Long 542d442699
chore CORE-4775: remove html page number metadata field (#2942) 
### Summary

Rip off page_number metadata fields until we have page counting for all
kinds of html files (not just limited to news articles with multiple
`<article>` tag)

### Test
Unit tests
`test_add_chunking_strategy_on_partition_html_respects_multipage` and
`test_add_chunking_strategy_title_on_partition_auto_respects_multipage`
removed since they relay on the `page_number` fields from the SEC html
file - now test moved to mock test for chunk_by_title -> revisit those
tests when we find test file for this

Also changed the element ids from partition outputs for html files -
element id change due to page number change (in element id hashing) ->
todo ticket: update other deterministic element id tests per crag's
comment

---------

Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: yuming-long <yuming-long@users.noreply.github.com>
2024-04-30 15:20:26 +00:00

294 lines
8.1 KiB
JSON
Raw Permalink Blame History

[
{
"element_id": "8cf6f327a51bcafbe61f759da949eae4",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Copy and paste this section for each week.",
"type": "Title"
},
{
"element_id": "0e80063a2a29a298f8f761e73d131d6c",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"emphasized_text_contents": [
"Win"
],
"emphasized_text_tags": [
"strong"
],
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Win",
"type": "Title"
},
{
"element_id": "3fa457c99713dacee34a094ca844d369",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "",
"type": "ListItem"
},
{
"element_id": "f147aba8b007b4bd8208e4572f5b9deb",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "",
"type": "ListItem"
},
{
"element_id": "6eb6112deae2548a83547e5ef9a21b88",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "",
"type": "ListItem"
},
{
"element_id": "0ec9faddd7d36aa48091805f98e064a0",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"emphasized_text_contents": [
"Needs input"
],
"emphasized_text_tags": [
"strong"
],
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Needs input",
"type": "Title"
},
{
"element_id": "10d0eeb46296850b4d495368cd20f70f",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "",
"type": "ListItem"
},
{
"element_id": "70f82254d37a60cc5c83e07d98cce166",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "",
"type": "ListItem"
},
{
"element_id": "93362aec18fca024916f21651c4fb019",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "",
"type": "ListItem"
},
{
"element_id": "2342f48f943338e6a30647e86ced5d81",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"emphasized_text_contents": [
"Focus"
],
"emphasized_text_tags": [
"strong"
],
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Focus",
"type": "Title"
},
{
"element_id": "e33fbc16d200926cce19fe943c3bbb62",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "",
"type": "ListItem"
},
{
"element_id": "1a2728decdac66085a26a43934e2c7cd",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "",
"type": "ListItem"
},
{
"element_id": "86e4984d57e4197d967dd0f0e866e9fd",
"metadata": {
"data_source": {
"date_created": "2023-07-09T12:54:45.226000",
"date_modified": "2023-07-09T12:54:45.226000",
"record_locator": {
"page_id": "1605942",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1605942",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng"
],
"text_as_html": "<table><tr><td>Notes</td><td></td></tr><tr><td>Important Links</td><td></td></tr></table>"
},
"text": "Notes Important Links",
"type": "Table"
}
]
Reference in New Issue View Git Blame Copy Permalink