yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-13 12:05:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured_ingest/expected-structured-output/confluence-diff/testteamsp/1802252.json
Yuming Long 542d442699
chore CORE-4775: remove html page number metadata field (#2942) 
### Summary

Rip off page_number metadata fields until we have page counting for all
kinds of html files (not just limited to news articles with multiple
`<article>` tag)

### Test
Unit tests
`test_add_chunking_strategy_on_partition_html_respects_multipage` and
`test_add_chunking_strategy_title_on_partition_auto_respects_multipage`
removed since they relay on the `page_number` fields from the SEC html
file - now test moved to mock test for chunk_by_title -> revisit those
tests when we find test file for this

Also changed the element ids from partition outputs for html files -
element id change due to page number change (in element id hashing) ->
todo ticket: update other deterministic element id tests per crag's
comment

---------

Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: yuming-long <yuming-long@users.noreply.github.com>
2024-04-30 15:20:26 +00:00

746 lines
22 KiB
JSON
Raw Permalink Blame History

[
{
"element_id": "1eba9da7f7ba3d80d060f638e240cc2c",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "74d66fc2066f7844294c8d162f443892",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "20c74c9c7e2f03ebdaa0cc475abc461e",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "13be5a443b462adf06733ac5f3c3f821",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "e3c59c23c9a8d3251fbe8c0b8bf06a4f",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "a97244e416b60752f3071e465dd63d41",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "dfd9cc8f70664dc0b785e1f2332a0993",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "d597dda3a2ba146bd314a4d3a92c4aac",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "5e75c9860459e175f1087efd0dc40972",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "c41a8ba74f19172536db4877b5e13f7e",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "29d99bc3b2a5fde6029ddfe8b1604f3a",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "589780ba10ade81f721303579ee9bce0",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "7bc5e9d84b41175c9ff8ad841394c2b3",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "caab6974e98b9e03c78191c02591775e",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2 testtext2",
"type": "Title"
},
{
"element_id": "fb956c2b3b6bc34871a42881a9bb7b63",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 List Item 1Testdoc2 List Item 1 Nested Item ATestdoc2 List Item 1 Nested Item B",
"type": "ListItem"
},
{
"element_id": "3ca8684b5eaa058dfeba0ce29a0f71cd",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 List Item 2",
"type": "ListItem"
},
{
"element_id": "7d021a80087130e7ac662b4218b11fd0",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 List Item 3",
"type": "ListItem"
},
{
"element_id": "550b644ec595cc8bf7e708d025d9cbfc",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 List Item 4",
"type": "ListItem"
},
{
"element_id": "893e516f50218b5fcc974bfbbfa09e54",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 List Item 5",
"type": "ListItem"
},
{
"element_id": "34ca1c89586f687361d40323d1d1e089",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
],
"link_start_indexes": [
0
],
"link_texts": [
"This is the link for unstructured . io."
],
"link_urls": [
"https://www.unstructured.io/"
]
},
"text": "This is the link for unstructured . io.",
"type": "NarrativeText"
},
{
"element_id": "cbcd197b1d044ea4b71da451273f6695",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"emphasized_text_contents": [
"Testdoc2 Checklist Item 1"
],
"emphasized_text_tags": [
"span"
],
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 Checklist Item 1",
"type": "ListItem"
},
{
"element_id": "cb303b5226d382a210692c8af82a209c",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"emphasized_text_contents": [
"Testdoc2 Checklist Item 2 (checked)"
],
"emphasized_text_tags": [
"span"
],
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 Checklist Item 2 (checked)",
"type": "ListItem"
},
{
"element_id": "74ce8c77f9e995b3a6a8f875b1c769a6",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"emphasized_text_contents": [
"Testdoc2 Checklist Item 3"
],
"emphasized_text_tags": [
"span"
],
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 Checklist Item 3",
"type": "ListItem"
},
{
"element_id": "7db875e9d61f74fe42dfd17feeaaa470",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "😃 😃 😃 😃 😃 😃 😃 😃 😃 😃 😃 😃",
"type": "UncategorizedText"
},
{
"element_id": "5b94af34bf8e667df31a179a4e9c4f57",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"emphasized_text_contents": [
"Testdoc2 bold text"
],
"emphasized_text_tags": [
"strong"
],
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 bold text",
"type": "NarrativeText"
},
{
"element_id": "86b950ab73fb5cff558bce55f97edad3",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"emphasized_text_contents": [
"Testdoc2 italic text"
],
"emphasized_text_tags": [
"em"
],
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 italic text",
"type": "Title"
},
{
"element_id": "6fd66af3449ab45ec24172d7004644af",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 Heading 1 Sized Text",
"type": "Title"
},
{
"element_id": "d0c615e86d95ed2535c1da2591e60df6",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 Heading 2 Sized Text",
"type": "Title"
},
{
"element_id": "2f445fe2b559a79b9105cafbb6dc9e6d",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 Heading 3 Sized Text",
"type": "Title"
},
{
"element_id": "dc21ed2230310b97f88d2204e35e0a54",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 Heading 4 Sized Text",
"type": "Title"
},
{
"element_id": "faebbfc5510e489bbab066d312c3b069",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
]
},
"text": "Testdoc2 Heading 5 Sized Text",
"type": "Title"
},
{
"element_id": "5deb9304a24bd612feb118b3942d7d12",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:39.240000",
"date_modified": "2023-07-11T17:01:47.340000",
"record_locator": {
"page_id": "1802252",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1802252",
"version": "1"
},
"filetype": "text/html",
"languages": [
"eng",
"fra"
],
"text_as_html": "<table><tr><td>Testdoc2 Table: Column 1 Row 0</td><td>Testdoc2 Table: Column 2 Row 0</td><td>Testdoc2 Table: Column 3 Row 0</td></tr><tr><td>Testdoc2 Table: Column 1 Row 1</td><td>Testdoc2 Table: Column 2 Row 1</td><td>Testdoc2 Table: Column 3 Row 1</td></tr><tr><td>Testdoc2 Table: Column 1 Row 2</td><td>Testdoc2 Table: Column 2 Row 2</td><td>Testdoc2 Table: Column 3 Row 2</td></tr></table>"
},
"text": "Testdoc2 Table: Column 1 Row 0 Testdoc2 Table: Column 2 Row 0 Testdoc2 Table: Column 3 Row 0 Testdoc2 Table: Column 1 Row 1 Testdoc2 Table: Column 2 Row 1 Testdoc2 Table: Column 3 Row 1 Testdoc2 Table: Column 1 Row 2 Testdoc2 Table: Column 2 Row 2 Testdoc2 Table: Column 3 Row 2",
"type": "Table"
}
]
Reference in New Issue View Git Blame Copy Permalink