yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-13 03:55:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured_ingest/expected-structured-output/confluence-diff/testteamsp/1867777.json
Yuming Long 542d442699
chore CORE-4775: remove html page number metadata field (#2942) 
### Summary

Rip off page_number metadata fields until we have page counting for all
kinds of html files (not just limited to news articles with multiple
`<article>` tag)

### Test
Unit tests
`test_add_chunking_strategy_on_partition_html_respects_multipage` and
`test_add_chunking_strategy_title_on_partition_auto_respects_multipage`
removed since they relay on the `page_number` fields from the SEC html
file - now test moved to mock test for chunk_by_title -> revisit those
tests when we find test file for this

Also changed the element ids from partition outputs for html files -
element id change due to page number change (in element id hashing) ->
todo ticket: update other deterministic element id tests per crag's
comment

---------

Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: yuming-long <yuming-long@users.noreply.github.com>
2024-04-30 15:20:26 +00:00

212 lines
6.3 KiB
JSON
Raw Permalink Blame History

[
{
"element_id": "05ef452e8c54005abc4c0666d8404dfd",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 1",
"type": "NarrativeText"
},
{
"element_id": "683293ea6931fb5021d4adaec2c4fce7",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 2",
"type": "NarrativeText"
},
{
"element_id": "54da27b49c6c112ed419e6b8895be2c4",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 3",
"type": "NarrativeText"
},
{
"element_id": "1a9848c57a6d51bdc462e33fbc95602d",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 4",
"type": "NarrativeText"
},
{
"element_id": "a30c9fa2274f7308a14fec6454deb88d",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 5",
"type": "NarrativeText"
},
{
"element_id": "b030841bd2c052c02beffba231c16613",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 6",
"type": "NarrativeText"
},
{
"element_id": "cfc808c5bcc358e14b673d6202c2f09d",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 7",
"type": "NarrativeText"
},
{
"element_id": "a51cb5b3992630d3bfffe014a285370f",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 8",
"type": "NarrativeText"
},
{
"element_id": "f05150f3d4675e976a42a5ea8055b1fe",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 9",
"type": "NarrativeText"
},
{
"element_id": "1ad195d0303ee0f0924b4f41b0335a56",
"metadata": {
"data_source": {
"date_created": "2023-07-11T17:01:19.072000",
"date_modified": "2023-07-11T17:02:21.081000",
"record_locator": {
"page_id": "1867777",
"url": "https://unstructured-ingest-test.atlassian.net"
},
"url": "https://unstructured-ingest-test.atlassian.net/wiki/rest/api/content/1867777",
"version": "2"
},
"filetype": "text/html",
"languages": [
"eng"
]
},
"text": "Testdoc1 has only this text for 10 times: 10",
"type": "NarrativeText"
}
]
Reference in New Issue View Git Blame Copy Permalink