yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-08 09:33:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured/documents/test_html.py

943 lines
32 KiB
Python
Raw Normal View History

rfctr(html): clean types and docs in prep for HTML table parsing fixes (#2104) There are a cluster of bugs in the HTML parsing code, particularly surrounding table behaviors but also inclusion of style elements, etc. Clean up typing and docstrings in that neighborhood as a way to familiarize myself with that part of the code-base.
2023-11-17 09:56:38 -08:00
# pyright: reportPrivateUsage=false
rfctr(html): improve SNR in HTMLDocument (#3162) **Summary** Remove dead code and organize helpers of HTMLDocument in preparation for improvements and bug-fixes to follow
2024-06-06 14:21:33 -07:00
# pyright: reportUnusedFunction=false
rfctr(html): clean types and docs in prep for HTML table parsing fixes (#2104) There are a cluster of bugs in the HTML parsing code, particularly surrounding table behaviors but also inclusion of style elements, etc. Clean up typing and docstrings in that neighborhood as a way to familiarize myself with that part of the code-base.
2023-11-17 09:56:38 -08:00
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
"""Test suite for `unstructured.documents.html` module."""
from __future__ import annotations
fix: Remove JavaScript from HTML reader output (#313) * Fixes an error causing JavaScript to appear in the output of `partition_html` sometimes.
2023-02-28 23:24:24 +01:00
import pathlib
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
from typing import Any
Initial Release
2022-06-29 14:35:19 -04:00
import pytest
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
from lxml import etree
Initial Release
2022-06-29 14:35:19 -04:00
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
from test_unstructured.unit_utils import (
FixtureRequest,
Mock,
example_doc_path,
function_mock,
property_mock,
)
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
from unstructured.documents import html
from unstructured.documents.elements import (
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
Address,
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
ListItem,
NarrativeText,
Adding table extraction to partition_html (#1324) Adding table extraction to HTML partitioning. This PR utilizes 'table' HTML elements to extract and parse HTML tables and return them in partitioning. ``` # checkout this branch, go into ipython shell In [1]: from unstructured.partition.html import partition_html In [2]: path_to_html = "{html sample file with table}" In [3]: elements = partition_html(path_to_html) ``` you should see the table in the elements list!
2023-09-11 11:14:11 -07:00
Table,
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
Text,
Title,
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
from unstructured.documents.html import HTMLDocument, HtmlPartitionerOptions
Initial Release
2022-06-29 14:35:19 -04:00
TAGS = (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
(
"<a><abbr><acronym><address><applet><area><article><aside><audio><b><base><basefont><bdi>"
"<bdo><big><blockquote><body><br><button><canvas><caption><center><cite><code><col>"
"<colgroup><data><datalist><dd><del><details><dfn><dialog><dir><div><dl><dt><em><embed>"
"<fieldset><figcaption><figure><font><footer><form><frame><frameset><h1><h2><h3><h4><h5>"
"<h6><head><header><hr><html><i><iframe><img><input><ins><kbd><label><legend><li><link>"
"<main><map><mark><meta><meter><nav><noframes><noscript><object><ol><optgroup><option>"
"<output><p><param><picture><pre><progress><q><rp><rt><ruby><s><samp><script><section>"
"<select><small><source><span><strike><strong><style><sub><summary><sup><table><tbody><td>"
"<template><textarea><tfoot><th><thead><time><title><tr><track><tt><u><ul><var><video><wbr>"
)
.replace(">", "")
.split("<")[1:]
Initial Release
2022-06-29 14:35:19 -04:00
)
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
VOID_TAGS = (
("<area><base><br><col><embed><hr><img><input><link><meta><param><source><track><wbr>")
.replace(">", "")
.split("<")[1:]
)
fix: Fix various cases of HTML text missing after partition (#1587) Fix 4 cases of text missing after partition: 1. Text immediately after `<body>` ```html <body> missing1 <div>hello</div> </body> ``` 2. Text inside container and immediately after `<br/>` ```html <div>hello<br/>missing2</div> ``` 3. Text immediately after a text opening tag, if said tag contains `<br/>` ```html <p>missing3<br/>hello</p> ``` 4. Text inside `<body>` if it is the only content (different cause from case 1) ```html <body>missing4</body> ``` Also fix problem causing `test_unstructured/documents/test_html.py::test_exclude_tag_types` to not work as intended. This will close GitHub Issue#1543
2023-10-03 12:17:51 +08:00
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
INCLUDED_TAGS = html.TEXT_TAGS + html.HEADING_TAGS + html.LIST_ITEM_TAGS + html.SECTION_TAGS
fix: Fix various cases of HTML text missing after partition (#1587) Fix 4 cases of text missing after partition: 1. Text immediately after `<body>` ```html <body> missing1 <div>hello</div> </body> ``` 2. Text inside container and immediately after `<br/>` ```html <div>hello<br/>missing2</div> ``` 3. Text immediately after a text opening tag, if said tag contains `<br/>` ```html <p>missing3<br/>hello</p> ``` 4. Text inside `<body>` if it is the only content (different cause from case 1) ```html <body>missing4</body> ``` Also fix problem causing `test_unstructured/documents/test_html.py::test_exclude_tag_types` to not work as intended. This will close GitHub Issue#1543
2023-10-03 12:17:51 +08:00
EXCLUDED_TAGS = [
tag
for tag in TAGS
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
if tag not in (INCLUDED_TAGS + html.TABLE_TAGS + VOID_TAGS + ["html", "head", "body"])
fix: Fix various cases of HTML text missing after partition (#1587) Fix 4 cases of text missing after partition: 1. Text immediately after `<body>` ```html <body> missing1 <div>hello</div> </body> ``` 2. Text inside container and immediately after `<br/>` ```html <div>hello<br/>missing2</div> ``` 3. Text immediately after a text opening tag, if said tag contains `<br/>` ```html <p>missing3<br/>hello</p> ``` 4. Text inside `<body>` if it is the only content (different cause from case 1) ```html <body>missing4</body> ``` Also fix problem causing `test_unstructured/documents/test_html.py::test_exclude_tag_types` to not work as intended. This will close GitHub Issue#1543
2023-10-03 12:17:51 +08:00
]
Initial Release
2022-06-29 14:35:19 -04:00
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
# -- table-extraction behaviors ------------------------------------------------------------------
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def test_it_can_parse_a_bare_bones_table_to_a_Table_element(opts_args: dict[str, Any]):
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"""Bare-bones means no `<thead>`, `<tbody>`, or `<tfoot>` elements."""
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts_args["text"] = (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"<html>\n"
"<body>\n"
" <table>\n"
" <tr><td>Lorem</td><td>Ipsum</td></tr>\n"
" <tr><td>Ut enim non</td><td>ad minim\nveniam quis</td></tr>\n"
" </table>\n"
"</body>\n"
"</html>"
Initial Release
2022-06-29 14:35:19 -04:00
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
# -- there is exactly one element and it's a Table instance --
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
(element,) = html_document.elements
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert isinstance(element, Table)
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
# -- table text is joined into a single string; no row or cell boundaries are represented --
assert element.text == "Lorem Ipsum Ut enim non ad minim\nveniam quis"
# -- An HTML representation is also available that is longer but represents table structure.
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert element.metadata.text_as_html == (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"<table>"
"<tr><td>Lorem</td><td>Ipsum</td></tr>"
"<tr><td>Ut enim non</td><td>ad minim<br/>veniam quis</td></tr>"
"</table>"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_it_accommodates_column_heading_cells_enclosed_in_thead_tbody_and_tfoot_elements(
opts_args: dict[str, Any]
):
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"""Cells within a `table/thead` element are included in the text and html.
The presence of a `<thead>` element in the original also determines whether a `<thead>` element
appears in `.text_as_html` or whether the first row of cells is simply in the body.
"""
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts_args["text"] = (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"<html>\n"
"<body>\n"
" <table>\n"
" <thead>\n"
" <tr><th>Lorem</th><th>Ipsum</th></tr>\n"
" </thead>\n"
" <tbody>\n"
" <tr><th>Lorem ipsum</th><td>dolor sit amet nulla</td></tr>\n"
" <tr><th>Ut enim non</th><td>ad minim\nveniam quis</td></tr>\n"
" </tbody>\n"
" <tfoot>\n"
" <tr><th>Dolor</th><td>Equis</td></tr>\n"
" </tfoot>\n"
" </table>\n"
"</body>\n"
"</html>"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
(element,) = html_document.elements
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert isinstance(element, Table)
assert element.metadata.text_as_html == (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"<table>"
"<tr><td>Lorem</td><td>Ipsum</td></tr>"
"<tr><td>Lorem ipsum</td><td>dolor sit amet nulla</td></tr>"
"<tr><td>Ut enim non</td><td>ad minim<br/>veniam quis</td></tr>"
"<tr><td>Dolor</td><td>Equis</td></tr>"
"</table>"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def test_it_does_not_emit_a_Table_element_for_a_table_with_no_text(opts_args: dict[str, Any]):
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts_args["text"] = (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"<html>\n"
"<body>\n"
" <table>\n"
" <tr><td> </td><td> </td></tr>\n"
" <tr><td> </td><td> </td></tr>\n"
" </table>\n"
"</body>\n"
"</html>"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
assert html_document.elements == []
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_it_grabs_bulleted_text_in_tables_as_ListItem_elements(opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
"<html>\n"
" <body>\n"
" <table>\n"
" <tbody>\n"
" <tr>\n"
" <td>&#8226;</td>\n"
" <td><p>Happy Groundhog's day!</p></td>\n"
" </tr>\n"
" <tr>\n"
" <td>&#8226;</td>\n"
" <td><p>Looks like six more weeks of winter ...</p></td>\n"
" </tr>\n"
" </tbody>\n"
" </table>\n"
" </body>\n"
"</html>\n"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert html_document.elements == [
ListItem(text="Happy Groundhog's day!"),
ListItem(text="Looks like six more weeks of winter ..."),
]
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_it_does_not_consider_an_empty_table_a_bulleted_text_table(opts_args: dict[str, Any]):
opts_args["text"] = (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"<html>\n"
"<body>\n"
" <table>\n"
" <tr><td> </td><td> </td></tr>\n"
" <tr><td> </td><td> </td></tr>\n"
" </table>\n"
"</body>\n"
"</html>"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
rfctr(html): drop now dead XMLDocument and Document (#3165) **Summary** `HTMLDocument` is the class handling the core of HTML parsing. This is critical code because 8 of the 20 file-type partitioners end up using this code (`partition_html()` + 7 brokering partitioners like EPUB, MD, and RST). For historical reasons, `HTMLDocument` subclassed `XMLDocument` which in turn subclassed `Document`, both of which are no longer relevant and unnecessarily complicate reasoning about `HTMLDocument` behavior. Remove that inheritance and dependency and drop both `XMLDocument` and `Document` modules which become dead code after no longer being used by `HTMLDocument`.
2024-06-08 00:36:18 -07:00
html_elem = html_document._document_tree
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
assert html_elem is not None
table = html_elem.find(".//table")
assert table is not None
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_document._is_bulleted_table(table) is False
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_it_provides_parseable_HTML_in_text_as_html(opts_args: dict[str, Any]):
opts_args["text"] = (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
"<html>\n"
"<body>\n"
" <table>\n"
" <thead>\n"
" <tr><th>Lorem</th><th>Ipsum</th></tr>\n"
" </thead>\n"
" <tbody>\n"
" <tr><th>Lorem ipsum</th><td>dolor sit amet nulla</td></tr>\n"
" <tr><th>Ut enim non</th><td>ad minim\nveniam quis</td></tr>\n"
" </tbody>\n"
" <tfoot>\n"
" <tr><th>Dolor</th><td>Equis</td></tr>\n"
" </tfoot>\n"
" </table>\n"
"</body>\n"
"</html>"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_document = HTMLDocument.load(HtmlPartitionerOptions(**opts_args))
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
(element,) = html_document.elements
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert isinstance(element, Table)
text_as_html = element.metadata.text_as_html
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
assert text_as_html is not None
html = etree.fromstring(text_as_html, etree.HTMLParser())
assert html is not None
# -- lxml adds the <html><body> container, that's not present in `.text_as_html` --
assert etree.tostring(html, encoding=str) == (
"<html><body>"
"<table>"
"<tr><td>Lorem</td><td>Ipsum</td></tr>"
"<tr><td>Lorem ipsum</td><td>dolor sit amet nulla</td></tr>"
"<tr><td>Ut enim non</td><td>ad minim<br/>veniam quis</td></tr>"
"<tr><td>Dolor</td><td>Equis</td></tr>"
"</table>"
"</body></html>"
)
fix(html): style (CSS) content appears in HTML text (#2132) Fixes #1958. `<style>` is invalid where it appears in the HTML of thw WSJ page mentioned by that issue but invalid has little meaning in the HTML world if Chrome accepts it. In any case, we have no use for the contents of a `<style>` tag wherever it appears so safe enough for us to just strip all those tags. Note we do not want to also strip the *tail text* which can contain text we're interested in.
2023-11-20 20:22:13 -08:00
# -- element-suppression behaviors ---------------------------------------------------------------
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_it_does_not_extract_text_in_script_tags(opts_args: dict[str, Any]):
opts_args["file_path"] = example_doc_path("example-with-scripts.html")
opts = HtmlPartitionerOptions(**opts_args)
doc = HTMLDocument.load(opts)
rfctr: skip CheckBox elements during chunking (#2253) `CheckBox` elements get special treatment during chunking. `CheckBox` does not derive from `Text` and can contribute no text to a chunk. It is considered "non-combinable" and so is emitted as-is as a chunk of its own. A consequence of this is it breaks an otherwise contiguous chunk into two wherever it occurs. This is problematic, but becomes much more so when overlap is introduced. Each chunk accepts a "tail" text fragment from its preceding element and contributes its own tail fragment to the next chunk. These tails represent the "overlap" between chunks. However, a non-text chunk can neither accept nor provide a tail-fragment and so interrupts the overlap. None of the possible solutions are terrific. Give `Element` a `.text` attribute such that _all_ elements have a `.text` attribute, even though its value is the empty-string for element-types such as CheckBox and PageBreak which inherently have no text. As a consequence, several `cast()` wrappers are no longer required to satisfy strict type-checking. This also allows a `CheckBox` element to be combined with `Text` subtypes during chunking, essentially the same way `PageBreak` is, contributing no text to the chunk. Also, remove the `_NonTextSection` object which previously wrapped a `CheckBox` element during pre-chunking as it is no longer required.
2023-12-13 12:22:25 -08:00
assert all("function (" not in element.text for element in doc.elements)
fix(html): style (CSS) content appears in HTML text (#2132) Fixes #1958. `<style>` is invalid where it appears in the HTML of thw WSJ page mentioned by that issue but invalid has little meaning in the HTML world if Chrome accepts it. In any case, we have no use for the contents of a `<style>` tag wherever it appears so safe enough for us to just strip all those tags. Note we do not want to also strip the *tail text* which can contain text we're interested in.
2023-11-20 20:22:13 -08:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_it_does_not_extract_text_in_style_tags(opts_args: dict[str, Any]):
opts_args["text"] = (
fix(html): style (CSS) content appears in HTML text (#2132) Fixes #1958. `<style>` is invalid where it appears in the HTML of thw WSJ page mentioned by that issue but invalid has little meaning in the HTML world if Chrome accepts it. In any case, we have no use for the contents of a `<style>` tag wherever it appears so safe enough for us to just strip all those tags. Note we do not want to also strip the *tail text* which can contain text we're interested in.
2023-11-20 20:22:13 -08:00
"<html>\n"
"<body>\n"
" <p><style> p { margin:0; padding:0; } </style>Lorem ipsum dolor</p>\n"
"</body>\n"
"</html>"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
fix(html): style (CSS) content appears in HTML text (#2132) Fixes #1958. `<style>` is invalid where it appears in the HTML of thw WSJ page mentioned by that issue but invalid has little meaning in the HTML world if Chrome accepts it. In any case, we have no use for the contents of a `<style>` tag wherever it appears so safe enough for us to just strip all those tags. Note we do not want to also strip the *tail text* which can contain text we're interested in.
2023-11-20 20:22:13 -08:00
(element,) = html_document.elements
assert isinstance(element, Text)
assert element.text == "Lorem ipsum dolor"
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# -- HTMLDocument.from_file() --------------------------------------------------------------------
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_read_html_doc(tmp_path: pathlib.Path, opts_args: dict[str, Any]):
file_path = str(tmp_path / "sample-doc.html")
with open(file_path, "w") as f:
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
f.write(
"<html>\n"
" <body>\n"
" <h1>A Great and Glorious Section</h1>\n"
" <p>Dear Leader is the best. He is such a wonderful engineer!</p>\n"
" <p></p>\n"
" <p>Another Magnificent Title</p>\n"
" <p><b>The prior element is a title based on its capitalization patterns!</b></p>\n"
" <table>\n"
" <tbody>\n"
" <tr>\n"
rfctr(html): improve SNR in HTMLDocument (#3162) **Summary** Remove dead code and organize helpers of HTMLDocument in preparation for improvements and bug-fixes to follow
2024-06-06 14:21:33 -07:00
" <td><p>I'm in a table</p></td>\n"
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
" </tr>\n"
" </tbody>\n"
" </table>\n"
" <h2>A New Beginning</h2>\n"
" <div>Here is the start of a new page.</div>\n"
" </body>\n"
"</html>\n"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts_args["file_path"] = file_path
opts = HtmlPartitionerOptions(**opts_args)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
elements = HTMLDocument.load(opts).elements
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
assert len(elements) == 7
assert elements == [
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
Title("A Great and Glorious Section"),
NarrativeText("Dear Leader is the best. He is such a wonderful engineer!"),
Title("Another Magnificent Title"),
NarrativeText("The prior element is a title based on its capitalization patterns!"),
rfctr(html): improve SNR in HTMLDocument (#3162) **Summary** Remove dead code and organize helpers of HTMLDocument in preparation for improvements and bug-fixes to follow
2024-06-06 14:21:33 -07:00
Table("I'm in a table"),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
Title("A New Beginning"),
NarrativeText("Here is the start of a new page."),
]
# -- HTMLDocument.elements -----------------------------------------------------------------------
Initial Release
2022-06-29 14:35:19 -04:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_nested_text_tags(opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): improve SNR in HTMLDocument (#3162) **Summary** Remove dead code and organize helpers of HTMLDocument in preparation for improvements and bug-fixes to follow
2024-06-06 14:21:33 -07:00
"<body>\n"
" <p>\n"
" <a>\n"
" There is some text here.\n"
" </a>\n"
" </p>\n"
"</body>\n"
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
elements = HTMLDocument.load(opts).elements
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
assert len(elements) == 1
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_containers_with_text_are_processed(opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
'<div dir=3D"ltr">Hi All,<div><br></div>\n'
" <div>Get excited for our first annual family day!</div>\n"
' <div>Best.<br clear=3D"all">\n'
" <div><br></div>\n"
" -- <br>\n"
' <div dir=3D"ltr">\n'
' <div dir=3D"ltr">Dino the Datasaur<div>\n'
" Unstructured Technologies<br>\n"
" <div>Data Scientist</div>\n"
" <div>Doylestown, PA 18901</div>\n"
" <div><br></div>\n"
" </div>\n"
" </div>\n"
" </div>\n"
" </div>\n"
"</div>\n"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert html_document.elements == [
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
Text(text="Hi All,"),
NarrativeText(text="Get excited for our first annual family day!"),
Title(text="Best."),
Text(text="\n -- "),
Title(text="Dino the Datasaur"),
Title(text="\n Unstructured Technologies"),
Title(text="Data Scientist"),
Address(text="Doylestown, PA 18901"),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
]
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_html_grabs_bulleted_text_in_tags(opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
"<html>\n"
" <body>\n"
" <ol>\n"
" <li>Happy Groundhog's day!</li>\n"
" <li>Looks like six more weeks of winter ...</li>\n"
" </ol>\n"
" </body>\n"
"</html>\n"
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
)
opts = HtmlPartitionerOptions(**opts_args)
assert HTMLDocument.load(opts).elements == [
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
ListItem(text="Happy Groundhog's day!"),
ListItem(text="Looks like six more weeks of winter ..."),
]
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_html_grabs_bulleted_text_in_paras(opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
"<html>\n"
" <body>\n"
" <p>\n"
" <span>&#8226; Happy Groundhog's day!</span>\n"
" </p>\n"
" <p>\n"
" <span>&#8226; Looks like six more weeks of winter ...</span>\n"
" </p>\n"
" </body>\n"
"</html>\n"
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
)
opts = HtmlPartitionerOptions(**opts_args)
assert HTMLDocument.load(opts).elements == [
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
ListItem(text="Happy Groundhog's day!"),
ListItem(text="Looks like six more weeks of winter ..."),
]
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_joins_tag_text_correctly(opts_args: dict[str, Any]):
opts_args["text"] = "<p>Hello again peet mag<i>ic</i>al</p>"
opts = HtmlPartitionerOptions(**opts_args)
doc = HTMLDocument.load(opts)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert doc.elements[0].text == "Hello again peet magical"
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_sample_doc_with_emoji(opts_args: dict[str, Any]):
opts_args["text"] = '<html charset="unicode">\n<p>Hello again 😀</p>\n</html>'
opts = HtmlPartitionerOptions(**opts_args)
doc = HTMLDocument.load(opts)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# NOTE(robinson) - unclear why right now, but the output is the emoji on the test runners
# and the byte string representation when running locally on mac
assert doc.elements[0].text in ["Hello again ð\x9f\x98\x80", "Hello again 😀"]
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_only_plain_text_in_body(opts_args: dict[str, Any]):
opts_args["text"] = "<body>Hello</body>"
opts = HtmlPartitionerOptions(**opts_args)
assert HTMLDocument.load(opts).elements[0].text == "Hello"
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_plain_text_before_anything_in_body(opts_args: dict[str, Any]):
opts_args["text"] = "<body>Hello<p>World</p></body>"
opts = HtmlPartitionerOptions(**opts_args)
doc = HTMLDocument.load(opts)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert doc.elements[0].text == "Hello"
assert doc.elements[1].text == "World"
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_line_break_in_container(opts_args: dict[str, Any]):
opts_args["text"] = "<div>Hello<br/>World</div>"
opts = HtmlPartitionerOptions(**opts_args)
doc = HTMLDocument.load(opts)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert doc.elements[0].text == "Hello"
assert doc.elements[1].text == "World"
@pytest.mark.parametrize("tag", html.TEXT_TAGS)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_line_break_in_text_tag(tag: str, opts_args: dict[str, Any]):
opts_args["text"] = f"<{tag}>Hello<br/>World</{tag}>"
opts = HtmlPartitionerOptions(**opts_args)
doc = HTMLDocument.load(opts)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert doc.elements[0].text == "Hello"
assert doc.elements[1].text == "World"
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
@pytest.mark.parametrize("tag", [tag for tag in html.TEXT_TAGS if tag not in html.TABLE_TAGS])
def test_tag_types(tag: str, opts_args: dict[str, Any]):
opts_args["text"] = f"<body>\n <{tag}>\n There is some text here.\n </{tag}>\n</body>\n"
opts = HtmlPartitionerOptions(**opts_args)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
elements = HTMLDocument.load(opts).elements
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
assert len(elements) == 1
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
@pytest.mark.parametrize("tag", EXCLUDED_TAGS)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_exclude_tag_types(tag: str, opts_args: dict[str, Any]):
opts_args["text"] = f"<body>\n <{tag}>\n There is some text here.\n </{tag}>\n</body>\n"
opts = HtmlPartitionerOptions(**opts_args)
elements = HTMLDocument.load(opts).elements
assert len(elements) == 0
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# -- _construct_text() ---------------------------------------------------------------------------
Initial Release
2022-06-29 14:35:19 -04:00
@pytest.mark.parametrize(
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
("doc", "expected"),
Initial Release
2022-06-29 14:35:19 -04:00
[
(
"<p>Hi there <span>my name is</span> <b><i>Matt</i></i></p>",
"Hi there my name is Matt",
),
("<p>I have a</p> tail", "I have a tail"),
],
)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
def test_construct_text(doc: str, expected: str):
Initial Release
2022-06-29 14:35:19 -04:00
document_tree = etree.fromstring(doc, etree.HTMLParser())
para = document_tree.find(".//p")
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert para is not None
Initial Release
2022-06-29 14:35:19 -04:00
text = html._construct_text(para)
assert text == expected
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# -- _get_emphasized_texts_from_tag() ------------------------------------------------------------
@pytest.mark.parametrize(
("doc", "root", "expected"),
[
(
"<p>Hello <strong>there</strong> I <em>am</em> a <b>very</b> <i>important</i> text</p>",
"p",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["there", "am", "very", "important"], ["strong", "em", "b", "i"]),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
),
(
"<p>Here is a <span>list</span> of <b>my <i>favorite</i> things</b></p>",
"p",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["list", "my favorite things", "favorite"], ["span", "b", "i"]),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
),
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
("<strong>A lone strong text!</strong>", "strong", (["A lone strong text!"], ["strong"])),
("<span>I have a</span> tail", "span", (["I have a"], ["span"])),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
],
)
def test_get_emphasized_texts_from_tag(doc: str, root: str, expected: list[dict[str, str]]):
document_tree = etree.fromstring(doc, etree.HTMLParser())
el = document_tree.find(f".//{root}")
assert el is not None
emphasized_texts = html._get_emphasized_texts_from_tag(el)
assert emphasized_texts == expected
# -- _get_links_from_tag() -----------------------------------------------------------------------
@pytest.mark.parametrize(
("doc", "root", "expected"),
[
(
"<a href='/loner'>A lone link!</a>",
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
"a",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["A lone link!"], ["/loner"], [-1]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
(
"<ul><li><a href='/wiki/Parrot'>Parrots</a></li><li>Dogs</li></ul>",
"ul",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["Parrots"], ["/wiki/Parrot"], [0]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
(
"<ul><li><a href='/parrot'>Parrots</a></li><li><a href='/dog'>Dogs</a></li></ul>",
"ul",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["Parrots", "Dogs"], ["/parrot", "/dog"], [0, 7]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
(
"<div>Here is <p>P tag</p> tail text. <a href='/link'>link!</a></div>",
"div",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["link!"], ["/link"], [25]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
(
"<div>Here is <p>P tag</p><a href='/link'>link!</a></div>",
"div",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["link!"], ["/link"], [13]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
],
)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
def test_get_links_from_tag(doc: str, root: str, expected: list[dict[str, str]]):
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
document_tree = etree.fromstring(doc, etree.HTMLParser())
el = document_tree.find(f".//{root}")
assert el is not None
links = html._get_links_from_tag(el)
assert links == expected
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# -- _is_text_tag() ------------------------------------------------------------------------------
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def test_adjacent_spans_are_text_tags(opts_args: dict[str, Any]):
html_str = "<div><span>&#8226;</span><span>A bullet!</span></div>"
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert el is not None
Initial Release
2022-06-29 14:35:19 -04:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_document._is_text_tag(el) is True
Initial Release
2022-06-29 14:35:19 -04:00
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
# -- unit-level tests ----------------------------------------------------------------------------
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
class DescribeHTMLDocument:
"""Unit-test suite for `unstructured.documents.html.HTMLDocument`."""
# -- ._main ----------------------------------
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def it_can_find_the_main_element_in_the_document(self, opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
"<header></header>\n"
"<body>\n"
" <p>Lots preamble stuff yada yada yada</p>\n"
" <main>\n"
" <article>\n"
" <section>\n"
" <h2>A Wonderful Section!</h2>\n"
" <p>Look at this amazing section!</p>\n"
" </section>\n"
" <section>\n"
" <h2>Another Wonderful Section!</h2>\n"
" <p>Look at this other amazing section!</p>\n"
" </section>\n"
" </article>\n"
" </main>\n"
"</body>\n"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
assert html_document._main.tag == "main"
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def but_it_returns_the_root_when_no_main_element_is_present(self, opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
"<header></header>\n"
"<body>\n"
" <p>Lots preamble stuff yada yada yada</p>\n"
" <article>\n"
" <section>\n"
" <h2>A Wonderful Section!</h2>\n"
" <p>Look at this amazing section!</p>\n"
" </section>\n"
" <section>\n"
" <h2>Another Wonderful Section!</h2>\n"
" <p>Look at this other amazing section!</p>\n"
" </section>\n"
" </article>\n"
"</body>\n"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
assert html_document._main.tag == "html"
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
# -- ._parse_bulleted_text_from_table() ------
def it_extracts_bullet_text_from_table_as_ListItem_elements(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<html>
<body>
<table>
<tbody>
<tr>
<td>•</td>
<td><p>Happy Groundhog's day!</p></td>
</tr>
<tr>
<td>•</td>
<td><p>Looks like six more weeks of winter ...</p></td>
</tr>
</tbody>
</table>
</body>
</html>
"""
html_document = HTMLDocument(html_str, opts)
table_elem = html_document._main.find(".//table")
assert table_elem is not None
list_item_iter = html_document._parse_bulleted_text_from_table(table_elem)
assert next(list_item_iter) == ListItem(text="Happy Groundhog's day!")
assert next(list_item_iter) == ListItem(text="Looks like six more weeks of winter ...")
with pytest.raises(StopIteration):
next(list_item_iter)
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
# -- ._parse_Table_from_table_elem() ---------
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def it_produces_one_cell_for_each_original_table_cell(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = (
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
# -- include formatting whitespace to make sure it is removed --
"<table>\n"
" <tr>\n"
" <td>foo</td>\n"
" <td>bar</td>\n"
" </tr>\n"
"</table>"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_document = HTMLDocument(html_str, opts)
table_elem = html_document._main.find(".//table")
assert table_elem is not None
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_table = html_document._parse_Table_from_table_elem(table_elem)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert isinstance(html_table, Table)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
assert html_table.text == "foo bar"
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_table.metadata.text_as_html == (
"<table><tr><td>foo</td><td>bar</td></tr></table>"
)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def it_accommodates_tds_with_child_elements(self, opts_args: dict[str, Any]):
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
"""Like this example from an SEC 10k filing."""
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_str = (
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
"<table>\n"
" <tr>\n"
" <td></td>\n"
" <td></td>\n"
" </tr>\n"
" <tr>\n"
" <td>\n"
" <p>\n"
" <span>\n"
' <ix:nonNumeric id="F_be4cc145-372a-4689-be60-d8a70b0c8b9a"'
' contextRef="C_1de69f73-df01-4830-8af0-0f11b469bc4a" name="dei:DocumentAnnualReport"'
' format="ixt-sec:boolballotbox">\n'
" <span>&#9746;</span>\n"
" </ix:nonNumeric>\n"
" </span>\n"
" </p>\n"
" </td>\n"
" <td>\n"
" <p>\n"
" <span>ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE"
" ACT OF 1934</span>\n"
" </p>\n"
" </td>\n"
" </tr>\n"
"</table>\n"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_document = HTMLDocument(html_str, opts)
table_elem = html_document._main.find(".//table")
assert table_elem is not None
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_table = html_document._parse_Table_from_table_elem(table_elem)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert isinstance(html_table, Table)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
assert html_table.text == (
"☒ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_table.metadata.text_as_html == (
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
"<table>"
"<tr><td></td><td></td></tr>"
"<tr><td>☒</td><td>ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES"
" EXCHANGE ACT OF 1934</td></tr>"
"</table>"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def it_reduces_a_nested_table_to_its_text_placed_in_the_cell_containing_the_nested_table(
self, opts_args: dict[str, Any]
):
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
"""Recursively ..."""
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
opts = HtmlPartitionerOptions(**opts_args)
# -- note <table> elements nested in <td> elements --
html_str = (
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
"<table>\n"
" <tr>\n"
" <td>\n"
" <table>\n"
" <tr><td>foo</td><td>bar</td></tr>\n"
" <tr><td>baz</td><td>bng</td></tr>\n"
" </table>\n"
" </td>\n"
" <td>\n"
" <table>\n"
" <tr><td>fizz</td><td>bang</td></tr>\n"
" </table>\n"
" </td>\n"
" </tr>\n"
"</table>"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_document = HTMLDocument(html_str, opts)
table_elem = html_document._main.find(".//table")
assert table_elem is not None
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_table = html_document._parse_Table_from_table_elem(table_elem)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert isinstance(html_table, Table)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
assert html_table.text == "foo bar baz bng fizz bang"
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_table.metadata.text_as_html == (
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
"<table><tr><td>foo bar baz bng</td><td>fizz bang</td></tr></table>"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
# -- ._parse_tag() ---------------------------
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def it_produces_a_Text_element_when_the_tag_contents_are_not_narrative_or_a_title(
self,
is_possible_title_: Mock,
opts_args: dict[str, Any],
):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p>NO PARTICULAR TYPE.</p>"
is_possible_title_.return_value = False
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_document._parse_tag(p) == Text(text="NO PARTICULAR TYPE.")
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def it_produces_a_ListItem_element_when_the_tag_contains_are_preceded_by_a_bullet_character(
self, opts_args: dict[str, Any]
):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p>● An excellent point!</p>"
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
assert html_document._parse_tag(p) == ListItem("An excellent point!")
def but_not_when_the_tag_contains_only_a_bullet_character_and_no_text(
self, opts_args: dict[str, Any]
):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p>●</p>"
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
assert html_document._parse_tag(p) is None
def it_returns_None_when_the_tag_has_no_content(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p></p>"
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
assert html_document._parse_tag(p) is None
def and_it_returns_None_when_the_tag_contains_only_a_stub(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p>$</p>"
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
assert html_document._parse_tag(p) is None
# -- ._process_list_item() -------------------------------------------------------------------
def it_continues_processing_elements_after_a_bulleted_list(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<div>
<p>●</p>
<p>●</p>
</div>
<div>
<p>An excellent point!</p>
</div>
"""
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
assert el is not None
parsed_el, _ = html_document._process_list_item(el, max_predecessor_len=10)
assert parsed_el == ListItem(text="An excellent point!")
def but_it_returns_None_when_no_element_follows_empty_bullets(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<div>
<p>●</p>
<p>●</p>
</div>
"""
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
assert el is not None
parsed_el, _ = html_document._process_list_item(el)
assert parsed_el is None
def and_it_returns_None_when_following_element_contains_no_text(
self, opts_args: dict[str, Any]
):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<div>
<p>●</p>
<p>●</p>
</div>
<div>
</div>
"""
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
assert el is not None
parsed_el, _ = html_document._process_list_item(el)
assert parsed_el is None
def it_ignores_deep_divs_when_so_instructed(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<div>
<p>●</p>
<p>●</p>
<p>●</p>
<p>●</p>
<p>●</p>
</div>
<div>
<p>An excellent point!</p>
</div>
"""
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
assert el is not None
parsed_el, _ = html_document._process_list_item(el, max_predecessor_len=2)
assert parsed_el is None
# -- module-level fixtures -----------------------------------------------------------------------
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
@pytest.fixture
def is_possible_narrative_text_(request: FixtureRequest):
return function_mock(request, "unstructured.documents.html.is_possible_narrative_text")
@pytest.fixture
def is_possible_title_(request: FixtureRequest):
return function_mock(request, "unstructured.documents.html.is_possible_title")
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
@pytest.fixture
def opts_args() -> dict[str, Any]:
"""All default arguments for `HtmlPartitionerOptions`.
Individual argument values can be changed to suit each test. Makes construction of opts more
compact for testing purposes.
"""
return {
"file": None,
"file_path": None,
"text": None,
"encoding": None,
"url": None,
"headers": {},
"ssl_verify": True,
"date_from_file_object": False,
"metadata_last_modified": None,
"skip_headers_and_footers": False,
"detection_origin": None,
}
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
@pytest.fixture
def pages_prop_(request: FixtureRequest):
return property_mock(request, HTMLDocument, "pages")
Reference in New Issue Copy Permalink