yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-08 01:22:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured/documents/test_html.py

517 lines
17 KiB
Python
Raw Normal View History

rfctr(html): clean types and docs in prep for HTML table parsing fixes (#2104) There are a cluster of bugs in the HTML parsing code, particularly surrounding table behaviors but also inclusion of style elements, etc. Clean up typing and docstrings in that neighborhood as a way to familiarize myself with that part of the code-base.
2023-11-17 09:56:38 -08:00
# pyright: reportPrivateUsage=false
rfctr(html): improve SNR in HTMLDocument (#3162) **Summary** Remove dead code and organize helpers of HTMLDocument in preparation for improvements and bug-fixes to follow
2024-06-06 14:21:33 -07:00
# pyright: reportUnusedFunction=false
rfctr(html): clean types and docs in prep for HTML table parsing fixes (#2104) There are a cluster of bugs in the HTML parsing code, particularly surrounding table behaviors but also inclusion of style elements, etc. Clean up typing and docstrings in that neighborhood as a way to familiarize myself with that part of the code-base.
2023-11-17 09:56:38 -08:00
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
"""Test suite for `unstructured.documents.html` module."""
from __future__ import annotations
fix: Remove JavaScript from HTML reader output (#313) * Fixes an error causing JavaScript to appear in the output of `partition_html` sometimes.
2023-02-28 23:24:24 +01:00
import pathlib
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
from typing import Any
Initial Release
2022-06-29 14:35:19 -04:00
import pytest
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
from lxml import etree
Initial Release
2022-06-29 14:35:19 -04:00
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
from test_unstructured.unit_utils import (
FixtureRequest,
Mock,
function_mock,
property_mock,
)
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
from unstructured.documents import html
from unstructured.documents.elements import (
ListItem,
NarrativeText,
Adding table extraction to partition_html (#1324) Adding table extraction to HTML partitioning. This PR utilizes 'table' HTML elements to extract and parse HTML tables and return them in partitioning. ``` # checkout this branch, go into ipython shell In [1]: from unstructured.partition.html import partition_html In [2]: path_to_html = "{html sample file with table}" In [3]: elements = partition_html(path_to_html) ``` you should see the table in the elements list!
2023-09-11 11:14:11 -07:00
Table,
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
Text,
Title,
)
rfctr(html): prepare for new html parser (#3257) **Summary** Extract as much mechanical refactoring from the HTML parser change-over into the PR as possible. This leaves the next PR focused on installing the new parser and the ingest-test impact. **Reviewers:** Commits are well groomed and reviewing commit-by-commit is probably easier. **Additional Context** This PR introduces the rewritten HTML parser. Its general design is recursive, consistent with the recursive structure of HTML (tree of elements). It also adds the unit tests for that parser but it does not _install_ the parser. So the behavior of `partition_html()` is unchanged by this PR. The next PR in this series will do that and handle the ingest and other unit test changes required to reflect the dozen or so bug-fixes the new parser provides.
2024-06-21 13:59:48 -07:00
from unstructured.documents.html import HTMLDocument
from unstructured.partition.html.partition import HtmlPartitionerOptions
Initial Release
2022-06-29 14:35:19 -04:00
TAGS = (
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
(
"<a><abbr><acronym><address><applet><area><article><aside><audio><b><base><basefont><bdi>"
"<bdo><big><blockquote><body><br><button><canvas><caption><center><cite><code><col>"
"<colgroup><data><datalist><dd><del><details><dfn><dialog><dir><div><dl><dt><em><embed>"
"<fieldset><figcaption><figure><font><footer><form><frame><frameset><h1><h2><h3><h4><h5>"
"<h6><head><header><hr><html><i><iframe><img><input><ins><kbd><label><legend><li><link>"
"<main><map><mark><meta><meter><nav><noframes><noscript><object><ol><optgroup><option>"
"<output><p><param><picture><pre><progress><q><rp><rt><ruby><s><samp><script><section>"
"<select><small><source><span><strike><strong><style><sub><summary><sup><table><tbody><td>"
"<template><textarea><tfoot><th><thead><time><title><tr><track><tt><u><ul><var><video><wbr>"
)
.replace(">", "")
.split("<")[1:]
Initial Release
2022-06-29 14:35:19 -04:00
)
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
VOID_TAGS = (
("<area><base><br><col><embed><hr><img><input><link><meta><param><source><track><wbr>")
.replace(">", "")
.split("<")[1:]
)
fix: Fix various cases of HTML text missing after partition (#1587) Fix 4 cases of text missing after partition: 1. Text immediately after `<body>` ```html <body> missing1 <div>hello</div> </body> ``` 2. Text inside container and immediately after `<br/>` ```html <div>hello<br/>missing2</div> ``` 3. Text immediately after a text opening tag, if said tag contains `<br/>` ```html <p>missing3<br/>hello</p> ``` 4. Text inside `<body>` if it is the only content (different cause from case 1) ```html <body>missing4</body> ``` Also fix problem causing `test_unstructured/documents/test_html.py::test_exclude_tag_types` to not work as intended. This will close GitHub Issue#1543
2023-10-03 12:17:51 +08:00
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
INCLUDED_TAGS = html.TEXT_TAGS + html.HEADING_TAGS + html.LIST_ITEM_TAGS + html.SECTION_TAGS
fix: Fix various cases of HTML text missing after partition (#1587) Fix 4 cases of text missing after partition: 1. Text immediately after `<body>` ```html <body> missing1 <div>hello</div> </body> ``` 2. Text inside container and immediately after `<br/>` ```html <div>hello<br/>missing2</div> ``` 3. Text immediately after a text opening tag, if said tag contains `<br/>` ```html <p>missing3<br/>hello</p> ``` 4. Text inside `<body>` if it is the only content (different cause from case 1) ```html <body>missing4</body> ``` Also fix problem causing `test_unstructured/documents/test_html.py::test_exclude_tag_types` to not work as intended. This will close GitHub Issue#1543
2023-10-03 12:17:51 +08:00
EXCLUDED_TAGS = [
tag
for tag in TAGS
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
if tag not in (INCLUDED_TAGS + html.TABLE_TAGS + VOID_TAGS + ["html", "head", "body"])
fix: Fix various cases of HTML text missing after partition (#1587) Fix 4 cases of text missing after partition: 1. Text immediately after `<body>` ```html <body> missing1 <div>hello</div> </body> ``` 2. Text inside container and immediately after `<br/>` ```html <div>hello<br/>missing2</div> ``` 3. Text immediately after a text opening tag, if said tag contains `<br/>` ```html <p>missing3<br/>hello</p> ``` 4. Text inside `<body>` if it is the only content (different cause from case 1) ```html <body>missing4</body> ``` Also fix problem causing `test_unstructured/documents/test_html.py::test_exclude_tag_types` to not work as intended. This will close GitHub Issue#1543
2023-10-03 12:17:51 +08:00
]
Initial Release
2022-06-29 14:35:19 -04:00
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# -- HTMLDocument.from_file() --------------------------------------------------------------------
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_read_html_doc(tmp_path: pathlib.Path, opts_args: dict[str, Any]):
file_path = str(tmp_path / "sample-doc.html")
with open(file_path, "w") as f:
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
f.write(
"<html>\n"
" <body>\n"
" <h1>A Great and Glorious Section</h1>\n"
" <p>Dear Leader is the best. He is such a wonderful engineer!</p>\n"
" <p></p>\n"
" <p>Another Magnificent Title</p>\n"
" <p><b>The prior element is a title based on its capitalization patterns!</b></p>\n"
" <table>\n"
" <tbody>\n"
" <tr>\n"
rfctr(html): improve SNR in HTMLDocument (#3162) **Summary** Remove dead code and organize helpers of HTMLDocument in preparation for improvements and bug-fixes to follow
2024-06-06 14:21:33 -07:00
" <td><p>I'm in a table</p></td>\n"
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
" </tr>\n"
" </tbody>\n"
" </table>\n"
" <h2>A New Beginning</h2>\n"
" <div>Here is the start of a new page.</div>\n"
" </body>\n"
"</html>\n"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts_args["file_path"] = file_path
opts = HtmlPartitionerOptions(**opts_args)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
elements = HTMLDocument.load(opts).elements
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
assert len(elements) == 7
assert elements == [
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
Title("A Great and Glorious Section"),
NarrativeText("Dear Leader is the best. He is such a wonderful engineer!"),
Title("Another Magnificent Title"),
NarrativeText("The prior element is a title based on its capitalization patterns!"),
rfctr(html): improve SNR in HTMLDocument (#3162) **Summary** Remove dead code and organize helpers of HTMLDocument in preparation for improvements and bug-fixes to follow
2024-06-06 14:21:33 -07:00
Table("I'm in a table"),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
Title("A New Beginning"),
NarrativeText("Here is the start of a new page."),
]
# -- HTMLDocument.elements -----------------------------------------------------------------------
Initial Release
2022-06-29 14:35:19 -04:00
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
@pytest.mark.parametrize("tag", html.TEXT_TAGS)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def test_line_break_in_text_tag(tag: str, opts_args: dict[str, Any]):
opts_args["text"] = f"<{tag}>Hello<br/>World</{tag}>"
opts = HtmlPartitionerOptions(**opts_args)
doc = HTMLDocument.load(opts)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert doc.elements[0].text == "Hello"
assert doc.elements[1].text == "World"
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
@pytest.mark.parametrize("tag", [tag for tag in html.TEXT_TAGS if tag not in html.TABLE_TAGS])
def test_tag_types(tag: str, opts_args: dict[str, Any]):
opts_args["text"] = f"<body>\n <{tag}>\n There is some text here.\n </{tag}>\n</body>\n"
opts = HtmlPartitionerOptions(**opts_args)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
elements = HTMLDocument.load(opts).elements
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
assert len(elements) == 1
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# -- _construct_text() ---------------------------------------------------------------------------
Initial Release
2022-06-29 14:35:19 -04:00
@pytest.mark.parametrize(
Resolve various style issues to improve overall code quality (#282) * Apply import sorting ruff . --select I --fix * Remove unnecessary open mode parameter ruff . --select UP015 --fix * Use f-string formatting rather than .format * Remove extraneous parentheses Also use "" instead of str() * Resolve missing trailing commas ruff . --select COM --fix * Rewrite list() and dict() calls using literals ruff . --select C4 --fix * Add () to pytest.fixture, use tuples for parametrize, etc. ruff . --select PT --fix * Simplify code: merge conditionals, context managers ruff . --select SIM --fix * Import without unnecessary alias ruff . --select PLR0402 --fix * Apply formatting via black * Rewrite ValueError somewhat Slightly unrelated to the rest of the PR * Apply formatting to tests via black * Update expected exception message to match 0d81564 * Satisfy E501 line too long in test * Update changelog & version * Add ruff to make tidy and test deps * Run 'make tidy' * Update changelog & version * Update changelog & version * Add ruff to 'check' target Doing so required me to also fix some non-auto-fixable issues. Two of them I fixed with a noqa: SIM115, but especially the one in __init__ may need some attention. That said, that refactor is out of scope of this PR.
2023-02-27 17:30:54 +01:00
("doc", "expected"),
Initial Release
2022-06-29 14:35:19 -04:00
[
(
"<p>Hi there <span>my name is</span> <b><i>Matt</i></i></p>",
"Hi there my name is Matt",
),
("<p>I have a</p> tail", "I have a tail"),
],
)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
def test_construct_text(doc: str, expected: str):
Initial Release
2022-06-29 14:35:19 -04:00
document_tree = etree.fromstring(doc, etree.HTMLParser())
para = document_tree.find(".//p")
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert para is not None
Initial Release
2022-06-29 14:35:19 -04:00
text = html._construct_text(para)
assert text == expected
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# -- _get_emphasized_texts_from_tag() ------------------------------------------------------------
@pytest.mark.parametrize(
("doc", "root", "expected"),
[
(
"<p>Hello <strong>there</strong> I <em>am</em> a <b>very</b> <i>important</i> text</p>",
"p",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["there", "am", "very", "important"], ["strong", "em", "b", "i"]),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
),
(
"<p>Here is a <span>list</span> of <b>my <i>favorite</i> things</b></p>",
"p",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["list", "my favorite things", "favorite"], ["span", "b", "i"]),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
),
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
("<strong>A lone strong text!</strong>", "strong", (["A lone strong text!"], ["strong"])),
("<span>I have a</span> tail", "span", (["I have a"], ["span"])),
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
],
)
def test_get_emphasized_texts_from_tag(doc: str, root: str, expected: list[dict[str, str]]):
document_tree = etree.fromstring(doc, etree.HTMLParser())
el = document_tree.find(f".//{root}")
assert el is not None
emphasized_texts = html._get_emphasized_texts_from_tag(el)
assert emphasized_texts == expected
# -- _get_links_from_tag() -----------------------------------------------------------------------
@pytest.mark.parametrize(
("doc", "root", "expected"),
[
(
"<a href='/loner'>A lone link!</a>",
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
"a",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["A lone link!"], ["/loner"], [-1]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
(
"<ul><li><a href='/wiki/Parrot'>Parrots</a></li><li>Dogs</li></ul>",
"ul",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["Parrots"], ["/wiki/Parrot"], [0]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
(
"<ul><li><a href='/parrot'>Parrots</a></li><li><a href='/dog'>Dogs</a></li></ul>",
"ul",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["Parrots", "Dogs"], ["/parrot", "/dog"], [0, 7]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
(
"<div>Here is <p>P tag</p> tail text. <a href='/link'>link!</a></div>",
"div",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["link!"], ["/link"], [25]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
(
"<div>Here is <p>P tag</p><a href='/link'>link!</a></div>",
"div",
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
(["link!"], ["/link"], [13]),
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
),
],
)
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
def test_get_links_from_tag(doc: str, root: str, expected: list[dict[str, str]]):
add support for `start_index` in `html` links extraction (#2600) add support for start_index in html links extraction (closes #2625) Testing ``` from unstructured.partition.html import partition_html from unstructured.staging.base import elements_to_json html_text = """<html> <p>Hello there I am a <a href="/link">very important link!</a></p> <p>Here is a list of my favorite things</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Parrot">Parrots</a></li> <li>Dogs</li> </ul> <a href="/loner">A lone link!</a> </html>""" elements = partition_html(text=html_text) print(elements_to_json(elements)) ``` --------- Co-authored-by: Michael Niestroj <michael.niestroj@unblu.com> Co-authored-by: christinestraub <christinemstraub@gmail.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-04-12 08:14:20 +02:00
document_tree = etree.fromstring(doc, etree.HTMLParser())
el = document_tree.find(f".//{root}")
assert el is not None
links = html._get_links_from_tag(el)
assert links == expected
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
# -- _is_text_tag() ------------------------------------------------------------------------------
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def test_adjacent_spans_are_text_tags(opts_args: dict[str, Any]):
html_str = "<div><span>&#8226;</span><span>A bullet!</span></div>"
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
assert el is not None
Initial Release
2022-06-29 14:35:19 -04:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_document._is_text_tag(el) is True
Initial Release
2022-06-29 14:35:19 -04:00
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
# -- unit-level tests ----------------------------------------------------------------------------
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
class DescribeHTMLDocument:
"""Unit-test suite for `unstructured.documents.html.HTMLDocument`."""
# -- ._main ----------------------------------
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def it_can_find_the_main_element_in_the_document(self, opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
"<header></header>\n"
"<body>\n"
" <p>Lots preamble stuff yada yada yada</p>\n"
" <main>\n"
" <article>\n"
" <section>\n"
" <h2>A Wonderful Section!</h2>\n"
" <p>Look at this amazing section!</p>\n"
" </section>\n"
" <section>\n"
" <h2>Another Wonderful Section!</h2>\n"
" <p>Look at this other amazing section!</p>\n"
" </section>\n"
" </article>\n"
" </main>\n"
"</body>\n"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
assert html_document._main.tag == "main"
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
def but_it_returns_the_root_when_no_main_element_is_present(self, opts_args: dict[str, Any]):
opts_args["text"] = (
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
"<header></header>\n"
"<body>\n"
" <p>Lots preamble stuff yada yada yada</p>\n"
" <article>\n"
" <section>\n"
" <h2>A Wonderful Section!</h2>\n"
" <p>Look at this amazing section!</p>\n"
" </section>\n"
" <section>\n"
" <h2>Another Wonderful Section!</h2>\n"
" <p>Look at this other amazing section!</p>\n"
" </section>\n"
" </article>\n"
"</body>\n"
)
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
opts = HtmlPartitionerOptions(**opts_args)
html_document = HTMLDocument.load(opts)
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
assert html_document._main.tag == "html"
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
# -- ._parse_bulleted_text_from_table() ------
def it_extracts_bullet_text_from_table_as_ListItem_elements(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<html>
<body>
<table>
<tbody>
<tr>
<td>•</td>
<td><p>Happy Groundhog's day!</p></td>
</tr>
<tr>
<td>•</td>
<td><p>Looks like six more weeks of winter ...</p></td>
</tr>
</tbody>
</table>
</body>
</html>
"""
html_document = HTMLDocument(html_str, opts)
table_elem = html_document._main.find(".//table")
assert table_elem is not None
list_item_iter = html_document._parse_bulleted_text_from_table(table_elem)
assert next(list_item_iter) == ListItem(text="Happy Groundhog's day!")
assert next(list_item_iter) == ListItem(text="Looks like six more weeks of winter ...")
with pytest.raises(StopIteration):
next(list_item_iter)
rfctr(html): promote HTMLDoc candidate methods (#3177) **Summary** Make `._find_articles()` and `._find_main` into `._articles` and `._main` properties on HTMLDocument, respectively. **Additional Context** After prior refactorings, these two functions now each require only `self` and can become `@lazyproperty`s on `HTMLDocument`. This ensures they are computed at most once. In addition, their close relationship to `HTMLDocument` is indicated by their membership as methods rather than "loose" functions.
2024-06-10 15:07:21 -07:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
# -- ._parse_Table_from_table_elem() ---------
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def it_produces_one_cell_for_each_original_table_cell(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = (
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
# -- include formatting whitespace to make sure it is removed --
"<table>\n"
" <tr>\n"
" <td>foo</td>\n"
" <td>bar</td>\n"
" </tr>\n"
"</table>"
)
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_document = HTMLDocument(html_str, opts)
table_elem = html_document._main.find(".//table")
assert table_elem is not None
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
html_table = html_document._parse_Table_from_table_elem(table_elem)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert isinstance(html_table, Table)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
assert html_table.text == "foo bar"
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_table.metadata.text_as_html == (
"<table><tr><td>foo</td><td>bar</td></tr></table>"
)
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
# -- ._parse_tag() ---------------------------
fix(html): unequal row lengths in HTMLTable.text_as_html (#2345) Fixes #2339 Fixes to HTML partitioning introduced with v0.11.0 removed the use of `tabulate` for forming the HTML placed in `HTMLTable.text_as_html`. This had several benefits, but part of `tabulate`'s behavior was to make row-length (cell-count) uniform across the rows of the table. Lacking this prior uniformity produced a downstream problem reported in On closer inspection, the method used to "harvest" cell-text was producing more text-nodes than there were cells and was sensitive to where whitespace was used to format the HTML. It also "moved" text to different columns in certain rows. Refine the cell-text gathering mechanism to get exactly one text string for each row cell, eliminating whitespace formatting nodes and producing strict correspondence between the number of cells in the original HTML table row and that placed in HTML.text_as_html. HTML tables that are uniform (every row has the same number of cells) will produce a uniform table in `.text_as_html`. Merged cells may still produce a non-uniform table in `.text_as_html` (because the source table is non-uniform).
2024-01-04 13:53:19 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def it_produces_a_Text_element_when_the_tag_contents_are_not_narrative_or_a_title(
self,
is_possible_title_: Mock,
opts_args: dict[str, Any],
):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p>NO PARTICULAR TYPE.</p>"
is_possible_title_.return_value = False
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
assert html_document._parse_tag(p) == Text(text="NO PARTICULAR TYPE.")
fix: assorted partition_html() bugs (#2113) Addresses a cluster of HTML-related bugs: - empty table is identified as bulleted-table - `partition_html()` emits empty (no text) tables (#1928) - `.text_as_html` contains inappropriate `<br>` elements in invalid locations. - cells enclosed in `<thead>` and `<tfoot>` elements are dropped (#1928) - `.text_as_html` contains whitespace padding Each of these is addressed in a separate commit below. Fixes #1928. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: scanny <scanny@users.noreply.github.com> Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-11-20 08:29:32 -08:00
rfctr(html): drop HTML-specific elements (#3207) **Summary** Remove HTML-specific element types and return "regular" elements like `Title` and `NarrativeText` from `partition_html()`. **Additional Context** - An aspect of the legacy HTML partitioner was the use of HTML-specific element types used to track metadata during partitioning. - That role is no longer necessary or desireable. - HTML-specific elements like `HTMLTitle` and `HTMLNarrativeText` were returned from partitioning HTML but also the seven other file-formats that broker partitioning to HTML (convert-to-HTML and partition_html()). This does not cause immediate breakage because these are still `Text` element subtypes, but it produces a confusing developer experience. - Remove the prior metadata roles from HTML-specific elements and remove those element types entirely.
2024-06-14 17:14:22 -07:00
def it_produces_a_ListItem_element_when_the_tag_contains_are_preceded_by_a_bullet_character(
self, opts_args: dict[str, Any]
):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p>● An excellent point!</p>"
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
assert html_document._parse_tag(p) == ListItem("An excellent point!")
def but_not_when_the_tag_contains_only_a_bullet_character_and_no_text(
self, opts_args: dict[str, Any]
):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p>●</p>"
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
assert html_document._parse_tag(p) is None
def it_returns_None_when_the_tag_has_no_content(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p></p>"
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
assert html_document._parse_tag(p) is None
def and_it_returns_None_when_the_tag_contains_only_a_stub(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = "<p>$</p>"
html_document = HTMLDocument(html_str, opts)
p = html_document._main.find(".//p")
assert p is not None
assert html_document._parse_tag(p) is None
# -- ._process_list_item() -------------------------------------------------------------------
def it_continues_processing_elements_after_a_bulleted_list(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<div>
<p>●</p>
<p>●</p>
</div>
<div>
<p>An excellent point!</p>
</div>
"""
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
assert el is not None
parsed_el, _ = html_document._process_list_item(el, max_predecessor_len=10)
assert parsed_el == ListItem(text="An excellent point!")
def but_it_returns_None_when_no_element_follows_empty_bullets(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<div>
<p>●</p>
<p>●</p>
</div>
"""
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
assert el is not None
parsed_el, _ = html_document._process_list_item(el)
assert parsed_el is None
def and_it_returns_None_when_following_element_contains_no_text(
self, opts_args: dict[str, Any]
):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<div>
<p>●</p>
<p>●</p>
</div>
<div>
</div>
"""
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
assert el is not None
parsed_el, _ = html_document._process_list_item(el)
assert parsed_el is None
def it_ignores_deep_divs_when_so_instructed(self, opts_args: dict[str, Any]):
opts = HtmlPartitionerOptions(**opts_args)
html_str = """
<div>
<p>●</p>
<p>●</p>
<p>●</p>
<p>●</p>
<p>●</p>
</div>
<div>
<p>An excellent point!</p>
</div>
"""
html_document = HTMLDocument(html_str, opts)
el = html_document._main.find(".//div")
assert el is not None
parsed_el, _ = html_document._process_list_item(el, max_predecessor_len=2)
assert parsed_el is None
# -- module-level fixtures -----------------------------------------------------------------------
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
@pytest.fixture
def is_possible_narrative_text_(request: FixtureRequest):
return function_mock(request, "unstructured.documents.html.is_possible_narrative_text")
@pytest.fixture
def is_possible_title_(request: FixtureRequest):
return function_mock(request, "unstructured.documents.html.is_possible_title")
rfctr(html): drop page concept (#3184) **Summary** Pagination of HTML documents is currently unused. The `Page` class and concept were deeply embedding in the legacy organization of HTML partitioning code due to the legacy `Document` (= pages of elements) domain model. Remove this concept from the code such that elements are available directly from the partitioner. **Additional Context** - Pagination can be re-added later if we decide we want it again. A re-implementation would be much simpler and much lower impact to the structure of the code and introduce much less additional complexity, similar to the approach we take in `partition_docx()`.
2024-06-13 11:19:42 -07:00
@pytest.fixture
def opts_args() -> dict[str, Any]:
"""All default arguments for `HtmlPartitionerOptions`.
Individual argument values can be changed to suit each test. Makes construction of opts more
compact for testing purposes.
"""
return {
"file": None,
"file_path": None,
"text": None,
"encoding": None,
"url": None,
"headers": {},
"ssl_verify": True,
"date_from_file_object": False,
"metadata_last_modified": None,
"skip_headers_and_footers": False,
"detection_origin": None,
}
rfctr(html): organize and improve HTMLDocument tests (#3161) **Summary** In preparation for further work on HTMLDocument, organize the organic growth in `documents/tests_html.py` and improving typing and expression. **Reviewers:** Commits are groomed and review is probably eased by going commit-by-commit
2024-06-06 11:16:02 -07:00
@pytest.fixture
def pages_prop_(request: FixtureRequest):
return property_mock(request, HTMLDocument, "pages")
Reference in New Issue Copy Permalink