yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-10-27 16:11:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: method to catch and classify overlapping bounding boxes (#1803)

Browse Source 
We have established that overlapping bounding boxes does not have a
one-fits-all solution, so different cases need to be handled differently
to avoid information loss. We have manually identified the
cases/categories of overlapping. Now we need a method to
programmatically classify overlapping-bboxes cases within detected
elements in a document, and return a report about it (list of cases with
metadata). This fits two purposes:

- **Evaluation**: We can have a pipeline using the DVC data registry
that assess the performance of a detection model against a set of
documents (PDF/Images), by analysing the overlapping-bboxes cases it
has. The metadata in the output can be used for generating metrics for
this.
- **Scope overlapping cases**: Manual inspection give us a clue about
currently present cases of overlapping bboxes. We need to propose
solutions to fix those on code. This method generates a report by
analysing several aspects of two overlapping regions. This data can be
used to profile and specify the necessary changes that will fix each
case.
- **Fix overlapping cases**: We could introduce this functionality in
the flow of a partition method (such as `partition_pdf`, to handle the
calls to post-processing methods to fix overlapping. Tested on ~331
documents, the worst time per page is around 5ms. For a document such as
`layout-parser-paper.pdf` it takes 4.46 ms.

Introduces functionality to take a list of unstructured elements (which
contain bounding boxes) and identify pairs of bounding boxes which
overlap and which case is pertinent to the pairing. This PR includes the
following methods in `utils.py`:

- **`ngrams(s, n)`**: Generate n-grams from a string
- **`calculate_shared_ngram_percentage(string_A, string_B, n)`**:
Calculate the percentage of `common_ngrams` between `string_A` and
`string_B` with reference to the total number of ngrams in `string_A`.
- **`calculate_largest_ngram_percentage(string_A, string_B)`**:
Iteratively call `calculate_shared_ngram_percentage` starting from the
biggest ngram possible until the shared percentage is >0.0%
- **`is_parent_box(parent_target, child_target, add=0)`**: True if the
`child_target` bounding box is nested in the `parent_target` Box format:
[`x_bottom_left`, `y_bottom_left`, `x_top_right`, `y_top_right`]. The
parameter 'add' is the pixel error tolerance for extra pixels outside
the parent region
- **`calculate_overlap_percentage(box1, box2,
intersection_ratio_method="total")`**: Box format: [`x_bottom_left`,
`y_bottom_left`, `x_top_right`, `y_top_right`]. Calculates the
percentage of overlapped region with reference to biggest element-region
(`intersection_ratio_method="parent"`), the smallest element-region
(`intersection_ratio_method="partial"`), or to the disjunctive union
region (`intersection_ratio_method="total"`).
- **`identify_overlapping_or_nesting_case`**: Identify if there are
nested or overlapping elements. If overlapping is present,
it identifies the case calling the method `identify_overlapping_case`.
- **`identify_overlapping_case`**: Classifies the overlapping case for
an element_pair input in one of 5 categories of overlapping.
- **`catch_overlapping_and_nested_bboxes`**: Catch overlapping and
nested bounding boxes cases across a list of elements. The params
`nested_error_tolerance_px` and `sm_overlap_threshold` help controling
the separation of the cases.

The overlapping/nested elements cases that are being caught are:
1. **Nested elements**
2. **Small partial overlap**
3. **Partial overlap with empty content**
4. **Partial overlap with duplicate text (sharing 100% of the text)**
5. **Partial overlap without sharing text**
6. **Partial overlap sharing**
{`calculate_largest_ngram_percentage(...)`}% **of the text**

Here is a snippet to test it:
```
from unstructured.partition.auto import partition

model_name = "yolox_quantized"
target = "sample-docs/layout-parser-paper-fast.pdf"
elements = partition(filename=file_path_i, strategy='hi_res', model_name=model_name)
overlapping_flag, overlapping_cases = catch_overlapping_bboxes(elements)
for case in overlapping_cases:
    print(case, "\n")
```
Here is a screenshot of a json built with the output list
`overlapping_cases`:
<img width="377" alt="image"
src="https://github.com/Unstructured-IO/unstructured/assets/38184042/a6fea64b-d40a-4e01-beda-27840f4f4b3a">
This commit is contained in:
Sebastian Laverde Alfonso 2023-10-25 05:17:34 -07:00 committed by GitHub
parent d8241cbcfc
commit c11a2ff478
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 622 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CHANGELOG.md
View File

@ -6,6 +6,8 @@
### Features
* **Functionality to catch and classify overlapping/nested elements** Method to identify overlapping-bboxes cases within detected elements in a document. It returns two values: a boolean defining if there are overlapping elements present, and a list reporting them with relevant metadata. The output includes information about the `overlapping_elements`, `overlapping_case`, `overlapping_percentage`, `largest_ngram_percentage`, `overlap_percentage_total`, `max_area`, `min_area`, and `total_area`.
* **Add Local connector source metadata** python's os module used to pull stats from local file when processing via the local connector and populates fields such as last modified time, created time.
* **Add Local connector source metadata.** python's os module used to pull stats from local file when processing via the local connector and populates fields such as last modified time, created time.
### Fixes

217
test_unstructured/test_utils.py
View File

@ -4,6 +4,8 @@ import os
import pytest
from unstructured import utils
from unstructured.documents.coordinates import PixelSpace
from unstructured.documents.elements import ElementMetadata, NarrativeText, Title
@pytest.fixture()
@ -110,3 +112,218 @@ def test_only_raises_when_len_more_than_1(iterator):
def test_only_raises_if_empty(iterator):
with pytest.raises(ValueError):
utils.only(iterator)
@pytest.mark.parametrize(
("elements", "nested_error_tolerance_px", "sm_overlap_threshold", "expectation"),
[
(
[
Title(
text="Some lovely title",
coordinates=((4, 5), (4, 8), (7, 8), (7, 5)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
NarrativeText(
text="Some lovely text",
coordinates=((2, 3), (2, 6), (5, 6), (5, 3)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
],
5,
10.0,
(
True,
[
{
"overlapping_elements": ["Title(ix=0)", "NarrativeText(ix=1)"],
"overlapping_case": "nested NarrativeText in Title",
"overlap_percentage": "100%",
"metadata": {
"largest_ngram_percentage": None,
"overlap_percentage_total": "5.88%",
"max_area": "9pxˆ2",
"min_area": "9pxˆ2",
"total_area": "18pxˆ2",
},
},
],
),
),
(
[
Title(
text="Some lovely title",
coordinates=((4, 5), (4, 8), (7, 8), (7, 5)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
NarrativeText(
text="Some lovely text",
coordinates=((2, 3), (2, 6), (5, 6), (5, 3)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
],
1,
10.0,
(
True,
[
{
"overlapping_elements": ["0. Title(ix=0)", "1. NarrativeText(ix=1)"],
"overlapping_case": "partial overlap sharing 50.0% of the text from1. "
"NarrativeText(2-gram)",
"overlap_percentage": "11.11%",
"metadata": {
"largest_ngram_percentage": 50.0,
"overlap_percentage_total": "5.88%",
"max_area": "9pxˆ2",
"min_area": "9pxˆ2",
"total_area": "18pxˆ2",
},
},
],
),
),
(
[
Title(
text="Some lovely title",
coordinates=((4, 5), (4, 8), (7, 8), (7, 5)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
NarrativeText(
text="Some lovely title",
coordinates=((2, 3), (2, 6), (5, 6), (5, 3)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
],
1,
10.0,
(
True,
[
{
"overlapping_elements": ["0. Title(ix=0)", "1. NarrativeText(ix=1)"],
"overlapping_case": "partial overlap with duplicate text",
"overlap_percentage": "11.11%",
"metadata": {
"largest_ngram_percentage": None,
"overlap_percentage_total": "5.88%",
"max_area": "9pxˆ2",
"min_area": "9pxˆ2",
"total_area": "18pxˆ2",
},
},
],
),
),
(
[
Title(
text="Some lovely title",
coordinates=((4, 5), (4, 8), (7, 8), (7, 5)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
NarrativeText(
text="Something totally different here",
coordinates=((2, 3), (2, 6), (5, 6), (5, 3)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
],
1,
10.0,
(
True,
[
{
"overlapping_elements": ["0. Title(ix=0)", "1. NarrativeText(ix=1)"],
"overlapping_case": "partial overlap without sharing text",
"overlap_percentage": "11.11%",
"metadata": {
"largest_ngram_percentage": 0,
"overlap_percentage_total": "5.88%",
"max_area": "9pxˆ2",
"min_area": "9pxˆ2",
"total_area": "18pxˆ2",
},
},
],
),
),
(
[
Title(
text="Some lovely title",
coordinates=((5, 6), (5, 10), (8, 10), (8, 6)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
NarrativeText(
text="Some lovely text",
coordinates=((1, 3), (2, 7), (6, 7), (5, 3)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
],
1,
10.0,
(
True,
[
{
"overlapping_elements": ["0. Title(ix=0)", "1. NarrativeText(ix=1)"],
"overlapping_case": "Small partial overlap",
"overlap_percentage": "8.33%",
"metadata": {
"largest_ngram_percentage": None,
"overlap_percentage_total": "3.23%",
"max_area": "20pxˆ2",
"min_area": "12pxˆ2",
"total_area": "32pxˆ2",
},
},
],
),
),
(
[
Title(
text="Some lovely title",
coordinates=((4, 6), (4, 7), (7, 7), (7, 6)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
NarrativeText(
text="Some lovely text",
coordinates=((6, 8), (6, 9), (9, 9), (9, 8)),
coordinate_system=PixelSpace(width=20, height=20),
metadata=ElementMetadata(page_number=1),
),
],
1,
10.0,
(False, []),
),
],
)
def test_catch_overlapping_and_nested_bboxes(
elements,
expectation,
nested_error_tolerance_px,
sm_overlap_threshold,
):
overlapping_flag, overlapping_cases = utils.catch_overlapping_and_nested_bboxes(
elements,
nested_error_tolerance_px,
sm_overlap_threshold,
)
assert overlapping_flag == expectation[0]
assert overlapping_cases == expectation[1]

404
unstructured/utils.py
View File

@ -6,6 +6,7 @@ import platform
import subprocess
from datetime import datetime
from functools import wraps
from itertools import combinations
from typing import (
Any,
Callable,
@ -25,10 +26,10 @@ import requests
from typing_extensions import ParamSpec
from unstructured.__version__ import __version__
from unstructured.documents.elements import Text
DATE_FORMATS = ("%Y-%m-%d", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d+%H:%M:%S", "%Y-%m-%dT%H:%M:%S%z")
_T = TypeVar("_T")
_P = ParamSpec("_P")
@ -280,3 +281,404 @@ def scarf_analytics():
)
except Exception:
pass
def ngrams(s: str, n: int) -> List:
"""Generate n-grams from a string s"""
ngrams_list = []
for i in range(len(s) - n + 1):
ngram = []
for j in range(n):
ngram.append(s[i + j])
ngrams_list.append(tuple(ngram))
return ngrams_list
def calculate_shared_ngram_percentage(
first_string: str,
second_string: str,
n: int,
) -> (float, List):
"""Calculate the percentage of common_ngrams between string_A and string_B
with reference to the total number of ngrams in string_A"""
if not n:
return 0, {}
first_string_ngrams = ngrams(first_string.split(), n)
second_string_ngrams = ngrams(second_string.split(), n)
if not first_string_ngrams:
return 0
common_ngrams = set(first_string_ngrams) & set(second_string_ngrams)
percentage = (len(common_ngrams) / len(first_string_ngrams)) * 100
return percentage, common_ngrams
def calculate_largest_ngram_percentage(first_string: str, second_string: str) -> (float, List, str):
"""Iteratively calculate_shared_ngram_percentage starting from the biggest
ngram possible until is >0.0%"""
shared_ngrams = []
if len(first_string.split()) < len(second_string.split()):
n = len(first_string.split()) - 1
else:
n = len(second_string.split()) - 1
first_string, second_string = second_string, first_string
ngram_percentage = 0
while not ngram_percentage:
ngram_percentage, shared_ngrams = calculate_shared_ngram_percentage(
first_string,
second_string,
n,
)
if n == 0:
break
else:
n -= 1
return round(ngram_percentage, 2), shared_ngrams, str(n + 1)
def is_parent_box(
parent_target: Union[List, Tuple],
child_target: Union[List, Tuple],
add: float = 0.0,
) -> bool:
"""True if the child_target bounding box is nested in the parent_target.
Box format: [x_bottom_left, y_bottom_left, x_top_right, y_top_right].
The parameter 'add' is the pixel error tolerance for extra pixels outside the parent region"""
if len(parent_target) != 4:
return False
if add and len(parent_target) == 4:
parent_target = list(parent_target)
parent_target[0] -= add
parent_target[1] -= add
parent_target[2] += add
parent_target[3] += add
if (
len(child_target) == 4
and (child_target[0] >= parent_target[0] and child_target[1] >= parent_target[1])
and (child_target[2] <= parent_target[2] and child_target[3] <= parent_target[3])
):
return True
if len(child_target) == 2 and (
parent_target[0] <= child_target[0] <= parent_target[2]
and parent_target[1] <= child_target[1] <= parent_target[3]
):
return True
return False
def calculate_overlap_percentage(
box1: Union[List, Tuple],
box2: Union[List, Tuple],
intersection_ratio_method: str = "total",
):
"""Box format: [x_bottom_left, y_bottom_left, x_top_right, y_top_right].
Calculates the percentage of overlapped region with reference to
the biggest element-region (intersection_ratio_method="parent"),
the smallest element-region (intersection_ratio_method="partial"), or to
the disjunctive union region (intersection_ratio_method="total")
"""
x1, y1 = box1[0]
x2, y2 = box1[2]
x3, y3 = box2[0]
x4, y4 = box2[2]
area_box1 = (x2 - x1) * (y2 - y1)
area_box2 = (x4 - x3) * (y4 - y3)
x_intersection1 = max(x1, x3)
y_intersection1 = max(y1, y3)
x_intersection2 = min(x2, x4)
y_intersection2 = min(y2, y4)
intersection_area = max(0, x_intersection2 - x_intersection1) * max(
0,
y_intersection2 - y_intersection1,
)
max_area = max(area_box1, area_box2)
min_area = min(area_box1, area_box2)
total_area = area_box1 + area_box2
if intersection_ratio_method == "parent":
if max_area == 0:
return 0
overlap_percentage = (intersection_area / max_area) * 100
elif intersection_ratio_method == "partial":
if min_area == 0:
return 0
overlap_percentage = (intersection_area / min_area) * 100
else:
if (area_box1 + area_box2) == 0:
return 0
overlap_percentage = (intersection_area / (area_box1 + area_box2 - intersection_area)) * 100
return round(overlap_percentage, 2), max_area, min_area, total_area
def identify_overlapping_case(
box_pair: Union[List[Union[List, Tuple]], Tuple[Union[List, Tuple]]],
label_pair: Union[List[str], Tuple[str]],
text_pair: Union[List[str], Tuple[str]],
ix_pair: Union[List[str], Tuple[str]],
sm_overlap_threshold: float = 10.0,
):
"""Classifies the overlapping case for an element_pair input.
There are 5 categories of overlapping:
'Small partial overlap', 'Partial overlap with empty content',
'Partial overlap with duplicate text (sharing 100% of the text)',
'Partial overlap without sharing text', and
'Partial overlap sharing {calculate_largest_ngram_percentage(...)}% of the text'
"""
overlapping_elements, overlapping_case, overlap_percentage, largest_ngram_percentage = (
None,
None,
None,
None,
)
box1, box2 = box_pair
type1, type2 = label_pair
text1, text2 = text_pair
ix_element1, ix_element2 = ix_pair
(
overlap_percentage,
max_area,
min_area,
total_area,
) = calculate_overlap_percentage(
box1,
box2,
intersection_ratio_method="partial",
)
if overlap_percentage < sm_overlap_threshold:
overlapping_elements = [
f"{type1}(ix={ix_element1})",
f"{type2}(ix={ix_element2})",
]
overlapping_case = "Small partial overlap"
else:
if not text1:
overlapping_elements = [
f"{type1}(ix={ix_element1})",
f"{type2}(ix={ix_element2})",
]
overlapping_case = f"partial overlap with empty content in {type1}"
elif not text2:
overlapping_elements = [
f"{type2}(ix={ix_element2})",
f"{type1}(ix={ix_element1})",
]
overlapping_case = f"partial overlap with empty content in {type2}"
elif text1 in text2 or text2 in text1:
overlapping_elements = [
f"{type1}(ix={ix_element1})",
f"{type2}(ix={ix_element2})",
]
overlapping_case = "partial overlap with duplicate text"
else:
(
largest_ngram_percentage,
largest_shared_ngrams_max,
largest_n,
) = calculate_largest_ngram_percentage(text1, text2)
largest_ngram_percentage = round(largest_ngram_percentage, 2)
if not largest_ngram_percentage:
overlapping_elements = [
f"{type1}(ix={ix_element1})",
f"{type2}(ix={ix_element2})",
]
overlapping_case = "partial overlap without sharing text"
else:
overlapping_elements = [
f"{type1}(ix={ix_element1})",
f"{type2}(ix={ix_element2})",
]
ref_type = type1 if len(text1.split()) < len(text2.split()) else type2
ref_type = "of the text from" + ref_type + f"({largest_n}-gram)"
overlapping_case = f"partial overlap sharing {largest_ngram_percentage}% {ref_type}"
return (
overlapping_elements,
overlapping_case,
overlap_percentage,
largest_ngram_percentage,
max_area,
min_area,
total_area,
)
# x1, y1 = box1[0]
def identify_overlapping_or_nesting_case(
box_pair: Union[List[Union[List, Tuple]], Tuple[Union[List, Tuple]]],
label_pair: Union[List[str], Tuple[str]],
text_pair: Union[List[str], Tuple[str]],
nested_error_tolerance_px: int = 5,
sm_overlap_threshold: float = 10.0,
):
"""Identify if there are nested or overlapping elements. If overlapping is present,
it identifies the case calling the method identify_overlapping_case"""
box1, box2 = box_pair
type1, type2 = label_pair
ix_element1 = "".join([ch for ch in type1 if ch.isnumeric()])
ix_element2 = "".join([ch for ch in type2 if ch.isnumeric()])
type1 = type1[3:].strip()
type2 = type2[3:].strip()
x_bottom_left_1, y_bottom_left_1 = box1[0]
x_top_right_1, y_top_right_1 = box1[2]
x_bottom_left_2, y_bottom_left_2 = box2[0]
x_top_right_2, y_top_right_2 = box2[2]
box1_corners = [x_bottom_left_1, y_bottom_left_1, x_top_right_1, y_top_right_1]
box2_corners = [x_bottom_left_2, y_bottom_left_2, x_top_right_2, y_top_right_2]
horizontal_overlap = x_bottom_left_1 < x_top_right_2 and x_top_right_1 > x_bottom_left_2
vertical_overlap = y_bottom_left_1 < y_top_right_2 and y_top_right_1 > y_bottom_left_2
(
overlapping_elements,
overlapping_case,
overlap_percentage,
overlap_percentage_total,
largest_ngram_percentage,
) = (
None,
None,
None,
None,
None,
)
max_area, min_area, total_area = None, None, None
if horizontal_overlap and vertical_overlap:
overlap_percentage_total, _, _, _ = calculate_overlap_percentage(
box1,
box2,
intersection_ratio_method="total",
)
overlap_percentage, max_area, min_area, total_area = calculate_overlap_percentage(
box1,
box2,
intersection_ratio_method="parent",
)
if is_parent_box(box1_corners, box2_corners, add=nested_error_tolerance_px):
overlapping_elements = [
f"{type1}(ix={ix_element1})",
f"{type2}(ix={ix_element2})",
]
overlapping_case = f"nested {type2} in {type1}"
overlap_percentage = 100
elif is_parent_box(box2_corners, box1_corners, add=nested_error_tolerance_px):
overlapping_elements = [
f"{type2}(ix={ix_element2})",
f"{type1}(ix={ix_element1})",
]
overlapping_case = f"nested {type1} in {type2}"
overlap_percentage = 100
else:
(
overlapping_elements,
overlapping_case,
overlap_percentage,
largest_ngram_percentage,
max_area,
min_area,
total_area,
) = identify_overlapping_case(
box_pair,
label_pair,
text_pair,
(ix_element1, ix_element2),
sm_overlap_threshold=sm_overlap_threshold,
)
return (
overlapping_elements,
overlapping_case,
overlap_percentage,
overlap_percentage_total,
largest_ngram_percentage,
max_area,
min_area,
total_area,
)
def catch_overlapping_and_nested_bboxes(
elements: List[Text],
nested_error_tolerance_px: int = 5,
sm_overlap_threshold: float = 10.0,
) -> (bool, List[Dict]):
"""Catch overlapping and nested bounding boxes cases across a list of elements."""
num_pages = elements[-1].metadata.page_number
bounding_boxes = [[] for _ in range(num_pages)]
text_labels = [[] for _ in range(num_pages)]
text_content = [[] for _ in range(num_pages)]
for ix, element in enumerate(elements):
n_page_to_ix = element.metadata.page_number - 1
bounding_boxes[n_page_to_ix].append(element.metadata.coordinates.to_dict()["points"])
text_labels[n_page_to_ix].append(f"{ix}. {element.category}")
text_content[n_page_to_ix].append(element.text)
document_with_overlapping_flag = False
overlapping_cases = []
for page_number, (page_bboxes, page_labels, page_text) in enumerate(
zip(bounding_boxes, text_labels, text_content),
start=1,
):
page_bboxes_combinations = list(combinations(page_bboxes, 2))
page_labels_combinations = list(combinations(page_labels, 2))
text_content_combinations = list(combinations(page_text, 2))
for box_pair, label_pair, text_pair in zip(
page_bboxes_combinations,
page_labels_combinations,
text_content_combinations,
):
(
overlapping_elements,
overlapping_case,
overlap_percentage,
overlap_percentage_total,
largest_ngram_percentage,
max_area,
min_area,
total_area,
) = identify_overlapping_or_nesting_case(
box_pair,
label_pair,
text_pair,
nested_error_tolerance_px,
sm_overlap_threshold,
)
if overlapping_case:
overlapping_cases.append(
{
"overlapping_elements": overlapping_elements,
"overlapping_case": overlapping_case,
"overlap_percentage": f"{overlap_percentage}%",
"metadata": {
"largest_ngram_percentage": largest_ngram_percentage,
"overlap_percentage_total": f"{overlap_percentage_total}%",
"max_area": f"{round(max_area, 2)}pxˆ2",
"min_area": f"{round(min_area, 2)}pxˆ2",
"total_area": f"{round(total_area, 2)}pxˆ2",
},
},
)
document_with_overlapping_flag = True
return document_with_overlapping_flag, overlapping_cases