2024-10-18 23:49:09 -07:00
|
|
|
|
EXPECTED_TABLE = (
|
|
|
|
|
|
"<table>"
|
|
|
|
|
|
"<tr><td>Stanley Cups</td><td/><td/></tr>"
|
|
|
|
|
|
"<tr><td>Team</td><td>Location</td><td>Stanley Cups</td></tr>"
|
|
|
|
|
|
"<tr><td>Blues</td><td>STL</td><td>1</td></tr>"
|
|
|
|
|
|
"<tr><td>Flyers</td><td>PHI</td><td>2</td></tr>"
|
|
|
|
|
|
"<tr><td>Maple Leafs</td><td>TOR</td><td>13</td></tr>"
|
|
|
|
|
|
"</table>"
|
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
EXPECTED_TABLE_SEMICOLON_DELIMITER = (
|
|
|
|
|
|
"<table>"
|
|
|
|
|
|
"<tr><td>Year</td><td>Month</td><td>Revenue</td><td>Costs</td><td/></tr>"
|
|
|
|
|
|
"<tr><td>2022</td><td>1</td><td>123</td><td>-123</td><td/></tr>"
|
|
|
|
|
|
"<tr><td>2023</td><td>2</td><td>143,1</td><td>-814,38</td><td/></tr>"
|
|
|
|
|
|
"<tr><td>2024</td><td>3</td><td>215,32</td><td>-11,08</td><td/></tr>"
|
|
|
|
|
|
"</table>"
|
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
EXPECTED_TABLE_WITH_EMOJI = (
|
|
|
|
|
|
"<table>"
|
|
|
|
|
|
"<tr><td>Stanley Cups</td><td/><td/></tr>"
|
|
|
|
|
|
"<tr><td>Team</td><td>Location</td><td>Stanley Cups</td></tr>"
|
|
|
|
|
|
"<tr><td>Blues</td><td>STL</td><td>1</td></tr>"
|
|
|
|
|
|
"<tr><td>Flyers</td><td>PHI</td><td>2</td></tr>"
|
|
|
|
|
|
"<tr><td>Maple Leafs</td><td>TOR</td><td>13</td></tr>"
|
|
|
|
|
|
"<tr><td>👨\\U+1F3FB🔧</td><td>TOR</td><td>15</td></tr>"
|
|
|
|
|
|
"</table>"
|
|
|
|
|
|
)
|
fix: stop csv and tsv dropping the first line of the file (#1530)
The current code assumes the first line of csv and tsv files are a
header line. Most csv and tsv files don't have a header line, and even
for those that do, dropping this line may not be the desired behavior.
Here is a snippet of code that demonstrates the current behavior and the
proposed fix
```
import pandas as pd
from lxml.html.soupparser import fromstring as soupparser_fromstring
c1 = """
Stanley Cups,,
Team,Location,Stanley Cups
Blues,STL,1
Flyers,PHI,2
Maple Leafs,TOR,13
"""
f = "./test.csv"
with open(f, 'w') as ff:
ff.write(c1)
print("Suggested Improvement Keep First Line")
table = pd.read_csv(f, header=None)
html_text = table.to_html(index=False, header=False, na_rep="")
text = soupparser_fromstring(html_text).text_content()
print(text)
print("\n\nOriginal Looses First Line")
table = pd.read_csv(f)
html_text = table.to_html(index=False, header=False, na_rep="")
text = soupparser_fromstring(html_text).text_content()
print(text)
```
---------
Co-authored-by: cragwolfe <crag@unstructured.io>
Co-authored-by: Yao You <theyaoyou@gmail.com>
Co-authored-by: Yao You <yao@unstructured.io>
2023-10-17 00:59:35 +02:00
|
|
|
|
|
2024-10-17 15:05:11 -07:00
|
|
|
|
EXPECTED_TABLE_XLSX = (
|
|
|
|
|
|
"<table>"
|
|
|
|
|
|
"<tr><td>Team</td><td>Location</td><td>Stanley Cups</td></tr>"
|
|
|
|
|
|
"<tr><td>Blues</td><td>STL</td><td>1</td></tr>"
|
|
|
|
|
|
"<tr><td>Flyers</td><td>PHI</td><td>2</td></tr>"
|
|
|
|
|
|
"<tr><td>Maple Leafs</td><td>TOR</td><td>13</td></tr>"
|
|
|
|
|
|
"</table>"
|
|
|
|
|
|
)
|
2023-06-15 13:50:53 -05:00
|
|
|
|
|
feat: xlsx subtable extraction (#1585)
**Executive Summary**
Unstructured is now able to capture subtables, along with other text
element types within the `.xlsx` sheet.
**Technical Details**
- The function now reads the excel *without* header as default
- Leverages the connected components search to find subtables within the
sheet. This search is based on dfs search
- It also handle the overlapping table or text cases
- Row with only single cell of data is considered not a table, and
therefore passed on the determine the element type as text
- In connected elements, it is possible to have table title, header, or
footer. We run the count for the first non-single empty rows from top
and bottom to determine those text
**Result**
This table now reads as:
<img width="747" alt="image"
src="https://github.com/Unstructured-IO/unstructured/assets/2177850/6b8e6d01-4ca5-43f4-ae88-6104b0174ed2">
```
[
{
"type": "Title",
"element_id": "3315afd97f7f2ebcd450e7c939878429",
"metadata": {
"filename": "vodafone.xlsx",
"file_directory": "example-docs",
"last_modified": "2023-10-03T17:51:34",
"filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"parent_id": "3315afd97f7f2ebcd450e7c939878429",
"languages": [
"spa",
"ita"
],
"page_number": 1,
"page_name": "Index",
"text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td>Topic</td>\n <td>Period</td>\n <td></td>\n <td></td>\n <td>Page</td>\n </tr>\n <tr>\n <td>Quarterly revenue</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>1</td>\n </tr>\n <tr>\n <td>Group financial performance</td>\n <td>FY 22</td>\n <td>FY 23</td>\n <td></td>\n <td>2</td>\n </tr>\n <tr>\n <td>Segmental results</td>\n <td>FY 22</td>\n <td>FY 23</td>\n <td></td>\n <td>3</td>\n </tr>\n <tr>\n <td>Segmental analysis</td>\n <td>FY 22</td>\n <td>FY 23</td>\n <td></td>\n <td>4</td>\n </tr>\n <tr>\n <td>Cash flow</td>\n <td>FY 22</td>\n <td>FY 23</td>\n <td></td>\n <td>5</td>\n </tr>\n </tbody>\n</table>"
},
"text": "Financial performance"
},
{
"type": "Table",
"element_id": "17f5d512705be6f8812e5dbb801ba727",
"metadata": {
"filename": "vodafone.xlsx",
"file_directory": "example-docs",
"last_modified": "2023-10-03T17:51:34",
"filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"parent_id": "3315afd97f7f2ebcd450e7c939878429",
"languages": [
"spa",
"ita"
],
"page_number": 1,
"page_name": "Index",
"text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td>Topic</td>\n <td>Period</td>\n <td></td>\n <td></td>\n <td>Page</td>\n </tr>\n <tr>\n <td>Quarterly revenue</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>1</td>\n </tr>\n <tr>\n <td>Group financial performance</td>\n <td>FY 22</td>\n <td>FY 23</td>\n <td></td>\n <td>2</td>\n </tr>\n <tr>\n <td>Segmental results</td>\n <td>FY 22</td>\n <td>FY 23</td>\n <td></td>\n <td>3</td>\n </tr>\n <tr>\n <td>Segmental analysis</td>\n <td>FY 22</td>\n <td>FY 23</td>\n <td></td>\n <td>4</td>\n </tr>\n <tr>\n <td>Cash flow</td>\n <td>FY 22</td>\n <td>FY 23</td>\n <td></td>\n <td>5</td>\n </tr>\n </tbody>\n</table>"
},
"text": "\n\n\nTopic\nPeriod\n\n\nPage\n\n\nQuarterly revenue\nNine quarters to 30 June 2023\n\n\n1\n\n\nGroup financial performance\nFY 22\nFY 23\n\n2\n\n\nSegmental results\nFY 22\nFY 23\n\n3\n\n\nSegmental analysis\nFY 22\nFY 23\n\n4\n\n\nCash flow\nFY 22\nFY 23\n\n5\n\n\n"
},
{
"type": "Title",
"element_id": "8a9db7161a02b427f8fda883656036e1",
"metadata": {
"filename": "vodafone.xlsx",
"file_directory": "example-docs",
"last_modified": "2023-10-03T17:51:34",
"filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"parent_id": "8a9db7161a02b427f8fda883656036e1",
"languages": [
"spa",
"ita"
],
"page_number": 1,
"page_name": "Index",
"text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td>Topic</td>\n <td>Period</td>\n <td></td>\n <td></td>\n <td>Page</td>\n </tr>\n <tr>\n <td>Mobile customers</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>6</td>\n </tr>\n <tr>\n <td>Fixed broadband customers</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>7</td>\n </tr>\n <tr>\n <td>Marketable homes passed</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>8</td>\n </tr>\n <tr>\n <td>TV customers</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>9</td>\n </tr>\n <tr>\n <td>Converged customers</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>10</td>\n </tr>\n <tr>\n <td>Mobile churn</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>11</td>\n </tr>\n <tr>\n <td>Mobile data usage</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>12</td>\n </tr>\n <tr>\n <td>Mobile ARPU</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>13</td>\n </tr>\n </tbody>\n</table>"
},
"text": "Operational metrics"
},
{
"type": "Table",
"element_id": "d5d16f7bf9c7950cd45fae06e12e5847",
"metadata": {
"filename": "vodafone.xlsx",
"file_directory": "example-docs",
"last_modified": "2023-10-03T17:51:34",
"filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"parent_id": "8a9db7161a02b427f8fda883656036e1",
"languages": [
"spa",
"ita"
],
"page_number": 1,
"page_name": "Index",
"text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td>Topic</td>\n <td>Period</td>\n <td></td>\n <td></td>\n <td>Page</td>\n </tr>\n <tr>\n <td>Mobile customers</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>6</td>\n </tr>\n <tr>\n <td>Fixed broadband customers</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>7</td>\n </tr>\n <tr>\n <td>Marketable homes passed</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>8</td>\n </tr>\n <tr>\n <td>TV customers</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>9</td>\n </tr>\n <tr>\n <td>Converged customers</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>10</td>\n </tr>\n <tr>\n <td>Mobile churn</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>11</td>\n </tr>\n <tr>\n <td>Mobile data usage</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>12</td>\n </tr>\n <tr>\n <td>Mobile ARPU</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>13</td>\n </tr>\n </tbody>\n</table>"
},
"text": "\n\n\nTopic\nPeriod\n\n\nPage\n\n\nMobile customers\nNine quarters to 30 June 2023\n\n\n6\n\n\nFixed broadband customers\nNine quarters to 30 June 2023\n\n\n7\n\n\nMarketable homes passed\nNine quarters to 30 June 2023\n\n\n8\n\n\nTV customers\nNine quarters to 30 June 2023\n\n\n9\n\n\nConverged customers\nNine quarters to 30 June 2023\n\n\n10\n\n\nMobile churn\nNine quarters to 30 June 2023\n\n\n11\n\n\nMobile data usage\nNine quarters to 30 June 2023\n\n\n12\n\n\nMobile ARPU\nNine quarters to 30 June 2023\n\n\n13\n\n\n"
},
{
"type": "Title",
"element_id": "f97e9da0e3b879f0a9df979ae260a5f7",
"metadata": {
"filename": "vodafone.xlsx",
"file_directory": "example-docs",
"last_modified": "2023-10-03T17:51:34",
"filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"parent_id": "f97e9da0e3b879f0a9df979ae260a5f7",
"languages": [
"spa",
"ita"
],
"page_number": 1,
"page_name": "Index",
"text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td>Topic</td>\n <td>Period</td>\n <td></td>\n <td></td>\n <td>Page</td>\n </tr>\n <tr>\n <td>Average foreign exchange rates</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>14</td>\n </tr>\n <tr>\n <td>Guidance rates</td>\n <td>FY 23/24</td>\n <td></td>\n <td></td>\n <td>14</td>\n </tr>\n </tbody>\n</table>"
},
"text": "Other"
},
{
"type": "Table",
"element_id": "080e1a745a2a3f2df22b6a08d33d59bb",
"metadata": {
"filename": "vodafone.xlsx",
"file_directory": "example-docs",
"last_modified": "2023-10-03T17:51:34",
"filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"parent_id": "f97e9da0e3b879f0a9df979ae260a5f7",
"languages": [
"spa",
"ita"
],
"page_number": 1,
"page_name": "Index",
"text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td>Topic</td>\n <td>Period</td>\n <td></td>\n <td></td>\n <td>Page</td>\n </tr>\n <tr>\n <td>Average foreign exchange rates</td>\n <td>Nine quarters to 30 June 2023</td>\n <td></td>\n <td></td>\n <td>14</td>\n </tr>\n <tr>\n <td>Guidance rates</td>\n <td>FY 23/24</td>\n <td></td>\n <td></td>\n <td>14</td>\n </tr>\n </tbody>\n</table>"
},
"text": "\n\n\nTopic\nPeriod\n\n\nPage\n\n\nAverage foreign exchange rates\nNine quarters to 30 June 2023\n\n\n14\n\n\nGuidance rates\nFY 23/24\n\n\n14\n\n\n"
}
]
```
2023-10-04 13:30:23 -04:00
|
|
|
|
EXPECTED_TITLE = "Stanley Cups"
|
2023-06-15 13:50:53 -05:00
|
|
|
|
|
fix: stop csv and tsv dropping the first line of the file (#1530)
The current code assumes the first line of csv and tsv files are a
header line. Most csv and tsv files don't have a header line, and even
for those that do, dropping this line may not be the desired behavior.
Here is a snippet of code that demonstrates the current behavior and the
proposed fix
```
import pandas as pd
from lxml.html.soupparser import fromstring as soupparser_fromstring
c1 = """
Stanley Cups,,
Team,Location,Stanley Cups
Blues,STL,1
Flyers,PHI,2
Maple Leafs,TOR,13
"""
f = "./test.csv"
with open(f, 'w') as ff:
ff.write(c1)
print("Suggested Improvement Keep First Line")
table = pd.read_csv(f, header=None)
html_text = table.to_html(index=False, header=False, na_rep="")
text = soupparser_fromstring(html_text).text_content()
print(text)
print("\n\nOriginal Looses First Line")
table = pd.read_csv(f)
html_text = table.to_html(index=False, header=False, na_rep="")
text = soupparser_fromstring(html_text).text_content()
print(text)
```
---------
Co-authored-by: cragwolfe <crag@unstructured.io>
Co-authored-by: Yao You <theyaoyou@gmail.com>
Co-authored-by: Yao You <yao@unstructured.io>
2023-10-17 00:59:35 +02:00
|
|
|
|
EXPECTED_TEXT = (
|
|
|
|
|
|
"Stanley Cups Team Location Stanley Cups Blues STL 1 Flyers PHI 2 Maple Leafs TOR 13"
|
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
EXPECTED_TEXT_XLSX = "Team Location Stanley Cups Blues STL 1 Flyers PHI 2 Maple Leafs TOR 13"
|
2023-08-10 16:28:57 -07:00
|
|
|
|
|
|
|
|
|
|
EXPECTED_TEXT_WITH_EMOJI = (
|
fix: stop csv and tsv dropping the first line of the file (#1530)
The current code assumes the first line of csv and tsv files are a
header line. Most csv and tsv files don't have a header line, and even
for those that do, dropping this line may not be the desired behavior.
Here is a snippet of code that demonstrates the current behavior and the
proposed fix
```
import pandas as pd
from lxml.html.soupparser import fromstring as soupparser_fromstring
c1 = """
Stanley Cups,,
Team,Location,Stanley Cups
Blues,STL,1
Flyers,PHI,2
Maple Leafs,TOR,13
"""
f = "./test.csv"
with open(f, 'w') as ff:
ff.write(c1)
print("Suggested Improvement Keep First Line")
table = pd.read_csv(f, header=None)
html_text = table.to_html(index=False, header=False, na_rep="")
text = soupparser_fromstring(html_text).text_content()
print(text)
print("\n\nOriginal Looses First Line")
table = pd.read_csv(f)
html_text = table.to_html(index=False, header=False, na_rep="")
text = soupparser_fromstring(html_text).text_content()
print(text)
```
---------
Co-authored-by: cragwolfe <crag@unstructured.io>
Co-authored-by: Yao You <theyaoyou@gmail.com>
Co-authored-by: Yao You <yao@unstructured.io>
2023-10-17 00:59:35 +02:00
|
|
|
|
"Stanley Cups "
|
2023-08-10 16:28:57 -07:00
|
|
|
|
"Team Location Stanley Cups Blues STL 1 Flyers PHI 2 Maple Leafs TOR 13 👨\\U+1F3FB🔧 TOR 15"
|
|
|
|
|
|
)
|
|
|
|
|
|
|
2023-12-13 18:57:46 -05:00
|
|
|
|
EXPECTED_TEXT_SEMICOLON_DELIMITER = (
|
|
|
|
|
|
"Year Month Revenue Costs 2022 1 123 -123 2023 2 143,1 -814,38 2024 3 215,32 -11,08"
|
|
|
|
|
|
)
|
|
|
|
|
|
|
2024-07-11 12:57:28 -07:00
|
|
|
|
EXPECTED_XLS_TABLE = (
|
2024-10-17 15:05:11 -07:00
|
|
|
|
"<table><tr>"
|
|
|
|
|
|
"<td>MC</td>"
|
|
|
|
|
|
"<td>What is 2+2?</td>"
|
|
|
|
|
|
"<td>4</td>"
|
|
|
|
|
|
"<td>correct</td>"
|
|
|
|
|
|
"<td>3</td>"
|
|
|
|
|
|
"<td>incorrect</td>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"</tr><tr>" # -----
|
|
|
|
|
|
"<td>MA</td>"
|
|
|
|
|
|
"<td>What C datatypes are 8 bits? (assume i386)</td>"
|
|
|
|
|
|
"<td>int</td>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td>float</td>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td>double</td>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td>char</td>"
|
|
|
|
|
|
"</tr><tr>" # -----
|
|
|
|
|
|
"<td>TF</td>"
|
|
|
|
|
|
"<td>Bagpipes are awesome.</td>"
|
|
|
|
|
|
"<td>true</td>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"</tr><tr>" # -----
|
|
|
|
|
|
"<td>ESS</td>"
|
|
|
|
|
|
"<td>How have the original Henry Hornbostel buildings influenced campus architecture and"
|
|
|
|
|
|
" design in the last 30 years?</td>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"</tr><tr>" # -----
|
|
|
|
|
|
"<td>ORD</td>"
|
|
|
|
|
|
"<td>Rank the following in their order of operation.</td>"
|
|
|
|
|
|
"<td>Parentheses</td>"
|
|
|
|
|
|
"<td>Exponents</td>"
|
|
|
|
|
|
"<td>Division</td>"
|
|
|
|
|
|
"<td>Addition</td>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"</tr><tr>" # -----
|
|
|
|
|
|
"<td>FIB</td>"
|
|
|
|
|
|
"<td>The student activities fee is</td>"
|
|
|
|
|
|
"<td>95</td>"
|
|
|
|
|
|
"<td>dollars for students enrolled in</td>"
|
|
|
|
|
|
"<td>19</td>"
|
|
|
|
|
|
"<td>units or more,</td>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"<td/>"
|
|
|
|
|
|
"</tr><tr>" # -----
|
|
|
|
|
|
"<td>MAT</td>"
|
|
|
|
|
|
"<td>Match the lower-case greek letter with its capital form.</td>"
|
|
|
|
|
|
"<td>λ</td>"
|
|
|
|
|
|
"<td>Λ</td>"
|
|
|
|
|
|
"<td>α</td>"
|
|
|
|
|
|
"<td>γ</td>"
|
|
|
|
|
|
"<td>Γ</td>"
|
|
|
|
|
|
"<td>φ</td>"
|
|
|
|
|
|
"<td>Φ</td>"
|
|
|
|
|
|
"</tr></table>"
|
2024-07-11 12:57:28 -07:00
|
|
|
|
)
|