| Operation Name | | | | Description | | block.pad(top, bottom, | right, | left) | | Enlarge the current block according to the input |
| block.scale(fx, fy) | | | Scale the current block given the ratio in x and y direction |
| block.shift(dx, dy) | | | Move the current block with the shift distances in x and y direction |
| blocki.is_in(block2) | | | | Whether block] is inside of block2 |
| blocki.intersect (block2) | | | Return the intersection region of blockl and block2. Coordinate type to be determined based on the inputs. |
| block1i.union(block2) | | | Return the union region of blockl and block2. Coordinate type to be determined based on the inputs. |
| blocki.relative_to(block2) | | | Convert the absolute coordinates of block] to relative coordinates to block2 |
| blocki.condition_on(block2) | | | Calculate the absolute coordinates of blockl given the canvas block2’s absolute coordinates |
"
},
- "text": "Operation Name Description block.pad(top, bottom, right, left) Enlarge the current block according to the input block.scale(fx, fy) Scale the current block given the ratio ion in x and y di block.shift(dx, dy) Move the current block with the shift distances in x and y direction block1.is_in(block2) Whether block] is inside of block2 ; Return the intersection region of block and block2. block1. intersect (block2) . . . Coordinate type to be determined based on the inputs. ; Return the union region of block1 and block2. block1.union(block2) . . . Coordinate type to be determined based on the inputs. Convert the absolute coordinates of block to block1.relative_to(block2) ' ' relative coordinates to block2 . Calculate the absolute coordinates of block1 given block1.condition_on(block2) . the canvas block2’s absolute coordinates block. crop_image (image) Obtain the image segments in the block region"
+ "text": "Operation Name Description block.pad(top, bottom, right, left) Enlarge the current block according to the input block.scale(fx, fy) block.shift(dx, dy) Scale the current block given the ratioin x and y direction Move the current block with the shiftdistances in x and y direction block1.is in(block2) Whether block1 is inside of block2 block1.intersect(block2) block1.union(block2) block1.relative to(block2) block1.condition on(block2) Return the intersection region of block1 and block2.Coordinate type to be determined based on the inputs. Return the union region of block1 and block2.Coordinate type to be determined based on the inputs. Convert the absolute coordinates of block1 torelative coordinates to block2 Calculate the absolute coordinates of block1 giventhe canvas block2’s absolute coordinates Obtain the image segments in the block region"
},
{
"type": "UncategorizedText",
@@ -1333,13 +1333,13 @@
},
{
"type": "FigureCaption",
- "element_id": "975d6cb141cb0a0313375630ae063fa8",
+ "element_id": "d10d70e48ed0066bd15dd133d09f61fd",
"metadata": {
"data_source": {},
"filetype": "application/pdf",
"page_number": 9
},
- "text": "x09 Burpunog uayor Aeydsiq 1 vondo 10g Guypunog usyoy apir:z uondo Mode I: Showing Layout on the Original Image Mode Il: Drawing OCR'd Text at the Correspoding Position"
+ "text": "a ESMode I: Showing Layout on the Original ImageayeMode Il: Drawing OCR'd Text at the Correspoding Position10g Bpunog vayoy feyds1q :1 vondo‘xog Burpunog vay apiH z word"
},
{
"type": "NarrativeText",
@@ -1423,13 +1423,13 @@
},
{
"type": "FigureCaption",
- "element_id": "2680b3c7a55754a3ba2738cb3d9d5e8b",
+ "element_id": "0ff9ad06a304818ae83b93c6f2b16309",
"metadata": {
"data_source": {},
"filetype": "application/pdf",
"page_number": 10
},
- "text": "et Intra-column reading order Token Categories tie (Adress 2) tee (NE sumber Variable HEE company type Column Categories (J tite we) adaress —_ (7) section Header by ‘e * Column reading order a a (a) Illustration of the original Japanese Maximum Allowed Height BRE B>e EER eR (b) Illustration of the recreated document with dense text structure for better OCR performance"
+ "text": "Column reading orderMaximum Allowed HeightZ. Shen et al.Intra-column reading ordert el 1 ili |.(a) Illustration of the original Japanese document with detected layout elements highlighted in colored boxesCe tans &iia! aaae oeRRbaeienases(b) Illustration of the recreated document with dense text structure for better OCR performance‘Token CategoriesMoteAddresstetNumberVaribiecompany typeColumn Categories(J tite| Aatress(tee[7] section adr"
},
{
"type": "NarrativeText",
@@ -1543,13 +1543,13 @@
},
{
"type": "FigureCaption",
- "element_id": "b33b2bc3b9c416673c7f74c6a00c49d8",
+ "element_id": "55f2474c66877608ca9b463a7076573e",
"metadata": {
"data_source": {},
"filetype": "application/pdf",
"page_number": 11
},
- "text": "(spe peepee, ‘Active Learning Layout Annotate Layout Dataset | + ‘Annotation Toolkit ¥ a Deep Leaming Layout Model Training & Inference, ¥ ; Handy Data Structures & Post-processing El Apis for Layout Det a LAR ror tye eats) 4 Text Recognition | <—— Default ane Customized ¥ ee Layout Structure Visualization & Export | <—— | visualization & Storage The Japanese Document Helpful LayoutParser Digitization Pipeline Modules"
+ "text": "(spe peepee,Active Learning Layout=Annotate Layout Dataset parte4zi Deep Learning LayoutLayout Detection Model Training & Inference,Post-processin Handy Data Structures &pl 9 APIs for Layout DataText Recognition Default and Customized: r OCR Models4Visualization & Export | <——Layout StructureVisualization & StorageThe Japanese Document Helpful LayoutParserDigitization Pipeline Modules"
},
{
"type": "UncategorizedText",
@@ -1723,13 +1723,13 @@
},
{
"type": "FigureCaption",
- "element_id": "7d42bb6af1404a95a6e8870d5c4d07bf",
+ "element_id": "f58d47bde7ebddd81c4a678c918a8f1b",
"metadata": {
"data_source": {},
"filetype": "application/pdf",
"page_number": 13
},
- "text": "(@) Partial table at the bottom (&) Full page table (6) Partial table at the top (d) Mis-detected tet line"
+ "text": "(2) Partial table atthe bottom (&) Full page table (6) Partial table at the top (d) Mis-detected tet line"
},
{
"type": "NarrativeText",
diff --git a/test_unstructured_ingest/expected-structured-output/s3/small-pdf-set/2023-Jan-economic-outlook.pdf.json b/test_unstructured_ingest/expected-structured-output/s3/small-pdf-set/2023-Jan-economic-outlook.pdf.json
index 89ce864c3..884200e49 100644
--- a/test_unstructured_ingest/expected-structured-output/s3/small-pdf-set/2023-Jan-economic-outlook.pdf.json
+++ b/test_unstructured_ingest/expected-structured-output/s3/small-pdf-set/2023-Jan-economic-outlook.pdf.json
@@ -721,13 +721,13 @@
},
{
"type": "Table",
- "element_id": "63bdc79def2500227001ac95d78727ab",
+ "element_id": "8dec233e9bc75c7256a28a899794709b",
"metadata": {
"data_source": {},
"filetype": "application/pdf",
"page_number": 7
},
- "text": "Difference from October 2022 Q4 over Q4 2/ Estimate___ Projections WEO Projections 1/ Estimate Projections 2021 2022 2023 2024 2023 2024 2022 2023 2024 World Output 6.2 34 29 34 0.2 0.1 1.9 3.2 3.0 Advanced Economies 5.4 27 1.2 14 04 0.2 1.3 14 1.6 United States 5.9 2.0 14 1.0 04 -0.2 07 1.0 13 Euro Area 5.3 3.5 07 16 0.2 -0.2 19 0.5 24 Germany 26 19 01 14 04 0.1 14 0.0 23 France 68 26 07 16 0.0 0.0 0.5 09 18 Italy 67 3.9 06 0.9 08 -04 21 0.1 1.0 Spain 5.5 5.2 14 24 -0.1 -0.2 21 13 28 Japan 21 14 18 0.9 0.2 -04 17 1.0 1.0 United Kingdom 76 41 -06 0.9 -0.9 03 04 -05 18 Canada 5.0 3.5 15 15 0.0 0.1 23 12 1.9 Other Advanced Economies 3/ 5.3 28 20 24 -03 02 14 2a 2.2 Emerging Market and Developing Economies 67 3.9 40 42 0.3 -0.1 25 5.0 4A Emerging and Developing Asia 74 43 5.3 5.2 04 0.0 3.4 6.2 49 China 84 3.0 5.2 45 08 0.0 29 5.9 41 India 4/ 87 68 61 68 0.0 0.0 43 70 7A Emerging and Developing Europe 69 07 15 26 0.9 01 -2.0 3.5 28 Russia 47 -2.2 0.3 21 26 06 441 1.0 2.0 Latin America and the Caribbean 7.0 3.9 18 2a 04 0.3 26 1.9 19 Brazil 5.0 34 12 15 0.2 -04 28 0.8 22 Mexico 47 34 47 16 05 -0.2 37 14 1.9 Middle East and Central Asia 45 5.3 3.2 37 -04 0.2 . . . Saudi Arabia 3.2 87 26 34 -11 0.5 46 27 35 Sub-Saharan Africa 47 38 38 41 04 0.0 = ao ao Nigeria 3.6 3.0 3.2 29 0.2 0.0 26 31 29 South Africa 49 26 12 13 01 0.0 3.0 0.5 18 Memorandum World Growth Based on Market Exchange Rates 6.0 3.41 24 25 03 -0.1 17 25 25 European Union 5.5 37 07 18 0.0 -0.3 18 1.2 2.0 ASEAN-5 5/ 3.8 5.2 43 47 0.2 -0.2 37 57 40 Middle East and North Africa 41 54 3.2 35 -04 0.2 a . . Emerging Market and Middle-Income Economies 70 38 40 44 04 0.0 25 5.0 44 Low-Income Developing Countries 441 49 49 56 0.0 01 World Trade Volume (goods and services) 6/ 10.4 5.4 24 3.4 -01 -0.3 Advanced Economies 94 66 23 27 0.0 -04 Emerging Market and Developing Economies 124 34 26 46 03 0.0 Commodity Prices Oil 7/ 65.8 39.8 -16.2 71 33 -0.9 11.2 -98 59 Nonfuel (average based on world commodity import weights) 26.4 70 -6.3 -0.4 -01 03 -2.0 14 -0.2 World Consumer Prices 8/ 47 88 6.6 43 04 0.2 9.2 5.0 3.5 Advanced Economies 9/ 34 73 46 26 0.2 02 78 31 23 Emerging Market and Developing Economies 8/ 5.9 99 84 5.5 0.0 02 10.4 66 45,"
+ "text": "Estimate2022 Projections 2023 2024 2021 WEO Projections 1/ 2023 2024 Estimate2022 Projections 2023 2024 Difference from October 2022 Q4 over Q4 2/ World Output Advanced Economies United States Euro Area Germany France Italy Spain Japan United Kingdom Canada Other Advanced Economies 3/ Emerging Market and Developing Economies Emerging and Developing Asia China India 4/ Emerging and Developing Europe Russia Latin America and the Caribbean Brazil Mexico Middle East and Central Asia Saudi Arabia Sub-Saharan Africa Nigeria South Africa Memorandum World Growth Based on Market Exchange Rates European Union ASEAN-5 5/ Middle East and North Africa Emerging Market and Middle-Income Economies Low-Income Developing Countries World Trade Volume (goods and services) 6/ Advanced Economies Emerging Market and Developing Economies Commodity Prices Oil 7/ Nonfuel (average based on world commodity import weights) World Consumer Prices 8/ Advanced Economies 9/ Emerging Market and Developing Economies 8/ 6.2 5.4 5.9 5.3 2.6 6.8 6.7 5.5 2.1 7.6 5.0 5.3 6.7 7.4 8.4 8.7 6.9 4.7 7.0 5.0 4.7 4.5 3.2 4.7 3.6 4.9 6.0 5.5 3.8 4.1 7.0 4.1 10.4 9.4 12.1 65.8 26.4 4.7 3.1 5.9 3.4 2.7 2.0 3.5 1.9 2.6 3.9 5.2 1.4 4.1 3.5 2.8 3.9 4.3 3.0 6.8 0.7 –2.2 3.9 3.1 3.1 5.3 8.7 3.8 3.0 2.6 3.1 3.7 5.2 5.4 3.8 4.9 5.4 6.6 3.4 39.8 7.0 8.8 7.3 9.9 2.9 1.2 1.4 0.7 0.1 0.7 0.6 1.1 1.8 –0.6 1.5 2.0 4.0 5.3 5.2 6.1 1.5 0.3 1.8 1.2 1.7 3.2 2.6 3.8 3.2 1.2 2.4 0.7 4.3 3.2 4.0 4.9 2.4 2.3 2.6 3.1 1.4 1.0 1.6 1.4 1.6 0.9 2.4 0.9 0.9 1.5 2.4 4.2 5.2 4.5 6.8 2.6 2.1 2.1 1.5 1.6 3.7 3.4 4.1 2.9 1.3 2.5 1.8 4.7 3.5 4.1 5.6 3.4 2.7 4.6 –16.2 –6.3 6.6 4.6 8.1 –7.1 –0.4 4.3 2.6 5.5 0.2 0.1 0.4 0.2 0.4 0.0 0.8 –0.1 0.2 –0.9 0.0 –0.3 0.3 0.4 0.8 0.0 0.9 2.6 0.1 0.2 0.5 –0.4 –1.1 0.1 0.2 0.1 0.3 0.0 –0.2 –0.4 0.4 0.0 –0.1 0.0 –0.3 –3.3 –0.1 0.1 0.2 0.0 –0.1 –0.2 –0.2 –0.2 –0.1 0.0 –0.4 –0.2 –0.4 0.3 –0.1 –0.2 –0.1 0.0 0.0 0.0 0.1 0.6 –0.3 –0.4 –0.2 0.2 0.5 0.0 0.0 0.0 –0.1 –0.3 –0.2 0.2 0.0 0.1 –0.3 –0.4 0.0 –0.9 0.3 0.2 0.2 0.2 1.9 1.3 0.7 1.9 1.4 0.5 2.1 2.1 1.7 0.4 2.3 1.4 2.5 3.4 2.9 4.3 –2.0 –4.1 2.6 2.8 3.7 . . . 4.6 . . . 2.6 3.0 1.7 1.8 3.7 . . . 2.5 . . . . . . . . . . . . 11.2 –2.0 9.2 7.8 10.4 3.2 1.1 1.0 0.5 0.0 0.9 0.1 1.3 1.0 –0.5 1.2 2.1 5.0 6.2 5.9 7.0 3.5 1.0 1.9 0.8 1.1 . . . 2.7 . . . 3.1 0.5 2.5 1.2 5.7 . . . 5.0 . . . . . . . . . . . . 3.0 1.6 1.3 2.1 2.3 1.8 1.0 2.8 1.0 1.8 1.9 2.2 4.1 4.9 4.1 7.1 2.8 2.0 1.9 2.2 1.9 . . . 3.5 . . . 2.9 1.8 2.5 2.0 4.0 . . . 4.1 . . . . . . . . . . . . –9.8 1.4 5.0 3.1 6.6 –5.9 –0.2 3.5 2.3 4.5 Q4 over Q4 2/ eee na RA Estimate_ le Projections Estimate_ le Projections Cetimatel Oo [re ee~T Se 2022 — 2022 — anna es battee eesaanShaeaan eeeanne anan aaa ——eeaaa World Output 6.2 0.2 19 eveee eve nmeAdvanced EconomiesTinta Chetan 54LO 27On 14an 04nA baal1.3nz 1.649 eee meeeUnited StatesCo Ae STN taeEuro Area‘Rareram) Vee5.3 vir0.2 ENGermanyCoen ww26no 2 « Ve04An eianyFranceleah, ov68 MAE07 vir0.0 ree0.5 vv0.9 ve67ce ev3.9— ve08na vwna vee24a4 vee0.14a NaySpain vee24no aNN+ “penJapanVithed Vinmdaee, we24— wwe04a voro!United KingdomPanada et76 Vw eeCanadaAbn Ndime nnd Camnaeeinn O) vee15nA vwnn vr23“aA INOSOther Advanced Economies 3/ wu5.3 ww28 Ld20 idl24 vw0.3 ow14 te24 Lindl22 MEE REEmerging Market and Developing Economiesnee a ree nee25 }et5.0 “eae a eee eeg and Developing Asia Me74 “~~04 ail3.4 EITC TSHTS GID MIC VOIOP HY PwictChinabese at vie45no Vt08An vt29aD ineIndia 4/ oT87 liad68 ve0.0 og43 Emerging and Developing EuropeDa. 15na 264 SET IMS MeN Pig SMRussiaLatin Amarina and the Carihhaan ve47mn al26na 44on 1.040 ENONLatin America and the CaribbeanDeasil iN oN04no nA OLEAN GM NENS aieBrazilMavien is ov28a7 eenMexicoRoeldia Cast end antral Asia ue0.5na eu3.7 oe19 bichinanbaiastelMiddle East and Central AsiaCad; Apahia 45ao vat-04aa ) veeAg 7 ac ee ON EN EENSaudi ArabiaCok Gahacen Af aia3.2az uw87— Ve26— ue3.4Aa Ne14na 46 27 35 ve eeSub-Saharan AfricaAlinaria ve47 on ofa4 0 ENNNigeriaOak AEinn ve3.0ne INIQe TeSouth Africa vw49 wu26 ve12 dl1.3 Ve041 vw0.0 ov3.0 vet0.5 og18 STMemorandumUae Pereath Beene an Marat Conhanns Dates BEES GEESIAISTWorld Growth Based on Market Exchange Ratesco -041 TN EE BO VN ene ee eeEuropean UnionACCA EE! vv5.5 vl37fo Lal0.0 aurea eatASEAN-5 5/Reebde Cred ered Rbesie Adslan ond38aa ve0.2nA dad0.2— ov4.0 rawMiddle East and North Africa ve44 del-04 vee vee NS EO EN EENEmerging Market and Middle-Income Economies 25 5.0 44 ee eee, Oe eee eeeLow-Income Developing Countries 44 awe nan eee eeeWorld Trade Volume (goods and services) 6/Aduannad Enanamineneo Teh10.4oO” Te2400 uM0.1ann ON ES NOES OE DESO NEAdvanced Economieser mee9.4“ana “rt23ne Mee0.0— iawn eensEmerging Market and Developing Economies oT12.1 vw3.4 ow26 of46 vw0.3 ~~ BESTT MIGINGL ait event eSCommodity Prices=) EIN!Oil 7/be® an! fesempnses became! nun taniied ananenelieeeS 65.8me A 39.8—- -16.2— 11.2an MirNonfuel (average based on world commodity import weights) V026.4 ve.0 I-0.4 Te-0.1 we Tew14 DMI! \\arvliagy Veo nN Ay aeWorld Consumer Prices 8/Aehiannedd Kamnemnian fii ae47nA Se MESES EIS OFAdvanced Economies 9/Emarninn Marbat and Navalaninn Erancs ee34 Lda0.2 02nd oe78ana ve2.3 rE RS OTEmerging Market and Developing Economies 8/ ) 10.4 fame i i al oe vt =: Lone aNate’ Ranl affective auchanna mtoe ara aeaamnad tn ramain oanatant atthe laugle nraunlinn dinna Oeinhar dA D00_Nevemherd] O00) Fenny nine noe ct Tho an _\"iT iartearhy"
},
{
"type": "UncategorizedText",
diff --git a/test_unstructured_ingest/expected-structured-output/s3/small-pdf-set/recalibrating-risk-report.pdf.json b/test_unstructured_ingest/expected-structured-output/s3/small-pdf-set/recalibrating-risk-report.pdf.json
index 0d3a1bc19..df30ef9bb 100644
--- a/test_unstructured_ingest/expected-structured-output/s3/small-pdf-set/recalibrating-risk-report.pdf.json
+++ b/test_unstructured_ingest/expected-structured-output/s3/small-pdf-set/recalibrating-risk-report.pdf.json
@@ -1099,6 +1099,16 @@
},
"text": "6"
},
+ {
+ "type": "FigureCaption",
+ "element_id": "f58b520072d30c4805940f5c99a306c3",
+ "metadata": {
+ "data_source": {},
+ "filetype": "application/pdf",
+ "page_number": 9
+ },
+ "text": "an ¥3 te,ay."
+ },
{
"type": "NarrativeText",
"element_id": "d754d8d468346f652657279272a11897",
diff --git a/unstructured/__version__.py b/unstructured/__version__.py
index 41643b78b..602eae017 100644
--- a/unstructured/__version__.py
+++ b/unstructured/__version__.py
@@ -1 +1 @@
-__version__ = "0.10.4-dev0" # pragma: no cover
+__version__ = "0.10.4" # pragma: no cover
diff --git a/unstructured/partition/pdf.py b/unstructured/partition/pdf.py
index 57196c50e..088184070 100644
--- a/unstructured/partition/pdf.py
+++ b/unstructured/partition/pdf.py
@@ -188,6 +188,7 @@ def partition_pdf_or_image(
infer_table_structure=infer_table_structure,
include_page_breaks=include_page_breaks,
ocr_languages=ocr_languages,
+ ocr_mode="individual_blocks",
metadata_last_modified=metadata_last_modified or last_modification_date,
**kwargs,
)
@@ -219,6 +220,7 @@ def _partition_pdf_or_image_local(
infer_table_structure: bool = False,
include_page_breaks: bool = False,
ocr_languages: str = "eng",
+ ocr_mode: str = "entire_page",
model_name: Optional[str] = None,
metadata_last_modified: Optional[str] = None,
**kwargs,
@@ -235,6 +237,7 @@ def _partition_pdf_or_image_local(
process_file_with_model_kwargs = {
"is_image": is_image,
"ocr_languages": ocr_languages,
+ "ocr_mode": ocr_mode,
"extract_tables": infer_table_structure,
"model_name": model_name,
}
@@ -249,6 +252,7 @@ def _partition_pdf_or_image_local(
file,
is_image=is_image,
ocr_languages=ocr_languages,
+ ocr_mode=ocr_mode,
extract_tables=infer_table_structure,
model_name=model_name,
)