2023-05-24 17:29:35 -05:00
|
|
|
####################################################################################################
|
|
|
|
|
# This file can house global constraints that aren't *direct* requirements of the package or any
|
|
|
|
|
# extras. Putting a dependency here will only affect dependency sets that contain them -- in other
|
|
|
|
|
# words, if something does not require a constraint, it will not be installed.
|
|
|
|
|
####################################################################################################
|
|
|
|
|
# NOTE(alan): Pinning to avoid conflicts with downstream ingest-s3
|
|
|
|
|
urllib3<1.27, >=1.25.4
|
2023-10-18 17:36:51 -07:00
|
|
|
boto3<1.28.18
|
|
|
|
|
botocore<1.31.18
|
2023-05-24 17:29:35 -05:00
|
|
|
# consistency with local-inference-pin
|
2023-07-12 10:41:02 -04:00
|
|
|
protobuf<4.24
|
2023-05-24 17:29:35 -05:00
|
|
|
# NOTE(robinson) - Required pins for security scans
|
|
|
|
|
jupyter-core>=4.11.2
|
|
|
|
|
wheel>=0.38.1
|
|
|
|
|
# NOTE(robinson) - The following pins are to address
|
|
|
|
|
# vulnerabilities in dependency scans
|
2023-08-01 11:31:13 -04:00
|
|
|
certifi>=2023.7.22
|
2023-07-10 14:52:25 -04:00
|
|
|
# From pycocotools in local-inference
|
|
|
|
|
pyparsing<3.1.0
|
2023-08-01 11:31:13 -04:00
|
|
|
# NOTE(robinson) - Numpy dropped Python 3.8 support in 1.25.0
|
|
|
|
|
numpy<1.25.0
|
|
|
|
|
scipy<1.11.0
|
|
|
|
|
IPython<8.13
|
2023-08-15 00:15:44 -05:00
|
|
|
# NOTE(alan) Pinned to avoid error that occurs with 2.4.3:
|
|
|
|
|
# AttributeError: 'ResourcePath' object has no attribute 'collection'
|
|
|
|
|
Office365-REST-Python-Client<2.4.3
|
2023-09-22 09:41:53 -07:00
|
|
|
# NOTE(trevor) `unstructured-inference` is set in extra-pdf-image.in to allow
|
|
|
|
|
# unstructured-inference to be upgraded when unstructured library is upgraded
|
|
|
|
|
# https://github.com/Unstructured-IO/unstructured/issues/1458
|
|
|
|
|
# unstructured-inference
|
2023-08-24 11:12:50 -04:00
|
|
|
# NOTE(klaijan) - Moved pin from test.in
|
|
|
|
|
# pinning to avoid error in argilla library
|
|
|
|
|
pydantic<2
|
2023-08-24 11:00:25 -07:00
|
|
|
# unable to build wheel for arm on 0.3.3+
|
|
|
|
|
safetensors<=0.3.2
|
2023-09-14 18:27:18 -05:00
|
|
|
# use the known compatible version of weaviate and unstructured.pytesseract
|
|
|
|
|
unstructured.pytesseract>=0.3.12
|
2023-09-13 18:19:20 -05:00
|
|
|
weaviate-client==3.23.2
|
2023-09-15 17:05:48 -07:00
|
|
|
# Note(yuming) - pining to avoid conflict with paddle install
|
|
|
|
|
matplotlib==3.7.2
|
2023-09-22 02:16:03 -07:00
|
|
|
# NOTE(crag) - pin to available pandas for python 3.8 (at least in CI)
|
2023-09-25 11:49:40 -07:00
|
|
|
fsspec==2023.9.1
|
2023-09-25 10:27:42 -04:00
|
|
|
pandas<2.0.4
|
2023-09-29 14:09:57 -05:00
|
|
|
# langchain limits anyio to below 4.0
|
|
|
|
|
anyio<4.0
|
2023-10-14 13:03:59 -07:00
|
|
|
# NOTE(crag): earlier versions fail in compilation step when pip installing the package
|
|
|
|
|
pycocotools>=2.0.7
|
|
|
|
|
# NOTE(crag): python3.8-python3.11 compat (if it ends up being required)
|
|
|
|
|
torch>2
|
2023-09-29 14:09:57 -05:00
|
|
|
# pinned in unstructured paddleocr
|
|
|
|
|
opencv-python==4.8.0.76
|
|
|
|
|
opencv-contrib-python==4.8.0.76
|
refactor: unstructured ingest as a pipeline (#1551)
### Description
As we add more and more steps to the pipeline (i.e. chunking, embedding,
table manipulation), it would help seperate the responsibility of each
of these into their own processes, running each in parallel using json
files to share data across. This will also help guarantee data is
serializable if this code was used in an actual pipeline. Following is a
flow diagram of the proposed changes. As part of this change:
* A parent pipeline class will be responsible for running each `node`,
which can optionally be run via multiprocessing if it supports it, or
not. Possible nodes at this moment:
* Doc factory: creates all the ingest docs via the source connector
* Source: reads/downloads all of the content to process to the local
filesystem to the location set by the `download_dir` parameter.
* Partition: runs partition on all of the downloaded content in json
format.
* Any number of reformat nodes that modify the partitioned content. This
can include chunking, embedding, etc.
* Write: push the final json into the destination via the destination
connector
* This pipeline relies on the information of the ingest docs to be
available via their serialization. An optimization was introduced with
the `IngestDocJsonMixin` which adds in all the `@property` fields to the
serialized json already being created via the `DataClassJsonMixin`
* For all intermediate steps (partitioning, reformatting), the content
is saved to a dedicated location on the local filesystem. Right now it's
set to `$HOME/.cache/unstructured/ingest/pipeline/STEP_NAME/`.
* Minor changes: made sense to move some of the config parameters
between the read and partition configs when I explicitly divided the
responsibility to download vs partition the content in the pipeline.
* The pipeline class only makes the doc factory, source and partition
nodes required, keeping with the logic that has been supported so far.
All reformatting nodes and write node are optional.
* Long term, there should also be some changes to the base configs
supported by the CLI to support pipeline specific configs, but for now
what exists was used to minimize changes in this PR.
* Final step to copy the final output to the location designated by the
`_output_filename` value of the ingest doc.
* Hashing occurs at each step by hashing the parameters of that step
(i.e. partition configs) along with the previous step via the filename
used. This allows each step to be the same _if_ all the parameters for
it have not changed and the content so far is the same.
* The only data that is shared and has writes to across processes is the
dictionary of ingest json data. This dict is created using the
`multiprocessing.manager.DictProxy` to make sure any interaction with it
is behind a lock.
### Minor refactors included:
* Utility methods added to extract configs from the click options
* Utility method to add common options to click commands.
* All writers moved to using the class approach which extracts a lot of
the common code so there's less copy-paste when new runners are added.
* Use `@property` for source metadata on base ingest doc to add logic to
call `update_source_metadata` if it's still `None` at the time it's
fetched.
### Additional bug fixes included
* Fsspec connectors were not serializable due to the `ingest_doc_cls`.
This was removed from the fields captured by the `@dataclass` decorator
and added in a `__post_init__` method.
* Various reddit connector params were missing. This doesn't have an
explicit ingest test at the moment so was never caught.
* Fsspec connector had the parent `update_source_metadata` misnamed as
`update_source_metadata_metadata` so it was never being called.
### Flow Diagram
2023-10-06 14:49:29 -04:00
|
|
|
onnxruntime==1.15.1
|
Refactor: support entire page OCR with `ocr_mode` and `ocr_languages` (#1579)
## Summary
Second part of OCR refactor to move it from inference repo to
unstructured repo, first part is done in
https://github.com/Unstructured-IO/unstructured-inference/pull/231. This
PR adds OCR process logics to entire page OCR, and support two OCR
modes, "entire_page" or "individual_blocks".
The updated workflow for `Hi_res` partition:
* pass the document as data/filename to inference repo to get
`inferred_layout` (DocumentLayout)
* pass the document as data/filename to OCR module, which first open the
document (create temp file/dir as needed), and split the document by
pages (convert PDF pages to image pages for PDF file)
* if ocr mode is `"entire_page"`
* OCR the entire image
* merge the OCR layout with inferred page layout
* if ocr mode is `"individual_blocks"`
* from inferred page layout, find element with no extracted text, crop
the entire image by the bboxes of the element
* replace empty text element with the text obtained from OCR the cropped
image
* return all merged PageLayouts and form a DocumentLayout subject for
later on process
This PR also bump `unstructured-inference==0.7.2` since the branch relay
on OCR refactor from unstructured-inference.
## Test
```
from unstructured.partition.auto import partition
entrie_page_ocr_mode_elements = partition(filename="example-docs/english-and-korean.png", ocr_mode="entire_page", ocr_languages="eng+kor", strategy="hi_res")
individual_blocks_ocr_mode_elements = partition(filename="example-docs/english-and-korean.png", ocr_mode="individual_blocks", ocr_languages="eng+kor", strategy="hi_res")
print([el.text for el in entrie_page_ocr_mode_elements])
print([el.text for el in individual_blocks_ocr_mode_elements])
```
latest output:
```
# entrie_page
['RULES AND INSTRUCTIONS 1. Template for day 1 (korean) , for day 2 (English) for day 3 both English and korean. 2. Use all your accounts. use different emails to send. Its better to have many email', 'accounts.', 'Note: Remember to write your own "OPENING MESSAGE" before you copy and paste the template. please always include [TREASURE HARUTO] for example:', '안녕하세요, 저 희 는 YGEAS 그룹 TREASUREWH HARUTOM|2] 팬 입니다. 팬 으 로서, HARUTO 씨 받 는 대 우 에 대해 의 구 심 과 불 공 평 함 을 LRU, 이 일 을 통해 저 희 의 의 혹 을 전 달 하여 귀 사 의 진지한 민 과 적극적인 답 변 을 받을 수 있 기 를 바랍니다.', '3. CC Harutonations@gmail.com so we can keep track of how many emails were', 'successfully sent', '4. Use the hashtag of Haruto on your tweet to show that vou have sent vour email]', '메 고']
# individual_blocks
['RULES AND INSTRUCTIONS 1. Template for day 1 (korean) , for day 2 (English) for day 3 both English and korean. 2. Use all your accounts. use different emails to send. Its better to have many email', 'Note: Remember to write your own "OPENING MESSAGE" before you copy and paste the template. please always include [TREASURE HARUTO] for example:', '안녕하세요, 저 희 는 YGEAS 그룹 TREASURES HARUTOM| 2] 팬 입니다. 팬 으로서, HARUTO 씨 받 는 대 우 에 대해 의 구 심 과 habe ERO, 이 머 일 을 적극 저 희 의 ASS 전 달 하여 귀 사 의 진지한 고 2 있 기 를 바랍니다.', '3. CC Harutonations@gmail.com so we can keep track of how many emails were ciiccecefisliy cant', 'VULLESSIULY Set 4. Use the hashtag of Haruto on your tweet to show that you have sent your email']
```
---------
Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: yuming-long <yuming-long@users.noreply.github.com>
Co-authored-by: christinestraub <christinemstraub@gmail.com>
Co-authored-by: christinestraub <christinestraub@users.noreply.github.com>
2023-10-06 18:54:49 -04:00
|
|
|
platformdirs==3.10.0
|
2023-10-26 12:22:40 -05:00
|
|
|
pillow<=10.0.1
|