yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-06 00:21:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/docs/source/apis/api_parameters.rst

145 lines
5.3 KiB
ReStructuredText
Raw Normal View History

Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
API Parameters
==============
The endpoint of the API provides several parameters to customize the processing of documents. Below are the details of these parameters:
files
-----
- **Type**: string (binary format)
- **Description**: The file to extract.
- **Required**: true
- **Example**: File to be partitioned. `Example File <https://github.com/Unstructured-IO/unstructured/blob/98d3541909f64290b5efb65a226fc3ee8a7cc5ee/example-docs/layout-parser-paper.pdf>`_
strategy
--------
- **Type**: string
- **Description**: The strategy to use for partitioning PDF/image. Options are fast, hi_res, auto. Default: auto.
- **Example**: hi_res
gz_uncompressed_content_type
-----------------------------
- **Type**: string
- **Description**: If file is gzipped, use this content type after unzipping.
- **Example**: application/pdf
output_format
-------------
- **Type**: string
- **Description**: The format of the response. Supported formats are application/json and text/csv. Default: application/json.
- **Example**: application/json
coordinates
-----------
- **Type**: boolean
- **Description**: If true, return coordinates for each element. Default: false.
encoding
--------
- **Type**: string
- **Description**: The encoding method used to decode the text input. Default: utf-8.
- **Example**: utf-8
feat(chunking): add metadata.orig_elements serde (#2680) **Summary** This final PR in the "orig_elements" series adds the needful such that `.metadata.orig_elements`, when present on a chunk (element), is serialized to JSON when the chunk is serialized, for instance, to be used in an HTTP response payload. It also provides for deserializing such a JSON payload into chunks that contain the `.orig_elements` metadata. **Additional Context** Note that `.metadata.orig_elements` is always `Optional[list[Element]]` when in memory. However, those original elements are serialized as Base64-encoded gzipped JSON and are in that form (str) when present as JSON or as "element-dicts" which is an intermediate serialization/deserialization format. That is, serialization is `Element -> dict -> JSON` and deserialization is `JSON -> dict -> Element` and `.orig_elements` are Base64-encoded in both the `dict` and `JSON` forms. --------- Co-authored-by: scanny <scanny@users.noreply.github.com>
2024-03-22 14:53:26 -07:00
extract_image_block_types
-------------------------
- **Type**: array
- **Description**: The types of image blocks to extract from the document. Supports various Element types.
- **Example**: ['Image', 'Table']
Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
hi_res_model_name
-----------------
- **Type**: string
- **Description**: The name of the inference model used when strategy is hi_res.
- **Example**: yolox
include_page_breaks
-------------------
- **Type**: boolean
feat(chunking): add metadata.orig_elements serde (#2680) **Summary** This final PR in the "orig_elements" series adds the needful such that `.metadata.orig_elements`, when present on a chunk (element), is serialized to JSON when the chunk is serialized, for instance, to be used in an HTTP response payload. It also provides for deserializing such a JSON payload into chunks that contain the `.orig_elements` metadata. **Additional Context** Note that `.metadata.orig_elements` is always `Optional[list[Element]]` when in memory. However, those original elements are serialized as Base64-encoded gzipped JSON and are in that form (str) when present as JSON or as "element-dicts" which is an intermediate serialization/deserialization format. That is, serialization is `Element -> dict -> JSON` and deserialization is `JSON -> dict -> Element` and `.orig_elements` are Base64-encoded in both the `dict` and `JSON` forms. --------- Co-authored-by: scanny <scanny@users.noreply.github.com>
2024-03-22 14:53:26 -07:00
- **Description**: When true, the output will include page break elements when the filetype supports
it. Default: false.
Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
languages
---------
- **Type**: array
- **Description**: The languages present in the document, for use in partitioning and/or OCR.
- **Default**: []
- **Example**: [eng]
pdf_infer_table_structure
-------------------------
- **Type**: boolean
chore: change table extraction defaults (#2588) Change default values for table extraction - works in pair with [this](https://github.com/Unstructured-IO/unstructured-api/pull/370) `unstructured-api` PR We want to move away from `pdf_infer_table_structure` parameter, in this PR: - We change how it's treated wrt `skip_infer_table_types` parameter. Whether to extract tables from pdf now follows from the rule: `pdf_infer_table_structure && "pdf" not in skip_infer_table_types` - We set it to `pdf_infer_table_structure=True` and `skip_infer_table_types=[]` by default - We remove it from the examples in documentation - We describe it as deprecated in favor of `skip_infer_table_types` in documentation More detailed description of how we want parameters to interact - if `pdf_infer_table_structure` is False tables will never extracted from pdf - if `pdf_infer_table_structure` is True tables will be extracted from pdf unless it's skipped via `skip_infer_table_types` - on default `pdf_infer_table_structure=True` and `skip_infer_table_types=[]` --------- Co-authored-by: Filip Knefel <filip@unstructured.io> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ds-filipknefel <ds-filipknefel@users.noreply.github.com> Co-authored-by: Ronny H <138828701+ron-unstructured@users.noreply.github.com>
2024-03-22 11:08:49 +01:00
- **Description**: Deprecated! Use skip_infer_table_types to opt out of table extraction for any file type. If False and strategy=hi_res, no Table Elements will be extracted from pdf files regardless of skip_infer_table_types contents.
Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
skip_infer_table_types
----------------------
- **Type**: array
feat: add support for partitioning .heic files (#2454) .heic files are an image filetype we have not supported. #### Testing ``` from unstructured.partition.image import partition_image png_filename = "example-docs/DA-1p.png" heic_filename = "example-docs/DA-1p.heic" png_elements = partition_image(png_filename, strategy="hi_res") heic_elements = partition_image(heic_filename, strategy="hi_res") for i in range(len(heic_elements)): print(heic_elements[i].text == png_elements[i].text) ``` --------- Co-authored-by: christinestraub <christinemstraub@gmail.com>
2024-01-29 22:49:00 -06:00
- **Description**: The document types that you want to skip table extraction with. Default: ['pdf', 'jpg', 'png', 'heic'].
Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
xml_keep_tags
-------------
- **Type**: boolean
- **Description**: If True, will retain the XML tags in the output. Otherwise it will simply extract the text from within the tags. Only applies to partition_xml.
feat(chunking): add metadata.orig_elements serde (#2680) **Summary** This final PR in the "orig_elements" series adds the needful such that `.metadata.orig_elements`, when present on a chunk (element), is serialized to JSON when the chunk is serialized, for instance, to be used in an HTTP response payload. It also provides for deserializing such a JSON payload into chunks that contain the `.orig_elements` metadata. **Additional Context** Note that `.metadata.orig_elements` is always `Optional[list[Element]]` when in memory. However, those original elements are serialized as Base64-encoded gzipped JSON and are in that form (str) when present as JSON or as "element-dicts" which is an intermediate serialization/deserialization format. That is, serialization is `Element -> dict -> JSON` and deserialization is `JSON -> dict -> Element` and `.orig_elements` are Base64-encoded in both the `dict` and `JSON` forms. --------- Co-authored-by: scanny <scanny@users.noreply.github.com>
2024-03-22 14:53:26 -07:00
Chunking Parameters
-------------------
The following parameters control chunking behavior. Chunking is automatically performed after
partitioning when a value is provided for the ``chunking_strategy`` argument. The remaining chunking
parameters are only operative when a chunking strategy is specified. Note that not all chunking
parameters apply to all chunking strategies. Any chunking arguments not supported by the selected
chunker are ignored.
Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
chunking_strategy
-----------------
- **Type**: string
feat(chunking): add metadata.orig_elements serde (#2680) **Summary** This final PR in the "orig_elements" series adds the needful such that `.metadata.orig_elements`, when present on a chunk (element), is serialized to JSON when the chunk is serialized, for instance, to be used in an HTTP response payload. It also provides for deserializing such a JSON payload into chunks that contain the `.orig_elements` metadata. **Additional Context** Note that `.metadata.orig_elements` is always `Optional[list[Element]]` when in memory. However, those original elements are serialized as Base64-encoded gzipped JSON and are in that form (str) when present as JSON or as "element-dicts" which is an intermediate serialization/deserialization format. That is, serialization is `Element -> dict -> JSON` and deserialization is `JSON -> dict -> Element` and `.orig_elements` are Base64-encoded in both the `dict` and `JSON` forms. --------- Co-authored-by: scanny <scanny@users.noreply.github.com>
2024-03-22 14:53:26 -07:00
- **Description**: Use one of the supported strategies to chunk the returned elements. When omitted,
no chunking is performed and any other chunking parameters provided are ignored.
- **Valid values**: ``"basic"``, ``"by_title"``
Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
combine_under_n_chars
---------------------
- **Type**: integer
feat(chunking): add metadata.orig_elements serde (#2680) **Summary** This final PR in the "orig_elements" series adds the needful such that `.metadata.orig_elements`, when present on a chunk (element), is serialized to JSON when the chunk is serialized, for instance, to be used in an HTTP response payload. It also provides for deserializing such a JSON payload into chunks that contain the `.orig_elements` metadata. **Additional Context** Note that `.metadata.orig_elements` is always `Optional[list[Element]]` when in memory. However, those original elements are serialized as Base64-encoded gzipped JSON and are in that form (str) when present as JSON or as "element-dicts" which is an intermediate serialization/deserialization format. That is, serialization is `Element -> dict -> JSON` and deserialization is `JSON -> dict -> Element` and `.orig_elements` are Base64-encoded in both the `dict` and `JSON` forms. --------- Co-authored-by: scanny <scanny@users.noreply.github.com>
2024-03-22 14:53:26 -07:00
- **Applicable Chunkers**: "by_title" only
- **Description**: When chunking strategy is set to "by_title", combine small chunks until the
combined chunk reaches a length of n chars. This can mitigate the appearance of small chunks
created by short paragraphs, not intended as section headings, being identified as ``Title``
elements in certain documents.
- **Default**: the same value as ``max_characters``
Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
- **Example**: 500
feat(chunking): add metadata.orig_elements serde (#2680) **Summary** This final PR in the "orig_elements" series adds the needful such that `.metadata.orig_elements`, when present on a chunk (element), is serialized to JSON when the chunk is serialized, for instance, to be used in an HTTP response payload. It also provides for deserializing such a JSON payload into chunks that contain the `.orig_elements` metadata. **Additional Context** Note that `.metadata.orig_elements` is always `Optional[list[Element]]` when in memory. However, those original elements are serialized as Base64-encoded gzipped JSON and are in that form (str) when present as JSON or as "element-dicts" which is an intermediate serialization/deserialization format. That is, serialization is `Element -> dict -> JSON` and deserialization is `JSON -> dict -> Element` and `.orig_elements` are Base64-encoded in both the `dict` and `JSON` forms. --------- Co-authored-by: scanny <scanny@users.noreply.github.com>
2024-03-22 14:53:26 -07:00
include_orig_elements
---------------------
- **Type**: boolean
- **Applicable Chunkers**: All
- **Description**: Add the elements used to form each chunk to ``.metadata.orig_elements`` for that
chunk. These can be used to recover the original text and metadata for individual elements when
that is required, for example to identify the page-numbers or coordinates spanned by a chunk.
When an element larger than ``max_characters`` is divided into two or more chunks via
text-splitting, each of those chunks will contain the entire original chunk as the only item in
its ``.metadata.orig_elements`` list.
- **Default**: true
Reorganized the Examples section in Documentation & add Databricks example (#1855) To test: > cd docs && make html Change logs: * Examples are reorganized to have its own page * Removed two old examples, ie. "file-utils" & "sentiment analysis". * Added two examples: "RAG with Unstructured, LangChain, and ChromaDB" & "Multi-Files Processing with S3 Connector and API" * Reorganized and added detailed API documentation: (i) usage, (ii) SDKs, (iii) Azure Marketplace, (iv) AWS Marketplace, (v) parameters and validation errors
2023-11-29 17:24:43 -08:00
max_characters
--------------
- **Type**: integer
feat(chunking): add metadata.orig_elements serde (#2680) **Summary** This final PR in the "orig_elements" series adds the needful such that `.metadata.orig_elements`, when present on a chunk (element), is serialized to JSON when the chunk is serialized, for instance, to be used in an HTTP response payload. It also provides for deserializing such a JSON payload into chunks that contain the `.orig_elements` metadata. **Additional Context** Note that `.metadata.orig_elements` is always `Optional[list[Element]]` when in memory. However, those original elements are serialized as Base64-encoded gzipped JSON and are in that form (str) when present as JSON or as "element-dicts" which is an intermediate serialization/deserialization format. That is, serialization is `Element -> dict -> JSON` and deserialization is `JSON -> dict -> Element` and `.orig_elements` are Base64-encoded in both the `dict` and `JSON` forms. --------- Co-authored-by: scanny <scanny@users.noreply.github.com>
2024-03-22 14:53:26 -07:00
- **Applicable Chunkers**: All
- **Description**: When chunking strategy is set, cut off new chunks after reaching a length of n
chars (hard max).
- **Default**: 500
Updated docs on API Params and Filetype Supports (#2433) To test: > cd docs && make html Changelogs: * Fixed sphinx error due to malformed rst table on partition page * Updated API Params, ie. `extract_image_block_types` and `extract_image_block_to_payload` * Updated image filetype supports
2024-01-19 16:07:57 -08:00
feat(chunking): add metadata.orig_elements serde (#2680) **Summary** This final PR in the "orig_elements" series adds the needful such that `.metadata.orig_elements`, when present on a chunk (element), is serialized to JSON when the chunk is serialized, for instance, to be used in an HTTP response payload. It also provides for deserializing such a JSON payload into chunks that contain the `.orig_elements` metadata. **Additional Context** Note that `.metadata.orig_elements` is always `Optional[list[Element]]` when in memory. However, those original elements are serialized as Base64-encoded gzipped JSON and are in that form (str) when present as JSON or as "element-dicts" which is an intermediate serialization/deserialization format. That is, serialization is `Element -> dict -> JSON` and deserialization is `JSON -> dict -> Element` and `.orig_elements` are Base64-encoded in both the `dict` and `JSON` forms. --------- Co-authored-by: scanny <scanny@users.noreply.github.com>
2024-03-22 14:53:26 -07:00
multipage_sections
------------------
- **Type**: boolean
- **Applicable Chunkers**: "by_title" only
- **Description**: When true and chunking strategy is set to "by_title", allows a chunk to include
elements from more than one page. Otherwise chunks are broken on page boundaries.
- **Default**: true
new_after_n_chars
-----------------
- **Type**: integer
- **Applicable Chunkers**: "basic", "by_title"
- **Description**: When chunking strategy is set, cut off new chunk after reaching a length of n
chars (soft max).
- **Default**: 1500
Reference in New Issue Copy Permalink