yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-12 19:45:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/docs/source/bricks.rst

23 lines
857 B
ReStructuredText
Raw Normal View History

Initial Release
2022-06-29 14:35:19 -04:00
Bricks
======
docs: more detailed bricks writeup; reoganize docs (#304) * add print statement in readme * elements before bricks * new preamble to bricks section * add preamble to bricks section * add preamble to cleaning section * descriptions of each documentation page * non-brick helper functions to the bottom * fix codeblock * includes some optional kwargs * code blocks * typo fix
2023-02-27 18:11:49 -05:00
Bricks are functions that live in ``unstructured`` and are the primary public API for the library.
Update docs jack (#1157) Documentation Overhaul - Added documentation hierarchy - Added options for Bash vs Python for API & Upstream Connectors - Added Introduction section (Overview, Key Concepts, Getting Started) - Redid connectors section - Installation is now broken up (needs further work)
2023-08-21 10:27:32 -07:00
There are several types of bricks in ``unstructured``, corresponding to the different stages of document pre-processing: partitioning, cleaning, chunking and staging.
docs: more detailed bricks writeup; reoganize docs (#304) * add print statement in readme * elements before bricks * new preamble to bricks section * add preamble to bricks section * add preamble to cleaning section * descriptions of each documentation page * non-brick helper functions to the bottom * fix codeblock * includes some optional kwargs * code blocks * typo fix
2023-02-27 18:11:49 -05:00
After reading this section, you should understand the following:
Update docs jack (#1157) Documentation Overhaul - Added documentation hierarchy - Added options for Bash vs Python for API & Upstream Connectors - Added Introduction section (Overview, Key Concepts, Getting Started) - Redid connectors section - Installation is now broken up (needs further work)
2023-08-21 10:27:32 -07:00
* How to partition a document into json or csv.
docs: more detailed bricks writeup; reoganize docs (#304) * add print statement in readme * elements before bricks * new preamble to bricks section * add preamble to bricks section * add preamble to cleaning section * descriptions of each documentation page * non-brick helper functions to the bottom * fix codeblock * includes some optional kwargs * code blocks * typo fix
2023-02-27 18:11:49 -05:00
* How to remove unwanted content from document elements using cleaning bricks.
Update docs jack (#1157) Documentation Overhaul - Added documentation hierarchy - Added options for Bash vs Python for API & Upstream Connectors - Added Introduction section (Overview, Key Concepts, Getting Started) - Redid connectors section - Installation is now broken up (needs further work)
2023-08-21 10:27:32 -07:00
* How to extract content from a document using the extraction bricks.
docs: more detailed bricks writeup; reoganize docs (#304) * add print statement in readme * elements before bricks * new preamble to bricks section * add preamble to bricks section * add preamble to cleaning section * descriptions of each documentation page * non-brick helper functions to the bottom * fix codeblock * includes some optional kwargs * code blocks * typo fix
2023-02-27 18:11:49 -05:00
* How to prepare data for downstream use cases using staging bricks
feat: chunk elements based on titles (#1222) ### Summary An initial pass on smart chunking for RAG applications. Breaks a document into sections based on the presence of `Title` elements. Also starts a new section under the following conditions: - If metadata changes, indicating a change in section or page or a switch to processing attachments. If `multipage_sections=True`, sections can span pages. `multipage_sections` defaults to True. - If the length of the section exceeds `new_after_n_chars` characters. The default is `1500`. The chunking function does not split individual elements, so it's possible for a section to exceed that threshold if an individual element if over `new_after_n_chars` characters, which could occur with a long `NarrativeText` element. - Section under `combine_under_n_chars` characters are combined. The default is `500`. ### Testing ```python from unstructured.partition.html import partition_html from unstructured.chunking.title import chunk_by_title url = "https://understandingwar.org/backgrounder/russian-offensive-campaign-assessment-august-27-2023-0" elements = partition_html(url=url) chunks = chunk_by_title(elements) for chunk in chunks: print(chunk) print("\n\n" + "-"*80) input() ```
2023-08-29 12:04:57 -04:00
* How to chunk partitioned documents for use cases such as Retrieval Augmented Generation (RAG).
docs: more detailed bricks writeup; reoganize docs (#304) * add print statement in readme * elements before bricks * new preamble to bricks section * add preamble to bricks section * add preamble to cleaning section * descriptions of each documentation page * non-brick helper functions to the bottom * fix codeblock * includes some optional kwargs * code blocks * typo fix
2023-02-27 18:11:49 -05:00
Update docs jack (#1157) Documentation Overhaul - Added documentation hierarchy - Added options for Bash vs Python for API & Upstream Connectors - Added Introduction section (Overview, Key Concepts, Getting Started) - Redid connectors section - Installation is now broken up (needs further work)
2023-08-21 10:27:32 -07:00
.. toctree::
:maxdepth: 1
Initial Release
2022-06-29 14:35:19 -04:00
Update docs jack (#1157) Documentation Overhaul - Added documentation hierarchy - Added options for Bash vs Python for API & Upstream Connectors - Added Introduction section (Overview, Key Concepts, Getting Started) - Redid connectors section - Installation is now broken up (needs further work)
2023-08-21 10:27:32 -07:00
bricks/partition
bricks/cleaning
bricks/extracting
bricks/staging
feat: chunk elements based on titles (#1222) ### Summary An initial pass on smart chunking for RAG applications. Breaks a document into sections based on the presence of `Title` elements. Also starts a new section under the following conditions: - If metadata changes, indicating a change in section or page or a switch to processing attachments. If `multipage_sections=True`, sections can span pages. `multipage_sections` defaults to True. - If the length of the section exceeds `new_after_n_chars` characters. The default is `1500`. The chunking function does not split individual elements, so it's possible for a section to exceed that threshold if an individual element if over `new_after_n_chars` characters, which could occur with a long `NarrativeText` element. - Section under `combine_under_n_chars` characters are combined. The default is `500`. ### Testing ```python from unstructured.partition.html import partition_html from unstructured.chunking.title import chunk_by_title url = "https://understandingwar.org/backgrounder/russian-offensive-campaign-assessment-august-27-2023-0" elements = partition_html(url=url) chunks = chunk_by_title(elements) for chunk in chunks: print(chunk) print("\n\n" + "-"*80) input() ```
2023-08-29 12:04:57 -04:00
bricks/chunking
Update Metadata and Installation Documentation (#1646) * Updated Metadata page: add common and additional metadata fields by document types and connectors * Updated specific installation extra by document types and connectors * Added embedding brick page in Sphinx TOC * Fixed Sphinx warnings in new pages
2023-10-04 18:25:41 -07:00
bricks/embedding
Reference in New Issue Copy Permalink