yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-19 07:02:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/requirements/ingest-openai.txt

117 lines
2.3 KiB
Plaintext
Raw Normal View History

Roman/azure cognitive embeddings (#1524) ### Description This PR is two-fold: **Embeddings:** * Embeddings incorporated into the sharepoint source connector, which will now call out to OpenAI and create embeddings if the flag is passed in and the api key provided. **Writing vector content (embeddings) to Azure cognitive search index:** * The schema for the index expected to exist in Azure has been updated to include the vector field type and a test script has been added to test the new content being produced from the Sharepoint connector to push the embedding content. Some important notes about other changes in here: * The embedding code had to be updated to patch the `to_dict` method on elements to add `embeddings` to the dict output if that was added. While the code originally added the embedding content, when `to_dict` was called to save the content as json, this was lost.
2023-09-26 19:24:21 -04:00
#
# This file is autogenerated by pip-compile with Python 3.8
# by the following command:
#
# pip-compile requirements/ingest-openai.in
#
aiohttp==3.8.5
# via
# langchain
# openai
aiosignal==1.3.1
# via aiohttp
async-timeout==4.0.3
# via
# aiohttp
# langchain
attrs==23.1.0
# via aiohttp
certifi==2023.7.22
# via
# -c requirements/base.txt
# -c requirements/constraints.in
# requests
charset-normalizer==3.2.0
# via
# -c requirements/base.txt
# aiohttp
# requests
dataclasses-json==0.6.1
# via
# -c requirements/base.txt
# langchain
frozenlist==1.4.0
# via
# aiohttp
# aiosignal
idna==3.4
# via
# -c requirements/base.txt
# requests
# yarl
langchain==0.0.298
# via -r requirements/ingest-openai.in
feat: get embedded url, associate text and start index for pdf (#1539) **Executive Summary** Adds PDF functionality to capture hyperlink (external or internal) for pdf fast strategy along with associate text. **Technical Details** - `pdfminer` associates `annotation` (links and uris) with bounding box rather than text. Therefore, the link and text matching is not a perfect pair but rather a logic-based and calculation matching from bounding box overlapping. - There is no word-level bounding box. Only character-level (access using `LTChar`). Thus in order to get to word-level, there is a window slicing through the text. The words are captured in alphanumeric and non-alphanumeric separately, meaning it will split the word if contains both, on the first encounter of non-alphanumeric.) - The bounding box calculation is calculated using start and stop coordinates for the corresponding word calculated from above. The calculation is simply using distance between two dots. The result now contains `links` in `metadata` as shown below: ``` "links": [ { "text": "link", "url": "https://github.com/Unstructured-IO/unstructured", "start_index": 12 }, { "text": "email", "url": "mailto:unstructuredai@earlygrowth.com", "start_index": 30 }, { "text": "phone number", "url": "tel:6505124019", "start_index": 49 } ] ``` --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Klaijan <Klaijan@users.noreply.github.com>
2023-09-27 13:43:32 -04:00
langsmith==0.0.41
Roman/azure cognitive embeddings (#1524) ### Description This PR is two-fold: **Embeddings:** * Embeddings incorporated into the sharepoint source connector, which will now call out to OpenAI and create embeddings if the flag is passed in and the api key provided. **Writing vector content (embeddings) to Azure cognitive search index:** * The schema for the index expected to exist in Azure has been updated to include the vector field type and a test script has been added to test the new content being produced from the Sharepoint connector to push the embedding content. Some important notes about other changes in here: * The embedding code had to be updated to patch the `to_dict` method on elements to add `embeddings` to the dict output if that was added. While the code originally added the embedding content, when `to_dict` was called to save the content as json, this was lost.
2023-09-26 19:24:21 -04:00
# via langchain
marshmallow==3.20.1
# via
# -c requirements/base.txt
# dataclasses-json
multidict==6.0.4
# via
# aiohttp
# yarl
mypy-extensions==1.0.0
# via
# -c requirements/base.txt
# typing-inspect
numexpr==2.8.6
# via langchain
numpy==1.24.4
# via
feat: get embedded url, associate text and start index for pdf (#1539) **Executive Summary** Adds PDF functionality to capture hyperlink (external or internal) for pdf fast strategy along with associate text. **Technical Details** - `pdfminer` associates `annotation` (links and uris) with bounding box rather than text. Therefore, the link and text matching is not a perfect pair but rather a logic-based and calculation matching from bounding box overlapping. - There is no word-level bounding box. Only character-level (access using `LTChar`). Thus in order to get to word-level, there is a window slicing through the text. The words are captured in alphanumeric and non-alphanumeric separately, meaning it will split the word if contains both, on the first encounter of non-alphanumeric.) - The bounding box calculation is calculated using start and stop coordinates for the corresponding word calculated from above. The calculation is simply using distance between two dots. The result now contains `links` in `metadata` as shown below: ``` "links": [ { "text": "link", "url": "https://github.com/Unstructured-IO/unstructured", "start_index": 12 }, { "text": "email", "url": "mailto:unstructuredai@earlygrowth.com", "start_index": 30 }, { "text": "phone number", "url": "tel:6505124019", "start_index": 49 } ] ``` --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Klaijan <Klaijan@users.noreply.github.com>
2023-09-27 13:43:32 -04:00
# -c requirements/base.txt
Roman/azure cognitive embeddings (#1524) ### Description This PR is two-fold: **Embeddings:** * Embeddings incorporated into the sharepoint source connector, which will now call out to OpenAI and create embeddings if the flag is passed in and the api key provided. **Writing vector content (embeddings) to Azure cognitive search index:** * The schema for the index expected to exist in Azure has been updated to include the vector field type and a test script has been added to test the new content being produced from the Sharepoint connector to push the embedding content. Some important notes about other changes in here: * The embedding code had to be updated to patch the `to_dict` method on elements to add `embeddings` to the dict output if that was added. While the code originally added the embedding content, when `to_dict` was called to save the content as json, this was lost.
2023-09-26 19:24:21 -04:00
# -c requirements/constraints.in
# langchain
# numexpr
openai==0.28.1
# via -r requirements/ingest-openai.in
packaging==23.1
# via
# -c requirements/base.txt
# marshmallow
pydantic==1.10.12
# via
# -c requirements/constraints.in
# langchain
# langsmith
pyyaml==6.0.1
# via langchain
regex==2023.8.8
# via
# -c requirements/base.txt
# tiktoken
requests==2.31.0
# via
# -c requirements/base.txt
# langchain
# langsmith
# openai
# tiktoken
sqlalchemy==2.0.21
# via langchain
tenacity==8.2.3
# via langchain
tiktoken==0.5.1
# via -r requirements/ingest-openai.in
tqdm==4.66.1
# via
# -c requirements/base.txt
# openai
typing-extensions==4.8.0
# via
# -c requirements/base.txt
# pydantic
# sqlalchemy
# typing-inspect
typing-inspect==0.9.0
# via
# -c requirements/base.txt
# dataclasses-json
urllib3==1.26.16
# via
# -c requirements/base.txt
# -c requirements/constraints.in
# requests
yarl==1.9.2
# via aiohttp
Reference in New Issue Copy Permalink