3 Commits

Author SHA1 Message Date
Roman Isecke
11cdd8d71f
roman/drop downloads in ingest tests (#1614)
### Description
In an effort to mitigate resource consumption when running CI tests,
cleanup download dir for ingest tests after each one.
2023-10-02 20:47:24 +00:00
Roman Isecke
9836235ead
Chunking support for SharePoint Connector (#1548)
### Description
Optionally adds in chunking to the CLI which adds a flag to trigger
chunking and exposes the parameters used by the `chunk_by_title` method.
Runs chunking before the embedding step.


Opened to replace original PR
https://github.com/Unstructured-IO/unstructured/pull/1531
2023-09-27 21:05:55 +00:00
Roman Isecke
5c7b4f586b
Roman/azure cognitive embeddings (#1524)
### Description
This PR is two-fold:  

**Embeddings:**
* Embeddings incorporated into the sharepoint source connector, which
will now call out to OpenAI and create embeddings if the flag is passed
in and the api key provided.

**Writing vector content (embeddings) to Azure cognitive search index:**
* The schema for the index expected to exist in Azure has been updated
to include the vector field type and a test script has been added to
test the new content being produced from the Sharepoint connector to
push the embedding content.

Some important notes about other changes in here:
* The embedding code had to be updated to patch the `to_dict` method on
elements to add `embeddings` to the dict output if that was added. While
the code originally added the embedding content, when `to_dict` was
called to save the content as json, this was lost.
2023-09-26 23:24:21 +00:00