mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-07-15 04:55:56 +00:00
bc791d53f4
Adds OpenSearch as a source and destination. Since OpenSearch is a fork of Elasticsearch, these connectors rely heavily on inheriting the Elasticsearch connectors whenever possible. - Adds OpenSearch source connector to be able to ingest documents from OpenSearch. - Adds OpenSearch destination connector to be able to ingest documents from any supported source, embed them and write the embeddings / documents into OpenSearch. - Defines an example unstructured elements schema for users to be able to setup their unstructured OpenSearch indexes easily. --------- Co-authored-by: potter-potter <david.potter@gmail.com>
36 lines
1.4 KiB
Bash
Executable File
36 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
# Processes a the file from local, chunks, embeds, and writes the results to an opensearch index.
|
|
|
|
# Structured outputs are stored in local-to-opensearch/
|
|
|
|
SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
|
|
cd "$SCRIPT_DIR"/../../.. || exit 1
|
|
|
|
# As an example we're using the local connector,
|
|
# however ingesting from any supported source connector is possible.
|
|
# shellcheck disable=2094
|
|
PYTHONPATH=. ./unstructured/ingest/main.py \
|
|
local \
|
|
--input-path example-docs/book-war-and-peace-1225p.txt \
|
|
--output-dir local-to-opensearch \
|
|
--strategy fast \
|
|
--chunk-elements \
|
|
--embedding-provider "<an unstructured embedding provider, ie. langchain-huggingface>" \
|
|
--num-processes 2 \
|
|
--verbose \
|
|
--work-dir "<directory for intermediate outputs to be saved>" \
|
|
opensearch \
|
|
--hosts "<List of URLs where opensearch index is served>" \
|
|
--index-name "<Index name to upload data in>" \
|
|
--username "<Username to authenticate into the index>" \
|
|
--password "<Password to authenticate into the index>" \
|
|
--batch-size-bytes "<Size limit for any batch to be uploaded, in bytes, ie. 15000000>" \
|
|
--num-processes "<Number of processes to be used to upload, ie. 2>" \
|
|
--ca-certs "<path/to/ca/certs>" \
|
|
--client-cert "<path/to/client/cert>" \
|
|
--client-key "<path/to/client/key>" \
|
|
--use-ssl \
|
|
--verify-certs \
|
|
--ssl-show-warn
|