mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-07-03 07:05:20 +00:00
4b8352e0f5
Adds Chroma (also known as ChromaDB) as a vector destination. Currently Chroma is an in-memory single-process oriented library with plans of a hosted and/or more production ready solution -https://docs.trychroma.com/deployment Though they now claim to support multiple Clients hitting the database at once, I found that it was inconsistent. Sometimes multiprocessing worked (maybe 1 out of 3 times) But the other times I would get different errors. So I kept it single process. --------- Co-authored-by: potter-potter <david.potter@gmail.com>
18 lines
483 B
Bash
18 lines
483 B
Bash
#!/usr/bin/env bash
|
|
|
|
unstructured-ingest \
|
|
local \
|
|
--input-path example-docs/book-war-and-peace-1p.txt \
|
|
--output-dir local-to-pinecone \
|
|
--strategy fast \
|
|
--chunk-elements \
|
|
--embedding-provider "<unstructured embedding provider, ie. langchain-huggingface>" \
|
|
--num-processes 2 \
|
|
--verbose \
|
|
--work-dir "<directory for intermediate outputs to be saved>" \
|
|
chroma \
|
|
--host "localhost" \
|
|
--port 8000 \
|
|
--collection-name "collection name" \
|
|
--batch-size 80
|