yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-08-06 07:50:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/requirements/ingest/embed-huggingface.in

6 lines
80 B
Plaintext
Raw Normal View History

fix: make pip compile (#2015) - add missing make file in ingest folder
2023-11-06 16:26:12 -06:00
-c ../constraints.in
-c ../base.txt
feat: Adds local embedding model (#1619) This PR adds a local embedding model option as an alternative to using our OpenAI embedding brick. This brick uses LangChain's HuggingFacEmbeddings.
2023-10-19 11:51:36 -05:00
huggingface
langchain
feat: extend ingest options to support multiple embedding modules, add deterministic ingest test for embeddings (#1918) Closes #1782 This PR: - Extends ingest pipeline so that it is possible to select an embedding provider from a range of providers - Modifies the ingest embedding test to be a diff test, since the embedding vectors are reproducible after supporting multiple providers Additional info on the chosen provider for the test: - Found `langchain.embeddings.HuggingFaceEmbeddings` to be deterministic even when there's no seed set - Took 6.84s to pass a unit test with the provider (without cache, including model download) - `langchain.embeddings.HuggingFaceEmbeddings` runs in local, making it zero cost For all these reasons, testing embedding modules with the Huggingface model seems to be making sense --------- Co-authored-by: cragwolfe <crag@unstructured.io> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
2023-11-06 12:26:12 +00:00
sentence_transformers
Reference in New Issue Copy Permalink