yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-17 22:15:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/docs/source/ingest/destination_connectors.rst

25 lines
784 B
ReStructuredText
Raw Normal View History

roman/delta lake dest connector (#1385) ### Description Add delta table downstream destination connector Closes https://github.com/Unstructured-IO/unstructured/issues/1415
2023-09-15 18:13:39 -04:00
Destination Connectors
======================
Connect to your favorite data storage platforms for effortless batch processing of your files.
We are constantly adding new data connectors and if you don't see your favorite platform let us know
fix: update slack link to link shortener (#2010) Per @tabossert we're now using a link shortener behind which we can rotate the link to keep it current. That way we (🤞 ) never have to update this here again. #### Testing: Links should work. No more links should exist in the documentation except this one.
2023-11-06 09:47:18 -06:00
in our community `Slack. <https://short.unstructured.io/pzw05l7>`_
roman/delta lake dest connector (#1385) ### Description Add delta table downstream destination connector Closes https://github.com/Unstructured-IO/unstructured/issues/1415
2023-09-15 18:13:39 -04:00
.. toctree::
:maxdepth: 1
chore: update sphinx ingest docs with new connectors (#2245) Replacing https://github.com/Unstructured-IO/unstructured/pull/2243
2023-12-11 16:29:41 -05:00
destination_connectors/azure
feat: adds Azure Cognitive Search (full text) destination connector (#1459) ### Description New [Azure Cognitive Search](https://azure.microsoft.com/en-us/products/ai-services/cognitive-search) destination connector added. Writes each json element from the created json files via partition and writes that content to an index. **Bonus bug fix:** Due to a recent change where the default version of python used in the repo was bumped to `3.10` from `3.8`, this means running `pip-compile` now runs it against that version rather than the lowest we support which is still `3.8`. This breaks the setup for those lower versions because some of the versions pulled in by `pip-compile` exist for `3.10` but not `3.8`. `pip-compile` was updates to run as a script that checks the version of python being used first, which helps guarantee that all dependencies meet the minimum python version requirement. Closes out https://github.com/Unstructured-IO/unstructured/issues/1466
2023-09-25 10:27:42 -04:00
destination_connectors/azure_cognitive_search
chore: update sphinx ingest docs with new connectors (#2245) Replacing https://github.com/Unstructured-IO/unstructured/pull/2243
2023-12-11 16:29:41 -05:00
destination_connectors/box
feat: add chroma destination connector (#2240) Adds Chroma (also known as ChromaDB) as a vector destination. Currently Chroma is an in-memory single-process oriented library with plans of a hosted and/or more production ready solution -https://docs.trychroma.com/deployment Though they now claim to support multiple Clients hitting the database at once, I found that it was inconsistent. Sometimes multiprocessing worked (maybe 1 out of 3 times) But the other times I would get different errors. So I kept it single process. --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2023-12-19 08:58:23 -08:00
destination_connectors/chroma
docs: add mongodb destination connector to ingest docs (#2136) The mongodb documentation page exists but is missing from the list of destination connectors, meaning it's not actually visible in the documentation.
2023-11-21 11:11:25 -08:00
destination_connectors/delta_table
chore: update sphinx ingest docs with new connectors (#2245) Replacing https://github.com/Unstructured-IO/unstructured/pull/2243
2023-12-11 16:29:41 -05:00
destination_connectors/dropbox
destination_connectors/gcs
docs: add mongodb destination connector to ingest docs (#2136) The mongodb documentation page exists but is missing from the list of destination connectors, meaning it's not actually visible in the documentation.
2023-11-21 11:11:25 -08:00
destination_connectors/mongodb
feat: add pinecone destination connector (#1774) Closes https://github.com/Unstructured-IO/unstructured/issues/1414 Closes #2039 This PR: - Uses Pinecone python cli to implement a destination connector for Pinecone and provides the ingest readme requirements [(here)](https://github.com/Unstructured-IO/unstructured/tree/main/unstructured/ingest#the-checklist) for the connector - Updates documentation for the s3 destination connector - Alphabetically sorts setup.py contents - Updates logs for the chunking node in ingest pipeline - Adds a baseline session handle implementation for destination connectors, to be able to parallelize their operations - For the [bug](https://github.com/Unstructured-IO/unstructured/issues/1892) related to persisting element data to ingest embedding nodes; this PR tests the [solution](https://github.com/Unstructured-IO/unstructured/pull/1893) with its ingest test - Solves a bug on ingest chunking params with [bugfix on chunking params and implementing related test](https://github.com/Unstructured-IO/unstructured/pull/1774/commits/69e1949a6fa806d14014eb68074929595f92360e) --------- Co-authored-by: Roman Isecke <136338424+rbiseck3@users.noreply.github.com>
2023-11-29 22:37:32 +00:00
destination_connectors/pinecone
feat: add Qdrant ingest destination connector (#2338) This PR intends to add [Qdrant](https://qdrant.tech/) as a supported ingestion destination. - Implements CLI and programmatic usage. - Documentation update - Integration test script --- Clone of #2315 to run with CI secrets --------- Co-authored-by: Anush008 <anushshetty90@gmail.com> Co-authored-by: Roman Isecke <136338424+rbiseck3@users.noreply.github.com>
2024-01-02 14:08:20 -08:00
destination_connectors/qdrant
feat: add pinecone destination connector (#1774) Closes https://github.com/Unstructured-IO/unstructured/issues/1414 Closes #2039 This PR: - Uses Pinecone python cli to implement a destination connector for Pinecone and provides the ingest readme requirements [(here)](https://github.com/Unstructured-IO/unstructured/tree/main/unstructured/ingest#the-checklist) for the connector - Updates documentation for the s3 destination connector - Alphabetically sorts setup.py contents - Updates logs for the chunking node in ingest pipeline - Adds a baseline session handle implementation for destination connectors, to be able to parallelize their operations - For the [bug](https://github.com/Unstructured-IO/unstructured/issues/1892) related to persisting element data to ingest embedding nodes; this PR tests the [solution](https://github.com/Unstructured-IO/unstructured/pull/1893) with its ingest test - Solves a bug on ingest chunking params with [bugfix on chunking params and implementing related test](https://github.com/Unstructured-IO/unstructured/pull/1774/commits/69e1949a6fa806d14014eb68074929595f92360e) --------- Co-authored-by: Roman Isecke <136338424+rbiseck3@users.noreply.github.com>
2023-11-29 22:37:32 +00:00
destination_connectors/s3
feat: adds postgresql/sqlite destination connector (#2005) - Adds a destination connector to upload processed output into a PostgreSQL/Sqlite database instance. - Users are responsible to provide their instances. This PR includes a couple of configuration examples. - Defines the scripts required to setup a PostgreSQL instance with the unstructured elements schema. - Validates postgres/pgvector embedding storage and retrieval --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2024-01-04 13:33:16 -06:00
destination_connectors/sql
chore: update sphinx ingest docs with new connectors (#2245) Replacing https://github.com/Unstructured-IO/unstructured/pull/2243
2023-12-11 16:29:41 -05:00
destination_connectors/weaviate
Reference in New Issue Copy Permalink