mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-11-01 10:33:09 +00:00
e8ec09c8b9
Thanks to Eric Hare @erichare at DataStax we have a new destination connector. This Pull Request implements an integration with [Astra DB](https://datastax.com) which allows for the Astra DB Vector Database to be compatible with Unstructured's set of integrations. To create your Astra account and authenticate with your `ASTRA_DB_APPLICATION_TOKEN`, and `ASTRA_DB_API_ENDPOINT`, follow these steps: 1. Create an account at https://astra.datastax.com 2. Login and create a new database 3. From the database page, in the right hand panel, you will find your API Endpoint 4. Beneath that, you can create a Token to be used Some notes about Astra DB: - Astra DB is a Vector Database which allows for high-performance database transactions, and enables modern GenAI apps [See here](https://docs.datastax.com/en/astra/astra-db-vector/get-started/concepts.html) - It supports similarity search via a number of methods [See here](https://docs.datastax.com/en/astra/astra-db-vector/get-started/concepts.html#metrics) - It also supports non-vector tables / collections
33 lines
1.1 KiB
ReStructuredText
33 lines
1.1 KiB
ReStructuredText
Astra
|
|
===========
|
|
|
|
Batch process all your records using ``unstructured-ingest`` to store structured outputs and embeddings locally on your filesystem and upload those to a Astra DB index.
|
|
|
|
First you'll need to install the Astra dependencies as shown here.
|
|
|
|
.. code:: shell
|
|
|
|
pip install "unstructured[astra]"
|
|
|
|
Run Locally
|
|
-----------
|
|
The upstream connector can be any of the ones supported, but for convenience here, showing a sample command using the
|
|
upstream local connector.
|
|
|
|
.. tabs::
|
|
|
|
.. tab:: Shell
|
|
|
|
.. literalinclude:: ./code/bash/astra.sh
|
|
:language: bash
|
|
|
|
.. tab:: Python
|
|
|
|
.. literalinclude:: ./code/python/astra.py
|
|
:language: python
|
|
|
|
|
|
For a full list of the options the CLI accepts check ``unstructured-ingest <upstream connector> astra --help``.
|
|
|
|
NOTE: Keep in mind that you will need to have all the appropriate extras and dependencies for the file types of the documents contained in your data storage platform if you're running this locally. You can find more information about this in the `installation guide <https://unstructured-io.github.io/unstructured/installing.html>`_.
|