yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-07-05 16:12:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured_ingest/test-ingest.sh

15 lines
504 B
Bash
Raw Normal View History

feat: Sample ingest project with S3 connector (#218)
2023-02-14 12:27:45 -08:00
#!/usr/bin/env bash
set -eux -o pipefail
feat: Ingest refactors, doc updates (#243) - Creates ABC's for ingest connectors - Updates the s3_connector classes to inherit from ABC's - Moves s3 test script to it's own file to establish pattern for additional connectors - Rewrites the Ingest.md doc, including instructions how how to add a connector - Updates the example s3 ingest script to use the new location for main.py Note that there were no logic changes, this is essentially a refactoring PR. Test instructions: Run ./test_unstructured_ingest/test-ingest.sh and ./examples/ingest/s3-small-batch/ingest.sh.
2023-02-21 10:15:33 -08:00
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd "$SCRIPT_DIR"/.. || exit 1
feat: Ingest CLI flags and test fixture updates (#227) * Many command line options added. The sample ingest project is now an easy to use CLI (no code editing necessary), capable of processing large numbers of files from S3 in a re-entrant manner. See Ingest.md. * Fixes issue where text fixtures had been truncated * Adds a check to make sure this doesn't happen again * Moves fixture outputs for the existing connector one subdir lower, to make room for future connector outputs.
2023-02-16 08:45:50 -08:00
feat: Ingest refactors, doc updates (#243) - Creates ABC's for ingest connectors - Updates the s3_connector classes to inherit from ABC's - Moves s3 test script to it's own file to establish pattern for additional connectors - Rewrites the Ingest.md doc, including instructions how how to add a connector - Updates the example s3 ingest script to use the new location for main.py Note that there were no logic changes, this is essentially a refactoring PR. Test instructions: Run ./test_unstructured_ingest/test-ingest.sh and ./examples/ingest/s3-small-batch/ingest.sh.
2023-02-21 10:15:33 -08:00
./test_unstructured_ingest/test-ingest-s3.sh
Add `AzureBlobStorageConnector` (#353) * Add `AzureBlobStorageConnector` based on its `fsspec` implementation inheriting from `FsspecConnector` * Start deprecation life cycle for `unstructured-ingest --s3-url` option, to be deprecated in favor of `--remote-url`. --------- Co-authored-by: Tom Aarsen <37621491+tomaarsen@users.noreply.github.com>
2023-03-11 00:43:40 +01:00
./test_unstructured_ingest/test-ingest-azure.sh
feat: Add GitHub data connector; add Markdown partitioner (#284)
2023-02-27 23:36:44 +01:00
./test_unstructured_ingest/test-ingest-github.sh
feat: Add GitLab ingest connector (#349) Add GitLab data connector for ingest. Involves more general Git functionality that is shared between the GitHub and GitLab data connectors. Prevent code duplication for functionality between GitHub and GitLab ingest connectors. Renamed github-access-token, github-branch and github-file-glob to git-access-token, git-branch and git-file-glob, respectively. These work for GitHub and GitLab.
2023-03-08 09:15:21 +01:00
./test_unstructured_ingest/test-ingest-gitlab.sh
feat: Add Wikipedia ingest connector (#299) The connector can process a Wikipedia page and output the HTML, the plain text contents, and the summary. No API key required Also add test case verifying that 3 files are indeed created (one for HTML, one for text, one for the summary).
2023-02-28 09:25:11 +01:00
./test_unstructured_ingest/test-ingest-wikipedia.sh
Connector for Biomedical Literature (#345) The implementation involves the introduction of SimpleBiomedConfig, BiomedIngestDoc and BiomedConnector which ingests documents from the PDF Download.
2023-03-11 01:09:54 +00:00
./test_unstructured_ingest/test-ingest-biomed-api.sh
./test_unstructured_ingest/test-ingest-biomed-path.sh
Reference in New Issue Copy Permalink