unstructured/test_unstructured_ingest/test-ingest.sh

#!/usr/bin/env bash

set -eux -o pipefail

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd "$SCRIPT_DIR"/.. || exit 1

./test_unstructured_ingest/test-ingest-s3.sh
./test_unstructured_ingest/test-ingest-github.sh
./test_unstructured_ingest/test-ingest-gitlab.sh
./test_unstructured_ingest/test-ingest-wikipedia.sh
feat: Sample ingest project with S3 connector (#218) 2023-02-14 12:27:45 -08:00			`#!/usr/bin/env bash`

			`set -eux -o pipefail`

feat: Ingest refactors, doc updates (#243) - Creates ABC's for ingest connectors - Updates the s3_connector classes to inherit from ABC's - Moves s3 test script to it's own file to establish pattern for additional connectors - Rewrites the Ingest.md doc, including instructions how how to add a connector - Updates the example s3 ingest script to use the new location for main.py Note that there were no logic changes, this is essentially a refactoring PR. Test instructions: Run ./test_unstructured_ingest/test-ingest.sh and ./examples/ingest/s3-small-batch/ingest.sh. 2023-02-21 10:15:33 -08:00			`SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )`
			`cd "$SCRIPT_DIR"/.. \|\| exit 1`
feat: Ingest CLI flags and test fixture updates (#227) * Many command line options added. The sample ingest project is now an easy to use CLI (no code editing necessary), capable of processing large numbers of files from S3 in a re-entrant manner. See Ingest.md. * Fixes issue where text fixtures had been truncated * Adds a check to make sure this doesn't happen again * Moves fixture outputs for the existing connector one subdir lower, to make room for future connector outputs. 2023-02-16 08:45:50 -08:00
feat: Ingest refactors, doc updates (#243) - Creates ABC's for ingest connectors - Updates the s3_connector classes to inherit from ABC's - Moves s3 test script to it's own file to establish pattern for additional connectors - Rewrites the Ingest.md doc, including instructions how how to add a connector - Updates the example s3 ingest script to use the new location for main.py Note that there were no logic changes, this is essentially a refactoring PR. Test instructions: Run ./test_unstructured_ingest/test-ingest.sh and ./examples/ingest/s3-small-batch/ingest.sh. 2023-02-21 10:15:33 -08:00			`./test_unstructured_ingest/test-ingest-s3.sh`
feat: Add GitHub data connector; add Markdown partitioner (#284) 2023-02-27 23:36:44 +01:00			`./test_unstructured_ingest/test-ingest-github.sh`
feat: Add GitLab ingest connector (#349) Add GitLab data connector for ingest. Involves more general Git functionality that is shared between the GitHub and GitLab data connectors. Prevent code duplication for functionality between GitHub and GitLab ingest connectors. Renamed github-access-token, github-branch and github-file-glob to git-access-token, git-branch and git-file-glob, respectively. These work for GitHub and GitLab. 2023-03-08 09:15:21 +01:00			`./test_unstructured_ingest/test-ingest-gitlab.sh`
feat: Add Wikipedia ingest connector (#299) The connector can process a Wikipedia page and output the HTML, the plain text contents, and the summary. No API key required Also add test case verifying that 3 files are indeed created (one for HTML, one for text, one for the summary). 2023-02-28 09:25:11 +01:00			`./test_unstructured_ingest/test-ingest-wikipedia.sh`