unstructured/test_unstructured_ingest/test-ingest-against-api.sh

#!/usr/bin/env bash

set -e

SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
cd "$SCRIPT_DIR"/.. || exit 1

PYTHONPATH=. ./unstructured/ingest/main.py \
    --local-input-path example-docs \
    --local-file-glob "*.pdf" \
    --structured-output-dir api-ingest-output \
    --partition-by-api \
    --verbose \
    --reprocess

set +e

if [ "$(find 'api-ingest-output' -type f -printf '.' | wc -c)" != 4 ]; then
   echo
   echo "4 files should have been created."
   exit 1
fi
feat: add --partition-by-api and --partition-host to unstructured-ingest (#443) * Add --partition-by-api and --partition-host args to ingest * Fix error in make check * Bump changelog * Add a test ingest script Also add a workaround for the test causing 400s from our api. Seems we need to make sure unstructured-api can handle getting a file.content_type of None. * Remove the content type workaround 2023-04-12 01:05:07 -04:00			`#!/usr/bin/env bash`

test: avoid misleading output in ingest tests (#488) Previously, if there was an error (non-zero exit code) in an ingest test script, the script would still complete and echo a warning about mismatched outputs and how to regenerate the fixtures. However, this statement is irrelevant and misleading: if the ingest failed with a non-zero exit code in the first place, that is the failure that should be debugged -- don't confuse the user with a comment about outputs. 2023-04-17 14:57:44 -07:00			`set -e`

feat: add --partition-by-api and --partition-host to unstructured-ingest (#443) * Add --partition-by-api and --partition-host args to ingest * Fix error in make check * Bump changelog * Add a test ingest script Also add a workaround for the test causing 400s from our api. Seems we need to make sure unstructured-api can handle getting a file.content_type of None. * Remove the content type workaround 2023-04-12 01:05:07 -04:00			`SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)`
			`cd "$SCRIPT_DIR"/.. \|\| exit 1`

			`PYTHONPATH=. ./unstructured/ingest/main.py \`
			`--local-input-path example-docs \`
			`--local-file-glob "*.pdf" \`
			`--structured-output-dir api-ingest-output \`
			`--partition-by-api \`
			`--verbose \`
			`--reprocess`

test: avoid misleading output in ingest tests (#488) Previously, if there was an error (non-zero exit code) in an ingest test script, the script would still complete and echo a warning about mismatched outputs and how to regenerate the fixtures. However, this statement is irrelevant and misleading: if the ingest failed with a non-zero exit code in the first place, that is the failure that should be debugged -- don't confuse the user with a comment about outputs. 2023-04-17 14:57:44 -07:00			`set +e`

fix: group together text from the same bounding box in `partition_pdf` with fast strategy (#542) * switch to using PDF objects * linting, linting, linting * couple more tweaks * added test for chevron-page * version and changelog * linting, linting, linting * now processing 4 files 2023-05-03 18:33:24 -04:00			`if [ "$(find 'api-ingest-output' -type f -printf '.' \| wc -c)" != 4 ]; then`
feat: add --partition-by-api and --partition-host to unstructured-ingest (#443) * Add --partition-by-api and --partition-host args to ingest * Fix error in make check * Bump changelog * Add a test ingest script Also add a workaround for the test causing 400s from our api. Seems we need to make sure unstructured-api can handle getting a file.content_type of None. * Remove the content type workaround 2023-04-12 01:05:07 -04:00			`echo`
fix: group together text from the same bounding box in `partition_pdf` with fast strategy (#542) * switch to using PDF objects * linting, linting, linting * couple more tweaks * added test for chevron-page * version and changelog * linting, linting, linting * now processing 4 files 2023-05-03 18:33:24 -04:00			`echo "4 files should have been created."`
feat: add --partition-by-api and --partition-host to unstructured-ingest (#443) * Add --partition-by-api and --partition-host args to ingest * Fix error in make check * Bump changelog * Add a test ingest script Also add a workaround for the test causing 400s from our api. Seems we need to make sure unstructured-api can handle getting a file.content_type of None. * Remove the content type workaround 2023-04-12 01:05:07 -04:00			`exit 1`
			`fi`