mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-11-04 12:03:15 +00:00
b7674fb97e
* Add confluence connector and an example script * add test script, add dependency installations * add authentication secret variables for ci tests and actions * add dependency installation commands for workflows * add dependency installation commands for workflows * Update ingest test fixtures (#907) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add add ingest test fixtures update workflow for python 3.10, update example script with dummy values * change workflow name to avoid confusion * change workflow name to avoid confusion * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions, remove 3.10 workflow for the test fixtures update * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions * Update ingest test fixtures (#911) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * revert back the test python version matrix * recompile dependencies * modifications for shellcheck * update changelog and version * changelog and version * remove comments * Update ingest test fixtures (#915) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add the option to state the number of spaces to be fetched * add scroll functionality, expose --confluence-num-of-spaces, --confluence-list-of-spaces and --confluence-num-of-docs-from-each-space to users * add help message * add docstrings for two tests, validate grabbing every doc in the fetched spaces, count number of files instead of diffing for confluence2 test * change test names * rename connector arg Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * change arg name for connector Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * add comment to example * change arg names * add new tests to ingest test * shellcheck remove redundant statement * Update ingest test fixtures (#932) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * Update ingest test fixtures (#936) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * linting * change file extensions to parse as html * Update ingest test fixtures (#943) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * remove old fixtures * update version to 0.8.2-dev3 * change file to trigger CI * change file to trigger CI * change file to trigger CI * change file to trigger CI --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
27 lines
823 B
Bash
27 lines
823 B
Bash
#!/usr/bin/env bash
|
|
|
|
# Description: Validate that the number of directories in the output directory is as expected.
|
|
#
|
|
# Arguments:
|
|
# - $1: The expected number of directories in the output directory.
|
|
# - $2: Name of the output folder. This is used to determine the structured output path.
|
|
|
|
set +e
|
|
|
|
EXPECTED_NUM_DIRS=$1
|
|
OUTPUT_FOLDER_NAME=$2
|
|
SCRIPT_DIR=$(dirname "$(realpath "$0")")
|
|
OUTPUT_DIR=$SCRIPT_DIR/structured-output/$OUTPUT_FOLDER_NAME
|
|
|
|
|
|
NUMBER_OF_FOUND_DIRS="$(find "$OUTPUT_DIR" -type d -exec printf '.' \; | wc -c | xargs)"
|
|
|
|
if [ "$NUMBER_OF_FOUND_DIRS" != "$EXPECTED_NUM_DIRS" ]; then
|
|
echo
|
|
echo "$EXPECTED_NUM_DIRS directories were expected to be found."
|
|
echo "$NUMBER_OF_FOUND_DIRS directories were found instead."
|
|
echo "Name of the directories found:"
|
|
find "$OUTPUT_DIR" -type d
|
|
exit 1
|
|
fi
|