mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-08-18 13:45:45 +00:00
627f78c16f
* add the first version of airtable connector * change imports as inline to fail gracefully in case of lacking dependency * parse tables as csv rather than plain text * add relevant logic to be able to use --airtable-list-of-paths * add script for creation of reseources for testing, add test script (large) for testing with a large number of tables to validate scroll functionality, update test script (diff) based on the new settings * fix ingest test names * add scripts for the large table test * remove large table test from diff test * make base and table ids explicit * add and remove comments * use -ne instead of != * update code based on the recent ingest refactor, update changelog and version * shellcheck fix * update comments * update check-num-rows-and-columns-output error message Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * update help comments * update help comments * update help comments * update workflows to set auth tokens and to run make install * add comments on create_scale_test_components * separate component ids from the test script, add comments to document test component creation * add LARGE_BASE test, implement LARGE_BASE component creation, replace component id * shellcheck fixes * shellcheck fixes * update docs * update comment * bump version * add wrongly deleted file * sort columns before saving to process * Update ingest test fixtures (#1098) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
23 lines
662 B
Bash
23 lines
662 B
Bash
#!/usr/bin/env bash
|
|
|
|
# Description: Validate that the number of rows in the output dataframe is as expected.
|
|
#
|
|
# Arguments:
|
|
# - $1: The expected number of rows in the dataframe.
|
|
# - $2: The path for the structured output file.
|
|
|
|
SCRIPT_PATH="scripts/airtable-test-helpers/print_num_rows_df.py"
|
|
|
|
EXPECTED_ROWS=$1
|
|
OUTPUT_FILE_NAME=$2
|
|
|
|
# Run the Python script and capture its output
|
|
ROWS=$(python "$SCRIPT_PATH" --structured-output-file-path "$OUTPUT_FILE_NAME")
|
|
|
|
# Compare the actual output with the expected output
|
|
if [[ $ROWS -ne $EXPECTED_ROWS ]]; then
|
|
echo
|
|
echo "ERROR: $ROWS rows created. $EXPECTED_ROWS rows should have been created."
|
|
exit 1
|
|
fi
|