yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-09-10 17:17:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/test_unstructured_ingest/src/local-single-file-with-encoding.sh

48 lines
1.4 KiB
Bash
Raw Normal View History

chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
#!/usr/bin/env bash
set -e
separate ingest tests (#1951) ### Description This splits the source ingest tests from the destination ingest tests since they share a different pattern: * src tests pull data from a source and compare the partitioned content to the expected results * destingation tests leverage the local connector to produce results to push to a destination and leverages overhead to create temporary locations at those destinations to write to and delete when done. Only the src tests create partitioned content that needs to be checked so the update ingest test CI job only needs to run these.
2023-11-01 15:23:44 -04:00
SRC_PATH=$(dirname "$(realpath "$0")")
SCRIPT_DIR=$(dirname "$SRC_PATH")
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
cd "$SCRIPT_DIR"/.. || exit 1
OUTPUT_FOLDER_NAME=local-single-file-with-encoding
test: parametrize ingest test scripts (#1979) This PR resolves [CORE-2453](https://unstructured-ai.atlassian.net/browse/CORE-2453): - parametrizes the output folder so that ingest output files can be saved other than the same place where the scripts are; this is set by env `OUTPUT_ROOT` - parametrize the python path `PYTHONPATH` to first check existing definition before default to `.`, the current folder - parametrize the run script that carries out ingest using `RUN_SCRIPT`, default is still `./unstructured/ingest/main.py` These changes allows us to run ingest test with more control. To test: - run `OUTPUT_ROOT=/tmp ./test_unstructured_ingest/src/local-single-file.sh`: the output now should be in `/tmp` instead of in the ingest test folder - run `RUN_SCRIPT=/hope/you/do/not/have/this/folder ./test_unstructured_ingest/src/local-single-file.sh` would raise an error because system can't find `/hope/you/do/not/have/this/folder` - run `RUN_SCRIPT=./unstructured/ingest/main.py ./test_unstructured_ingest/src/local-single-file.sh` should run as normal - do the following ```bash cp ./unstructured/ingest/main.py /tmp/main.py OUTPUT_ROOT=/tmp PYTHONPATH=$(pwd) RUN_SCRIPT=./unstructured/ingest/main.py ./test_unstructured_ingest/src/local-single-file.sh ``` This will run and generate output at `/tmp` [CORE-2453]: https://unstructured-ai.atlassian.net/browse/CORE-2453?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
2023-11-02 16:41:56 -05:00
OUTPUT_ROOT=${OUTPUT_ROOT:-$SCRIPT_DIR}
OUTPUT_DIR=$OUTPUT_ROOT/structured-output/$OUTPUT_FOLDER_NAME
WORK_DIR=$OUTPUT_ROOT/workdir/$OUTPUT_FOLDER_NAME
build(image): call python3 not python for image compat (#1574) Fixes docker exec unstructured-smoke-test /bin/bash -c /home/notebook-user/test_unstructured_ingest/test-ingest-wikipedia.sh /home/notebook-user/test_unstructured_ingest/test-ingest-wikipedia.sh: line 10: python: command not found in https://github.com/Unstructured-IO/unstructured/blob/6ad4971/scripts/docker-smoke-test.sh#L43 that was preventing docker images from being built.
2023-09-28 21:48:19 -07:00
max_processes=${MAX_PROCESSES:=$(python3 -c "import os; print(os.cpu_count())")}
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
chore: ingest test file cleanup (#1366)
2023-09-21 14:51:08 -04:00
# shellcheck disable=SC1091
source "$SCRIPT_DIR"/cleanup.sh
chore: for ingest-test metrics, also check subdirs (#2079) - Copy script only went through one layer of subdirectory so it did not found the match between manifest file and structured output. Now edited to search all subdirectories. - `set -e` causes the script to exit at any exit rather than `exit 0`, fix all scripts that needs to run the copy script to be `set +e` right before the check diff, then back to `set -e` after - Edit the default evaluation metrics output from `metrics` to `metrics-tmp` to account for diff check - Add a script that checks the differences between old eval metric output (metrics) and new eval metrics output (metrics-tmp)
2023-11-16 00:02:43 -05:00
# shellcheck disable=SC2317
refactor: unstructured ingest as a pipeline (#1551) ### Description As we add more and more steps to the pipeline (i.e. chunking, embedding, table manipulation), it would help seperate the responsibility of each of these into their own processes, running each in parallel using json files to share data across. This will also help guarantee data is serializable if this code was used in an actual pipeline. Following is a flow diagram of the proposed changes. As part of this change: * A parent pipeline class will be responsible for running each `node`, which can optionally be run via multiprocessing if it supports it, or not. Possible nodes at this moment: * Doc factory: creates all the ingest docs via the source connector * Source: reads/downloads all of the content to process to the local filesystem to the location set by the `download_dir` parameter. * Partition: runs partition on all of the downloaded content in json format. * Any number of reformat nodes that modify the partitioned content. This can include chunking, embedding, etc. * Write: push the final json into the destination via the destination connector * This pipeline relies on the information of the ingest docs to be available via their serialization. An optimization was introduced with the `IngestDocJsonMixin` which adds in all the `@property` fields to the serialized json already being created via the `DataClassJsonMixin` * For all intermediate steps (partitioning, reformatting), the content is saved to a dedicated location on the local filesystem. Right now it's set to `$HOME/.cache/unstructured/ingest/pipeline/STEP_NAME/`. * Minor changes: made sense to move some of the config parameters between the read and partition configs when I explicitly divided the responsibility to download vs partition the content in the pipeline. * The pipeline class only makes the doc factory, source and partition nodes required, keeping with the logic that has been supported so far. All reformatting nodes and write node are optional. * Long term, there should also be some changes to the base configs supported by the CLI to support pipeline specific configs, but for now what exists was used to minimize changes in this PR. * Final step to copy the final output to the location designated by the `_output_filename` value of the ingest doc. * Hashing occurs at each step by hashing the parameters of that step (i.e. partition configs) along with the previous step via the filename used. This allows each step to be the same _if_ all the parameters for it have not changed and the content so far is the same. * The only data that is shared and has writes to across processes is the dictionary of ingest json data. This dict is created using the `multiprocessing.manager.DictProxy` to make sure any interaction with it is behind a lock. ### Minor refactors included: * Utility methods added to extract configs from the click options * Utility method to add common options to click commands. * All writers moved to using the class approach which extracts a lot of the common code so there's less copy-paste when new runners are added. * Use `@property` for source metadata on base ingest doc to add logic to call `update_source_metadata` if it's still `None` at the time it's fetched. ### Additional bug fixes included * Fsspec connectors were not serializable due to the `ingest_doc_cls`. This was removed from the fields captured by the `@dataclass` decorator and added in a `__post_init__` method. * Various reddit connector params were missing. This doesn't have an explicit ingest test at the moment so was never caught. * Fsspec connector had the parent `update_source_metadata` misnamed as `update_source_metadata_metadata` so it was never being called. ### Flow Diagram ![ingest_pipeline](https://github.com/Unstructured-IO/unstructured/assets/136338424/be485606-cfe0-4931-8b81-c2bf569cf1e2)
2023-10-06 14:49:29 -04:00
function cleanup() {
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called.
2023-12-11 20:04:15 -05:00
cleanup_dir "$OUTPUT_DIR"
cleanup_dir "$WORK_DIR"
refactor: unstructured ingest as a pipeline (#1551) ### Description As we add more and more steps to the pipeline (i.e. chunking, embedding, table manipulation), it would help seperate the responsibility of each of these into their own processes, running each in parallel using json files to share data across. This will also help guarantee data is serializable if this code was used in an actual pipeline. Following is a flow diagram of the proposed changes. As part of this change: * A parent pipeline class will be responsible for running each `node`, which can optionally be run via multiprocessing if it supports it, or not. Possible nodes at this moment: * Doc factory: creates all the ingest docs via the source connector * Source: reads/downloads all of the content to process to the local filesystem to the location set by the `download_dir` parameter. * Partition: runs partition on all of the downloaded content in json format. * Any number of reformat nodes that modify the partitioned content. This can include chunking, embedding, etc. * Write: push the final json into the destination via the destination connector * This pipeline relies on the information of the ingest docs to be available via their serialization. An optimization was introduced with the `IngestDocJsonMixin` which adds in all the `@property` fields to the serialized json already being created via the `DataClassJsonMixin` * For all intermediate steps (partitioning, reformatting), the content is saved to a dedicated location on the local filesystem. Right now it's set to `$HOME/.cache/unstructured/ingest/pipeline/STEP_NAME/`. * Minor changes: made sense to move some of the config parameters between the read and partition configs when I explicitly divided the responsibility to download vs partition the content in the pipeline. * The pipeline class only makes the doc factory, source and partition nodes required, keeping with the logic that has been supported so far. All reformatting nodes and write node are optional. * Long term, there should also be some changes to the base configs supported by the CLI to support pipeline specific configs, but for now what exists was used to minimize changes in this PR. * Final step to copy the final output to the location designated by the `_output_filename` value of the ingest doc. * Hashing occurs at each step by hashing the parameters of that step (i.e. partition configs) along with the previous step via the filename used. This allows each step to be the same _if_ all the parameters for it have not changed and the content so far is the same. * The only data that is shared and has writes to across processes is the dictionary of ingest json data. This dict is created using the `multiprocessing.manager.DictProxy` to make sure any interaction with it is behind a lock. ### Minor refactors included: * Utility methods added to extract configs from the click options * Utility method to add common options to click commands. * All writers moved to using the class approach which extracts a lot of the common code so there's less copy-paste when new runners are added. * Use `@property` for source metadata on base ingest doc to add logic to call `update_source_metadata` if it's still `None` at the time it's fetched. ### Additional bug fixes included * Fsspec connectors were not serializable due to the `ingest_doc_cls`. This was removed from the fields captured by the `@dataclass` decorator and added in a `__post_init__` method. * Various reddit connector params were missing. This doesn't have an explicit ingest test at the moment so was never caught. * Fsspec connector had the parent `update_source_metadata` misnamed as `update_source_metadata_metadata` so it was never being called. ### Flow Diagram ![ingest_pipeline](https://github.com/Unstructured-IO/unstructured/assets/136338424/be485606-cfe0-4931-8b81-c2bf569cf1e2)
2023-10-06 14:49:29 -04:00
}
trap cleanup EXIT
chore: ingest test file cleanup (#1366)
2023-09-21 14:51:08 -04:00
test: parametrize ingest test scripts (#1979) This PR resolves [CORE-2453](https://unstructured-ai.atlassian.net/browse/CORE-2453): - parametrizes the output folder so that ingest output files can be saved other than the same place where the scripts are; this is set by env `OUTPUT_ROOT` - parametrize the python path `PYTHONPATH` to first check existing definition before default to `.`, the current folder - parametrize the run script that carries out ingest using `RUN_SCRIPT`, default is still `./unstructured/ingest/main.py` These changes allows us to run ingest test with more control. To test: - run `OUTPUT_ROOT=/tmp ./test_unstructured_ingest/src/local-single-file.sh`: the output now should be in `/tmp` instead of in the ingest test folder - run `RUN_SCRIPT=/hope/you/do/not/have/this/folder ./test_unstructured_ingest/src/local-single-file.sh` would raise an error because system can't find `/hope/you/do/not/have/this/folder` - run `RUN_SCRIPT=./unstructured/ingest/main.py ./test_unstructured_ingest/src/local-single-file.sh` should run as normal - do the following ```bash cp ./unstructured/ingest/main.py /tmp/main.py OUTPUT_ROOT=/tmp PYTHONPATH=$(pwd) RUN_SCRIPT=./unstructured/ingest/main.py ./test_unstructured_ingest/src/local-single-file.sh ``` This will run and generate output at `/tmp` [CORE-2453]: https://unstructured-ai.atlassian.net/browse/CORE-2453?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
2023-11-02 16:41:56 -05:00
RUN_SCRIPT=${RUN_SCRIPT:-./unstructured/ingest/main.py}
PYTHONPATH=${PYTHONPATH:-.} "$RUN_SCRIPT" \
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called.
2023-12-11 20:04:15 -05:00
local \
--num-processes "$max_processes" \
--metadata-exclude filename,file_directory,metadata.data_source.date_created,metadata.data_source.date_modified,metadata.data_source.date_processed,metadata.last_modified,metadata.detection_class_prob,metadata.parent_id,metadata.category_depth \
--output-dir "$OUTPUT_DIR" \
--encoding cp1252 \
--verbose \
--reprocess \
--input-path example-docs/fake-html-cp1252.html \
--work-dir "$WORK_DIR"
chore: Add encoding param to ingest (#955) * Add encoding param to ingest
2023-07-24 10:06:13 -07:00
set +e
chore: ingest test file cleanup (#1366)
2023-09-21 14:51:08 -04:00
"$SCRIPT_DIR"/check-diff-expected-output.sh $OUTPUT_FOLDER_NAME
chore: for ingest-test metrics, also check subdirs (#2079) - Copy script only went through one layer of subdirectory so it did not found the match between manifest file and structured output. Now edited to search all subdirectories. - `set -e` causes the script to exit at any exit rather than `exit 0`, fix all scripts that needs to run the copy script to be `set +e` right before the check diff, then back to `set -e` after - Edit the default evaluation metrics output from `metrics` to `metrics-tmp` to account for diff check - Add a script that checks the differences between old eval metric output (metrics) and new eval metrics output (metrics-tmp)
2023-11-16 00:02:43 -05:00
EXIT_CODE=$?
set -e
if [ "$EXIT_CODE" -ne 0 ]; then
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called.
2023-12-11 20:04:15 -05:00
echo "The last script run exited with a non-zero exit code: $EXIT_CODE."
# Handle the error or exit
chore: for ingest-test metrics, also check subdirs (#2079) - Copy script only went through one layer of subdirectory so it did not found the match between manifest file and structured output. Now edited to search all subdirectories. - `set -e` causes the script to exit at any exit rather than `exit 0`, fix all scripts that needs to run the copy script to be `set +e` right before the check diff, then back to `set -e` after - Edit the default evaluation metrics output from `metrics` to `metrics-tmp` to account for diff check - Add a script that checks the differences between old eval metric output (metrics) and new eval metrics output (metrics-tmp)
2023-11-16 00:02:43 -05:00
fi
build: element type frequency evaluation metrics workflow in ci (#1862) **Executive Summary** Measured element type frequency accuracy from the current version of code with the expected output. The performance is reported as tsv file under `metrics`. **Technical Details** - The evaluation measures element type frequencies from `structured-output-eval` against `expected-structured-output` - `evaluation.py` has been edited to support function calling using `click.group()` and `command()` - `evaluation-ingest-cp.sh` is now added to all the `test-ingest-xx.sh` scripts **Outputs** 2 tsv files is saved ![image](https://github.com/Unstructured-IO/unstructured/assets/2177850/b4458094-a9fc-48f9-a0bd-2ccd6985440a) ![image](https://github.com/Unstructured-IO/unstructured/assets/2177850/6d785736-bcaf-4275-bf2d-ab511cdfb3f4) 9-0e05-41d4-b69f-841a2aa131ec) and aggregated score is displayed. ![image](https://github.com/Unstructured-IO/unstructured/assets/2177850/9d42bd0c-a0dd-41c2-a2e5-b675a40f35cc) --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: Klaijan <Klaijan@users.noreply.github.com> Co-authored-by: Yao You <theyaoyou@gmail.com>
2023-10-27 00:36:36 -04:00
"$SCRIPT_DIR"/evaluation-ingest-cp.sh "$OUTPUT_DIR" "$OUTPUT_FOLDER_NAME"
chore: for ingest-test metrics, also check subdirs (#2079) - Copy script only went through one layer of subdirectory so it did not found the match between manifest file and structured output. Now edited to search all subdirectories. - `set -e` causes the script to exit at any exit rather than `exit 0`, fix all scripts that needs to run the copy script to be `set +e` right before the check diff, then back to `set -e` after - Edit the default evaluation metrics output from `metrics` to `metrics-tmp` to account for diff check - Add a script that checks the differences between old eval metric output (metrics) and new eval metrics output (metrics-tmp)
2023-11-16 00:02:43 -05:00
exit $EXIT_CODE
Reference in New Issue Copy Permalink