unstructured/examples/ingest/azure_cognitive_search/ingest.sh

#!/usr/bin/env bash

SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
cd "$SCRIPT_DIR"/../../.. || exit 1

PYTHONPATH=. ./unstructured/ingest/main.py \
  s3 \
  --remote-url s3://utic-dev-tech-fixtures/small-pdf-set/ \
  --anonymous \
  --output-dir s3-small-batch-output-to-azure \
  --num-processes 2 \
  --verbose \
  --strategy fast \
  azure-cognitive-search \
  --key "$AZURE_SEARCH_API_KEY" \
  --endpoint "$AZURE_SEARCH_ENDPOINT" \
  --index utic-test-ingest-fixtures-output
feat: adds Azure Cognitive Search (full text) destination connector (#1459) ### Description New [Azure Cognitive Search](https://azure.microsoft.com/en-us/products/ai-services/cognitive-search) destination connector added. Writes each json element from the created json files via partition and writes that content to an index. Bonus bug fix: Due to a recent change where the default version of python used in the repo was bumped to `3.10` from `3.8`, this means running `pip-compile` now runs it against that version rather than the lowest we support which is still `3.8`. This breaks the setup for those lower versions because some of the versions pulled in by `pip-compile` exist for `3.10` but not `3.8`. `pip-compile` was updates to run as a script that checks the version of python being used first, which helps guarantee that all dependencies meet the minimum python version requirement. Closes out https://github.com/Unstructured-IO/unstructured/issues/1466 2023-09-25 10:27:42 -04:00			`#!/usr/bin/env bash`

chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called. 2023-12-11 20:04:15 -05:00			`SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)`
feat: adds Azure Cognitive Search (full text) destination connector (#1459) ### Description New [Azure Cognitive Search](https://azure.microsoft.com/en-us/products/ai-services/cognitive-search) destination connector added. Writes each json element from the created json files via partition and writes that content to an index. Bonus bug fix: Due to a recent change where the default version of python used in the repo was bumped to `3.10` from `3.8`, this means running `pip-compile` now runs it against that version rather than the lowest we support which is still `3.8`. This breaks the setup for those lower versions because some of the versions pulled in by `pip-compile` exist for `3.10` but not `3.8`. `pip-compile` was updates to run as a script that checks the version of python being used first, which helps guarantee that all dependencies meet the minimum python version requirement. Closes out https://github.com/Unstructured-IO/unstructured/issues/1466 2023-09-25 10:27:42 -04:00			`cd "$SCRIPT_DIR"/../../.. \|\| exit 1`

			`PYTHONPATH=. ./unstructured/ingest/main.py \`
chore: shell scripts default indent of 2 instead of 4 (#2287) Given the tendency for shell scripts to easily enter into a few levels of indentation and long line lengths, update the default to 2 spaces. 2023-12-18 23:48:21 -08:00			`s3 \`
			`--remote-url s3://utic-dev-tech-fixtures/small-pdf-set/ \`
			`--anonymous \`
			`--output-dir s3-small-batch-output-to-azure \`
			`--num-processes 2 \`
			`--verbose \`
			`--strategy fast \`
			`azure-cognitive-search \`
			`--key "$AZURE_SEARCH_API_KEY" \`
			`--endpoint "$AZURE_SEARCH_ENDPOINT" \`
			`--index utic-test-ingest-fixtures-output`