unstructured/scripts/sftp-test-helpers/create-and-check-sftp.sh

#!/usr/bin/env bash

SCRIPT_DIR=$(dirname "$(realpath "$0")")

# Upload nested folder path to sftp server
function upload() {
  docker cp "$SCRIPT_DIR"/folder1/ sftp-test:/home/foo/upload/
}

# Create sftp server
docker compose version
docker compose -f "$SCRIPT_DIR"/docker-compose.yaml up --wait
docker compose -f "$SCRIPT_DIR"/docker-compose.yaml ps

echo "Cluster is live."
upload
feat: Add sftp source connector (#2163) Adds source connector for SFTP which uses fsspec and paramiko via fsspec. Paramiko is the standard sftp package for python used in pysftp etc... ``` --username foo \ --password bar \ --remote-url sftp://localhost:47474/upload/ ``` Will only download a specifically requested file if it has an extension. (i.e. `--remote-url sftp://localhost:47474/upload/bob.zip`) It will treat any other remote_url as a folder path. This is intentional. --------- Co-authored-by: potter-potter <david.potter@gmail.com> 2023-12-07 11:33:19 -08:00			`#!/usr/bin/env bash`

			`SCRIPT_DIR=$(dirname "$(realpath "$0")")`

			`# Upload nested folder path to sftp server`
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called. 2023-12-11 20:04:15 -05:00			`function upload() {`
chore: shell scripts default indent of 2 instead of 4 (#2287) Given the tendency for shell scripts to easily enter into a few levels of indentation and long line lengths, update the default to 2 spaces. 2023-12-18 23:48:21 -08:00			`docker cp "$SCRIPT_DIR"/folder1/ sftp-test:/home/foo/upload/`
feat: Add sftp source connector (#2163) Adds source connector for SFTP which uses fsspec and paramiko via fsspec. Paramiko is the standard sftp package for python used in pysftp etc... ``` --username foo \ --password bar \ --remote-url sftp://localhost:47474/upload/ ``` Will only download a specifically requested file if it has an extension. (i.e. `--remote-url sftp://localhost:47474/upload/bob.zip`) It will treat any other remote_url as a folder path. This is intentional. --------- Co-authored-by: potter-potter <david.potter@gmail.com> 2023-12-07 11:33:19 -08:00			`}`

			`# Create sftp server`
			`docker compose version`
			`docker compose -f "$SCRIPT_DIR"/docker-compose.yaml up --wait`
			`docker compose -f "$SCRIPT_DIR"/docker-compose.yaml ps`

			`echo "Cluster is live."`
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called. 2023-12-11 20:04:15 -05:00			`upload`