unstructured/examples/ingest/discord/ingest.sh

#!/usr/bin/env bash

# Ingests a discord text channel into a file.

# Structured outputs are stored in discord-example/

SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
cd "$SCRIPT_DIR"/../../.. || exit 1

PYTHONPATH=. ./unstructured/ingest/main.py \
  discord \
  --channels 12345678 \
  --token "$DISCORD_TOKEN" \
  --download-dir discord-ingest-download \
  --output-dir discord-example \
  --preserve-downloads \
  --verbose
Feat: Discord connector (#515) * Initial commit of discord connector based off of initial work by @tnachen with modifications https://github.com/tnachen/unstructured/tree/tnachen/discord_connector * Add test file change format of imports * working version of the connector More work to be done to tidy it up and add any additional options * add to test fixtures update * fix spacing * tests working, switching to bot testing channel * add additional channel add reprocess to tests * add try clause to allow for exit on error Update changelog and bump version * add updated expected output filtes * add logic to check if —discord-period is an integer Add more to option description * fix lint error * Update discord reqs * PR feedback * add newline * another newline --------- Co-authored-by: Justin Bossert <packerbacker21@hotmail.com> 2023-05-16 11:46:30 -07:00			`#!/usr/bin/env bash`

			`# Ingests a discord text channel into a file.`

			`# Structured outputs are stored in discord-example/`

chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called. 2023-12-11 20:04:15 -05:00			`SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)`
Feat: Discord connector (#515) * Initial commit of discord connector based off of initial work by @tnachen with modifications https://github.com/tnachen/unstructured/tree/tnachen/discord_connector * Add test file change format of imports * working version of the connector More work to be done to tidy it up and add any additional options * add to test fixtures update * fix spacing * tests working, switching to bot testing channel * add additional channel add reprocess to tests * add try clause to allow for exit on error Update changelog and bump version * add updated expected output filtes * add logic to check if —discord-period is an integer Add more to option description * fix lint error * Update discord reqs * PR feedback * add newline * another newline --------- Co-authored-by: Justin Bossert <packerbacker21@hotmail.com> 2023-05-16 11:46:30 -07:00			`cd "$SCRIPT_DIR"/../../.. \|\| exit 1`

			`PYTHONPATH=. ./unstructured/ingest/main.py \`
chore: shell scripts default indent of 2 instead of 4 (#2287) Given the tendency for shell scripts to easily enter into a few levels of indentation and long line lengths, update the default to 2 spaces. 2023-12-18 23:48:21 -08:00			`discord \`
			`--channels 12345678 \`
			`--token "$DISCORD_TOKEN" \`
			`--download-dir discord-ingest-download \`
			`--output-dir discord-example \`
			`--preserve-downloads \`
			`--verbose`