yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-06-27 02:30:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
unstructured/Makefile

494 lines
14 KiB
Makefile
Raw Normal View History

Initial Release
2022-06-29 14:35:19 -04:00
PACKAGE_NAME := unstructured
Update pip in makefile (#981) * update pip in makefile * merge and update requirements * update version * update outlook requirements
2023-07-27 14:38:51 -07:00
PIP_VERSION := 23.2.1
chore: add tests for docker (#373)
2023-03-21 13:46:09 -07:00
CURRENT_DIR := $(shell pwd)
Fixes issue where detectron2 would not install on OSX (#552) * Fixes issue where detectron2 would not install on OSX Tested on Apple silicon based MacBook Pro. This installs tensorboard which is required on OSX and arm based cpu’s for detectron2. * Improve Arch detection for tensorboard * remove makefile from commands in readme pin tensorboard version
2023-05-05 17:16:28 -07:00
ARCH := $(shell uname -m)
Initial Release
2022-06-29 14:35:19 -04:00
.PHONY: help
help: Makefile
@sed -n 's/^\(## \)\([a-zA-Z]\)/\2/p' $<
###########
# Install #
###########
## install-base: installs core requirements needed for text processing bricks
.PHONY: install-base
install-base: install-base-pip-packages install-nltk-models
## install: installs all test, dev, and experimental requirements
.PHONY: install
build(deps): split up dependencies by document type (#986) * split dependencies by document type * make pip-compile with new requirements * add extra requirements to setup.py * add in all docs; re pip-compile * extra for all docs * add pandas to xlsx * dependency requires for tsv and csv * handling for doc, docx and odt * dependency check for pypandoc * required dependencies for pandoc files * xml and html * markdown * msg * add in pdf * add in pptx * add in excel * add lxml as base req * extra all docs for local inference * local inference installs all * pin pillow version * fixes for plain text tests * fixes for doc * update make commands * changelog and version * add xlrd * update pip-compile * pin numpy for python 3.8 support * more constraints * contraint on scipy * update install docs * constrain ipython * add outlook to pip-compile * more ipython constraints * add extras to dockerfile * pin office365 client * few doc tweaks * types as strings * last pip-compile * re pip-comple * make tidy * make tidy
2023-08-01 11:31:13 -04:00
install: install-base-pip-packages install-dev install-nltk-models install-test install-huggingface install-all-docs
Initial Release
2022-06-29 14:35:19 -04:00
.PHONY: install-ci
fix: fix ingest paddle hanging issue (#1441) ## Summary Ingest tests are having paddle OOM issue which cause the tests to hang forever. The fix here is to remove paddle from ci and set both OCR env `TABLE_OCR` and `ENTIRE_PAGE_OCR` to `tesseract`. (will have follow up PR to investigate why this is failing) ## Test please check ingest tests in CI
2023-09-19 13:20:23 -04:00
install-ci: install-base-pip-packages install-nltk-models install-huggingface install-all-docs install-test
Initial Release
2022-06-29 14:35:19 -04:00
test: create CI pipelines for verifying base and extras pass respective tests (#1137) **Summary** Closes #747 * Create CI Pipeline for running text, xml, email, and html doc tests against the library installed without extras * Create CI Pipeline for running each library extra against their respective tests
2023-08-19 12:56:13 -04:00
.PHONY: install-base-ci
install-base-ci: install-base-pip-packages install-nltk-models install-test
Initial Release
2022-06-29 14:35:19 -04:00
.PHONY: install-base-pip-packages
install-base-pip-packages:
python3 -m pip install pip==${PIP_VERSION}
Remove extra index url from docs (#1711) It’s no longer required to specify the extra index url as we utilize a different method of gathering install anonymous analytics.
2023-10-11 12:34:49 -07:00
python3 -m pip install -r requirements/base.txt
Initial Release
2022-06-29 14:35:19 -04:00
feat: Staging brick for attention window chunking (#34) * add huggingface dependencies and re pip-compile * first pass on chunk by attention window * test for chunking function * completed tests for chunk_by_attention_window * change default buffer size to 2 * wrapper function for staging * added docs for transformers * fix wording and typos * updated change log and bumped the version * added docs on huggingface dependencies * fix typo * re pip-compile
2022-10-13 11:18:27 -04:00
.PHONY: install-huggingface
install-huggingface:
python3 -m pip install pip==${PIP_VERSION}
build: script to update all ingest fixtures, add azure ingest fixtures (#367) - Updates CI to install tesseract version 5.3.0 (better than 4.x in various ways incl. perf.). - Adds azure expected output fixtures for more useful reference points and as a repro for Some PDF's with scanned images return empty elements #346 . - Adds a script to regenerate ingest test fixtures that is run in an ubuntu docker container (like CI), with the same version of tesseract. See the comments in scripts/ingest-test-fixtures-update.sh for details. - Updates expected outputs with above script. - Updates individual test-ingest scripts to update expected .json output if OVERWRITE_FIXTURES=true.
2023-04-11 00:11:50 -07:00
python3 -m pip install -r requirements/huggingface.txt
feat: Staging brick for attention window chunking (#34) * add huggingface dependencies and re pip-compile * first pass on chunk by attention window * test for chunking function * completed tests for chunk_by_attention_window * change default buffer size to 2 * wrapper function for staging * added docs for transformers * fix wording and typos * updated change log and bumped the version * added docs on huggingface dependencies * fix typo * re pip-compile
2022-10-13 11:18:27 -04:00
bug: fix typo in makefile (#2474) fix typo in makefile: `.PHONE` -> `.PHONY`
2024-01-30 12:12:35 -06:00
.PHONY: install-nltk-models
Initial Release
2022-06-29 14:35:19 -04:00
install-nltk-models:
python -c "import nltk; nltk.download('punkt')"
python -c "import nltk; nltk.download('averaged_perceptron_tagger')"
.PHONY: install-test
install-test:
build: script to update all ingest fixtures, add azure ingest fixtures (#367) - Updates CI to install tesseract version 5.3.0 (better than 4.x in various ways incl. perf.). - Adds azure expected output fixtures for more useful reference points and as a repro for Some PDF's with scanned images return empty elements #346 . - Adds a script to regenerate ingest test fixtures that is run in an ubuntu docker container (like CI), with the same version of tesseract. See the comments in scripts/ingest-test-fixtures-update.sh for details. - Updates expected outputs with above script. - Updates individual test-ingest scripts to update expected .json output if OVERWRITE_FIXTURES=true.
2023-04-11 00:11:50 -07:00
python3 -m pip install -r requirements/test.txt
[CORE-1741] use forked pytesseract to reduce calls to tesseract (#1298) This PR resolves [CORE-1741](https://unstructured-ai.atlassian.net/browse/CORE-1741) by using a new function `pytesseract.run_and_get_multiple_output`, see forked repo for more details: https://github.com/Unstructured-IO/unstructured.pytesseract/releases/tag/0.3.11-dev1 This reduces the call to `tesseract` by half per page of PDF/image during partition, roughly reducing the runtime by 48%. The new function is in forked `unstructured.pytesseract`. A PR has been made to the upstream repo and once that is merged we should switch to the up stream version. For now we add a new dependency: `unstructured.pytesseract`. ## testing Existing unit tests should serve as tests to the new function. To demonstrate the changes in performance: - checkout main - run `./scripts/performance/profile.sh` and select `ocr_only` strategy, using the 10th document (16 page layout paper in pdf format) - examine the speedscope profile or time profile in flamegraph -> should see two dominant time spenders are `pytesseract.image_to_text` and `pytesseract.image_to_boxes`, with both about the same total time (see attached first image) - checkout this branch - run the same `profile.sh` with the same options - examine the profile again and this time should notice 1) total runtime is reduced by more than 40%; 2) only `unstructured_pytesseract.run_and_get_multiple_output` is the top time spender and its total time is about the same as either the `pytesseract.image_to_text` or `pytesseract.image_to_boxes` time (see second image below) ![Screenshot 2023-09-06 at 9 45 10 AM](https://github.com/Unstructured-IO/unstructured/assets/647930/fed6118b-a0dc-493d-bef8-85d73027c968) ![Screenshot 2023-09-06 at 9 46 37 AM](https://github.com/Unstructured-IO/unstructured/assets/647930/dd1d6369-cfba-43d4-b1c6-87a8a98b2e16) [CORE-1741]: https://unstructured-ai.atlassian.net/browse/CORE-1741?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Benjamin Torres <benjats07@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io>
2023-09-14 18:27:18 -05:00
# NOTE(yao) - CI seem to always install tesseract to test so it would make sense to also require
# pytesseract installation into the virtual env for testing
python3 -m pip install unstructured.pytesseract -c requirements/constraints.in
python3 -m pip install argilla -c requirements/constraints.in
feat: Add `stage_for_weaviate` and schema creation function (#672) * add weaviate docker compose * added staging brick and tests for weaviate * initial notebook and requirements file * add commentary to weaviate notebook * weaviate readme * update docs * version and change log * install weaviate client * install weaviate; skip for docker * linting, linting, linting * install weaviate client with deps * comments on weaviate client * fix module not found error for docker container * skipped wrong test in docker * fix typos * add in local-inference
2023-06-01 16:48:54 -04:00
# NOTE(robinson) - Installing weaviate-client separately here because the requests
# version conflicts with label_studio_sdk
update constraints to pin weaviate during ci (#1408) This PR ensures the version for `weaviate` is consistent in CI testing. Latest (3.24.1) is not compatible with our test needs and last version that run successfully in CI is 3.23.2.
2023-09-13 18:19:20 -05:00
python3 -m pip install weaviate-client -c requirements/constraints.in
# TODO (yao): find out if how to constrain argilla properly without causing conflicts
Roman/downstream connector cli subcommand (#1302) ### Description Update all other connectors to use the new downstream architecture that was recently introduced for the s3 connector. Closes #1313 and #1311
2023-09-11 11:40:56 -04:00
python3 -m pip install argilla
Initial Release
2022-06-29 14:35:19 -04:00
.PHONY: install-dev
install-dev:
build: script to update all ingest fixtures, add azure ingest fixtures (#367) - Updates CI to install tesseract version 5.3.0 (better than 4.x in various ways incl. perf.). - Adds azure expected output fixtures for more useful reference points and as a repro for Some PDF's with scanned images return empty elements #346 . - Adds a script to regenerate ingest test fixtures that is run in an ubuntu docker container (like CI), with the same version of tesseract. See the comments in scripts/ingest-test-fixtures-update.sh for details. - Updates expected outputs with above script. - Updates individual test-ingest scripts to update expected .json output if OVERWRITE_FIXTURES=true.
2023-04-11 00:11:50 -07:00
python3 -m pip install -r requirements/dev.txt
Initial Release
2022-06-29 14:35:19 -04:00
.PHONY: install-build
install-build:
build: script to update all ingest fixtures, add azure ingest fixtures (#367) - Updates CI to install tesseract version 5.3.0 (better than 4.x in various ways incl. perf.). - Adds azure expected output fixtures for more useful reference points and as a repro for Some PDF's with scanned images return empty elements #346 . - Adds a script to regenerate ingest test fixtures that is run in an ubuntu docker container (like CI), with the same version of tesseract. See the comments in scripts/ingest-test-fixtures-update.sh for details. - Updates expected outputs with above script. - Updates individual test-ingest scripts to update expected .json output if OVERWRITE_FIXTURES=true.
2023-04-11 00:11:50 -07:00
python3 -m pip install -r requirements/build.txt
Initial Release
2022-06-29 14:35:19 -04:00
build(deps): split up dependencies by document type (#986) * split dependencies by document type * make pip-compile with new requirements * add extra requirements to setup.py * add in all docs; re pip-compile * extra for all docs * add pandas to xlsx * dependency requires for tsv and csv * handling for doc, docx and odt * dependency check for pypandoc * required dependencies for pandoc files * xml and html * markdown * msg * add in pdf * add in pptx * add in excel * add lxml as base req * extra all docs for local inference * local inference installs all * pin pillow version * fixes for plain text tests * fixes for doc * update make commands * changelog and version * add xlrd * update pip-compile * pin numpy for python 3.8 support * more constraints * contraint on scipy * update install docs * constrain ipython * add outlook to pip-compile * more ipython constraints * add extras to dockerfile * pin office365 client * few doc tweaks * types as strings * last pip-compile * re pip-comple * make tidy * make tidy
2023-08-01 11:31:13 -04:00
.PHONY: install-csv
install-csv:
python3 -m pip install -r requirements/extra-csv.txt
.PHONY: install-docx
install-docx:
python3 -m pip install -r requirements/extra-docx.txt
Capture section and chapter in the metadata for epubs under `epub_section` (#1005) Capture section and chapter in the metadata for epubs under epub_section. Closes Github issue #459
2023-08-12 16:02:06 -05:00
.PHONY: install-epub
install-epub:
python3 -m pip install -r requirements/extra-epub.txt
build(deps): split up dependencies by document type (#986) * split dependencies by document type * make pip-compile with new requirements * add extra requirements to setup.py * add in all docs; re pip-compile * extra for all docs * add pandas to xlsx * dependency requires for tsv and csv * handling for doc, docx and odt * dependency check for pypandoc * required dependencies for pandoc files * xml and html * markdown * msg * add in pdf * add in pptx * add in excel * add lxml as base req * extra all docs for local inference * local inference installs all * pin pillow version * fixes for plain text tests * fixes for doc * update make commands * changelog and version * add xlrd * update pip-compile * pin numpy for python 3.8 support * more constraints * contraint on scipy * update install docs * constrain ipython * add outlook to pip-compile * more ipython constraints * add extras to dockerfile * pin office365 client * few doc tweaks * types as strings * last pip-compile * re pip-comple * make tidy * make tidy
2023-08-01 11:31:13 -04:00
.PHONY: install-odt
install-odt:
python3 -m pip install -r requirements/extra-odt.txt
.PHONY: install-pypandoc
install-pypandoc:
python3 -m pip install -r requirements/extra-pandoc.txt
.PHONY: install-markdown
install-markdown:
python3 -m pip install -r requirements/extra-markdown.txt
.PHONY: install-msg
install-msg:
python3 -m pip install -r requirements/extra-msg.txt
.PHONY: install-pdf-image
install-pdf-image:
python3 -m pip install -r requirements/extra-pdf-image.txt
.PHONY: install-pptx
install-pptx:
python3 -m pip install -r requirements/extra-pptx.txt
.PHONY: install-xlsx
install-xlsx:
python3 -m pip install -r requirements/extra-xlsx.txt
.PHONY: install-all-docs
Capture section and chapter in the metadata for epubs under `epub_section` (#1005) Capture section and chapter in the metadata for epubs under epub_section. Closes Github issue #459
2023-08-12 16:02:06 -05:00
install-all-docs: install-base install-csv install-docx install-epub install-odt install-pypandoc install-markdown install-msg install-pdf-image install-pptx install-xlsx
build(deps): split up dependencies by document type (#986) * split dependencies by document type * make pip-compile with new requirements * add extra requirements to setup.py * add in all docs; re pip-compile * extra for all docs * add pandas to xlsx * dependency requires for tsv and csv * handling for doc, docx and odt * dependency check for pypandoc * required dependencies for pandoc files * xml and html * markdown * msg * add in pdf * add in pptx * add in excel * add lxml as base req * extra all docs for local inference * local inference installs all * pin pillow version * fixes for plain text tests * fixes for doc * update make commands * changelog and version * add xlrd * update pip-compile * pin numpy for python 3.8 support * more constraints * contraint on scipy * update install docs * constrain ipython * add outlook to pip-compile * more ipython constraints * add extras to dockerfile * pin office365 client * few doc tweaks * types as strings * last pip-compile * re pip-comple * make tidy * make tidy
2023-08-01 11:31:13 -04:00
Roman/optimize ingest ci (#1799) ### Description Currently the CI caches the CI dependencies but uses the hash of all files in `requirements/`. This isn't completely accurate since the ingest dependencies are installed in a later step and don't affect the cached environment. As part of this PR: * ingest dependencies were isolated into their own folder in `requirements/ingest/` * A new cache setup was introduced in the CI to restore the base cache -> install ingest dependencies -> cache it with a new id * new make target created to install all ingest dependencies via `pip install -r ...` * updates to Dockerfile to use `find ...` to install all dependencies, avoiding the need to update this when new deps are added. * update to pip-compile script to run over all `*.in` files in `requirements/`
2023-10-24 10:54:00 -04:00
.PHONY: install-all-ingest
install-all-ingest:
find requirements/ingest -type f -name "*.txt" -exec python3 -m pip install -r '{}' ';'
Connector for Google Drive (#294) Implements issue #244
2023-03-07 06:01:02 +00:00
.PHONY: install-ingest-google-drive
install-ingest-google-drive:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/google-drive.txt
Connector for Google Drive (#294) Implements issue #244
2023-03-07 06:01:02 +00:00
feat: Sample ingest project with S3 connector (#218)
2023-02-14 12:27:45 -08:00
## install-ingest-s3: install requirements for the s3 connector
.PHONY: install-ingest-s3
install-ingest-s3:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/s3.txt
feat: Sample ingest project with S3 connector (#218)
2023-02-14 12:27:45 -08:00
feat: add google cloud storage connector (#746)
2023-06-21 15:14:50 -07:00
.PHONY: install-ingest-gcs
install-ingest-gcs:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/gcs.txt
feat: add google cloud storage connector (#746)
2023-06-21 15:14:50 -07:00
feat: add Dropbox connector (#844)
2023-06-30 17:08:27 -07:00
.PHONY: install-ingest-dropbox
install-ingest-dropbox:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/dropbox.txt
feat: add Dropbox connector (#844)
2023-06-30 17:08:27 -07:00
Add `AzureBlobStorageConnector` (#353) * Add `AzureBlobStorageConnector` based on its `fsspec` implementation inheriting from `FsspecConnector` * Start deprecation life cycle for `unstructured-ingest --s3-url` option, to be deprecated in favor of `--remote-url`. --------- Co-authored-by: Tom Aarsen <37621491+tomaarsen@users.noreply.github.com>
2023-03-11 00:43:40 +01:00
.PHONY: install-ingest-azure
install-ingest-azure:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/azure.txt
Add `AzureBlobStorageConnector` (#353) * Add `AzureBlobStorageConnector` based on its `fsspec` implementation inheriting from `FsspecConnector` * Start deprecation life cycle for `unstructured-ingest --s3-url` option, to be deprecated in favor of `--remote-url`. --------- Co-authored-by: Tom Aarsen <37621491+tomaarsen@users.noreply.github.com>
2023-03-11 00:43:40 +01:00
feat: adds Box connector (#996)
2023-07-31 18:10:10 -07:00
.PHONY: install-ingest-box
install-ingest-box:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/box.txt
feat: adds Box connector (#996)
2023-07-31 18:10:10 -07:00
Roman/delta table connector (#1132) ### Description Add delta table connector and test against a delta table generated via delta.io and uploaded to s3. Shows an example of how to use the connection options to leverage s3. I was able to get this to work with s3 if I pass in the access and secret keys as storage options. Even though the s3 bucket being used is public, would not work without those. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-08-22 10:19:46 -04:00
.PHONY: install-ingest-delta-table
install-ingest-delta-table:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/delta-table.txt
Roman/delta table connector (#1132) ### Description Add delta table connector and test against a delta table generated via delta.io and uploaded to s3. Shows an example of how to use the connection options to leverage s3. I was able to get this to work with s3 if I pass in the access and secret keys as storage options. Even though the s3 bucket being used is public, would not work without those. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-08-22 10:19:46 -04:00
Feat: Discord connector (#515) * Initial commit of discord connector based off of initial work by @tnachen with modifications https://github.com/tnachen/unstructured/tree/tnachen/discord_connector * Add test file change format of imports * working version of the connector More work to be done to tidy it up and add any additional options * add to test fixtures update * fix spacing * tests working, switching to bot testing channel * add additional channel add reprocess to tests * add try clause to allow for exit on error Update changelog and bump version * add updated expected output filtes * add logic to check if —discord-period is an integer Add more to option description * fix lint error * Update discord reqs * PR feedback * add newline * another newline --------- Co-authored-by: Justin Bossert <packerbacker21@hotmail.com>
2023-05-16 11:46:30 -07:00
.PHONY: install-ingest-discord
install-ingest-discord:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
pip install -r requirements/ingest/discord.txt
Feat: Discord connector (#515) * Initial commit of discord connector based off of initial work by @tnachen with modifications https://github.com/tnachen/unstructured/tree/tnachen/discord_connector * Add test file change format of imports * working version of the connector More work to be done to tidy it up and add any additional options * add to test fixtures update * fix spacing * tests working, switching to bot testing channel * add additional channel add reprocess to tests * add try clause to allow for exit on error Update changelog and bump version * add updated expected output filtes * add logic to check if —discord-period is an integer Add more to option description * fix lint error * Update discord reqs * PR feedback * add newline * another newline --------- Co-authored-by: Justin Bossert <packerbacker21@hotmail.com>
2023-05-16 11:46:30 -07:00
feat: Add GitHub data connector; add Markdown partitioner (#284)
2023-02-27 23:36:44 +01:00
.PHONY: install-ingest-github
install-ingest-github:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/github.txt
feat: Add GitHub data connector; add Markdown partitioner (#284)
2023-02-27 23:36:44 +01:00
Roman/notion tests (#1072) ### Description * Add ingest test for Notion docs * Update default cache dir for connectors to include connector name. Makes debugging the cached content easier. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-08-21 15:16:50 -04:00
.PHONY: install-ingest-biomed
install-ingest-biomed:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/biomed.txt
Roman/notion tests (#1072) ### Description * Add ingest test for Notion docs * Update default cache dir for connectors to include connector name. Makes debugging the cached content easier. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-08-21 15:16:50 -04:00
feat: Add GitLab ingest connector (#349) Add GitLab data connector for ingest. Involves more general Git functionality that is shared between the GitHub and GitLab data connectors. Prevent code duplication for functionality between GitHub and GitLab ingest connectors. Renamed github-access-token, github-branch and github-file-glob to git-access-token, git-branch and git-file-glob, respectively. These work for GitHub and GitLab.
2023-03-08 09:15:21 +01:00
.PHONY: install-ingest-gitlab
install-ingest-gitlab:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/gitlab.txt
feat: Add GitLab ingest connector (#349) Add GitLab data connector for ingest. Involves more general Git functionality that is shared between the GitHub and GitLab data connectors. Prevent code duplication for functionality between GitHub and GitLab ingest connectors. Renamed github-access-token, github-branch and github-file-glob to git-access-token, git-branch and git-file-glob, respectively. These work for GitHub and GitLab.
2023-03-08 09:15:21 +01:00
feat: add OneDrive connector (#834)
2023-07-13 14:57:54 -06:00
.PHONY: install-ingest-onedrive
install-ingest-onedrive:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/onedrive.txt
feat: add OneDrive connector (#834)
2023-07-13 14:57:54 -06:00
feat: adds Outlook connector (#939) * bonus: fixes issue with email partitioning where From field was being assigned the To field value.
2023-07-25 21:09:26 -07:00
.PHONY: install-ingest-outlook
install-ingest-outlook:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/outlook.txt
feat: adds Outlook connector (#939) * bonus: fixes issue with email partitioning where From field was being assigned the To field value.
2023-07-25 21:09:26 -07:00
feat: Add Reddit ingest connector (#293) Add Reddit data connector for ingest. * The connector can process a subreddit. * Either via a search query, * or via hot posts. * The texts in the submissions are converted to markdown files including the post title and the text body, if any (i.e. no images or videos). * The number of posts to fetch can be changed with the CLI.
2023-02-27 09:11:04 +01:00
.PHONY: install-ingest-reddit
install-ingest-reddit:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/reddit.txt
feat: Add Reddit ingest connector (#293) Add Reddit data connector for ingest. * The connector can process a subreddit. * Either via a search query, * or via hot posts. * The texts in the submissions are converted to markdown files including the post title and the text body, if any (i.e. no images or videos). * The number of posts to fetch can be changed with the CLI.
2023-02-27 09:11:04 +01:00
Slack connector (#462) This connector takes a slack channel id, token and other options to pull conversation history for a channel and store it as a text file that is then processed by unstructured into expected output.
2023-04-16 12:34:43 -07:00
.PHONY: install-ingest-slack
install-ingest-slack:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
pip install -r requirements/ingest/slack.txt
Slack connector (#462) This connector takes a slack channel id, token and other options to pull conversation history for a channel and store it as a text file that is then processed by unstructured into expected output.
2023-04-16 12:34:43 -07:00
feat: Add Wikipedia ingest connector (#299) The connector can process a Wikipedia page and output the HTML, the plain text contents, and the summary. No API key required Also add test case verifying that 3 files are indeed created (one for HTML, one for text, one for the summary).
2023-02-28 09:25:11 +01:00
.PHONY: install-ingest-wikipedia
install-ingest-wikipedia:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/wikipedia.txt
feat: Add Wikipedia ingest connector (#299) The connector can process a Wikipedia page and output the HTML, the plain text contents, and the summary. No API key required Also add test case verifying that 3 files are indeed created (one for HTML, one for text, one for the summary).
2023-02-28 09:25:11 +01:00
feat: elasticsearch connector (#817)
2023-07-01 18:45:28 +01:00
.PHONY: install-ingest-elasticsearch
install-ingest-elasticsearch:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/elasticsearch.txt
feat: confluence connector (cloud) (#906) * Add confluence connector and an example script * add test script, add dependency installations * add authentication secret variables for ci tests and actions * add dependency installation commands for workflows * add dependency installation commands for workflows * Update ingest test fixtures (#907) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add add ingest test fixtures update workflow for python 3.10, update example script with dummy values * change workflow name to avoid confusion * change workflow name to avoid confusion * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions, remove 3.10 workflow for the test fixtures update * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions * Update ingest test fixtures (#911) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * revert back the test python version matrix * recompile dependencies * modifications for shellcheck * update changelog and version * changelog and version * remove comments * Update ingest test fixtures (#915) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add the option to state the number of spaces to be fetched * add scroll functionality, expose --confluence-num-of-spaces, --confluence-list-of-spaces and --confluence-num-of-docs-from-each-space to users * add help message * add docstrings for two tests, validate grabbing every doc in the fetched spaces, count number of files instead of diffing for confluence2 test * change test names * rename connector arg Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * change arg name for connector Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * add comment to example * change arg names * add new tests to ingest test * shellcheck remove redundant statement * Update ingest test fixtures (#932) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * Update ingest test fixtures (#936) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * linting * change file extensions to parse as html * Update ingest test fixtures (#943) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * remove old fixtures * update version to 0.8.2-dev3 * change file to trigger CI * change file to trigger CI * change file to trigger CI * change file to trigger CI --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
2023-07-18 19:29:41 +01:00
feat: add opensearch source and destination connector (#2349) Adds OpenSearch as a source and destination. Since OpenSearch is a fork of Elasticsearch, these connectors rely heavily on inheriting the Elasticsearch connectors whenever possible. - Adds OpenSearch source connector to be able to ingest documents from OpenSearch. - Adds OpenSearch destination connector to be able to ingest documents from any supported source, embed them and write the embeddings / documents into OpenSearch. - Defines an example unstructured elements schema for users to be able to setup their unstructured OpenSearch indexes easily. --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2024-01-16 20:31:49 -08:00
.PHONY: install-ingest-opensearch
install-ingest-opensearch:
python3 -m pip install -r requirements/ingest/opensearch.txt
feat: confluence connector (cloud) (#906) * Add confluence connector and an example script * add test script, add dependency installations * add authentication secret variables for ci tests and actions * add dependency installation commands for workflows * add dependency installation commands for workflows * Update ingest test fixtures (#907) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add add ingest test fixtures update workflow for python 3.10, update example script with dummy values * change workflow name to avoid confusion * change workflow name to avoid confusion * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions, remove 3.10 workflow for the test fixtures update * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions * Update ingest test fixtures (#911) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * revert back the test python version matrix * recompile dependencies * modifications for shellcheck * update changelog and version * changelog and version * remove comments * Update ingest test fixtures (#915) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add the option to state the number of spaces to be fetched * add scroll functionality, expose --confluence-num-of-spaces, --confluence-list-of-spaces and --confluence-num-of-docs-from-each-space to users * add help message * add docstrings for two tests, validate grabbing every doc in the fetched spaces, count number of files instead of diffing for confluence2 test * change test names * rename connector arg Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * change arg name for connector Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * add comment to example * change arg names * add new tests to ingest test * shellcheck remove redundant statement * Update ingest test fixtures (#932) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * Update ingest test fixtures (#936) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * linting * change file extensions to parse as html * Update ingest test fixtures (#943) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * remove old fixtures * update version to 0.8.2-dev3 * change file to trigger CI * change file to trigger CI * change file to trigger CI * change file to trigger CI --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
2023-07-18 19:29:41 +01:00
.PHONY: install-ingest-confluence
install-ingest-confluence:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/confluence.txt
feat: confluence connector (cloud) (#906) * Add confluence connector and an example script * add test script, add dependency installations * add authentication secret variables for ci tests and actions * add dependency installation commands for workflows * add dependency installation commands for workflows * Update ingest test fixtures (#907) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add add ingest test fixtures update workflow for python 3.10, update example script with dummy values * change workflow name to avoid confusion * change workflow name to avoid confusion * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions, remove 3.10 workflow for the test fixtures update * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions * Update ingest test fixtures (#911) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * revert back the test python version matrix * recompile dependencies * modifications for shellcheck * update changelog and version * changelog and version * remove comments * Update ingest test fixtures (#915) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add the option to state the number of spaces to be fetched * add scroll functionality, expose --confluence-num-of-spaces, --confluence-list-of-spaces and --confluence-num-of-docs-from-each-space to users * add help message * add docstrings for two tests, validate grabbing every doc in the fetched spaces, count number of files instead of diffing for confluence2 test * change test names * rename connector arg Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * change arg name for connector Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * add comment to example * change arg names * add new tests to ingest test * shellcheck remove redundant statement * Update ingest test fixtures (#932) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * Update ingest test fixtures (#936) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * linting * change file extensions to parse as html * Update ingest test fixtures (#943) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * remove old fixtures * update version to 0.8.2-dev3 * change file to trigger CI * change file to trigger CI * change file to trigger CI * change file to trigger CI --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
2023-07-18 19:29:41 +01:00
feat: airtable connector (#1012) * add the first version of airtable connector * change imports as inline to fail gracefully in case of lacking dependency * parse tables as csv rather than plain text * add relevant logic to be able to use --airtable-list-of-paths * add script for creation of reseources for testing, add test script (large) for testing with a large number of tables to validate scroll functionality, update test script (diff) based on the new settings * fix ingest test names * add scripts for the large table test * remove large table test from diff test * make base and table ids explicit * add and remove comments * use -ne instead of != * update code based on the recent ingest refactor, update changelog and version * shellcheck fix * update comments * update check-num-rows-and-columns-output error message Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * update help comments * update help comments * update help comments * update workflows to set auth tokens and to run make install * add comments on create_scale_test_components * separate component ids from the test script, add comments to document test component creation * add LARGE_BASE test, implement LARGE_BASE component creation, replace component id * shellcheck fixes * shellcheck fixes * update docs * update comment * bump version * add wrongly deleted file * sort columns before saving to process * Update ingest test fixtures (#1098) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
2023-08-11 22:02:51 +03:00
.PHONY: install-ingest-airtable
install-ingest-airtable:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/airtable.txt
feat: airtable connector (#1012) * add the first version of airtable connector * change imports as inline to fail gracefully in case of lacking dependency * parse tables as csv rather than plain text * add relevant logic to be able to use --airtable-list-of-paths * add script for creation of reseources for testing, add test script (large) for testing with a large number of tables to validate scroll functionality, update test script (diff) based on the new settings * fix ingest test names * add scripts for the large table test * remove large table test from diff test * make base and table ids explicit * add and remove comments * use -ne instead of != * update code based on the recent ingest refactor, update changelog and version * shellcheck fix * update comments * update check-num-rows-and-columns-output error message Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * update help comments * update help comments * update help comments * update workflows to set auth tokens and to run make install * add comments on create_scale_test_components * separate component ids from the test script, add comments to document test component creation * add LARGE_BASE test, implement LARGE_BASE component creation, replace component id * shellcheck fixes * shellcheck fixes * update docs * update comment * bump version * add wrongly deleted file * sort columns before saving to process * Update ingest test fixtures (#1098) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
2023-08-11 22:02:51 +03:00
build(deps): dependency cleanup (#1102) Cleans up some pins that were prone to conflicts. All pins belong in constraints.in.
2023-08-15 00:15:44 -05:00
.PHONY: install-ingest-sharepoint
install-ingest-sharepoint:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/sharepoint.txt
build(deps): dependency cleanup (#1102) Cleans up some pins that were prone to conflicts. All pins belong in constraints.in.
2023-08-15 00:15:44 -05:00
feat: Weaviate destination connector (#1963) Closes #1781. - Adds a Weaviate destination connector - The connector receives a host for the weaviate instance and a weaviate class name. - Defines a weaviate schema for json elements. - Defines the pre-processing to conform unstructured's schema to the proposed weaviate schema.
2023-12-01 16:27:41 -06:00
.PHONY: install-ingest-weaviate
install-ingest-weaviate:
python3 -m pip install -r requirements/ingest/weaviate.txt
Roman/notion tests (#1072) ### Description * Add ingest test for Notion docs * Update default cache dir for connectors to include connector name. Makes debugging the cached content easier. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-08-21 15:16:50 -04:00
.PHONY: install-ingest-local
install-ingest-local:
echo "no unique dependencies for local connector"
.PHONY: install-ingest-notion
install-ingest-notion:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/notion.txt
Roman/notion tests (#1072) ### Description * Add ingest test for Notion docs * Update default cache dir for connectors to include connector name. Makes debugging the cached content easier. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
2023-08-21 15:16:50 -04:00
feat: add salesforce connector (#1168)
2023-09-02 08:50:31 -07:00
.PHONY: install-ingest-salesforce
install-ingest-salesforce:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/salesforce.txt
feat: add salesforce connector (#1168)
2023-09-02 08:50:31 -07:00
feat: jira connector (cloud) (#1238) This connector: - takes a Jira Cloud URL, user email and api token; to authenticate into Jira Cloud - ingests: - either all issues in all projects in a Jira Cloud Organization - or - issues in user specified projects, boards - user specified issues - processes this kind of data: - text fields such as issue summary, description, and comments - dropdown fields such as issue type, status, priority, assignee, reporter, labels, and components - other data such as issue id, issue key, project id, information on subtasks - notes down attachment URLs, however does not process attachments - stores each downloaded issue in a txt file, in a predefined template form (consisting of the data above) - then processes each downloaded issue document into elements using unstructured library - related to: https://github.com/Unstructured-IO/unstructured/issues/263 To test the changes, make the necessary setups and run the relevant ingest test scripts. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
2023-09-06 13:10:48 +03:00
.PHONY: install-ingest-jira
install-ingest-jira:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/jira.txt
feat: jira connector (cloud) (#1238) This connector: - takes a Jira Cloud URL, user email and api token; to authenticate into Jira Cloud - ingests: - either all issues in all projects in a Jira Cloud Organization - or - issues in user specified projects, boards - user specified issues - processes this kind of data: - text fields such as issue summary, description, and comments - dropdown fields such as issue type, status, priority, assignee, reporter, labels, and components - other data such as issue id, issue key, project id, information on subtasks - notes down attachment URLs, however does not process attachments - stores each downloaded issue in a txt file, in a predefined template form (consisting of the data above) - then processes each downloaded issue document into elements using unstructured library - related to: https://github.com/Unstructured-IO/unstructured/issues/263 To test the changes, make the necessary setups and run the relevant ingest test scripts. --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
2023-09-06 13:10:48 +03:00
rvztz/hubspot ingest connector (#1760) Closes #1843 Ingest connector for HubSpot. Supports: - Calls: Logs from calls related to contacts, companies and tickets - Communications: Logs from SMS/Whatsapp related to contacts, companies and tickets - Notes: Notes related to CRM notes - Products: CRM products - Emails: Logs from emails sent to CRM objects. - Tasks: CRM tasks From each record, `body/`description`information is grabbed. When a title property is available, this is registered at the beggining of the output file. The CLI receives three params: - `api-token`: [Private app](https://developers.hubspot.com/docs/api/private-apps) token. - `object-types: One of the noted supported objects in the form of a comma separated list: `calls,products,tasks` - `custom-properties`: Custom properties to grab information from. Must be in the form `<object_type>:<custom_property_id>,<object_type>:<custom_property_id>` --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rvztz <rvztz@users.noreply.github.com>
2023-11-28 17:07:57 -06:00
.PHONY: install-ingest-hubspot
install-ingest-hubspot:
feat: Add sftp source connector (#2163) Adds source connector for SFTP which uses fsspec and paramiko via fsspec. Paramiko is the standard sftp package for python used in pysftp etc... ``` --username foo \ --password bar \ --remote-url sftp://localhost:47474/upload/ ``` Will only download a specifically requested file if it has an extension. (i.e. `--remote-url sftp://localhost:47474/upload/bob.zip`) It will treat any other remote_url as a folder path. This is intentional. --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2023-12-07 11:33:19 -08:00
python3 -m pip install -r requirements/ingest/hubspot.txt
.PHONY: install-ingest-sftp
install-ingest-sftp:
python3 -m pip install -r requirements/ingest/sftp.txt
rvztz/hubspot ingest connector (#1760) Closes #1843 Ingest connector for HubSpot. Supports: - Calls: Logs from calls related to contacts, companies and tickets - Communications: Logs from SMS/Whatsapp related to contacts, companies and tickets - Notes: Notes related to CRM notes - Products: CRM products - Emails: Logs from emails sent to CRM objects. - Tasks: CRM tasks From each record, `body/`description`information is grabbed. When a title property is available, this is registered at the beggining of the output file. The CLI receives three params: - `api-token`: [Private app](https://developers.hubspot.com/docs/api/private-apps) token. - `object-types: One of the noted supported objects in the form of a comma separated list: `calls,products,tasks` - `custom-properties`: Custom properties to grab information from. Must be in the form `<object_type>:<custom_property_id>,<object_type>:<custom_property_id>` --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: rvztz <rvztz@users.noreply.github.com>
2023-11-28 17:07:57 -06:00
feat: add pinecone destination connector (#1774) Closes https://github.com/Unstructured-IO/unstructured/issues/1414 Closes #2039 This PR: - Uses Pinecone python cli to implement a destination connector for Pinecone and provides the ingest readme requirements [(here)](https://github.com/Unstructured-IO/unstructured/tree/main/unstructured/ingest#the-checklist) for the connector - Updates documentation for the s3 destination connector - Alphabetically sorts setup.py contents - Updates logs for the chunking node in ingest pipeline - Adds a baseline session handle implementation for destination connectors, to be able to parallelize their operations - For the [bug](https://github.com/Unstructured-IO/unstructured/issues/1892) related to persisting element data to ingest embedding nodes; this PR tests the [solution](https://github.com/Unstructured-IO/unstructured/pull/1893) with its ingest test - Solves a bug on ingest chunking params with [bugfix on chunking params and implementing related test](https://github.com/Unstructured-IO/unstructured/pull/1774/commits/69e1949a6fa806d14014eb68074929595f92360e) --------- Co-authored-by: Roman Isecke <136338424+rbiseck3@users.noreply.github.com>
2023-11-29 22:37:32 +00:00
.PHONY: install-ingest-pinecone
install-ingest-pinecone:
feat: add chroma destination connector (#2240) Adds Chroma (also known as ChromaDB) as a vector destination. Currently Chroma is an in-memory single-process oriented library with plans of a hosted and/or more production ready solution -https://docs.trychroma.com/deployment Though they now claim to support multiple Clients hitting the database at once, I found that it was inconsistent. Sometimes multiprocessing worked (maybe 1 out of 3 times) But the other times I would get different errors. So I kept it single process. --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2023-12-19 08:58:23 -08:00
python3 -m pip install -r requirements/ingest/pinecone.txt
feat: add Qdrant ingest destination connector (#2338) This PR intends to add [Qdrant](https://qdrant.tech/) as a supported ingestion destination. - Implements CLI and programmatic usage. - Documentation update - Integration test script --- Clone of #2315 to run with CI secrets --------- Co-authored-by: Anush008 <anushshetty90@gmail.com> Co-authored-by: Roman Isecke <136338424+rbiseck3@users.noreply.github.com>
2024-01-02 14:08:20 -08:00
.PHONY: install-ingest-qdrant
install-ingest-qdrant:
python3 -m pip install -r requirements/ingest/qdrant.txt
feat: add chroma destination connector (#2240) Adds Chroma (also known as ChromaDB) as a vector destination. Currently Chroma is an in-memory single-process oriented library with plans of a hosted and/or more production ready solution -https://docs.trychroma.com/deployment Though they now claim to support multiple Clients hitting the database at once, I found that it was inconsistent. Sometimes multiprocessing worked (maybe 1 out of 3 times) But the other times I would get different errors. So I kept it single process. --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2023-12-19 08:58:23 -08:00
.PHONY: install-ingest-chroma
install-ingest-chroma:
python3 -m pip install -r requirements/ingest/chroma.txt
feat: add pinecone destination connector (#1774) Closes https://github.com/Unstructured-IO/unstructured/issues/1414 Closes #2039 This PR: - Uses Pinecone python cli to implement a destination connector for Pinecone and provides the ingest readme requirements [(here)](https://github.com/Unstructured-IO/unstructured/tree/main/unstructured/ingest#the-checklist) for the connector - Updates documentation for the s3 destination connector - Alphabetically sorts setup.py contents - Updates logs for the chunking node in ingest pipeline - Adds a baseline session handle implementation for destination connectors, to be able to parallelize their operations - For the [bug](https://github.com/Unstructured-IO/unstructured/issues/1892) related to persisting element data to ingest embedding nodes; this PR tests the [solution](https://github.com/Unstructured-IO/unstructured/pull/1893) with its ingest test - Solves a bug on ingest chunking params with [bugfix on chunking params and implementing related test](https://github.com/Unstructured-IO/unstructured/pull/1774/commits/69e1949a6fa806d14014eb68074929595f92360e) --------- Co-authored-by: Roman Isecke <136338424+rbiseck3@users.noreply.github.com>
2023-11-29 22:37:32 +00:00
feat: adds postgresql/sqlite destination connector (#2005) - Adds a destination connector to upload processed output into a PostgreSQL/Sqlite database instance. - Users are responsible to provide their instances. This PR includes a couple of configuration examples. - Defines the scripts required to setup a PostgreSQL instance with the unstructured elements schema. - Validates postgres/pgvector embedding storage and retrieval --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2024-01-04 13:33:16 -06:00
.PHONY: install-ingest-postgres
install-ingest-postgres:
feat: databricks volumes dest added (#2391) ### Description This adds in a destination connector to write content to the Databricks Unity Catalog Volumes service. Currently there is an internal account that can be used for testing manually but there is not dedicated account to use for testing so this is not being added to the automated ingest tests that get run in the CI. To test locally: ```shell #!/usr/bin/env bash path="testpath/$(uuidgen)" PYTHONPATH=. python ./unstructured/ingest/main.py local \ --num-processes 4 \ --output-dir azure-test \ --strategy fast \ --verbose \ --input-path example-docs/fake-memo.pdf \ --recursive \ databricks-volumes \ --catalog "utic-dev-tech-fixtures" \ --volume "small-pdf-set" \ --volume-path "$path" \ --username "$DATABRICKS_USERNAME" \ --password "$DATABRICKS_PASSWORD" \ --host "$DATABRICKS_HOST" ```
2024-01-22 20:25:51 -05:00
python3 -m pip install -r requirements/ingest/postgres.txt
feat: adds postgresql/sqlite destination connector (#2005) - Adds a destination connector to upload processed output into a PostgreSQL/Sqlite database instance. - Users are responsible to provide their instances. This PR includes a couple of configuration examples. - Defines the scripts required to setup a PostgreSQL instance with the unstructured elements schema. - Validates postgres/pgvector embedding storage and retrieval --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2024-01-04 13:33:16 -06:00
feat: add MongoDB source connector (#2393) Adds MongoDB as a source (we already had it as a destination connector) --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2024-01-16 12:56:29 -08:00
.PHONY: install-ingest-mongodb
install-ingest-mongodb:
feat: databricks volumes dest added (#2391) ### Description This adds in a destination connector to write content to the Databricks Unity Catalog Volumes service. Currently there is an internal account that can be used for testing manually but there is not dedicated account to use for testing so this is not being added to the automated ingest tests that get run in the CI. To test locally: ```shell #!/usr/bin/env bash path="testpath/$(uuidgen)" PYTHONPATH=. python ./unstructured/ingest/main.py local \ --num-processes 4 \ --output-dir azure-test \ --strategy fast \ --verbose \ --input-path example-docs/fake-memo.pdf \ --recursive \ databricks-volumes \ --catalog "utic-dev-tech-fixtures" \ --volume "small-pdf-set" \ --volume-path "$path" \ --username "$DATABRICKS_USERNAME" \ --password "$DATABRICKS_PASSWORD" \ --host "$DATABRICKS_HOST" ```
2024-01-22 20:25:51 -05:00
python3 -m pip install -r requirements/ingest/mongodb.txt
.PHONY: install-ingest-databricks-volumes
install-ingest-databricks-volumes:
python3 -m pip install -r requirements/ingest/databricks-volumes.txt
feat: add MongoDB source connector (#2393) Adds MongoDB as a source (we already had it as a destination connector) --------- Co-authored-by: potter-potter <david.potter@gmail.com>
2024-01-16 12:56:29 -08:00
feat: astra dest connector (#2571) Thanks to Eric Hare @erichare at DataStax we have a new destination connector. This Pull Request implements an integration with [Astra DB](https://datastax.com) which allows for the Astra DB Vector Database to be compatible with Unstructured's set of integrations. To create your Astra account and authenticate with your `ASTRA_DB_APPLICATION_TOKEN`, and `ASTRA_DB_API_ENDPOINT`, follow these steps: 1. Create an account at https://astra.datastax.com 2. Login and create a new database 3. From the database page, in the right hand panel, you will find your API Endpoint 4. Beneath that, you can create a Token to be used Some notes about Astra DB: - Astra DB is a Vector Database which allows for high-performance database transactions, and enables modern GenAI apps [See here](https://docs.datastax.com/en/astra/astra-db-vector/get-started/concepts.html) - It supports similarity search via a number of methods [See here](https://docs.datastax.com/en/astra/astra-db-vector/get-started/concepts.html#metrics) - It also supports non-vector tables / collections
2024-02-23 12:50:50 -08:00
.PHONY: install-ingest-astra
install-ingest-astra:
python3 -m pip install -r requirements/ingest/astra.txt
feat: Adds local embedding model (#1619) This PR adds a local embedding model option as an alternative to using our OpenAI embedding brick. This brick uses LangChain's HuggingFacEmbeddings.
2023-10-19 11:51:36 -05:00
.PHONY: install-embed-huggingface
install-embed-huggingface:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/embed-huggingface.txt
feat: Adds local embedding model (#1619) This PR adds a local embedding model option as an alternative to using our OpenAI embedding brick. This brick uses LangChain's HuggingFacEmbeddings.
2023-10-19 11:51:36 -05:00
feat: local inference (#125) Splits partition_pdf into two paths, one used for local inference when url is None, another for inference via api when url is a string.
2023-01-04 16:19:05 -06:00
.PHONY: install-unstructured-inference
install-unstructured-inference:
chore: fix Makefile ingest targets (#2051) Fixes the Makefile `ingest-` targets were broken in https://github.com/Unstructured-IO/unstructured/pull/1799/files. **Test Instructions** for maketarget in $(grep .PHONY Makefile | grep install-ingest | perl -p -e 's/.PHONY://' | tr -d '\n'); do echo $maketarget; make $maketarget done
2023-11-09 21:55:27 -08:00
python3 -m pip install -r requirements/ingest/local-inference.txt
feat: local inference (#125) Splits partition_pdf into two paths, one used for local inference when url is None, another for inference via api when url is a string.
2023-01-04 16:19:05 -06:00
## install-local-inference: installs requirements for local inference
.PHONY: install-local-inference
build(deps): split up dependencies by document type (#986) * split dependencies by document type * make pip-compile with new requirements * add extra requirements to setup.py * add in all docs; re pip-compile * extra for all docs * add pandas to xlsx * dependency requires for tsv and csv * handling for doc, docx and odt * dependency check for pypandoc * required dependencies for pandoc files * xml and html * markdown * msg * add in pdf * add in pptx * add in excel * add lxml as base req * extra all docs for local inference * local inference installs all * pin pillow version * fixes for plain text tests * fixes for doc * update make commands * changelog and version * add xlrd * update pip-compile * pin numpy for python 3.8 support * more constraints * contraint on scipy * update install docs * constrain ipython * add outlook to pip-compile * more ipython constraints * add extras to dockerfile * pin office365 client * few doc tweaks * types as strings * last pip-compile * re pip-comple * make tidy * make tidy
2023-08-01 11:31:13 -04:00
install-local-inference: install install-all-docs
feat: local inference (#125) Splits partition_pdf into two paths, one used for local inference when url is None, another for inference via api when url is a string.
2023-01-04 16:19:05 -06:00
Chore: Support epub tests in docker image (#630) * docker works * more epub tests * changelog version * support epub + odt + rtf * update dockerfile * revert.. * install pandoc on ci env * pandoc docker grab bashed on arch * move arch into image * move back to base image
2023-05-26 15:38:48 -04:00
.PHONY: install-pandoc
install-pandoc:
ARCH=${ARCH} ./scripts/install-pandoc.sh
Feat: CORE-1269 - Install paddlepaddle wheel dependent on arch, supporting aarch64 (#1350) Testing instructions on Apple silicon ``` make docker-build docker run -it unstructured:dev bash python3 ``` Then run the test in this PR https://unstructured-ai.atlassian.net/browse/CORE-1269 You should get output like shown in ticket Run the same process on your local machine (not inside docker) with same test to verify the non aarch64 paddlepaddle got installed correctly --------- Co-authored-by: Yuming Long <63475068+yuming-long@users.noreply.github.com>
2023-09-15 17:05:48 -07:00
.PHONY: install-paddleocr
install-paddleocr:
ARCH=${ARCH} ./scripts/install-paddleocr.sh
Chore: Support epub tests in docker image (#630) * docker works * more epub tests * changelog version * support epub + odt + rtf * update dockerfile * revert.. * install pandoc on ci env * pandoc docker grab bashed on arch * move arch into image * move back to base image
2023-05-26 15:38:48 -04:00
Initial Release
2022-06-29 14:35:19 -04:00
## pip-compile: compiles all base/dev/test requirements
.PHONY: pip-compile
pip-compile:
feat: adds Azure Cognitive Search (full text) destination connector (#1459) ### Description New [Azure Cognitive Search](https://azure.microsoft.com/en-us/products/ai-services/cognitive-search) destination connector added. Writes each json element from the created json files via partition and writes that content to an index. **Bonus bug fix:** Due to a recent change where the default version of python used in the repo was bumped to `3.10` from `3.8`, this means running `pip-compile` now runs it against that version rather than the lowest we support which is still `3.8`. This breaks the setup for those lower versions because some of the versions pulled in by `pip-compile` exist for `3.10` but not `3.8`. `pip-compile` was updates to run as a script that checks the version of python being used first, which helps guarantee that all dependencies meet the minimum python version requirement. Closes out https://github.com/Unstructured-IO/unstructured/issues/1466
2023-09-25 10:27:42 -04:00
@scripts/pip-compile.sh
Add s3 writer (#1223) ### Description Convert s3 cli code to also support writing to s3. Writers are added as optional subcommands to the parent command with their own arguments. Custom `click.Group` introduced to add some custom formatting and text in help messages. To limit the scope of this PR, most existing files were not touched but instead new files were added for the new flow. This allowed _only_ the s3 connector to be updated without breaking any other ones.
2023-08-31 18:19:53 -04:00
Initial Release
2022-06-29 14:35:19 -04:00
## install-project-local: install unstructured into your local python environment
.PHONY: install-project-local
install-project-local: install
# MAYBE TODO: fail if already exists?
pip install -e .
## uninstall-project-local: uninstall unstructured from your local python environment
.PHONY: uninstall-project-local
uninstall-project-local:
pip uninstall ${PACKAGE_NAME}
#################
# Test and Lint #
#################
fix: use api key where needed in tests (#843) * passes api key for unstructured-api to unit and ingest tests as needed. * adds check for env var CI to otherwise skip tests that require an api key
2023-06-29 10:31:01 -07:00
export CI ?= false
feat/add sources from unstructured inference (#1538) This PR adds support for `source` property from `unstructured_inference`, allowing the user to be able to see the origin of the data under `detection_origin`field environment variable UNSTRUCTURED_INCLUDE_DEBUG_METADATA=true In order to try this feature you can use this code: ``` from unstructured.partition.pdf import partition_pdf_or_image yolox_elements = partition_pdf_or_image(filename='example-docs/loremipsum-flat.pdf', strategy='hi_res', model_name='yolox') sources = [e.detection_origin for e in yolox_elements] print(sources) ``` And will print 'yolox' as source for all the elements
2023-10-05 15:26:47 -05:00
export UNSTRUCTURED_INCLUDE_DEBUG_METADATA ?= false
fix: use api key where needed in tests (#843) * passes api key for unstructured-api to unit and ingest tests as needed. * adds check for env var CI to otherwise skip tests that require an api key
2023-06-29 10:31:01 -07:00
Initial Release
2022-06-29 14:35:19 -04:00
## test: runs all unittests
.PHONY: test
test:
feat/add sources from unstructured inference (#1538) This PR adds support for `source` property from `unstructured_inference`, allowing the user to be able to see the origin of the data under `detection_origin`field environment variable UNSTRUCTURED_INCLUDE_DEBUG_METADATA=true In order to try this feature you can use this code: ``` from unstructured.partition.pdf import partition_pdf_or_image yolox_elements = partition_pdf_or_image(filename='example-docs/loremipsum-flat.pdf', strategy='hi_res', model_name='yolox') sources = [e.detection_origin for e in yolox_elements] print(sources) ``` And will print 'yolox' as source for all the elements
2023-10-05 15:26:47 -05:00
PYTHONPATH=. CI=$(CI) \
tests: separate chipper tests (#1939) Separates chipper tests to speed up testing and CI.
2023-10-31 16:02:00 -05:00
UNSTRUCTURED_INCLUDE_DEBUG_METADATA=$(UNSTRUCTURED_INCLUDE_DEBUG_METADATA) pytest test_${PACKAGE_NAME} -m "not chipper" --cov=${PACKAGE_NAME} --cov-report term-missing --durations=40
.PHONY: test-chipper
test-chipper:
PYTHONPATH=. CI=$(CI) \
UNSTRUCTURED_INCLUDE_DEBUG_METADATA=$(UNSTRUCTURED_INCLUDE_DEBUG_METADATA) pytest test_${PACKAGE_NAME} -m "chipper" --cov=${PACKAGE_NAME} --cov-report term-missing --durations=40
Initial Release
2022-06-29 14:35:19 -04:00
Chore: add uns api repo unittests (#954) * stage * git clone * ci ignore markdown file * make install * use env instead * remove md * add script * wrong env value * add note * maybe don't rm * no cd../ --------- Co-authored-by: cragwolfe <crag@unstructured.io>
2023-07-26 16:55:35 -04:00
.PHONY: test-unstructured-api-unit
test-unstructured-api-unit:
scripts/test-unstructured-api-unit.sh
test: create CI pipelines for verifying base and extras pass respective tests (#1137) **Summary** Closes #747 * Create CI Pipeline for running text, xml, email, and html doc tests against the library installed without extras * Create CI Pipeline for running each library extra against their respective tests
2023-08-19 12:56:13 -04:00
.PHONY: test-no-extras
# TODO(newelh) Add json test when fixed
test-no-extras:
feat/add sources from unstructured inference (#1538) This PR adds support for `source` property from `unstructured_inference`, allowing the user to be able to see the origin of the data under `detection_origin`field environment variable UNSTRUCTURED_INCLUDE_DEBUG_METADATA=true In order to try this feature you can use this code: ``` from unstructured.partition.pdf import partition_pdf_or_image yolox_elements = partition_pdf_or_image(filename='example-docs/loremipsum-flat.pdf', strategy='hi_res', model_name='yolox') sources = [e.detection_origin for e in yolox_elements] print(sources) ``` And will print 'yolox' as source for all the elements
2023-10-05 15:26:47 -05:00
PYTHONPATH=. CI=$(CI) \
UNSTRUCTURED_INCLUDE_DEBUG_METADATA=$(UNSTRUCTURED_INCLUDE_DEBUG_METADATA) pytest \
test: create CI pipelines for verifying base and extras pass respective tests (#1137) **Summary** Closes #747 * Create CI Pipeline for running text, xml, email, and html doc tests against the library installed without extras * Create CI Pipeline for running each library extra against their respective tests
2023-08-19 12:56:13 -04:00
test_${PACKAGE_NAME}/partition/test_text.py \
test_${PACKAGE_NAME}/partition/test_email.py \
test_${PACKAGE_NAME}/partition/test_html_partition.py \
Add s3 writer (#1223) ### Description Convert s3 cli code to also support writing to s3. Writers are added as optional subcommands to the parent command with their own arguments. Custom `click.Group` introduced to add some custom formatting and text in help messages. To limit the scope of this PR, most existing files were not touched but instead new files were added for the new flow. This allowed _only_ the s3 connector to be updated without breaking any other ones.
2023-08-31 18:19:53 -04:00
test_${PACKAGE_NAME}/partition/test_xml_partition.py
test: create CI pipelines for verifying base and extras pass respective tests (#1137) **Summary** Closes #747 * Create CI Pipeline for running text, xml, email, and html doc tests against the library installed without extras * Create CI Pipeline for running each library extra against their respective tests
2023-08-19 12:56:13 -04:00
.PHONY: test-extra-csv
test-extra-csv:
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/csv
.PHONY: test-extra-docx
test-extra-docx:
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/docx
.PHONY: test-extra-markdown
test-extra-markdown:
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/markdown
.PHONY: test-extra-msg
test-extra-msg:
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/msg
.PHONY: test-extra-odt
test-extra-odt:
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/odt
.PHONY: test-extra-pdf-image
test-extra-pdf-image:
PYTHONPATH=. CI=$(CI) pytest \
serde tests round-trip through JSON (#1681) Each partitioner has a test like `test_partition_x_with_json()`. What these do is serialize the elements produced by the partitioner to JSON, then read them back in from JSON and compare the before and after elements. Because our element equality (`Element.__eq__()`) is shallow, this doesn't tell us a lot, but if we take it one more step, like `List[Element] -> JSON -> List[Element] -> JSON` and then compare the JSON, it gives us some confidence that the serialized elements can be "re-hydrated" without losing any information. This actually showed up a few problems, all in the serialization/deserialization (serde) code that all elements share.
2023-10-12 12:47:55 -07:00
test_${PACKAGE_NAME}/partition/pdf_image
test: create CI pipelines for verifying base and extras pass respective tests (#1137) **Summary** Closes #747 * Create CI Pipeline for running text, xml, email, and html doc tests against the library installed without extras * Create CI Pipeline for running each library extra against their respective tests
2023-08-19 12:56:13 -04:00
.PHONY: test-extra-pptx
test-extra-pptx:
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/pptx
.PHONY: test-extra-epub
fix: remove duplicate target in makefile (#1235) Removed a duplicate make target in the `Makefile`.
2023-08-29 01:49:18 -05:00
test-extra-epub:
test: create CI pipelines for verifying base and extras pass respective tests (#1137) **Summary** Closes #747 * Create CI Pipeline for running text, xml, email, and html doc tests against the library installed without extras * Create CI Pipeline for running each library extra against their respective tests
2023-08-19 12:56:13 -04:00
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/epub
.PHONY: test-extra-pypandoc
test-extra-pypandoc:
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/pypandoc
.PHONY: test-extra-xlsx
test-extra-xlsx:
PYTHONPATH=. CI=$(CI) pytest \
test_${PACKAGE_NAME}/partition/xlsx
Initial Release
2022-06-29 14:35:19 -04:00
## check: runs linters (includes tests)
.PHONY: check
chore: lint for print statements in ingest code (#2215) ### Description Given the filtering in the ingest logger, anything going to console should go through that. This adds a linter that only checks for `print()` statements in the ingest code and ignored it elsewhere for now.
2023-12-05 11:42:23 -05:00
check: check-ruff check-black check-flake8 check-version check-flake8-print
Initial Release
2022-06-29 14:35:19 -04:00
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called.
2023-12-11 20:04:15 -05:00
.PHONY: check-shfmt
check-shfmt:
chore: shell scripts default indent of 2 instead of 4 (#2287) Given the tendency for shell scripts to easily enter into a few levels of indentation and long line lengths, update the default to 2 spaces.
2023-12-18 23:48:21 -08:00
shfmt -i 2 -d .
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called.
2023-12-11 20:04:15 -05:00
refactoring linting (#1739) ### Description Currently linting only takes place over the base unstructured directory but we support python files throughout the repo. It makes sense for all those files to also abide by the same linting rules so the entire repo was set to be inspected when the linters are run. Along with that autoflake was added as a linter which has a lot of added benefits such as removing unused imports for you that would currently break flake and require manual intervention. The only real relevant changes in this PR are in the `Makefile`, `setup.cfg`, and `requirements/test.in`. The rest is the result of running the linters.
2023-10-17 08:45:12 -04:00
.PHONY: check-black
check-black:
black . --check
.PHONY: check-flake8
check-flake8:
flake8 .
chore: lint for print statements in ingest code (#2215) ### Description Given the filtering in the ingest logger, anything going to console should go through that. This adds a linter that only checks for `print()` statements in the ingest code and ignored it elsewhere for now.
2023-12-05 11:42:23 -05:00
# Check for print statements in ingest since anything going to console should be using the ingest logger
# as it has a built in filter to redact sensitive information
.PHONY: check-flake8-print
check-flake8-print:
flake8 --per-file-ignores "" ./unstructured/ingest
refactoring linting (#1739) ### Description Currently linting only takes place over the base unstructured directory but we support python files throughout the repo. It makes sense for all those files to also abide by the same linting rules so the entire repo was set to be inspected when the linters are run. Along with that autoflake was added as a linter which has a lot of added benefits such as removing unused imports for you that would currently break flake and require manual intervention. The only real relevant changes in this PR are in the `Makefile`, `setup.cfg`, and `requirements/test.in`. The rest is the result of running the linters.
2023-10-17 08:45:12 -04:00
.PHONY: check-ruff
check-ruff:
update .pre-commit-config to match linting used by CI (#1906) Closes #1905 .pre-commit-config.yaml does not match pyproject.toml, which causes unnecessary/undesirable formatting changes. These changes are not required by CI, so they should not have to be made. **To Reproduce** Install pre-commit configuration as described [here](https://github.com/Unstructured-IO/unstructured#installation-instructions-for-local-development). Make a commit and something like the following will be logged: ``` check for added large files..............................................Passed check toml...........................................(no files to check)Skipped check yaml...........................................(no files to check)Skipped check json...........................................(no files to check)Skipped check xml............................................(no files to check)Skipped fix end of files.........................................................Passed trim trailing whitespace.................................................Passed mixed line ending........................................................Passed black....................................................................Passed ruff.....................................................................Failed - hook id: ruff - files were modified by this hook ``` --------- Co-authored-by: Yao You <theyaoyou@gmail.com>
2023-10-27 13:24:55 -05:00
ruff . --select C4,COM,E,F,I,PLR0402,PT,SIM,UP015,UP018,UP032,UP034 --ignore COM812,PT011,PT012,SIM117
Initial Release
2022-06-29 14:35:19 -04:00
refactoring linting (#1739) ### Description Currently linting only takes place over the base unstructured directory but we support python files throughout the repo. It makes sense for all those files to also abide by the same linting rules so the entire repo was set to be inspected when the linters are run. Along with that autoflake was added as a linter which has a lot of added benefits such as removing unused imports for you that would currently break flake and require manual intervention. The only real relevant changes in this PR are in the `Makefile`, `setup.cfg`, and `requirements/test.in`. The rest is the result of running the linters.
2023-10-17 08:45:12 -04:00
.PHONY: check-autoflake
check-autoflake:
autoflake --check-diff .
Initial Release
2022-06-29 14:35:19 -04:00
feat: Add shellcheck to CI and Make target (#10)
2022-09-29 15:24:28 -04:00
## check-scripts: run shellcheck
.PHONY: check-scripts
check-scripts:
# Fail if any of these files have warnings
scripts/shellcheck.sh
feat: keep version synchronized (#25) * Added script to check/sync versions using CHANGELOG.md as a source of truth. * Script currently only syncs __version__.py but can easily be extended to cover other files by adding the files to an array in the script. * Also updated sphinx conf.py to get version dynamically from __version__.py
2022-10-10 13:11:48 -05:00
## check-version: run check to ensure version in CHANGELOG.md matches version in package
.PHONY: check-version
check-version:
# Fail if syncing version would produce changes
sync version-sync.sh with other repos (#508)
2023-04-21 05:48:38 +09:00
scripts/version-sync.sh -c \
-f "unstructured/__version__.py" semver
feat: keep version synchronized (#25) * Added script to check/sync versions using CHANGELOG.md as a source of truth. * Script currently only syncs __version__.py but can easily be extended to cover other files by adding the files to an array in the script. * Also updated sphinx conf.py to get version dynamically from __version__.py
2022-10-10 13:11:48 -05:00
Initial Release
2022-06-29 14:35:19 -04:00
## tidy: run black
.PHONY: tidy
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called.
2023-12-11 20:04:15 -05:00
tidy: tidy-python
.PHONY: tidy_shell
tidy-shell:
chore: shell scripts default indent of 2 instead of 4 (#2287) Given the tendency for shell scripts to easily enter into a few levels of indentation and long line lengths, update the default to 2 spaces.
2023-12-18 23:48:21 -08:00
shfmt -i 2 -l -w .
chore: add shfmt (#2246) ### Description Given all the shell files that now exist in the repo, would be nice to have linting/formatting around them (in addition to the existing shellcheck which doesn't do anything to format the shell code). This PR introduces `shfmt` to both check for changes and apply formatting when the associated make targets are called.
2023-12-11 20:04:15 -05:00
.PHONY: tidy-python
tidy-python:
update .pre-commit-config to match linting used by CI (#1906) Closes #1905 .pre-commit-config.yaml does not match pyproject.toml, which causes unnecessary/undesirable formatting changes. These changes are not required by CI, so they should not have to be made. **To Reproduce** Install pre-commit configuration as described [here](https://github.com/Unstructured-IO/unstructured#installation-instructions-for-local-development). Make a commit and something like the following will be logged: ``` check for added large files..............................................Passed check toml...........................................(no files to check)Skipped check yaml...........................................(no files to check)Skipped check json...........................................(no files to check)Skipped check xml............................................(no files to check)Skipped fix end of files.........................................................Passed trim trailing whitespace.................................................Passed mixed line ending........................................................Passed black....................................................................Passed ruff.....................................................................Failed - hook id: ruff - files were modified by this hook ``` --------- Co-authored-by: Yao You <theyaoyou@gmail.com>
2023-10-27 13:24:55 -05:00
ruff . --select C4,COM,E,F,I,PLR0402,PT,SIM,UP015,UP018,UP032,UP034 --fix-only --ignore COM812,PT011,PT012,SIM117 || true
refactoring linting (#1739) ### Description Currently linting only takes place over the base unstructured directory but we support python files throughout the repo. It makes sense for all those files to also abide by the same linting rules so the entire repo was set to be inspected when the linters are run. Along with that autoflake was added as a linter which has a lot of added benefits such as removing unused imports for you that would currently break flake and require manual intervention. The only real relevant changes in this PR are in the `Makefile`, `setup.cfg`, and `requirements/test.in`. The rest is the result of running the linters.
2023-10-17 08:45:12 -04:00
autoflake --in-place .
Add check for duplicate click options (#1775) ### Description Given that many of the options associated with the `Click` based cli ingest commands are added dynamically from a number of configs, a check was incorporated to make sure there were no duplicate entries to prevent new configs from overwriting already added options. ### Issues that were found and fixes: * duplicate api-key option set on Notion command conflicts with api key used for unstructured api. Added notion prefix. * retry logic configs had duplicates in biomed. Removed since this is not handled by the pipeline.
2023-10-20 10:00:19 -04:00
black .
Initial Release
2022-06-29 14:35:19 -04:00
feat: keep version synchronized (#25) * Added script to check/sync versions using CHANGELOG.md as a source of truth. * Script currently only syncs __version__.py but can easily be extended to cover other files by adding the files to an array in the script. * Also updated sphinx conf.py to get version dynamically from __version__.py
2022-10-10 13:11:48 -05:00
## version-sync: update __version__.py with most recent version from CHANGELOG.md
.PHONY: version-sync
version-sync:
sync version-sync.sh with other repos (#508)
2023-04-21 05:48:38 +09:00
scripts/version-sync.sh \
-f "unstructured/__version__.py" semver
feat: keep version synchronized (#25) * Added script to check/sync versions using CHANGELOG.md as a source of truth. * Script currently only syncs __version__.py but can easily be extended to cover other files by adding the files to an array in the script. * Also updated sphinx conf.py to get version dynamically from __version__.py
2022-10-10 13:11:48 -05:00
Initial Release
2022-06-29 14:35:19 -04:00
.PHONY: check-coverage
check-coverage:
coverage report --fail-under=95
adding Dockerfile (#359)
2023-03-14 13:40:01 -07:00
build(deps): avoid version conflicts (#636) Addresses #631. * Uses constraints to keep dependency versions more consistent. * Moves all dependencies to .in files which are then ingested by setup.py. * Adds script to check consistency of all extras. * Adds consistency check to CI. I should note that while it shouldn't be possible to cause a conflict between base.txt and any of the extras (because base.txt constrains all the extras) it is possible to get a conflict between two of the extras files. There are ways of trying to avoid that (like constraining each file by all the files that have already been processed before it in the order given in the make pip-compile target) but the ones I could think of seemed a little overwrought, and come with problems of their own. If a conflict arises, it should be flagged by CI or locally with make check-deps. When/if that happens, you can resolve the conflict by adding appropriate global constraints in requirements/constraints.txt. Also note that if fileA.in is constrained by fileB.txt, then fileB.in should be compiled before fileA.in in the make pip-compile target. Otherwise fileA.in will be compiled with the old version of fileB.txt which can cause conflicts or keep dependencies from being updated properly.
2023-05-24 17:29:35 -05:00
## check-deps: check consistency of dependencies
.PHONY: check-deps
check-deps:
scripts/consistent-deps.sh
adding Dockerfile (#359)
2023-03-14 13:40:01 -07:00
##########
# Docker #
##########
# Docker targets are provided for convenience only and are not required in a standard development environment
chore: build with registry as cache (#454)
2023-04-06 00:34:07 -07:00
DOCKER_IMAGE ?= unstructured:dev
ci: publish amd and arm images (#404)
2023-03-29 00:02:39 -07:00
adding Dockerfile (#359)
2023-03-14 13:40:01 -07:00
.PHONY: docker-build
docker-build:
chore: build with registry as cache (#454)
2023-04-06 00:34:07 -07:00
PIP_VERSION=${PIP_VERSION} DOCKER_IMAGE_NAME=${DOCKER_IMAGE} ./scripts/docker-build.sh
adding Dockerfile (#359)
2023-03-14 13:40:01 -07:00
.PHONY: docker-start-bash
docker-start-bash:
chore: build with registry as cache (#454)
2023-04-06 00:34:07 -07:00
docker run -ti --rm ${DOCKER_IMAGE}
chore: add tests for docker (#373)
2023-03-21 13:46:09 -07:00
dev: add py-spy profiling (#1251) This PR adds a new developer tool for profiling performance: `py-spy`. Additionally it adds a new make command to start a docker with your local `unstructured` repo mounted for quick testing code in a Rocky Linux environment (see usage below for intent). ### py-spy It is a sampling profiler https://github.com/benfred/py-spy and in practice usually provides more readily usable information than commonly used `cProfiler`. It also supports output to `speedscope` format, [which](https://github.com/jlfwong/speedscope#usage) provides a rich view of the profiling result. ### usage The new tool is added to the existing `profile.sh` script and is readily discoverable in the interactive interface. When select to view the new speedscope format profile it would show up in your local browser if you followed the readme to install speedscope locally via `npm install -g speedscope`. On macOS the profiling tool needs superuser privilege. If you are not comfortable with that feel free to run the profiling inside a Linux container if your local dev env is macOS.
2023-08-31 14:26:29 -05:00
.PHONY: docker-start-dev
docker-start-dev:
docker run --rm \
-v ${CURRENT_DIR}:/mnt/local_unstructued \
-ti ${DOCKER_IMAGE}
chore: add tests for docker (#373)
2023-03-21 13:46:09 -07:00
.PHONY: docker-test
docker-test:
chore: build with registry as cache (#454)
2023-04-06 00:34:07 -07:00
docker run --rm \
Set user for container to same as api image. (#1239) This is security best practice, a user can override this with their own Dockerfile if required.
2023-08-29 18:01:44 -07:00
-v ${CURRENT_DIR}/test_unstructured:/home/notebook-user/test_unstructured \
-v ${CURRENT_DIR}/test_unstructured_ingest:/home/notebook-user/test_unstructured_ingest \
fix: use api key where needed in tests (#843) * passes api key for unstructured-api to unit and ingest tests as needed. * adds check for env var CI to otherwise skip tests that require an api key
2023-06-29 10:31:01 -07:00
$(if $(wildcard uns_test_env_file),--env-file uns_test_env_file,) \
ci: publish amd and arm images (#404)
2023-03-29 00:02:39 -07:00
$(DOCKER_IMAGE) \
feat/add sources from unstructured inference (#1538) This PR adds support for `source` property from `unstructured_inference`, allowing the user to be able to see the origin of the data under `detection_origin`field environment variable UNSTRUCTURED_INCLUDE_DEBUG_METADATA=true In order to try this feature you can use this code: ``` from unstructured.partition.pdf import partition_pdf_or_image yolox_elements = partition_pdf_or_image(filename='example-docs/loremipsum-flat.pdf', strategy='hi_res', model_name='yolox') sources = [e.detection_origin for e in yolox_elements] print(sources) ``` And will print 'yolox' as source for all the elements
2023-10-05 15:26:47 -05:00
bash -c "CI=$(CI) \
UNSTRUCTURED_INCLUDE_DEBUG_METADATA=$(UNSTRUCTURED_INCLUDE_DEBUG_METADATA) \
tests: separate chipper tests (#1939) Separates chipper tests to speed up testing and CI.
2023-10-31 16:02:00 -05:00
pytest -m 'not chipper' $(if $(TEST_FILE),$(TEST_FILE),test_unstructured)"
chore: install all extras in Dockerfile (#419) * Adds step to install all extras * Adds smoke test of wikipedia ingest to validate in CI
2023-03-30 13:23:30 -07:00
.PHONY: docker-smoke-test
docker-smoke-test:
fix: docker smoke test on build (#457)
2023-04-06 10:03:42 -07:00
DOCKER_IMAGE=${DOCKER_IMAGE} ./scripts/docker-smoke-test.sh
feat: add jupyter make commands (#651) Co-authored-by: Matt Robinson <mrobinson@unstructured.io>
2023-05-31 17:01:23 +03:00
###########
# Jupyter #
###########
.PHONY: docker-jupyter-notebook
docker-jupyter-notebook:
docker run -p 8888:8888 --mount type=bind,source=$(realpath .),target=/home --entrypoint jupyter-notebook -t --rm ${DOCKER_IMAGE} --allow-root --port 8888 --ip 0.0.0.0 --NotebookApp.token='' --NotebookApp.password=''
.PHONY: run-jupyter
run-jupyter:
feat: Add `stage_for_weaviate` and schema creation function (#672) * add weaviate docker compose * added staging brick and tests for weaviate * initial notebook and requirements file * add commentary to weaviate notebook * weaviate readme * update docs * version and change log * install weaviate client * install weaviate; skip for docker * linting, linting, linting * install weaviate client with deps * comments on weaviate client * fix module not found error for docker container * skipped wrong test in docker * fix typos * add in local-inference
2023-06-01 16:48:54 -04:00
PYTHONPATH=$(realpath .) JUPYTER_PATH=$(realpath .) jupyter-notebook --NotebookApp.token='' --NotebookApp.password=''
Reference in New Issue Copy Permalink