mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-08-17 05:02:25 +00:00
b7674fb97e
* Add confluence connector and an example script * add test script, add dependency installations * add authentication secret variables for ci tests and actions * add dependency installation commands for workflows * add dependency installation commands for workflows * Update ingest test fixtures (#907) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add add ingest test fixtures update workflow for python 3.10, update example script with dummy values * change workflow name to avoid confusion * change workflow name to avoid confusion * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions, remove 3.10 workflow for the test fixtures update * only leave 3.8 in ingest test matrix to test consistent partitioning among python versions * Update ingest test fixtures (#911) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * revert back the test python version matrix * recompile dependencies * modifications for shellcheck * update changelog and version * changelog and version * remove comments * Update ingest test fixtures (#915) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * add the option to state the number of spaces to be fetched * add scroll functionality, expose --confluence-num-of-spaces, --confluence-list-of-spaces and --confluence-num-of-docs-from-each-space to users * add help message * add docstrings for two tests, validate grabbing every doc in the fetched spaces, count number of files instead of diffing for confluence2 test * change test names * rename connector arg Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * change arg name for connector Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> * add comment to example * change arg names * add new tests to ingest test * shellcheck remove redundant statement * Update ingest test fixtures (#932) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * Update ingest test fixtures (#936) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * linting * change file extensions to parse as html * Update ingest test fixtures (#943) Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com> * remove old fixtures * update version to 0.8.2-dev3 * change file to trigger CI * change file to trigger CI * change file to trigger CI * change file to trigger CI --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
27 lines
737 B
Docker
27 lines
737 B
Docker
# Dockerfile that approximates the CI image
|
|
#
|
|
# Mainly useful for updating test-ingest fixtures
|
|
|
|
FROM ubuntu:22.04
|
|
|
|
COPY scripts/setup_ubuntu.sh scripts/setup_ubuntu.sh
|
|
|
|
RUN bash scripts/setup_ubuntu.sh root
|
|
|
|
COPY requirements/ requirements/
|
|
COPY Makefile Makefile
|
|
|
|
SHELL ["/bin/bash", "-c"]
|
|
|
|
RUN source ~/.bashrc && pyenv virtualenv 3.8.17 unstructured && \
|
|
source ~/.pyenv/versions/unstructured/bin/activate && \
|
|
make install-ci && \
|
|
make install-ingest-s3 && \
|
|
make install-ingest-azure && \
|
|
make install-ingest-github && \
|
|
make install-ingest-gitlab && \
|
|
make install-ingest-wikipedia && \
|
|
make install-ingest-discord && \
|
|
make install install-ingest-slack && \
|
|
make install-ingest-confluence
|