unstructured/examples/ingest/sharepoint/ingest.sh

#!/usr/bin/env bash

# Processes the Unstructured-IO/unstructured repository
# through Unstructured's library in 2 processes.

# Structured outputs are stored in sharepoint-ingest-output/

# NOTE, this script is not ready-to-run!
# You must enter a MS Sharepoint app client-id, client secret and sharepoint site url
# before running.

# To get the credentials for your Sharepoint app, follow these steps:
# https://github.com/vgrem/Office365-REST-Python-Client/wiki/How-to-connect-to-SharePoint-Online-and-and-SharePoint-2013-2016-2019-on-premises--with-app-principal

# To optionally set up your application and obtain permissions related variables (--permissions-application-id, --permissions-client-cred, --permissions-tenant), follow these steps:
# https://tsmatz.wordpress.com/2016/10/07/application-permission-with-v2-endpoint-and-microsoft-graph

SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
cd "$SCRIPT_DIR"/../../.. || exit 1

PYTHONPATH=. ./unstructured/ingest/main.py \
  sharepoint \
  --client-id "<Microsoft Sharepoint app client-id>" \
  --client-cred "<Microsoft Sharepoint app client-secret>" \
  --site "<e.g https://contoso.sharepoint.com or https://contoso.admin.sharepoint.com to process all sites within tenant>" \
  --permissions-application-id "<Microsoft Graph API application id to process per-file access permissions>" \
  --permissions-client-cred "<Microsoft Graph API application credentials to process per-file access permissions>" \
  --permissions-tenant "<e.g https://contoso.onmicrosoft.com to process per-file access permissions>" \
  --files-only "Flag to process only files within the site(s)" \
  --output-dir sharepoint-ingest-output \
  --num-processes 2 \
  --path "Shared Documents" \
  --verbose
-												feat: Sharepoint connector (#918)


											
										
										
											2023-08-10 10:37:58 -06:00
+								#!/usr/bin/env bash
 								# Processes the Unstructured-IO/unstructured repository
 								# through Unstructured's library in 2 processes.
 								# Structured outputs are stored in sharepoint-ingest-output/
 								# NOTE, this script is not ready-to-run!
 								# You must enter a MS Sharepoint app client-id, client secret and sharepoint site url
-												Roman/downstream connector cli subcommand (#1302)

### Description
Update all other connectors to use the new downstream architecture that
was recently introduced for the s3 connector.

Closes #1313 and #1311
											
										
										
											2023-09-11 11:40:56 -04:00
+								# before running.
-												feat: Sharepoint connector (#918)


											
										
										
											2023-08-10 10:37:58 -06:00
 								# To get the credentials for your Sharepoint app, follow these steps:
 								# https://github.com/vgrem/Office365-REST-Python-Client/wiki/How-to-connect-to-SharePoint-Online-and-and-SharePoint-2013-2016-2019-on-premises--with-app-principal
-												feat: add file-based access permissions for SharePoint ingest (#1628)

This PR:

- defines rbac_data as a SourceMetadata field,
- manages connections to an external api for obtaining rbac data with
ConnectorRBAC class,
- serializes rbac data and saves it to the disk,
- matches the rbac_data in the disk to each IngestDoc, using a common
field,
- forwards rbac data to Elements, via the partition() function

To test the changes, run `examples/ingest/sharepoint/ingest.sh` with the
relevant rbac & connector credentials

---------

Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: ahmetmeleq <ahmetmeleq@users.noreply.github.com>
											
										
										
											2023-10-13 01:38:08 +01:00
+								# To optionally set up your application and obtain permissions related variables (--permissions-application-id, --permissions-client-cred, --permissions-tenant), follow these steps:
 								# https://tsmatz.wordpress.com/2016/10/07/application-permission-with-v2-endpoint-and-microsoft-graph
-												feat: Sharepoint connector (#918)


											
										
										
											2023-08-10 10:37:58 -06:00
-												chore: add shfmt (#2246)

### Description
Given all the shell files that now exist in the repo, would be nice to
have linting/formatting around them (in addition to the existing
shellcheck which doesn't do anything to format the shell code). This PR
introduces `shfmt` to both check for changes and apply formatting when
the associated make targets are called.
											
										
										
											2023-12-11 20:04:15 -05:00
+								SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
-												feat: Sharepoint connector (#918)


											
										
										
											2023-08-10 10:37:58 -06:00
+								cd "$SCRIPT_DIR"/../../.. || exit 1
 								PYTHONPATH=. ./unstructured/ingest/main.py \
-												chore: shell scripts default indent of 2 instead of 4 (#2287)

Given the tendency for shell scripts to easily enter into a few levels
of indentation and long line lengths, update the default to 2 spaces.
											
										
										
											2023-12-18 23:48:21 -08:00
+								  sharepoint \
 								  --client-id "<Microsoft Sharepoint app client-id>" \
 								  --client-cred "<Microsoft Sharepoint app client-secret>" \
 								  --site "<e.g https://contoso.sharepoint.com or https://contoso.admin.sharepoint.com to process all sites within tenant>" \
 								  --permissions-application-id "<Microsoft Graph API application id to process per-file access permissions>" \
 								  --permissions-client-cred "<Microsoft Graph API application credentials to process per-file access permissions>" \
 								  --permissions-tenant "<e.g https://contoso.onmicrosoft.com to process per-file access permissions>" \
 								  --files-only "Flag to process only files within the site(s)" \
 								  --output-dir sharepoint-ingest-output \
 								  --num-processes 2 \
 								  --path "Shared Documents" \
 								  --verbose