yujunjun/unstructured
1
0
Fork 0
mirror of https://github.com/Unstructured-IO/unstructured.git synced 2025-11-14 01:17:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: make Elasticsearch Destination connector write settings optional (#2398)

Browse Source 
* set required=False to all write config options
* update num_processes to default to 1 since that will always work
This commit is contained in:
ryannikolaidis 2024-01-14 14:31:05 -08:00 committed by GitHub
parent 2ce829ddd0
commit f07fc6e03a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 8 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@ -1,4 +1,4 @@
## 0.12.1-dev5 ## 0.12.1-dev6
### Enhancements ### Enhancements
@ -6,6 +6,7 @@
* **Add overlap option for chunking.** Add option to overlap chunks. Intra-chunk and inter-chunk overlap are requested separately. Intra-chunk overlap is applied only to the second and later chunks formed by text-splitting an oversized chunk. Inter-chunk overlap may also be specified; this applies overlap between "normal" (not-oversized) chunks. * **Add overlap option for chunking.** Add option to overlap chunks. Intra-chunk and inter-chunk overlap are requested separately. Intra-chunk overlap is applied only to the second and later chunks formed by text-splitting an oversized chunk. Inter-chunk overlap may also be specified; this applies overlap between "normal" (not-oversized) chunks.
* **Salesforce connector accepts private key path or value.** Salesforce parameter `private-key-file` has been renamed to `private-key`. Private key can be provided as path to file or file contents. * **Salesforce connector accepts private key path or value.** Salesforce parameter `private-key-file` has been renamed to `private-key`. Private key can be provided as path to file or file contents.
* **Add "basic" chunking to ingest CLI.** Add options to ingest CLI allowing access to the new "basic" chunking strategy and overlap options. * **Add "basic" chunking to ingest CLI.** Add options to ingest CLI allowing access to the new "basic" chunking strategy and overlap options.
* **Make Elasticsearch Destination connector arguments optional.** Elasticsearch Destination connector write settings are made optional and will rely on default values when not specified.
### Features ### Features

2
unstructured/__version__.py
View File

@ -1 +1 @@
__version__ = "0.12.1-dev5" # pragma: no cover __version__ = "0.12.1-dev6" # pragma: no cover

6
unstructured/ingest/cli/cmds/elasticsearch.py
View File

@ -92,7 +92,7 @@ class ElasticsearchCliWriteConfig(ElasticsearchWriteConfig, CliConfig):
options = [ options = [
click.Option( click.Option(
["--batch-size-bytes"], ["--batch-size-bytes"],
required=True, required=False,
default=15_000_000, default=15_000_000,
type=int, type=int,
help="Size limit (in bytes) for each batch of items to be uploaded. Check" help="Size limit (in bytes) for each batch of items to be uploaded. Check"
@ -101,8 +101,8 @@ class ElasticsearchCliWriteConfig(ElasticsearchWriteConfig, CliConfig):
), ),
click.Option( click.Option(
["--num-processes"], ["--num-processes"],
required=True, required=False,
default=2, default=1,
type=int, type=int,
help="Number of processes to be used while uploading content", help="Number of processes to be used while uploading content",
), ),

4
unstructured/ingest/connector/elasticsearch.py
View File

@ -308,8 +308,8 @@ class ElasticsearchSourceConnector(SourceConnectorCleanupMixin, BaseSourceConnec
@dataclass @dataclass
class ElasticsearchWriteConfig(WriteConfig): class ElasticsearchWriteConfig(WriteConfig):
batch_size_bytes: int batch_size_bytes: int = 15_000_000
num_processes: int num_processes: int = 1
@dataclass @dataclass