mirror of
https://github.com/Unstructured-IO/unstructured.git
synced 2025-11-01 10:33:09 +00:00
chore: make Elasticsearch Destination connector write settings optional (#2398)
* set required=False to all write config options * update num_processes to default to 1 since that will always work
This commit is contained in:
parent
2ce829ddd0
commit
f07fc6e03a
@ -1,4 +1,4 @@
|
||||
## 0.12.1-dev5
|
||||
## 0.12.1-dev6
|
||||
|
||||
### Enhancements
|
||||
|
||||
@ -6,6 +6,7 @@
|
||||
* **Add overlap option for chunking.** Add option to overlap chunks. Intra-chunk and inter-chunk overlap are requested separately. Intra-chunk overlap is applied only to the second and later chunks formed by text-splitting an oversized chunk. Inter-chunk overlap may also be specified; this applies overlap between "normal" (not-oversized) chunks.
|
||||
* **Salesforce connector accepts private key path or value.** Salesforce parameter `private-key-file` has been renamed to `private-key`. Private key can be provided as path to file or file contents.
|
||||
* **Add "basic" chunking to ingest CLI.** Add options to ingest CLI allowing access to the new "basic" chunking strategy and overlap options.
|
||||
* **Make Elasticsearch Destination connector arguments optional.** Elasticsearch Destination connector write settings are made optional and will rely on default values when not specified.
|
||||
|
||||
### Features
|
||||
|
||||
|
||||
@ -1 +1 @@
|
||||
__version__ = "0.12.1-dev5" # pragma: no cover
|
||||
__version__ = "0.12.1-dev6" # pragma: no cover
|
||||
|
||||
@ -92,7 +92,7 @@ class ElasticsearchCliWriteConfig(ElasticsearchWriteConfig, CliConfig):
|
||||
options = [
|
||||
click.Option(
|
||||
["--batch-size-bytes"],
|
||||
required=True,
|
||||
required=False,
|
||||
default=15_000_000,
|
||||
type=int,
|
||||
help="Size limit (in bytes) for each batch of items to be uploaded. Check"
|
||||
@ -101,8 +101,8 @@ class ElasticsearchCliWriteConfig(ElasticsearchWriteConfig, CliConfig):
|
||||
),
|
||||
click.Option(
|
||||
["--num-processes"],
|
||||
required=True,
|
||||
default=2,
|
||||
required=False,
|
||||
default=1,
|
||||
type=int,
|
||||
help="Number of processes to be used while uploading content",
|
||||
),
|
||||
|
||||
@ -308,8 +308,8 @@ class ElasticsearchSourceConnector(SourceConnectorCleanupMixin, BaseSourceConnec
|
||||
|
||||
@dataclass
|
||||
class ElasticsearchWriteConfig(WriteConfig):
|
||||
batch_size_bytes: int
|
||||
num_processes: int
|
||||
batch_size_bytes: int = 15_000_000
|
||||
num_processes: int = 1
|
||||
|
||||
|
||||
@dataclass
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user