chore: make Elasticsearch Destination connector write settings optional (#2398)

* set required=False to all write config options
* update num_processes to default to 1 since that will always work
This commit is contained in:
ryannikolaidis 2024-01-14 14:31:05 -08:00 committed by GitHub
parent 2ce829ddd0
commit f07fc6e03a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 8 additions and 7 deletions

View File

@ -1,4 +1,4 @@
## 0.12.1-dev5
## 0.12.1-dev6
### Enhancements
@ -6,6 +6,7 @@
* **Add overlap option for chunking.** Add option to overlap chunks. Intra-chunk and inter-chunk overlap are requested separately. Intra-chunk overlap is applied only to the second and later chunks formed by text-splitting an oversized chunk. Inter-chunk overlap may also be specified; this applies overlap between "normal" (not-oversized) chunks.
* **Salesforce connector accepts private key path or value.** Salesforce parameter `private-key-file` has been renamed to `private-key`. Private key can be provided as path to file or file contents.
* **Add "basic" chunking to ingest CLI.** Add options to ingest CLI allowing access to the new "basic" chunking strategy and overlap options.
* **Make Elasticsearch Destination connector arguments optional.** Elasticsearch Destination connector write settings are made optional and will rely on default values when not specified.
### Features

View File

@ -1 +1 @@
__version__ = "0.12.1-dev5" # pragma: no cover
__version__ = "0.12.1-dev6" # pragma: no cover

View File

@ -92,7 +92,7 @@ class ElasticsearchCliWriteConfig(ElasticsearchWriteConfig, CliConfig):
options = [
click.Option(
["--batch-size-bytes"],
required=True,
required=False,
default=15_000_000,
type=int,
help="Size limit (in bytes) for each batch of items to be uploaded. Check"
@ -101,8 +101,8 @@ class ElasticsearchCliWriteConfig(ElasticsearchWriteConfig, CliConfig):
),
click.Option(
["--num-processes"],
required=True,
default=2,
required=False,
default=1,
type=int,
help="Number of processes to be used while uploading content",
),

View File

@ -308,8 +308,8 @@ class ElasticsearchSourceConnector(SourceConnectorCleanupMixin, BaseSourceConnec
@dataclass
class ElasticsearchWriteConfig(WriteConfig):
batch_size_bytes: int
num_processes: int
batch_size_bytes: int = 15_000_000
num_processes: int = 1
@dataclass