Fix an issue where the idleTimeBetweenReadInMillis configured
via MultiLangDaemon was not taking effect because it used
the auto-generated setter from Lombok to set the configured value,
while there is a custom setter that must be invoked to set the
value correctly.
There is also a general confusion between using Lombok's setter vs
custom setter in java.
Unifying the approach to use the custom Lombok-fluent-style setter
and deprecating the previously added custom setIdleTimeBetweenReadsInMillis
Correct way to configure idleTimeBetweenReadsInMillis for MultiLang is
to add this in the properties file:
idleTimeBetweenReadsInMillis = 10000 # 10 seconds
Correct way to configure for java:
configsBuilder.retrievalConfig().retrievalSpecificConfig(
new PollingConfig(streamName, kinesisClient)
.idleTimeBetweenReadsInMillis(Duration.ofSeconds(10).toMillis())
Issues: #999, #950, #515
* [#367] Enhanced multi-lang `AWSCredentialsProvider=...` decoder and construction.
+ added support for external ids (issue #367)
+ added support for endpoint+region (e.g., STS via VPC)
* Multiple multi-lang edits to introduce logging and additional tests.
+ added `ENDPOINT_REGION` nested key for a simpler Cx experience
+ deduplicated, and improved, logic w.r.t. CredentialsProvider
construction to NOT swallow Exceptions
* Relocated `multilang.properties` from `main/resources` to `test/resources`
* Updated multilang to support streamArn
* Updated arn import to use software.amzon instead of com.amazonaws, also updated unit tests to be more explicit with the expected exceptions
* Updated exception wording for region validation in StreamArn to be more consistent with other error messages
* reverted spacing change
* Updated StreamArn in multilang to only replace streamName (not region as well). Also updated unit tests and added Region validation
* Updated region validation in multilang to be more readible
* Refactored multilang unit tests to be more simple
* Updated multilang daemon to validate streamArn based on pattern rather than individual section
* removed region validation as this was not a requirement for stringArn support in multilangdaemon
* removed spacing and removed unit test assertion on exception message
* removed unnecessary param from unit test
* removed unused imports from multilang unit tests
* simplified the assertion for multilang daemon unit tests
* Cleaned up unit test code following best practices for spacing/naming conventions and simplied kinesisClientLibConfiguration
* Updated region code in unit tests for multilang daemon
---------
Co-authored-by: Ryan Pelaez <rmpelaez@amazon.com>
The AWS SDK depends on being able to find which region it should
operate in. In many cases this is either explicitly set or retrieved
from a variety of sources. For these test cases we don't actually
care what the region. To ensure that the tests operate as expected the
region is set before the test runs, and reset upon completion.
* Introducing MultiLangDaemon support for Enhanced Fan-Out.
* MultiLangDaemon now supports the following command line options.
* `--properties-file`: Properties file that the KCL should use to set up the Scheduler.
* `--log-configuration`: logback.xml that the KCL should use for logging.
* Updated AWS SDK dependency to 2.2.0.
* MultiLangDaemon now uses logback for logging.
* Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
* Records are now delivered via streaming, reducing end-to-end latency.
* The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
* New configuration options are available to configure Enhanced Fan Out.
* `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
__WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__
| Name | Default | Description |
|-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
| consumerArn | Unset | The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. |
| streamName | Unset | The name of the stream that a consumer should be create for if necessary |
| consumerName | Unset | The name of the consumer to create. If this is not set the applicationName will be used instead. |
| applicationName | Unset | The name of the application. This is used as the name of the consumer unless consumerName is set. |
* Modular Configuration of the Kinesis Client
The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from the `ConfigsBuilder`.
Please see the migration guide for more information
* `CheckpointConfig`
* `CoordinatorConfig`
* `LeaseManagementConfig`
* `LifecycleConfig`
* `MetricsConfig`
* `ProcessorConfig`
* `RetrievalConfig`
* Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
All configurations will only accept 2.0 clients.
* When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
* __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
__When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
Please see the migration guide for more information
* MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`. To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
