* Moved sequence number validation to an experimental feature
Moved the sequence number validation to become an experimental feature
that can be removed in the future.
Added an annotation for experimental features.
* Delete merge conflict again?
* Add some reminder that this stuff is experimental
* Added a reason field, and some reasons
Added a reason value to the annotation, and updated two of the unusual places.
* Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
* Records are now delivered via streaming, reducing end-to-end latency.
* The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
* New configuration options are available to configure Enhanced Fan Out.
* `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
__WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__
| Name | Default | Description |
|-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
| consumerArn | Unset | The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. |
| streamName | Unset | The name of the stream that a consumer should be create for if necessary |
| consumerName | Unset | The name of the consumer to create. If this is not set the applicationName will be used instead. |
| applicationName | Unset | The name of the application. This is used as the name of the consumer unless consumerName is set. |
* Modular Configuration of the Kinesis Client
The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from the `ConfigsBuilder`.
Please see the migration guide for more information
* `CheckpointConfig`
* `CoordinatorConfig`
* `LeaseManagementConfig`
* `LifecycleConfig`
* `MetricsConfig`
* `ProcessorConfig`
* `RetrievalConfig`
* Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
All configurations will only accept 2.0 clients.
* When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
* __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
__When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
Please see the migration guide for more information
* MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`. To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
* Added cache updating behavior for GetShard
Customer are occasionally seeing messages about being unable to
retrieve shard information, which is logged as a warning. This change
will allow the shard map to be updated even when there is no re-shard
operation.
This now triggers a shard list update if there is 1000 cache misses,
or a cache miss occurs when the cache is more than 30 seconds old.
For Kinesis the updates will use ListShards, and for DynamoDB Streams
it will continue to use DescribeStream.
* Adjust some logging, and the zeroing of cache misses a bit
Only log about cache refresh if it's the thread doing the cache
refresh. If after synchronizing the shard is present, accept that
someone else loaded the shard map, and move on.
If the cache was reloaded, and the shard was found the current thread
will reset the cache misses.
The warnings for the cache miss was using a modulo of 1000 which is
the maximum value for cache misses, so wasn't to useful.
Fixes#48
* Fixing issue with NullMetrics warning messages when trying to checkpoint on a separate thread.
* Adding testing to validate the MetricsScope setting during checkpoiniting.
* Added IKinesisProxy injector in Worker.Builder to allow injecting custom proxy implementations
* Added unit tests for IKinesisProxy injection in Worker Builder
* Revert "Added unit tests for IKinesisProxy injection in Worker Builder"
This reverts commit aa944c1706.
Reverting to undo changes to import ordering.
* Added unit tests for IKinesisProxy injection in Worker Builder
Re-added unit tests after reverting changes to import ordering.
* Revert "Added unit tests for IKinesisProxy injection in Worker Builder"
This reverts commit 91e445774b.
Reverting to refactor unit tests.
* Added unit tests for Worker Builder IKinesisProxy injection validation
Refactored unit tests as per comments in the pull request.
* Added debug logs in KinesisLocalFileDataCreator
* Revert "Added debug logs in KinesisLocalFileDataCreator"
This reverts commit 1ff00d0b01.
* Edited JavaDoc for Worker Builder kinesisProxy
Allow unexpected child shards to be ignored
now instead of always throwing an assertion if a child shard has an
open parent, consider worker configuration before doing so. if
configured to ignore such shards, do not create leases for them during
shard sync. this is intended to mitigate failing worker init when
processing dynamodb streams with many thousands of shards (which can
happen for tables with thousands of partitions).
this new behavior can be enabled by adding the following to a
configuration/properties file:
```
ignoreUnexpectedChildShards = true
```
* Shutdown that throws an exception will be retried.
Without this change a transient error on shutdown with reason terminate prevents
child shards from starting.
* Fixing the tests for the Shutdown fix.
Fixes#262
Changing the signture of SingleRecordsFetcherFactory to no longer take maxRecords as the parameter to the constructor. Changed the createRecordsFetcher signature to take maxRecords as a parameter. (#264)
* Handle spurious lease renewal failures gracefully.
If the request to conditionally update a lease counter in DynamoDB fails, it's
considered a failure to renew the lease. This is a good thing, except if the
request failure was just because of connectivity problems. In this case the
counter *did* update in DynamoDB, but the Dynamo client retries the request
which then fails the update condition (since the lease counter no longer
matches expected value).
To handle this gracefully we opt to get the lease record from Dynamo and
examine the lease owner and counter. If it matches what we were expecting,
then we consider renewal a success.
This adds that ability for the KCL to fetch records while the record processor is busy. This can help smooth out delays in record process, or retrieving data from Kinesis. Enabling this does require extra threads for background retrieval.
Settings
* dataFetchingStrategy: Which strategy to use to retrieve records. This can be either DEFAULT or PREFETCH_CACHED
* maxCacheByteSize: Retrieval will be paused when the total number of bytes in the cache exceeds this value
* maxRecordsCount: Retrieval will be paused when the total number of records in the cache exceeds this value
* maxPendingProcessRecordsInput: Retrieval will be paused when the total number of fulfilled requests in the cache exceeds this value
This changes the retriever strategy to only accept the shard iterator
when we have accepted a result to return. This is for the
asynchronous retriever where multiple threads may contend for the same
iterator slot. This ensures only the one selected for the response will
advance the shard iterator.
* Add support for two phase checkpoints
Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
* PR #188
* Support timeouts, and retry for GetRecords calls.
Applications can now set timeouts for GetRecord calls to Kinesis. As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
* PR #214
* Notification when the lease table is throttled
When writes, or reads, to the lease table are throttled a warning will be emitted. If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
* PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown. The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
* PR #204
It's now possible to configure GetRecords calls to timeout if they
take to long. This can be used to terminate a long running request to
ensure that record processors continue to make progress
This feature was added with contributions from @pfifer, @sahilpalvia,
and @BtXin.
Adds support for making two phase commits. When a checkpoint is started, but fails to complete the next instance of the record processor will receive the attempted checkpoint position at initialization time.
* Adding timeout to waitForStatusMessage future call. Introducing new config properties timeoutEnabled and timeoutInSeconds. Halting the JVM if timeout is reached.
* Adding test cases for halt jvm code. Made the configuration objects for timeout optional.
* Addressing code review comments and making appropriate changes.
* Initial start of fix for requested shutdown
* Execute the requested shutdown in a separate thread.
Fix for Issue #167
* Reworked some of the shutdown logic to make the relationships clearer.
* Added/Updated Copyright Statements
* Add Missing License Statements
When the parent Java process receives a SIGTERM it will now trigger a graceful shutdown of the worker, which dispatches a ShutdownRequestedMessage to all of the record processors. This will allow record processors a final chance to checkpoint before the lease is terminated.
This changes is incompatible with current versions of the MultiLang Clients. A future change will allow older versions MultiLang clients to use newer versions of the Java KCL.
Improve the handling of describe stream throttling by no longer
triggering a null pointer exception when all requests are throttled.
Also store the last position reached, and always restart from
there.
* Reduce Throttling Error Reports
Only report a throttling error if their are 6 consecutive throttles.
Moved the logging of throttling messages to the throttling
reporter.
* Handle No Lease, and Lease Loss in requestShutdown.
Ensure that the Worker is shutdown if there are no leases. Handle the
case where a lease is lost during the notification process.
* Add some more comments around possible race conditions.
Updated the MultiLangDaemon to use the v2 record processor interfaces, and added features to messages passed to MultiLangDaemon clients.
These changes will require updates to the various MultiLangDaemon clients. The changes for the Python version are complete, and other versions will be updated later.
Add a new method to the worker requestShutdown that allows the worker to
gracefully shutdown all record processors. The graceful shutdown gives
the record processors a last chance to checkpoint before they're
terminated.
To use these new features the record processor must implement
IShutdownnotificationaware.
* Add support for configuring DynamoDB endpoint
Adding a new field named `dynamoDBEndpoint` to the .properties file
that gets passed into the KCL multi-lang daemon. We need this ability
to point the KCL worker at a local instance of DynamoDB rather than in
AWS.
* Added the ability to use AWSCredentialsProvider's that require non-empty contructor args e.g. ProfileCredentialsProvider where you provide the profile name to use from your ~/.aws/credentials file
* Created a constructor without the dynamoDBEndpoint argument i.e. same arguments before the dynamo change, for backwards compatibility
Allow disabling the shard sync at startup if the lease table already
contains leases. This will reduce the startup load for larger streams
when restarted the Kinesis application.
* Add ability to specify different credential providers for Kinesis, DynamoDB, and CloudWatch. This is needed when accessing a cross-account Kineses stream using an assumed role.
* Fix copy/paste mistake.
* Update tests.
Thanks to rgfindl@
Clean up check for ShardIteratorType to prevent emitting a spurious message for every call to GetIterator
Add missing KinesisProxyTest, and add 3 new tests for getIterator.
Don't include checkpoint in hashCode/equality for ShardInfo, since it
changes. Checkpointing would cause the Worker to recreate the
ShardConsumer. Add unit tests that verify the equality constraints.
Remove the equality test for checkpoint from ShardInfoTests. Nothing
appears to rely on the checkpoint being part of ShardInfo.
Fix WorkerTest broken in overzealous simplification.
