* Changes to support injection of ShardSyncer, LeaseTaker, and LeaseRenewer into KCL Worker
* Additional checks around injection of LeaseRenewer and LeaseRenewerThreadPool
* Changed accessor on InitialPositionInStreamExtended to public to allow ShardSyncer injection
* Changed ShardSyncer to a public interface. Renamed implementation to KinesisShardSyncer.
* Removed wild card imports introduced in previous commit
* Minor refactoring in Worker Builder
* Added license info to ShardSyncer interface. Minor refactoring
* Changes to chain constructor in LeaseCoordinator
* Changed accessor on InitialPositionInStreamExtended factory methods. Minor changes in Worker builder.
* Changes to support periodic shard sync
* Patching changes left out in merge
* Overriding shard-sync idle time to 0 for periodic shard-sync
* Addressed PR feedback
* Addresed PR #579 review comments
* Modified constructor for DeterministicShuffleShardSyncLeaderDecider
* Addressed PR comments
* Fixed failing test
* Removed redundant member varible
* Handle spurious lease renewal failures gracefully.
If the request to conditionally update a lease counter in DynamoDB fails, it's
considered a failure to renew the lease. This is a good thing, except if the
request failure was just because of connectivity problems. In this case the
counter *did* update in DynamoDB, but the Dynamo client retries the request
which then fails the update condition (since the lease counter no longer
matches expected value).
To handle this gracefully we opt to get the lease record from Dynamo and
examine the lease owner and counter. If it matches what we were expecting,
then we consider renewal a success.
* Add support for two phase checkpoints
Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
* PR #188
* Support timeouts, and retry for GetRecords calls.
Applications can now set timeouts for GetRecord calls to Kinesis. As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
* PR #214
* Notification when the lease table is throttled
When writes, or reads, to the lease table are throttled a warning will be emitted. If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
* PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown. The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
* PR #204
* Adding logging to ProvisionedThroughputExceededException.
* Addressing CR comments and changing log level from error to warn
* Updated the comments as per cr comments.
Adds support for making two phase commits. When a checkpoint is started, but fails to complete the next instance of the record processor will receive the attempted checkpoint position at initialization time.
Allow configuration of the lease renewer thread pool size. Users can
now control the size of the pool. Additionally core threads are now
a quarter of the thread pool size, and aren't allowed to timeout.
The existing ThreadPoolExecutor was misconfigured and caused
the thread pool to recycle idle threads continuously. VisualVM
showed that a thread got recycled about every 3 seconds, creating
and destroying several hundred threads in about 15 minutes.
Use ExecutorService.newFixedThreadPool instead, as recommended
by the javadoc for ExecutorService.
Add a new method to the worker requestShutdown that allows the worker to
gracefully shutdown all record processors. The graceful shutdown gives
the record processors a last chance to checkpoint before they're
terminated.
To use these new features the record processor must implement
IShutdownnotificationaware.
Allow disabling the shard sync at startup if the lease table already
contains leases. This will reduce the startup load for larger streams
when restarted the Kinesis application.
