* Execute graceful shutdown on its own thread
* PR #191
* Issue #167
* Added support for controlling the size of the lease renewer thread pool
* PR #177
* Issue #171
* Require Java 8 and later
Java 8 is now required for versions 1.8.0 of the amazon-kinesis-client and later.
* PR #176
* Initial start of fix for requested shutdown
* Execute the requested shutdown in a separate thread.
Fix for Issue #167
* Reworked some of the shutdown logic to make the relationships clearer.
* Added/Updated Copyright Statements
* Add Missing License Statements
Allow configuration of the lease renewer thread pool size. Users can
now control the size of the pool. Additionally core threads are now
a quarter of the thread pool size, and aren't allowed to timeout.
* Advance version, and drop Java 7 support
Advanced the version to 1.8.0 as Java 7 support is being removed.
* Remove build settings for Java 7
Removed the configuration for Java 7, and switched to trusty for access
to openjdk8.
* Added support for graceful shutdown in MultiLang Clients
* PR #174
* PR #182
* Updated documentation for `v2.IRecordProcessor#shutdown`, and `KinesisClientLibConfiguration#idleTimeBetweenReadsMillis`
* PR #170
* Updated to version 1.11.151 of the AWS Java SDK
* PR #183
When the parent Java process receives a SIGTERM it will now trigger a graceful shutdown of the worker, which dispatches a ShutdownRequestedMessage to all of the record processors. This will allow record processors a final chance to checkpoint before the lease is terminated.
This changes is incompatible with current versions of the MultiLang Clients. A future change will allow older versions MultiLang clients to use newer versions of the Java KCL.
* Correctly handle throttling for DescribeStream, and save accumulated progress from individual calls.
* PR #152
* Upgrade to version 1.11.115 of the AWS Java SDK
* PR #155
Improve the handling of describe stream throttling by no longer
triggering a null pointer exception when all requests are throttled.
Also store the last position reached, and always restart from
there.
When the dynamoDBEndpoint configuration property was added, not all paths that
should have made use of it were updated. Both paths now use the config value to
set up the DynamoDB client.
The existing ThreadPoolExecutor was misconfigured and caused
the thread pool to recycle idle threads continuously. VisualVM
showed that a thread got recycled about every 3 seconds, creating
and destroying several hundred threads in about 15 minutes.
Use ExecutorService.newFixedThreadPool instead, as recommended
by the javadoc for ExecutorService.
Disable the JavaDoc doclint when building with Java 8. Java 8 became
much more pedantic on bad practices, and the documentation hasn't been
cleaned up.
* Reduce Throttling Error Reports
Only report a throttling error if their are 6 consecutive throttles.
Moved the logging of throttling messages to the throttling
reporter.
* Handle No Lease, and Lease Loss in requestShutdown.
Ensure that the Worker is shutdown if there are no leases. Handle the
case where a lease is lost during the notification process.
* Add some more comments around possible race conditions.
Made the ShardInfo class public, and updated the documentation for a
number of the properties.
This will allow users to implement custom shard prioritization
strategies.
Fixes:
Issue #120
Updated to the newest version of the AWS Java SDK.
Fixes:
Amazon Kinesis Client for Python Issue awslabs/amazon-kinesis-client-python#27
Related:
PR #125
* Add Direct Dependency on Commons Log
Commons log was a transient dependency, but should have been a direct
dependency. Now take a direct dependency on it.
* Move to Next Release Version Snapshot
Changed version to 1.7.3-SNAPSHOT for next release.
MultiLangDaemon Feature Updates
The MultiLangDaemon has been upgraded to use the v2 interfaces, which allows access to enhanced checkpointing, and more information during record processor initialization. The MultiLangDaemon clients must be updated before they can take advantage of these new features.
Updated the MultiLangDaemon to use the v2 record processor interfaces, and added features to messages passed to MultiLangDaemon clients.
These changes will require updates to the various MultiLangDaemon clients. The changes for the Python version are complete, and other versions will be updated later.
Add a new method to the worker requestShutdown that allows the worker to
gracefully shutdown all record processors. The graceful shutdown gives
the record processors a last chance to checkpoint before they're
terminated.
To use these new features the record processor must implement
IShutdownnotificationaware.
* Add support for configuring DynamoDB endpoint
Adding a new field named `dynamoDBEndpoint` to the .properties file
that gets passed into the KCL multi-lang daemon. We need this ability
to point the KCL worker at a local instance of DynamoDB rather than in
AWS.
* Added the ability to use AWSCredentialsProvider's that require non-empty contructor args e.g. ProfileCredentialsProvider where you provide the profile name to use from your ~/.aws/credentials file
* Created a constructor without the dynamoDBEndpoint argument i.e. same arguments before the dynamo change, for backwards compatibility
Allow disabling the shard sync at startup if the lease table already
contains leases. This will reduce the startup load for larger streams
when restarted the Kinesis application.
* Add ability to specify different credential providers for Kinesis, DynamoDB, and CloudWatch. This is needed when accessing a cross-account Kineses stream using an assumed role.
* Fix copy/paste mistake.
* Update tests.
Thanks to rgfindl@
Clean up check for ShardIteratorType to prevent emitting a spurious message for every call to GetIterator
Add missing KinesisProxyTest, and add 3 new tests for getIterator.
Don't include checkpoint in hashCode/equality for ShardInfo, since it
changes. Checkpointing would cause the Worker to recreate the
ShardConsumer. Add unit tests that verify the equality constraints.
Remove the equality test for checkpoint from ShardInfoTests. Nothing
appears to rely on the checkpoint being part of ShardInfo.
Fix WorkerTest broken in overzealous simplification.
Added a new interface that allows the worker to prioritize which lease
assignment it will work on next. When using the
ParentsFirstshardprioritization the worker will select parents for
processing before selecting children. This will prevent ShardConsumers
from spending time sleeping in the WAITING_ON_PARENT_SHARDS state.
