* Add support for two phase checkpoints
Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
* PR #188
* Support timeouts, and retry for GetRecords calls.
Applications can now set timeouts for GetRecord calls to Kinesis. As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
* PR #214
* Notification when the lease table is throttled
When writes, or reads, to the lease table are throttled a warning will be emitted. If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
* PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown. The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
* PR #204
It's now possible to configure GetRecords calls to timeout if they
take to long. This can be used to terminate a long running request to
ensure that record processors continue to make progress
This feature was added with contributions from @pfifer, @sahilpalvia,
and @BtXin.
* Adding logging to ProvisionedThroughputExceededException.
* Addressing CR comments and changing log level from error to warn
* Updated the comments as per cr comments.
Adds support for making two phase commits. When a checkpoint is started, but fails to complete the next instance of the record processor will receive the attempted checkpoint position at initialization time.
Support timeouts for calls to the MultiLang Daemon
This adds support for setting a timeout when dispatching records to
the client record processor. If the record processor doesn't respond
within the timeout the parent Java process will be terminated. This
is a temporary fix to handle cases where the KCL becomes blocked
while waiting for a client record processor.
The timeout for the this can be set by adding
`timeoutInSeconds = <timeout value>`.
The default for this is no timeout.
Setting this can cause the KCL to exit suddenly, before using this
ensure that you have an automated restart for your application
Related #195
Related #185
* Adding timeout to waitForStatusMessage future call. Introducing new config properties timeoutEnabled and timeoutInSeconds. Halting the JVM if timeout is reached.
* Adding test cases for halt jvm code. Made the configuration objects for timeout optional.
* Addressing code review comments and making appropriate changes.
* Execute graceful shutdown on its own thread
* PR #191
* Issue #167
* Added support for controlling the size of the lease renewer thread pool
* PR #177
* Issue #171
* Require Java 8 and later
Java 8 is now required for versions 1.8.0 of the amazon-kinesis-client and later.
* PR #176
* Initial start of fix for requested shutdown
* Execute the requested shutdown in a separate thread.
Fix for Issue #167
* Reworked some of the shutdown logic to make the relationships clearer.
* Added/Updated Copyright Statements
* Add Missing License Statements
Allow configuration of the lease renewer thread pool size. Users can
now control the size of the pool. Additionally core threads are now
a quarter of the thread pool size, and aren't allowed to timeout.
When the parent Java process receives a SIGTERM it will now trigger a graceful shutdown of the worker, which dispatches a ShutdownRequestedMessage to all of the record processors. This will allow record processors a final chance to checkpoint before the lease is terminated.
This changes is incompatible with current versions of the MultiLang Clients. A future change will allow older versions MultiLang clients to use newer versions of the Java KCL.
Improve the handling of describe stream throttling by no longer
triggering a null pointer exception when all requests are throttled.
Also store the last position reached, and always restart from
there.
When the dynamoDBEndpoint configuration property was added, not all paths that
should have made use of it were updated. Both paths now use the config value to
set up the DynamoDB client.
The existing ThreadPoolExecutor was misconfigured and caused
the thread pool to recycle idle threads continuously. VisualVM
showed that a thread got recycled about every 3 seconds, creating
and destroying several hundred threads in about 15 minutes.
Use ExecutorService.newFixedThreadPool instead, as recommended
by the javadoc for ExecutorService.
* Reduce Throttling Error Reports
Only report a throttling error if their are 6 consecutive throttles.
Moved the logging of throttling messages to the throttling
reporter.
* Handle No Lease, and Lease Loss in requestShutdown.
Ensure that the Worker is shutdown if there are no leases. Handle the
case where a lease is lost during the notification process.
* Add some more comments around possible race conditions.
Made the ShardInfo class public, and updated the documentation for a
number of the properties.
This will allow users to implement custom shard prioritization
strategies.
Fixes:
Issue #120
Updated the MultiLangDaemon to use the v2 record processor interfaces, and added features to messages passed to MultiLangDaemon clients.
These changes will require updates to the various MultiLangDaemon clients. The changes for the Python version are complete, and other versions will be updated later.
Add a new method to the worker requestShutdown that allows the worker to
gracefully shutdown all record processors. The graceful shutdown gives
the record processors a last chance to checkpoint before they're
terminated.
To use these new features the record processor must implement
IShutdownnotificationaware.
* Add support for configuring DynamoDB endpoint
Adding a new field named `dynamoDBEndpoint` to the .properties file
that gets passed into the KCL multi-lang daemon. We need this ability
to point the KCL worker at a local instance of DynamoDB rather than in
AWS.
* Added the ability to use AWSCredentialsProvider's that require non-empty contructor args e.g. ProfileCredentialsProvider where you provide the profile name to use from your ~/.aws/credentials file
* Created a constructor without the dynamoDBEndpoint argument i.e. same arguments before the dynamo change, for backwards compatibility
Allow disabling the shard sync at startup if the lease table already
contains leases. This will reduce the startup load for larger streams
when restarted the Kinesis application.
* Add ability to specify different credential providers for Kinesis, DynamoDB, and CloudWatch. This is needed when accessing a cross-account Kineses stream using an assumed role.
* Fix copy/paste mistake.
* Update tests.
Thanks to rgfindl@
Clean up check for ShardIteratorType to prevent emitting a spurious message for every call to GetIterator
Add missing KinesisProxyTest, and add 3 new tests for getIterator.
Don't include checkpoint in hashCode/equality for ShardInfo, since it
changes. Checkpointing would cause the Worker to recreate the
ShardConsumer. Add unit tests that verify the equality constraints.
Remove the equality test for checkpoint from ShardInfoTests. Nothing
appears to rely on the checkpoint being part of ShardInfo.
Fix WorkerTest broken in overzealous simplification.
Added a new interface that allows the worker to prioritize which lease
assignment it will work on next. When using the
ParentsFirstshardprioritization the worker will select parents for
processing before selecting children. This will prevent ShardConsumers
from spending time sleeping in the WAITING_ON_PARENT_SHARDS state.
Added support for time based iterators. Time based iterators are only
used if there is no current checkpoint for that shard, otherwise the
sequence number of the checkpoint is used.
The test 'testWorkerForcefulShutdown' was using Thread.sleep() to create
the conditions for an interrupted state. It was possible for the test
to take enough time for the sleep to actually exit before the interrupt
was sent. This would cause the test to fail. Changing to a pair of
sempahores ensures that the test record processor will remain blocked
for the expected amount of time.
Merge, and slight adjustment, of
https://github.com/awslabs/amazon-kinesis-client/pull/61.
This allows configuration to explicitly set the name of the lease
table. The table name defaults to the application name which is the
current behavior.
Added unit tests for many of the classes in the client. Added new
dependencies for the unit tests to the POM file. Unit tests can be
executed the `surefire:test` goal, and are executed during the test phase.
Added integration tests for the client. The tests require access to
credentials that can interact with Kinesis, and DynamoDB. The
integration tests can be run by executing the
`failsafe:integration-test` goal, or are run as part of the
integration-test phase
**Resources in the integration tests are not currently released on
completion of the integration tests.**
