Updated the MultiLangDaemon to use the v2 record processor interfaces, and added features to messages passed to MultiLangDaemon clients.
These changes will require updates to the various MultiLangDaemon clients. The changes for the Python version are complete, and other versions will be updated later.
Add a new method to the worker requestShutdown that allows the worker to
gracefully shutdown all record processors. The graceful shutdown gives
the record processors a last chance to checkpoint before they're
terminated.
To use these new features the record processor must implement
IShutdownnotificationaware.
* Add support for configuring DynamoDB endpoint
Adding a new field named `dynamoDBEndpoint` to the .properties file
that gets passed into the KCL multi-lang daemon. We need this ability
to point the KCL worker at a local instance of DynamoDB rather than in
AWS.
* Added the ability to use AWSCredentialsProvider's that require non-empty contructor args e.g. ProfileCredentialsProvider where you provide the profile name to use from your ~/.aws/credentials file
* Created a constructor without the dynamoDBEndpoint argument i.e. same arguments before the dynamo change, for backwards compatibility
Allow disabling the shard sync at startup if the lease table already
contains leases. This will reduce the startup load for larger streams
when restarted the Kinesis application.
* Add ability to specify different credential providers for Kinesis, DynamoDB, and CloudWatch. This is needed when accessing a cross-account Kineses stream using an assumed role.
* Fix copy/paste mistake.
* Update tests.
Thanks to rgfindl@
Clean up check for ShardIteratorType to prevent emitting a spurious message for every call to GetIterator
Add missing KinesisProxyTest, and add 3 new tests for getIterator.
Don't include checkpoint in hashCode/equality for ShardInfo, since it
changes. Checkpointing would cause the Worker to recreate the
ShardConsumer. Add unit tests that verify the equality constraints.
Remove the equality test for checkpoint from ShardInfoTests. Nothing
appears to rely on the checkpoint being part of ShardInfo.
Fix WorkerTest broken in overzealous simplification.
Added a new interface that allows the worker to prioritize which lease
assignment it will work on next. When using the
ParentsFirstshardprioritization the worker will select parents for
processing before selecting children. This will prevent ShardConsumers
from spending time sleeping in the WAITING_ON_PARENT_SHARDS state.
Added support for time based iterators. Time based iterators are only
used if there is no current checkpoint for that shard, otherwise the
sequence number of the checkpoint is used.
Merge, and slight adjustment, of
https://github.com/awslabs/amazon-kinesis-client/pull/61.
This allows configuration to explicitly set the name of the lease
table. The table name defaults to the application name which is the
current behavior.
