* KCL shardend fix for V1
* Address Comments
* Address more comments
* Force lease lost before shutting down ShardConsumer with Zombie state
* Updating version
* Addressing comments
* Addressing comments
* Fixing unit test
* Addressing comments
* Adding default implementation for onShardConsumerShutDown in ShardSyncStrategy interface
* Method name changes
* Addressing comments
* Addressing comments
* Addressing comments
* Revert the access change for getShardList method
* Changes to support injection of ShardSyncer, LeaseTaker, and LeaseRenewer into KCL Worker
* Additional checks around injection of LeaseRenewer and LeaseRenewerThreadPool
* Changed accessor on InitialPositionInStreamExtended to public to allow ShardSyncer injection
* Changed ShardSyncer to a public interface. Renamed implementation to KinesisShardSyncer.
* Removed wild card imports introduced in previous commit
* Minor refactoring in Worker Builder
* Added license info to ShardSyncer interface. Minor refactoring
* Changes to chain constructor in LeaseCoordinator
* Changed accessor on InitialPositionInStreamExtended factory methods. Minor changes in Worker builder.
* Changes to support periodic shard sync
* Patching changes left out in merge
* Overriding shard-sync idle time to 0 for periodic shard-sync
* Addressed PR feedback
* Addresed PR #579 review comments
* Modified constructor for DeterministicShuffleShardSyncLeaderDecider
* Addressed PR comments
* Fixed failing test
* Removed redundant member varible
* Re-enable metrics for shard-sync
* Removed unused method from ShardSyncTaskManager
* Changes to support injection of ShardSyncer, LeaseTaker, and LeaseRenewer into KCL Worker
* Additional checks around injection of LeaseRenewer and LeaseRenewerThreadPool
* Changed accessor on InitialPositionInStreamExtended to public to allow ShardSyncer injection
* Changed ShardSyncer to a public interface. Renamed implementation to KinesisShardSyncer.
* Removed wild card imports introduced in previous commit
* Minor refactoring in Worker Builder
* Added license info to ShardSyncer interface. Minor refactoring
* Changes to chain constructor in LeaseCoordinator
* Changed accessor on InitialPositionInStreamExtended factory methods. Minor changes in Worker builder.
* Changes to support periodic shard sync
* Patching changes left out in merge
* Overriding shard-sync idle time to 0 for periodic shard-sync
* Addressed PR feedback
* Addresed PR #579 review comments
* Modified constructor for DeterministicShuffleShardSyncLeaderDecider
* Addressed PR comments
* Fixed failing test
* Removed redundant member varible
* Add proxy support
Read proxy info from application.properties file first,
then java system settings, and finally from ENV vars.
* Formatted code according to AWS scheme.
Import specific classes, not *.
* Add proxy config unit tests
* Changed per @sahilpalvia comments
* Fix failing test
* Changed per @sahilpalvia comments
* Fixing missed http_proxy string
* Changed per @sahilpalvia comments
* Added cache updating behavior for GetShard
Customer are occasionally seeing messages about being unable to
retrieve shard information, which is logged as a warning. This change
will allow the shard map to be updated even when there is no re-shard
operation.
This now triggers a shard list update if there is 1000 cache misses,
or a cache miss occurs when the cache is more than 30 seconds old.
For Kinesis the updates will use ListShards, and for DynamoDB Streams
it will continue to use DescribeStream.
* Adjust some logging, and the zeroing of cache misses a bit
Only log about cache refresh if it's the thread doing the cache
refresh. If after synchronizing the shard is present, accept that
someone else loaded the shard map, and move on.
If the cache was reloaded, and the shard was found the current thread
will reset the cache misses.
The warnings for the cache miss was using a modulo of 1000 which is
the maximum value for cache misses, so wasn't to useful.
* Release 1.9.1 of the Amazon Kinesis Client Library for Java
Updated versions, and added release notes.
* Added note about the update of the AWS Java SDK
* Reformatting the constructors to be together in the Worker class.
* Using lombok annotations to replace setters in the Worker.Builder.
* Deprecating all the public constructors for the Worker class. Recommending to use the Worker.Builder to create the Worker object.
* Including annotation usage for leaseManager in the Worker.Builder
* Adding capability to support Immutable clients by the Worker and Worker.Builder
* Adding annotation support for WorkerStateChangeListener.
Fixes#48
* Fixing issue with NullMetrics warning messages when trying to checkpoint on a separate thread.
* Adding testing to validate the MetricsScope setting during checkpoiniting.
* Added IKinesisProxy injector in Worker.Builder to allow injecting custom proxy implementations
* Added unit tests for IKinesisProxy injection in Worker Builder
* Revert "Added unit tests for IKinesisProxy injection in Worker Builder"
This reverts commit aa944c1706.
Reverting to undo changes to import ordering.
* Added unit tests for IKinesisProxy injection in Worker Builder
Re-added unit tests after reverting changes to import ordering.
* Revert "Added unit tests for IKinesisProxy injection in Worker Builder"
This reverts commit 91e445774b.
Reverting to refactor unit tests.
* Added unit tests for Worker Builder IKinesisProxy injection validation
Refactored unit tests as per comments in the pull request.
* Added debug logs in KinesisLocalFileDataCreator
* Revert "Added debug logs in KinesisLocalFileDataCreator"
This reverts commit 1ff00d0b01.
* Edited JavaDoc for Worker Builder kinesisProxy
Allow unexpected child shards to be ignored
now instead of always throwing an assertion if a child shard has an
open parent, consider worker configuration before doing so. if
configured to ignore such shards, do not create leases for them during
shard sync. this is intended to mitigate failing worker init when
processing dynamodb streams with many thousands of shards (which can
happen for tables with thousands of partitions).
this new behavior can be enabled by adding the following to a
configuration/properties file:
```
ignoreUnexpectedChildShards = true
```
Release 1.8.8
* Fixed issues with leases losses due to `ExpiredIteratorException` in `PrefetchGetRecordsCache` and `AsynchronousFetchingStrategy`.
PrefetchGetRecordsCache will request for a new iterator and start fetching data again.
* PR#263
* Added warning message for long running tasks.
* PR#259
* Handling spurious lease renewal failures gracefully.
Added better handling of DynamoDB failures when updating leases. These failures would occur when a request to DynamoDB appeared to fail, but was actually successful.
* PR#247
* ShutdownTask gets retried if the previous attempt on the ShutdownTask fails.
* PR#267
* Fix for using maxRecords from `KinesisClientLibConfiguration` in `GetRecordsCache` for fetching records.
* PR#264
* Shutdown that throws an exception will be retried.
Without this change a transient error on shutdown with reason terminate prevents
child shards from starting.
* Fixing the tests for the Shutdown fix.
Fixes#262
Changing the signture of SingleRecordsFetcherFactory to no longer take maxRecords as the parameter to the constructor. Changed the createRecordsFetcher signature to take maxRecords as a parameter. (#264)
* Handle spurious lease renewal failures gracefully.
If the request to conditionally update a lease counter in DynamoDB fails, it's
considered a failure to renew the lease. This is a good thing, except if the
request failure was just because of connectivity problems. In this case the
counter *did* update in DynamoDB, but the Dynamo client retries the request
which then fails the update condition (since the lease counter no longer
matches expected value).
To handle this gracefully we opt to get the lease record from Dynamo and
examine the lease owner and counter. If it matches what we were expecting,
then we consider renewal a success.
