* Making DynamoDBLeaseCoordinator take IOPs configuration in the constructor
* InitialLeaseTableReadCapacity and InitialLeaseTableWriteCapacity for the DynamoDBLeaseCoordinator class throws UnsupportedException
* Calling onNext and onComplete if throwable is of the kind ResourceNotFound.
* Adding testing for ResourceNotFound
* Updating version to 2.0.1-SNAPSHOT
* Added note about the groupId change
The groupId of the Amazon Kinesis Client changed from com.amazonaws to
software.amazon.kinesis
Fixes#354
* Reorder items, and add message about new API's
Moved the configuration message to just above the configuration.
Added messages about the new API's that the KCL uses.
Fixes#353
* Updated CHANGELOG with the updated release notes
* Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
* Records are now delivered via streaming, reducing end-to-end latency.
* The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
* New configuration options are available to configure Enhanced Fan Out.
* `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
__WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__
| Name | Default | Description |
|-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
| consumerArn | Unset | The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. |
| streamName | Unset | The name of the stream that a consumer should be create for if necessary |
| consumerName | Unset | The name of the consumer to create. If this is not set the applicationName will be used instead. |
| applicationName | Unset | The name of the application. This is used as the name of the consumer unless consumerName is set. |
* Modular Configuration of the Kinesis Client
The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from the `ConfigsBuilder`.
Please see the migration guide for more information
* `CheckpointConfig`
* `CoordinatorConfig`
* `LeaseManagementConfig`
* `LifecycleConfig`
* `MetricsConfig`
* `ProcessorConfig`
* `RetrievalConfig`
* Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
All configurations will only accept 2.0 clients.
* When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
* __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
__When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
Please see the migration guide for more information
* MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`. To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
* Added cache updating behavior for GetShard
Customer are occasionally seeing messages about being unable to
retrieve shard information, which is logged as a warning. This change
will allow the shard map to be updated even when there is no re-shard
operation.
This now triggers a shard list update if there is 1000 cache misses,
or a cache miss occurs when the cache is more than 30 seconds old.
For Kinesis the updates will use ListShards, and for DynamoDB Streams
it will continue to use DescribeStream.
* Adjust some logging, and the zeroing of cache misses a bit
Only log about cache refresh if it's the thread doing the cache
refresh. If after synchronizing the shard is present, accept that
someone else loaded the shard map, and move on.
If the cache was reloaded, and the shard was found the current thread
will reset the cache misses.
The warnings for the cache miss was using a modulo of 1000 which is
the maximum value for cache misses, so wasn't to useful.
* Release 1.9.1 of the Amazon Kinesis Client Library for Java
Updated versions, and added release notes.
* Added note about the update of the AWS Java SDK
* Reformatting the constructors to be together in the Worker class.
* Using lombok annotations to replace setters in the Worker.Builder.
* Deprecating all the public constructors for the Worker class. Recommending to use the Worker.Builder to create the Worker object.
* Including annotation usage for leaseManager in the Worker.Builder
* Adding capability to support Immutable clients by the Worker and Worker.Builder
* Adding annotation support for WorkerStateChangeListener.
* Add a warning about ListShards and IAM Polices
* Corrected some spelling errors
* Add the same warning to the changelog
Added the warning from the README.md to the CHANGELOG.md.
* Added support for ListShards API. This API is used in place of DescribeStream API to provide more throughput during ShardSyncTask.
* ListShards supoorts higher call rate, which should reduce instances of throttling when attempting to synchronize the shard list.
* Updated the version of AWS Java SDK to 1.11.272.
* Version 1.11.272 is now the minimum support versiojn of the SDK.
* Deprecated the following methods, and classes. These methods, and classes will be removed in a future release.
* Deprecated IKinesisProxy#getStreamInfo.
* Deprecated IKinesisProxyFactory.
* Deprecated KinesisProxyFactory.
* Deprecated certain KinesisProxy constructors.
* Allow providing a custom IKinesisProxy implementation.
* PR #274
* Checkpointing on a different thread should no longer emit a warning about NullMetricsScope.
* PR #284
* Issue #48
* Upgraded the AWS Java SDK to version 1.11.271
* PR #287
Fixes#48
* Fixing issue with NullMetrics warning messages when trying to checkpoint on a separate thread.
* Adding testing to validate the MetricsScope setting during checkpoiniting.
* Added IKinesisProxy injector in Worker.Builder to allow injecting custom proxy implementations
* Added unit tests for IKinesisProxy injection in Worker Builder
* Revert "Added unit tests for IKinesisProxy injection in Worker Builder"
This reverts commit aa944c1706.
Reverting to undo changes to import ordering.
* Added unit tests for IKinesisProxy injection in Worker Builder
Re-added unit tests after reverting changes to import ordering.
* Revert "Added unit tests for IKinesisProxy injection in Worker Builder"
This reverts commit 91e445774b.
Reverting to refactor unit tests.
* Added unit tests for Worker Builder IKinesisProxy injection validation
Refactored unit tests as per comments in the pull request.
* Added debug logs in KinesisLocalFileDataCreator
* Revert "Added debug logs in KinesisLocalFileDataCreator"
This reverts commit 1ff00d0b01.
* Edited JavaDoc for Worker Builder kinesisProxy
* Allow disabling check for the case where a child shard has an open parent shard.
There is a race condition where it's possible for the a parent shard
to appear open, while having child shards. This check can now be
disabled by setting ignoreUnexpectedChildShards in the
KinesisClientLibConfiguration to true.
* PR #240
* Issue #210
* Upgraded the AWS SDK for Java to 1.11.261
* PR #281
Allow unexpected child shards to be ignored
now instead of always throwing an assertion if a child shard has an
open parent, consider worker configuration before doing so. if
configured to ignore such shards, do not create leases for them during
shard sync. this is intended to mitigate failing worker init when
processing dynamodb streams with many thousands of shards (which can
happen for tables with thousands of partitions).
this new behavior can be enabled by adding the following to a
configuration/properties file:
```
ignoreUnexpectedChildShards = true
```
Release 1.8.8
* Fixed issues with leases losses due to `ExpiredIteratorException` in `PrefetchGetRecordsCache` and `AsynchronousFetchingStrategy`.
PrefetchGetRecordsCache will request for a new iterator and start fetching data again.
* PR#263
* Added warning message for long running tasks.
* PR#259
* Handling spurious lease renewal failures gracefully.
Added better handling of DynamoDB failures when updating leases. These failures would occur when a request to DynamoDB appeared to fail, but was actually successful.
* PR#247
* ShutdownTask gets retried if the previous attempt on the ShutdownTask fails.
* PR#267
* Fix for using maxRecords from `KinesisClientLibConfiguration` in `GetRecordsCache` for fetching records.
* PR#264
* Shutdown that throws an exception will be retried.
Without this change a transient error on shutdown with reason terminate prevents
child shards from starting.
* Fixing the tests for the Shutdown fix.
Fixes#262
Changing the signture of SingleRecordsFetcherFactory to no longer take maxRecords as the parameter to the constructor. Changed the createRecordsFetcher signature to take maxRecords as a parameter. (#264)
