* SequenceNumberValidator for verifying shardId's
Added a SequenceNumberValidator that will can extract, and verify
shardId's from a v2 sequence number.
* Added documentation and bit length test
Added documentation for the public methods.
Added a bit length test for the reader that will reject sequence
numbers that don't fit the expectations.
* Added more comments and further document public operations
Added comments in the only SequenceNumberReader explaining how things
are expected to work.
Further documented the class and operations with expectations and outcomes.
* Added configuration to allow failing on mismatched records
Allow configuration which will cause the FanOutRecordsPublisher to
throw an exception when it detects records that aren't for the shard
it's processing.
* Some debug logging to understand mismatched sequence numbers
Added some logging messages for sequence numbers when starting up.
* Added debug support and logging
Added @ToString to InitialPositionInStreamExtended for debugging
purposes.
Added a debug log about the initialization of the
FanOutRecordsPublisher to ensure that the publisher is being
initialized as expected.
Changed the CoordinatorFactory methods to be default methods. This
will not require an implementer to implement the deprecated methods.
Updated the documentation on the methods to indicate resolve order as
a list.
Cleaning up configuration
Deprecating createGracefulShutdownCoordinator from CoordinatorFactory
Deprecating createWorkerStateChangeListener from CoordinatorFactory
Introduing gracefulShutdownCoordinator and workerStateChangeListener configurations to CoordinatorConfig
Switching to use CoordinatorFactory only if the new configurations in code are set to null
* Deprecating createGracefulShutdownCoordinator from CoordinatorFactory
* Deprecating createWorkerStateChangeListener from CoordinatorFactory
* Introduing gracefulShutdownCoordinator and workerStateChangeListener configurations to CoordinatorConfig
* Switching to use CoordinatorFactory only if the new configurations in code are set to null
* Update release notes for release 2.0.1
* Updated version to 2.0.1
* Added notes for PR #371, and cleaned up formatting
* Reordered release notes, and added additional fixes
Ordered the release notes in ascending order of their issue #
Added release notes for #374 and #375
* Add release notes to the change log
outstandingRequests was actually representing the available space in
the RxJava queue. This renames it to better match reality.
Also changed to only make the request if there is available queue
space. We now decrement availableQueueSpace ahead of determine
whether to request another item.
* Added missing lock around the call to request.
Calls to Subscription#request weren't synchronized correctly. This
was only really an issue if there is a large number of errors
occurring.
* Reject operations where the subscriber doesn't match.
If the original subscriber doesn't match the current subscriber reject
operations completely.
If the flow is null, but the subscriber still matches error out the
subscription. The original subscriber will restart.
For canceling only accept the cancel request if the original
subscriber matches the current subscriber.
* Remove unneeded if statement
Don't really need to check if the subscriber is still current, as this
is synchronized.
Subscribe to shard ends periodically and the KCL needs to reconnect at
the last continuation sequence number. If the continuation sequence
number happens to be the last record returned using AT_SEQUENCE_NUMBER
will cause the record to be returned again.
* Cleaning up LeaseManagementConfig:
* Removing unused metrics factory variable from LeaseManagementConfig.
* Revert "Cleaning up LeaseManagementConfig:"
This reverts commit b16ba37966.
* Deprecating metrics factory in LeaseManagementConfig
* Marking metrics factory in LeaseManagementConfig for deprecation
* Making DynamoDBLeaseCoordinator take IOPs configuration in the constructor
* InitialLeaseTableReadCapacity and InitialLeaseTableWriteCapacity for the DynamoDBLeaseCoordinator class throws UnsupportedException
* Calling onNext and onComplete if throwable is of the kind ResourceNotFound.
* Adding testing for ResourceNotFound
* Updating version to 2.0.1-SNAPSHOT
* Added note about the groupId change
The groupId of the Amazon Kinesis Client changed from com.amazonaws to
software.amazon.kinesis
Fixes#354
* Reorder items, and add message about new API's
Moved the configuration message to just above the configuration.
Added messages about the new API's that the KCL uses.
Fixes#353
* Updated CHANGELOG with the updated release notes
* Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
* Records are now delivered via streaming, reducing end-to-end latency.
* The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
* New configuration options are available to configure Enhanced Fan Out.
* `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
__WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__
| Name | Default | Description |
|-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
| consumerArn | Unset | The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. |
| streamName | Unset | The name of the stream that a consumer should be create for if necessary |
| consumerName | Unset | The name of the consumer to create. If this is not set the applicationName will be used instead. |
| applicationName | Unset | The name of the application. This is used as the name of the consumer unless consumerName is set. |
* Modular Configuration of the Kinesis Client
The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from the `ConfigsBuilder`.
Please see the migration guide for more information
* `CheckpointConfig`
* `CoordinatorConfig`
* `LeaseManagementConfig`
* `LifecycleConfig`
* `MetricsConfig`
* `ProcessorConfig`
* `RetrievalConfig`
* Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
All configurations will only accept 2.0 clients.
* When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
* __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
__When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
Please see the migration guide for more information
* MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`. To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
* Added cache updating behavior for GetShard
Customer are occasionally seeing messages about being unable to
retrieve shard information, which is logged as a warning. This change
will allow the shard map to be updated even when there is no re-shard
operation.
This now triggers a shard list update if there is 1000 cache misses,
or a cache miss occurs when the cache is more than 30 seconds old.
For Kinesis the updates will use ListShards, and for DynamoDB Streams
it will continue to use DescribeStream.
* Adjust some logging, and the zeroing of cache misses a bit
Only log about cache refresh if it's the thread doing the cache
refresh. If after synchronizing the shard is present, accept that
someone else loaded the shard map, and move on.
If the cache was reloaded, and the shard was found the current thread
will reset the cache misses.
The warnings for the cache miss was using a modulo of 1000 which is
the maximum value for cache misses, so wasn't to useful.
