* Advance version to 2.1.3-SNAPSHOT
* Added timeouts for Kinesis and DynamoDB calls
Added a timeout to prevent an issue where the Kinesis or DynamoDB call
never completes.
For Kinesis call to GetRecords the timeout defaults to 30 seconds, and can be configured
on the PollingConfig.
For DynamoDB and Kinesis (when calling ListShards) the timeout
defaults to 60 seconds and can be configured on LeaseManagementConfig.
* Introducing MultiLangDaemon support for Enhanced Fan-Out.
* MultiLangDaemon now supports the following command line options.
* `--properties-file`: Properties file that the KCL should use to set up the Scheduler.
* `--log-configuration`: logback.xml that the KCL should use for logging.
* Updated AWS SDK dependency to 2.2.0.
* MultiLangDaemon now uses logback for logging.
Reverted 3 commits:
Revert "Change version number to 2.0.3-experimental"
Revert: 54c171dc2a.
Revert "Experimental support for sequence number validation in the publisher (#401)"
Revert: 592499f7bc.
Revert "Support Validating Records are From to the Expected Shard (#400)"
Revert: 01f5db8049.
* Update release notes for release 2.0.1
* Updated version to 2.0.1
* Added notes for PR #371, and cleaned up formatting
* Reordered release notes, and added additional fixes
Ordered the release notes in ascending order of their issue #
Added release notes for #374 and #375
* Add release notes to the change log
* Calling onNext and onComplete if throwable is of the kind ResourceNotFound.
* Adding testing for ResourceNotFound
* Updating version to 2.0.1-SNAPSHOT
* Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
* Records are now delivered via streaming, reducing end-to-end latency.
* The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
* New configuration options are available to configure Enhanced Fan Out.
* `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
__WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__
| Name | Default | Description |
|-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
| consumerArn | Unset | The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. |
| streamName | Unset | The name of the stream that a consumer should be create for if necessary |
| consumerName | Unset | The name of the consumer to create. If this is not set the applicationName will be used instead. |
| applicationName | Unset | The name of the application. This is used as the name of the consumer unless consumerName is set. |
* Modular Configuration of the Kinesis Client
The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from the `ConfigsBuilder`.
Please see the migration guide for more information
* `CheckpointConfig`
* `CoordinatorConfig`
* `LeaseManagementConfig`
* `LifecycleConfig`
* `MetricsConfig`
* `ProcessorConfig`
* `RetrievalConfig`
* Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
All configurations will only accept 2.0 clients.
* When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
* __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
__When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
Please see the migration guide for more information
* MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`. To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
* Release 1.9.1 of the Amazon Kinesis Client Library for Java
Updated versions, and added release notes.
* Added note about the update of the AWS Java SDK
* Added support for ListShards API. This API is used in place of DescribeStream API to provide more throughput during ShardSyncTask.
* ListShards supoorts higher call rate, which should reduce instances of throttling when attempting to synchronize the shard list.
* Updated the version of AWS Java SDK to 1.11.272.
* Version 1.11.272 is now the minimum support versiojn of the SDK.
* Deprecated the following methods, and classes. These methods, and classes will be removed in a future release.
* Deprecated IKinesisProxy#getStreamInfo.
* Deprecated IKinesisProxyFactory.
* Deprecated KinesisProxyFactory.
* Deprecated certain KinesisProxy constructors.
* Allow providing a custom IKinesisProxy implementation.
* PR #274
* Checkpointing on a different thread should no longer emit a warning about NullMetricsScope.
* PR #284
* Issue #48
* Upgraded the AWS Java SDK to version 1.11.271
* PR #287
* Allow disabling check for the case where a child shard has an open parent shard.
There is a race condition where it's possible for the a parent shard
to appear open, while having child shards. This check can now be
disabled by setting ignoreUnexpectedChildShards in the
KinesisClientLibConfiguration to true.
* PR #240
* Issue #210
* Upgraded the AWS SDK for Java to 1.11.261
* PR #281
This adds that ability for the KCL to fetch records while the record processor is busy. This can help smooth out delays in record process, or retrieving data from Kinesis. Enabling this does require extra threads for background retrieval.
Settings
* dataFetchingStrategy: Which strategy to use to retrieve records. This can be either DEFAULT or PREFETCH_CACHED
* maxCacheByteSize: Retrieval will be paused when the total number of bytes in the cache exceeds this value
* maxRecordsCount: Retrieval will be paused when the total number of records in the cache exceeds this value
* maxPendingProcessRecordsInput: Retrieval will be paused when the total number of fulfilled requests in the cache exceeds this value
* Release 1.8.5 of the Amazon Kinesis Client for Java
Release 1.8.5 (September 26, 2017)
* Only advance the shard iterator for the accepted response.
This fixes a race condition in the `KinesisDataFetcher` when it's being used to make asynchronous requests. The shard iterator is now only advanced when the retriever calls `DataFetcherResult#accept()`.
* PR #230
* Issue #231
This changes the retriever strategy to only accept the shard iterator
when we have accepted a result to return. This is for the
asynchronous retriever where multiple threads may contend for the same
iterator slot. This ensures only the one selected for the response will
advance the shard iterator.
* Create a new completion service for each request.
This ensures that canceled tasks are discarded. This will prevent a cancellation exception causing issues processing records.
* PR #227
* Issue #226
* Call shutdown on the retriever when the record processor is being shutdown
This fixes a bug that could leak threads if using the
AsynchronousGetRecordsRetrievalStrategy is being used.
The asynchronous retriever is only used when
KinesisClientLibConfiguration#retryGetRecordsInSeconds, and
KinesisClientLibConfiguration#maxGetRecordsThreadPool are set.
* PR #222
* Add support for two phase checkpoints
Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
* PR #188
* Support timeouts, and retry for GetRecords calls.
Applications can now set timeouts for GetRecord calls to Kinesis. As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
* PR #214
* Notification when the lease table is throttled
When writes, or reads, to the lease table are throttled a warning will be emitted. If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
* PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown. The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
* PR #204
Support timeouts for calls to the MultiLang Daemon
This adds support for setting a timeout when dispatching records to
the client record processor. If the record processor doesn't respond
within the timeout the parent Java process will be terminated. This
is a temporary fix to handle cases where the KCL becomes blocked
while waiting for a client record processor.
The timeout for the this can be set by adding
`timeoutInSeconds = <timeout value>`.
The default for this is no timeout.
Setting this can cause the KCL to exit suddenly, before using this
ensure that you have an automated restart for your application
Related #195
Related #185
* Execute graceful shutdown on its own thread
* PR #191
* Issue #167
* Added support for controlling the size of the lease renewer thread pool
* PR #177
* Issue #171
* Require Java 8 and later
Java 8 is now required for versions 1.8.0 of the amazon-kinesis-client and later.
* PR #176
* Advance version, and drop Java 7 support
Advanced the version to 1.8.0 as Java 7 support is being removed.
* Remove build settings for Java 7
Removed the configuration for Java 7, and switched to trusty for access
to openjdk8.
* Added support for graceful shutdown in MultiLang Clients
* PR #174
* PR #182
* Updated documentation for `v2.IRecordProcessor#shutdown`, and `KinesisClientLibConfiguration#idleTimeBetweenReadsMillis`
* PR #170
* Updated to version 1.11.151 of the AWS Java SDK
* PR #183
* Correctly handle throttling for DescribeStream, and save accumulated progress from individual calls.
* PR #152
* Upgrade to version 1.11.115 of the AWS Java SDK
* PR #155
Disable the JavaDoc doclint when building with Java 8. Java 8 became
much more pedantic on bad practices, and the documentation hasn't been
cleaned up.
Updated to the newest version of the AWS Java SDK.
Fixes:
Amazon Kinesis Client for Python Issue awslabs/amazon-kinesis-client-python#27
Related:
PR #125
* Add Direct Dependency on Commons Log
Commons log was a transient dependency, but should have been a direct
dependency. Now take a direct dependency on it.
* Move to Next Release Version Snapshot
Changed version to 1.7.3-SNAPSHOT for next release.
MultiLangDaemon Feature Updates
The MultiLangDaemon has been upgraded to use the v2 interfaces, which allows access to enhanced checkpointing, and more information during record processor initialization. The MultiLangDaemon clients must be updated before they can take advantage of these new features.
Updated the MultiLangDaemon to use the v2 record processor interfaces, and added features to messages passed to MultiLangDaemon clients.
These changes will require updates to the various MultiLangDaemon clients. The changes for the Python version are complete, and other versions will be updated later.
Added support for time based iterators. Time based iterators are only
used if there is no current checkpoint for that shard, otherwise the
sequence number of the checkpoint is used.
Added the Travis CI configuration to enable continous integration
testing for common JDK version > 1.7.
Move gpg signing to its own profile to avoid causing issues with the
test, and integration-test phases.
Added unit tests for many of the classes in the client. Added new
dependencies for the unit tests to the POM file. Unit tests can be
executed the `surefire:test` goal, and are executed during the test phase.
Added integration tests for the client. The tests require access to
credentials that can interact with Kinesis, and DynamoDB. The
integration tests can be run by executing the
`failsafe:integration-test` goal, or are run as part of the
integration-test phase
**Resources in the integration tests are not currently released on
completion of the integration tests.**
