luciano/amazon-kinesis-client
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Revert back to 0a91e6faa5

Browse source
This commit is contained in:
glarwood 2018-12-11 18:53:20 -08:00
parent a05e22f782
commit a116817710
389 changed files with 22966 additions and 31568 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
.github/PULL_REQUEST_TEMPLATE.md vendored
View file

@ -1,6 +0,0 @@
*Issue #, if available:*
*Description of changes:*
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

6
.travis.yml
View file

@ -1,6 +1,6 @@
language: java
jdk:
- openjdk8
- openjdk7
- oraclejdk7
- oraclejdk8
sudo: false
dist: trusty
sudo: false

460
CHANGELOG.md
View file

@ -1,460 +0,0 @@
# Changelog
### Release 2.0.5 (November 12, 2018)
[Milestone #26](https://github.com/awslabs/amazon-kinesis-client/milestone/26?closed=1)
* Fixed a deadlock condition that could occur when using the polling model.
When using the `PollingConfig` and a slower record processor it was possible to hit a deadlock in the retrieval of records.
* [PR #462](https://github.com/awslabs/amazon-kinesis-client/pull/462)
* [Issue #448](https://github.com/awslabs/amazon-kinesis-client/issues/448)
* Adjusted `RetrievalConfig`, and `FanOutConfig` to use accessors instead of direct member access.
* [PR #453](https://github.com/awslabs/amazon-kinesis-client/pull/453)
### Release 2.0.4 (October 18, 2018)
[Milestone #25](https://github.com/awslabs/amazon-kinesis-client/milestone/25)
* Added method to retrieve leases from the LeaseCoordinator and LeaseTaker.
* [PR #428](https://github.com/awslabs/amazon-kinesis-client/pull/428)
* Fixed a race condition shutting down the Scheduler before it has completed initialization.
* [PR #439](https://github.com/awslabs/amazon-kinesis-client/pull/439)
* [Issue #427](https://github.com/awslabs/amazon-kinesis-client/issues/427)
* Added `HierarchicalShardSyncer` which replaces the static `ShardSyncer`.
`HierarchicalShardSyncer` removes the contention between multiple instances of the Scheduler when running under a single JVM.
* [PR #395](https://github.com/awslabs/amazon-kinesis-client/pull/395)
* [Issue #415](https://github.com/awslabs/amazon-kinesis-client/issues/415)
* Added `TaskExecutionListener` which allows monitoring of tasks being executed by the `ShardConsumer`.
The listener is invoked before and after a task is executed by the `ShardConsumer`.
* [PR #417](https://github.com/awslabs/amazon-kinesis-client/pull/417)
### Release 2.0.3 (October 8, 2018)
[Milestone #23](https://github.com/awslabs/amazon-kinesis-client/milestone/23)
* Fixed an issue where the `KinesisAsyncClient` could be misconfigured to use HTTP 1.1.
Using HTTP 1.1 with `SubscribeToShard` is unsupported, and could cause misdelivery of records to the record processor.
* [Issue #391](https://github.com/awslabs/amazon-kinesis-client/issues/391)
* [PR #434](https://github.com/awslabs/amazon-kinesis-client/pull/434)
* [PR #433](https://github.com/awslabs/amazon-kinesis-client/pull/433)
* Lower the severity of `ReadTimeout` exceptions.
`ReadTimeout` exceptions can occur if the client is unable to request data from Kinesis for more than client timeout, which defaults to 30 seconds. This can occur if the record processor blocks for more than the timeout period. `ReadTimeout` could also occur as part of [Issue #391](https://github.com/awslabs/amazon-kinesis-client/issues/391).
* [Issue #399](https://github.com/awslabs/amazon-kinesis-client/issues/399)
* [PR #403](https://github.com/awslabs/amazon-kinesis-client/pull/403)
* Added a callback that allows applications to take actions after DynamoDB table creation.
Applications can now install a callback that is called after creating the DynamoDB table by implementing `TableCreatorCallback`.
* [PR #413](https://github.com/awslabs/amazon-kinesis-client/pull/413)
* Updated the guava dependency to 26.0-jre.
* [PR #420](https://github.com/awslabs/amazon-kinesis-client/pull/420)
* [Issue #416](https://github.com/awslabs/amazon-kinesis-client/issues/416)
* Added some additional debug logging around the initialization of the `FanOutRecordsPublisher`.
* [PR #398](https://github.com/awslabs/amazon-kinesis-client/pull/398)
* Upgraded AWS SDK version to 2.0.6
* [PR #434](https://github.com/awslabs/amazon-kinesis-client/pull/434)
### Release 2.0.2 (September 4, 2018)
[Milestone #22](https://github.com/awslabs/amazon-kinesis-client/milestone/22)
* Fixed an issue where the a warning would be logged every second if `logWarningForTaskAfterMillis` was set.
The logging for last time of data arrival now respects the value of `logWarningForTaskAfterMillis`.
* [PR #383](https://github.com/awslabs/amazon-kinesis-client/pull/383)
* [Issue #381](https://github.com/awslabs/amazon-kinesis-client/issues/381)
* Moved creation of `WorkerStateChangedListener` and `GracefulShutdownCoordinator` to the `CoordinatorConfig`.
Originally the `WorkerStateChangedListener` and `GracefulShutdownCoordinator` were created by methods on the `SchedulerCoordinatorFactory`, but they should have been configuration options.
The original methods have been deprecated, and may be removed at a later date.
* [PR #385](https://github.com/awslabs/amazon-kinesis-client/pull/385)
* [PR #388](https://github.com/awslabs/amazon-kinesis-client/pull/388)
* Removed dependency on Apache Commons Lang 2.6.
The dependency on Apache Commons Lang 2.6 has removed, and all usages updated to use Apache Commons Lang 3.7.
* [PR #386](https://github.com/awslabs/amazon-kinesis-client/pull/386)
* [Issue #370](https://github.com/awslabs/amazon-kinesis-client/issues/370)
* Fixed a typo in the MutliLang Daemon shutdown hook.
* [PR #387](https://github.com/awslabs/amazon-kinesis-client/pull/387)
* Added method `onAllInitializationAttemptsFailed(Throwable)` to `WorkerStateChangedListener` to report when all initialization attempts have failed.
This method is a default method, and it isn't require to implement the method. This method is only called after all attempts to initialize the `Scheduler` have failed.
* [PR #369](https://github.com/awslabs/amazon-kinesis-client/pull/369)
### Release 2.0.1 (August 21, 2018)
* Mark certain internal components with `@KinesisClientInternalApi` attribute.
Components marked as internal may be deprecated at a faster rate than public components.
* [PR #358](https://github.com/awslabs/amazon-kinesis-client/pull/358)
* Fixed an issue where `ResourceNotFoundException` on subscription to a shard was not triggering end of shard handling.
If a lease table contains a shard that is no longer present in the stream attempt to subscribe to that shard will trigger a `ResourceNotFoundException`. These exception are treated the same as reaching the end of a shard.
* [PR #359](https://github.com/awslabs/amazon-kinesis-client/pull/359)
* Fixed an issue where the KCL would not Use the configured DynamoDB IOPs when creating the lease table.
* [PR #360](https://github.com/awslabs/amazon-kinesis-client/pull/360)
* Make the maximum number of Scheduler initialization attempts configurable.
The maximum number of `Scheduler` initialization attempts can be configured via `CoordinatorConfig#maxInitializationAttempts`.
* [PR #363](https://github.com/awslabs/amazon-kinesis-client/pull/363)
* [PR #368](https://github.com/awslabs/amazon-kinesis-client/pull/368)
* Fixed an issue where it was possible to get a duplicate record when resubscribing to a shard.
Subscribe to shard requires periodic resubscribing, and uses a new concept of a continuation sequence number. If the continuation sequence number was equal to the last record that record would be processed a second time. Resubscribing now uses `AFTER_SEQUENCE_NUMBER` to ensure that only later records are returned.
* [PR #371](https://github.com/awslabs/amazon-kinesis-client/pull/371)
* Upgraded to AWS SDK 2.0.1
* [PR #372](https://github.com/awslabs/amazon-kinesis-client/pull/372)
* Fixed an issue where time based restart of the subscription wasn't resetting the `lastRequestTime`.
If a subscription hasn't delivered any data for more than 30 seconds it will be canceled and restarted. This detection is based of the `lastRequestTime` which wasn't getting reset after the restart was triggered.
* [PR #373](https://github.com/awslabs/amazon-kinesis-client/pull/373)
* Fixed an issue where requesting on the subscription from the `FanOutRecordsPublisher` could trigger an unexpected failure.
Due to a race condition the underlying flow in the subscription could be set to something else. The method is now synchronized, and verifies that the subscriber it was created with is still the subscriber in affect.
This issue generally would only appear when multiple errors were occurring while connecting to Kinesis.
* [PR #374](https://github.com/awslabs/amazon-kinesis-client/pull/374)
* Fixed an issue where the number of requested items could exceed the capacity of the RxJava queue.
There was an off by one issue when determining whether to make a request to the SDK subscription. This changes the calculation to represent the capacity as a queue.
* [PR #375](https://github.com/awslabs/amazon-kinesis-client/pull/375)
### Release 2.0.0 (August 02, 2018)
* The Maven `groupId`, along with the `version`, for the Amazon Kinesis Client has changed from `com.amazonaws` to `software.amazon.kinesis`.
To add a dependency on the new version of the Amazon Kinesis Client:
``` xml
<dependency>
<groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>2.0.0</version>
</dependency>
```
* Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
* Records are now delivered via streaming, reducing end-to-end latency.
* The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
* `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
__WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__
* The Amazon Kinesis Client now uses 3 additional Kinesis API's:
__WARNING: If using a restrictive Kinesis IAM policy you may need to add the following API methods to the policy.__
* [`SubscribeToShard`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html)
* [`DescribeStreamSummary`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStreamSummary.html)
* [`DescribeStreamConsumer`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStreamConsumer.html)
* [`RegisterStreamConsumer`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_RegisterStreamConsumer.html)
* New configuration options are available to configure Enhanced Fan Out.
| Name | Default | Description |
|-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
| consumerArn | Unset | The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. |
| streamName | Unset | The name of the stream that a consumer should be create for if necessary |
| consumerName | Unset | The name of the consumer to create. If this is not set the applicationName will be used instead. |
| applicationName | Unset | The name of the application. This is used as the name of the consumer unless consumerName is set. |
* Modular Configuration of the Kinesis Client
The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from the `ConfigsBuilder`.
Please [see the migration guide for more information][migration-guide].
* `CheckpointConfig`
* `CoordinatorConfig`
* `LeaseManagementConfig`
* `LifecycleConfig`
* `MetricsConfig`
* `ProcessorConfig`
* `RetrievalConfig`
* Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
All configurations will only accept 2.0 clients.
* When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
* __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
__When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
[Please see the migration guide for more information][migration-guide]
* MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`. To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
## Release 1.9.1 (April 30, 2018)
* Added the ability to create a prepared checkpoint when at `SHARD_END`.
* [PR #301](https://github.com/awslabs/amazon-kinesis-client/pull/301)
* Added the ability to subscribe to worker state change events.
* [PR #291](https://github.com/awslabs/amazon-kinesis-client/pull/291)
* Added support for custom lease managers.
A custom `LeaseManager` can be provided to `Worker.Builder` that will be used to provide lease services.
This makes it possible to implement custom lease management systems in addition to the default DynamoDB system.
* [PR #297](https://github.com/awslabs/amazon-kinesis-client/pull/297)
* Updated the version of the AWS Java SDK to 1.11.219
## Release 1.9.0 (February 6, 2018)
* Introducing support for ListShards API. This API is used in place of DescribeStream API to provide more throughput during ShardSyncTask. Please consult the [AWS Documentation for ListShards](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ListShards.html) for more information.
* ListShards supports higher call rate, which should reduce instances of throttling when attempting to synchronize the shard list.
* __WARNING: `ListShards` is a new API, and may require updating any explicit IAM policies__
* Added configuration parameters for ListShards usage
| Name | Default | Description |
| ---- | ------- | ----------- |
| [listShardsBackoffTimeInMillis](https://github.com/awslabs/amazon-kinesis-client/blob/3ae916c5fcdccd6b835c86ba7f6f53dd5b4c8b04/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1412) | 1500 ms | This is the default backoff time between 2 ListShards calls when throttled. |
| [listShardsRetryAttempts](https://github.com/awslabs/amazon-kinesis-client/blob/3ae916c5fcdccd6b835c86ba7f6f53dd5b4c8b04/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1423) | 50 | This is the maximum number of times the KinesisProxy will retry to make ListShards calls on being throttled. |
* Updating the version of AWS Java SDK to 1.11.272.
* Version 1.11.272 is now the minimum support version of the SDK.
* Deprecating the following methods, and classes. These methods, and classes will be removed in a future release.
* Deprecated [IKinesisProxy#getStreamInfo](https://github.com/awslabs/amazon-kinesis-client/blob/3ae916c5fcdccd6b835c86ba7f6f53dd5b4c8b04/src/main/java/com/amazonaws/services/kinesis/clientlibrary/proxies/IKinesisProxy.java#L48-L62).
* Deprecated [IKinesisProxyFactory](https://github.com/awslabs/amazon-kinesis-client/blob/3ae916c5fcdccd6b835c86ba7f6f53dd5b4c8b04/src/main/java/com/amazonaws/services/kinesis/clientlibrary/proxies/IKinesisProxyFactory.java).
* Deprecated [KinesisProxyFactory](https://github.com/awslabs/amazon-kinesis-client/blob/3ae916c5fcdccd6b835c86ba7f6f53dd5b4c8b04/src/main/java/com/amazonaws/services/kinesis/clientlibrary/proxies/KinesisProxyFactory.java).
* Deprecated certain [KinesisProxy](https://github.com/awslabs/amazon-kinesis-client/blob/3ae916c5fcdccd6b835c86ba7f6f53dd5b4c8b04/src/main/java/com/amazonaws/services/kinesis/clientlibrary/proxies/KinesisProxy.java) constructors.
* [PR #293](https://github.com/awslabs/amazon-kinesis-client/pull/293)
## Release 1.8.10
* Allow providing a custom IKinesisProxy implementation.
* [PR #274](https://github.com/awslabs/amazon-kinesis-client/pull/274)
* Checkpointing on a different thread should no longer emit a warning about NullMetricsScope.
* [PR #284](https://github.com/awslabs/amazon-kinesis-client/pull/284)
* [Issue #48](https://github.com/awslabs/amazon-kinesis-client/issues/48)
* Upgraded the AWS Java SDK to version 1.11.271
* [PR #287](https://github.com/awslabs/amazon-kinesis-client/pull/287)
## Release 1.8.9
* Allow disabling check for the case where a child shard has an open parent shard.
There is a race condition where it's possible for the a parent shard to appear open, while having child shards. This check can now be disabled by setting [`ignoreUnexpectedChildShards`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1037) to true.
* [PR #240](https://github.com/awslabs/amazon-kinesis-client/pull/240)
* [Issue #210](https://github.com/awslabs/amazon-kinesis-client/issues/210)
* Upgraded the AWS SDK for Java to 1.11.261
* [PR #281](https://github.com/awslabs/amazon-kinesis-client/pull/281)
## Release 1.8.8
* Fixed issues with leases losses due to `ExpiredIteratorException` in `PrefetchGetRecordsCache` and `AsynchronousFetchingStrategy`.
PrefetchGetRecordsCache will request for a new iterator and start fetching data again.
* [PR#263](https://github.com/awslabs/amazon-kinesis-client/pull/263)
* Added warning message for long running tasks.
Logging long running tasks can be enabled by setting the following configuration property:
| Name | Default | Description |
| ---- | ------- | ----------- |
| [`logWarningForTaskAfterMillis`](https://github.com/awslabs/amazon-kinesis-client/blob/3de901ea9327370ed732af86c4d4999c8d99541c/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1367) | Not set | Milliseconds after which the logger will log a warning message for the long running task |
* [PR#259](https://github.com/awslabs/amazon-kinesis-client/pull/259)
* Handling spurious lease renewal failures gracefully.
Added better handling of DynamoDB failures when updating leases. These failures would occur when a request to DynamoDB appeared to fail, but was actually successful.
* [PR#247](https://github.com/awslabs/amazon-kinesis-client/pull/247)
* ShutdownTask gets retried if the previous attempt on the ShutdownTask fails.
* [PR#267](https://github.com/awslabs/amazon-kinesis-client/pull/267)
* Fix for using maxRecords from `KinesisClientLibConfiguration` in `GetRecordsCache` for fetching records.
* [PR#264](https://github.com/awslabs/amazon-kinesis-client/pull/264)
## Release 1.8.7
* Don't add a delay for synchronous requests to Kinesis
Removes a delay that had been added for synchronous `GetRecords` calls to Kinesis.
* [PR #256](https://github.com/awslabs/amazon-kinesis-client/pull/256)
## Release 1.8.6
* Add prefetching of records from Kinesis
Prefetching will retrieve and queue additional records from Kinesis while the application is processing existing records.
Prefetching can be enabled by setting [`dataFetchingStrategy`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1317) to `PREFETCH_CACHED`. Once enabled an additional fetching thread will be started to retrieve records from Kinesis. Retrieved records will be held in a queue until the application is ready to process them.
Pre-fetching supports the following configuration values:
| Name | Default | Description |
| ---- | ------- | ----------- |
| [`dataFetchingStrategy`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1317) | `DEFAULT` | Which data fetching strategy to use |
| [`maxPendingProcessRecordsInput`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1296) | 3 | The maximum number of process records input that can be queued |
| [`maxCacheByteSize`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1307) | 8 MiB | The maximum number of bytes that can be queued |
| [`maxRecordsCount`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1326) | 30,000 | The maximum number of records that can be queued |
| [`idleMillisBetweenCalls`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1353) | 1,500 ms | The amount of time to wait between calls to Kinesis |
* [PR #246](https://github.com/awslabs/amazon-kinesis-client/pull/246)
## Release 1.8.5 (September 26, 2017)
* Only advance the shard iterator for the accepted response.
This fixes a race condition in the `KinesisDataFetcher` when it's being used to make asynchronous requests. The shard iterator is now only advanced when the retriever calls `DataFetcherResult#accept()`.
* [PR #230](https://github.com/awslabs/amazon-kinesis-client/pull/230)
* [Issue #231](https://github.com/awslabs/amazon-kinesis-client/issues/231)
## Release 1.8.4 (September 22, 2017)
* Create a new completion service for each request.
This ensures that canceled tasks are discarded. This will prevent a cancellation exception causing issues processing records.
* [PR #227](https://github.com/awslabs/amazon-kinesis-client/pull/227)
* [Issue #226](https://github.com/awslabs/amazon-kinesis-client/issues/226)
## Release 1.8.3 (September 22, 2017)
* Call shutdown on the retriever when the record processor is being shutdown
This fixes a bug that could leak threads if using the [`AsynchronousGetRecordsRetrievalStrategy`](https://github.com/awslabs/amazon-kinesis-client/blob/9a82b6bd05b3c9c5f8581af007141fa6d5f0fc4e/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/AsynchronousGetRecordsRetrievalStrategy.java#L42) is being used.
The asynchronous retriever is only used when [`KinesisClientLibConfiguration#retryGetRecordsInSeconds`](https://github.com/awslabs/amazon-kinesis-client/blob/01d2688bc6e68fd3fe5cb698cb65299d67ac930d/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L227), and [`KinesisClientLibConfiguration#maxGetRecordsThreadPool`](https://github.com/awslabs/amazon-kinesis-client/blob/01d2688bc6e68fd3fe5cb698cb65299d67ac930d/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L230) are set.
* [PR #222](https://github.com/awslabs/amazon-kinesis-client/pull/222)
## Release 1.8.2 (September 20, 2017)
* Add support for two phase checkpoints
Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
Should the checkpoint fail the attempted sequence number is provided in the [`InitializationInput#getPendingCheckpointSequenceNumber`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/types/InitializationInput.java#L81) otherwise the value will be null.
* [PR #188](https://github.com/awslabs/amazon-kinesis-client/pull/188)
* Support timeouts, and retry for GetRecords calls.
Applications can now set timeouts for GetRecord calls to Kinesis. As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
* [PR #214](https://github.com/awslabs/amazon-kinesis-client/pull/214)
* Notification when the lease table is throttled
When writes, or reads, to the lease table are throttled a warning will be emitted. If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
* [PR #212](https://github.com/awslabs/amazon-kinesis-client/pull/212)
* Support configuring the graceful shutdown timeout for MultiLang Clients
This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown. The timeout can be configured by adding `shutdownGraceMillis` to the properties file set to the number of milliseconds to wait.
* [PR #204](https://github.com/awslabs/amazon-kinesis-client/pull/204)
## Release 1.8.1 (August 2, 2017)
* Support timeouts for calls to the MultiLang Daemon
This adds support for setting a timeout when dispatching records to the client record processor. If the record processor doesn't respond within the timeout the parent Java process will be terminated. This is a temporary fix to handle cases where the KCL becomes blocked while waiting for a client record processor.
The timeout for the this can be set by adding `timeoutInSeconds = <timeout value>`. The default for this is no timeout.
__Setting this can cause the KCL to exit suddenly, before using this ensure that you have an automated restart for your application__
* [PR #195](https://github.com/awslabs/amazon-kinesis-client/pull/195)
* [Issue #185](https://github.com/awslabs/amazon-kinesis-client/issues/185)
## Release 1.8.0 (July 25, 2017)
* Execute graceful shutdown on its own thread
* [PR #191](https://github.com/awslabs/amazon-kinesis-client/pull/191)
* [Issue #167](https://github.com/awslabs/amazon-kinesis-client/issues/167)
* Added support for controlling the size of the lease renewer thread pool
* [PR #177](https://github.com/awslabs/amazon-kinesis-client/pull/177)
* [Issue #171](https://github.com/awslabs/amazon-kinesis-client/issues/171)
* Require Java 8 and later
__Java 8 is now required for versions 1.8.0 of the amazon-kinesis-client and later.__
* [PR #176](https://github.com/awslabs/amazon-kinesis-client/issues/176)
## Release 1.7.6 (June 21, 2017)
* Added support for graceful shutdown in MultiLang Clients
* [PR #174](https://github.com/awslabs/amazon-kinesis-client/pull/174)
* [PR #182](https://github.com/awslabs/amazon-kinesis-client/pull/182)
* Updated documentation for `v2.IRecordProcessor#shutdown`, and `KinesisClientLibConfiguration#idleTimeBetweenReadsMillis`
* [PR #170](https://github.com/awslabs/amazon-kinesis-client/pull/170)
* Updated to version 1.11.151 of the AWS Java SDK
* [PR #183](https://github.com/awslabs/amazon-kinesis-client/pull/183)
## Release 1.7.5 (April 7, 2017)
* Correctly handle throttling for DescribeStream, and save accumulated progress from individual calls.
* [PR #152](https://github.com/awslabs/amazon-kinesis-client/pull/152)
* Upgrade to version 1.11.115 of the AWS Java SDK
* [PR #155](https://github.com/awslabs/amazon-kinesis-client/pull/155)
## Release 1.7.4 (February 27, 2017)
* Fixed an issue building JavaDoc for Java 8.
* [Issue #18](https://github.com/awslabs/amazon-kinesis-client/issues/18)
* [PR #141](https://github.com/awslabs/amazon-kinesis-client/pull/141)
* Reduce Throttling Messages to WARN, unless throttling occurs 6 times consecutively.
* [Issue #4](https://github.com/awslabs/amazon-kinesis-client/issues/4)
* [PR #140](https://github.com/awslabs/amazon-kinesis-client/pull/140)
* Fixed two bugs occurring in requestShutdown.
* Fixed a bug that prevented the worker from shutting down, via requestShutdown, when no leases were held.
* [Issue #128](https://github.com/awslabs/amazon-kinesis-client/issues/128)
* Fixed a bug that could trigger a NullPointerException if leases changed during requestShutdown.
* [Issue #129](https://github.com/awslabs/amazon-kinesis-client/issues/129)
* [PR #139](https://github.com/awslabs/amazon-kinesis-client/pull/139)
* Upgraded the AWS SDK Version to 1.11.91
* [PR #138](https://github.com/awslabs/amazon-kinesis-client/pull/138)
* Use an executor returned from `ExecutorService.newFixedThreadPool` instead of constructing it by hand.
* [PR #135](https://github.com/awslabs/amazon-kinesis-client/pull/135)
* Correctly initialize DynamoDB client, when endpoint is explicitly set.
* [PR #142](https://github.com/awslabs/amazon-kinesis-client/pull/142)
## Release 1.7.3 (January 9, 2017)
* Upgrade to the newest AWS Java SDK.
* [Amazon Kinesis Client Issue #27](https://github.com/awslabs/amazon-kinesis-client-python/issues/27)
* [PR #126](https://github.com/awslabs/amazon-kinesis-client/pull/126)
* [PR #125](https://github.com/awslabs/amazon-kinesis-client/pull/125)
* Added a direct dependency on commons-logging.
* [Issue #123](https://github.com/awslabs/amazon-kinesis-client/issues/123)
* [PR #124](https://github.com/awslabs/amazon-kinesis-client/pull/124)
* Make ShardInfo public to allow for custom ShardPrioritization strategies.
* [Issue #120](https://github.com/awslabs/amazon-kinesis-client/issues/120)
* [PR #127](https://github.com/awslabs/amazon-kinesis-client/pull/127)
## Release 1.7.2 (November 7, 2016)
* MultiLangDaemon Feature Updates
The MultiLangDaemon has been upgraded to use the v2 interfaces, which allows access to enhanced checkpointing, and more information during record processor initialization. The MultiLangDaemon clients must be updated before they can take advantage of these new features.
## Release 1.7.1 (November 3, 2016)
* General
* Allow disabling shard synchronization at startup.
* Applications can disable shard synchronization at startup. Disabling shard synchronization can application startup times for very large streams.
* [PR #102](https://github.com/awslabs/amazon-kinesis-client/pull/102)
* Applications can now request a graceful shutdown, and record processors that implement the IShutdownNotificationAware will be given a chance to checkpoint before being shutdown.
* This adds a [new interface](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/interfaces/v2/IShutdownNotificationAware.java), and a [new method on Worker](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java#L539).
* [PR #109](https://github.com/awslabs/amazon-kinesis-client/pull/109)
* Solves [Issue #79](https://github.com/awslabs/amazon-kinesis-client/issues/79)
* MultiLangDaemon
* Applications can now use credential provides that accept string parameters.
* [PR #99](https://github.com/awslabs/amazon-kinesis-client/pull/99)
* Applications can now use different credentials for each service.
* [PR #111](https://github.com/awslabs/amazon-kinesis-client/pull/111)
## Release 1.7.0 (August 22, 2016)
* Add support for time based iterators ([See GetShardIterator Documentation](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html))
* [PR #94](https://github.com/awslabs/amazon-kinesis-client/pull/94)
The `KinesisClientLibConfiguration` now supports providing an initial time stamp position.
* This position is only used if there is no current checkpoint for the shard.
* This setting cannot be used with DynamoDB Streams
Resolves [Issue #88](https://github.com/awslabs/amazon-kinesis-client/issues/88)
* Allow Prioritization of Parent Shards for Task Assignment
* [PR #95](https://github.com/awslabs/amazon-kinesis-client/pull/95)
The `KinesisClientLibconfiguration` now supports providing a `ShardPrioritization` strategy. This strategy controls how the `Worker` determines which `ShardConsumer` to call next. This can improve processing for streams that split often, such as DynamoDB Streams.
* Remove direct dependency on `aws-java-sdk-core`, to allow independent versioning.
* [PR #92](https://github.com/awslabs/amazon-kinesis-client/pull/92)
**You may need to add a direct dependency on aws-java-sdk-core if other dependencies include an older version.**
## Release 1.6.5 (July 25, 2016)
* Change LeaseManager to call DescribeTable before attempting to create the lease table.
* [Issue #36](https://github.com/awslabs/amazon-kinesis-client/issues/36)
* [PR #41](https://github.com/awslabs/amazon-kinesis-client/pull/41)
* [PR #67](https://github.com/awslabs/amazon-kinesis-client/pull/67)
* Allow DynamoDB lease table name to be specified
* [PR #61](https://github.com/awslabs/amazon-kinesis-client/pull/61)
* Add approximateArrivalTimestamp for JsonFriendlyRecord
* [PR #86](https://github.com/awslabs/amazon-kinesis-client/pull/86)
* Shutdown lease renewal thread pool on exit.
* [PR #84](https://github.com/awslabs/amazon-kinesis-client/pull/84)
* Wait for CloudWatch publishing thread to finish before exiting.
* [PR #82](https://github.com/awslabs/amazon-kinesis-client/pull/82)
* Added unit, and integration tests for the library.
## Release 1.6.4 (July 6, 2016)
* Upgrade to AWS SDK for Java 1.11.14
* [Issue #74](https://github.com/awslabs/amazon-kinesis-client/issues/74)
* [Issue #73](https://github.com/awslabs/amazon-kinesis-client/issues/73)
* **Maven Artifact Signing Change**
* Artifacts are now signed by the identity `Amazon Kinesis Tools <amazon-kinesis-tools@amazon.com>`
## Release 1.6.3 (May 12, 2016)
* Fix format exception caused by DEBUG log in LeaseTaker [Issue # 68](https://github.com/awslabs/amazon-kinesis-client/issues/68)
## Release 1.6.2 (March 23, 2016)
* Support for specifying max leases per worker and max leases to steal at a time.
* Support for specifying initial DynamoDB table read and write capacity.
* Support for parallel lease renewal.
* Support for graceful worker shutdown.
* Change DefaultCWMetricsPublisher log level to debug. [PR # 49](https://github.com/awslabs/amazon-kinesis-client/pull/49)
* Avoid NPE in MLD record processor shutdown if record processor was not initialized. [Issue # 29](https://github.com/awslabs/amazon-kinesis-client/issues/29)
## Release 1.6.1 (September 23, 2015)
* Expose [approximateArrivalTimestamp](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) for Records in processRecords API call.
## Release 1.6.0 (July 31, 2015)
* Restores compatibility with [dynamodb-streams-kinesis-adapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter) (which was broken in 1.4.0).
## Release 1.5.1 (July 20, 2015)
* KCL maven artifact 1.5.0 does not work with JDK 7. This release addresses this issue.
## Release 1.5.0 (July 9, 2015)
* **[Metrics Enhancements][kinesis-guide-monitoring-with-kcl]**
* Support metrics level and dimension configurations to control CloudWatch metrics emitted by the KCL.
* Add new metrics that track time spent in record processor methods.
* Disable WorkerIdentifier dimension by default.
* **Exception Reporting** &mdash; Do not silently ignore exceptions in ShardConsumer.
* **AWS SDK Component Dependencies** &mdash; Depend only on AWS SDK components that are used.
## Release 1.4.0 (June 2, 2015)
* Integration with the **[Kinesis Producer Library (KPL)][kinesis-guide-kpl]**
* Automatically de-aggregate records put into the Kinesis stream using the KPL.
* Support checkpointing at the individual user record level when multiple user records are aggregated into one Kinesis record using the KPL.
See [Consumer De-aggregation with the KCL][kinesis-guide-consumer-deaggregation] for details.
## Release 1.3.0 (May 22, 2015)
* A new metric called "MillisBehindLatest", which tracks how far consumers are from real time, is now uploaded to CloudWatch.
## Release 1.2.1 (January 26, 2015)
* **MultiLangDaemon** &mdash; Changes to the MultiLangDaemon to make it easier to provide a custom worker.
## Release 1.2 (October 21, 2014)
* **Multi-Language Support** &mdash; Amazon KCL now supports implementing record processors in any language by communicating with the daemon over [STDIN and STDOUT][multi-lang-protocol]. Python developers can directly use the [Amazon Kinesis Client Library for Python][kclpy] to write their data processing applications.
## Release 1.1 (June 30, 2014)
* **Checkpointing at a specific sequence number** &mdash; The IRecordProcessorCheckpointer interface now supports checkpointing at a sequence number specified by the record processor.
* **Set region** &mdash; KinesisClientLibConfiguration now supports setting the region name to indicate the location of the Amazon Kinesis service. The Amazon DynamoDB table and Amazon CloudWatch metrics associated with your application will also use this region setting.
[kinesis]: http://aws.amazon.com/kinesis
[kinesis-forum]: http://developer.amazonwebservices.com/connect/forum.jspa?forumID=169
[kinesis-client-library-issues]: https://github.com/awslabs/amazon-kinesis-client/issues
[docs-signup]: http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-setup.html
[kinesis-guide]: http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html
[kinesis-guide-begin]: http://docs.aws.amazon.com/kinesis/latest/dev/before-you-begin.html
[kinesis-guide-create]: http://docs.aws.amazon.com/kinesis/latest/dev/step-one-create-stream.html
[kinesis-guide-applications]: http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-app.html
[kinesis-guide-monitoring-with-kcl]: http://docs.aws.amazon.com//kinesis/latest/dev/monitoring-with-kcl.html
[kinesis-guide-kpl]: http://docs.aws.amazon.com//kinesis/latest/dev/developing-producers-with-kpl.html
[kinesis-guide-consumer-deaggregation]: http://docs.aws.amazon.com//kinesis/latest/dev/kinesis-kpl-consumer-deaggregation.html
[kclpy]: https://github.com/awslabs/amazon-kinesis-client-python
[multi-lang-protocol]: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/multilang/package-info.java

4
CODE_OF_CONDUCT.md
View file

@ -1,4 +0,0 @@
## Code of Conduct
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
opensource-codeofconduct@amazon.com with any additional questions or comments.

61
CONTRIBUTING.md
View file

@ -1,61 +0,0 @@
# Contributing Guidelines
Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
documentation, we greatly value feedback and contributions from our community.
Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
information to effectively respond to your bug report or contribution.
## Reporting Bugs/Feature Requests
We welcome you to use the GitHub issue tracker to report bugs or suggest features.
When filing an issue, please check [existing open](https://github.com/awslabs/amazon-kinesis-client/issues), or [recently closed](https://github.com/awslabs/amazon-kinesis-client/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already
reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
* A reproducible test case or series of steps
* The version of our code being used
* Any modifications you've made relevant to the bug
* Anything unusual about your environment or deployment
## Contributing via Pull Requests
Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
1. You are working against the latest source on the *master* branch.
2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
To send us a pull request, please:
1. Fork the repository.
2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
3. Ensure local tests pass.
4. Commit to your fork using clear commit messages.
5. Send us a pull request, answering any default questions in the pull request interface.
6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
[creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
## Finding contributions to work on
Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/amazon-kinesis-client/labels/help%20wanted) issues is a great place to start.
## Code of Conduct
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
opensource-codeofconduct@amazon.com with any additional questions or comments.
## Security issue notifications
If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
## Licensing
See the [LICENSE](https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.

10
META-INF/MANIFEST.MF
View file

@ -2,9 +2,9 @@ Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Amazon Kinesis Client Library for Java
Bundle-SymbolicName: com.amazonaws.kinesisclientlibrary;singleton:=true
Bundle-Version: 2.0.0
Bundle-Version: 1.7.4
Bundle-Vendor: Amazon Technologies, Inc
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Bundle-RequiredExecutionEnvironment: JavaSE-1.7
Require-Bundle: org.apache.commons.codec;bundle-version="1.6",
org.apache.commons.logging;bundle-version="1.1.3";visibility:=reexport,
com.fasterxml.jackson.core.jackson-databind;bundle-version="2.5.3",
@ -12,16 +12,16 @@ Require-Bundle: org.apache.commons.codec;bundle-version="1.6",
com.fasterxml.jackson.core.jackson-annotations;bundle-version="2.5.0",
org.apache.httpcomponents.httpcore;bundle-version="4.3.3",
org.apache.httpcomponents.httpclient;bundle-version="4.3.6"
com.amazonaws.sdk;bundle-version="1.11.319",
com.amazonaws.sdk;bundle-version="1.11.14",
Export-Package: com.amazonaws.services.kinesis,
com.amazonaws.services.kinesis.clientlibrary,
com.amazonaws.services.kinesis.clientlibrary.kinesisClientLibConfiguration,
com.amazonaws.services.kinesis.clientlibrary.config,
com.amazonaws.services.kinesis.clientlibrary.exceptions,
com.amazonaws.services.kinesis.clientlibrary.exceptions.internal,
com.amazonaws.services.kinesis.clientlibrary.interfaces,
com.amazonaws.services.kinesis.clientlibrary.lib,
com.amazonaws.services.kinesis.clientlibrary.lib.checkpoint,
com.amazonaws.services.kinesis.clientlibrary.lib.scheduler,
com.amazonaws.services.kinesis.clientlibrary.lib.worker,
com.amazonaws.services.kinesis.clientlibrary.proxies,
com.amazonaws.services.kinesis.clientlibrary.types,
com.amazonaws.services.kinesis.leases,

181
README.md
View file

@ -1,5 +1,4 @@
# Amazon Kinesis Client Library for Java
[![Build Status](https://travis-ci.org/awslabs/amazon-kinesis-client.svg?branch=master)](https://travis-ci.org/awslabs/amazon-kinesis-client) ![BuildStatus](https://codebuild.us-west-2.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiaWo4bDYyUkpWaG9ZTy9zeFVoaVlWbEwxazdicDJLcmZwUUpFWVVBM0ZueEJSeFIzNkhURzdVbUd6WUZHcGNxa3BEUzNrL0I5Nzc4NE9rbXhvdEpNdlFRPSIsIml2UGFyYW1ldGVyU3BlYyI6IlZDaVZJSTM1QW95bFRTQnYiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=master)
# Amazon Kinesis Client Library for Java [![Build Status](https://travis-ci.org/awslabs/amazon-kinesis-client.svg?branch=master)](https://travis-ci.org/awslabs/amazon-kinesis-client)
The **Amazon Kinesis Client Library for Java** (Amazon KCL) enables Java developers to easily consume and process data from [Amazon Kinesis][kinesis].
@ -7,13 +6,6 @@ The **Amazon Kinesis Client Library for Java** (Amazon KCL) enables Java develop
* [Forum][kinesis-forum]
* [Issues][kinesis-client-library-issues]
### Recommended Upgrade for All Users of the 2.x Amazon Kinesis Client
**:warning: It's highly recommended for users of version 2.0 of the Amazon Kinesis Client to upgrade to version 2.0.3 or later. A [bug has been](https://github.com/awslabs/amazon-kinesis-client/issues/391) identified in versions prior to 2.0.3 that could cause records to be delivered to the wrong record processor.**
**:information_source: Amazon Kinesis Client versions 1.x are not impacted.**
Please open an issue if you have any questions.
## Features
* Provides an easy-to-use programming model for processing data using Amazon Kinesis
@ -23,7 +15,7 @@ Please open an issue if you have any questions.
1. **Sign up for AWS** &mdash; Before you begin, you need an AWS account. For more information about creating an AWS account and retrieving your AWS credentials, see [AWS Account and Credentials][docs-signup] in the AWS SDK for Java Developer Guide.
1. **Sign up for Amazon Kinesis** &mdash; Go to the Amazon Kinesis console to sign up for the service and create an Amazon Kinesis stream. For more information, see [Create an Amazon Kinesis Stream][kinesis-guide-create] in the Amazon Kinesis Developer Guide.
1. **Minimum requirements** &mdash; To use the Amazon Kinesis Client Library, you'll need **Java 1.8+**. For more information about Amazon Kinesis Client Library requirements, see [Before You Begin][kinesis-guide-begin] in the Amazon Kinesis Developer Guide.
1. **Minimum requirements** &mdash; To use the Amazon Kinesis Client Library, you'll need **Java 1.7+**. For more information about Amazon Kinesis Client Library requirements, see [Before You Begin][kinesis-guide-begin] in the Amazon Kinesis Developer Guide.
1. **Using the Amazon Kinesis Client Library** &mdash; The best way to get familiar with the Amazon Kinesis Client Library is to read [Developing Record Consumer Applications][kinesis-guide-applications] in the Amazon Kinesis Developer Guide.
## Building from Source
@ -31,46 +23,146 @@ Please open an issue if you have any questions.
After you've downloaded the code from GitHub, you can build it using Maven. To disable GPG signing in the build, use this command: `mvn clean install -Dgpg.skip=true`
## Integration with the Kinesis Producer Library
For producer-side developers using the **[Kinesis Producer Library (KPL)][kinesis-guide-kpl]**, the KCL integrates without additional effort. When the KCL retrieves an aggregated Amazon Kinesis record consisting of multiple KPL user records, it will automatically invoke the KPL to extract the individual user records before returning them to the user.
For producer-side developers using the **[Kinesis Producer Library (KPL)][kinesis-guide-kpl]**, the KCL integrates without additional effort.  When the KCL retrieves an aggregated Amazon Kinesis record consisting of multiple KPL user records, it will automatically invoke the KPL to extract the individual user records before returning them to the user.
## Amazon KCL support for other languages
To make it easier for developers to write record processors in other languages, we have implemented a Java based daemon, called MultiLangDaemon that does all the heavy lifting. Our approach has the daemon spawn a sub-process, which in turn runs the record processor, which can be written in any language. The MultiLangDaemon process and the record processor sub-process communicate with each other over [STDIN and STDOUT using a defined protocol][multi-lang-protocol]. There will be a one to one correspondence amongst record processors, child processes, and shards. For Python developers specifically, we have abstracted these implementation details away and [expose an interface][kclpy] that enables you to focus on writing record processing logic in Python. This approach enables KCL to be language agnostic, while providing identical features and similar parallel processing model across all languages.
## Using the KCL
The recommended way to use the KCL for Java is to consume it from Maven.
### Version 2.x
``` xml
<dependency>
<groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>2.0.4</version>
</dependency>
```
### Version 1.x
[Version 1.x tracking branch](https://github.com/awslabs/amazon-kinesis-client/tree/v1.x)
``` xml
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>1.9.2</version>
</dependency>
```
## Release Notes
### Release 1.7.4 (February 27, 2017)
* Fixed an issue building JavaDoc for Java 8.
* [Issue #18](https://github.com/awslabs/amazon-kinesis-client/issues/18)
* [PR #141](https://github.com/awslabs/amazon-kinesis-client/pull/141)
* Reduce Throttling Messages to WARN, unless throttling occurs 6 times consecutively.
* [Issue #4](https://github.com/awslabs/amazon-kinesis-client/issues/4)
* [PR #140](https://github.com/awslabs/amazon-kinesis-client/pull/140)
* Fixed two bugs occurring in requestShutdown.
* Fixed a bug that prevented the worker from shutting down, via requestShutdown, when no leases were held.
* [Issue #128](https://github.com/awslabs/amazon-kinesis-client/issues/128)
* Fixed a bug that could trigger a NullPointerException if leases changed during requestShutdown.
* [Issue #129](https://github.com/awslabs/amazon-kinesis-client/issues/129)
* [PR #139](https://github.com/awslabs/amazon-kinesis-client/pull/139)
* Upgraded the AWS SDK Version to 1.11.91
* [PR #138](https://github.com/awslabs/amazon-kinesis-client/pull/138)
* Use an executor returned from `ExecutorService.newFixedThreadPool` instead of constructing it by hand.
* [PR #135](https://github.com/awslabs/amazon-kinesis-client/pull/135)
* Correctly initialize DynamoDB client, when endpoint is explicitly set.
* [PR #142](https://github.com/awslabs/amazon-kinesis-client/pull/142)
### Latest Release (2.0.5 - November 12, 2018)
[Milestone #26](https://github.com/awslabs/amazon-kinesis-client/milestone/26?closed=1)
* Fixed a deadlock condition that could occur when using the polling model.
It was possible to hit a deadlock in the retrieval of records When using the `PollingConfig` and a slow running record processor.
* [PR #462](https://github.com/awslabs/amazon-kinesis-client/pull/462)
* [Issue #448](https://github.com/awslabs/amazon-kinesis-client/issues/448)
* Adjusted `RetrievalConfig`, and `FanOutConfig` to use accessors instead of direct member access.
* [PR #453](https://github.com/awslabs/amazon-kinesis-client/pull/453)
### Release 1.7.3 (January 9, 2017)
* Upgrade to the newest AWS Java SDK.
* [Amazon Kinesis Client Issue #27](https://github.com/awslabs/amazon-kinesis-client-python/issues/27)
* [PR #126](https://github.com/awslabs/amazon-kinesis-client/pull/126)
* [PR #125](https://github.com/awslabs/amazon-kinesis-client/pull/125)
* Added a direct dependency on commons-logging.
* [Issue #123](https://github.com/awslabs/amazon-kinesis-client/issues/123)
* [PR #124](https://github.com/awslabs/amazon-kinesis-client/pull/124)
* Make ShardInfo public to allow for custom ShardPrioritization strategies.
* [Issue #120](https://github.com/awslabs/amazon-kinesis-client/issues/120)
* [PR #127](https://github.com/awslabs/amazon-kinesis-client/pull/127)
### For remaining release notes check **[CHANGELOG.md][changelog-md]**.
### Release 1.7.2 (November 7, 2016)
* MultiLangDaemon Feature Updates
The MultiLangDaemon has been upgraded to use the v2 interfaces, which allows access to enhanced checkpointing, and more information during record processor initialization. The MultiLangDaemon clients must be updated before they can take advantage of these new features.
### Release 1.7.1 (November 3, 2016)
* General
* Allow disabling shard synchronization at startup.
* Applications can disable shard synchronization at startup. Disabling shard synchronization can application startup times for very large streams.
* [PR #102](https://github.com/awslabs/amazon-kinesis-client/pull/102)
* Applications can now request a graceful shutdown, and record processors that implement the IShutdownNotificationAware will be given a chance to checkpoint before being shutdown.
* This adds a [new interface](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/interfaces/v2/IShutdownNotificationAware.java), and a [new method on Worker](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java#L539).
* [PR #109](https://github.com/awslabs/amazon-kinesis-client/pull/109)
* Solves [Issue #79](https://github.com/awslabs/amazon-kinesis-client/issues/79)
* MultiLangDaemon
* Applications can now use credential provides that accept string parameters.
* [PR #99](https://github.com/awslabs/amazon-kinesis-client/pull/99)
* Applications can now use different credentials for each service.
* [PR #111](https://github.com/awslabs/amazon-kinesis-client/pull/111)
### Release 1.7.0 (August 22, 2016)
* Add support for time based iterators ([See GetShardIterator Documentation](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html))
* [PR #94](https://github.com/awslabs/amazon-kinesis-client/pull/94)
The `KinesisClientLibConfiguration` now supports providing an initial time stamp position.
* This position is only used if there is no current checkpoint for the shard.
* This setting cannot be used with DynamoDB Streams
Resolves [Issue #88](https://github.com/awslabs/amazon-kinesis-client/issues/88)
* Allow Prioritization of Parent Shards for Task Assignment
* [PR #95](https://github.com/awslabs/amazon-kinesis-client/pull/95)
The `KinesisClientLibconfiguration` now supports providing a `ShardPrioritization` strategy. This strategy controls how the `Worker` determines which `ShardConsumer` to call next. This can improve processing for streams that split often, such as DynamoDB Streams.
* Remove direct dependency on `aws-java-sdk-core`, to allow independent versioning.
* [PR #92](https://github.com/awslabs/amazon-kinesis-client/pull/92)
**You may need to add a direct dependency on aws-java-sdk-core if other dependencies include an older version.**
### Release 1.6.5 (July 25, 2016)
* Change LeaseManager to call DescribeTable before attempting to create the lease table.
* [Issue #36](https://github.com/awslabs/amazon-kinesis-client/issues/36)
* [PR #41](https://github.com/awslabs/amazon-kinesis-client/pull/41)
* [PR #67](https://github.com/awslabs/amazon-kinesis-client/pull/67)
* Allow DynamoDB lease table name to be specified
* [PR #61](https://github.com/awslabs/amazon-kinesis-client/pull/61)
* Add approximateArrivalTimestamp for JsonFriendlyRecord
* [PR #86](https://github.com/awslabs/amazon-kinesis-client/pull/86)
* Shutdown lease renewal thread pool on exit.
* [PR #84](https://github.com/awslabs/amazon-kinesis-client/pull/84)
* Wait for CloudWatch publishing thread to finish before exiting.
* [PR #82](https://github.com/awslabs/amazon-kinesis-client/pull/82)
* Added unit, and integration tests for the library.
### Release 1.6.4 (July 6, 2016)
* Upgrade to AWS SDK for Java 1.11.14
* [Issue #74](https://github.com/awslabs/amazon-kinesis-client/issues/74)
* [Issue #73](https://github.com/awslabs/amazon-kinesis-client/issues/73)
* **Maven Artifact Signing Change**
* Artifacts are now signed by the identity `Amazon Kinesis Tools <amazon-kinesis-tools@amazon.com>`
### Release 1.6.3 (May 12, 2016)
* Fix format exception caused by DEBUG log in LeaseTaker [Issue # 68](https://github.com/awslabs/amazon-kinesis-client/issues/68)
### Release 1.6.2 (March 23, 2016)
* Support for specifying max leases per worker and max leases to steal at a time.
* Support for specifying initial DynamoDB table read and write capacity.
* Support for parallel lease renewal.
* Support for graceful worker shutdown.
* Change DefaultCWMetricsPublisher log level to debug. [PR # 49](https://github.com/awslabs/amazon-kinesis-client/pull/49)
* Avoid NPE in MLD record processor shutdown if record processor was not initialized. [Issue # 29](https://github.com/awslabs/amazon-kinesis-client/issues/29)
### Release 1.6.1 (September 23, 2015)
* Expose [approximateArrivalTimestamp](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) for Records in processRecords API call.
### Release 1.6.0 (July 31, 2015)
* Restores compatibility with [dynamodb-streams-kinesis-adapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter) (which was broken in 1.4.0).
### Release 1.5.1 (July 20, 2015)
* KCL maven artifact 1.5.0 does not work with JDK 7. This release addresses this issue.
### Release 1.5.0 (July 9, 2015)
* **[Metrics Enhancements][kinesis-guide-monitoring-with-kcl]**
* Support metrics level and dimension configurations to control CloudWatch metrics emitted by the KCL.
* Add new metrics that track time spent in record processor methods.
* Disable WorkerIdentifier dimension by default.
* **Exception Reporting** &mdash; Do not silently ignore exceptions in ShardConsumer.
* **AWS SDK Component Dependencies** &mdash; Depend only on AWS SDK components that are used.
### Release 1.4.0 (June 2, 2015)
* Integration with the **[Kinesis Producer Library (KPL)][kinesis-guide-kpl]**
* Automatically de-aggregate records put into the Kinesis stream using the KPL.
* Support checkpointing at the individual user record level when multiple user records are aggregated into one Kinesis record using the KPL.
See [Consumer De-aggregation with the KCL][kinesis-guide-consumer-deaggregation] for details.
### Release 1.3.0 (May 22, 2015)
* A new metric called "MillisBehindLatest", which tracks how far consumers are from real time, is now uploaded to CloudWatch.
### Release 1.2.1 (January 26, 2015)
* **MultiLangDaemon** &mdash; Changes to the MultiLangDaemon to make it easier to provide a custom worker.
### Release 1.2 (October 21, 2014)
* **Multi-Language Support** &mdash; Amazon KCL now supports implementing record processors in any language by communicating with the daemon over [STDIN and STDOUT][multi-lang-protocol]. Python developers can directly use the [Amazon Kinesis Client Library for Python][kclpy] to write their data processing applications.
### Release 1.1 (June 30, 2014)
* **Checkpointing at a specific sequence number** &mdash; The IRecordProcessorCheckpointer interface now supports checkpointing at a sequence number specified by the record processor.
* **Set region** &mdash; KinesisClientLibConfiguration now supports setting the region name to indicate the location of the Amazon Kinesis service. The Amazon DynamoDB table and Amazon CloudWatch metrics associated with your application will also use this region setting.
[kinesis]: http://aws.amazon.com/kinesis
[kinesis-forum]: http://developer.amazonwebservices.com/connect/forum.jspa?forumID=169
@ -85,5 +177,4 @@ The recommended way to use the KCL for Java is to consume it from Maven.
[kinesis-guide-consumer-deaggregation]: http://docs.aws.amazon.com//kinesis/latest/dev/kinesis-kpl-consumer-deaggregation.html
[kclpy]: https://github.com/awslabs/amazon-kinesis-client-python
[multi-lang-protocol]: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/multilang/package-info.java
[changelog-md]: https://github.com/awslabs/amazon-kinesis-client/blob/master/CHANGELOG.md
[migration-guide]: https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html

130
amazon-kinesis-client-multilang/pom.xml
View file

@ -1,130 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
~
~ Licensed under the Amazon Software License (the "License").
~ You may not use this file except in compliance with the License.
~ A copy of the License is located at
~
~ http://aws.amazon.com/asl/
~
~ or in the "license" file accompanying this file. This file is distributed
~ on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
~ express or implied. See the License for the specific language governing
~ permissions and limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>amazon-kinesis-client-pom</artifactId>
<groupId>software.amazon.kinesis</groupId>
<version>2.0.5</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>amazon-kinesis-client-multilang</artifactId>
<dependencies>
<dependency>
<groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.16.20</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.1.7</version>
</dependency>
<!-- Test -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.19</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.3</version>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>disable-java8-doclint</id>
<activation>
<jdk>[1.8,)</jdk>
</activation>
<properties>
<additionalparam>-Xdoclint:none</additionalparam>
</properties>
</profile>
</profiles>
</project>

78
amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangRecordProcessorFactory.java
View file

@ -1,78 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang;
import java.util.concurrent.ExecutorService;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.processor.ShardRecordProcessorFactory;
import software.amazon.kinesis.processor.ShardRecordProcessor;
/**
* Creates {@link MultiLangShardRecordProcessor}'s.
*/
@Slf4j
public class MultiLangRecordProcessorFactory implements ShardRecordProcessorFactory {
private static final String COMMAND_DELIMETER_REGEX = " +";
private final String command;
private final String[] commandArray;
private final ObjectMapper objectMapper;
private final ExecutorService executorService;
private final KinesisClientLibConfiguration configuration;
/**
* @param command The command that will do processing for this factory's record processors.
* @param executorService An executor service to use while processing inputs and outputs of the child process.
*/
public MultiLangRecordProcessorFactory(String command, ExecutorService executorService,
KinesisClientLibConfiguration configuration) {
this(command, executorService, new ObjectMapper(), configuration);
}
/**
* @param command The command that will do processing for this factory's record processors.
* @param executorService An executor service to use while processing inputs and outputs of the child process.
* @param objectMapper An object mapper used to convert messages to json to be written to the child process
*/
public MultiLangRecordProcessorFactory(String command, ExecutorService executorService, ObjectMapper objectMapper,
KinesisClientLibConfiguration configuration) {
this.command = command;
this.commandArray = command.split(COMMAND_DELIMETER_REGEX);
this.executorService = executorService;
this.objectMapper = objectMapper;
this.configuration = configuration;
}
@Override
public ShardRecordProcessor shardRecordProcessor() {
log.debug("Creating new record processor for client executable: {}", command);
/*
* Giving ProcessBuilder the command as an array of Strings allows users to specify command line arguments.
*/
return new MultiLangShardRecordProcessor(new ProcessBuilder(commandArray), executorService, this.objectMapper,
this.configuration);
}
String[] getCommandArray() {
return commandArray;
}
}

48
amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/BooleanPropertyValueDecoder.java
View file

@ -1,48 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.config;
import java.util.Arrays;
import java.util.List;
/**
* Provide boolean property.
*/
class BooleanPropertyValueDecoder implements IPropertyValueDecoder<Boolean> {
/**
* Constructor.
*/
BooleanPropertyValueDecoder() {
}
/**
* @param value property value as String
* @return corresponding variable in correct type
*/
@Override
public Boolean decodeValue(String value) {
return Boolean.parseBoolean(value);
}
/**
* @return list of supported types
*/
@Override
public List<Class<Boolean>> getSupportedTypes() {
return Arrays.asList(boolean.class, Boolean.class);
}
}

61
amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/InitializeMessage.java
View file

@ -1,61 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.messages;
import lombok.Getter;
import lombok.Setter;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
/**
* An initialize message is sent to the client's subprocess to indicate that it should perform its initialization steps.
*/
@Getter
@Setter
public class InitializeMessage extends Message {
/**
* The name used for the action field in {@link Message}.
*/
public static final String ACTION = "initialize";
/**
* The shard id that this processor is getting initialized for.
*/
private String shardId;
private String sequenceNumber;
private Long subSequenceNumber;
/**
* Default constructor.
*/
public InitializeMessage() {
}
/**
* Convenience constructor.
*
* @param initializationInput {@link InitializationInput}
*/
public InitializeMessage(InitializationInput initializationInput) {
this.shardId = initializationInput.shardId();
if (initializationInput.extendedSequenceNumber() != null) {
this.sequenceNumber = initializationInput.extendedSequenceNumber().sequenceNumber();
this.subSequenceNumber = initializationInput.extendedSequenceNumber().subSequenceNumber();
} else {
this.sequenceNumber = null;
this.subSequenceNumber = null;
}
}
}

60
amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/JsonFriendlyRecord.java
View file

@ -1,60 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.messages;
import java.time.Instant;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.AllArgsConstructor;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.NonNull;
import lombok.Setter;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
/**
* Class for encoding Record objects to json. Needed because Records have byte buffers for their data field which causes
* problems for the json library we're using.
*/
@NoArgsConstructor
@AllArgsConstructor
@Getter
@Setter
@EqualsAndHashCode
@ToString
public class JsonFriendlyRecord {
private byte[] data;
private String partitionKey;
private String sequenceNumber;
private Instant approximateArrivalTimestamp;
private Long subSequenceNumber;
public static String ACTION = "record";
public static JsonFriendlyRecord fromKinesisClientRecord(@NonNull final KinesisClientRecord record) {
byte[] data = record.data() == null ? null : record.data().array();
return new JsonFriendlyRecord(data, record.partitionKey(), record.sequenceNumber(),
record.approximateArrivalTimestamp(), record.subSequenceNumber());
}
@JsonProperty
public String getAction() {
return ACTION;
}
}

28
amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/ShutdownRequestedMessage.java
View file

@ -1,28 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.messages;
import lombok.NoArgsConstructor;
/**
* A message to indicate to the client's process that shutdown is requested.
*/
@NoArgsConstructor
public class ShutdownRequestedMessage extends Message {
/**
* The name used for the action field in {@link Message}.
*/
public static final String ACTION = "shutdownRequested";
}

1384
amazon-kinesis-client-multilang/src/main/java/software/amazon/kinesis/coordinator/KinesisClientLibConfiguration.java
View file

File diff suppressed because it is too large Load diff

26
amazon-kinesis-client-multilang/src/main/resources/logback.xml
View file

@ -1,26 +0,0 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
~ Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
~
~ Licensed under the Amazon Software License (the "License").
~ You may not use this file except in compliance with the License.
~ A copy of the License is located at
~
~ http://aws.amazon.com/asl/
~
~ or in the "license" file accompanying this file. This file is distributed
~ on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
~ express or implied. See the License for the specific language governing
~ permissions and limitations under the License.
-->
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d [%thread] %-5level %logger{36} [%mdc{ShardId:-NONE}] - %msg %n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>

259
amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/MultiLangProtocolTest.java
View file

@ -1,259 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.anyInt;
import static org.mockito.Matchers.anyLong;
import static org.mockito.Matchers.anyString;
import static org.mockito.Matchers.argThat;
import static org.mockito.Matchers.eq;
import static org.mockito.Mockito.timeout;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.runners.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import com.amazonaws.services.kinesis.multilang.messages.CheckpointMessage;
import com.amazonaws.services.kinesis.multilang.messages.Message;
import com.amazonaws.services.kinesis.multilang.messages.ProcessRecordsMessage;
import com.amazonaws.services.kinesis.multilang.messages.StatusMessage;
import com.google.common.util.concurrent.SettableFuture;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
@RunWith(MockitoJUnitRunner.class)
public class MultiLangProtocolTest {
private static final List<KinesisClientRecord> EMPTY_RECORD_LIST = Collections.emptyList();
@Mock
private MultiLangProtocol protocol;
@Mock
private MessageWriter messageWriter;
@Mock
private MessageReader messageReader;
private String shardId;
@Mock
private RecordProcessorCheckpointer checkpointer;
@Mock
private KinesisClientLibConfiguration configuration;
@Before
public void setup() {
this.shardId = "shard-id-123";
protocol = new MultiLangProtocolForTesting(messageReader, messageWriter,
InitializationInput.builder().shardId(shardId).build(), configuration);
when(configuration.getTimeoutInSeconds()).thenReturn(Optional.empty());
}
private <T> Future<T> buildFuture(T value) {
SettableFuture<T> future = SettableFuture.create();
future.set(value);
return future;
}
private <T> Future<T> buildFuture(T value, Class<T> clazz) {
SettableFuture<T> future = SettableFuture.create();
future.set(value);
return future;
}
@Test
public void initializeTest() throws InterruptedException, ExecutionException {
when(messageWriter
.writeInitializeMessage(argThat(Matchers.withInit(InitializationInput.builder()
.shardId(shardId).build())))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("initialize"), Message.class));
assertThat(protocol.initialize(), equalTo(true));
}
@Test
public void processRecordsTest() throws InterruptedException, ExecutionException {
when(messageWriter.writeProcessRecordsMessage(any(ProcessRecordsInput.class))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("processRecords"), Message.class));
assertThat(protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST).build()),
equalTo(true));
}
@Test
public void shutdownTest() throws InterruptedException, ExecutionException {
when(messageWriter.writeShutdownMessage(any(ShutdownReason.class))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("shutdown"), Message.class));
Mockito.doReturn(buildFuture(true)).when(messageWriter)
.writeShutdownMessage(any(ShutdownReason.class));
Mockito.doReturn(buildFuture(new StatusMessage("shutdown")))
.when(messageReader).getNextMessageFromSTDOUT();
assertThat(protocol.shutdown(null, ShutdownReason.LEASE_LOST), equalTo(true));
}
@Test
public void shutdownRequestedTest() {
when(messageWriter.writeShutdownRequestedMessage()).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("shutdownRequested"), Message.class));
Mockito.doReturn(buildFuture(true)).when(messageWriter)
.writeShutdownRequestedMessage();
Mockito.doReturn(buildFuture(new StatusMessage("shutdownRequested")))
.when(messageReader).getNextMessageFromSTDOUT();
assertThat(protocol.shutdownRequested(null), equalTo(true));
}
private Answer<Future<Message>> buildMessageAnswers(List<Message> messages) {
return new Answer<Future<Message>>() {
Iterator<Message> messageIterator;
Message message;
Answer<Future<Message>> init(List<Message> messages) {
messageIterator = messages.iterator();
return this;
}
@Override
public Future<Message> answer(InvocationOnMock invocation) throws Throwable {
if (this.messageIterator.hasNext()) {
message = this.messageIterator.next();
}
return buildFuture(message);
}
}.init(messages);
}
@Test
public void processRecordsWithCheckpointsTest() throws InterruptedException, ExecutionException,
KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
when(messageWriter.writeProcessRecordsMessage(any(ProcessRecordsInput.class))).thenReturn(buildFuture(true));
when(messageWriter.writeCheckpointMessageWithError(anyString(), anyLong(), any(Throwable.class))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenAnswer(buildMessageAnswers(new ArrayList<Message>() {
{
this.add(new CheckpointMessage("123", 0L, null));
this.add(new CheckpointMessage(null, 0L, null));
/*
* This procesRecords message will be ignored by the read loop which only cares about status and
* checkpoint messages. All other lines and message types are ignored. By inserting it here, we check
* that this test succeeds even with unexpected messaging.
*/
this.add(new ProcessRecordsMessage());
this.add(new StatusMessage("processRecords"));
}
}));
boolean result = protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST)
.checkpointer(checkpointer).build());
assertThat(result, equalTo(true));
verify(checkpointer, timeout(1)).checkpoint();
verify(checkpointer, timeout(1)).checkpoint("123", 0L);
}
@Test
public void processRecordsWithABadCheckpointTest() throws InterruptedException, ExecutionException {
when(messageWriter.writeProcessRecordsMessage(any(ProcessRecordsInput.class))).thenReturn(buildFuture(true));
when(messageWriter.writeCheckpointMessageWithError(anyString(), anyLong(), any(Throwable.class))).thenReturn(buildFuture(false));
when(messageReader.getNextMessageFromSTDOUT()).thenAnswer(buildMessageAnswers(new ArrayList<Message>() {
{
this.add(new CheckpointMessage("456", 0L, null));
this.add(new StatusMessage("processRecords"));
}
}));
assertThat(protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST)
.checkpointer(checkpointer).build()), equalTo(false));
}
@Test(expected = NullPointerException.class)
public void waitForStatusMessageTimeoutTest() throws InterruptedException, TimeoutException, ExecutionException {
when(messageWriter.writeProcessRecordsMessage(any(ProcessRecordsInput.class))).thenReturn(buildFuture(true));
Future<Message> future = Mockito.mock(Future.class);
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(future);
when(configuration.getTimeoutInSeconds()).thenReturn(Optional.of(5));
when(future.get(anyInt(), eq(TimeUnit.SECONDS))).thenThrow(TimeoutException.class);
protocol = new MultiLangProtocolForTesting(messageReader,
messageWriter,
InitializationInput.builder().shardId(shardId).build(),
configuration);
protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST).build());
}
@Test
public void waitForStatusMessageSuccessTest() {
when(messageWriter.writeProcessRecordsMessage(any(ProcessRecordsInput.class))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("processRecords"), Message.class));
when(configuration.getTimeoutInSeconds()).thenReturn(Optional.of(5));
assertTrue(protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST).build()));
}
private class MultiLangProtocolForTesting extends MultiLangProtocol {
/**
* Constructor.
*
* @param messageReader A message reader.
* @param messageWriter A message writer.
* @param initializationInput
* @param configuration
*/
MultiLangProtocolForTesting(final MessageReader messageReader,
final MessageWriter messageWriter,
final InitializationInput initializationInput,
final KinesisClientLibConfiguration configuration) {
super(messageReader, messageWriter, initializationInput, configuration);
}
@Override
protected void haltJvm(final int exitStatus) {
throw new NullPointerException();
}
}
}

40
amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/StreamingShardRecordProcessorFactoryTest.java
View file

@ -1,40 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import org.junit.Assert;
import org.junit.Test;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;
@RunWith(MockitoJUnitRunner.class)
public class StreamingShardRecordProcessorFactoryTest {
@Mock
private KinesisClientLibConfiguration configuration;
@Test
public void createProcessorTest() {
MultiLangRecordProcessorFactory factory = new MultiLangRecordProcessorFactory("somecommand", null, configuration);
ShardRecordProcessor processor = factory.shardRecordProcessor();
Assert.assertEquals("Should have constructed a StreamingRecordProcessor", MultiLangShardRecordProcessor.class,
processor.getClass());
}
}

103
amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/config/AWSCredentialsProviderPropertyValueDecoderTest.java
View file

@ -1,103 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.config;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
public class AWSCredentialsProviderPropertyValueDecoderTest {
private static final String TEST_ACCESS_KEY_ID = "123";
private static final String TEST_SECRET_KEY = "456";
private String credentialName1 = "com.amazonaws.services.kinesis.multilang.config.AWSCredentialsProviderPropertyValueDecoderTest$AlwaysSucceedCredentialsProvider";
private String credentialName2 = "com.amazonaws.services.kinesis.multilang.config.AWSCredentialsProviderPropertyValueDecoderTest$ConstructorCredentialsProvider";
private AWSCredentialsProviderPropertyValueDecoder decoder = new AWSCredentialsProviderPropertyValueDecoder();
@Test
public void testSingleProvider() {
AwsCredentialsProvider provider = decoder.decodeValue(credentialName1);
assertEquals(provider.getClass(), AwsCredentialsProviderChain.class);
assertEquals(provider.resolveCredentials().accessKeyId(), TEST_ACCESS_KEY_ID);
assertEquals(provider.resolveCredentials().secretAccessKey(), TEST_SECRET_KEY);
}
@Test
public void testTwoProviders() {
AwsCredentialsProvider provider = decoder.decodeValue(credentialName1 + "," + credentialName1);
assertEquals(provider.getClass(), AwsCredentialsProviderChain.class);
assertEquals(provider.resolveCredentials().accessKeyId(), TEST_ACCESS_KEY_ID);
assertEquals(provider.resolveCredentials().secretAccessKey(), TEST_SECRET_KEY);
}
@Test
public void testProfileProviderWithOneArg() {
AwsCredentialsProvider provider = decoder.decodeValue(credentialName2 + "|arg");
assertEquals(provider.getClass(), AwsCredentialsProviderChain.class);
assertEquals(provider.resolveCredentials().accessKeyId(), "arg");
assertEquals(provider.resolveCredentials().secretAccessKey(), "blank");
}
@Test
public void testProfileProviderWithTwoArgs() {
AwsCredentialsProvider provider = decoder.decodeValue(credentialName2 + "|arg1|arg2");
assertEquals(provider.getClass(), AwsCredentialsProviderChain.class);
assertEquals(provider.resolveCredentials().accessKeyId(), "arg1");
assertEquals(provider.resolveCredentials().secretAccessKey(), "arg2");
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProvider implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create(TEST_ACCESS_KEY_ID, TEST_SECRET_KEY);
}
}
/**
* This credentials provider needs a constructor call to instantiate it
*/
public static class ConstructorCredentialsProvider implements AwsCredentialsProvider {
private String arg1;
private String arg2;
public ConstructorCredentialsProvider(String arg1) {
this.arg1 = arg1;
this.arg2 = "blank";
}
public ConstructorCredentialsProvider(String arg1, String arg2) {
this.arg1 = arg1;
this.arg2 = arg2;
}
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create(arg1, arg2);
}
}
}

50
amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/config/DatePropertyValueDecoderTest.java
View file

@ -1,50 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.config;
import static org.junit.Assert.assertEquals;
import java.util.Date;
import org.junit.Test;
public class DatePropertyValueDecoderTest {
private DatePropertyValueDecoder decoder = new DatePropertyValueDecoder();
private static final String TEST_VALUE = "1527267472";
@Test
public void testNumericValue() {
Date timestamp = decoder.decodeValue(TEST_VALUE);
assertEquals(timestamp.getClass(), Date.class);
assertEquals(timestamp, new Date(Long.parseLong(TEST_VALUE) * 1000L));
}
@Test(expected = IllegalArgumentException.class)
public void testEmptyValue() {
Date timestamp = decoder.decodeValue("");
}
@Test(expected = IllegalArgumentException.class)
public void testNullValue() {
Date timestamp = decoder.decodeValue(null);
}
@Test(expected = IllegalArgumentException.class)
public void testNonNumericValue() {
Date timestamp = decoder.decodeValue("123abc");
}
}

419
amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/config/KinesisClientLibConfiguratorTest.java
View file

@ -1,419 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.config;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.Optional;
import java.util.Set;
import org.apache.commons.lang3.StringUtils;
import org.junit.Ignore;
import org.junit.Test;
import com.google.common.collect.ImmutableSet;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.kinesis.common.InitialPositionInStream;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.metrics.MetricsLevel;
public class KinesisClientLibConfiguratorTest {
private String credentialName1 = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysSucceedCredentialsProvider";
private String credentialName2 = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysFailCredentialsProvider";
private String credentialNameKinesis = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysSucceedCredentialsProviderKinesis";
private String credentialNameDynamoDB = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysSucceedCredentialsProviderDynamoDB";
private String credentialNameCloudWatch = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysSucceedCredentialsProviderCloudWatch";
private KinesisClientLibConfigurator configurator = new KinesisClientLibConfigurator();
@Test
public void testWithBasicSetup() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = " + credentialName1, "workerId = 123" }, '\n'));
assertEquals(config.getApplicationName(), "b");
assertEquals(config.getStreamName(), "a");
assertEquals(config.getWorkerIdentifier(), "123");
assertEquals(config.getMaxGetRecordsThreadPool(), Optional.empty());
assertEquals(config.getRetryGetRecordsInSeconds(), Optional.empty());
}
@Test
public void testWithLongVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "applicationName = app",
"streamName = 123", "AWSCredentialsProvider = " + credentialName1 + ", " + credentialName2,
"workerId = 123", "failoverTimeMillis = 100", "shardSyncIntervalMillis = 500" }, '\n'));
assertEquals(config.getApplicationName(), "app");
assertEquals(config.getStreamName(), "123");
assertEquals(config.getWorkerIdentifier(), "123");
assertEquals(config.getFailoverTimeMillis(), 100);
assertEquals(config.getShardSyncIntervalMillis(), 500);
}
@Test
public void testWithUnsupportedClientConfigurationVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(
new String[] { "AWSCredentialsProvider = " + credentialName1 + ", " + credentialName2, "workerId = id",
"kinesisClientConfig = {}", "streamName = stream", "applicationName = b" },
'\n'));
assertEquals(config.getApplicationName(), "b");
assertEquals(config.getStreamName(), "stream");
assertEquals(config.getWorkerIdentifier(), "id");
// by setting the configuration there is no effect on kinesisClientConfiguration variable.
}
@Test
public void testWithIntVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = kinesis",
"AWSCredentialsProvider = " + credentialName2 + ", " + credentialName1, "workerId = w123",
"maxRecords = 10", "metricsMaxQueueSize = 20", "applicationName = kinesis",
"retryGetRecordsInSeconds = 2", "maxGetRecordsThreadPool = 1" }, '\n'));
assertEquals(config.getApplicationName(), "kinesis");
assertEquals(config.getStreamName(), "kinesis");
assertEquals(config.getWorkerIdentifier(), "w123");
assertEquals(config.getMaxRecords(), 10);
assertEquals(config.getMetricsMaxQueueSize(), 20);
assertEquals(config.getRetryGetRecordsInSeconds(), Optional.of(2));
assertEquals(config.getMaxGetRecordsThreadPool(), Optional.of(1));
}
@Test
public void testWithBooleanVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD, " + credentialName1, "workerId = 0",
"cleanupLeasesUponShardCompletion = false", "validateSequenceNumberBeforeCheckpointing = true" },
'\n'));
assertEquals(config.getApplicationName(), "b");
assertEquals(config.getStreamName(), "a");
assertEquals(config.getWorkerIdentifier(), "0");
assertFalse(config.shouldCleanupLeasesUponShardCompletion());
assertTrue(config.shouldValidateSequenceNumberBeforeCheckpointing());
}
@Test
public void testWithStringVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 1",
"kinesisEndpoint = https://kinesis", "metricsLevel = SUMMARY" }, '\n'));
assertEquals(config.getWorkerIdentifier(), "1");
assertEquals(config.getKinesisEndpoint(), "https://kinesis");
assertEquals(config.getMetricsLevel(), MetricsLevel.SUMMARY);
}
@Test
public void testWithSetVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 1",
"metricsEnabledDimensions = ShardId, WorkerIdentifier" }, '\n'));
Set<String> expectedMetricsEnabledDimensions = ImmutableSet.<String> builder()
.add("ShardId", "WorkerIdentifier")
.addAll(KinesisClientLibConfiguration.METRICS_ALWAYS_ENABLED_DIMENSIONS).build();
assertEquals(config.getMetricsEnabledDimensions(), expectedMetricsEnabledDimensions);
}
@Test
public void testWithInitialPositionInStreamVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 123",
"initialPositionInStream = TriM_Horizon" }, '\n'));
assertEquals(config.getInitialPositionInStream(), InitialPositionInStream.TRIM_HORIZON);
}
@Test
public void testSkippingNonKCLVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 123",
"initialPositionInStream = TriM_Horizon", "abc = 1" }, '\n'));
assertEquals(config.getApplicationName(), "b");
assertEquals(config.getStreamName(), "a");
assertEquals(config.getWorkerIdentifier(), "123");
assertEquals(config.getInitialPositionInStream(), InitialPositionInStream.TRIM_HORIZON);
}
@Test
public void testEmptyOptionalVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 123",
"initialPositionInStream = TriM_Horizon", "maxGetRecordsThreadPool = 1" }, '\n'));
assertEquals(config.getMaxGetRecordsThreadPool(), Optional.of(1));
assertEquals(config.getRetryGetRecordsInSeconds(), Optional.empty());
}
@Test
public void testWithZeroValue() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 123",
"initialPositionInStream = TriM_Horizon", "maxGetRecordsThreadPool = 0",
"retryGetRecordsInSeconds = 0" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
try {
configurator.getConfiguration(input);
} catch (Exception e) {
fail("Don't expect to fail on invalid variable value");
}
}
@Test
public void testWithInvalidIntValue() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = " + credentialName1, "workerId = 123", "failoverTimeMillis = 100nf" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
try {
configurator.getConfiguration(input);
} catch (Exception e) {
fail("Don't expect to fail on invalid variable value");
}
}
@Test
public void testWithNegativeIntValue() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = " + credentialName1, "workerId = 123", "failoverTimeMillis = -12" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
configurator.getConfiguration(input);
} catch (Exception e) {
fail("Don't expect to fail on invalid variable value");
}
}
@Test
public void testWithMissingCredentialsProvider() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b", "workerId = 123",
"failoverTimeMillis = 100", "shardSyncIntervalMillis = 500" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
configurator.getConfiguration(input);
fail("expect failure with no credentials provider variables");
} catch (Exception e) {
// succeed
}
}
@Test
public void testWithMissingWorkerId() {
String test = StringUtils.join(
new String[] { "streamName = a", "applicationName = b", "AWSCredentialsProvider = " + credentialName1,
"failoverTimeMillis = 100", "shardSyncIntervalMillis = 500" },
'\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
// if workerId is not provided, configurator should assign one for it automatically
assertNotNull(config.getWorkerIdentifier());
assertFalse(config.getWorkerIdentifier().isEmpty());
}
@Test
public void testWithMissingStreamName() {
String test = StringUtils.join(new String[] { "applicationName = b",
"AWSCredentialsProvider = " + credentialName1, "workerId = 123", "failoverTimeMillis = 100" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
configurator.getConfiguration(input);
fail("expect failure with no stream name variables");
} catch (Exception e) {
// succeed
}
}
@Test
public void testWithMissingApplicationName() {
String test = StringUtils.join(new String[] { "streamName = a", "AWSCredentialsProvider = " + credentialName1,
"workerId = 123", "failoverTimeMillis = 100" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
configurator.getConfiguration(input);
fail("expect failure with no application variables");
} catch (Exception e) {
// succeed
}
}
@Test
public void testWithAWSCredentialsFailed() {
String test = StringUtils.join(
new String[] { "streamName = a", "applicationName = b", "AWSCredentialsProvider = " + credentialName2,
"failoverTimeMillis = 100", "shardSyncIntervalMillis = 500" },
'\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
config.getKinesisCredentialsProvider().resolveCredentials();
fail("expect failure with wrong credentials provider");
} catch (Exception e) {
// succeed
}
}
// TODO: fix this test
@Test
@Ignore
public void testWithDifferentAWSCredentialsForDynamoDBAndCloudWatch() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = " + credentialNameKinesis,
"AWSCredentialsProviderDynamoDB = " + credentialNameDynamoDB,
"AWSCredentialsProviderCloudWatch = " + credentialNameCloudWatch, "failoverTimeMillis = 100",
"shardSyncIntervalMillis = 500" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
try {
config.getKinesisCredentialsProvider().resolveCredentials();
} catch (Exception e) {
fail("Kinesis credential providers should not fail.");
}
try {
config.getDynamoDBCredentialsProvider().resolveCredentials();
} catch (Exception e) {
fail("DynamoDB credential providers should not fail.");
}
try {
config.getCloudWatchCredentialsProvider().resolveCredentials();
} catch (Exception e) {
fail("CloudWatch credential providers should not fail.");
}
}
// TODO: fix this test
@Test
@Ignore
public void testWithDifferentAWSCredentialsForDynamoDBAndCloudWatchFailed() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = " + credentialNameKinesis,
"AWSCredentialsProviderDynamoDB = " + credentialName1,
"AWSCredentialsProviderCloudWatch = " + credentialName1, "failoverTimeMillis = 100",
"shardSyncIntervalMillis = 500" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
try {
config.getKinesisCredentialsProvider().resolveCredentials();
} catch (Exception e) {
fail("Kinesis credential providers should not fail.");
}
try {
config.getDynamoDBCredentialsProvider().resolveCredentials();
fail("DynamoDB credential providers should fail.");
} catch (Exception e) {
// succeed
}
try {
config.getCloudWatchCredentialsProvider().resolveCredentials();
fail("CloudWatch credential providers should fail.");
} catch (Exception e) {
// succeed
}
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProvider implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return null;
}
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProviderKinesis implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create("", "");
}
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProviderDynamoDB implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create("", "");
}
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProviderCloudWatch implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create("", "");
}
}
/**
* This credentials provider will always fail
*/
public static class AlwaysFailCredentialsProvider implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
throw new IllegalArgumentException();
}
}
private KinesisClientLibConfiguration getConfiguration(String configString) {
InputStream input = new ByteArrayInputStream(configString.getBytes());
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
return config;
}
}

83
amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/messages/MessageTest.java
View file

@ -1,83 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.messages;
import java.nio.ByteBuffer;
import java.util.Collections;
import org.junit.Assert;
import org.junit.Test;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
public class MessageTest {
@Test
public void toStringTest() {
Message[] messages = new Message[]{
new CheckpointMessage("1234567890", 0L, null),
new InitializeMessage(InitializationInput.builder().shardId("shard-123").build()),
new ProcessRecordsMessage(ProcessRecordsInput.builder()
.records(Collections.singletonList(
KinesisClientRecord.builder()
.data(ByteBuffer.wrap("cat".getBytes()))
.partitionKey("cat")
.sequenceNumber("555")
.build()))
.build()),
new ShutdownMessage(ShutdownReason.LEASE_LOST),
new StatusMessage("processRecords"),
new InitializeMessage(),
new ProcessRecordsMessage(),
new ShutdownRequestedMessage()
};
// TODO: fix this
for (int i = 0; i < messages.length; i++) {
System.out.println(messages[i].toString());
Assert.assertTrue("Each message should contain the action field", messages[i].toString().contains("action"));
}
// Hit this constructor
KinesisClientRecord defaultJsonFriendlyRecord = KinesisClientRecord.builder().build();
Assert.assertNull(defaultJsonFriendlyRecord.partitionKey());
Assert.assertNull(defaultJsonFriendlyRecord.data());
Assert.assertNull(defaultJsonFriendlyRecord.sequenceNumber());
Assert.assertNull(new ShutdownMessage(null).getReason());
// Hit the bad object mapping path
Message withBadMapper = new Message() {
}.withObjectMapper(new ObjectMapper() {
/**
*
*/
private static final long serialVersionUID = 1L;
@Override
public String writeValueAsString(Object m) throws JsonProcessingException {
throw new JsonProcessingException(new Throwable()) {
};
}
});
String s = withBadMapper.toString();
Assert.assertNotNull(s);
}
}

26
amazon-kinesis-client-multilang/src/test/resources/logback.xml
View file

@ -1,26 +0,0 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
~ Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
~
~ Licensed under the Amazon Software License (the "License").
~ You may not use this file except in compliance with the License.
~ A copy of the License is located at
~
~ http://aws.amazon.com/asl/
~
~ or in the "license" file accompanying this file. This file is distributed
~ on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
~ express or implied. See the License for the specific language governing
~ permissions and limitations under the License.
-->
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d [%thread] %-5level %logger{36} [%mdc{ShardId:-NONE}] - %msg %n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>

332
amazon-kinesis-client/pom.xml
View file

@ -1,332 +0,0 @@
<!--
~ Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
~
~ Licensed under the Amazon Software License (the "License").
~ You may not use this file except in compliance with the License.
~ A copy of the License is located at
~
~ http://aws.amazon.com/asl/
~
~ or in the "license" file accompanying this file. This file is distributed
~ on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
~ express or implied. See the License for the specific language governing
~ permissions and limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-client-pom</artifactId>
<version>2.0.5</version>
</parent>
<artifactId>amazon-kinesis-client</artifactId>
<packaging>jar</packaging>
<name>Amazon Kinesis Client Library for Java</name>
<description>The Amazon Kinesis Client Library for Java enables Java developers to easily consume and process data
from Amazon Kinesis.
</description>
<url>https://aws.amazon.com/kinesis</url>
<scm>
<url>https://github.com/awslabs/amazon-kinesis-client.git</url>
</scm>
<licenses>
<license>
<name>Amazon Software License</name>
<url>https://aws.amazon.com/asl</url>
<distribution>repo</distribution>
</license>
</licenses>
<properties>
<aws-java-sdk.version>1.11.272</aws-java-sdk.version>
<awssdk.version>2.0.6</awssdk.version>
<sqlite4java.version>1.0.392</sqlite4java.version>
<sqlite4java.native>libsqlite4java</sqlite4java.native>
<sqlite4java.libpath>${project.build.directory}/test-lib</sqlite4java.libpath>
<slf4j.version>1.7.25</slf4j.version>
</properties>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>kinesis</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>dynamodb</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>cloudwatch</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>26.0-jre</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.7</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>io.reactivex.rxjava2</groupId>
<artifactId>rxjava</artifactId>
<version>2.1.14</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.16.20</version>
<scope>provided</scope>
</dependency>
<!-- Test -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.19</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
<!--<dependency>-->
<!--<groupId>com.amazonaws</groupId>-->
<!--<artifactId>DynamoDBLocal</artifactId>-->
<!--<version>1.11.86</version>-->
<!--<scope>test</scope>-->
<!--</dependency>-->
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.1.7</version>
<scope>test</scope>
</dependency>
</dependencies>
<!--<repositories>-->
<!--<repository>-->
<!--<id>dynamodblocal</id>-->
<!--<name>AWS DynamoDB Local Release Repository</name>-->
<!--<url>https://s3-us-west-2.amazonaws.com/dynamodb-local/release</url>-->
<!--</repository>-->
<!--</repositories>-->
<developers>
<developer>
<id>amazonwebservices</id>
<organization>Amazon Web Services</organization>
<organizationUrl>https://aws.amazon.com</organizationUrl>
<roles>
<role>developer</role>
</roles>
</developer>
</developers>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<excludes>
<exclude>**/*IntegrationTest.java</exclude>
</excludes>
<systemProperties>
<property>
<name>sqlite4java.library.path</name>
<value>${sqlite4java.libpath}</value>
</property>
</systemProperties>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<includes>
<include>**/*IntegrationTest.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy</id>
<phase>test-compile</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<!-- Mac OS X -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>${sqlite4java.native}-osx</artifactId>
<version>${sqlite4java.version}</version>
<type>dylib</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
<!-- Linux -->
<!-- i386 -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>${sqlite4java.native}-linux-i386</artifactId>
<version>${sqlite4java.version}</version>
<type>so</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
<!-- amd64 -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>${sqlite4java.native}-linux-amd64</artifactId>
<version>${sqlite4java.version}</version>
<type>so</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
<!-- Windows -->
<!-- x86 -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>sqlite4java-win32-x86</artifactId>
<version>${sqlite4java.version}</version>
<type>dll</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
<!-- x64 -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>sqlite4java-win32-x64</artifactId>
<version>${sqlite4java.version}</version>
<type>dll</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.3</version>
<configuration>
<excludePackageNames>com.amazonaws.services.kinesis.producer.protobuf</excludePackageNames>
</configuration>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>disable-java8-doclint</id>
<activation>
<jdk>[1.8,)</jdk>
</activation>
<properties>
<additionalparam>-Xdoclint:none</additionalparam>
</properties>
</profile>
</profiles>
</project>

26
amazon-kinesis-client/src/main/java/software/amazon/kinesis/annotations/KinesisClientInternalApi.java
View file

@ -1,26 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.annotations;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
/**
* Any class/method/variable marked with this annotation is subject to breaking changes between minor releases.
*/
@Retention(RetentionPolicy.CLASS)
public @interface KinesisClientInternalApi {
}

43
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/Checkpoint.java
View file

@ -1,43 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import lombok.Data;
import lombok.experimental.Accessors;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* A class encapsulating the 2 pieces of state stored in a checkpoint.
*/
@Data
@Accessors(fluent = true)
public class Checkpoint {
private final ExtendedSequenceNumber checkpoint;
private final ExtendedSequenceNumber pendingCheckpoint;
/**
* Constructor.
*
* @param checkpoint the checkpoint sequence number - cannot be null or empty.
* @param pendingCheckpoint the pending checkpoint sequence number - can be null.
*/
public Checkpoint(final ExtendedSequenceNumber checkpoint, final ExtendedSequenceNumber pendingCheckpoint) {
if (checkpoint == null || checkpoint.sequenceNumber().isEmpty()) {
throw new IllegalArgumentException("Checkpoint cannot be null or empty");
}
this.checkpoint = checkpoint;
this.pendingCheckpoint = pendingCheckpoint;
}
}

30
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/CheckpointConfig.java
View file

@ -1,30 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import lombok.Data;
import lombok.experimental.Accessors;
import software.amazon.kinesis.checkpoint.dynamodb.DynamoDBCheckpointFactory;
/**
* Used by the KCL to manage checkpointing.
*/
@Data
@Accessors(fluent = true)
public class CheckpointConfig {
private CheckpointFactory checkpointFactory = new DynamoDBCheckpointFactory();
}

27
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/CheckpointFactory.java
View file

@ -1,27 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.processor.Checkpointer;
/**
*
*/
public interface CheckpointFactory {
Checkpointer createCheckpointer(LeaseCoordinator leaseCoordinator, LeaseRefresher leaseRefresher);
}

68
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/DoesNothingPreparedCheckpointer.java
View file

@ -1,68 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.kinesis.processor.PreparedCheckpointer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* A special PreparedCheckpointer that does nothing, which can be used when preparing a checkpoint at the current
* checkpoint sequence number where it is never necessary to do another checkpoint.
* This simplifies programming by preventing application developers from having to reason about whether
* their application has processed records before calling prepareCheckpoint
*
* Here's why it's safe to do nothing:
* The only way to checkpoint at current checkpoint value is to have a record processor that gets
* initialized, processes 0 records, then calls prepareCheckpoint(). The value in the table is the same, so there's
* no reason to overwrite it with another copy of itself.
*/
@KinesisClientInternalApi
public class DoesNothingPreparedCheckpointer implements PreparedCheckpointer {
private final ExtendedSequenceNumber sequenceNumber;
/**
* Constructor.
* @param sequenceNumber the sequence number value
*/
public DoesNothingPreparedCheckpointer(ExtendedSequenceNumber sequenceNumber) {
this.sequenceNumber = sequenceNumber;
}
/**
* {@inheritDoc}
*/
@Override
public ExtendedSequenceNumber pendingCheckpoint() {
return sequenceNumber;
}
/**
* {@inheritDoc}
*/
@Override
public void checkpoint()
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException,
IllegalArgumentException {
// This method does nothing
}
}

188
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/SequenceNumberValidator.java
View file

@ -1,188 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import java.math.BigInteger;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
import org.apache.commons.lang3.StringUtils;
import lombok.Data;
import lombok.experimental.Accessors;
/**
* This supports extracting the shardId from a sequence number.
*
* <h2>Warning</h2>
* <strong>Sequence numbers are an opaque value used by Kinesis, and maybe changed at any time. Should validation stop
* working you may need to update your version of the KCL</strong>
*
*/
public class SequenceNumberValidator {
@Data
@Accessors(fluent = true)
private static class SequenceNumberComponents {
final int version;
final int shardId;
}
private interface SequenceNumberReader {
Optional<SequenceNumberComponents> read(String sequenceNumber);
}
/**
* Reader for the v2 sequence number format. v1 sequence numbers are no longer used or available.
*/
private static class V2SequenceNumberReader implements SequenceNumberReader {
private static final int VERSION = 2;
private static final int EXPECTED_BIT_LENGTH = 186;
private static final int VERSION_OFFSET = 184;
private static final long VERSION_MASK = (1 << 4) - 1;
private static final int SHARD_ID_OFFSET = 4;
private static final long SHARD_ID_MASK = (1L << 32) - 1;
@Override
public Optional<SequenceNumberComponents> read(String sequenceNumberString) {
BigInteger sequenceNumber = new BigInteger(sequenceNumberString, 10);
//
// If the bit length of the sequence number isn't 186 it's impossible for the version numbers
// to be where we expect them. We treat this the same as an unknown version of the sequence number
//
// If the sequence number length isn't what we expect it's due to a new version of the sequence number or
// an invalid sequence number. This
//
if (sequenceNumber.bitLength() != EXPECTED_BIT_LENGTH) {
return Optional.empty();
}
//
// Read the 4 most significant bits of the sequence number, the 2 most significant bits are implicitly 0
// (2 == 0b0011). If the version number doesn't match we give up and say we can't parse the sequence number
//
int version = readOffset(sequenceNumber, VERSION_OFFSET, VERSION_MASK);
if (version != VERSION) {
return Optional.empty();
}
//
// If we get here the sequence number is big enough, and the version matches so the shardId should be valid.
//
int shardId = readOffset(sequenceNumber, SHARD_ID_OFFSET, SHARD_ID_MASK);
return Optional.of(new SequenceNumberComponents(version, shardId));
}
private int readOffset(BigInteger sequenceNumber, int offset, long mask) {
long value = sequenceNumber.shiftRight(offset).longValue() & mask;
return (int) value;
}
}
private static final List<SequenceNumberReader> SEQUENCE_NUMBER_READERS = Collections
.singletonList(new V2SequenceNumberReader());
private Optional<SequenceNumberComponents> retrieveComponentsFor(String sequenceNumber) {
return SEQUENCE_NUMBER_READERS.stream().map(r -> r.read(sequenceNumber)).filter(Optional::isPresent).map(Optional::get).findFirst();
}
/**
* Attempts to retrieve the version for a sequence number. If no reader can be found for the sequence number this
* will return an empty Optional.
*
* <p>
* <strong>This will return an empty Optional if the it's unable to extract the version number. This can occur for
* multiple reasons including:
* <ul>
* <li>Kinesis has started using a new version of sequence numbers</li>
* <li>The provided sequence number isn't a valid Kinesis sequence number.</li>
* </ul>
* </strong>
* </p>
*
* @param sequenceNumber
* the sequence number to extract the version from
* @return an Optional containing the version if a compatible sequence number reader can be found, an empty Optional
* otherwise.
*/
public Optional<Integer> versionFor(String sequenceNumber) {
return retrieveComponentsFor(sequenceNumber).map(SequenceNumberComponents::version);
}
/**
* Attempts to retrieve the shardId from a sequence number. If the version of the sequence number is unsupported
* this will return an empty optional.
*
* <strong>This will return an empty Optional if the sequence number isn't recognized. This can occur for multiple
* reasons including:
* <ul>
* <li>Kinesis has started using a new version of sequence numbers</li>
* <li>The provided sequence number isn't a valid Kinesis sequence number.</li>
* </ul>
* </strong>
* <p>
* This should always return a value if {@link #versionFor(String)} returns a value
* </p>
*
* @param sequenceNumber
* the sequence number to extract the shardId from
* @return an Optional containing the shardId if the version is supported, an empty Optional otherwise.
*/
public Optional<String> shardIdFor(String sequenceNumber) {
return retrieveComponentsFor(sequenceNumber).map(s -> String.format("shardId-%012d", s.shardId()));
}
/**
* Validates that the sequence number provided contains the given shardId. If the sequence number is unsupported
* this will return an empty Optional.
*
* <p>
* Validation of a sequence number will only occur if the sequence number can be parsed. It's possible to use
* {@link #versionFor(String)} to verify that the given sequence number is supported by this class. There are 3
* possible validation states:
* <dl>
* <dt>Some(True)</dt>
* <dd>The sequence number can be parsed, and the shardId matches the one in the sequence number</dd>
* <dt>Some(False)</dt>
* <dd>THe sequence number can be parsed, and the shardId doesn't match the one in the sequence number</dd>
* <dt>None</dt>
* <dd>It wasn't possible to parse the sequence number so the validity of the sequence number is unknown</dd>
* </dl>
* </p>
*
* <p>
* <strong>Handling unknown validation causes is application specific, and not specific handling is
* provided.</strong>
* </p>
*
* @param sequenceNumber
* the sequence number to verify the shardId
* @param shardId
* the shardId that the sequence is expected to contain
* @return true if the sequence number contains the shardId, false if it doesn't. If the sequence number version is
* unsupported this will return an empty Optional
*/
public Optional<Boolean> validateSequenceNumberForShard(String sequenceNumber, String shardId) {
return shardIdFor(sequenceNumber).map(s -> StringUtils.equalsIgnoreCase(s, shardId));
}
}

65
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/ShardPreparedCheckpointer.java
View file

@ -1,65 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.kinesis.processor.PreparedCheckpointer;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Objects of this class are prepared to checkpoint at a specific sequence number. They use an
* RecordProcessorCheckpointer to do the actual checkpointing, so their checkpoint is subject to the same 'didn't go
* backwards' validation as a normal checkpoint.
*/
public class ShardPreparedCheckpointer implements PreparedCheckpointer {
private final ExtendedSequenceNumber pendingCheckpointSequenceNumber;
private final RecordProcessorCheckpointer checkpointer;
/**
* Constructor.
*
* @param pendingCheckpointSequenceNumber sequence number to checkpoint at
* @param checkpointer checkpointer to use
*/
public ShardPreparedCheckpointer(ExtendedSequenceNumber pendingCheckpointSequenceNumber,
RecordProcessorCheckpointer checkpointer) {
this.pendingCheckpointSequenceNumber = pendingCheckpointSequenceNumber;
this.checkpointer = checkpointer;
}
/**
* {@inheritDoc}
*/
@Override
public ExtendedSequenceNumber pendingCheckpoint() {
return pendingCheckpointSequenceNumber;
}
/**
* {@inheritDoc}
*/
@Override
public void checkpoint()
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException,
IllegalArgumentException {
checkpointer.checkpoint(pendingCheckpointSequenceNumber.sequenceNumber(),
pendingCheckpointSequenceNumber.subSequenceNumber());
}
}

323
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/ShardRecordProcessorCheckpointer.java
View file

@ -1,323 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import lombok.Getter;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.kinesis.model.Record;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.KinesisClientLibException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.processor.PreparedCheckpointer;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* This class is used to enable RecordProcessors to checkpoint their progress.
* The Amazon Kinesis Client Library will instantiate an object and provide a reference to the application
* ShardRecordProcessor instance. Amazon Kinesis Client Library will create one instance per shard assignment.
*/
@RequiredArgsConstructor
@Slf4j
public class ShardRecordProcessorCheckpointer implements RecordProcessorCheckpointer {
@NonNull
private final ShardInfo shardInfo;
@NonNull
@Getter @Accessors(fluent = true)
private final Checkpointer checkpointer;
// Set to the last value set via checkpoint().
// Sample use: verify application shutdown() invoked checkpoint() at the end of a shard.
@Getter @Accessors(fluent = true)
private ExtendedSequenceNumber lastCheckpointValue;
@Getter @Accessors(fluent = true)
private ExtendedSequenceNumber largestPermittedCheckpointValue;
private ExtendedSequenceNumber sequenceNumberAtShardEnd;
/**
* {@inheritDoc}
*/
@Override
public synchronized void checkpoint()
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
if (log.isDebugEnabled()) {
log.debug("Checkpointing {}, token {} at largest permitted value {}", shardInfo.shardId(),
shardInfo.concurrencyToken(), this.largestPermittedCheckpointValue);
}
advancePosition(this.largestPermittedCheckpointValue);
}
/**
* {@inheritDoc}
*/
@Override
public synchronized void checkpoint(Record record)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException,
IllegalArgumentException {
// TODO: UserRecord Deprecation
if (record == null) {
throw new IllegalArgumentException("Could not checkpoint a null record");
} /* else if (record instanceof UserRecord) {
checkpoint(record.sequenceNumber(), ((UserRecord) record).subSequenceNumber());
} */ else {
checkpoint(record.sequenceNumber(), 0);
}
}
/**
* {@inheritDoc}
*/
@Override
public synchronized void checkpoint(String sequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException,
IllegalArgumentException {
checkpoint(sequenceNumber, 0);
}
/**
* {@inheritDoc}
*/
@Override
public synchronized void checkpoint(String sequenceNumber, long subSequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException,
IllegalArgumentException {
if (subSequenceNumber < 0) {
throw new IllegalArgumentException("Could not checkpoint at invalid, negative subsequence number "
+ subSequenceNumber);
}
/*
* If there isn't a last checkpoint value, we only care about checking the upper bound.
* If there is a last checkpoint value, we want to check both the lower and upper bound.
*/
ExtendedSequenceNumber newCheckpoint = new ExtendedSequenceNumber(sequenceNumber, subSequenceNumber);
if ((lastCheckpointValue == null || lastCheckpointValue.compareTo(newCheckpoint) <= 0)
&& newCheckpoint.compareTo(largestPermittedCheckpointValue) <= 0) {
if (log.isDebugEnabled()) {
log.debug("Checkpointing {}, token {} at specific extended sequence number {}", shardInfo.shardId(),
shardInfo.concurrencyToken(), newCheckpoint);
}
this.advancePosition(newCheckpoint);
} else {
throw new IllegalArgumentException(String.format(
"Could not checkpoint at extended sequence number %s as it did not fall into acceptable range "
+ "between the last checkpoint %s and the greatest extended sequence number passed to this "
+ "record processor %s",
newCheckpoint, this.lastCheckpointValue, this.largestPermittedCheckpointValue));
}
}
/**
* {@inheritDoc}
*/
@Override
public synchronized PreparedCheckpointer prepareCheckpoint()
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
return this.prepareCheckpoint(
this.largestPermittedCheckpointValue.sequenceNumber(),
this.largestPermittedCheckpointValue.subSequenceNumber());
}
/**
* {@inheritDoc}
*/
@Override
public synchronized PreparedCheckpointer prepareCheckpoint(Record record)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
//
// TODO: UserRecord Deprecation
//
if (record == null) {
throw new IllegalArgumentException("Could not prepare checkpoint a null record");
} /*else if (record instanceof UserRecord) {
return prepareCheckpoint(record.sequenceNumber(), ((UserRecord) record).subSequenceNumber());
} */ else {
return prepareCheckpoint(record.sequenceNumber(), 0);
}
}
/**
* {@inheritDoc}
*/
@Override
public synchronized PreparedCheckpointer prepareCheckpoint(String sequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
return prepareCheckpoint(sequenceNumber, 0);
}
/**
* {@inheritDoc}
*/
@Override
public synchronized PreparedCheckpointer prepareCheckpoint(String sequenceNumber, long subSequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
if (subSequenceNumber < 0) {
throw new IllegalArgumentException("Could not checkpoint at invalid, negative subsequence number "
+ subSequenceNumber);
}
/*
* If there isn't a last checkpoint value, we only care about checking the upper bound.
* If there is a last checkpoint value, we want to check both the lower and upper bound.
*/
ExtendedSequenceNumber pendingCheckpoint = new ExtendedSequenceNumber(sequenceNumber, subSequenceNumber);
if ((lastCheckpointValue == null || lastCheckpointValue.compareTo(pendingCheckpoint) <= 0)
&& pendingCheckpoint.compareTo(largestPermittedCheckpointValue) <= 0) {
if (log.isDebugEnabled()) {
log.debug("Preparing checkpoint {}, token {} at specific extended sequence number {}",
shardInfo.shardId(), shardInfo.concurrencyToken(), pendingCheckpoint);
}
return doPrepareCheckpoint(pendingCheckpoint);
} else {
throw new IllegalArgumentException(String.format(
"Could not prepare checkpoint at extended sequence number %s as it did not fall into acceptable "
+ "range between the last checkpoint %s and the greatest extended sequence number passed "
+ "to this record processor %s",
pendingCheckpoint, this.lastCheckpointValue, this.largestPermittedCheckpointValue));
}
}
public synchronized void setInitialCheckpointValue(ExtendedSequenceNumber initialCheckpoint) {
lastCheckpointValue = initialCheckpoint;
}
/**
* @param largestPermittedCheckpointValue the largest permitted checkpoint
*/
public synchronized void largestPermittedCheckpointValue(ExtendedSequenceNumber largestPermittedCheckpointValue) {
this.largestPermittedCheckpointValue = largestPermittedCheckpointValue;
}
/**
* Used to remember the last extended sequence number before SHARD_END to allow us to prevent the checkpointer
* from checkpointing at the end of the shard twice (i.e. at the last extended sequence number and then again
* at SHARD_END).
*
* @param extendedSequenceNumber
*/
public synchronized void sequenceNumberAtShardEnd(ExtendedSequenceNumber extendedSequenceNumber) {
this.sequenceNumberAtShardEnd = extendedSequenceNumber;
}
/**
* Internal API - has package level access only for testing purposes.
*
* @param sequenceNumber
*
* @throws KinesisClientLibDependencyException
* @throws ThrottlingException
* @throws ShutdownException
* @throws InvalidStateException
*/
void advancePosition(String sequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
advancePosition(new ExtendedSequenceNumber(sequenceNumber));
}
void advancePosition(ExtendedSequenceNumber extendedSequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
ExtendedSequenceNumber checkpointToRecord = extendedSequenceNumber;
if (sequenceNumberAtShardEnd != null && sequenceNumberAtShardEnd.equals(extendedSequenceNumber)) {
// If we are about to checkpoint the very last sequence number for this shard, we might as well
// just checkpoint at SHARD_END
checkpointToRecord = ExtendedSequenceNumber.SHARD_END;
}
// Don't checkpoint a value we already successfully checkpointed
if (extendedSequenceNumber != null && !extendedSequenceNumber.equals(lastCheckpointValue)) {
try {
if (log.isDebugEnabled()) {
log.debug("Setting {}, token {} checkpoint to {}", shardInfo.shardId(),
shardInfo.concurrencyToken(), checkpointToRecord);
}
checkpointer.setCheckpoint(shardInfo.shardId(), checkpointToRecord, shardInfo.concurrencyToken());
lastCheckpointValue = checkpointToRecord;
} catch (ThrottlingException | ShutdownException | InvalidStateException
| KinesisClientLibDependencyException e) {
throw e;
} catch (KinesisClientLibException e) {
log.warn("Caught exception setting checkpoint.", e);
throw new KinesisClientLibDependencyException("Caught exception while checkpointing", e);
}
}
}
/**
* This method stores the given sequenceNumber as a pending checkpoint in the lease table without overwriting the
* current checkpoint, then returns a PreparedCheckpointer that is ready to checkpoint at the given sequence number.
*
* This method does not advance lastCheckpointValue, but calls to PreparedCheckpointer.checkpoint() on the returned
* objects do. This allows customers to 'discard' prepared checkpoints by calling any of the 4 checkpoint methods on
* this class before calling PreparedCheckpointer.checkpoint(). Some examples:
*
* 1) prepareCheckpoint(snA); checkpoint(snB). // this works regardless of whether snA or snB is bigger. It discards
* the prepared checkpoint at snA.
* 2) prepareCheckpoint(snA); prepareCheckpoint(snB). // this works regardless of whether snA or snB is bigger. It
* replaces the preparedCheckpoint at snA with a new one at snB.
* 3) checkpointA = prepareCheckpoint(snA); checkpointB = prepareCheckpoint(snB); checkpointB.checkpoint();
* checkpointerA.checkpoint(); // This replaces the prepared checkpoint at snA with a new one at snB, then
* checkpoints at snB regardless of whether snA or snB is bigger. The checkpoint at snA only succeeds if snA > snB.
*
* @param extendedSequenceNumber the sequence number for the prepared checkpoint
* @return a prepared checkpoint that is ready to checkpoint at the given sequence number.
* @throws KinesisClientLibDependencyException
* @throws InvalidStateException
* @throws ThrottlingException
* @throws ShutdownException
*/
private PreparedCheckpointer doPrepareCheckpoint(ExtendedSequenceNumber extendedSequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
ExtendedSequenceNumber newPrepareCheckpoint = extendedSequenceNumber;
if (sequenceNumberAtShardEnd != null && sequenceNumberAtShardEnd.equals(extendedSequenceNumber)) {
// If we are about to checkpoint the very last sequence number for this shard, we might as well
// just checkpoint at SHARD_END
newPrepareCheckpoint = ExtendedSequenceNumber.SHARD_END;
}
// Don't actually prepare a checkpoint if they're trying to checkpoint at the current checkpointed value.
// The only way this can happen is if they call prepareCheckpoint() in a record processor that was initialized
// AND that has not processed any records since initialization.
if (newPrepareCheckpoint.equals(lastCheckpointValue)) {
return new DoesNothingPreparedCheckpointer(newPrepareCheckpoint);
}
try {
checkpointer.prepareCheckpoint(shardInfo.shardId(), newPrepareCheckpoint, shardInfo.concurrencyToken());
} catch (ThrottlingException | ShutdownException | InvalidStateException
| KinesisClientLibDependencyException e) {
throw e;
} catch (KinesisClientLibException e) {
log.warn("Caught exception setting prepareCheckpoint.", e);
throw new KinesisClientLibDependencyException("Caught exception while prepareCheckpointing", e);
}
ShardPreparedCheckpointer result = new ShardPreparedCheckpointer(newPrepareCheckpoint, this);
return result;
}
}

37
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/dynamodb/DynamoDBCheckpointFactory.java
View file

@ -1,37 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint.dynamodb;
import lombok.Data;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.checkpoint.CheckpointFactory;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.processor.Checkpointer;
/**
*
*/
@Data
@KinesisClientInternalApi
public class DynamoDBCheckpointFactory implements CheckpointFactory {
@Override
public Checkpointer createCheckpointer(final LeaseCoordinator leaseLeaseCoordinator,
final LeaseRefresher leaseRefresher) {
return new DynamoDBCheckpointer(leaseLeaseCoordinator, leaseRefresher);
}
}

159
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/dynamodb/DynamoDBCheckpointer.java
View file

@ -1,159 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint.dynamodb;
import java.util.Objects;
import java.util.UUID;
import com.google.common.annotations.VisibleForTesting;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.checkpoint.Checkpoint;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.KinesisClientLibException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.kinesis.exceptions.internal.KinesisClientLibIOException;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
*
*/
@RequiredArgsConstructor
@Slf4j
@KinesisClientInternalApi
public class DynamoDBCheckpointer implements Checkpointer {
@NonNull
private final LeaseCoordinator leaseCoordinator;
@NonNull
private final LeaseRefresher leaseRefresher;
private String operation;
@Override
public void setCheckpoint(final String shardId, final ExtendedSequenceNumber checkpointValue,
final String concurrencyToken) throws KinesisClientLibException {
try {
boolean wasSuccessful = setCheckpoint(shardId, checkpointValue, UUID.fromString(concurrencyToken));
if (!wasSuccessful) {
throw new ShutdownException("Can't update checkpoint - instance doesn't hold the lease for this shard");
}
} catch (ProvisionedThroughputException e) {
throw new ThrottlingException("Got throttled while updating checkpoint.", e);
} catch (InvalidStateException e) {
String message = "Unable to save checkpoint for shardId " + shardId;
log.error(message, e);
throw new software.amazon.kinesis.exceptions.InvalidStateException(message, e);
} catch (DependencyException e) {
throw new KinesisClientLibDependencyException("Unable to save checkpoint for shardId " + shardId, e);
}
}
@Override
public ExtendedSequenceNumber getCheckpoint(final String shardId) throws KinesisClientLibException {
try {
return leaseRefresher.getLease(shardId).checkpoint();
} catch (DependencyException | InvalidStateException | ProvisionedThroughputException e) {
String message = "Unable to fetch checkpoint for shardId " + shardId;
log.error(message, e);
throw new KinesisClientLibIOException(message, e);
}
}
@Override
public Checkpoint getCheckpointObject(final String shardId) throws KinesisClientLibException {
try {
Lease lease = leaseRefresher.getLease(shardId);
log.debug("[{}] Retrieved lease => {}", shardId, lease);
return new Checkpoint(lease.checkpoint(), lease.pendingCheckpoint());
} catch (DependencyException | InvalidStateException | ProvisionedThroughputException e) {
String message = "Unable to fetch checkpoint for shardId " + shardId;
log.error(message, e);
throw new KinesisClientLibIOException(message, e);
}
}
@Override
public void prepareCheckpoint(final String shardId, final ExtendedSequenceNumber pendingCheckpoint,
final String concurrencyToken) throws KinesisClientLibException {
try {
boolean wasSuccessful =
prepareCheckpoint(shardId, pendingCheckpoint, UUID.fromString(concurrencyToken));
if (!wasSuccessful) {
throw new ShutdownException(
"Can't prepare checkpoint - instance doesn't hold the lease for this shard");
}
} catch (ProvisionedThroughputException e) {
throw new ThrottlingException("Got throttled while preparing checkpoint.", e);
} catch (InvalidStateException e) {
String message = "Unable to prepare checkpoint for shardId " + shardId;
log.error(message, e);
throw new software.amazon.kinesis.exceptions.InvalidStateException(message, e);
} catch (DependencyException e) {
throw new KinesisClientLibDependencyException("Unable to prepare checkpoint for shardId " + shardId, e);
}
}
@VisibleForTesting
public boolean setCheckpoint(String shardId, ExtendedSequenceNumber checkpoint, UUID concurrencyToken)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
Lease lease = leaseCoordinator.getCurrentlyHeldLease(shardId);
if (lease == null) {
log.info("Worker {} could not update checkpoint for shard {} because it does not hold the lease",
leaseCoordinator.workerIdentifier(), shardId);
return false;
}
lease.checkpoint(checkpoint);
lease.pendingCheckpoint(null);
lease.ownerSwitchesSinceCheckpoint(0L);
return leaseCoordinator.updateLease(lease, concurrencyToken, operation, shardId);
}
boolean prepareCheckpoint(String shardId, ExtendedSequenceNumber pendingCheckpoint, UUID concurrencyToken)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
Lease lease = leaseCoordinator.getCurrentlyHeldLease(shardId);
if (lease == null) {
log.info("Worker {} could not prepare checkpoint for shard {} because it does not hold the lease",
leaseCoordinator.workerIdentifier(), shardId);
return false;
}
lease.pendingCheckpoint(Objects.requireNonNull(pendingCheckpoint, "pendingCheckpoint should not be null"));
return leaseCoordinator.updateLease(lease, concurrencyToken, operation, shardId);
}
@Override
public void operation(@NonNull final String operation) {
this.operation = operation;
}
@Override
public String operation() {
return operation;
}
}

175
amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/ConfigsBuilder.java
View file

@ -1,175 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.common;
import org.apache.commons.lang3.StringUtils;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import software.amazon.awssdk.services.cloudwatch.CloudWatchAsyncClient;
import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.kinesis.checkpoint.CheckpointConfig;
import software.amazon.kinesis.coordinator.CoordinatorConfig;
import software.amazon.kinesis.leases.LeaseManagementConfig;
import software.amazon.kinesis.lifecycle.LifecycleConfig;
import software.amazon.kinesis.metrics.MetricsConfig;
import software.amazon.kinesis.processor.ProcessorConfig;
import software.amazon.kinesis.processor.ShardRecordProcessorFactory;
import software.amazon.kinesis.retrieval.RetrievalConfig;
/**
* This Builder is useful to create all configurations for the KCL with default values.
*/
@Data
@Accessors(fluent = true)
public class ConfigsBuilder {
/**
* Name of the stream to consume records from
*/
@NonNull
private final String streamName;
/**
* Application name for the KCL Worker
*/
@NonNull
private final String applicationName;
/**
* KinesisClient to be used to consumer records from Kinesis
*/
@NonNull
private final KinesisAsyncClient kinesisClient;
/**
* DynamoDBClient to be used to interact with DynamoDB service for lease management and checkpoiniting
*/
@NonNull
private final DynamoDbAsyncClient dynamoDBClient;
/**
* CloudWatchClient to be used to push KCL metrics to CloudWatch service
*/
@NonNull
private final CloudWatchAsyncClient cloudWatchClient;
/**
* KCL worker identifier to distinguish between 2 unique workers
*/
@NonNull
private final String workerIdentifier;
/**
* ShardRecordProcessorFactory to be used to create ShardRecordProcesor for processing records
*/
@NonNull
private final ShardRecordProcessorFactory shardRecordProcessorFactory;
/**
* Lease table name used for lease management and checkpointing.
*/
private String tableName;
/**
* Lease table name used for lease management and checkpointing.
*
* @return DynamoDB table name
*/
public String tableName() {
if (StringUtils.isEmpty(tableName)) {
tableName = applicationName();
}
return tableName;
}
/**
* CloudWatch namespace for KCL metrics.
*/
private String namespace;
/**
* CloudWatch namespace for KCL metrics.
*
* @return CloudWatch namespace
*/
public String namespace() {
if (StringUtils.isEmpty(namespace)) {
namespace = applicationName();
}
return namespace;
}
/**
* Creates a new instance of CheckpointConfig
*
* @return CheckpointConfig
*/
public CheckpointConfig checkpointConfig() {
return new CheckpointConfig();
}
/**
* Creates a new instance of CoordinatorConfig
*
* @return CoordinatorConfig
*/
public CoordinatorConfig coordinatorConfig() {
return new CoordinatorConfig(applicationName());
}
/**
* Creates a new instance of LeaseManagementConfig
*
* @return LeaseManagementConfig
*/
public LeaseManagementConfig leaseManagementConfig() {
return new LeaseManagementConfig(tableName(), dynamoDBClient(), kinesisClient(), streamName(),
workerIdentifier());
}
/**
* Creates a new instance of LifecycleConfig
*
* @return LifecycleConfig
*/
public LifecycleConfig lifecycleConfig() {
return new LifecycleConfig();
}
/**
* Creates a new instance of MetricsConfig
*
* @return MetricsConfig
*/
public MetricsConfig metricsConfig() {
return new MetricsConfig(cloudWatchClient(), namespace());
}
/**
* Creates a new instance of ProcessorConfig
*
* @return ProcessorConfigConfig
*/
public ProcessorConfig processorConfig() {
return new ProcessorConfig(shardRecordProcessorFactory());
}
/**
* Creates a new instance of RetrievalConfig
*
* @return RetrievalConfig
*/
public RetrievalConfig retrievalConfig() {
return new RetrievalConfig(kinesisClient(), streamName(), applicationName());
}
}

36
amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java
View file

@ -1,36 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.common;
/**
* Used to specify the position in the stream where a new application should start from.
* This is used during initial application bootstrap (when a checkpoint doesn't exist for a shard or its parents).
*/
public enum InitialPositionInStream {
/**
* Start after the most recent data record (fetch new data).
*/
LATEST,
/**
* Start from the oldest available data record.
*/
TRIM_HORIZON,
/**
* Start from the record at or after the specified server-side timestamp.
*/
AT_TIMESTAMP
}

37
amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/KinesisClientUtil.java
View file

@ -1,37 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.common;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClientBuilder;
/**
* Utility to setup KinesisAsyncClient to be used with KCL.
*/
public class KinesisClientUtil {
/**
* Creates a client from a builder.
*
* @param clientBuilder
* @return
*/
public static KinesisAsyncClient createKinesisAsyncClient(KinesisAsyncClientBuilder clientBuilder) {
return clientBuilder.httpClientBuilder(NettyNioAsyncHttpClient.builder().maxConcurrency(Integer.MAX_VALUE))
.build();
}
}

74
amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/KinesisRequestsBuilder.java
View file

@ -1,74 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.common;
import software.amazon.awssdk.awscore.AwsRequest;
import software.amazon.awssdk.awscore.AwsRequestOverrideConfiguration;
import software.amazon.awssdk.core.ApiName;
import software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerRequest;
import software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryRequest;
import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
import software.amazon.awssdk.services.kinesis.model.RegisterStreamConsumerRequest;
import software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.retrieval.RetrievalConfig;
/**
*
*/
@KinesisClientInternalApi
public class KinesisRequestsBuilder {
public static ListShardsRequest.Builder listShardsRequestBuilder() {
return appendUserAgent(ListShardsRequest.builder());
}
public static SubscribeToShardRequest.Builder subscribeToShardRequestBuilder() {
return appendUserAgent(SubscribeToShardRequest.builder());
}
public static GetRecordsRequest.Builder getRecordsRequestBuilder() {
return appendUserAgent(GetRecordsRequest.builder());
}
public static GetShardIteratorRequest.Builder getShardIteratorRequestBuilder() {
return appendUserAgent(GetShardIteratorRequest.builder());
}
public static DescribeStreamSummaryRequest.Builder describeStreamSummaryRequestBuilder() {
return appendUserAgent(DescribeStreamSummaryRequest.builder());
}
public static RegisterStreamConsumerRequest.Builder registerStreamConsumerRequestBuilder() {
return appendUserAgent(RegisterStreamConsumerRequest.builder());
}
public static DescribeStreamConsumerRequest.Builder describeStreamConsumerRequestBuilder() {
return appendUserAgent(DescribeStreamConsumerRequest.builder());
}
@SuppressWarnings("unchecked")
private static <T extends AwsRequest.Builder> T appendUserAgent(final T builder) {
return (T) builder
.overrideConfiguration(
AwsRequestOverrideConfiguration.builder()
.addApiName(ApiName.builder().name(RetrievalConfig.KINESIS_CLIENT_LIB_USER_AGENT)
.version(RetrievalConfig.KINESIS_CLIENT_LIB_USER_AGENT_VERSION).build())
.build());
}
}

93
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/CoordinatorConfig.java
View file

@ -1,93 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import software.amazon.kinesis.leases.NoOpShardPrioritization;
import software.amazon.kinesis.leases.ShardPrioritization;
/**
* Used by the KCL to configure the coordinator.
*/
@Data
@Accessors(fluent = true)
public class CoordinatorConfig {
/**
* Application name used by checkpointer to checkpoint.
*
* @return String
*/
@NonNull
private final String applicationName;
/**
* The maximum number of attempts to initialize the Scheduler
*
* <p>Default value: 20</p>
*/
private int maxInitializationAttempts = 20;
/**
* Interval in milliseconds between polling to check for parent shard completion.
* Polling frequently will take up more DynamoDB IOPS (when there are leases for shards waiting on
* completion of parent shards).
*
* <p>Default value: 10000L</p>
*/
private long parentShardPollIntervalMillis = 10000L;
/**
* The Scheduler will skip shard sync during initialization if there are one or more leases in the lease table. This
* assumes that the shards and leases are in-sync. This enables customers to choose faster startup times (e.g.
* during incremental deployments of an application).
*
* <p>Default value: false</p>
*/
private boolean skipShardSyncAtWorkerInitializationIfLeasesExist = false;
/**
* The number of milliseconds between polling of the shard consumer for triggering state changes, and health checks.
*
* <p>Default value: 1000 milliseconds</p>
*/
private long shardConsumerDispatchPollIntervalMillis = 1000L;
/**
* Shard prioritization strategy.
*
* <p>Default value: {@link NoOpShardPrioritization}</p>
*/
private ShardPrioritization shardPrioritization = new NoOpShardPrioritization();
/**
* WorkerStateChangeListener to be used by the Scheduler.
*
* <p>Default value: {@link NoOpWorkerStateChangeListener}</p>
*/
private WorkerStateChangeListener workerStateChangeListener = new NoOpWorkerStateChangeListener();
/**
* GracefulShutdownCoordinator to be used by the Scheduler.
*
* <p>Default value: {@link GracefulShutdownCoordinator}</p>
*/
private GracefulShutdownCoordinator gracefulShutdownCoordinator = new GracefulShutdownCoordinator();
private CoordinatorFactory coordinatorFactory = new SchedulerCoordinatorFactory();
}

89
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/CoordinatorFactory.java
View file

@ -1,89 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import java.util.concurrent.ExecutorService;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.processor.Checkpointer;
/**
* Used in the process of configuring and providing instances to the {@link Scheduler}
*/
public interface CoordinatorFactory {
/**
* Creates the executor service to be used by the Scheduler.
*
* @return ExecutorService
*/
ExecutorService createExecutorService();
/**
* Creates GracefulShutdownCoordinator to be used by the Scheduler.
*
* <h3>Method Deprecated</h3>
* <p>
* <strong>Note: This method has been deprecated, and will be removed in a future release. Use the configuration in
* {@link CoordinatorConfig#gracefulShutdownCoordinator}. Set the
* {@link CoordinatorConfig#gracefulShutdownCoordinator} to null in order to use this method.</strong>
* </p>
* <h4>Resolution Order</h3>
* <ol>
* <li>{@link CoordinatorConfig#gracefulShutdownCoordinator()}</li>
* <li>{@link CoordinatorFactory#createGracefulShutdownCoordinator()}</li>
* </ol>
*
*
* @return a {@link GracefulShutdownCoordinator} that manages the process of shutting down the scheduler.
*/
@Deprecated
default GracefulShutdownCoordinator createGracefulShutdownCoordinator() {
return new GracefulShutdownCoordinator();
}
/**
* Creates a WorkerStateChangeListener to be used by the Scheduler.
*
* <h3>Method Deprecated</h3>
* <p>
* <strong>Note: This method has been deprecated, and will be removed in a future release. Use the configuration in
* {@link CoordinatorConfig#workerStateChangeListener}. Set the
* {@link CoordinatorConfig#workerStateChangeListener} to null in order to use this method.</strong>
* </p>
*
* <h4>Resolution Order</h3>
* <ol>
* <li>{@link CoordinatorConfig#workerStateChangeListener()}</li>
* <li>{@link CoordinatorFactory#createWorkerStateChangeListener()}</li>
* </ol>
*
* @return a {@link WorkerStateChangeListener} instance that will be notified for specific {@link Scheduler} steps.
*/
@Deprecated
default WorkerStateChangeListener createWorkerStateChangeListener() {
return new NoOpWorkerStateChangeListener();
}
/**
* Creates a RecordProcessorChedckpointer to be used by the Scheduler.
*
* @param shardInfo ShardInfo to be used in order to create the ShardRecordProcessorCheckpointer
* @param checkpoint Checkpointer to be used in order to create Shardthe RecordProcessorCheckpointer
* @return ShardRecordProcessorCheckpointer
*/
ShardRecordProcessorCheckpointer createRecordProcessorCheckpointer(ShardInfo shardInfo, Checkpointer checkpoint);
}

35
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/GracefulShutdownContext.java
View file

@ -1,35 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import lombok.Data;
import lombok.experimental.Accessors;
import java.util.concurrent.CountDownLatch;
@Data
@Accessors(fluent = true)
class GracefulShutdownContext {
private final CountDownLatch shutdownCompleteLatch;
private final CountDownLatch notificationCompleteLatch;
private final Scheduler scheduler;
static GracefulShutdownContext SHUTDOWN_ALREADY_COMPLETED = new GracefulShutdownContext(null, null, null);
boolean isShutdownAlreadyCompleted() {
return shutdownCompleteLatch == null && notificationCompleteLatch == null && scheduler == null;
}
}

160
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/GracefulShutdownCoordinator.java
View file

@ -1,160 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import java.util.concurrent.Callable;
import java.util.concurrent.Future;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
class GracefulShutdownCoordinator {
Future<Boolean> startGracefulShutdown(Callable<Boolean> shutdownCallable) {
FutureTask<Boolean> task = new FutureTask<>(shutdownCallable);
Thread shutdownThread = new Thread(task, "RequestedShutdownThread");
shutdownThread.start();
return task;
}
Callable<Boolean> createGracefulShutdownCallable(Callable<GracefulShutdownContext> startWorkerShutdown) {
return new GracefulShutdownCallable(startWorkerShutdown);
}
@Slf4j
static class GracefulShutdownCallable implements Callable<Boolean> {
private final Callable<GracefulShutdownContext> startWorkerShutdown;
GracefulShutdownCallable(Callable<GracefulShutdownContext> startWorkerShutdown) {
this.startWorkerShutdown = startWorkerShutdown;
}
private boolean isWorkerShutdownComplete(GracefulShutdownContext context) {
return context.scheduler().shutdownComplete() || context.scheduler().shardInfoShardConsumerMap().isEmpty();
}
private String awaitingLogMessage(GracefulShutdownContext context) {
long awaitingNotification = context.notificationCompleteLatch().getCount();
long awaitingFinalShutdown = context.shutdownCompleteLatch().getCount();
return String.format(
"Waiting for %d record process to complete shutdown notification, and %d record processor to complete final shutdown ",
awaitingNotification, awaitingFinalShutdown);
}
private String awaitingFinalShutdownMessage(GracefulShutdownContext context) {
long outstanding = context.shutdownCompleteLatch().getCount();
return String.format("Waiting for %d record processors to complete final shutdown", outstanding);
}
private boolean waitForRecordProcessors(GracefulShutdownContext context) {
//
// Awaiting for all ShardConsumer/RecordProcessors to be notified that a shutdown has been requested.
// There is the possibility of a race condition where a lease is terminated after the shutdown request
// notification is started, but before the ShardConsumer is sent the notification. In this case the
// ShardConsumer would start the lease loss shutdown, and may never call the notification methods.
//
try {
while (!context.notificationCompleteLatch().await(1, TimeUnit.SECONDS)) {
if (Thread.interrupted()) {
throw new InterruptedException();
}
log.info(awaitingLogMessage(context));
if (workerShutdownWithRemaining(context.shutdownCompleteLatch().getCount(), context)) {
return false;
}
}
} catch (InterruptedException ie) {
log.warn("Interrupted while waiting for notification complete, terminating shutdown. {}",
awaitingLogMessage(context));
return false;
}
if (Thread.interrupted()) {
log.warn("Interrupted before worker shutdown, terminating shutdown");
return false;
}
//
// Once all record processors have been notified of the shutdown it is safe to allow the worker to
// start its shutdown behavior. Once shutdown starts it will stop renewer, and drop any remaining leases.
//
context.scheduler().shutdown();
if (Thread.interrupted()) {
log.warn("Interrupted after worker shutdown, terminating shutdown");
return false;
}
//
// Want to wait for all the remaining ShardConsumers/ShardRecordProcessor's to complete their final shutdown
// processing. This should really be a no-op since as part of the notification completion the lease for
// ShardConsumer is terminated.
//
try {
while (!context.shutdownCompleteLatch().await(1, TimeUnit.SECONDS)) {
if (Thread.interrupted()) {
throw new InterruptedException();
}
log.info(awaitingFinalShutdownMessage(context));
if (workerShutdownWithRemaining(context.shutdownCompleteLatch().getCount(), context)) {
return false;
}
}
} catch (InterruptedException ie) {
log.warn("Interrupted while waiting for shutdown completion, terminating shutdown. {}",
awaitingFinalShutdownMessage(context));
return false;
}
return true;
}
/**
* This checks to see if the worker has already hit it's shutdown target, while there is outstanding record
* processors. This maybe a little racy due to when the value of outstanding is retrieved. In general though the
* latch should be decremented before the shutdown completion.
*
* @param outstanding
* the number of record processor still awaiting shutdown.
*/
private boolean workerShutdownWithRemaining(long outstanding, GracefulShutdownContext context) {
if (isWorkerShutdownComplete(context)) {
if (outstanding != 0) {
log.info("Shutdown completed, but shutdownCompleteLatch still had outstanding {} with a current"
+ " value of {}. shutdownComplete: {} -- Consumer Map: {}", outstanding,
context.shutdownCompleteLatch().getCount(), context.scheduler().shutdownComplete(),
context.scheduler().shardInfoShardConsumerMap().size());
return true;
}
}
return false;
}
@Override
public Boolean call() throws Exception {
GracefulShutdownContext context;
try {
context = startWorkerShutdown.call();
} catch (Exception ex) {
log.warn("Caught exception while requesting initial worker shutdown.", ex);
throw ex;
}
return context.isShutdownAlreadyCompleted() || waitForRecordProcessors(context);
}
}
}

30
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/NoOpWorkerStateChangeListener.java
View file

@ -1,30 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
public class NoOpWorkerStateChangeListener implements WorkerStateChangeListener {
/**
* Empty constructor for NoOp Worker State Change Listener
*/
public NoOpWorkerStateChangeListener() {
}
@Override
public void onWorkerStateChange(WorkerState newState) {
}
}

657
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/Scheduler.java
View file

@ -1,657 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import com.google.common.annotations.VisibleForTesting;
import lombok.AccessLevel;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.NonNull;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.checkpoint.CheckpointConfig;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseManagementConfig;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.ShardDetector;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.leases.ShardPrioritization;
import software.amazon.kinesis.leases.ShardSyncTask;
import software.amazon.kinesis.leases.ShardSyncTaskManager;
import software.amazon.kinesis.leases.HierarchicalShardSyncer;
import software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator;
import software.amazon.kinesis.leases.exceptions.LeasingException;
import software.amazon.kinesis.lifecycle.LifecycleConfig;
import software.amazon.kinesis.lifecycle.ShardConsumer;
import software.amazon.kinesis.lifecycle.ShardConsumerArgument;
import software.amazon.kinesis.lifecycle.ShardConsumerShutdownNotification;
import software.amazon.kinesis.lifecycle.ShutdownNotification;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.lifecycle.TaskResult;
import software.amazon.kinesis.metrics.CloudWatchMetricsFactory;
import software.amazon.kinesis.metrics.MetricsCollectingTaskDecorator;
import software.amazon.kinesis.metrics.MetricsConfig;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.processor.ProcessorConfig;
import software.amazon.kinesis.processor.ShardRecordProcessorFactory;
import software.amazon.kinesis.processor.ShutdownNotificationAware;
import software.amazon.kinesis.retrieval.AggregatorUtil;
import software.amazon.kinesis.retrieval.RecordsPublisher;
import software.amazon.kinesis.retrieval.RetrievalConfig;
/**
*
*/
@Getter
@Accessors(fluent = true)
@Slf4j
public class Scheduler implements Runnable {
private SchedulerLog slog = new SchedulerLog();
private final CheckpointConfig checkpointConfig;
private final CoordinatorConfig coordinatorConfig;
private final LeaseManagementConfig leaseManagementConfig;
private final LifecycleConfig lifecycleConfig;
private final MetricsConfig metricsConfig;
private final ProcessorConfig processorConfig;
private final RetrievalConfig retrievalConfig;
private final String applicationName;
private final int maxInitializationAttempts;
private final Checkpointer checkpoint;
private final long shardConsumerDispatchPollIntervalMillis;
// Backoff time when polling to check if application has finished processing
// parent shards
private final long parentShardPollIntervalMillis;
private final ExecutorService executorService;
// private final GetRecordsRetrievalStrategy getRecordsRetrievalStrategy;
private final LeaseCoordinator leaseCoordinator;
private final ShardSyncTaskManager shardSyncTaskManager;
private final ShardPrioritization shardPrioritization;
private final boolean cleanupLeasesUponShardCompletion;
private final boolean skipShardSyncAtWorkerInitializationIfLeasesExist;
private final GracefulShutdownCoordinator gracefulShutdownCoordinator;
private final WorkerStateChangeListener workerStateChangeListener;
private final InitialPositionInStreamExtended initialPosition;
private final MetricsFactory metricsFactory;
private final long failoverTimeMillis;
private final long taskBackoffTimeMillis;
private final String streamName;
private final long listShardsBackoffTimeMillis;
private final int maxListShardsRetryAttempts;
private final LeaseRefresher leaseRefresher;
private final ShardDetector shardDetector;
private final boolean ignoreUnexpetedChildShards;
private final AggregatorUtil aggregatorUtil;
private final HierarchicalShardSyncer hierarchicalShardSyncer;
// Holds consumers for shards the worker is currently tracking. Key is shard
// info, value is ShardConsumer.
private ConcurrentMap<ShardInfo, ShardConsumer> shardInfoShardConsumerMap = new ConcurrentHashMap<>();
private volatile boolean shutdown;
private volatile long shutdownStartTimeMillis;
private volatile boolean shutdownComplete = false;
private final Object lock = new Object();
/**
* Used to ensure that only one requestedShutdown is in progress at a time.
*/
private Future<Boolean> gracefulShutdownFuture;
@VisibleForTesting
protected boolean gracefuleShutdownStarted = false;
public Scheduler(@NonNull final CheckpointConfig checkpointConfig,
@NonNull final CoordinatorConfig coordinatorConfig,
@NonNull final LeaseManagementConfig leaseManagementConfig,
@NonNull final LifecycleConfig lifecycleConfig,
@NonNull final MetricsConfig metricsConfig,
@NonNull final ProcessorConfig processorConfig,
@NonNull final RetrievalConfig retrievalConfig) {
this.checkpointConfig = checkpointConfig;
this.coordinatorConfig = coordinatorConfig;
this.leaseManagementConfig = leaseManagementConfig;
this.lifecycleConfig = lifecycleConfig;
this.metricsConfig = metricsConfig;
this.processorConfig = processorConfig;
this.retrievalConfig = retrievalConfig;
this.applicationName = this.coordinatorConfig.applicationName();
this.maxInitializationAttempts = this.coordinatorConfig.maxInitializationAttempts();
this.metricsFactory = this.metricsConfig.metricsFactory();
this.leaseCoordinator = this.leaseManagementConfig.leaseManagementFactory()
.createLeaseCoordinator(this.metricsFactory);
this.leaseRefresher = this.leaseCoordinator.leaseRefresher();
//
// TODO: Figure out what to do with lease manage <=> checkpoint relationship
//
this.checkpoint = this.checkpointConfig.checkpointFactory().createCheckpointer(this.leaseCoordinator,
this.leaseRefresher);
//
// TODO: Move this configuration to lifecycle
//
this.shardConsumerDispatchPollIntervalMillis = this.coordinatorConfig.shardConsumerDispatchPollIntervalMillis();
this.parentShardPollIntervalMillis = this.coordinatorConfig.parentShardPollIntervalMillis();
this.executorService = this.coordinatorConfig.coordinatorFactory().createExecutorService();
this.shardSyncTaskManager = this.leaseManagementConfig.leaseManagementFactory()
.createShardSyncTaskManager(this.metricsFactory);
this.shardPrioritization = this.coordinatorConfig.shardPrioritization();
this.cleanupLeasesUponShardCompletion = this.leaseManagementConfig.cleanupLeasesUponShardCompletion();
this.skipShardSyncAtWorkerInitializationIfLeasesExist =
this.coordinatorConfig.skipShardSyncAtWorkerInitializationIfLeasesExist();
if (coordinatorConfig.gracefulShutdownCoordinator() != null) {
this.gracefulShutdownCoordinator = coordinatorConfig.gracefulShutdownCoordinator();
} else {
this.gracefulShutdownCoordinator = this.coordinatorConfig.coordinatorFactory()
.createGracefulShutdownCoordinator();
}
if (coordinatorConfig.workerStateChangeListener() != null) {
this.workerStateChangeListener = coordinatorConfig.workerStateChangeListener();
} else {
this.workerStateChangeListener = this.coordinatorConfig.coordinatorFactory()
.createWorkerStateChangeListener();
}
this.initialPosition = retrievalConfig.initialPositionInStreamExtended();
this.failoverTimeMillis = this.leaseManagementConfig.failoverTimeMillis();
this.taskBackoffTimeMillis = this.lifecycleConfig.taskBackoffTimeMillis();
// this.retryGetRecordsInSeconds = this.retrievalConfig.retryGetRecordsInSeconds();
// this.maxGetRecordsThreadPool = this.retrievalConfig.maxGetRecordsThreadPool();
this.streamName = this.retrievalConfig.streamName();
this.listShardsBackoffTimeMillis = this.retrievalConfig.listShardsBackoffTimeInMillis();
this.maxListShardsRetryAttempts = this.retrievalConfig.maxListShardsRetryAttempts();
this.shardDetector = this.shardSyncTaskManager.shardDetector();
this.ignoreUnexpetedChildShards = this.leaseManagementConfig.ignoreUnexpectedChildShards();
this.aggregatorUtil = this.lifecycleConfig.aggregatorUtil();
this.hierarchicalShardSyncer = leaseManagementConfig.hierarchicalShardSyncer();
}
/**
* Start consuming data from the stream, and pass it to the application record processors.
*/
@Override
public void run() {
if (shutdown) {
return;
}
try {
initialize();
log.info("Initialization complete. Starting worker loop.");
} catch (RuntimeException e) {
log.error("Unable to initialize after {} attempts. Shutting down.", maxInitializationAttempts, e);
workerStateChangeListener.onAllInitializationAttemptsFailed(e);
shutdown();
}
while (!shouldShutdown()) {
runProcessLoop();
}
finalShutdown();
log.info("Worker loop is complete. Exiting from worker.");
}
private void initialize() {
synchronized (lock) {
workerStateChangeListener.onWorkerStateChange(WorkerStateChangeListener.WorkerState.INITIALIZING);
boolean isDone = false;
Exception lastException = null;
for (int i = 0; (!isDone) && (i < maxInitializationAttempts); i++) {
try {
log.info("Initialization attempt {}", (i + 1));
log.info("Initializing LeaseCoordinator");
leaseCoordinator.initialize();
TaskResult result = null;
if (!skipShardSyncAtWorkerInitializationIfLeasesExist || leaseRefresher.isLeaseTableEmpty()) {
log.info("Syncing Kinesis shard info");
ShardSyncTask shardSyncTask = new ShardSyncTask(shardDetector, leaseRefresher, initialPosition,
cleanupLeasesUponShardCompletion, ignoreUnexpetedChildShards, 0L, hierarchicalShardSyncer,
metricsFactory);
result = new MetricsCollectingTaskDecorator(shardSyncTask, metricsFactory).call();
} else {
log.info("Skipping shard sync per configuration setting (and lease table is not empty)");
}
if (result == null || result.getException() == null) {
if (!leaseCoordinator.isRunning()) {
log.info("Starting LeaseCoordinator");
leaseCoordinator.start();
} else {
log.info("LeaseCoordinator is already running. No need to start it.");
}
isDone = true;
} else {
lastException = result.getException();
}
} catch (LeasingException e) {
log.error("Caught exception when initializing LeaseCoordinator", e);
lastException = e;
} catch (Exception e) {
lastException = e;
}
try {
Thread.sleep(parentShardPollIntervalMillis);
} catch (InterruptedException e) {
log.debug("Sleep interrupted while initializing worker.");
}
}
if (!isDone) {
throw new RuntimeException(lastException);
}
workerStateChangeListener.onWorkerStateChange(WorkerStateChangeListener.WorkerState.STARTED);
}
}
@VisibleForTesting
void runProcessLoop() {
try {
boolean foundCompletedShard = false;
Set<ShardInfo> assignedShards = new HashSet<>();
for (ShardInfo shardInfo : getShardInfoForAssignments()) {
ShardConsumer shardConsumer = createOrGetShardConsumer(shardInfo,
processorConfig.shardRecordProcessorFactory());
if (shardConsumer.isShutdown() && shardConsumer.shutdownReason().equals(ShutdownReason.SHARD_END)) {
foundCompletedShard = true;
} else {
shardConsumer.executeLifecycle();
}
assignedShards.add(shardInfo);
}
if (foundCompletedShard) {
shardSyncTaskManager.syncShardAndLeaseInfo();
}
// clean up shard consumers for unassigned shards
cleanupShardConsumers(assignedShards);
slog.info("Sleeping ...");
Thread.sleep(shardConsumerDispatchPollIntervalMillis);
} catch (Exception e) {
log.error("Worker.run caught exception, sleeping for {} milli seconds!",
String.valueOf(shardConsumerDispatchPollIntervalMillis), e);
try {
Thread.sleep(shardConsumerDispatchPollIntervalMillis);
} catch (InterruptedException ex) {
log.info("Worker: sleep interrupted after catching exception ", ex);
}
}
slog.resetInfoLogging();
}
/**
* Returns whether worker can shutdown immediately. Note that this method is called from Worker's {{@link #run()}
* method before every loop run, so method must do minimum amount of work to not impact shard processing timings.
*
* @return Whether worker should shutdown immediately.
*/
@VisibleForTesting
boolean shouldShutdown() {
if (executorService.isShutdown()) {
log.error("Worker executor service has been shutdown, so record processors cannot be shutdown.");
return true;
}
if (shutdown) {
if (shardInfoShardConsumerMap.isEmpty()) {
log.info("All record processors have been shutdown successfully.");
return true;
}
if ((System.currentTimeMillis() - shutdownStartTimeMillis) >= failoverTimeMillis) {
log.info("Lease failover time is reached, so forcing shutdown.");
return true;
}
}
return false;
}
/**
* Requests a graceful shutdown of the worker, notifying record processors, that implement
* {@link ShutdownNotificationAware}, of the impending shutdown. This gives the record processor a final chance to
* checkpoint.
*
* This will only create a single shutdown future. Additional attempts to start a graceful shutdown will return the
* previous future.
*
* <b>It's possible that a record processor won't be notify before being shutdown. This can occur if the lease is
* lost after requesting shutdown, but before the notification is dispatched.</b>
*
* <h2>Requested Shutdown Process</h2> When a shutdown process is requested it operates slightly differently to
* allow the record processors a chance to checkpoint a final time.
* <ol>
* <li>Call to request shutdown invoked.</li>
* <li>Worker stops attempting to acquire new leases</li>
* <li>Record Processor Shutdown Begins
* <ol>
* <li>Record processor is notified of the impending shutdown, and given a final chance to checkpoint</li>
* <li>The lease for the record processor is then dropped.</li>
* <li>The record processor enters into an idle state waiting for the worker to complete final termination</li>
* <li>The worker will detect a record processor that has lost it's lease, and will terminate the record processor
* with {@link ShutdownReason#LEASE_LOST}</li>
* </ol>
* </li>
* <li>The worker will shutdown all record processors.</li>
* <li>Once all record processors have been terminated, the worker will terminate all owned resources.</li>
* <li>Once the worker shutdown is complete, the returned future is completed.</li>
* </ol>
*
* @return a future that will be set once the shutdown has completed. True indicates that the graceful shutdown
* completed successfully. A false value indicates that a non-exception case caused the shutdown process to
* terminate early.
*/
public Future<Boolean> startGracefulShutdown() {
synchronized (this) {
if (gracefulShutdownFuture == null) {
gracefulShutdownFuture = gracefulShutdownCoordinator
.startGracefulShutdown(createGracefulShutdownCallable());
}
}
return gracefulShutdownFuture;
}
/**
* Creates a callable that will execute the graceful shutdown process. This callable can be used to execute graceful
* shutdowns in your own executor, or execute the shutdown synchronously.
*
* @return a callable that run the graceful shutdown process. This may return a callable that return true if the
* graceful shutdown has already been completed.
* @throws IllegalStateException
* thrown by the callable if another callable has already started the shutdown process.
*/
public Callable<Boolean> createGracefulShutdownCallable() {
if (shutdownComplete()) {
return () -> true;
}
Callable<GracefulShutdownContext> startShutdown = createWorkerShutdownCallable();
return gracefulShutdownCoordinator.createGracefulShutdownCallable(startShutdown);
}
public boolean hasGracefulShutdownStarted() {
return gracefuleShutdownStarted;
}
@VisibleForTesting
Callable<GracefulShutdownContext> createWorkerShutdownCallable() {
return () -> {
synchronized (this) {
if (this.gracefuleShutdownStarted) {
throw new IllegalStateException("Requested shutdown has already been started");
}
this.gracefuleShutdownStarted = true;
}
//
// Stop accepting new leases. Once we do this we can be sure that
// no more leases will be acquired.
//
leaseCoordinator.stopLeaseTaker();
Collection<Lease> leases = leaseCoordinator.getAssignments();
if (leases == null || leases.isEmpty()) {
//
// If there are no leases notification is already completed, but we still need to shutdown the worker.
//
this.shutdown();
return GracefulShutdownContext.SHUTDOWN_ALREADY_COMPLETED;
}
CountDownLatch shutdownCompleteLatch = new CountDownLatch(leases.size());
CountDownLatch notificationCompleteLatch = new CountDownLatch(leases.size());
for (Lease lease : leases) {
ShutdownNotification shutdownNotification = new ShardConsumerShutdownNotification(leaseCoordinator,
lease, notificationCompleteLatch, shutdownCompleteLatch);
ShardInfo shardInfo = DynamoDBLeaseCoordinator.convertLeaseToAssignment(lease);
ShardConsumer consumer = shardInfoShardConsumerMap.get(shardInfo);
if (consumer != null) {
consumer.gracefulShutdown(shutdownNotification);
} else {
//
// There is a race condition between retrieving the current assignments, and creating the
// notification. If the a lease is lost in between these two points, we explicitly decrement the
// notification latches to clear the shutdown.
//
notificationCompleteLatch.countDown();
shutdownCompleteLatch.countDown();
}
}
return new GracefulShutdownContext(shutdownCompleteLatch, notificationCompleteLatch, this);
};
}
/**
* Signals worker to shutdown. Worker will try initiating shutdown of all record processors. Note that if executor
* services were passed to the worker by the user, worker will not attempt to shutdown those resources.
*
* <h2>Shutdown Process</h2> When called this will start shutdown of the record processor, and eventually shutdown
* the worker itself.
* <ol>
* <li>Call to start shutdown invoked</li>
* <li>Lease coordinator told to stop taking leases, and to drop existing leases.</li>
* <li>Worker discovers record processors that no longer have leases.</li>
* <li>Worker triggers shutdown with state {@link ShutdownReason#LEASE_LOST}.</li>
* <li>Once all record processors are shutdown, worker terminates owned resources.</li>
* <li>Shutdown complete.</li>
* </ol>
*/
public void shutdown() {
synchronized (lock) {
if (shutdown) {
log.warn("Shutdown requested a second time.");
return;
}
log.info("Worker shutdown requested.");
// Set shutdown flag, so Worker.run can start shutdown process.
shutdown = true;
shutdownStartTimeMillis = System.currentTimeMillis();
// Stop lease coordinator, so leases are not renewed or stolen from other workers.
// Lost leases will force Worker to begin shutdown process for all shard consumers in
// Worker.run().
leaseCoordinator.stop();
workerStateChangeListener.onWorkerStateChange(WorkerStateChangeListener.WorkerState.SHUT_DOWN);
}
}
/**
* Perform final shutdown related tasks for the worker including shutting down worker owned executor services,
* threads, etc.
*/
private void finalShutdown() {
log.info("Starting worker's final shutdown.");
if (executorService instanceof SchedulerCoordinatorFactory.SchedulerThreadPoolExecutor) {
// This should interrupt all active record processor tasks.
executorService.shutdownNow();
}
if (metricsFactory instanceof CloudWatchMetricsFactory) {
((CloudWatchMetricsFactory) metricsFactory).shutdown();
}
shutdownComplete = true;
}
private List<ShardInfo> getShardInfoForAssignments() {
List<ShardInfo> assignedStreamShards = leaseCoordinator.getCurrentAssignments();
List<ShardInfo> prioritizedShards = shardPrioritization.prioritize(assignedStreamShards);
if ((prioritizedShards != null) && (!prioritizedShards.isEmpty())) {
if (slog.isInfoEnabled()) {
StringBuilder builder = new StringBuilder();
boolean firstItem = true;
for (ShardInfo shardInfo : prioritizedShards) {
if (!firstItem) {
builder.append(", ");
}
builder.append(shardInfo.shardId());
firstItem = false;
}
slog.info("Current stream shard assignments: " + builder.toString());
}
} else {
slog.info("No activities assigned");
}
return prioritizedShards;
}
/**
* NOTE: This method is internal/private to the Worker class. It has package access solely for testing.
*
* @param shardInfo
* Kinesis shard info
* @return ShardConsumer for the shard
*/
ShardConsumer createOrGetShardConsumer(@NonNull final ShardInfo shardInfo,
@NonNull final ShardRecordProcessorFactory shardRecordProcessorFactory) {
ShardConsumer consumer = shardInfoShardConsumerMap.get(shardInfo);
// Instantiate a new consumer if we don't have one, or the one we
// had was from an earlier
// lease instance (and was shutdown). Don't need to create another
// one if the shard has been
// completely processed (shutdown reason terminate).
if ((consumer == null)
|| (consumer.isShutdown() && consumer.shutdownReason().equals(ShutdownReason.LEASE_LOST))) {
consumer = buildConsumer(shardInfo, shardRecordProcessorFactory);
shardInfoShardConsumerMap.put(shardInfo, consumer);
slog.infoForce("Created new shardConsumer for : " + shardInfo);
}
return consumer;
}
protected ShardConsumer buildConsumer(@NonNull final ShardInfo shardInfo,
@NonNull final ShardRecordProcessorFactory shardRecordProcessorFactory) {
RecordsPublisher cache = retrievalConfig.retrievalFactory().createGetRecordsCache(shardInfo, metricsFactory);
ShardRecordProcessorCheckpointer checkpointer = coordinatorConfig.coordinatorFactory().createRecordProcessorCheckpointer(shardInfo,
checkpoint);
ShardConsumerArgument argument = new ShardConsumerArgument(shardInfo,
streamName,
leaseRefresher,
executorService,
cache,
shardRecordProcessorFactory.shardRecordProcessor(),
checkpoint,
checkpointer,
parentShardPollIntervalMillis,
taskBackoffTimeMillis,
skipShardSyncAtWorkerInitializationIfLeasesExist,
listShardsBackoffTimeMillis,
maxListShardsRetryAttempts,
processorConfig.callProcessRecordsEvenForEmptyRecordList(),
shardConsumerDispatchPollIntervalMillis,
initialPosition,
cleanupLeasesUponShardCompletion,
ignoreUnexpetedChildShards,
shardDetector,
aggregatorUtil,
hierarchicalShardSyncer,
metricsFactory);
return new ShardConsumer(cache, executorService, shardInfo, lifecycleConfig.logWarningForTaskAfterMillis(),
argument, lifecycleConfig.taskExecutionListener());
}
/**
* NOTE: This method is internal/private to the Worker class. It has package access solely for testing.
*
* This method relies on ShardInfo.equals() method returning true for ShardInfo objects which may have been
* instantiated with parentShardIds in a different order (and rest of the fields being the equal). For example
* shardInfo1.equals(shardInfo2) should return true with shardInfo1 and shardInfo2 defined as follows. ShardInfo
* shardInfo1 = new ShardInfo(shardId1, concurrencyToken1, Arrays.asList("parent1", "parent2")); ShardInfo
* shardInfo2 = new ShardInfo(shardId1, concurrencyToken1, Arrays.asList("parent2", "parent1"));
*/
void cleanupShardConsumers(Set<ShardInfo> assignedShards) {
for (ShardInfo shard : shardInfoShardConsumerMap.keySet()) {
if (!assignedShards.contains(shard)) {
// Shutdown the consumer since we are no longer responsible for
// the shard.
ShardConsumer consumer = shardInfoShardConsumerMap.get(shard);
if (consumer.leaseLost()) {
shardInfoShardConsumerMap.remove(shard);
log.debug("Removed consumer for {} as lease has been lost", shard.shardId());
} else {
consumer.executeLifecycle();
}
}
}
}
/**
* Logger for suppressing too much INFO logging. To avoid too much logging information Worker will output logging at
* INFO level for a single pass through the main loop every minute. At DEBUG level it will output all INFO logs on
* every pass.
*/
@NoArgsConstructor(access = AccessLevel.PRIVATE)
private static class SchedulerLog {
private long reportIntervalMillis = TimeUnit.MINUTES.toMillis(1);
private long nextReportTime = System.currentTimeMillis() + reportIntervalMillis;
private boolean infoReporting;
void info(Object message) {
if (this.isInfoEnabled()) {
log.info("{}", message);
}
}
void infoForce(Object message) {
log.info("{}", message);
}
private boolean isInfoEnabled() {
return infoReporting;
}
private void resetInfoLogging() {
if (infoReporting) {
// We just logged at INFO level for a pass through worker loop
if (log.isInfoEnabled()) {
infoReporting = false;
nextReportTime = System.currentTimeMillis() + reportIntervalMillis;
} // else is DEBUG or TRACE so leave reporting true
} else if (nextReportTime <= System.currentTimeMillis()) {
infoReporting = true;
}
}
}
@Deprecated
public Future<Void> requestShutdown() {
return null;
}
}

64
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/SchedulerCoordinatorFactory.java
View file

@ -1,64 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import lombok.Data;
import lombok.NonNull;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.processor.Checkpointer;
/**
*
*/
@Data
@KinesisClientInternalApi
public class SchedulerCoordinatorFactory implements CoordinatorFactory {
/**
* {@inheritDoc}
*/
@Override
public ExecutorService createExecutorService() {
return new SchedulerThreadPoolExecutor(
new ThreadFactoryBuilder().setNameFormat("ShardRecordProcessor-%04d").build());
}
static class SchedulerThreadPoolExecutor extends ThreadPoolExecutor {
private static final long DEFAULT_KEEP_ALIVE = 60L;
SchedulerThreadPoolExecutor(ThreadFactory threadFactory) {
super(0, Integer.MAX_VALUE, DEFAULT_KEEP_ALIVE, TimeUnit.SECONDS, new SynchronousQueue<>(),
threadFactory);
}
}
/**
* {@inheritDoc}
*/
@Override
public ShardRecordProcessorCheckpointer createRecordProcessorCheckpointer(@NonNull final ShardInfo shardInfo,
@NonNull final Checkpointer checkpoint) {
return new ShardRecordProcessorCheckpointer(shardInfo, checkpoint);
}
}

33
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/WorkerStateChangeListener.java
View file

@ -1,33 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
/**
* A listener for callbacks on changes worker state
*/
@FunctionalInterface
public interface WorkerStateChangeListener {
enum WorkerState {
CREATED,
INITIALIZING,
STARTED,
SHUT_DOWN
}
void onWorkerStateChange(WorkerState newState);
default void onAllInitializationAttemptsFailed(Throwable e) {
}
}

39
amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/ShutdownException.java
View file

@ -1,39 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.exceptions;
/**
* The ShardRecordProcessor instance has been shutdown (e.g. and attempts a checkpoint).
*/
public class ShutdownException extends KinesisClientLibNonRetryableException {
private static final long serialVersionUID = 1L;
/**
* @param message provides more details about the cause and potential ways to debug/address.
*/
public ShutdownException(String message) {
super(message);
}
/**
* @param message provides more details about the cause and potential ways to debug/address.
* @param e Cause of the exception
*/
public ShutdownException(String message, Exception e) {
super(message, e);
}
}

44
amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/internal/KinesisClientLibIOException.java
View file

@ -1,44 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.exceptions.internal;
import software.amazon.kinesis.exceptions.KinesisClientLibRetryableException;
/**
* Thrown when we encounter issues when reading/writing information (e.g. shard information from Kinesis may not be
* current/complete).
*/
public class KinesisClientLibIOException extends KinesisClientLibRetryableException {
private static final long serialVersionUID = 1L;
/**
* Constructor.
*
* @param message Error message.
*/
public KinesisClientLibIOException(String message) {
super(message);
}
/**
* Constructor.
*
* @param message Error message.
* @param e Cause.
*/
public KinesisClientLibIOException(String message, Exception e) {
super(message, e);
}
}

756
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/HierarchicalShardSyncer.java
View file

@ -1,756 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.io.Serializable;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Comparator;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.function.Function;
import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.kinesis.model.Shard;
import software.amazon.awssdk.utils.CollectionUtils;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.common.InitialPositionInStream;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.exceptions.internal.KinesisClientLibIOException;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.metrics.MetricsLevel;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsUtil;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Helper class to sync leases with shards of the Kinesis stream.
* It will create new leases/activities when it discovers new Kinesis shards (bootstrap/resharding).
* It deletes leases for shards that have been trimmed from Kinesis, or if we've completed processing it
* and begun processing it's child shards.
*/
@Slf4j
@KinesisClientInternalApi
public class HierarchicalShardSyncer {
/**
* Check and create leases for any new shards (e.g. following a reshard operation). Sync leases with Kinesis shards
* (e.g. at startup, or when we reach end of a shard).
*
* @param shardDetector
* @param leaseRefresher
* @param initialPosition
* @param cleanupLeasesOfCompletedShards
* @param ignoreUnexpectedChildShards
* @param scope
* @throws DependencyException
* @throws InvalidStateException
* @throws ProvisionedThroughputException
* @throws KinesisClientLibIOException
*/
// CHECKSTYLE:OFF CyclomaticComplexity
public synchronized void checkAndCreateLeaseForNewShards(@NonNull final ShardDetector shardDetector,
final LeaseRefresher leaseRefresher, final InitialPositionInStreamExtended initialPosition,
final boolean cleanupLeasesOfCompletedShards, final boolean ignoreUnexpectedChildShards,
final MetricsScope scope) throws DependencyException, InvalidStateException,
ProvisionedThroughputException, KinesisClientLibIOException {
final List<Shard> shards = getShardList(shardDetector);
log.debug("Num shards: {}", shards.size());
final Map<String, Shard> shardIdToShardMap = constructShardIdToShardMap(shards);
final Map<String, Set<String>> shardIdToChildShardIdsMap = constructShardIdToChildShardIdsMap(
shardIdToShardMap);
final Set<String> inconsistentShardIds = findInconsistentShardIds(shardIdToChildShardIdsMap, shardIdToShardMap);
if (!ignoreUnexpectedChildShards) {
assertAllParentShardsAreClosed(inconsistentShardIds);
}
final List<Lease> currentLeases = leaseRefresher.listLeases();
final List<Lease> newLeasesToCreate = determineNewLeasesToCreate(shards, currentLeases, initialPosition,
inconsistentShardIds);
log.debug("Num new leases to create: {}", newLeasesToCreate.size());
for (Lease lease : newLeasesToCreate) {
long startTime = System.currentTimeMillis();
boolean success = false;
try {
leaseRefresher.createLeaseIfNotExists(lease);
success = true;
} finally {
MetricsUtil.addSuccessAndLatency(scope, "CreateLease", success, startTime, MetricsLevel.DETAILED);
}
}
final List<Lease> trackedLeases = new ArrayList<>(currentLeases);
trackedLeases.addAll(newLeasesToCreate);
cleanupGarbageLeases(shardDetector, shards, trackedLeases, leaseRefresher);
if (cleanupLeasesOfCompletedShards) {
cleanupLeasesOfFinishedShards(currentLeases, shardIdToShardMap, shardIdToChildShardIdsMap, trackedLeases,
leaseRefresher);
}
}
// CHECKSTYLE:ON CyclomaticComplexity
/** Helper method to detect a race condition between fetching the shards via paginated DescribeStream calls
* and a reshard operation.
* @param inconsistentShardIds
* @throws KinesisClientLibIOException
*/
private static void assertAllParentShardsAreClosed(final Set<String> inconsistentShardIds)
throws KinesisClientLibIOException {
if (!CollectionUtils.isNullOrEmpty(inconsistentShardIds)) {
final String ids = StringUtils.join(inconsistentShardIds, ' ');
throw new KinesisClientLibIOException(String.format(
"%d open child shards (%s) are inconsistent. This can happen due to a race condition between describeStream and a reshard operation.",
inconsistentShardIds.size(), ids));
}
}
/**
* Helper method to construct the list of inconsistent shards, which are open shards with non-closed ancestor
* parent(s).
* @param shardIdToChildShardIdsMap
* @param shardIdToShardMap
* @return Set of inconsistent open shard ids for shards having open parents.
*/
private static Set<String> findInconsistentShardIds(final Map<String, Set<String>> shardIdToChildShardIdsMap,
final Map<String, Shard> shardIdToShardMap) {
return shardIdToChildShardIdsMap.entrySet().stream()
.filter(entry -> entry.getKey() == null
|| shardIdToShardMap.get(entry.getKey()).sequenceNumberRange().endingSequenceNumber() == null)
.flatMap(entry -> shardIdToChildShardIdsMap.get(entry.getKey()).stream()).collect(Collectors.toSet());
}
/**
* Note: this has package level access for testing purposes.
* Useful for asserting that we don't have an incomplete shard list following a reshard operation.
* We verify that if the shard is present in the shard list, it is closed and its hash key range
* is covered by its child shards.
* @param shardIdsOfClosedShards Id of the shard which is expected to be closed
* @return ShardIds of child shards (children of the expectedClosedShard)
* @throws KinesisClientLibIOException
*/
synchronized void assertClosedShardsAreCoveredOrAbsent(final Map<String, Shard> shardIdToShardMap,
final Map<String, Set<String>> shardIdToChildShardIdsMap, final Set<String> shardIdsOfClosedShards)
throws KinesisClientLibIOException {
final String exceptionMessageSuffix = "This can happen if we constructed the list of shards "
+ " while a reshard operation was in progress.";
for (String shardId : shardIdsOfClosedShards) {
final Shard shard = shardIdToShardMap.get(shardId);
if (shard == null) {
log.info("Shard {} is not present in Kinesis anymore.", shardId);
continue;
}
final String endingSequenceNumber = shard.sequenceNumberRange().endingSequenceNumber();
if (endingSequenceNumber == null) {
throw new KinesisClientLibIOException("Shard " + shardIdsOfClosedShards
+ " is not closed. " + exceptionMessageSuffix);
}
final Set<String> childShardIds = shardIdToChildShardIdsMap.get(shardId);
if (childShardIds == null) {
throw new KinesisClientLibIOException("Incomplete shard list: Closed shard " + shardId
+ " has no children." + exceptionMessageSuffix);
}
assertHashRangeOfClosedShardIsCovered(shard, shardIdToShardMap, childShardIds);
}
}
private synchronized void assertHashRangeOfClosedShardIsCovered(final Shard closedShard,
final Map<String, Shard> shardIdToShardMap, final Set<String> childShardIds)
throws KinesisClientLibIOException {
BigInteger minStartingHashKeyOfChildren = null;
BigInteger maxEndingHashKeyOfChildren = null;
final BigInteger startingHashKeyOfClosedShard = new BigInteger(closedShard.hashKeyRange().startingHashKey());
final BigInteger endingHashKeyOfClosedShard = new BigInteger(closedShard.hashKeyRange().endingHashKey());
for (String childShardId : childShardIds) {
final Shard childShard = shardIdToShardMap.get(childShardId);
final BigInteger startingHashKey = new BigInteger(childShard.hashKeyRange().startingHashKey());
if (minStartingHashKeyOfChildren == null || startingHashKey.compareTo(minStartingHashKeyOfChildren) < 0) {
minStartingHashKeyOfChildren = startingHashKey;
}
final BigInteger endingHashKey = new BigInteger(childShard.hashKeyRange().endingHashKey());
if (maxEndingHashKeyOfChildren == null || endingHashKey.compareTo(maxEndingHashKeyOfChildren) > 0) {
maxEndingHashKeyOfChildren = endingHashKey;
}
}
if (minStartingHashKeyOfChildren == null || maxEndingHashKeyOfChildren == null
|| minStartingHashKeyOfChildren.compareTo(startingHashKeyOfClosedShard) > 0
|| maxEndingHashKeyOfChildren.compareTo(endingHashKeyOfClosedShard) < 0) {
throw new KinesisClientLibIOException(String.format(
"Incomplete shard list: hash key range of shard %s is not covered by its child shards.",
closedShard.shardId()));
}
}
/**
* Helper method to construct shardId->setOfChildShardIds map.
* Note: This has package access for testing purposes only.
* @param shardIdToShardMap
* @return
*/
static Map<String, Set<String>> constructShardIdToChildShardIdsMap(final Map<String, Shard> shardIdToShardMap) {
final Map<String, Set<String>> shardIdToChildShardIdsMap = new HashMap<>();
for (final Map.Entry<String, Shard> entry : shardIdToShardMap.entrySet()) {
final String shardId = entry.getKey();
final Shard shard = entry.getValue();
final String parentShardId = shard.parentShardId();
if (parentShardId != null && shardIdToShardMap.containsKey(parentShardId)) {
final Set<String> childShardIds = shardIdToChildShardIdsMap.computeIfAbsent(parentShardId,
key -> new HashSet<>());
childShardIds.add(shardId);
}
final String adjacentParentShardId = shard.adjacentParentShardId();
if (adjacentParentShardId != null && shardIdToShardMap.containsKey(adjacentParentShardId)) {
final Set<String> childShardIds = shardIdToChildShardIdsMap.computeIfAbsent(adjacentParentShardId,
key -> new HashSet<>());
childShardIds.add(shardId);
}
}
return shardIdToChildShardIdsMap;
}
private static List<Shard> getShardList(@NonNull final ShardDetector shardDetector) throws KinesisClientLibIOException {
final List<Shard> shards = shardDetector.listShards();
if (shards == null) {
throw new KinesisClientLibIOException(
"Stream is not in ACTIVE OR UPDATING state - will retry getting the shard list.");
}
return shards;
}
/**
* Determine new leases to create and their initial checkpoint.
* Note: Package level access only for testing purposes.
*
* For each open (no ending sequence number) shard without open parents that doesn't already have a lease,
* determine if it is a descendent of any shard which is or will be processed (e.g. for which a lease exists):
* If so, set checkpoint of the shard to TrimHorizon and also create leases for ancestors if needed.
* If not, set checkpoint of the shard to the initial position specified by the client.
* To check if we need to create leases for ancestors, we use the following rules:
* * If we began (or will begin) processing data for a shard, then we must reach end of that shard before
* we begin processing data from any of its descendants.
* * A shard does not start processing data until data from all its parents has been processed.
* Note, if the initial position is LATEST and a shard has two parents and only one is a descendant - we'll create
* leases corresponding to both the parents - the parent shard which is not a descendant will have
* its checkpoint set to Latest.
*
* We assume that if there is an existing lease for a shard, then either:
* * we have previously created a lease for its parent (if it was needed), or
* * the parent shard has expired.
*
* For example:
* Shard structure (each level depicts a stream segment):
* 0 1 2 3 4 5 - shards till epoch 102
* \ / \ / | |
* 6 7 4 5 - shards from epoch 103 - 205
* \ / | / \
* 8 4 9 10 - shards from epoch 206 (open - no ending sequenceNumber)
* Current leases: (3, 4, 5)
* New leases to create: (2, 6, 7, 8, 9, 10)
*
* The leases returned are sorted by the starting sequence number - following the same order
* when persisting the leases in DynamoDB will ensure that we recover gracefully if we fail
* before creating all the leases.
*
* If a shard has no existing lease, is open, and is a descendant of a parent which is still open, we ignore it
* here; this happens when the list of shards is inconsistent, which could be due to pagination delay for very
* high shard count streams (i.e., dynamodb streams for tables with thousands of partitions). This can only
* currently happen here if ignoreUnexpectedChildShards was true in syncShardleases.
*
*
* @param shards List of all shards in Kinesis (we'll create new leases based on this set)
* @param currentLeases List of current leases
* @param initialPosition One of LATEST, TRIM_HORIZON, or AT_TIMESTAMP. We'll start fetching records from that
* location in the shard (when an application starts up for the first time - and there are no checkpoints).
* @param inconsistentShardIds Set of child shard ids having open parents.
* @return List of new leases to create sorted by starting sequenceNumber of the corresponding shard
*/
static List<Lease> determineNewLeasesToCreate(final List<Shard> shards, final List<Lease> currentLeases,
final InitialPositionInStreamExtended initialPosition, final Set<String> inconsistentShardIds) {
final Map<String, Lease> shardIdToNewLeaseMap = new HashMap<>();
final Map<String, Shard> shardIdToShardMapOfAllKinesisShards = constructShardIdToShardMap(shards);
final Set<String> shardIdsOfCurrentLeases = currentLeases.stream()
.peek(lease -> log.debug("Existing lease: {}", lease)).map(Lease::leaseKey).collect(Collectors.toSet());
final List<Shard> openShards = getOpenShards(shards);
final Map<String, Boolean> memoizationContext = new HashMap<>();
// Iterate over the open shards and find those that don't have any lease entries.
for (Shard shard : openShards) {
final String shardId = shard.shardId();
log.debug("Evaluating leases for open shard {} and its ancestors.", shardId);
if (shardIdsOfCurrentLeases.contains(shardId)) {
log.debug("Lease for shardId {} already exists. Not creating a lease", shardId);
} else if (inconsistentShardIds.contains(shardId)) {
log.info("shardId {} is an inconsistent child. Not creating a lease", shardId);
} else {
log.debug("Need to create a lease for shardId {}", shardId);
final Lease newLease = newKCLLease(shard);
final boolean isDescendant = checkIfDescendantAndAddNewLeasesForAncestors(shardId, initialPosition,
shardIdsOfCurrentLeases, shardIdToShardMapOfAllKinesisShards, shardIdToNewLeaseMap,
memoizationContext);
/**
* If the shard is a descendant and the specified initial position is AT_TIMESTAMP, then the
* checkpoint should be set to AT_TIMESTAMP, else to TRIM_HORIZON. For AT_TIMESTAMP, we will add a
* lease just like we do for TRIM_HORIZON. However we will only return back records with server-side
* timestamp at or after the specified initial position timestamp.
*
* Shard structure (each level depicts a stream segment):
* 0 1 2 3 4 5 - shards till epoch 102
* \ / \ / | |
* 6 7 4 5 - shards from epoch 103 - 205
* \ / | /\
* 8 4 9 10 - shards from epoch 206 (open - no ending sequenceNumber)
*
* Current leases: empty set
*
* For the above example, suppose the initial position in stream is set to AT_TIMESTAMP with
* timestamp value 206. We will then create new leases for all the shards (with checkpoint set to
* AT_TIMESTAMP), including the ancestor shards with epoch less than 206. However as we begin
* processing the ancestor shards, their checkpoints would be updated to SHARD_END and their leases
* would then be deleted since they won't have records with server-side timestamp at/after 206. And
* after that we will begin processing the descendant shards with epoch at/after 206 and we will
* return the records that meet the timestamp requirement for these shards.
*/
if (isDescendant
&& !initialPosition.getInitialPositionInStream().equals(InitialPositionInStream.AT_TIMESTAMP)) {
newLease.checkpoint(ExtendedSequenceNumber.TRIM_HORIZON);
} else {
newLease.checkpoint(convertToCheckpoint(initialPosition));
}
log.debug("Set checkpoint of {} to {}", newLease.leaseKey(), newLease.checkpoint());
shardIdToNewLeaseMap.put(shardId, newLease);
}
}
final List<Lease> newLeasesToCreate = new ArrayList<>(shardIdToNewLeaseMap.values());
final Comparator<Lease> startingSequenceNumberComparator = new StartingSequenceNumberAndShardIdBasedComparator(
shardIdToShardMapOfAllKinesisShards);
newLeasesToCreate.sort(startingSequenceNumberComparator);
return newLeasesToCreate;
}
/**
* Determine new leases to create and their initial checkpoint.
* Note: Package level access only for testing purposes.
*/
static List<Lease> determineNewLeasesToCreate(final List<Shard> shards, final List<Lease> currentLeases,
final InitialPositionInStreamExtended initialPosition) {
final Set<String> inconsistentShardIds = new HashSet<>();
return determineNewLeasesToCreate(shards, currentLeases, initialPosition, inconsistentShardIds);
}
/**
* Note: Package level access for testing purposes only.
* Check if this shard is a descendant of a shard that is (or will be) processed.
* Create leases for the ancestors of this shard as required.
* See javadoc of determineNewLeasesToCreate() for rules and example.
*
* @param shardId The shardId to check.
* @param initialPosition One of LATEST, TRIM_HORIZON, or AT_TIMESTAMP. We'll start fetching records from that
* location in the shard (when an application starts up for the first time - and there are no checkpoints).
* @param shardIdsOfCurrentLeases The shardIds for the current leases.
* @param shardIdToShardMapOfAllKinesisShards ShardId->Shard map containing all shards obtained via DescribeStream.
* @param shardIdToLeaseMapOfNewShards Add lease POJOs corresponding to ancestors to this map.
* @param memoizationContext Memoization of shards that have been evaluated as part of the evaluation
* @return true if the shard is a descendant of any current shard (lease already exists)
*/
// CHECKSTYLE:OFF CyclomaticComplexity
static boolean checkIfDescendantAndAddNewLeasesForAncestors(final String shardId,
final InitialPositionInStreamExtended initialPosition, final Set<String> shardIdsOfCurrentLeases,
final Map<String, Shard> shardIdToShardMapOfAllKinesisShards,
final Map<String, Lease> shardIdToLeaseMapOfNewShards, final Map<String, Boolean> memoizationContext) {
final Boolean previousValue = memoizationContext.get(shardId);
if (previousValue != null) {
return previousValue;
}
boolean isDescendant = false;
final Set<String> descendantParentShardIds = new HashSet<>();
if (shardId != null && shardIdToShardMapOfAllKinesisShards.containsKey(shardId)) {
if (shardIdsOfCurrentLeases.contains(shardId)) {
// This shard is a descendant of a current shard.
isDescendant = true;
// We don't need to add leases of its ancestors,
// because we'd have done it when creating a lease for this shard.
} else {
final Shard shard = shardIdToShardMapOfAllKinesisShards.get(shardId);
final Set<String> parentShardIds = getParentShardIds(shard, shardIdToShardMapOfAllKinesisShards);
for (String parentShardId : parentShardIds) {
// Check if the parent is a descendant, and include its ancestors.
if (checkIfDescendantAndAddNewLeasesForAncestors(parentShardId, initialPosition,
shardIdsOfCurrentLeases, shardIdToShardMapOfAllKinesisShards, shardIdToLeaseMapOfNewShards,
memoizationContext)) {
isDescendant = true;
descendantParentShardIds.add(parentShardId);
log.debug("Parent shard {} is a descendant.", parentShardId);
} else {
log.debug("Parent shard {} is NOT a descendant.", parentShardId);
}
}
// If this is a descendant, create leases for its parent shards (if they don't exist)
if (isDescendant) {
for (String parentShardId : parentShardIds) {
if (!shardIdsOfCurrentLeases.contains(parentShardId)) {
log.debug("Need to create a lease for shardId {}", parentShardId);
Lease lease = shardIdToLeaseMapOfNewShards.get(parentShardId);
if (lease == null) {
lease = newKCLLease(shardIdToShardMapOfAllKinesisShards.get(parentShardId));
shardIdToLeaseMapOfNewShards.put(parentShardId, lease);
}
if (descendantParentShardIds.contains(parentShardId)
&& !initialPosition.getInitialPositionInStream()
.equals(InitialPositionInStream.AT_TIMESTAMP)) {
lease.checkpoint(ExtendedSequenceNumber.TRIM_HORIZON);
} else {
lease.checkpoint(convertToCheckpoint(initialPosition));
}
}
}
} else {
// This shard should be included, if the customer wants to process all records in the stream or
// if the initial position is AT_TIMESTAMP. For AT_TIMESTAMP, we will add a lease just like we do
// for TRIM_HORIZON. However we will only return back records with server-side timestamp at or
// after the specified initial position timestamp.
if (initialPosition.getInitialPositionInStream().equals(InitialPositionInStream.TRIM_HORIZON)
|| initialPosition.getInitialPositionInStream()
.equals(InitialPositionInStream.AT_TIMESTAMP)) {
isDescendant = true;
}
}
}
}
memoizationContext.put(shardId, isDescendant);
return isDescendant;
}
// CHECKSTYLE:ON CyclomaticComplexity
/**
* Helper method to get parent shardIds of the current shard - includes the parent shardIds if:
* a/ they are not null
* b/ if they exist in the current shard map (i.e. haven't expired)
*
* @param shard Will return parents of this shard
* @param shardIdToShardMapOfAllKinesisShards ShardId->Shard map containing all shards obtained via DescribeStream.
* @return Set of parentShardIds
*/
static Set<String> getParentShardIds(final Shard shard,
final Map<String, Shard> shardIdToShardMapOfAllKinesisShards) {
final Set<String> parentShardIds = new HashSet<>(2);
final String parentShardId = shard.parentShardId();
if (parentShardId != null && shardIdToShardMapOfAllKinesisShards.containsKey(parentShardId)) {
parentShardIds.add(parentShardId);
}
final String adjacentParentShardId = shard.adjacentParentShardId();
if (adjacentParentShardId != null && shardIdToShardMapOfAllKinesisShards.containsKey(adjacentParentShardId)) {
parentShardIds.add(adjacentParentShardId);
}
return parentShardIds;
}
/**
* Delete leases corresponding to shards that no longer exist in the stream. Current scheme: Delete a lease if:
* <ul>
* <li>The corresponding shard is not present in the list of Kinesis shards</li>
* <li>The parentShardIds listed in the lease are also not present in the list of Kinesis shards.</li>
* </ul>
*
* @param shards
* List of all Kinesis shards (assumed to be a consistent snapshot - when stream is in Active state).
* @param trackedLeases
* List of
* @param leaseRefresher
* @throws KinesisClientLibIOException
* Thrown if we couldn't get a fresh shard list from Kinesis.
* @throws ProvisionedThroughputException
* @throws InvalidStateException
* @throws DependencyException
*/
private static void cleanupGarbageLeases(@NonNull final ShardDetector shardDetector, final List<Shard> shards,
final List<Lease> trackedLeases, final LeaseRefresher leaseRefresher) throws KinesisClientLibIOException,
DependencyException, InvalidStateException, ProvisionedThroughputException {
final Set<String> kinesisShards = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
// Check if there are leases for non-existent shards
final List<Lease> garbageLeases = trackedLeases.stream()
.filter(lease -> isCandidateForCleanup(lease, kinesisShards)).collect(Collectors.toList());
if (!CollectionUtils.isNullOrEmpty(garbageLeases)) {
log.info("Found {} candidate leases for cleanup. Refreshing list of"
+ " Kinesis shards to pick up recent/latest shards", garbageLeases.size());
final Set<String> currentKinesisShardIds = getShardList(shardDetector).stream().map(Shard::shardId)
.collect(Collectors.toSet());
for (Lease lease : garbageLeases) {
if (isCandidateForCleanup(lease, currentKinesisShardIds)) {
log.info("Deleting lease for shard {} as it is not present in Kinesis stream.", lease.leaseKey());
leaseRefresher.deleteLease(lease);
}
}
}
}
/**
* Note: This method has package level access, solely for testing purposes.
*
* @param lease Candidate shard we are considering for deletion.
* @param currentKinesisShardIds
* @return true if neither the shard (corresponding to the lease), nor its parents are present in
* currentKinesisShardIds
* @throws KinesisClientLibIOException Thrown if currentKinesisShardIds contains a parent shard but not the child
* shard (we are evaluating for deletion).
*/
static boolean isCandidateForCleanup(final Lease lease, final Set<String> currentKinesisShardIds)
throws KinesisClientLibIOException {
boolean isCandidateForCleanup = true;
if (currentKinesisShardIds.contains(lease.leaseKey())) {
isCandidateForCleanup = false;
} else {
log.info("Found lease for non-existent shard: {}. Checking its parent shards", lease.leaseKey());
final Set<String> parentShardIds = lease.parentShardIds();
for (String parentShardId : parentShardIds) {
// Throw an exception if the parent shard exists (but the child does not).
// This may be a (rare) race condition between fetching the shard list and Kinesis expiring shards.
if (currentKinesisShardIds.contains(parentShardId)) {
final String message = String.format("Parent shard %s exists but not the child shard %s",
parentShardId, lease.leaseKey());
log.info(message);
throw new KinesisClientLibIOException(message);
}
}
}
return isCandidateForCleanup;
}
/**
* Private helper method.
* Clean up leases for shards that meet the following criteria:
* a/ the shard has been fully processed (checkpoint is set to SHARD_END)
* b/ we've begun processing all the child shards: we have leases for all child shards and their checkpoint is not
* TRIM_HORIZON.
*
* @param currentLeases List of leases we evaluate for clean up
* @param shardIdToShardMap Map of shardId->Shard (assumed to include all Kinesis shards)
* @param shardIdToChildShardIdsMap Map of shardId->childShardIds (assumed to include all Kinesis shards)
* @param trackedLeases List of all leases we are tracking.
* @param leaseRefresher Lease refresher (will be used to delete leases)
* @throws DependencyException
* @throws InvalidStateException
* @throws ProvisionedThroughputException
* @throws KinesisClientLibIOException
*/
private synchronized void cleanupLeasesOfFinishedShards(final Collection<Lease> currentLeases,
final Map<String, Shard> shardIdToShardMap, final Map<String, Set<String>> shardIdToChildShardIdsMap,
final List<Lease> trackedLeases, final LeaseRefresher leaseRefresher) throws DependencyException,
InvalidStateException, ProvisionedThroughputException, KinesisClientLibIOException {
final List<Lease> leasesOfClosedShards = currentLeases.stream()
.filter(lease -> lease.checkpoint().equals(ExtendedSequenceNumber.SHARD_END))
.collect(Collectors.toList());
final Set<String> shardIdsOfClosedShards = leasesOfClosedShards.stream().map(Lease::leaseKey)
.collect(Collectors.toSet());
if (!CollectionUtils.isNullOrEmpty(leasesOfClosedShards)) {
assertClosedShardsAreCoveredOrAbsent(shardIdToShardMap, shardIdToChildShardIdsMap, shardIdsOfClosedShards);
Comparator<? super Lease> startingSequenceNumberComparator = new StartingSequenceNumberAndShardIdBasedComparator(
shardIdToShardMap);
leasesOfClosedShards.sort(startingSequenceNumberComparator);
final Map<String, Lease> trackedLeaseMap = trackedLeases.stream()
.collect(Collectors.toMap(Lease::leaseKey, Function.identity()));
for (Lease leaseOfClosedShard : leasesOfClosedShards) {
final String closedShardId = leaseOfClosedShard.leaseKey();
final Set<String> childShardIds = shardIdToChildShardIdsMap.get(closedShardId);
if (closedShardId != null && !CollectionUtils.isNullOrEmpty(childShardIds)) {
cleanupLeaseForClosedShard(closedShardId, childShardIds, trackedLeaseMap, leaseRefresher);
}
}
}
}
/**
* Delete lease for the closed shard. Rules for deletion are:
* a/ the checkpoint for the closed shard is SHARD_END,
* b/ there are leases for all the childShardIds and their checkpoint is NOT TRIM_HORIZON
* Note: This method has package level access solely for testing purposes.
*
* @param closedShardId Identifies the closed shard
* @param childShardIds ShardIds of children of the closed shard
* @param trackedLeases shardId->Lease map with all leases we are tracking (should not be null)
* @param leaseRefresher
* @throws ProvisionedThroughputException
* @throws InvalidStateException
* @throws DependencyException
*/
synchronized void cleanupLeaseForClosedShard(final String closedShardId, final Set<String> childShardIds,
final Map<String, Lease> trackedLeases, final LeaseRefresher leaseRefresher)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
final Lease leaseForClosedShard = trackedLeases.get(closedShardId);
final List<Lease> childShardLeases = childShardIds.stream().map(trackedLeases::get).filter(Objects::nonNull)
.collect(Collectors.toList());
if (leaseForClosedShard != null && leaseForClosedShard.checkpoint().equals(ExtendedSequenceNumber.SHARD_END)
&& childShardLeases.size() == childShardIds.size()) {
boolean okayToDelete = true;
for (Lease lease : childShardLeases) {
if (lease.checkpoint().equals(ExtendedSequenceNumber.TRIM_HORIZON)) {
okayToDelete = false;
break;
}
}
if (okayToDelete) {
log.info("Deleting lease for shard {} as it has been completely processed and processing of child "
+ "shards has begun.", leaseForClosedShard.leaseKey());
leaseRefresher.deleteLease(leaseForClosedShard);
}
}
}
/**
* Helper method to create a new Lease POJO for a shard.
* Note: Package level access only for testing purposes
*
* @param shard
* @return
*/
private static Lease newKCLLease(final Shard shard) {
Lease newLease = new Lease();
newLease.leaseKey(shard.shardId());
List<String> parentShardIds = new ArrayList<>(2);
if (shard.parentShardId() != null) {
parentShardIds.add(shard.parentShardId());
}
if (shard.adjacentParentShardId() != null) {
parentShardIds.add(shard.adjacentParentShardId());
}
newLease.parentShardIds(parentShardIds);
newLease.ownerSwitchesSinceCheckpoint(0L);
return newLease;
}
/**
* Helper method to construct a shardId->Shard map for the specified list of shards.
*
* @param shards List of shards
* @return ShardId->Shard map
*/
static Map<String, Shard> constructShardIdToShardMap(final List<Shard> shards) {
return shards.stream().collect(Collectors.toMap(Shard::shardId, Function.identity()));
}
/**
* Helper method to return all the open shards for a stream.
* Note: Package level access only for testing purposes.
*
* @param allShards All shards returved via DescribeStream. We assume this to represent a consistent shard list.
* @return List of open shards (shards at the tip of the stream) - may include shards that are not yet active.
*/
static List<Shard> getOpenShards(final List<Shard> allShards) {
return allShards.stream().filter(shard -> shard.sequenceNumberRange().endingSequenceNumber() == null)
.peek(shard -> log.debug("Found open shard: {}", shard.shardId())).collect(Collectors.toList());
}
private static ExtendedSequenceNumber convertToCheckpoint(final InitialPositionInStreamExtended position) {
ExtendedSequenceNumber checkpoint = null;
if (position.getInitialPositionInStream().equals(InitialPositionInStream.TRIM_HORIZON)) {
checkpoint = ExtendedSequenceNumber.TRIM_HORIZON;
} else if (position.getInitialPositionInStream().equals(InitialPositionInStream.LATEST)) {
checkpoint = ExtendedSequenceNumber.LATEST;
} else if (position.getInitialPositionInStream().equals(InitialPositionInStream.AT_TIMESTAMP)) {
checkpoint = ExtendedSequenceNumber.AT_TIMESTAMP;
}
return checkpoint;
}
/** Helper class to compare leases based on starting sequence number of the corresponding shards.
*
*/
@RequiredArgsConstructor
private static class StartingSequenceNumberAndShardIdBasedComparator implements Comparator<Lease>, Serializable {
private static final long serialVersionUID = 1L;
private final Map<String, Shard> shardIdToShardMap;
/**
* Compares two leases based on the starting sequence number of corresponding shards.
* If shards are not found in the shardId->shard map supplied, we do a string comparison on the shardIds.
* We assume that lease1 and lease2 are:
* a/ not null,
* b/ shards (if found) have non-null starting sequence numbers
*
* {@inheritDoc}
*/
@Override
public int compare(final Lease lease1, final Lease lease2) {
int result = 0;
final String shardId1 = lease1.leaseKey();
final String shardId2 = lease2.leaseKey();
final Shard shard1 = shardIdToShardMap.get(shardId1);
final Shard shard2 = shardIdToShardMap.get(shardId2);
// If we found shards for the two leases, use comparison of the starting sequence numbers
if (shard1 != null && shard2 != null) {
BigInteger sequenceNumber1 = new BigInteger(shard1.sequenceNumberRange().startingSequenceNumber());
BigInteger sequenceNumber2 = new BigInteger(shard2.sequenceNumberRange().startingSequenceNumber());
result = sequenceNumber1.compareTo(sequenceNumber2);
}
if (result == 0) {
result = shardId1.compareTo(shardId2);
}
return result;
}
}
}

219
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/KinesisShardDetector.java
View file

@ -1,219 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.time.Duration;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Function;
import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;
import lombok.AccessLevel;
import lombok.Getter;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.Synchronized;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.awssdk.services.kinesis.model.KinesisException;
import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
import software.amazon.awssdk.services.kinesis.model.ResourceInUseException;
import software.amazon.awssdk.services.kinesis.model.Shard;
import software.amazon.awssdk.utils.CollectionUtils;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.common.KinesisRequestsBuilder;
import software.amazon.kinesis.retrieval.AWSExceptionManager;
/**
*
*/
@RequiredArgsConstructor
@Slf4j
@Accessors(fluent = true)
@KinesisClientInternalApi
public class KinesisShardDetector implements ShardDetector {
@NonNull
private final KinesisAsyncClient kinesisClient;
@NonNull
private final String streamName;
private final long listShardsBackoffTimeInMillis;
private final int maxListShardsRetryAttempts;
private final long listShardsCacheAllowedAgeInSeconds;
private final int maxCacheMissesBeforeReload;
private final int cacheMissWarningModulus;
private volatile Map<String, Shard> cachedShardMap = null;
private volatile Instant lastCacheUpdateTime;
@Getter(AccessLevel.PACKAGE)
private AtomicInteger cacheMisses = new AtomicInteger(0);
@Override
public Shard shard(@NonNull final String shardId) {
if (CollectionUtils.isNullOrEmpty(this.cachedShardMap)) {
synchronized (this) {
if (CollectionUtils.isNullOrEmpty(this.cachedShardMap)) {
listShards();
}
}
}
Shard shard = cachedShardMap.get(shardId);
if (shard == null) {
if (cacheMisses.incrementAndGet() > maxCacheMissesBeforeReload || shouldRefreshCache()) {
synchronized (this) {
shard = cachedShardMap.get(shardId);
if (shard == null) {
log.info("Too many shard map cache misses or cache is out of date -- forcing a refresh");
listShards();
shard = cachedShardMap.get(shardId);
if (shard == null) {
log.warn("Even after cache refresh shard '{}' wasn't found. This could indicate a bigger"
+ " problem.", shardId);
}
cacheMisses.set(0);
} else {
//
// If the shardmap got updated, go ahead and set cache misses to 0
//
cacheMisses.set(0);
}
}
}
}
if (shard == null) {
final String message = String.format("Cannot find the shard given the shardId %s. Cache misses: %s",
shardId, cacheMisses);
if (cacheMisses.get() % cacheMissWarningModulus == 0) {
log.warn(message);
} else {
log.debug(message);
}
}
return shard;
}
@Override
@Synchronized
public List<Shard> listShards() {
final List<Shard> shards = new ArrayList<>();
ListShardsResponse result;
String nextToken = null;
do {
result = listShards(nextToken);
if (result == null) {
/*
* If listShards ever returns null, we should bail and return null. This indicates the stream is not
* in ACTIVE or UPDATING state and we may not have accurate/consistent information about the stream.
*/
return null;
} else {
shards.addAll(result.shards());
nextToken = result.nextToken();
}
} while (StringUtils.isNotEmpty(result.nextToken()));
cachedShardMap(shards);
return shards;
}
private ListShardsResponse listShards(final String nextToken) {
final AWSExceptionManager exceptionManager = new AWSExceptionManager();
exceptionManager.add(LimitExceededException.class, t -> t);
exceptionManager.add(ResourceInUseException.class, t -> t);
exceptionManager.add(KinesisException.class, t -> t);
ListShardsRequest.Builder request = KinesisRequestsBuilder.listShardsRequestBuilder();
if (StringUtils.isEmpty(nextToken)) {
request = request.streamName(streamName);
} else {
request = request.nextToken(nextToken);
}
ListShardsResponse result = null;
LimitExceededException lastException = null;
int remainingRetries = maxListShardsRetryAttempts;
while (result == null) {
try {
try {
result = kinesisClient.listShards(request.build()).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: check if this is the correct behavior for Interrupted Exception
log.debug("Interrupted exception caught, shutdown initiated, returning null");
return null;
}
} catch (ResourceInUseException e) {
log.info("Stream is not in Active/Updating status, returning null (wait until stream is in"
+ " Active or Updating)");
return null;
} catch (LimitExceededException e) {
log.info("Got LimitExceededException when listing shards {}. Backing off for {} millis.", streamName,
listShardsBackoffTimeInMillis);
try {
Thread.sleep(listShardsBackoffTimeInMillis);
} catch (InterruptedException ie) {
log.debug("Stream {} : Sleep was interrupted ", streamName, ie);
}
lastException = e;
}
remainingRetries--;
if (remainingRetries <= 0 && result == null) {
if (lastException != null) {
throw lastException;
}
throw new IllegalStateException("Received null from ListShards call.");
}
}
return result;
}
void cachedShardMap(final List<Shard> shards) {
cachedShardMap = shards.stream().collect(Collectors.toMap(Shard::shardId, Function.identity()));
lastCacheUpdateTime = Instant.now();
}
private boolean shouldRefreshCache() {
final Duration secondsSinceLastUpdate = Duration.between(lastCacheUpdateTime, Instant.now());
final String message = String.format("Shard map cache is %d seconds old", secondsSinceLastUpdate.getSeconds());
if (secondsSinceLastUpdate.compareTo(Duration.of(listShardsCacheAllowedAgeInSeconds, ChronoUnit.SECONDS)) > 0) {
log.info("{}. Age exceeds limit of {} seconds -- Refreshing.", message, listShardsCacheAllowedAgeInSeconds);
return true;
}
log.debug("{}. Age doesn't exceed limit of {} seconds.", message, listShardsCacheAllowedAgeInSeconds);
return false;
}
}

152
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseCoordinator.java
View file

@ -1,152 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.UUID;
import software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
/**
*
*/
public interface LeaseCoordinator {
/**
* Initialize the lease coordinator (create the lease table if needed).
* @throws DependencyException
* @throws ProvisionedThroughputException
*/
void initialize() throws ProvisionedThroughputException, DependencyException, IllegalStateException;
/**
* Start background LeaseHolder and LeaseTaker threads.
* @throws ProvisionedThroughputException If we can't talk to DynamoDB due to insufficient capacity.
* @throws InvalidStateException If the lease table doesn't exist
* @throws DependencyException If we encountered exception taking to DynamoDB
*/
void start() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Runs a single iteration of the lease taker - used by integration tests.
*
* @throws InvalidStateException
* @throws DependencyException
*/
void runLeaseTaker() throws DependencyException, InvalidStateException;
/**
* Runs a single iteration of the lease renewer - used by integration tests.
*
* @throws InvalidStateException
* @throws DependencyException
*/
void runLeaseRenewer() throws DependencyException, InvalidStateException;
/**
* @return true if this LeaseCoordinator is running
*/
boolean isRunning();
/**
* @return workerIdentifier
*/
String workerIdentifier();
/**
* @return {@link LeaseRefresher}
*/
LeaseRefresher leaseRefresher();
/**
* @return currently held leases
*/
Collection<Lease> getAssignments();
/**
* @param leaseKey lease key to fetch currently held lease for
*
* @return deep copy of currently held Lease for given key, or null if we don't hold the lease for that key
*/
Lease getCurrentlyHeldLease(String leaseKey);
/**
* Updates application-specific lease values in DynamoDB.
*
* @param lease lease object containing updated values
* @param concurrencyToken obtained by calling Lease.concurrencyToken for a currently held lease
*
* @return true if update succeeded, false otherwise
*
* @throws InvalidStateException if lease table does not exist
* @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
* @throws DependencyException if DynamoDB update fails in an unexpected way
*/
boolean updateLease(Lease lease, UUID concurrencyToken, String operation, String shardId)
throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Requests the cancellation of the lease taker.
*/
void stopLeaseTaker();
/**
* Requests that renewals for the given lease are stopped.
*
* @param lease the lease to stop renewing.
*/
void dropLease(Lease lease);
/**
* Stops background threads and waits for specific amount of time for all background tasks to complete.
* If tasks are not completed after this time, method will shutdown thread pool forcefully and return.
*/
void stop();
/**
* @return Current shard/lease assignments
*/
List<ShardInfo> getCurrentAssignments();
/**
* Default implementation returns an empty list and concrete implementation is expected to return all leases
* for the application that are in the lease table. This enables application managing Kcl Scheduler to take care of
* horizontal scaling for example.
*
* @return all leases for the application that are in the lease table
*/
default List<Lease> allLeases() {
return Collections.emptyList();
}
/**
* @param writeCapacity The DynamoDB table used for tracking leases will be provisioned with the specified initial
* write capacity
* @return LeaseCoordinator
*/
DynamoDBLeaseCoordinator initialLeaseTableWriteCapacity(long writeCapacity);
/**
* @param readCapacity The DynamoDB table used for tracking leases will be provisioned with the specified initial
* read capacity
* @return LeaseCoordinator
*/
DynamoDBLeaseCoordinator initialLeaseTableReadCapacity(long readCapacity);
}

269
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseManagementConfig.java
View file

@ -1,269 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.kinesis.common.InitialPositionInStream;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseManagementFactory;
import software.amazon.kinesis.leases.dynamodb.TableCreatorCallback;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.NullMetricsFactory;
/**
* Used by the KCL to configure lease management.
*/
@Data
@Accessors(fluent = true)
public class LeaseManagementConfig {
/**
* Name of the table to use in DynamoDB
*
* @return String
*/
@NonNull
private final String tableName;
/**
* Client to be used to access DynamoDB service.
*
* @return {@link DynamoDbAsyncClient}
*/
@NonNull
private final DynamoDbAsyncClient dynamoDBClient;
/**
* Client to be used to access Kinesis Data Streams service.
*
* @return {@link KinesisAsyncClient}
*/
@NonNull
private final KinesisAsyncClient kinesisClient;
/**
* Name of the Kinesis Data Stream to read records from.
*/
@NonNull
private final String streamName;
/**
* Used to distinguish different workers/processes of a KCL application.
*
* @return String
*/
@NonNull
private final String workerIdentifier;
/**
* Fail over time in milliseconds. A worker which does not renew it's lease within this time interval
* will be regarded as having problems and it's shards will be assigned to other workers.
* For applications that have a large number of shards, this may be set to a higher number to reduce
* the number of DynamoDB IOPS required for tracking leases.
*
* <p>Default value: 10000L</p>
*/
private long failoverTimeMillis = 10000L;
/**
* Shard sync interval in milliseconds - e.g. wait for this long between shard sync tasks.
*
* <p>Default value: 60000L</p>
*/
private long shardSyncIntervalMillis = 60000L;
/**
* Cleanup leases upon shards completion (don't wait until they expire in Kinesis).
* Keeping leases takes some tracking/resources (e.g. they need to be renewed, assigned), so by default we try
* to delete the ones we don't need any longer.
*
* <p>Default value: true</p>
*/
private boolean cleanupLeasesUponShardCompletion = true;
/**
* The max number of leases (shards) this worker should process.
* This can be useful to avoid overloading (and thrashing) a worker when a host has resource constraints
* or during deployment.
*
* <p>NOTE: Setting this to a low value can cause data loss if workers are not able to pick up all shards in the
* stream due to the max limit.</p>
*
* <p>Default value: {@link Integer#MAX_VALUE}</p>
*/
private int maxLeasesForWorker = Integer.MAX_VALUE;;
/**
* Max leases to steal from another worker at one time (for load balancing).
* Setting this to a higher number can allow for faster load convergence (e.g. during deployments, cold starts),
* but can cause higher churn in the system.
*
* <p>Default value: 1</p>
*/
private int maxLeasesToStealAtOneTime = 1;
/**
* The Amazon DynamoDB table used for tracking leases will be provisioned with this read capacity.
*
* <p>Default value: 10</p>
*/
private int initialLeaseTableReadCapacity = 10;
/**
* The Amazon DynamoDB table used for tracking leases will be provisioned with this write capacity.
*
* <p>Default value: 10</p>
*/
private int initialLeaseTableWriteCapacity = 10;
/**
* The size of the thread pool to create for the lease renewer to use.
*
* <p>Default value: 20</p>
*/
private int maxLeaseRenewalThreads = 20;
/**
*
*/
private boolean ignoreUnexpectedChildShards = false;
/**
*
*/
private boolean consistentReads = false;
private long listShardsBackoffTimeInMillis = 1500L;
private int maxListShardsRetryAttempts = 50;
public long epsilonMillis = 25L;
/**
* The initial position for getting records from Kinesis streams.
*
* <p>Default value: {@link InitialPositionInStream#TRIM_HORIZON}</p>
*/
private InitialPositionInStreamExtended initialPositionInStream =
InitialPositionInStreamExtended.newInitialPosition(InitialPositionInStream.TRIM_HORIZON);
private int maxCacheMissesBeforeReload = 1000;
private long listShardsCacheAllowedAgeInSeconds = 30;
private int cacheMissWarningModulus = 250;
private MetricsFactory metricsFactory = new NullMetricsFactory();
/**
* Returns the metrics factory.
*
* <p>
* NOTE: This method is deprecated and will be removed in a future release. This metrics factory is not being used
* in the KCL.
* </p>
*
* @return
*/
@Deprecated
public MetricsFactory metricsFactory() {
return metricsFactory;
}
/**
* Sets the metrics factory.
*
* <p>
* NOTE: This method is deprecated and will be removed in a future release. This metrics factory is not being used
* in the KCL.
* </p>
*
* @param metricsFactory
*/
@Deprecated
public LeaseManagementConfig metricsFactory(final MetricsFactory metricsFactory) {
this.metricsFactory = metricsFactory;
return this;
}
/**
* The {@link ExecutorService} to be used by {@link ShardSyncTaskManager}.
*
* <p>Default value: {@link LeaseManagementThreadPool}</p>
*/
private ExecutorService executorService = new LeaseManagementThreadPool(
new ThreadFactoryBuilder().setNameFormat("ShardSyncTaskManager-%04d").build());
static class LeaseManagementThreadPool extends ThreadPoolExecutor {
private static final long DEFAULT_KEEP_ALIVE_TIME = 60L;
LeaseManagementThreadPool(ThreadFactory threadFactory) {
super(0, Integer.MAX_VALUE, DEFAULT_KEEP_ALIVE_TIME, TimeUnit.SECONDS, new SynchronousQueue<>(),
threadFactory);
}
}
/**
* Callback used with DynamoDB lease management. Callback is invoked once the table is newly created and is in the
* active status.
*
* <p>
* Default value: {@link TableCreatorCallback#NOOP_TABLE_CREATOR_CALLBACK}
* </p>
*/
private TableCreatorCallback tableCreatorCallback = TableCreatorCallback.NOOP_TABLE_CREATOR_CALLBACK;
private HierarchicalShardSyncer hierarchicalShardSyncer = new HierarchicalShardSyncer();
private LeaseManagementFactory leaseManagementFactory;
public LeaseManagementFactory leaseManagementFactory() {
if (leaseManagementFactory == null) {
leaseManagementFactory = new DynamoDBLeaseManagementFactory(kinesisClient(),
streamName(),
dynamoDBClient(),
tableName(),
workerIdentifier(),
executorService(),
initialPositionInStream(),
failoverTimeMillis(),
epsilonMillis(),
maxLeasesForWorker(),
maxLeasesToStealAtOneTime(),
maxLeaseRenewalThreads(),
cleanupLeasesUponShardCompletion(),
ignoreUnexpectedChildShards(),
shardSyncIntervalMillis(),
consistentReads(),
listShardsBackoffTimeInMillis(),
maxListShardsRetryAttempts(),
maxCacheMissesBeforeReload(),
listShardsCacheAllowedAgeInSeconds(),
cacheMissWarningModulus(),
initialLeaseTableReadCapacity(),
initialLeaseTableWriteCapacity(),
hierarchicalShardSyncer(),
tableCreatorCallback());
}
return leaseManagementFactory;
}
}

32
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseManagementFactory.java
View file

@ -1,32 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseRefresher;
import software.amazon.kinesis.metrics.MetricsFactory;
/**
*
*/
public interface LeaseManagementFactory {
LeaseCoordinator createLeaseCoordinator(MetricsFactory metricsFactory);
ShardSyncTaskManager createShardSyncTaskManager(MetricsFactory metricsFactory);
DynamoDBLeaseRefresher createLeaseRefresher();
ShardDetector createShardDetector();
}

115
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseSerializer.java
View file

@ -1,115 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.Collection;
import java.util.Map;
import software.amazon.awssdk.services.dynamodb.model.AttributeDefinition;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.AttributeValueUpdate;
import software.amazon.awssdk.services.dynamodb.model.ExpectedAttributeValue;
import software.amazon.awssdk.services.dynamodb.model.KeySchemaElement;
import software.amazon.kinesis.leases.Lease;
/**
* Utility class that manages the mapping of Lease objects/operations to records in DynamoDB.
*/
public interface LeaseSerializer {
/**
* Construct a DynamoDB record out of a Lease object
*
* @param lease lease object to serialize
* @return an attribute value map representing the lease object
*/
Map<String, AttributeValue> toDynamoRecord(Lease lease);
/**
* Construct a Lease object out of a DynamoDB record.
*
* @param dynamoRecord attribute value map from DynamoDB
* @return a deserialized lease object representing the attribute value map
*/
Lease fromDynamoRecord(Map<String, AttributeValue> dynamoRecord);
/**
* @param lease
* @return the attribute value map representing a Lease's hash key given a Lease object.
*/
Map<String, AttributeValue> getDynamoHashKey(Lease lease);
/**
* Special getDynamoHashKey implementation used by {@link LeaseRefresher#getLease(String)}.
*
* @param leaseKey
* @return the attribute value map representing a Lease's hash key given a string.
*/
Map<String, AttributeValue> getDynamoHashKey(String leaseKey);
/**
* @param lease
* @return the attribute value map asserting that a lease counter is what we expect.
*/
Map<String, ExpectedAttributeValue> getDynamoLeaseCounterExpectation(Lease lease);
/**
* @param lease
* @return the attribute value map asserting that the lease owner is what we expect.
*/
Map<String, ExpectedAttributeValue> getDynamoLeaseOwnerExpectation(Lease lease);
/**
* @return the attribute value map asserting that a lease does not exist.
*/
Map<String, ExpectedAttributeValue> getDynamoNonexistantExpectation();
/**
* @param lease
* @return the attribute value map that increments a lease counter
*/
Map<String, AttributeValueUpdate> getDynamoLeaseCounterUpdate(Lease lease);
/**
* @param lease
* @param newOwner
* @return the attribute value map that takes a lease for a new owner
*/
Map<String, AttributeValueUpdate> getDynamoTakeLeaseUpdate(Lease lease, String newOwner);
/**
* @param lease
* @return the attribute value map that voids a lease
*/
Map<String, AttributeValueUpdate> getDynamoEvictLeaseUpdate(Lease lease);
/**
* @param lease
* @return the attribute value map that updates application-specific data for a lease and increments the lease
* counter
*/
Map<String, AttributeValueUpdate> getDynamoUpdateLeaseUpdate(Lease lease);
/**
* @return the key schema for creating a DynamoDB table to store leases
*/
Collection<KeySchemaElement> getKeySchema();
/**
* @return attribute definitions for creating a DynamoDB table to store leases
*/
Collection<AttributeDefinition> getAttributeDefinitions();
}

59
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseTaker.java
View file

@ -1,59 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
/**
* ILeaseTaker is used by LeaseCoordinator to take new leases, or leases that other workers fail to renew. Each
* LeaseCoordinator instance corresponds to one worker and uses exactly one ILeaseTaker to take leases for that worker.
*/
public interface LeaseTaker {
/**
* Compute the set of leases available to be taken and attempt to take them. Lease taking rules are:
*
* 1) If a lease's counter hasn't changed in long enough, try to take it.
* 2) If we see a lease we've never seen before, take it only if owner == null. If it's owned, odds are the owner is
* holding it. We can't tell until we see it more than once.
* 3) For load balancing purposes, you may violate rules 1 and 2 for EXACTLY ONE lease per call of takeLeases().
*
* @return map of shardId to Lease object for leases we just successfully took.
*
* @throws DependencyException on unexpected DynamoDB failures
* @throws InvalidStateException if lease table does not exist
*/
Map<String, Lease> takeLeases() throws DependencyException, InvalidStateException;
/**
* @return workerIdentifier for this LeaseTaker
*/
String getWorkerIdentifier();
/**
* Default implementation returns an empty list and concrete implementaion is expected to return all leases
* for the application that are in the lease table either by reading lease table or from an internal cache.
*
* @return all leases for the application that are in the lease table
*/
default List<Lease> allLeases() {
return Collections.emptyList();
}
}

35
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/NoOpShardPrioritization.java
View file

@ -1,35 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.List;
/**
* Shard Prioritization that returns the same original list of shards without any modifications.
*/
public class NoOpShardPrioritization implements
ShardPrioritization {
/**
* Empty constructor for NoOp Shard Prioritization.
*/
public NoOpShardPrioritization() {
}
@Override
public List<ShardInfo> prioritize(List<ShardInfo> original) {
return original;
}
}

30
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardDetector.java
View file

@ -1,30 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import software.amazon.awssdk.services.kinesis.model.Shard;
import java.util.List;
/**
*
*/
public interface ShardDetector {
Shard shard(String shardId);
List<Shard> listShards();
}

33
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardPrioritization.java
View file

@ -1,33 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.List;
/**
* Provides logic to prioritize or filter shards before their execution.
*/
public interface ShardPrioritization {
/**
* Returns new list of shards ordered based on their priority.
* Resulted list may have fewer shards compared to original list
*
* @param original
* list of shards needed to be prioritized
* @return new list that contains only shards that should be processed
*/
List<ShardInfo> prioritize(List<ShardInfo> original);
}

91
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardSyncTask.java
View file

@ -1,91 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.lifecycle.ConsumerTask;
import software.amazon.kinesis.lifecycle.TaskResult;
import software.amazon.kinesis.lifecycle.TaskType;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsUtil;
/**
* This task syncs leases/activies with shards of the stream.
* It will create new leases/activites when it discovers new shards (e.g. setup/resharding).
* It will clean up leases/activities for shards that have been completely processed (if
* cleanupLeasesUponShardCompletion is true).
*/
@RequiredArgsConstructor
@Slf4j
@KinesisClientInternalApi
public class ShardSyncTask implements ConsumerTask {
private final String SHARD_SYNC_TASK_OPERATION = "ShardSyncTask";
@NonNull
private final ShardDetector shardDetector;
@NonNull
private final LeaseRefresher leaseRefresher;
@NonNull
private final InitialPositionInStreamExtended initialPosition;
private final boolean cleanupLeasesUponShardCompletion;
private final boolean ignoreUnexpectedChildShards;
private final long shardSyncTaskIdleTimeMillis;
@NonNull
private final HierarchicalShardSyncer hierarchicalShardSyncer;
@NonNull
private final MetricsFactory metricsFactory;
private final TaskType taskType = TaskType.SHARDSYNC;
/*
* (non-Javadoc)
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#call()
*/
@Override
public TaskResult call() {
Exception exception = null;
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, SHARD_SYNC_TASK_OPERATION);
try {
hierarchicalShardSyncer.checkAndCreateLeaseForNewShards(shardDetector, leaseRefresher, initialPosition,
cleanupLeasesUponShardCompletion, ignoreUnexpectedChildShards, scope);
if (shardSyncTaskIdleTimeMillis > 0) {
Thread.sleep(shardSyncTaskIdleTimeMillis);
}
} catch (Exception e) {
log.error("Caught exception while sync'ing Kinesis shards and leases", e);
exception = e;
} finally {
MetricsUtil.endScope(scope);
}
return new TaskResult(exception);
}
/*
* (non-Javadoc)
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#taskType()
*/
@Override
public TaskType taskType() {
return taskType;
}
}

162
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardSyncTaskManager.java
View file

@ -1,162 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.lifecycle.ConsumerTask;
import software.amazon.kinesis.lifecycle.TaskResult;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsCollectingTaskDecorator;
/**
* The ShardSyncTaskManager is used to track the task to sync shards with leases (create leases for new
* Kinesis shards, remove obsolete leases). We'll have at most one outstanding sync task at any time.
* Worker will use this class to kick off a sync task when it finds shards which have been completely processed.
*/
@Data
@Accessors(fluent = true)
@Slf4j
public class ShardSyncTaskManager {
@NonNull
private final ShardDetector shardDetector;
@NonNull
private final LeaseRefresher leaseRefresher;
@NonNull
private final InitialPositionInStreamExtended initialPositionInStream;
private final boolean cleanupLeasesUponShardCompletion;
private final boolean ignoreUnexpectedChildShards;
private final long shardSyncIdleTimeMillis;
@NonNull
private final ExecutorService executorService;
@NonNull
private final HierarchicalShardSyncer hierarchicalShardSyncer;
@NonNull
private final MetricsFactory metricsFactory;
/**
* Constructor.
*
* <p>NOTE: This constructor is deprecated and will be removed in a future release.</p>
*
* @param shardDetector
* @param leaseRefresher
* @param initialPositionInStream
* @param cleanupLeasesUponShardCompletion
* @param ignoreUnexpectedChildShards
* @param shardSyncIdleTimeMillis
* @param executorService
* @param metricsFactory
*/
@Deprecated
public ShardSyncTaskManager(ShardDetector shardDetector, LeaseRefresher leaseRefresher,
InitialPositionInStreamExtended initialPositionInStream, boolean cleanupLeasesUponShardCompletion,
boolean ignoreUnexpectedChildShards, long shardSyncIdleTimeMillis, ExecutorService executorService,
MetricsFactory metricsFactory) {
this.shardDetector = shardDetector;
this.leaseRefresher = leaseRefresher;
this.initialPositionInStream = initialPositionInStream;
this.cleanupLeasesUponShardCompletion = cleanupLeasesUponShardCompletion;
this.ignoreUnexpectedChildShards = ignoreUnexpectedChildShards;
this.shardSyncIdleTimeMillis = shardSyncIdleTimeMillis;
this.executorService = executorService;
this.hierarchicalShardSyncer = new HierarchicalShardSyncer();
this.metricsFactory = metricsFactory;
}
/**
* Constructor.
*
* @param shardDetector
* @param leaseRefresher
* @param initialPositionInStream
* @param cleanupLeasesUponShardCompletion
* @param ignoreUnexpectedChildShards
* @param shardSyncIdleTimeMillis
* @param executorService
* @param hierarchicalShardSyncer
* @param metricsFactory
*/
public ShardSyncTaskManager(ShardDetector shardDetector, LeaseRefresher leaseRefresher,
InitialPositionInStreamExtended initialPositionInStream, boolean cleanupLeasesUponShardCompletion,
boolean ignoreUnexpectedChildShards, long shardSyncIdleTimeMillis, ExecutorService executorService,
HierarchicalShardSyncer hierarchicalShardSyncer, MetricsFactory metricsFactory) {
this.shardDetector = shardDetector;
this.leaseRefresher = leaseRefresher;
this.initialPositionInStream = initialPositionInStream;
this.cleanupLeasesUponShardCompletion = cleanupLeasesUponShardCompletion;
this.ignoreUnexpectedChildShards = ignoreUnexpectedChildShards;
this.shardSyncIdleTimeMillis = shardSyncIdleTimeMillis;
this.executorService = executorService;
this.hierarchicalShardSyncer = hierarchicalShardSyncer;
this.metricsFactory = metricsFactory;
}
private ConsumerTask currentTask;
private Future<TaskResult> future;
public synchronized boolean syncShardAndLeaseInfo() {
return checkAndSubmitNextTask();
}
private synchronized boolean checkAndSubmitNextTask() {
boolean submittedNewTask = false;
if ((future == null) || future.isCancelled() || future.isDone()) {
if ((future != null) && future.isDone()) {
try {
TaskResult result = future.get();
if (result.getException() != null) {
log.error("Caught exception running {} task: ", currentTask.taskType(),
result.getException());
}
} catch (InterruptedException | ExecutionException e) {
log.warn("{} task encountered exception.", currentTask.taskType(), e);
}
}
currentTask =
new MetricsCollectingTaskDecorator(
new ShardSyncTask(shardDetector,
leaseRefresher,
initialPositionInStream,
cleanupLeasesUponShardCompletion,
ignoreUnexpectedChildShards,
shardSyncIdleTimeMillis,
hierarchicalShardSyncer,
metricsFactory),
metricsFactory);
future = executorService.submit(currentTask);
submittedNewTask = true;
if (log.isDebugEnabled()) {
log.debug("Submitted new {} task.", currentTask.taskType());
}
} else {
if (log.isDebugEnabled()) {
log.debug("Previous {} task still pending. Not submitting new task.", currentTask.taskType());
}
}
return submittedNewTask;
}
}

414
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseCoordinator.java
View file

@ -1,414 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedTransferQueue;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.LeaseRenewer;
import software.amazon.kinesis.leases.LeaseTaker;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.LeasingException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsLevel;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsUtil;
/**
* LeaseCoordinator abstracts away LeaseTaker and LeaseRenewer from the application code that's using leasing. It owns
* the scheduling of the two previously mentioned components as well as informing LeaseRenewer when LeaseTaker takes new
* leases.
*
*/
@Slf4j
@KinesisClientInternalApi
public class DynamoDBLeaseCoordinator implements LeaseCoordinator {
// Time to wait for in-flight Runnables to finish when calling .stop();
private static final long STOP_WAIT_TIME_MILLIS = 2000L;
private static final ThreadFactory LEASE_COORDINATOR_THREAD_FACTORY = new ThreadFactoryBuilder()
.setNameFormat("LeaseCoordinator-%04d").setDaemon(true).build();
private static final ThreadFactory LEASE_RENEWAL_THREAD_FACTORY = new ThreadFactoryBuilder()
.setNameFormat("LeaseRenewer-%04d").setDaemon(true).build();
private final LeaseRenewer leaseRenewer;
private final LeaseTaker leaseTaker;
private final long renewerIntervalMillis;
private final long takerIntervalMillis;
private final ExecutorService leaseRenewalThreadpool;
private final LeaseRefresher leaseRefresher;
private long initialLeaseTableReadCapacity;
private long initialLeaseTableWriteCapacity;
protected final MetricsFactory metricsFactory;
private final Object shutdownLock = new Object();
private ScheduledExecutorService leaseCoordinatorThreadPool;
private ScheduledFuture<?> takerFuture;
private volatile boolean running = false;
/**
* Constructor.
*
* <p>NOTE: This constructor is deprecated and will be removed in a future release.</p>
*
* @param leaseRefresher
* LeaseRefresher instance to use
* @param workerIdentifier
* Identifies the worker (e.g. useful to track lease ownership)
* @param leaseDurationMillis
* Duration of a lease
* @param epsilonMillis
* Allow for some variance when calculating lease expirations
* @param maxLeasesForWorker
* Max leases this Worker can handle at a time
* @param maxLeasesToStealAtOneTime
* Steal up to these many leases at a time (for load balancing)
* @param metricsFactory
* Used to publish metrics about lease operations
*/
@Deprecated
public DynamoDBLeaseCoordinator(final LeaseRefresher leaseRefresher,
final String workerIdentifier,
final long leaseDurationMillis,
final long epsilonMillis,
final int maxLeasesForWorker,
final int maxLeasesToStealAtOneTime,
final int maxLeaseRenewerThreadCount,
final MetricsFactory metricsFactory) {
this(leaseRefresher, workerIdentifier, leaseDurationMillis, epsilonMillis, maxLeasesForWorker,
maxLeasesToStealAtOneTime, maxLeaseRenewerThreadCount,
TableConstants.DEFAULT_INITIAL_LEASE_TABLE_READ_CAPACITY,
TableConstants.DEFAULT_INITIAL_LEASE_TABLE_WRITE_CAPACITY, metricsFactory);
}
/**
* Constructor.
*
* @param leaseRefresher
* LeaseRefresher instance to use
* @param workerIdentifier
* Identifies the worker (e.g. useful to track lease ownership)
* @param leaseDurationMillis
* Duration of a lease
* @param epsilonMillis
* Allow for some variance when calculating lease expirations
* @param maxLeasesForWorker
* Max leases this Worker can handle at a time
* @param maxLeasesToStealAtOneTime
* Steal up to these many leases at a time (for load balancing)
* @param initialLeaseTableReadCapacity
* Initial dynamodb lease table read iops if creating the lease table
* @param initialLeaseTableWriteCapacity
* Initial dynamodb lease table write iops if creating the lease table
* @param metricsFactory
* Used to publish metrics about lease operations
*/
public DynamoDBLeaseCoordinator(final LeaseRefresher leaseRefresher,
final String workerIdentifier,
final long leaseDurationMillis,
final long epsilonMillis,
final int maxLeasesForWorker,
final int maxLeasesToStealAtOneTime,
final int maxLeaseRenewerThreadCount,
final long initialLeaseTableReadCapacity,
final long initialLeaseTableWriteCapacity,
final MetricsFactory metricsFactory) {
this.leaseRefresher = leaseRefresher;
this.leaseRenewalThreadpool = getLeaseRenewalExecutorService(maxLeaseRenewerThreadCount);
this.leaseTaker = new DynamoDBLeaseTaker(leaseRefresher, workerIdentifier, leaseDurationMillis, metricsFactory)
.withMaxLeasesForWorker(maxLeasesForWorker)
.withMaxLeasesToStealAtOneTime(maxLeasesToStealAtOneTime);
this.leaseRenewer = new DynamoDBLeaseRenewer(
leaseRefresher, workerIdentifier, leaseDurationMillis, leaseRenewalThreadpool, metricsFactory);
this.renewerIntervalMillis = leaseDurationMillis / 3 - epsilonMillis;
this.takerIntervalMillis = (leaseDurationMillis + epsilonMillis) * 2;
if (initialLeaseTableReadCapacity <= 0) {
throw new IllegalArgumentException("readCapacity should be >= 1");
}
this.initialLeaseTableReadCapacity = initialLeaseTableReadCapacity;
if (initialLeaseTableWriteCapacity <= 0) {
throw new IllegalArgumentException("writeCapacity should be >= 1");
}
this.initialLeaseTableWriteCapacity = initialLeaseTableWriteCapacity;
this.metricsFactory = metricsFactory;
log.info("With failover time {} ms and epsilon {} ms, LeaseCoordinator will renew leases every {} ms, take"
+ "leases every {} ms, process maximum of {} leases and steal {} lease(s) at a time.",
leaseDurationMillis,
epsilonMillis,
renewerIntervalMillis,
takerIntervalMillis,
maxLeasesForWorker,
maxLeasesToStealAtOneTime);
}
private class TakerRunnable implements Runnable {
@Override
public void run() {
try {
runLeaseTaker();
} catch (LeasingException e) {
log.error("LeasingException encountered in lease taking thread", e);
} catch (Throwable t) {
log.error("Throwable encountered in lease taking thread", t);
}
}
}
private class RenewerRunnable implements Runnable {
@Override
public void run() {
try {
runLeaseRenewer();
} catch (LeasingException e) {
log.error("LeasingException encountered in lease renewing thread", e);
} catch (Throwable t) {
log.error("Throwable encountered in lease renewing thread", t);
}
}
}
@Override
public void initialize() throws ProvisionedThroughputException, DependencyException, IllegalStateException {
final boolean newTableCreated =
leaseRefresher.createLeaseTableIfNotExists(initialLeaseTableReadCapacity, initialLeaseTableWriteCapacity);
if (newTableCreated) {
log.info("Created new lease table for coordinator with initial read capacity of {} and write capacity of {}.",
initialLeaseTableReadCapacity, initialLeaseTableWriteCapacity);
}
// Need to wait for table in active state.
final long secondsBetweenPolls = 10L;
final long timeoutSeconds = 600L;
final boolean isTableActive = leaseRefresher.waitUntilLeaseTableExists(secondsBetweenPolls, timeoutSeconds);
if (!isTableActive) {
throw new DependencyException(new IllegalStateException("Creating table timeout"));
}
}
@Override
public void start() throws DependencyException, InvalidStateException, ProvisionedThroughputException {
leaseRenewer.initialize();
// 2 because we know we'll have at most 2 concurrent tasks at a time.
leaseCoordinatorThreadPool = Executors.newScheduledThreadPool(2, LEASE_COORDINATOR_THREAD_FACTORY);
// Taker runs with fixed DELAY because we want it to run slower in the event of performance degredation.
takerFuture = leaseCoordinatorThreadPool.scheduleWithFixedDelay(new TakerRunnable(),
0L,
takerIntervalMillis,
TimeUnit.MILLISECONDS);
// Renewer runs at fixed INTERVAL because we want it to run at the same rate in the event of degredation.
leaseCoordinatorThreadPool.scheduleAtFixedRate(new RenewerRunnable(),
0L,
renewerIntervalMillis,
TimeUnit.MILLISECONDS);
running = true;
}
@Override
public void runLeaseTaker() throws DependencyException, InvalidStateException {
MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, "TakeLeases");
long startTime = System.currentTimeMillis();
boolean success = false;
try {
Map<String, Lease> takenLeases = leaseTaker.takeLeases();
// Only add taken leases to renewer if coordinator is still running.
synchronized (shutdownLock) {
if (running) {
leaseRenewer.addLeasesToRenew(takenLeases.values());
}
}
success = true;
} finally {
MetricsUtil.addWorkerIdentifier(scope, workerIdentifier());
MetricsUtil.addSuccessAndLatency(scope, success, startTime, MetricsLevel.SUMMARY);
MetricsUtil.endScope(scope);
}
}
@Override
public void runLeaseRenewer() throws DependencyException, InvalidStateException {
leaseRenewer.renewLeases();
}
@Override
public Collection<Lease> getAssignments() {
return leaseRenewer.getCurrentlyHeldLeases().values();
}
@Override
public List<Lease> allLeases() {
return leaseTaker.allLeases();
}
@Override
public Lease getCurrentlyHeldLease(String leaseKey) {
return leaseRenewer.getCurrentlyHeldLease(leaseKey);
}
@Override
public String workerIdentifier() {
return leaseTaker.getWorkerIdentifier();
}
@Override
public LeaseRefresher leaseRefresher() {
return leaseRefresher;
}
@Override
public void stop() {
if (leaseCoordinatorThreadPool != null) {
leaseCoordinatorThreadPool.shutdown();
try {
if (leaseCoordinatorThreadPool.awaitTermination(STOP_WAIT_TIME_MILLIS, TimeUnit.MILLISECONDS)) {
log.info("Worker {} has successfully stopped lease-tracking threads",
leaseTaker.getWorkerIdentifier());
} else {
leaseCoordinatorThreadPool.shutdownNow();
log.info("Worker {} stopped lease-tracking threads {} ms after stop",
leaseTaker.getWorkerIdentifier(),
STOP_WAIT_TIME_MILLIS);
}
} catch (InterruptedException e) {
log.debug("Encountered InterruptedException when awaiting threadpool termination");
}
} else {
log.debug("Threadpool was null, no need to shutdown/terminate threadpool.");
}
leaseRenewalThreadpool.shutdownNow();
synchronized (shutdownLock) {
leaseRenewer.clearCurrentlyHeldLeases();
running = false;
}
}
@Override
public void stopLeaseTaker() {
takerFuture.cancel(false);
}
@Override
public void dropLease(final Lease lease) {
synchronized (shutdownLock) {
if (lease != null) {
leaseRenewer.dropLease(lease);
}
}
}
@Override
public boolean isRunning() {
return running;
}
@Override
public boolean updateLease(final Lease lease, final UUID concurrencyToken, final String operation,
final String shardId) throws DependencyException, InvalidStateException, ProvisionedThroughputException {
return leaseRenewer.updateLease(lease, concurrencyToken, operation, shardId);
}
/**
* Returns executor service that should be used for lease renewal.
* @param maximumPoolSize Maximum allowed thread pool size
* @return Executor service that should be used for lease renewal.
*/
private static ExecutorService getLeaseRenewalExecutorService(int maximumPoolSize) {
int coreLeaseCount = Math.max(maximumPoolSize / 4, 2);
return new ThreadPoolExecutor(coreLeaseCount, maximumPoolSize, 60, TimeUnit.SECONDS,
new LinkedTransferQueue<>(), LEASE_RENEWAL_THREAD_FACTORY);
}
@Override
public List<ShardInfo> getCurrentAssignments() {
Collection<Lease> leases = getAssignments();
return convertLeasesToAssignments(leases);
}
private static List<ShardInfo> convertLeasesToAssignments(final Collection<Lease> leases) {
if (leases == null) {
return Collections.emptyList();
}
return leases.stream().map(DynamoDBLeaseCoordinator::convertLeaseToAssignment).collect(Collectors.toList());
}
public static ShardInfo convertLeaseToAssignment(final Lease lease) {
return new ShardInfo(lease.leaseKey(), lease.concurrencyToken().toString(), lease.parentShardIds(),
lease.checkpoint());
}
/**
* {@inheritDoc}
*
* <p>NOTE: This method is deprecated. Please set the initial capacity through the constructor.</p>
*/
@Override
@Deprecated
public DynamoDBLeaseCoordinator initialLeaseTableReadCapacity(long readCapacity) {
if (readCapacity <= 0) {
throw new IllegalArgumentException("readCapacity should be >= 1");
}
initialLeaseTableReadCapacity = readCapacity;
return this;
}
/**
* {@inheritDoc}
*
* <p>NOTE: This method is deprecated. Please set the initial capacity through the constructor.</p>
*/
@Override
@Deprecated
public DynamoDBLeaseCoordinator initialLeaseTableWriteCapacity(long writeCapacity) {
if (writeCapacity <= 0) {
throw new IllegalArgumentException("writeCapacity should be >= 1");
}
initialLeaseTableWriteCapacity = writeCapacity;
return this;
}
}

279
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseManagementFactory.java
View file

@ -1,279 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
import java.util.concurrent.ExecutorService;
import lombok.Data;
import lombok.NonNull;
import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.leases.HierarchicalShardSyncer;
import software.amazon.kinesis.leases.KinesisShardDetector;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseManagementFactory;
import software.amazon.kinesis.leases.ShardDetector;
import software.amazon.kinesis.leases.ShardSyncTaskManager;
import software.amazon.kinesis.metrics.MetricsFactory;
/**
*
*/
@Data
@KinesisClientInternalApi
public class DynamoDBLeaseManagementFactory implements LeaseManagementFactory {
@NonNull
private final KinesisAsyncClient kinesisClient;
@NonNull
private final String streamName;
@NonNull
private final DynamoDbAsyncClient dynamoDBClient;
@NonNull
private final String tableName;
@NonNull
private final String workerIdentifier;
@NonNull
private final ExecutorService executorService;
@NonNull
private final InitialPositionInStreamExtended initialPositionInStream;
@NonNull
private final HierarchicalShardSyncer hierarchicalShardSyncer;
private final long failoverTimeMillis;
private final long epsilonMillis;
private final int maxLeasesForWorker;
private final int maxLeasesToStealAtOneTime;
private final int maxLeaseRenewalThreads;
private final boolean cleanupLeasesUponShardCompletion;
private final boolean ignoreUnexpectedChildShards;
private final long shardSyncIntervalMillis;
private final boolean consistentReads;
private final long listShardsBackoffTimeMillis;
private final int maxListShardsRetryAttempts;
private final int maxCacheMissesBeforeReload;
private final long listShardsCacheAllowedAgeInSeconds;
private final int cacheMissWarningModulus;
private final long initialLeaseTableReadCapacity;
private final long initialLeaseTableWriteCapacity;
private final TableCreatorCallback tableCreatorCallback;
/**
* Constructor.
*
* <p>NOTE: This constructor is deprecated and will be removed in a future release.</p>
*
* @param kinesisClient
* @param streamName
* @param dynamoDBClient
* @param tableName
* @param workerIdentifier
* @param executorService
* @param initialPositionInStream
* @param failoverTimeMillis
* @param epsilonMillis
* @param maxLeasesForWorker
* @param maxLeasesToStealAtOneTime
* @param maxLeaseRenewalThreads
* @param cleanupLeasesUponShardCompletion
* @param ignoreUnexpectedChildShards
* @param shardSyncIntervalMillis
* @param consistentReads
* @param listShardsBackoffTimeMillis
* @param maxListShardsRetryAttempts
* @param maxCacheMissesBeforeReload
* @param listShardsCacheAllowedAgeInSeconds
* @param cacheMissWarningModulus
*/
@Deprecated
public DynamoDBLeaseManagementFactory(final KinesisAsyncClient kinesisClient, final String streamName,
final DynamoDbAsyncClient dynamoDBClient, final String tableName, final String workerIdentifier,
final ExecutorService executorService, final InitialPositionInStreamExtended initialPositionInStream,
final long failoverTimeMillis, final long epsilonMillis, final int maxLeasesForWorker,
final int maxLeasesToStealAtOneTime, final int maxLeaseRenewalThreads,
final boolean cleanupLeasesUponShardCompletion, final boolean ignoreUnexpectedChildShards,
final long shardSyncIntervalMillis, final boolean consistentReads, final long listShardsBackoffTimeMillis,
final int maxListShardsRetryAttempts, final int maxCacheMissesBeforeReload,
final long listShardsCacheAllowedAgeInSeconds, final int cacheMissWarningModulus) {
this(kinesisClient, streamName, dynamoDBClient, tableName, workerIdentifier, executorService,
initialPositionInStream, failoverTimeMillis, epsilonMillis, maxLeasesForWorker,
maxLeasesToStealAtOneTime, maxLeaseRenewalThreads, cleanupLeasesUponShardCompletion,
ignoreUnexpectedChildShards, shardSyncIntervalMillis, consistentReads, listShardsBackoffTimeMillis,
maxListShardsRetryAttempts, maxCacheMissesBeforeReload, listShardsCacheAllowedAgeInSeconds,
cacheMissWarningModulus, TableConstants.DEFAULT_INITIAL_LEASE_TABLE_READ_CAPACITY,
TableConstants.DEFAULT_INITIAL_LEASE_TABLE_WRITE_CAPACITY);
}
/**
* Constructor.
*
* <p>
* NOTE: This constructor is deprecated and will be removed in a future release.
* </p>
*
* @param kinesisClient
* @param streamName
* @param dynamoDBClient
* @param tableName
* @param workerIdentifier
* @param executorService
* @param initialPositionInStream
* @param failoverTimeMillis
* @param epsilonMillis
* @param maxLeasesForWorker
* @param maxLeasesToStealAtOneTime
* @param maxLeaseRenewalThreads
* @param cleanupLeasesUponShardCompletion
* @param ignoreUnexpectedChildShards
* @param shardSyncIntervalMillis
* @param consistentReads
* @param listShardsBackoffTimeMillis
* @param maxListShardsRetryAttempts
* @param maxCacheMissesBeforeReload
* @param listShardsCacheAllowedAgeInSeconds
* @param cacheMissWarningModulus
* @param initialLeaseTableReadCapacity
* @param initialLeaseTableWriteCapacity
*/
@Deprecated
public DynamoDBLeaseManagementFactory(final KinesisAsyncClient kinesisClient, final String streamName,
final DynamoDbAsyncClient dynamoDBClient, final String tableName, final String workerIdentifier,
final ExecutorService executorService, final InitialPositionInStreamExtended initialPositionInStream,
final long failoverTimeMillis, final long epsilonMillis, final int maxLeasesForWorker,
final int maxLeasesToStealAtOneTime, final int maxLeaseRenewalThreads,
final boolean cleanupLeasesUponShardCompletion, final boolean ignoreUnexpectedChildShards,
final long shardSyncIntervalMillis, final boolean consistentReads, final long listShardsBackoffTimeMillis,
final int maxListShardsRetryAttempts, final int maxCacheMissesBeforeReload,
final long listShardsCacheAllowedAgeInSeconds, final int cacheMissWarningModulus,
final long initialLeaseTableReadCapacity, final long initialLeaseTableWriteCapacity) {
this(kinesisClient, streamName, dynamoDBClient, tableName, workerIdentifier, executorService,
initialPositionInStream, failoverTimeMillis, epsilonMillis, maxLeasesForWorker,
maxLeasesToStealAtOneTime, maxLeaseRenewalThreads, cleanupLeasesUponShardCompletion,
ignoreUnexpectedChildShards, shardSyncIntervalMillis, consistentReads, listShardsBackoffTimeMillis,
maxListShardsRetryAttempts, maxCacheMissesBeforeReload, listShardsCacheAllowedAgeInSeconds,
cacheMissWarningModulus, initialLeaseTableReadCapacity, initialLeaseTableWriteCapacity,
new HierarchicalShardSyncer(), TableCreatorCallback.NOOP_TABLE_CREATOR_CALLBACK);
}
/**
* Constructor.
*
* @param kinesisClient
* @param streamName
* @param dynamoDBClient
* @param tableName
* @param workerIdentifier
* @param executorService
* @param initialPositionInStream
* @param failoverTimeMillis
* @param epsilonMillis
* @param maxLeasesForWorker
* @param maxLeasesToStealAtOneTime
* @param maxLeaseRenewalThreads
* @param cleanupLeasesUponShardCompletion
* @param ignoreUnexpectedChildShards
* @param shardSyncIntervalMillis
* @param consistentReads
* @param listShardsBackoffTimeMillis
* @param maxListShardsRetryAttempts
* @param maxCacheMissesBeforeReload
* @param listShardsCacheAllowedAgeInSeconds
* @param cacheMissWarningModulus
* @param initialLeaseTableReadCapacity
* @param initialLeaseTableWriteCapacity
* @param hierarchicalShardSyncer
* @param tableCreatorCallback
*/
public DynamoDBLeaseManagementFactory(final KinesisAsyncClient kinesisClient, final String streamName,
final DynamoDbAsyncClient dynamoDBClient, final String tableName, final String workerIdentifier,
final ExecutorService executorService, final InitialPositionInStreamExtended initialPositionInStream,
final long failoverTimeMillis, final long epsilonMillis, final int maxLeasesForWorker,
final int maxLeasesToStealAtOneTime, final int maxLeaseRenewalThreads,
final boolean cleanupLeasesUponShardCompletion, final boolean ignoreUnexpectedChildShards,
final long shardSyncIntervalMillis, final boolean consistentReads, final long listShardsBackoffTimeMillis,
final int maxListShardsRetryAttempts, final int maxCacheMissesBeforeReload,
final long listShardsCacheAllowedAgeInSeconds, final int cacheMissWarningModulus,
final long initialLeaseTableReadCapacity, final long initialLeaseTableWriteCapacity,
final HierarchicalShardSyncer hierarchicalShardSyncer,
final TableCreatorCallback tableCreatorCallback) {
this.kinesisClient = kinesisClient;
this.streamName = streamName;
this.dynamoDBClient = dynamoDBClient;
this.tableName = tableName;
this.workerIdentifier = workerIdentifier;
this.executorService = executorService;
this.initialPositionInStream = initialPositionInStream;
this.failoverTimeMillis = failoverTimeMillis;
this.epsilonMillis = epsilonMillis;
this.maxLeasesForWorker = maxLeasesForWorker;
this.maxLeasesToStealAtOneTime = maxLeasesToStealAtOneTime;
this.maxLeaseRenewalThreads = maxLeaseRenewalThreads;
this.cleanupLeasesUponShardCompletion = cleanupLeasesUponShardCompletion;
this.ignoreUnexpectedChildShards = ignoreUnexpectedChildShards;
this.shardSyncIntervalMillis = shardSyncIntervalMillis;
this.consistentReads = consistentReads;
this.listShardsBackoffTimeMillis = listShardsBackoffTimeMillis;
this.maxListShardsRetryAttempts = maxListShardsRetryAttempts;
this.maxCacheMissesBeforeReload = maxCacheMissesBeforeReload;
this.listShardsCacheAllowedAgeInSeconds = listShardsCacheAllowedAgeInSeconds;
this.cacheMissWarningModulus = cacheMissWarningModulus;
this.initialLeaseTableReadCapacity = initialLeaseTableReadCapacity;
this.initialLeaseTableWriteCapacity = initialLeaseTableWriteCapacity;
this.hierarchicalShardSyncer = hierarchicalShardSyncer;
this.tableCreatorCallback = tableCreatorCallback;
}
@Override
public LeaseCoordinator createLeaseCoordinator(@NonNull final MetricsFactory metricsFactory) {
return new DynamoDBLeaseCoordinator(this.createLeaseRefresher(),
workerIdentifier,
failoverTimeMillis,
epsilonMillis,
maxLeasesForWorker,
maxLeasesToStealAtOneTime,
maxLeaseRenewalThreads,
initialLeaseTableReadCapacity,
initialLeaseTableWriteCapacity,
metricsFactory);
}
@Override
public ShardSyncTaskManager createShardSyncTaskManager(@NonNull final MetricsFactory metricsFactory) {
return new ShardSyncTaskManager(this.createShardDetector(),
this.createLeaseRefresher(),
initialPositionInStream,
cleanupLeasesUponShardCompletion,
ignoreUnexpectedChildShards,
shardSyncIntervalMillis,
executorService,
hierarchicalShardSyncer,
metricsFactory);
}
@Override
public DynamoDBLeaseRefresher createLeaseRefresher() {
return new DynamoDBLeaseRefresher(tableName, dynamoDBClient, new DynamoDBLeaseSerializer(), consistentReads,
tableCreatorCallback);
}
@Override
public ShardDetector createShardDetector() {
return new KinesisShardDetector(kinesisClient, streamName, listShardsBackoffTimeMillis,
maxListShardsRetryAttempts, listShardsCacheAllowedAgeInSeconds, maxCacheMissesBeforeReload,
cacheMissWarningModulus);
}
}

646
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseRefresher.java
View file

@ -1,646 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.AttributeValueUpdate;
import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
import software.amazon.awssdk.services.dynamodb.model.CreateTableRequest;
import software.amazon.awssdk.services.dynamodb.model.DeleteItemRequest;
import software.amazon.awssdk.services.dynamodb.model.DescribeTableRequest;
import software.amazon.awssdk.services.dynamodb.model.DescribeTableResponse;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import software.amazon.awssdk.services.dynamodb.model.GetItemRequest;
import software.amazon.awssdk.services.dynamodb.model.GetItemResponse;
import software.amazon.awssdk.services.dynamodb.model.LimitExceededException;
import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput;
import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughputExceededException;
import software.amazon.awssdk.services.dynamodb.model.PutItemRequest;
import software.amazon.awssdk.services.dynamodb.model.ResourceInUseException;
import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
import software.amazon.awssdk.services.dynamodb.model.ScanRequest;
import software.amazon.awssdk.services.dynamodb.model.ScanResponse;
import software.amazon.awssdk.services.dynamodb.model.TableStatus;
import software.amazon.awssdk.services.dynamodb.model.UpdateItemRequest;
import software.amazon.awssdk.utils.CollectionUtils;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.LeaseSerializer;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.retrieval.AWSExceptionManager;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* An implementation of {@link LeaseRefresher} that uses DynamoDB.
*/
@Slf4j
@KinesisClientInternalApi
public class DynamoDBLeaseRefresher implements LeaseRefresher {
protected final String table;
protected final DynamoDbAsyncClient dynamoDBClient;
protected final LeaseSerializer serializer;
protected final boolean consistentReads;
private final TableCreatorCallback tableCreatorCallback;
private boolean newTableCreated = false;
/**
* Constructor.
*
* <p>
* NOTE: This constructor is deprecated and will be removed in a future release.
* </p>
*
* @param table
* @param dynamoDBClient
* @param serializer
* @param consistentReads
*/
@Deprecated
public DynamoDBLeaseRefresher(final String table, final DynamoDbAsyncClient dynamoDBClient,
final LeaseSerializer serializer, final boolean consistentReads) {
this(table, dynamoDBClient, serializer, consistentReads, TableCreatorCallback.NOOP_TABLE_CREATOR_CALLBACK);
}
/**
* Constructor.
*
* @param table
* @param dynamoDBClient
* @param serializer
* @param consistentReads
* @param tableCreatorCallback
*/
public DynamoDBLeaseRefresher(final String table, final DynamoDbAsyncClient dynamoDBClient,
final LeaseSerializer serializer, final boolean consistentReads,
@NonNull final TableCreatorCallback tableCreatorCallback) {
this.table = table;
this.dynamoDBClient = dynamoDBClient;
this.serializer = serializer;
this.consistentReads = consistentReads;
this.tableCreatorCallback = tableCreatorCallback;
}
/**
* {@inheritDoc}
*/
@Override
public boolean createLeaseTableIfNotExists(@NonNull final Long readCapacity, @NonNull final Long writeCapacity)
throws ProvisionedThroughputException, DependencyException {
try {
if (tableStatus() != null) {
return newTableCreated;
}
} catch (DependencyException de) {
//
// Something went wrong with DynamoDB
//
log.error("Failed to get table status for {}", table, de);
}
ProvisionedThroughput throughput = ProvisionedThroughput.builder().readCapacityUnits(readCapacity)
.writeCapacityUnits(writeCapacity).build();
CreateTableRequest request = CreateTableRequest.builder().tableName(table).keySchema(serializer.getKeySchema())
.attributeDefinitions(serializer.getAttributeDefinitions()).provisionedThroughput(throughput).build();
final AWSExceptionManager exceptionManager = createExceptionManager();
exceptionManager.add(ResourceInUseException.class, t -> t);
exceptionManager.add(LimitExceededException.class, t -> t);
try {
try {
dynamoDBClient.createTable(request).get();
newTableCreated = true;
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
throw new DependencyException(e);
}
} catch (ResourceInUseException e) {
log.info("Table {} already exists.", table);
return newTableCreated;
} catch (LimitExceededException e) {
throw new ProvisionedThroughputException("Capacity exceeded when creating table " + table, e);
} catch (DynamoDbException e) {
throw new DependencyException(e);
}
return newTableCreated;
}
/**
* {@inheritDoc}
*/
@Override
public boolean leaseTableExists() throws DependencyException {
return TableStatus.ACTIVE == tableStatus();
}
private TableStatus tableStatus() throws DependencyException {
DescribeTableRequest request = DescribeTableRequest.builder().tableName(table).build();
final AWSExceptionManager exceptionManager = createExceptionManager();
exceptionManager.add(ResourceNotFoundException.class, t -> t);
DescribeTableResponse result;
try {
try {
result = dynamoDBClient.describeTable(request).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: Check if this is the correct behavior
throw new DependencyException(e);
}
} catch (ResourceNotFoundException e) {
log.debug("Got ResourceNotFoundException for table {} in leaseTableExists, returning false.", table);
return null;
} catch (DynamoDbException e) {
throw new DependencyException(e);
}
TableStatus tableStatus = result.table().tableStatus();
log.debug("Lease table exists and is in status {}", tableStatus);
return tableStatus;
}
@Override
public boolean waitUntilLeaseTableExists(long secondsBetweenPolls, long timeoutSeconds) throws DependencyException {
long sleepTimeRemaining = TimeUnit.SECONDS.toMillis(timeoutSeconds);
while (!leaseTableExists()) {
if (sleepTimeRemaining <= 0) {
return false;
}
long timeToSleepMillis = Math.min(TimeUnit.SECONDS.toMillis(secondsBetweenPolls), sleepTimeRemaining);
sleepTimeRemaining -= sleep(timeToSleepMillis);
}
if (newTableCreated) {
log.debug("Lease table was recently created, will perform post table creation actions");
performPostTableCreationAction();
}
return true;
}
/**
* Exposed for testing purposes.
*
* @param timeToSleepMillis time to sleep in milliseconds
*
* @return actual time slept in millis
*/
long sleep(long timeToSleepMillis) {
long startTime = System.currentTimeMillis();
try {
Thread.sleep(timeToSleepMillis);
} catch (InterruptedException e) {
log.debug("Interrupted while sleeping");
}
return System.currentTimeMillis() - startTime;
}
/**
* {@inheritDoc}
*/
@Override
public List<Lease> listLeases() throws DependencyException, InvalidStateException, ProvisionedThroughputException {
return list(null);
}
/**
* {@inheritDoc}
*/
@Override
public boolean isLeaseTableEmpty()
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
return list(1).isEmpty();
}
/**
* List with the given page size. Package access for integration testing.
*
* @param limit number of items to consider at a time - used by integration tests to force paging.
* @return list of leases
* @throws InvalidStateException if table does not exist
* @throws DependencyException if DynamoDB scan fail in an unexpected way
* @throws ProvisionedThroughputException if DynamoDB scan fail due to exceeded capacity
*/
List<Lease> list(Integer limit) throws DependencyException, InvalidStateException, ProvisionedThroughputException {
log.debug("Listing leases from table {}", table);
ScanRequest.Builder scanRequestBuilder = ScanRequest.builder().tableName(table);
if (limit != null) {
scanRequestBuilder = scanRequestBuilder.limit(limit);
}
ScanRequest scanRequest = scanRequestBuilder.build();
final AWSExceptionManager exceptionManager = createExceptionManager();
exceptionManager.add(ResourceNotFoundException.class, t -> t);
exceptionManager.add(ProvisionedThroughputExceededException.class, t -> t);
try {
try {
ScanResponse scanResult = dynamoDBClient.scan(scanRequest).get();
List<Lease> result = new ArrayList<>();
while (scanResult != null) {
for (Map<String, AttributeValue> item : scanResult.items()) {
log.debug("Got item {} from DynamoDB.", item.toString());
result.add(serializer.fromDynamoRecord(item));
}
Map<String, AttributeValue> lastEvaluatedKey = scanResult.lastEvaluatedKey();
if (CollectionUtils.isNullOrEmpty(lastEvaluatedKey)) {
// Signify that we're done.
scanResult = null;
log.debug("lastEvaluatedKey was null - scan finished.");
} else {
// Make another request, picking up where we left off.
scanRequest = scanRequest.toBuilder().exclusiveStartKey(lastEvaluatedKey).build();
log.debug("lastEvaluatedKey was {}, continuing scan.", lastEvaluatedKey);
scanResult = dynamoDBClient.scan(scanRequest).get();
}
}
log.debug("Listed {} leases from table {}", result.size(), table);
return result;
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: Check if this is the correct behavior
throw new DependencyException(e);
}
} catch (ResourceNotFoundException e) {
throw new InvalidStateException("Cannot scan lease table " + table + " because it does not exist.", e);
} catch (ProvisionedThroughputExceededException e) {
throw new ProvisionedThroughputException(e);
} catch (DynamoDbException e) {
throw new DependencyException(e);
}
}
/**
* {@inheritDoc}
*/
@Override
public boolean createLeaseIfNotExists(@NonNull final Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
log.debug("Creating lease {}", lease);
PutItemRequest request = PutItemRequest.builder().tableName(table).item(serializer.toDynamoRecord(lease))
.expected(serializer.getDynamoNonexistantExpectation()).build();
final AWSExceptionManager exceptionManager = createExceptionManager();
exceptionManager.add(ConditionalCheckFailedException.class, t -> t);
try {
try {
dynamoDBClient.putItem(request).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: Check if this is the correct behavior
throw new DependencyException(e);
}
} catch (ConditionalCheckFailedException e) {
log.debug("Did not create lease {} because it already existed", lease);
return false;
} catch (DynamoDbException e) {
throw convertAndRethrowExceptions("create", lease.leaseKey(), e);
}
return true;
}
/**
* {@inheritDoc}
*/
@Override
public Lease getLease(@NonNull final String leaseKey)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
log.debug("Getting lease with key {}", leaseKey);
GetItemRequest request = GetItemRequest.builder().tableName(table).key(serializer.getDynamoHashKey(leaseKey))
.consistentRead(consistentReads).build();
final AWSExceptionManager exceptionManager = createExceptionManager();
try {
try {
GetItemResponse result = dynamoDBClient.getItem(request).get();
Map<String, AttributeValue> dynamoRecord = result.item();
if (CollectionUtils.isNullOrEmpty(dynamoRecord)) {
log.debug("No lease found with key {}, returning null.", leaseKey);
return null;
} else {
final Lease lease = serializer.fromDynamoRecord(dynamoRecord);
log.debug("Got lease {}", lease);
return lease;
}
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: check behavior
throw new DependencyException(e);
}
} catch (DynamoDbException e) {
throw convertAndRethrowExceptions("get", leaseKey, e);
}
}
/**
* {@inheritDoc}
*/
@Override
public boolean renewLease(@NonNull final Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
log.debug("Renewing lease with key {}", lease.leaseKey());
UpdateItemRequest request = UpdateItemRequest.builder().tableName(table).key(serializer.getDynamoHashKey(lease))
.expected(serializer.getDynamoLeaseCounterExpectation(lease))
.attributeUpdates(serializer.getDynamoLeaseCounterUpdate(lease)).build();
final AWSExceptionManager exceptionManager = createExceptionManager();
exceptionManager.add(ConditionalCheckFailedException.class, t -> t);
try {
try {
dynamoDBClient.updateItem(request).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: Check if this is correct behavior
throw new DependencyException(e);
}
} catch (ConditionalCheckFailedException e) {
log.debug("Lease renewal failed for lease with key {} because the lease counter was not {}",
lease.leaseKey(), lease.leaseCounter());
// If we had a spurious retry during the Dynamo update, then this conditional PUT failure
// might be incorrect. So, we get the item straight away and check if the lease owner + lease
// counter are what we expected.
String expectedOwner = lease.leaseOwner();
Long expectedCounter = lease.leaseCounter() + 1;
final Lease updatedLease = getLease(lease.leaseKey());
if (updatedLease == null || !expectedOwner.equals(updatedLease.leaseOwner())
|| !expectedCounter.equals(updatedLease.leaseCounter())) {
return false;
}
log.info("Detected spurious renewal failure for lease with key {}, but recovered", lease.leaseKey());
} catch (DynamoDbException e) {
throw new DependencyException(e);
}
lease.leaseCounter(lease.leaseCounter() + 1);
return true;
}
/**
* {@inheritDoc}
*/
@Override
public boolean takeLease(@NonNull final Lease lease, @NonNull final String owner)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
final String oldOwner = lease.leaseOwner();
log.debug("Taking lease with leaseKey {} from {} to {}", lease.leaseKey(),
lease.leaseOwner() == null ? "nobody" : lease.leaseOwner(), owner);
final AWSExceptionManager exceptionManager = createExceptionManager();
exceptionManager.add(ConditionalCheckFailedException.class, t -> t);
Map<String, AttributeValueUpdate> updates = serializer.getDynamoLeaseCounterUpdate(lease);
updates.putAll(serializer.getDynamoTakeLeaseUpdate(lease, owner));
UpdateItemRequest request = UpdateItemRequest.builder().tableName(table).key(serializer.getDynamoHashKey(lease))
.expected(serializer.getDynamoLeaseCounterExpectation(lease)).attributeUpdates(updates).build();
try {
try {
dynamoDBClient.updateItem(request).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: Check behavior
throw new DependencyException(e);
}
} catch (ConditionalCheckFailedException e) {
log.debug("Lease renewal failed for lease with key {} because the lease counter was not {}",
lease.leaseKey(), lease.leaseCounter());
return false;
} catch (DynamoDbException e) {
throw convertAndRethrowExceptions("take", lease.leaseKey(), e);
}
lease.leaseCounter(lease.leaseCounter() + 1);
lease.leaseOwner(owner);
if (oldOwner != null && !oldOwner.equals(owner)) {
lease.ownerSwitchesSinceCheckpoint(lease.ownerSwitchesSinceCheckpoint() + 1);
}
return true;
}
/**
* {@inheritDoc}
*/
@Override
public boolean evictLease(@NonNull final Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
log.debug("Evicting lease with leaseKey {} owned by {}", lease.leaseKey(), lease.leaseOwner());
final AWSExceptionManager exceptionManager = createExceptionManager();
exceptionManager.add(ConditionalCheckFailedException.class, t -> t);
Map<String, AttributeValueUpdate> updates = serializer.getDynamoLeaseCounterUpdate(lease);
updates.putAll(serializer.getDynamoEvictLeaseUpdate(lease));
UpdateItemRequest request = UpdateItemRequest.builder().tableName(table).key(serializer.getDynamoHashKey(lease))
.expected(serializer.getDynamoLeaseOwnerExpectation(lease)).attributeUpdates(updates).build();
try {
try {
dynamoDBClient.updateItem(request).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: check behavior
throw new DependencyException(e);
}
} catch (ConditionalCheckFailedException e) {
log.debug("Lease eviction failed for lease with key {} because the lease owner was not {}",
lease.leaseKey(), lease.leaseOwner());
return false;
} catch (DynamoDbException e) {
throw convertAndRethrowExceptions("evict", lease.leaseKey(), e);
}
lease.leaseOwner(null);
lease.leaseCounter(lease.leaseCounter() + 1);
return true;
}
/**
* {@inheritDoc}
*/
public void deleteAll() throws DependencyException, InvalidStateException, ProvisionedThroughputException {
List<Lease> allLeases = listLeases();
log.warn("Deleting {} items from table {}", allLeases.size(), table);
final AWSExceptionManager exceptionManager = createExceptionManager();
for (final Lease lease : allLeases) {
DeleteItemRequest deleteRequest = DeleteItemRequest.builder().tableName(table)
.key(serializer.getDynamoHashKey(lease)).build();
try {
try {
dynamoDBClient.deleteItem(deleteRequest).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: check the behavior
throw new DependencyException(e);
}
} catch (DynamoDbException e) {
throw convertAndRethrowExceptions("deleteAll", lease.leaseKey(), e);
}
}
}
/**
* {@inheritDoc}
*/
@Override
public void deleteLease(@NonNull final Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
log.debug("Deleting lease with leaseKey {}", lease.leaseKey());
DeleteItemRequest deleteRequest = DeleteItemRequest.builder().tableName(table)
.key(serializer.getDynamoHashKey(lease)).build();
final AWSExceptionManager exceptionManager = createExceptionManager();
try {
try {
dynamoDBClient.deleteItem(deleteRequest).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: Check if this is the correct behavior
throw new DependencyException(e);
}
} catch (DynamoDbException e) {
throw convertAndRethrowExceptions("delete", lease.leaseKey(), e);
}
}
/**
* {@inheritDoc}
*/
@Override
public boolean updateLease(@NonNull final Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
log.debug("Updating lease {}", lease);
final AWSExceptionManager exceptionManager = createExceptionManager();
exceptionManager.add(ConditionalCheckFailedException.class, t -> t);
Map<String, AttributeValueUpdate> updates = serializer.getDynamoLeaseCounterUpdate(lease);
updates.putAll(serializer.getDynamoUpdateLeaseUpdate(lease));
UpdateItemRequest request = UpdateItemRequest.builder().tableName(table).key(serializer.getDynamoHashKey(lease))
.expected(serializer.getDynamoLeaseCounterExpectation(lease)).attributeUpdates(updates).build();
try {
try {
dynamoDBClient.updateItem(request).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
throw new DependencyException(e);
}
} catch (ConditionalCheckFailedException e) {
log.debug("Lease update failed for lease with key {} because the lease counter was not {}",
lease.leaseKey(), lease.leaseCounter());
return false;
} catch (DynamoDbException e) {
throw convertAndRethrowExceptions("update", lease.leaseKey(), e);
}
lease.leaseCounter(lease.leaseCounter() + 1);
return true;
}
/**
* {@inheritDoc}
*/
@Override
public ExtendedSequenceNumber getCheckpoint(String shardId)
throws ProvisionedThroughputException, InvalidStateException, DependencyException {
ExtendedSequenceNumber checkpoint = null;
Lease lease = getLease(shardId);
if (lease != null) {
checkpoint = lease.checkpoint();
}
return checkpoint;
}
/*
* This method contains boilerplate exception handling - it throws or returns something to be thrown. The
* inconsistency there exists to satisfy the compiler when this method is used at the end of non-void methods.
*/
protected DependencyException convertAndRethrowExceptions(String operation, String leaseKey, Exception e)
throws ProvisionedThroughputException, InvalidStateException {
if (e instanceof ProvisionedThroughputExceededException) {
log.warn("Provisioned Throughput on the lease table has been exceeded. It's recommended that you increase"
+ " the IOPs on the table. Failure to increase the IOPs may cause the application to not make"
+ " progress.");
throw new ProvisionedThroughputException(e);
} else if (e instanceof ResourceNotFoundException) {
throw new InvalidStateException(
String.format("Cannot %s lease with key %s because table %s does not exist.",
operation, leaseKey, table),
e);
} else {
return new DependencyException(e);
}
}
private AWSExceptionManager createExceptionManager() {
final AWSExceptionManager exceptionManager = new AWSExceptionManager();
exceptionManager.add(DynamoDbException.class, t -> t);
return exceptionManager;
}
void performPostTableCreationAction() {
tableCreatorCallback.performAction(
TableCreatorCallbackInput.builder().dynamoDbClient(dynamoDBClient).tableName(table).build());
}
}

421
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseRenewer.java
View file

@ -1,421 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentNavigableMap;
import java.util.concurrent.ConcurrentSkipListMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import org.apache.commons.lang3.StringUtils;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.cloudwatch.model.StandardUnit;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.LeaseRenewer;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsLevel;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsUtil;
/**
* An implementation of {@link LeaseRenewer} that uses DynamoDB via {@link LeaseRefresher}.
*/
@Slf4j
@KinesisClientInternalApi
public class DynamoDBLeaseRenewer implements LeaseRenewer {
private static final int RENEWAL_RETRIES = 2;
private static final String RENEW_ALL_LEASES_DIMENSION = "RenewAllLeases";
private final LeaseRefresher leaseRefresher;
private final String workerIdentifier;
private final long leaseDurationNanos;
private final ExecutorService executorService;
private final MetricsFactory metricsFactory;
private final ConcurrentNavigableMap<String, Lease> ownedLeases = new ConcurrentSkipListMap<>();
/**
* Constructor.
*
* @param leaseRefresher
* LeaseRefresher to use
* @param workerIdentifier
* identifier of this worker
* @param leaseDurationMillis
* duration of a lease in milliseconds
* @param executorService
* ExecutorService to use for renewing leases in parallel
*/
public DynamoDBLeaseRenewer(final LeaseRefresher leaseRefresher, final String workerIdentifier,
final long leaseDurationMillis, final ExecutorService executorService,
final MetricsFactory metricsFactory) {
this.leaseRefresher = leaseRefresher;
this.workerIdentifier = workerIdentifier;
this.leaseDurationNanos = TimeUnit.MILLISECONDS.toNanos(leaseDurationMillis);
this.executorService = executorService;
this.metricsFactory = metricsFactory;
}
/**
* {@inheritDoc}
*/
@Override
public void renewLeases() throws DependencyException, InvalidStateException {
// Due to the eventually consistent nature of ConcurrentNavigableMap iterators, this log entry may become
// inaccurate during iteration.
log.debug("Worker {} holding {} leases: {}", workerIdentifier, ownedLeases.size(), ownedLeases);
/*
* Lease renewals are done in parallel so many leases can be renewed for short lease fail over time
* configuration. In this case, metrics scope is also shared across different threads, so scope must be thread
* safe.
*/
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, RENEW_ALL_LEASES_DIMENSION);
long startTime = System.currentTimeMillis();
boolean success = false;
try {
/*
* We iterate in descending order here so that the synchronized(lease) inside renewLease doesn't "lead" calls
* to getCurrentlyHeldLeases. They'll still cross paths, but they won't interleave their executions.
*/
int lostLeases = 0;
List<Future<Boolean>> renewLeaseTasks = new ArrayList<>();
for (Lease lease : ownedLeases.descendingMap().values()) {
renewLeaseTasks.add(executorService.submit(new RenewLeaseTask(lease)));
}
int leasesInUnknownState = 0;
Exception lastException = null;
for (Future<Boolean> renewLeaseTask : renewLeaseTasks) {
try {
if (!renewLeaseTask.get()) {
lostLeases++;
}
} catch (InterruptedException e) {
log.info("Interrupted while waiting for a lease to renew.");
leasesInUnknownState += 1;
Thread.currentThread().interrupt();
} catch (ExecutionException e) {
log.error("Encountered an exception while renewing a lease.", e.getCause());
leasesInUnknownState += 1;
lastException = e;
}
}
scope.addData("LostLeases", lostLeases, StandardUnit.COUNT, MetricsLevel.SUMMARY);
scope.addData("CurrentLeases", ownedLeases.size(), StandardUnit.COUNT, MetricsLevel.SUMMARY);
if (leasesInUnknownState > 0) {
throw new DependencyException(
String.format("Encountered an exception while renewing leases. The number"
+ " of leases which might not have been renewed is %d", leasesInUnknownState),
lastException);
}
success = true;
} finally {
MetricsUtil.addWorkerIdentifier(scope, workerIdentifier);
MetricsUtil.addSuccessAndLatency(scope, success, startTime, MetricsLevel.SUMMARY);
MetricsUtil.endScope(scope);
}
}
@RequiredArgsConstructor
private class RenewLeaseTask implements Callable<Boolean> {
private final Lease lease;
@Override
public Boolean call() throws Exception {
return renewLease(lease);
}
}
private boolean renewLease(Lease lease) throws DependencyException, InvalidStateException {
return renewLease(lease, false);
}
private boolean renewLease(Lease lease, boolean renewEvenIfExpired) throws DependencyException, InvalidStateException {
String leaseKey = lease.leaseKey();
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, RENEW_ALL_LEASES_DIMENSION);
boolean success = false;
boolean renewedLease = false;
long startTime = System.currentTimeMillis();
try {
for (int i = 1; i <= RENEWAL_RETRIES; i++) {
try {
synchronized (lease) {
// Don't renew expired lease during regular renewals. getCopyOfHeldLease may have returned null
// triggering the application processing to treat this as a lost lease (fail checkpoint with
// ShutdownException).
boolean isLeaseExpired = lease.isExpired(leaseDurationNanos, System.nanoTime());
if (renewEvenIfExpired || !isLeaseExpired) {
renewedLease = leaseRefresher.renewLease(lease);
}
if (renewedLease) {
lease.lastCounterIncrementNanos(System.nanoTime());
}
}
if (renewedLease) {
if (log.isDebugEnabled()) {
log.debug("Worker {} successfully renewed lease with key {}", workerIdentifier, leaseKey);
}
} else {
log.info("Worker {} lost lease with key {}", workerIdentifier, leaseKey);
ownedLeases.remove(leaseKey);
}
success = true;
break;
} catch (ProvisionedThroughputException e) {
log.info("Worker {} could not renew lease with key {} on try {} out of {} due to capacity",
workerIdentifier, leaseKey, i, RENEWAL_RETRIES);
}
}
} finally {
MetricsUtil.addWorkerIdentifier(scope, workerIdentifier);
MetricsUtil.addSuccessAndLatency(scope, "RenewLease", success, startTime, MetricsLevel.DETAILED);
MetricsUtil.endScope(scope);
}
return renewedLease;
}
/**
* {@inheritDoc}
*/
@Override
public Map<String, Lease> getCurrentlyHeldLeases() {
Map<String, Lease> result = new HashMap<>();
long now = System.nanoTime();
for (String leaseKey : ownedLeases.keySet()) {
Lease copy = getCopyOfHeldLease(leaseKey, now);
if (copy != null) {
result.put(copy.leaseKey(), copy);
}
}
return result;
}
/**
* {@inheritDoc}
*/
@Override
public Lease getCurrentlyHeldLease(String leaseKey) {
return getCopyOfHeldLease(leaseKey, System.nanoTime());
}
/**
* Internal method to return a lease with a specific lease key only if we currently hold it.
*
* @param leaseKey key of lease to return
* @param now current timestamp for old-ness checking
* @return non-authoritative copy of the held lease, or null if we don't currently hold it
*/
private Lease getCopyOfHeldLease(String leaseKey, long now) {
Lease authoritativeLease = ownedLeases.get(leaseKey);
if (authoritativeLease == null) {
return null;
} else {
Lease copy = null;
synchronized (authoritativeLease) {
copy = authoritativeLease.copy();
}
if (copy.isExpired(leaseDurationNanos, now)) {
log.info("getCurrentlyHeldLease not returning lease with key {} because it is expired",
copy.leaseKey());
return null;
} else {
return copy;
}
}
}
/**
* {@inheritDoc}
*/
@Override
public boolean updateLease(Lease lease, UUID concurrencyToken, @NonNull String operation, String shardId)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
verifyNotNull(lease, "lease cannot be null");
verifyNotNull(lease.leaseKey(), "leaseKey cannot be null");
verifyNotNull(concurrencyToken, "concurrencyToken cannot be null");
String leaseKey = lease.leaseKey();
Lease authoritativeLease = ownedLeases.get(leaseKey);
if (authoritativeLease == null) {
log.info("Worker {} could not update lease with key {} because it does not hold it", workerIdentifier,
leaseKey);
return false;
}
/*
* If the passed-in concurrency token doesn't match the concurrency token of the authoritative lease, it means
* the lease was lost and regained between when the caller acquired his concurrency token and when the caller
* called update.
*/
if (!authoritativeLease.concurrencyToken().equals(concurrencyToken)) {
log.info("Worker {} refusing to update lease with key {} because concurrency tokens don't match",
workerIdentifier, leaseKey);
return false;
}
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, operation);
if (StringUtils.isNotEmpty(shardId)) {
MetricsUtil.addShardId(scope, shardId);
}
long startTime = System.currentTimeMillis();
boolean success = false;
try {
synchronized (authoritativeLease) {
authoritativeLease.update(lease);
boolean updatedLease = leaseRefresher.updateLease(authoritativeLease);
if (updatedLease) {
// Updates increment the counter
authoritativeLease.lastCounterIncrementNanos(System.nanoTime());
} else {
/*
* If updateLease returns false, it means someone took the lease from us. Remove the lease
* from our set of owned leases pro-actively rather than waiting for a run of renewLeases().
*/
log.info("Worker {} lost lease with key {} - discovered during update", workerIdentifier, leaseKey);
/*
* Remove only if the value currently in the map is the same as the authoritative lease. We're
* guarding against a pause after the concurrency token check above. It plays out like so:
*
* 1) Concurrency token check passes
* 2) Pause. Lose lease, re-acquire lease. This requires at least one lease counter update.
* 3) Unpause. leaseRefresher.updateLease fails conditional write due to counter updates, returns
* false.
* 4) ownedLeases.remove(key, value) doesn't do anything because authoritativeLease does not
* .equals() the re-acquired version in the map on the basis of lease counter. This is what we want.
* If we just used ownedLease.remove(key), we would have pro-actively removed a lease incorrectly.
*
* Note that there is a subtlety here - Lease.equals() deliberately does not check the concurrency
* token, but it does check the lease counter, so this scheme works.
*/
ownedLeases.remove(leaseKey, authoritativeLease);
}
success = true;
return updatedLease;
}
} finally {
MetricsUtil.addSuccessAndLatency(scope, "UpdateLease", success, startTime, MetricsLevel.DETAILED);
MetricsUtil.endScope(scope);
}
}
/**
* {@inheritDoc}
*/
@Override
public void addLeasesToRenew(Collection<Lease> newLeases) {
verifyNotNull(newLeases, "newLeases cannot be null");
for (Lease lease : newLeases) {
if (lease.lastCounterIncrementNanos() == null) {
log.info("addLeasesToRenew ignoring lease with key {} because it does not have lastRenewalNanos set",
lease.leaseKey());
continue;
}
Lease authoritativeLease = lease.copy();
/*
* Assign a concurrency token when we add this to the set of currently owned leases. This ensures that
* every time we acquire a lease, it gets a new concurrency token.
*/
authoritativeLease.concurrencyToken(UUID.randomUUID());
ownedLeases.put(authoritativeLease.leaseKey(), authoritativeLease);
}
}
/**
* {@inheritDoc}
*/
@Override
public void clearCurrentlyHeldLeases() {
ownedLeases.clear();
}
/**
* {@inheritDoc}
* @param lease the lease to drop.
*/
@Override
public void dropLease(Lease lease) {
ownedLeases.remove(lease.leaseKey());
}
/**
* {@inheritDoc}
*/
@Override
public void initialize() throws DependencyException, InvalidStateException, ProvisionedThroughputException {
Collection<Lease> leases = leaseRefresher.listLeases();
List<Lease> myLeases = new LinkedList<>();
boolean renewEvenIfExpired = true;
for (Lease lease : leases) {
if (workerIdentifier.equals(lease.leaseOwner())) {
log.info(" Worker {} found lease {}", workerIdentifier, lease);
// Okay to renew even if lease is expired, because we start with an empty list and we add the lease to
// our list only after a successful renew. So we don't need to worry about the edge case where we could
// continue renewing a lease after signaling a lease loss to the application.
if (renewLease(lease, renewEvenIfExpired)) {
myLeases.add(lease);
}
} else {
log.debug("Worker {} ignoring lease {} ", workerIdentifier, lease);
}
}
addLeasesToRenew(myLeases);
}
private void verifyNotNull(Object object, String message) {
if (object == null) {
throw new IllegalArgumentException(message);
}
}
}

238
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseSerializer.java
View file

@ -1,238 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.google.common.base.Strings;
import software.amazon.awssdk.services.dynamodb.model.AttributeAction;
import software.amazon.awssdk.services.dynamodb.model.AttributeDefinition;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.AttributeValueUpdate;
import software.amazon.awssdk.services.dynamodb.model.ExpectedAttributeValue;
import software.amazon.awssdk.services.dynamodb.model.KeySchemaElement;
import software.amazon.awssdk.services.dynamodb.model.KeyType;
import software.amazon.awssdk.services.dynamodb.model.ScalarAttributeType;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.leases.DynamoUtils;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseSerializer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* An implementation of ILeaseSerializer for basic Lease objects. Can also instantiate subclasses of Lease so that
* LeaseSerializer can be decorated by other classes if you need to add fields to leases.
*/
@KinesisClientInternalApi
public class DynamoDBLeaseSerializer implements LeaseSerializer {
private static final String LEASE_KEY_KEY = "leaseKey";
private static final String LEASE_OWNER_KEY = "leaseOwner";
private static final String LEASE_COUNTER_KEY = "leaseCounter";
private static final String OWNER_SWITCHES_KEY = "ownerSwitchesSinceCheckpoint";
private static final String CHECKPOINT_SEQUENCE_NUMBER_KEY = "checkpoint";
private static final String CHECKPOINT_SUBSEQUENCE_NUMBER_KEY = "checkpointSubSequenceNumber";
private static final String PENDING_CHECKPOINT_SEQUENCE_KEY = "pendingCheckpoint";
private static final String PENDING_CHECKPOINT_SUBSEQUENCE_KEY = "pendingCheckpointSubSequenceNumber";
private static final String PARENT_SHARD_ID_KEY = "parentShardId";
@Override
public Map<String, AttributeValue> toDynamoRecord(final Lease lease) {
Map<String, AttributeValue> result = new HashMap<>();
result.put(LEASE_KEY_KEY, DynamoUtils.createAttributeValue(lease.leaseKey()));
result.put(LEASE_COUNTER_KEY, DynamoUtils.createAttributeValue(lease.leaseCounter()));
if (lease.leaseOwner() != null) {
result.put(LEASE_OWNER_KEY, DynamoUtils.createAttributeValue(lease.leaseOwner()));
}
result.put(OWNER_SWITCHES_KEY, DynamoUtils.createAttributeValue(lease.ownerSwitchesSinceCheckpoint()));
result.put(CHECKPOINT_SEQUENCE_NUMBER_KEY, DynamoUtils.createAttributeValue(lease.checkpoint().sequenceNumber()));
result.put(CHECKPOINT_SUBSEQUENCE_NUMBER_KEY, DynamoUtils.createAttributeValue(lease.checkpoint().subSequenceNumber()));
if (lease.parentShardIds() != null && !lease.parentShardIds().isEmpty()) {
result.put(PARENT_SHARD_ID_KEY, DynamoUtils.createAttributeValue(lease.parentShardIds()));
}
if (lease.pendingCheckpoint() != null && !lease.pendingCheckpoint().sequenceNumber().isEmpty()) {
result.put(PENDING_CHECKPOINT_SEQUENCE_KEY, DynamoUtils.createAttributeValue(lease.pendingCheckpoint().sequenceNumber()));
result.put(PENDING_CHECKPOINT_SUBSEQUENCE_KEY, DynamoUtils.createAttributeValue(lease.pendingCheckpoint().subSequenceNumber()));
}
return result;
}
@Override
public Lease fromDynamoRecord(final Map<String, AttributeValue> dynamoRecord) {
Lease result = new Lease();
result.leaseKey(DynamoUtils.safeGetString(dynamoRecord, LEASE_KEY_KEY));
result.leaseOwner(DynamoUtils.safeGetString(dynamoRecord, LEASE_OWNER_KEY));
result.leaseCounter(DynamoUtils.safeGetLong(dynamoRecord, LEASE_COUNTER_KEY));
result.ownerSwitchesSinceCheckpoint(DynamoUtils.safeGetLong(dynamoRecord, OWNER_SWITCHES_KEY));
result.checkpoint(
new ExtendedSequenceNumber(
DynamoUtils.safeGetString(dynamoRecord, CHECKPOINT_SEQUENCE_NUMBER_KEY),
DynamoUtils.safeGetLong(dynamoRecord, CHECKPOINT_SUBSEQUENCE_NUMBER_KEY))
);
result.parentShardIds(DynamoUtils.safeGetSS(dynamoRecord, PARENT_SHARD_ID_KEY));
if (!Strings.isNullOrEmpty(DynamoUtils.safeGetString(dynamoRecord, PENDING_CHECKPOINT_SEQUENCE_KEY))) {
result.pendingCheckpoint(
new ExtendedSequenceNumber(
DynamoUtils.safeGetString(dynamoRecord, PENDING_CHECKPOINT_SEQUENCE_KEY),
DynamoUtils.safeGetLong(dynamoRecord, PENDING_CHECKPOINT_SUBSEQUENCE_KEY))
);
}
return result;
}
@Override
public Map<String, AttributeValue> getDynamoHashKey(final String leaseKey) {
Map<String, AttributeValue> result = new HashMap<>();
result.put(LEASE_KEY_KEY, DynamoUtils.createAttributeValue(leaseKey));
return result;
}
@Override
public Map<String, AttributeValue> getDynamoHashKey(final Lease lease) {
return getDynamoHashKey(lease.leaseKey());
}
@Override
public Map<String, ExpectedAttributeValue> getDynamoLeaseCounterExpectation(final Lease lease) {
return getDynamoLeaseCounterExpectation(lease.leaseCounter());
}
public Map<String, ExpectedAttributeValue> getDynamoLeaseCounterExpectation(final Long leaseCounter) {
Map<String, ExpectedAttributeValue> result = new HashMap<>();
ExpectedAttributeValue eav = ExpectedAttributeValue.builder().value(DynamoUtils.createAttributeValue(leaseCounter)).build();
result.put(LEASE_COUNTER_KEY, eav);
return result;
}
@Override
public Map<String, ExpectedAttributeValue> getDynamoLeaseOwnerExpectation(final Lease lease) {
Map<String, ExpectedAttributeValue> result = new HashMap<>();
ExpectedAttributeValue.Builder eavBuilder = ExpectedAttributeValue.builder();
if (lease.leaseOwner() == null) {
eavBuilder = eavBuilder.exists(false);
} else {
eavBuilder = eavBuilder.value(DynamoUtils.createAttributeValue(lease.leaseOwner()));
}
result.put(LEASE_OWNER_KEY, eavBuilder.build());
return result;
}
@Override
public Map<String, ExpectedAttributeValue> getDynamoNonexistantExpectation() {
Map<String, ExpectedAttributeValue> result = new HashMap<>();
ExpectedAttributeValue expectedAV = ExpectedAttributeValue.builder().exists(false).build();
result.put(LEASE_KEY_KEY, expectedAV);
return result;
}
@Override
public Map<String, AttributeValueUpdate> getDynamoLeaseCounterUpdate(final Lease lease) {
return getDynamoLeaseCounterUpdate(lease.leaseCounter());
}
public Map<String, AttributeValueUpdate> getDynamoLeaseCounterUpdate(Long leaseCounter) {
Map<String, AttributeValueUpdate> result = new HashMap<>();
AttributeValueUpdate avu =
AttributeValueUpdate.builder().value(DynamoUtils.createAttributeValue(leaseCounter + 1)).action(AttributeAction.PUT).build();
result.put(LEASE_COUNTER_KEY, avu);
return result;
}
@Override
public Map<String, AttributeValueUpdate> getDynamoTakeLeaseUpdate(final Lease lease, String owner) {
Map<String, AttributeValueUpdate> result = new HashMap<>();
result.put(LEASE_OWNER_KEY, AttributeValueUpdate.builder().value(DynamoUtils.createAttributeValue(owner)).action(AttributeAction.PUT).build());
String oldOwner = lease.leaseOwner();
if (oldOwner != null && !oldOwner.equals(owner)) {
result.put(OWNER_SWITCHES_KEY, AttributeValueUpdate.builder().value(DynamoUtils.createAttributeValue(1L)).action(AttributeAction.ADD).build());
}
return result;
}
@Override
public Map<String, AttributeValueUpdate> getDynamoEvictLeaseUpdate(final Lease lease) {
Map<String, AttributeValueUpdate> result = new HashMap<>();
AttributeValue value = null;
result.put(LEASE_OWNER_KEY, AttributeValueUpdate.builder().value(value).action(AttributeAction.DELETE).build());
return result;
}
private AttributeValueUpdate putUpdate(AttributeValue attributeValue) {
return AttributeValueUpdate.builder().value(attributeValue).action(AttributeAction.PUT).build();
}
@Override
public Map<String, AttributeValueUpdate> getDynamoUpdateLeaseUpdate(final Lease lease) {
Map<String, AttributeValueUpdate> result = new HashMap<>();
result.put(CHECKPOINT_SEQUENCE_NUMBER_KEY, putUpdate(DynamoUtils.createAttributeValue(lease.checkpoint().sequenceNumber())));
result.put(CHECKPOINT_SUBSEQUENCE_NUMBER_KEY, putUpdate(DynamoUtils.createAttributeValue(lease.checkpoint().subSequenceNumber())));
result.put(OWNER_SWITCHES_KEY, putUpdate(DynamoUtils.createAttributeValue(lease.ownerSwitchesSinceCheckpoint())));
if (lease.pendingCheckpoint() != null && !lease.pendingCheckpoint().sequenceNumber().isEmpty()) {
result.put(PENDING_CHECKPOINT_SEQUENCE_KEY, putUpdate(DynamoUtils.createAttributeValue(lease.pendingCheckpoint().sequenceNumber())));
result.put(PENDING_CHECKPOINT_SUBSEQUENCE_KEY, putUpdate(DynamoUtils.createAttributeValue(lease.pendingCheckpoint().subSequenceNumber())));
} else {
result.put(PENDING_CHECKPOINT_SEQUENCE_KEY, AttributeValueUpdate.builder().action(AttributeAction.DELETE).build());
result.put(PENDING_CHECKPOINT_SUBSEQUENCE_KEY, AttributeValueUpdate.builder().action(AttributeAction.DELETE).build());
}
return result;
}
@Override
public Collection<KeySchemaElement> getKeySchema() {
List<KeySchemaElement> keySchema = new ArrayList<>();
keySchema.add(KeySchemaElement.builder().attributeName(LEASE_KEY_KEY).keyType(KeyType.HASH).build());
return keySchema;
}
@Override
public Collection<AttributeDefinition> getAttributeDefinitions() {
List<AttributeDefinition> definitions = new ArrayList<>();
definitions.add(AttributeDefinition.builder().attributeName(LEASE_KEY_KEY)
.attributeType(ScalarAttributeType.S).build());
return definitions;
}
}

541
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java
View file

@ -1,541 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.cloudwatch.model.StandardUnit;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.LeaseTaker;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsLevel;
import software.amazon.kinesis.metrics.MetricsUtil;
/**
* An implementation of {@link LeaseTaker} that uses DynamoDB via {@link LeaseRefresher}.
*/
@Slf4j
@KinesisClientInternalApi
public class DynamoDBLeaseTaker implements LeaseTaker {
private static final int TAKE_RETRIES = 3;
private static final int SCAN_RETRIES = 1;
// See note on TAKE_LEASES_DIMENSION(Callable) for why we have this callable.
private static final Callable<Long> SYSTEM_CLOCK_CALLABLE = System::nanoTime;
private static final String TAKE_LEASES_DIMENSION = "TakeLeases";
private final LeaseRefresher leaseRefresher;
private final String workerIdentifier;
private final long leaseDurationNanos;
private final MetricsFactory metricsFactory;
private final Map<String, Lease> allLeases = new HashMap<>();
// TODO: Remove these defaults and use the defaults in the config
private int maxLeasesForWorker = Integer.MAX_VALUE;
private int maxLeasesToStealAtOneTime = 1;
private long lastScanTimeNanos = 0L;
public DynamoDBLeaseTaker(LeaseRefresher leaseRefresher, String workerIdentifier, long leaseDurationMillis,
final MetricsFactory metricsFactory) {
this.leaseRefresher = leaseRefresher;
this.workerIdentifier = workerIdentifier;
this.leaseDurationNanos = TimeUnit.MILLISECONDS.toNanos(leaseDurationMillis);
this.metricsFactory = metricsFactory;
}
/**
* Worker will not acquire more than the specified max number of leases even if there are more
* shards that need to be processed. This can be used in scenarios where a worker is resource constrained or
* to prevent lease thrashing when small number of workers pick up all leases for small amount of time during
* deployment.
* Note that setting a low value may cause data loss (e.g. if there aren't enough Workers to make progress on all
* shards). When setting the value for this property, one must ensure enough workers are present to process
* shards and should consider future resharding, child shards that may be blocked on parent shards, some workers
* becoming unhealthy, etc.
*
* @param maxLeasesForWorker Max leases this Worker can handle at a time
* @return LeaseTaker
*/
public DynamoDBLeaseTaker withMaxLeasesForWorker(int maxLeasesForWorker) {
if (maxLeasesForWorker <= 0) {
throw new IllegalArgumentException("maxLeasesForWorker should be >= 1");
}
this.maxLeasesForWorker = maxLeasesForWorker;
return this;
}
/**
* Max leases to steal from a more loaded Worker at one time (for load balancing).
* Setting this to a higher number can allow for faster load convergence (e.g. during deployments, cold starts),
* but can cause higher churn in the system.
*
* @param maxLeasesToStealAtOneTime Steal up to this many leases at one time (for load balancing)
* @return LeaseTaker
*/
public DynamoDBLeaseTaker withMaxLeasesToStealAtOneTime(int maxLeasesToStealAtOneTime) {
if (maxLeasesToStealAtOneTime <= 0) {
throw new IllegalArgumentException("maxLeasesToStealAtOneTime should be >= 1");
}
this.maxLeasesToStealAtOneTime = maxLeasesToStealAtOneTime;
return this;
}
/**
* {@inheritDoc}
*/
@Override
public Map<String, Lease> takeLeases() throws DependencyException, InvalidStateException {
return takeLeases(SYSTEM_CLOCK_CALLABLE);
}
/**
* Internal implementation of TAKE_LEASES_DIMENSION. Takes a callable that can provide the time to enable test cases
* without Thread.sleep. Takes a callable instead of a raw time value because the time needs to be computed as-of
* immediately after the scan.
*
* @param timeProvider
* Callable that will supply the time
*
* @return map of lease key to taken lease
*
* @throws DependencyException
* @throws InvalidStateException
*/
synchronized Map<String, Lease> takeLeases(Callable<Long> timeProvider)
throws DependencyException, InvalidStateException {
// Key is leaseKey
Map<String, Lease> takenLeases = new HashMap<>();
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, TAKE_LEASES_DIMENSION);
long startTime = System.currentTimeMillis();
boolean success = false;
ProvisionedThroughputException lastException = null;
try {
try {
for (int i = 1; i <= SCAN_RETRIES; i++) {
try {
updateAllLeases(timeProvider);
success = true;
} catch (ProvisionedThroughputException e) {
log.info("Worker {} could not find expired leases on try {} out of {}", workerIdentifier, i,
TAKE_RETRIES);
lastException = e;
}
}
} finally {
MetricsUtil.addWorkerIdentifier(scope, workerIdentifier);
MetricsUtil.addSuccessAndLatency(scope, "ListLeases", success, startTime, MetricsLevel.DETAILED);
}
if (lastException != null) {
log.error("Worker {} could not scan leases table, aborting TAKE_LEASES_DIMENSION. Exception caught by"
+ " last retry:", workerIdentifier, lastException);
return takenLeases;
}
List<Lease> expiredLeases = getExpiredLeases();
Set<Lease> leasesToTake = computeLeasesToTake(expiredLeases);
Set<String> untakenLeaseKeys = new HashSet<>();
for (Lease lease : leasesToTake) {
String leaseKey = lease.leaseKey();
startTime = System.currentTimeMillis();
success = false;
try {
for (int i = 1; i <= TAKE_RETRIES; i++) {
try {
if (leaseRefresher.takeLease(lease, workerIdentifier)) {
lease.lastCounterIncrementNanos(System.nanoTime());
takenLeases.put(leaseKey, lease);
} else {
untakenLeaseKeys.add(leaseKey);
}
success = true;
break;
} catch (ProvisionedThroughputException e) {
log.info("Could not take lease with key {} for worker {} on try {} out of {} due to"
+ " capacity", leaseKey, workerIdentifier, i, TAKE_RETRIES);
}
}
} finally {
MetricsUtil.addSuccessAndLatency(scope, "TakeLease", success, startTime, MetricsLevel.DETAILED);
}
}
if (takenLeases.size() > 0) {
log.info("Worker {} successfully took {} leases: {}", workerIdentifier, takenLeases.size(),
stringJoin(takenLeases.keySet(), ", "));
}
if (untakenLeaseKeys.size() > 0) {
log.info("Worker {} failed to take {} leases: {}", workerIdentifier, untakenLeaseKeys.size(),
stringJoin(untakenLeaseKeys, ", "));
}
scope.addData("TakenLeases", takenLeases.size(), StandardUnit.COUNT, MetricsLevel.SUMMARY);
} finally {
MetricsUtil.endScope(scope);
}
return takenLeases;
}
/** Package access for testing purposes.
*
* @param strings
* @param delimiter
* @return Joined string.
*/
static String stringJoin(Collection<String> strings, String delimiter) {
StringBuilder builder = new StringBuilder();
boolean needDelimiter = false;
for (String string : strings) {
if (needDelimiter) {
builder.append(delimiter);
}
builder.append(string);
needDelimiter = true;
}
return builder.toString();
}
/**
* Scan all leases and update lastRenewalTime. Add new leases and delete old leases.
*
* @param timeProvider callable that supplies the current time
*
* @return list of expired leases, possibly empty, never null.
*
* @throws ProvisionedThroughputException if listLeases fails due to lack of provisioned throughput
* @throws InvalidStateException if the lease table does not exist
* @throws DependencyException if listLeases fails in an unexpected way
*/
private void updateAllLeases(Callable<Long> timeProvider)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
List<Lease> freshList = leaseRefresher.listLeases();
try {
lastScanTimeNanos = timeProvider.call();
} catch (Exception e) {
throw new DependencyException("Exception caught from timeProvider", e);
}
// This set will hold the lease keys not updated by the previous listLeases call.
Set<String> notUpdated = new HashSet<>(allLeases.keySet());
// Iterate over all leases, finding ones to try to acquire that haven't changed since the last iteration
for (Lease lease : freshList) {
String leaseKey = lease.leaseKey();
Lease oldLease = allLeases.get(leaseKey);
allLeases.put(leaseKey, lease);
notUpdated.remove(leaseKey);
if (oldLease != null) {
// If we've seen this lease before...
if (oldLease.leaseCounter().equals(lease.leaseCounter())) {
// ...and the counter hasn't changed, propagate the lastRenewalNanos time from the old lease
lease.lastCounterIncrementNanos(oldLease.lastCounterIncrementNanos());
} else {
// ...and the counter has changed, set lastRenewalNanos to the time of the scan.
lease.lastCounterIncrementNanos(lastScanTimeNanos);
}
} else {
if (lease.leaseOwner() == null) {
// if this new lease is unowned, it's never been renewed.
lease.lastCounterIncrementNanos(0L);
if (log.isDebugEnabled()) {
log.debug("Treating new lease with key {} as never renewed because it is new and unowned.",
leaseKey);
}
} else {
// if this new lease is owned, treat it as renewed as of the scan
lease.lastCounterIncrementNanos(lastScanTimeNanos);
if (log.isDebugEnabled()) {
log.debug("Treating new lease with key {} as recently renewed because it is new and owned.",
leaseKey);
}
}
}
}
// Remove dead leases from allLeases
for (String key : notUpdated) {
allLeases.remove(key);
}
}
/**
* @return list of leases that were expired as of our last scan.
*/
private List<Lease> getExpiredLeases() {
List<Lease> expiredLeases = new ArrayList<>();
for (Lease lease : allLeases.values()) {
if (lease.isExpired(leaseDurationNanos, lastScanTimeNanos)) {
expiredLeases.add(lease);
}
}
return expiredLeases;
}
/**
* Compute the number of leases I should try to take based on the state of the system.
*
* @param expiredLeases list of leases we determined to be expired
* @return set of leases to take.
*/
private Set<Lease> computeLeasesToTake(List<Lease> expiredLeases) {
Map<String, Integer> leaseCounts = computeLeaseCounts(expiredLeases);
Set<Lease> leasesToTake = new HashSet<>();
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, TAKE_LEASES_DIMENSION);
MetricsUtil.addWorkerIdentifier(scope, workerIdentifier);
try {
int numLeases = allLeases.size();
int numWorkers = leaseCounts.size();
if (numLeases == 0) {
// If there are no leases, I shouldn't try to take any.
return leasesToTake;
}
int target;
if (numWorkers >= numLeases) {
// If we have n leases and n or more workers, each worker can have up to 1 lease, including myself.
target = 1;
} else {
/*
* numWorkers must be < numLeases.
*
* Our target for each worker is numLeases / numWorkers (+1 if numWorkers doesn't evenly divide numLeases)
*/
target = numLeases / numWorkers + (numLeases % numWorkers == 0 ? 0 : 1);
// Spill over is the number of leases this worker should have claimed, but did not because it would
// exceed the max allowed for this worker.
int leaseSpillover = Math.max(0, target - maxLeasesForWorker);
if (target > maxLeasesForWorker) {
log.warn(
"Worker {} target is {} leases and maxLeasesForWorker is {}. Resetting target to {},"
+ " lease spillover is {}. Note that some shards may not be processed if no other "
+ "workers are able to pick them up.",
workerIdentifier, target, maxLeasesForWorker, maxLeasesForWorker, leaseSpillover);
target = maxLeasesForWorker;
}
scope.addData("LeaseSpillover", leaseSpillover, StandardUnit.COUNT, MetricsLevel.SUMMARY);
}
int myCount = leaseCounts.get(workerIdentifier);
int numLeasesToReachTarget = target - myCount;
if (numLeasesToReachTarget <= 0) {
// If we don't need anything, return the empty set.
return leasesToTake;
}
// Shuffle expiredLeases so workers don't all try to contend for the same leases.
Collections.shuffle(expiredLeases);
int originalExpiredLeasesSize = expiredLeases.size();
if (expiredLeases.size() > 0) {
// If we have expired leases, get up to <needed> leases from expiredLeases
for (; numLeasesToReachTarget > 0 && expiredLeases.size() > 0; numLeasesToReachTarget--) {
leasesToTake.add(expiredLeases.remove(0));
}
} else {
// If there are no expired leases and we need a lease, consider stealing.
List<Lease> leasesToSteal = chooseLeasesToSteal(leaseCounts, numLeasesToReachTarget, target);
for (Lease leaseToSteal : leasesToSteal) {
log.info("Worker {} needed {} leases but none were expired, so it will steal lease {} from {}",
workerIdentifier, numLeasesToReachTarget, leaseToSteal.leaseKey(),
leaseToSteal.leaseOwner());
leasesToTake.add(leaseToSteal);
}
}
if (!leasesToTake.isEmpty()) {
log.info(
"Worker {} saw {} total leases, {} available leases, {} "
+ "workers. Target is {} leases, I have {} leases, I will take {} leases",
workerIdentifier, numLeases, originalExpiredLeasesSize, numWorkers, target, myCount,
leasesToTake.size());
}
scope.addData("TotalLeases", numLeases, StandardUnit.COUNT, MetricsLevel.DETAILED);
scope.addData("ExpiredLeases", originalExpiredLeasesSize, StandardUnit.COUNT, MetricsLevel.SUMMARY);
scope.addData("NumWorkers", numWorkers, StandardUnit.COUNT, MetricsLevel.SUMMARY);
scope.addData("NeededLeases", numLeasesToReachTarget, StandardUnit.COUNT, MetricsLevel.DETAILED);
scope.addData("LeasesToTake", leasesToTake.size(), StandardUnit.COUNT, MetricsLevel.DETAILED);
} finally {
MetricsUtil.endScope(scope);
}
return leasesToTake;
}
/**
* Choose leases to steal by randomly selecting one or more (up to max) from the most loaded worker.
* Stealing rules:
*
* Steal up to maxLeasesToStealAtOneTime leases from the most loaded worker if
* a) he has > target leases and I need >= 1 leases : steal min(leases needed, maxLeasesToStealAtOneTime)
* b) he has == target leases and I need > 1 leases : steal 1
*
* @param leaseCounts map of workerIdentifier to lease count
* @param needed # of leases needed to reach the target leases for the worker
* @param target target # of leases per worker
* @return Leases to steal, or empty list if we should not steal
*/
private List<Lease> chooseLeasesToSteal(Map<String, Integer> leaseCounts, int needed, int target) {
List<Lease> leasesToSteal = new ArrayList<>();
Entry<String, Integer> mostLoadedWorker = null;
// Find the most loaded worker
for (Entry<String, Integer> worker : leaseCounts.entrySet()) {
if (mostLoadedWorker == null || mostLoadedWorker.getValue() < worker.getValue()) {
mostLoadedWorker = worker;
}
}
int numLeasesToSteal = 0;
if ((mostLoadedWorker.getValue() >= target) && (needed > 0)) {
int leasesOverTarget = mostLoadedWorker.getValue() - target;
numLeasesToSteal = Math.min(needed, leasesOverTarget);
// steal 1 if we need > 1 and max loaded worker has target leases.
if ((needed > 1) && (numLeasesToSteal == 0)) {
numLeasesToSteal = 1;
}
numLeasesToSteal = Math.min(numLeasesToSteal, maxLeasesToStealAtOneTime);
}
if (numLeasesToSteal <= 0) {
if (log.isDebugEnabled()) {
log.debug(String.format("Worker %s not stealing from most loaded worker %s. He has %d,"
+ " target is %d, and I need %d",
workerIdentifier,
mostLoadedWorker.getKey(),
mostLoadedWorker.getValue(),
target,
needed));
}
return leasesToSteal;
} else {
if (log.isDebugEnabled()) {
log.debug("Worker {} will attempt to steal {} leases from most loaded worker {}. "
+ " He has {} leases, target is {}, I need {}, maxLeasesToSteatAtOneTime is {}.",
workerIdentifier,
numLeasesToSteal,
mostLoadedWorker.getKey(),
mostLoadedWorker.getValue(),
target,
needed,
maxLeasesToStealAtOneTime);
}
}
String mostLoadedWorkerIdentifier = mostLoadedWorker.getKey();
List<Lease> candidates = new ArrayList<>();
// Collect leases belonging to that worker
for (Lease lease : allLeases.values()) {
if (mostLoadedWorkerIdentifier.equals(lease.leaseOwner())) {
candidates.add(lease);
}
}
// Return random ones
Collections.shuffle(candidates);
int toIndex = Math.min(candidates.size(), numLeasesToSteal);
leasesToSteal.addAll(candidates.subList(0, toIndex));
return leasesToSteal;
}
/**
* Count leases by host. Always includes myself, but otherwise only includes hosts that are currently holding
* leases.
*
* @param expiredLeases list of leases that are currently expired
* @return map of workerIdentifier to lease count
*/
private Map<String, Integer> computeLeaseCounts(List<Lease> expiredLeases) {
Map<String, Integer> leaseCounts = new HashMap<>();
// Compute the number of leases per worker by looking through allLeases and ignoring leases that have expired.
for (Lease lease : allLeases.values()) {
if (!expiredLeases.contains(lease)) {
String leaseOwner = lease.leaseOwner();
Integer oldCount = leaseCounts.get(leaseOwner);
if (oldCount == null) {
leaseCounts.put(leaseOwner, 1);
} else {
leaseCounts.put(leaseOwner, oldCount + 1);
}
}
}
// If I have no leases, I wasn't represented in leaseCounts. Let's fix that.
Integer myCount = leaseCounts.get(workerIdentifier);
if (myCount == null) {
myCount = 0;
leaseCounts.put(workerIdentifier, myCount);
}
return leaseCounts;
}
/**
* {@inheritDoc}
*/
@Override
public String getWorkerIdentifier() {
return workerIdentifier;
}
/**
* {@inheritDoc}
*/
@Override
public synchronized List<Lease> allLeases() {
return new ArrayList<>(allLeases.values());
}
}

29
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/TableConstants.java
View file

@ -1,29 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
import lombok.AccessLevel;
import lombok.NoArgsConstructor;
/**
* This class is just a holder for initial lease table IOPs units. This class will be removed in a future release.
*/
@Deprecated
@NoArgsConstructor(access = AccessLevel.PRIVATE)
public class TableConstants {
public static final long DEFAULT_INITIAL_LEASE_TABLE_READ_CAPACITY = 10L;
public static final long DEFAULT_INITIAL_LEASE_TABLE_WRITE_CAPACITY = 10L;
}

39
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/TableCreatorCallback.java
View file

@ -1,39 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
/**
* Callback interface for interacting with the DynamoDB lease table post creation.
*/
@FunctionalInterface
public interface TableCreatorCallback {
/**
* NoOp implemetation for TableCreatorCallback
*/
TableCreatorCallback NOOP_TABLE_CREATOR_CALLBACK = (TableCreatorCallbackInput tableCreatorCallbackInput) -> {
// Do nothing
};
/**
* Actions needed to be performed on the DynamoDB lease table once the table has been created and is in the ACTIVE
* status. Will not be called if the table previously exists.
*
* @param tableCreatorCallbackInput
* Input object for table creator
*/
void performAction(TableCreatorCallbackInput tableCreatorCallbackInput);
}

39
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/TableCreatorCallbackInput.java
View file

@ -1,39 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.dynamodb;
import lombok.Builder;
import lombok.Data;
import lombok.EqualsAndHashCode;
import lombok.NonNull;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
/**
*
*/
@Builder(toBuilder = true)
@Data
@Accessors(fluent = true)
@ToString
@EqualsAndHashCode
public class TableCreatorCallbackInput {
@NonNull
private final DynamoDbAsyncClient dynamoDbClient;
@NonNull
private final String tableName;
}

34
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/exceptions/DependencyException.java
View file

@ -1,34 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.exceptions;
/**
* Indicates that a lease operation has failed because a dependency of the leasing system has failed. This will happen
* if DynamoDB throws an InternalServerException or a generic AmazonClientException (the specific subclasses of
* AmazonClientException are all handled more gracefully).
*/
public class DependencyException extends LeasingException {
private static final long serialVersionUID = 1L;
public DependencyException(Throwable e) {
super(e);
}
public DependencyException(String message, Throwable e) {
super(message, e);
}
}

37
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/exceptions/InvalidStateException.java
View file

@ -1,37 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.exceptions;
/**
* Indicates that a lease operation has failed because DynamoDB is an invalid state. The most common example is failing
* to create the DynamoDB table before doing any lease operations.
*/
public class InvalidStateException extends LeasingException {
private static final long serialVersionUID = 1L;
public InvalidStateException(Throwable e) {
super(e);
}
public InvalidStateException(String message, Throwable e) {
super(message, e);
}
public InvalidStateException(String message) {
super(message);
}
}

36
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/exceptions/LeasingException.java
View file

@ -1,36 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.exceptions;
/**
* Top-level exception type for all exceptions thrown by the leasing code.
*/
public class LeasingException extends Exception {
public LeasingException(Throwable e) {
super(e);
}
public LeasingException(String message, Throwable e) {
super(message, e);
}
public LeasingException(String message) {
super(message);
}
private static final long serialVersionUID = 1L;
}

32
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/exceptions/ProvisionedThroughputException.java
View file

@ -1,32 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases.exceptions;
/**
* Indicates that a lease operation has failed due to lack of provisioned throughput for a DynamoDB table.
*/
public class ProvisionedThroughputException extends LeasingException {
private static final long serialVersionUID = 1L;
public ProvisionedThroughputException(Throwable e) {
super(e);
}
public ProvisionedThroughputException(String message, Throwable e) {
super(message, e);
}
}

46
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/exceptions/ShardSyncer.java
View file

@ -1,46 +0,0 @@
package software.amazon.kinesis.leases.exceptions;
import lombok.NonNull;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.exceptions.internal.KinesisClientLibIOException;
import software.amazon.kinesis.leases.HierarchicalShardSyncer;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.ShardDetector;
import software.amazon.kinesis.metrics.MetricsScope;
/**
* Helper class to sync leases with shards of the Kinesis stream.
* It will create new leases/activities when it discovers new Kinesis shards (bootstrap/resharding).
* It deletes leases for shards that have been trimmed from Kinesis, or if we've completed processing it
* and begun processing it's child shards.
*
* <p>NOTE: This class is deprecated and will be removed in a future release.</p>
*/
@Deprecated
public class ShardSyncer {
private static final HierarchicalShardSyncer HIERARCHICAL_SHARD_SYNCER = new HierarchicalShardSyncer();
/**
* <p>NOTE: This method is deprecated and will be removed in a future release.</p>
*
* @param shardDetector
* @param leaseRefresher
* @param initialPosition
* @param cleanupLeasesOfCompletedShards
* @param ignoreUnexpectedChildShards
* @param scope
* @throws DependencyException
* @throws InvalidStateException
* @throws ProvisionedThroughputException
* @throws KinesisClientLibIOException
*/
@Deprecated
public static synchronized void checkAndCreateLeasesForNewShards(@NonNull final ShardDetector shardDetector,
final LeaseRefresher leaseRefresher, final InitialPositionInStreamExtended initialPosition,
final boolean cleanupLeasesOfCompletedShards, final boolean ignoreUnexpectedChildShards,
final MetricsScope scope) throws DependencyException, InvalidStateException, ProvisionedThroughputException,
KinesisClientLibIOException {
HIERARCHICAL_SHARD_SYNCER.checkAndCreateLeaseForNewShards(shardDetector, leaseRefresher, initialPosition,
cleanupLeasesOfCompletedShards, ignoreUnexpectedChildShards, scope);
}
}

104
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/BlockOnParentShardTask.java
View file

@ -1,104 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import lombok.AccessLevel;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.exceptions.internal.BlockedOnParentShardException;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Task to block until processing of all data records in the parent shard(s) is completed.
* We check if we have checkpoint(s) for the parent shard(s).
* If a checkpoint for a parent shard is found, we poll and wait until the checkpoint value is SHARD_END
* (application has checkpointed after processing all records in the shard).
* If we don't find a checkpoint for the parent shard(s), we assume they have been trimmed and directly
* proceed with processing data from the shard.
*/
@RequiredArgsConstructor(access = AccessLevel.PACKAGE)
@Slf4j
@KinesisClientInternalApi
// TODO: Check for non null values
public class BlockOnParentShardTask implements ConsumerTask {
@NonNull
private final ShardInfo shardInfo;
private final LeaseRefresher leaseRefresher;
// Sleep for this duration if the parent shards have not completed processing, or we encounter an exception.
private final long parentShardPollIntervalMillis;
private final TaskType taskType = TaskType.BLOCK_ON_PARENT_SHARDS;
/*
* (non-Javadoc)
*
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#call()
*/
@Override
public TaskResult call() {
Exception exception = null;
try {
boolean blockedOnParentShard = false;
for (String shardId : shardInfo.parentShardIds()) {
Lease lease = leaseRefresher.getLease(shardId);
if (lease != null) {
ExtendedSequenceNumber checkpoint = lease.checkpoint();
if ((checkpoint == null) || (!checkpoint.equals(ExtendedSequenceNumber.SHARD_END))) {
log.debug("Shard {} is not yet done. Its current checkpoint is {}", shardId, checkpoint);
blockedOnParentShard = true;
exception = new BlockedOnParentShardException("Parent shard not yet done");
break;
} else {
log.debug("Shard {} has been completely processed.", shardId);
}
} else {
log.info("No lease found for shard {}. Not blocking on completion of this shard.", shardId);
}
}
if (!blockedOnParentShard) {
log.info("No need to block on parents {} of shard {}", shardInfo.parentShardIds(), shardInfo.shardId());
return new TaskResult(null);
}
} catch (Exception e) {
log.error("Caught exception when checking for parent shard checkpoint", e);
exception = e;
}
try {
Thread.sleep(parentShardPollIntervalMillis);
} catch (InterruptedException e) {
log.error("Sleep interrupted when waiting on parent shard(s) of {}", shardInfo.shardId(), e);
}
return new TaskResult(exception);
}
/*
* (non-Javadoc)
*
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#taskType()
*/
@Override
public TaskType taskType() {
return taskType;
}
}

108
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ConsumerState.java
View file

@ -1,108 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
/**
* Represents a the current state of the consumer. This handles the creation of tasks for the consumer, and what to
* do when a transition occurs.
*
*/
interface ConsumerState {
/**
* Creates a new task for this state using the passed in consumer to build the task. If there is no task
* required for this state it may return a null value. {@link ConsumerState}'s are allowed to modify the
* consumer during the execution of this method.
*
* @param consumerArgument
* configuration specific to the task being created
* @param consumer
* the consumer to use build the task, or execute state.
* @param input
* the process input received, this may be null if it's a control message
* @return a valid task for this state or null if there is no task required.
*/
ConsumerTask createTask(ShardConsumerArgument consumerArgument, ShardConsumer consumer, ProcessRecordsInput input);
/**
* Provides the next state of the consumer upon success of the task return by
* {@link ConsumerState#createTask(ShardConsumerArgument, ShardConsumer, ProcessRecordsInput)}.
*
* @return the next state that the consumer should transition to, this may be the same object as the current
* state.
*/
ConsumerState successTransition();
/**
* Provides the next state of the consumer if the task failed. This defaults to no state change.
*
* @return the state to change to upon a task failure
*/
default ConsumerState failureTransition() {
return this;
}
/**
* Provides the next state of the consumer when a shutdown has been requested. The returned state is dependent
* on the current state, and the shutdown reason.
*
* @param shutdownReason
* the reason that a shutdown was requested
* @return the next state that the consumer should transition to, this may be the same object as the current
* state.
*/
ConsumerState shutdownTransition(ShutdownReason shutdownReason);
/**
* The type of task that {@link ConsumerState#createTask(ShardConsumerArgument, ShardConsumer, ProcessRecordsInput)}
* would return. This is always a valid state
* even if createTask would return a null value.
*
* @return the type of task that this state represents.
*/
TaskType taskType();
/**
* An enumeration represent the type of this state. Different consumer states may return the same
* {@link ConsumerStates.ShardConsumerState}.
*
* @return the type of consumer state this represents.
*/
ConsumerStates.ShardConsumerState state();
boolean isTerminal();
/**
* Whether this state requires data to be available before the task can be created
*
* @return true if the task requires data to be available before creation, false otherwise
*/
default boolean requiresDataAvailability() {
return false;
}
/**
* Indicates whether a state requires an external event to re-awaken for processing.
*
* @return true if the state is some external event to restart processing, false if events can be immediately
* dispatched.
*/
default boolean requiresAwake() {
return false;
}
}

139
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/InitializeTask.java
View file

@ -1,139 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.checkpoint.Checkpoint;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsLevel;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsUtil;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.retrieval.RecordsPublisher;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Task for initializing shard position and invoking the ShardRecordProcessor initialize() API.
*/
@RequiredArgsConstructor
@Slf4j
@KinesisClientInternalApi
public class InitializeTask implements ConsumerTask {
private static final String INITIALIZE_TASK_OPERATION = "InitializeTask";
private static final String RECORD_PROCESSOR_INITIALIZE_METRIC = "RecordProcessor.initialize";
@NonNull
private final ShardInfo shardInfo;
@NonNull
private final ShardRecordProcessor shardRecordProcessor;
@NonNull
private final Checkpointer checkpoint;
@NonNull
private final ShardRecordProcessorCheckpointer recordProcessorCheckpointer;
@NonNull
private final InitialPositionInStreamExtended initialPositionInStream;
@NonNull
private final RecordsPublisher cache;
// Back off for this interval if we encounter a problem (exception)
private final long backoffTimeMillis;
@NonNull
private final MetricsFactory metricsFactory;
private final TaskType taskType = TaskType.INITIALIZE;
/*
* Initializes the data fetcher (position in shard) and invokes the ShardRecordProcessor initialize() API.
* (non-Javadoc)
*
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#call()
*/
@Override
public TaskResult call() {
boolean applicationException = false;
Exception exception = null;
try {
log.debug("Initializing ShardId {}", shardInfo);
Checkpoint initialCheckpointObject = checkpoint.getCheckpointObject(shardInfo.shardId());
ExtendedSequenceNumber initialCheckpoint = initialCheckpointObject.checkpoint();
log.debug("[{}]: Checkpoint: {} -- Initial Position: {}", shardInfo.shardId(), initialCheckpoint,
initialPositionInStream);
cache.start(initialCheckpoint, initialPositionInStream);
recordProcessorCheckpointer.largestPermittedCheckpointValue(initialCheckpoint);
recordProcessorCheckpointer.setInitialCheckpointValue(initialCheckpoint);
log.debug("Calling the record processor initialize().");
final InitializationInput initializationInput = InitializationInput.builder()
.shardId(shardInfo.shardId())
.extendedSequenceNumber(initialCheckpoint)
.pendingCheckpointSequenceNumber(initialCheckpointObject.pendingCheckpoint())
.build();
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory,
INITIALIZE_TASK_OPERATION);
final long startTime = System.currentTimeMillis();
try {
shardRecordProcessor.initialize(initializationInput);
log.debug("Record processor initialize() completed.");
} catch (Exception e) {
applicationException = true;
throw e;
} finally {
MetricsUtil.addLatency(scope, RECORD_PROCESSOR_INITIALIZE_METRIC, startTime, MetricsLevel.SUMMARY);
MetricsUtil.endScope(scope);
}
return new TaskResult(null);
} catch (Exception e) {
if (applicationException) {
log.error("Application initialize() threw exception: ", e);
} else {
log.error("Caught exception: ", e);
}
exception = e;
// backoff if we encounter an exception.
try {
Thread.sleep(this.backoffTimeMillis);
} catch (InterruptedException ie) {
log.debug("Interrupted sleep", ie);
}
}
return new TaskResult(exception);
}
/*
* (non-Javadoc)
*
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#taskType()
*/
@Override
public TaskType taskType() {
return taskType;
}
}

55
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/LifecycleConfig.java
View file

@ -1,55 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import java.util.Optional;
import lombok.Data;
import lombok.experimental.Accessors;
import software.amazon.kinesis.retrieval.AggregatorUtil;
/**
* Used by the KCL to configure the lifecycle.
*/
@Data
@Accessors(fluent = true)
public class LifecycleConfig {
/**
* Logs warn message if as task is held in a task for more than the set time.
*
* <p>Default value: {@link Optional#empty()}</p>
*/
private Optional<Long> logWarningForTaskAfterMillis = Optional.empty();
/**
* Backoff time in milliseconds for Amazon Kinesis Client Library tasks (in the event of failures).
*
* <p>Default value: 500L</p>
*/
private long taskBackoffTimeMillis = 500L;
/**
* AggregatorUtil is responsible for deaggregating KPL records.
*/
private AggregatorUtil aggregatorUtil = new AggregatorUtil();
/**
* TaskExecutionListener to be used to handle events during task execution lifecycle for a shard.
*
* <p>Default value: {@link NoOpTaskExecutionListener}</p>
*/
private TaskExecutionListener taskExecutionListener = new NoOpTaskExecutionListener();
}

31
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/NoOpTaskExecutionListener.java
View file

@ -1,31 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import software.amazon.kinesis.lifecycle.events.TaskExecutionListenerInput;
/**
* NoOp implementation of {@link TaskExecutionListener} interface that takes no action on task execution.
*/
public class NoOpTaskExecutionListener implements TaskExecutionListener {
@Override
public void beforeTaskExecution(TaskExecutionListenerInput input) {
}
@Override
public void afterTaskExecution(TaskExecutionListenerInput input) {
}
}

292
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ProcessTask.java
View file

@ -1,292 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import java.util.List;
import java.util.ListIterator;
import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.cloudwatch.model.StandardUnit;
import software.amazon.awssdk.services.kinesis.model.Shard;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.leases.ShardDetector;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsLevel;
import software.amazon.kinesis.metrics.MetricsUtil;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.retrieval.AggregatorUtil;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
import software.amazon.kinesis.retrieval.ThrottlingReporter;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Task for fetching data records and invoking processRecords() on the record processor instance.
*/
@Slf4j
@KinesisClientInternalApi
public class ProcessTask implements ConsumerTask {
private static final String PROCESS_TASK_OPERATION = "ProcessTask";
private static final String DATA_BYTES_PROCESSED_METRIC = "DataBytesProcessed";
private static final String RECORDS_PROCESSED_METRIC = "RecordsProcessed";
private static final String RECORD_PROCESSOR_PROCESS_RECORDS_METRIC = "RecordProcessor.processRecords";
private static final String MILLIS_BEHIND_LATEST_METRIC = "MillisBehindLatest";
private final ShardInfo shardInfo;
private final ShardRecordProcessor shardRecordProcessor;
private final ShardRecordProcessorCheckpointer recordProcessorCheckpointer;
private final TaskType taskType = TaskType.PROCESS;
private final long backoffTimeMillis;
private final Shard shard;
private final ThrottlingReporter throttlingReporter;
private final boolean shouldCallProcessRecordsEvenForEmptyRecordList;
private final long idleTimeInMilliseconds;
private final ProcessRecordsInput processRecordsInput;
private final MetricsFactory metricsFactory;
private final AggregatorUtil aggregatorUtil;
public ProcessTask(@NonNull ShardInfo shardInfo,
@NonNull ShardRecordProcessor shardRecordProcessor,
@NonNull ShardRecordProcessorCheckpointer recordProcessorCheckpointer,
long backoffTimeMillis,
boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
ShardDetector shardDetector,
@NonNull ThrottlingReporter throttlingReporter,
ProcessRecordsInput processRecordsInput,
boolean shouldCallProcessRecordsEvenForEmptyRecordList,
long idleTimeInMilliseconds,
@NonNull AggregatorUtil aggregatorUtil,
@NonNull MetricsFactory metricsFactory) {
this.shardInfo = shardInfo;
this.shardRecordProcessor = shardRecordProcessor;
this.recordProcessorCheckpointer = recordProcessorCheckpointer;
this.backoffTimeMillis = backoffTimeMillis;
this.throttlingReporter = throttlingReporter;
this.processRecordsInput = processRecordsInput;
this.shouldCallProcessRecordsEvenForEmptyRecordList = shouldCallProcessRecordsEvenForEmptyRecordList;
this.idleTimeInMilliseconds = idleTimeInMilliseconds;
this.metricsFactory = metricsFactory;
if (!skipShardSyncAtWorkerInitializationIfLeasesExist) {
this.shard = shardDetector.shard(shardInfo.shardId());
} else {
this.shard = null;
}
if (this.shard == null && !skipShardSyncAtWorkerInitializationIfLeasesExist) {
log.warn("Cannot get the shard for this ProcessTask, so duplicate KPL user records "
+ "in the event of resharding will not be dropped during deaggregation of Amazon "
+ "Kinesis records.");
}
this.aggregatorUtil = aggregatorUtil;
this.recordProcessorCheckpointer.checkpointer().operation(PROCESS_TASK_OPERATION);
}
/*
* (non-Javadoc)
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#call()
*/
@Override
public TaskResult call() {
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, PROCESS_TASK_OPERATION);
MetricsUtil.addShardId(scope, shardInfo.shardId());
long startTimeMillis = System.currentTimeMillis();
boolean success = false;
try {
scope.addData(RECORDS_PROCESSED_METRIC, 0, StandardUnit.COUNT, MetricsLevel.SUMMARY);
scope.addData(DATA_BYTES_PROCESSED_METRIC, 0, StandardUnit.BYTES, MetricsLevel.SUMMARY);
Exception exception = null;
try {
if (processRecordsInput.millisBehindLatest() != null) {
scope.addData(MILLIS_BEHIND_LATEST_METRIC, processRecordsInput.millisBehindLatest(),
StandardUnit.MILLISECONDS, MetricsLevel.SUMMARY);
}
if (processRecordsInput.isAtShardEnd() && processRecordsInput.records().isEmpty()) {
log.info("Reached end of shard {} and have no records to process", shardInfo.shardId());
return new TaskResult(null, true);
}
throttlingReporter.success();
List<KinesisClientRecord> records = deaggregateAnyKplRecords(processRecordsInput.records());
if (!records.isEmpty()) {
scope.addData(RECORDS_PROCESSED_METRIC, records.size(), StandardUnit.COUNT, MetricsLevel.SUMMARY);
}
recordProcessorCheckpointer.largestPermittedCheckpointValue(filterAndGetMaxExtendedSequenceNumber(
scope, records, recordProcessorCheckpointer.lastCheckpointValue(),
recordProcessorCheckpointer.largestPermittedCheckpointValue()));
if (shouldCallProcessRecords(records)) {
callProcessRecords(processRecordsInput, records);
}
success = true;
} catch (RuntimeException e) {
log.error("ShardId {}: Caught exception: ", shardInfo.shardId(), e);
exception = e;
backoff();
}
if (processRecordsInput.isAtShardEnd()) {
log.info("Reached end of shard {}, and processed {} records", shardInfo.shardId(), processRecordsInput.records().size());
return new TaskResult(null, true);
}
return new TaskResult(exception);
} finally {
MetricsUtil.addSuccessAndLatency(scope, success, startTimeMillis, MetricsLevel.SUMMARY);
MetricsUtil.endScope(scope);
}
}
private List<KinesisClientRecord> deaggregateAnyKplRecords(List<KinesisClientRecord> records) {
if (shard == null) {
return aggregatorUtil.deaggregate(records);
} else {
return aggregatorUtil.deaggregate(records, shard.hashKeyRange().startingHashKey(), shard.hashKeyRange().endingHashKey());
}
}
/**
* Sleeps for the configured backoff period. This is usually only called when an exception occurs.
*/
private void backoff() {
// backoff if we encounter an exception.
try {
Thread.sleep(this.backoffTimeMillis);
} catch (InterruptedException ie) {
log.debug("{}: Sleep was interrupted", shardInfo.shardId(), ie);
}
}
/**
* Dispatches a batch of records to the record processor, and handles any fallout from that.
*
* @param input
* the result of the last call to Kinesis
* @param records
* the records to be dispatched. It's possible the records have been adjusted by KPL deaggregation.
*/
private void callProcessRecords(ProcessRecordsInput input, List<KinesisClientRecord> records) {
log.debug("Calling application processRecords() with {} records from {}", records.size(),
shardInfo.shardId());
final ProcessRecordsInput processRecordsInput = ProcessRecordsInput.builder().records(records).cacheExitTime(input.cacheExitTime()).cacheEntryTime(input.cacheEntryTime())
.checkpointer(recordProcessorCheckpointer).millisBehindLatest(input.millisBehindLatest()).build();
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, PROCESS_TASK_OPERATION);
MetricsUtil.addShardId(scope, shardInfo.shardId());
final long startTime = System.currentTimeMillis();
try {
shardRecordProcessor.processRecords(processRecordsInput);
} catch (Exception e) {
log.error("ShardId {}: Application processRecords() threw an exception when processing shard ",
shardInfo.shardId(), e);
log.error("ShardId {}: Skipping over the following data records: {}", shardInfo.shardId(), records);
} finally {
MetricsUtil.addLatency(scope, RECORD_PROCESSOR_PROCESS_RECORDS_METRIC, startTime, MetricsLevel.SUMMARY);
MetricsUtil.endScope(scope);
}
}
/**
* Whether we should call process records or not
*
* @param records
* the records returned from the call to Kinesis, and/or deaggregation
* @return true if the set of records should be dispatched to the record process, false if they should not.
*/
private boolean shouldCallProcessRecords(List<KinesisClientRecord> records) {
return (!records.isEmpty()) || shouldCallProcessRecordsEvenForEmptyRecordList;
}
/**
* Emits metrics, and sleeps if there are no records available
*
* @param startTimeMillis
* the time when the task started
*/
private void handleNoRecords(long startTimeMillis) {
log.debug("Kinesis didn't return any records for shard {}", shardInfo.shardId());
long sleepTimeMillis = idleTimeInMilliseconds - (System.currentTimeMillis() - startTimeMillis);
if (sleepTimeMillis > 0) {
sleepTimeMillis = Math.max(sleepTimeMillis, idleTimeInMilliseconds);
try {
log.debug("Sleeping for {} ms since there were no new records in shard {}", sleepTimeMillis,
shardInfo.shardId());
Thread.sleep(sleepTimeMillis);
} catch (InterruptedException e) {
log.debug("ShardId {}: Sleep was interrupted", shardInfo.shardId());
}
}
}
@Override
public TaskType taskType() {
return taskType;
}
/**
* Scans a list of records to filter out records up to and including the most recent checkpoint value and to get the
* greatest extended sequence number from the retained records. Also emits metrics about the records.
*
* @param scope
* metrics scope to emit metrics into
* @param records
* list of records to scan and change in-place as needed
* @param lastCheckpointValue
* the most recent checkpoint value
* @param lastLargestPermittedCheckpointValue
* previous largest permitted checkpoint value
* @return the largest extended sequence number among the retained records
*/
private ExtendedSequenceNumber filterAndGetMaxExtendedSequenceNumber(final MetricsScope scope,
final List<KinesisClientRecord> records,
final ExtendedSequenceNumber lastCheckpointValue,
final ExtendedSequenceNumber lastLargestPermittedCheckpointValue) {
ExtendedSequenceNumber largestExtendedSequenceNumber = lastLargestPermittedCheckpointValue;
ListIterator<KinesisClientRecord> recordIterator = records.listIterator();
while (recordIterator.hasNext()) {
KinesisClientRecord record = recordIterator.next();
ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber(record.sequenceNumber(),
record.subSequenceNumber());
if (extendedSequenceNumber.compareTo(lastCheckpointValue) <= 0) {
recordIterator.remove();
log.debug("removing record with ESN {} because the ESN is <= checkpoint ({})", extendedSequenceNumber,
lastCheckpointValue);
continue;
}
if (largestExtendedSequenceNumber == null
|| largestExtendedSequenceNumber.compareTo(extendedSequenceNumber) < 0) {
largestExtendedSequenceNumber = extendedSequenceNumber;
}
scope.addData(DATA_BYTES_PROCESSED_METRIC, record.data().limit(), StandardUnit.BYTES,
MetricsLevel.SUMMARY);
}
return largestExtendedSequenceNumber;
}
}

535
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ShardConsumer.java
View file

@ -1,535 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import java.time.Duration;
import java.time.Instant;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Function;
import org.reactivestreams.Subscriber;
import org.reactivestreams.Subscription;
import com.google.common.annotations.VisibleForTesting;
import io.reactivex.Flowable;
import io.reactivex.Scheduler;
import io.reactivex.schedulers.Schedulers;
import lombok.AccessLevel;
import lombok.Getter;
import lombok.NonNull;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.exceptions.internal.BlockedOnParentShardException;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.events.TaskExecutionListenerInput;
import software.amazon.kinesis.metrics.MetricsCollectingTaskDecorator;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.retrieval.RecordsPublisher;
import software.amazon.kinesis.retrieval.RetryableRetrievalException;
/**
* Responsible for consuming data records of a (specified) shard.
* The instance should be shutdown when we lose the primary responsibility for a shard.
* A new instance should be created if the primary responsibility is reassigned back to this process.
*/
@Getter(AccessLevel.PACKAGE)
@Accessors(fluent = true)
@Slf4j
@KinesisClientInternalApi
public class ShardConsumer {
public static final int MAX_TIME_BETWEEN_REQUEST_RESPONSE = 35000;
private final RecordsPublisher recordsPublisher;
private final ExecutorService executorService;
private final Scheduler scheduler;
private final ShardInfo shardInfo;
private final ShardConsumerArgument shardConsumerArgument;
@NonNull
private final Optional<Long> logWarningForTaskAfterMillis;
private final Function<ConsumerTask, ConsumerTask> taskMetricsDecorator;
private final int bufferSize;
private final TaskExecutionListener taskExecutionListener;
private ConsumerTask currentTask;
private TaskOutcome taskOutcome;
private final AtomicReference<Throwable> processFailure = new AtomicReference<>(null);
private final AtomicReference<Throwable> dispatchFailure = new AtomicReference<>(null);
private CompletableFuture<Boolean> stateChangeFuture;
private boolean needsInitialization = true;
private volatile Instant taskDispatchedAt;
private volatile boolean taskIsRunning = false;
/*
* Tracks current state. It is only updated via the consumeStream/shutdown APIs. Therefore we don't do
* much coordination/synchronization to handle concurrent reads/updates.
*/
private ConsumerState currentState;
/*
* Used to track if we lost the primary responsibility. Once set to true, we will start shutting down.
* If we regain primary responsibility before shutdown is complete, Worker should create a new ShardConsumer object.
*/
@Getter(AccessLevel.PUBLIC)
private volatile ShutdownReason shutdownReason;
private volatile ShutdownNotification shutdownNotification;
private final InternalSubscriber subscriber;
public ShardConsumer(RecordsPublisher recordsPublisher, ExecutorService executorService, ShardInfo shardInfo,
Optional<Long> logWarningForTaskAfterMillis, ShardConsumerArgument shardConsumerArgument,
TaskExecutionListener taskExecutionListener) {
this(recordsPublisher, executorService, shardInfo, logWarningForTaskAfterMillis, shardConsumerArgument,
ConsumerStates.INITIAL_STATE,
ShardConsumer.metricsWrappingFunction(shardConsumerArgument.metricsFactory()), 8, taskExecutionListener);
}
//
// TODO: Make bufferSize configurable
//
public ShardConsumer(RecordsPublisher recordsPublisher, ExecutorService executorService, ShardInfo shardInfo,
Optional<Long> logWarningForTaskAfterMillis, ShardConsumerArgument shardConsumerArgument,
ConsumerState initialState, Function<ConsumerTask, ConsumerTask> taskMetricsDecorator,
int bufferSize, TaskExecutionListener taskExecutionListener) {
this.recordsPublisher = recordsPublisher;
this.executorService = executorService;
this.shardInfo = shardInfo;
this.shardConsumerArgument = shardConsumerArgument;
this.logWarningForTaskAfterMillis = logWarningForTaskAfterMillis;
this.taskExecutionListener = taskExecutionListener;
this.currentState = initialState;
this.taskMetricsDecorator = taskMetricsDecorator;
scheduler = Schedulers.from(executorService);
subscriber = new InternalSubscriber();
this.bufferSize = bufferSize;
if (this.shardInfo.isCompleted()) {
markForShutdown(ShutdownReason.SHARD_END);
}
}
private void startSubscriptions() {
Flowable.fromPublisher(recordsPublisher).subscribeOn(scheduler).observeOn(scheduler, true, bufferSize)
.subscribe(subscriber);
}
private final Object lockObject = new Object();
private Instant lastRequestTime = null;
private class InternalSubscriber implements Subscriber<ProcessRecordsInput> {
private Subscription subscription;
private volatile Instant lastDataArrival;
@Override
public void onSubscribe(Subscription s) {
subscription = s;
subscription.request(1);
}
@Override
public void onNext(ProcessRecordsInput input) {
try {
synchronized (lockObject) {
lastRequestTime = null;
}
lastDataArrival = Instant.now();
handleInput(input.toBuilder().cacheExitTime(Instant.now()).build(), subscription);
} catch (Throwable t) {
log.warn("{}: Caught exception from handleInput", shardInfo.shardId(), t);
dispatchFailure.set(t);
} finally {
subscription.request(1);
synchronized (lockObject) {
lastRequestTime = Instant.now();
}
}
}
@Override
public void onError(Throwable t) {
log.warn("{}: onError(). Cancelling subscription, and marking self as failed.", shardInfo.shardId(), t);
subscription.cancel();
processFailure.set(t);
}
@Override
public void onComplete() {
log.debug("{}: onComplete(): Received onComplete. Activity should be triggered externally", shardInfo.shardId());
}
public void cancel() {
if (subscription != null) {
subscription.cancel();
}
}
}
private synchronized void handleInput(ProcessRecordsInput input, Subscription subscription) {
if (isShutdownRequested()) {
subscription.cancel();
return;
}
processData(input);
if (taskOutcome == TaskOutcome.END_OF_SHARD) {
markForShutdown(ShutdownReason.SHARD_END);
subscription.cancel();
return;
}
subscription.request(1);
}
public void executeLifecycle() {
if (isShutdown()) {
return;
}
if (stateChangeFuture != null && !stateChangeFuture.isDone()) {
return;
}
try {
if (isShutdownRequested()) {
stateChangeFuture = shutdownComplete();
} else if (needsInitialization) {
if (stateChangeFuture != null) {
if (stateChangeFuture.get()) {
subscribe();
needsInitialization = false;
}
}
stateChangeFuture = initializeComplete();
}
} catch (InterruptedException e) {
//
// Ignored should be handled by scheduler
//
} catch (ExecutionException e) {
throw new RuntimeException(e);
}
if (ConsumerStates.ShardConsumerState.PROCESSING.equals(currentState.state())) {
Throwable t = healthCheck();
if (t instanceof Error) {
throw (Error) t;
}
}
}
@VisibleForTesting
Throwable healthCheck() {
logNoDataRetrievedAfterTime();
logLongRunningTask();
Throwable failure = processFailure.get();
if (!processFailure.compareAndSet(failure, null) && failure != null) {
log.error("{}: processFailure was updated while resetting, this shouldn't happen. " +
"Will retry on next health check", shardInfo.shardId());
return null;
}
if (failure != null) {
String logMessage = String.format("%s: Failure occurred in retrieval. Restarting data requests", shardInfo.shardId());
if (failure instanceof RetryableRetrievalException) {
log.debug(logMessage, failure.getCause());
} else {
log.warn(logMessage, failure);
}
startSubscriptions();
return failure;
}
Throwable expectedDispatchFailure = dispatchFailure.get();
if (expectedDispatchFailure != null) {
if (!dispatchFailure.compareAndSet(expectedDispatchFailure, null)) {
log.info("{}: Unable to reset the dispatch failure, this can happen if the record processor is failing aggressively.", shardInfo.shardId());
return null;
}
log.warn("Exception occurred while dispatching incoming data. The incoming data has been skipped", expectedDispatchFailure);
return expectedDispatchFailure;
}
synchronized (lockObject) {
if (lastRequestTime != null) {
Instant now = Instant.now();
Duration timeSinceLastResponse = Duration.between(lastRequestTime, now);
if (timeSinceLastResponse.toMillis() > MAX_TIME_BETWEEN_REQUEST_RESPONSE) {
log.error(
"{}: Last request was dispatched at {}, but no response as of {} ({}). Cancelling subscription, and restarting.",
shardInfo.shardId(), lastRequestTime, now, timeSinceLastResponse);
if (subscriber != null) {
subscriber.cancel();
}
//
// Set the last request time to now, we specifically don't null it out since we want it to trigger a
// restart if the subscription still doesn't start producing.
//
lastRequestTime = Instant.now();
startSubscriptions();
}
}
}
return null;
}
Duration taskRunningTime() {
if (taskDispatchedAt != null && taskIsRunning) {
return Duration.between(taskDispatchedAt, Instant.now());
}
return null;
}
String longRunningTaskMessage(Duration taken) {
if (taken != null) {
return String.format("Previous %s task still pending for shard %s since %s ago. ", currentTask.taskType(),
shardInfo.shardId(), taken);
}
return null;
}
private void logNoDataRetrievedAfterTime() {
logWarningForTaskAfterMillis.ifPresent(value -> {
Instant lastDataArrival = subscriber.lastDataArrival;
if (lastDataArrival != null) {
Instant now = Instant.now();
Duration timeSince = Duration.between(subscriber.lastDataArrival, now);
if (timeSince.toMillis() > value) {
log.warn("Last time data arrived: {} ({})", lastDataArrival, timeSince);
}
}
});
}
private void logLongRunningTask() {
Duration taken = taskRunningTime();
if (taken != null) {
String message = longRunningTaskMessage(taken);
if (log.isDebugEnabled()) {
log.debug("{} Not submitting new task.", message);
}
logWarningForTaskAfterMillis.ifPresent(value -> {
if (taken.toMillis() > value) {
log.warn(message);
}
});
}
}
@VisibleForTesting
void subscribe() {
startSubscriptions();
}
@VisibleForTesting
synchronized CompletableFuture<Boolean> initializeComplete() {
if (taskOutcome != null) {
updateState(taskOutcome);
}
if (currentState.state() == ConsumerStates.ShardConsumerState.PROCESSING) {
return CompletableFuture.completedFuture(true);
}
return CompletableFuture.supplyAsync(() -> {
if (isShutdownRequested()) {
throw new IllegalStateException("Shutdown requested while initializing");
}
executeTask(null);
if (isShutdownRequested()) {
throw new IllegalStateException("Shutdown requested while initializing");
}
return false;
}, executorService);
}
@VisibleForTesting
synchronized CompletableFuture<Boolean> shutdownComplete() {
if (taskOutcome != null) {
updateState(taskOutcome);
} else {
//
// ShardConsumer has been asked to shutdown before the first task even had a chance to run.
// In this case generate a successful task outcome, and allow the shutdown to continue. This should only
// happen if the lease was lost before the initial state had a chance to run.
//
updateState(TaskOutcome.SUCCESSFUL);
}
if (isShutdown()) {
return CompletableFuture.completedFuture(true);
}
return CompletableFuture.supplyAsync(() -> {
executeTask(null);
return false;
});
}
private synchronized void processData(ProcessRecordsInput input) {
executeTask(input);
}
private synchronized void executeTask(ProcessRecordsInput input) {
TaskExecutionListenerInput taskExecutionListenerInput = TaskExecutionListenerInput.builder()
.shardInfo(shardInfo)
.taskType(currentState.taskType())
.build();
taskExecutionListener.beforeTaskExecution(taskExecutionListenerInput);
ConsumerTask task = currentState.createTask(shardConsumerArgument, ShardConsumer.this, input);
if (task != null) {
taskDispatchedAt = Instant.now();
currentTask = task;
taskIsRunning = true;
TaskResult result;
try {
result = task.call();
} finally {
taskIsRunning = false;
}
taskOutcome = resultToOutcome(result);
taskExecutionListenerInput = taskExecutionListenerInput.toBuilder().taskOutcome(taskOutcome).build();
}
taskExecutionListener.afterTaskExecution(taskExecutionListenerInput);
}
private TaskOutcome resultToOutcome(TaskResult result) {
if (result.getException() == null) {
if (result.isShardEndReached()) {
return TaskOutcome.END_OF_SHARD;
}
return TaskOutcome.SUCCESSFUL;
}
logTaskException(result);
return TaskOutcome.FAILURE;
}
private synchronized void updateState(TaskOutcome outcome) {
ConsumerState nextState = currentState;
switch (outcome) {
case SUCCESSFUL:
nextState = currentState.successTransition();
break;
case END_OF_SHARD:
markForShutdown(ShutdownReason.SHARD_END);
break;
case FAILURE:
nextState = currentState.failureTransition();
break;
default:
log.error("No handler for outcome of {}", outcome.name());
nextState = currentState.failureTransition();
break;
}
nextState = handleShutdownTransition(outcome, nextState);
currentState = nextState;
}
private ConsumerState handleShutdownTransition(TaskOutcome outcome, ConsumerState nextState) {
if (isShutdownRequested() && outcome != TaskOutcome.FAILURE) {
return currentState.shutdownTransition(shutdownReason);
}
return nextState;
}
private void logTaskException(TaskResult taskResult) {
if (log.isDebugEnabled()) {
Exception taskException = taskResult.getException();
if (taskException instanceof BlockedOnParentShardException) {
// No need to log the stack trace for this exception (it is very specific).
log.debug("Shard {} is blocked on completion of parent shard.", shardInfo.shardId());
} else {
log.debug("Caught exception running {} task: ", currentTask.taskType(), taskResult.getException());
}
}
}
/**
* Requests the shutdown of the this ShardConsumer. This should give the record processor a chance to checkpoint
* before being shutdown.
*
* @param shutdownNotification
* used to signal that the record processor has been given the chance to shutdown.
*/
public void gracefulShutdown(ShutdownNotification shutdownNotification) {
if (subscriber != null) {
subscriber.cancel();
}
this.shutdownNotification = shutdownNotification;
markForShutdown(ShutdownReason.REQUESTED);
}
/**
* Shutdown this ShardConsumer (including invoking the ShardRecordProcessor shutdown API).
* This is called by Worker when it loses responsibility for a shard.
*
* @return true if shutdown is complete (false if shutdown is still in progress)
*/
public boolean leaseLost() {
log.debug("Shutdown({}): Lease lost triggered.", shardInfo.shardId());
if (subscriber != null) {
subscriber.cancel();
log.debug("Shutdown({}): Subscriber cancelled.", shardInfo.shardId());
}
markForShutdown(ShutdownReason.LEASE_LOST);
return isShutdown();
}
synchronized void markForShutdown(ShutdownReason reason) {
//
// ShutdownReason.LEASE_LOST takes precedence over SHARD_END
// (we won't be able to save checkpoint at end of shard)
//
if (shutdownReason == null || shutdownReason.canTransitionTo(reason)) {
shutdownReason = reason;
}
}
/**
* Used (by Worker) to check if this ShardConsumer instance has been shutdown
* ShardRecordProcessor shutdown() has been invoked, as appropriate.
*
* @return true if shutdown is complete
*/
public boolean isShutdown() {
return currentState.isTerminal();
}
@VisibleForTesting
public boolean isShutdownRequested() {
return shutdownReason != null;
}
/**
* Default task wrapping function for metrics
*
* @param metricsFactory
* the factory used for reporting metrics
* @return a function that will wrap the task with a metrics reporter
*/
private static Function<ConsumerTask, ConsumerTask> metricsWrappingFunction(MetricsFactory metricsFactory) {
return (task) -> {
if (task == null) {
return null;
} else {
return new MetricsCollectingTaskDecorator(task, metricsFactory);
}
};
}
}

73
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ShardConsumerArgument.java
View file

@ -1,73 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.ShardDetector;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.leases.HierarchicalShardSyncer;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.retrieval.AggregatorUtil;
import software.amazon.kinesis.retrieval.RecordsPublisher;
import java.util.concurrent.ExecutorService;
@Data
@Accessors(fluent = true)
@KinesisClientInternalApi
public class ShardConsumerArgument {
@NonNull
private final ShardInfo shardInfo;
@NonNull
private final String streamName;
@NonNull
private final LeaseRefresher leaseRefresher;
@NonNull
private final ExecutorService executorService;
@NonNull
private final RecordsPublisher recordsPublisher;
@NonNull
private final ShardRecordProcessor shardRecordProcessor;
@NonNull
private final Checkpointer checkpoint;
@NonNull
private final ShardRecordProcessorCheckpointer recordProcessorCheckpointer;
private final long parentShardPollIntervalMillis;
private final long taskBackoffTimeMillis;
private final boolean skipShardSyncAtWorkerInitializationIfLeasesExist;
private final long listShardsBackoffTimeInMillis;
private final int maxListShardsRetryAttempts;
private final boolean shouldCallProcessRecordsEvenForEmptyRecordList;
private final long idleTimeInMilliseconds;
@NonNull
private final InitialPositionInStreamExtended initialPositionInStream;
private final boolean cleanupLeasesOfCompletedShards;
private final boolean ignoreUnexpectedChildShards;
@NonNull
private final ShardDetector shardDetector;
private final AggregatorUtil aggregatorUtil;
private final HierarchicalShardSyncer hierarchicalShardSyncer;
@NonNull
private final MetricsFactory metricsFactory;
}

54
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ShutdownInput.java
View file

@ -1,54 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
/**
* Container for the parameters to the IRecordProcessor's
* {@link ShardRecordProcessor#shutdown(ShutdownInput
* shutdownInput) shutdown} method.
*/
@Builder
@Getter
@Accessors(fluent = true)
@EqualsAndHashCode
@ToString
public class ShutdownInput {
/**
* Get shutdown reason.
*
* -- GETTER --
* @return Reason for the shutdown (ShutdownReason.SHARD_END indicates the shard is closed and there are no
* more records to process. Shutdown.LEASE_LOST indicates a fail over has occurred).
*/
private final ShutdownReason shutdownReason;
/**
* Get Checkpointer.
*
* -- GETTER --
* @return The checkpointer object that the record processor should use to checkpoint
*/
private final RecordProcessorCheckpointer checkpointer;
}

36
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ShutdownNotification.java
View file

@ -1,36 +0,0 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import software.amazon.kinesis.processor.ShardRecordProcessor;
/**
* A shutdown request to the ShardConsumer
*/
public interface ShutdownNotification {
/**
* Used to indicate that the record processor has been notified of a requested shutdown, and given the chance to
* checkpoint.
*
*/
void shutdownNotificationComplete();
/**
* Used to indicate that the record processor has completed the call to
* {@link ShardRecordProcessor#shutdown(ShutdownInput)} has
* completed.
*/
void shutdownComplete();
}

59
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ShutdownNotificationTask.java
View file

@ -1,59 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.lifecycle.events.ShutdownRequestedInput;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.processor.ShardRecordProcessor;
/**
* Notifies record processor of incoming shutdown request, and gives them a chance to checkpoint.
*/
@RequiredArgsConstructor(access = AccessLevel.PACKAGE)
@Slf4j
@KinesisClientInternalApi
public class ShutdownNotificationTask implements ConsumerTask {
private final ShardRecordProcessor shardRecordProcessor;
private final RecordProcessorCheckpointer recordProcessorCheckpointer;
private final ShutdownNotification shutdownNotification;
// TODO: remove if not used
private final ShardInfo shardInfo;
@Override
public TaskResult call() {
try {
try {
shardRecordProcessor.shutdownRequested(ShutdownRequestedInput.builder().checkpointer(recordProcessorCheckpointer).build());
} catch (Exception ex) {
return new TaskResult(ex);
}
return new TaskResult(null);
} finally {
shutdownNotification.shutdownNotificationComplete();
}
}
@Override
public TaskType taskType() {
return TaskType.SHUTDOWN_NOTIFICATION;
}
}

171
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ShutdownTask.java
View file

@ -1,171 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import com.google.common.annotations.VisibleForTesting;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.annotations.KinesisClientInternalApi;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.ShardDetector;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.leases.HierarchicalShardSyncer;
import software.amazon.kinesis.lifecycle.events.LeaseLostInput;
import software.amazon.kinesis.lifecycle.events.ShardEndedInput;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsLevel;
import software.amazon.kinesis.metrics.MetricsUtil;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.retrieval.RecordsPublisher;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Task for invoking the ShardRecordProcessor shutdown() callback.
*/
@RequiredArgsConstructor
@Slf4j
@KinesisClientInternalApi
public class ShutdownTask implements ConsumerTask {
private static final String SHUTDOWN_TASK_OPERATION = "ShutdownTask";
private static final String RECORD_PROCESSOR_SHUTDOWN_METRIC = "RecordProcessor.shutdown";
@NonNull
private final ShardInfo shardInfo;
@NonNull
private final ShardDetector shardDetector;
@NonNull
private final ShardRecordProcessor shardRecordProcessor;
@NonNull
private final ShardRecordProcessorCheckpointer recordProcessorCheckpointer;
@NonNull
private final ShutdownReason reason;
@NonNull
private final InitialPositionInStreamExtended initialPositionInStream;
private final boolean cleanupLeasesOfCompletedShards;
private final boolean ignoreUnexpectedChildShards;
@NonNull
private final LeaseRefresher leaseRefresher;
private final long backoffTimeMillis;
@NonNull
private final RecordsPublisher recordsPublisher;
@NonNull
private final HierarchicalShardSyncer hierarchicalShardSyncer;
@NonNull
private final MetricsFactory metricsFactory;
private final TaskType taskType = TaskType.SHUTDOWN;
/*
* Invokes ShardRecordProcessor shutdown() API.
* (non-Javadoc)
*
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#call()
*/
@Override
public TaskResult call() {
recordProcessorCheckpointer.checkpointer().operation(SHUTDOWN_TASK_OPERATION);
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, SHUTDOWN_TASK_OPERATION);
Exception exception;
boolean applicationException = false;
try {
try {
// If we reached end of the shard, set sequence number to SHARD_END.
if (reason == ShutdownReason.SHARD_END) {
recordProcessorCheckpointer
.sequenceNumberAtShardEnd(recordProcessorCheckpointer.largestPermittedCheckpointValue());
recordProcessorCheckpointer.largestPermittedCheckpointValue(ExtendedSequenceNumber.SHARD_END);
}
log.debug("Invoking shutdown() for shard {}, concurrencyToken {}. Shutdown reason: {}",
shardInfo.shardId(), shardInfo.concurrencyToken(), reason);
final ShutdownInput shutdownInput = ShutdownInput.builder().shutdownReason(reason)
.checkpointer(recordProcessorCheckpointer).build();
final long startTime = System.currentTimeMillis();
try {
if (reason == ShutdownReason.SHARD_END) {
shardRecordProcessor.shardEnded(ShardEndedInput.builder().checkpointer(recordProcessorCheckpointer).build());
ExtendedSequenceNumber lastCheckpointValue = recordProcessorCheckpointer.lastCheckpointValue();
if (lastCheckpointValue == null
|| !lastCheckpointValue.equals(ExtendedSequenceNumber.SHARD_END)) {
throw new IllegalArgumentException(
"Application didn't checkpoint at end of shard " + shardInfo.shardId());
}
} else {
shardRecordProcessor.leaseLost(LeaseLostInput.builder().build());
}
log.debug("Shutting down retrieval strategy.");
recordsPublisher.shutdown();
log.debug("Record processor completed shutdown() for shard {}", shardInfo.shardId());
} catch (Exception e) {
applicationException = true;
throw e;
} finally {
MetricsUtil.addLatency(scope, RECORD_PROCESSOR_SHUTDOWN_METRIC, startTime, MetricsLevel.SUMMARY);
}
if (reason == ShutdownReason.SHARD_END) {
log.debug("Looking for child shards of shard {}", shardInfo.shardId());
// create leases for the child shards
hierarchicalShardSyncer.checkAndCreateLeaseForNewShards(shardDetector, leaseRefresher,
initialPositionInStream, cleanupLeasesOfCompletedShards, ignoreUnexpectedChildShards, scope);
log.debug("Finished checking for child shards of shard {}", shardInfo.shardId());
}
return new TaskResult(null);
} catch (Exception e) {
if (applicationException) {
log.error("Application exception. ", e);
} else {
log.error("Caught exception: ", e);
}
exception = e;
// backoff if we encounter an exception.
try {
Thread.sleep(this.backoffTimeMillis);
} catch (InterruptedException ie) {
log.debug("Interrupted sleep", ie);
}
}
} finally {
MetricsUtil.endScope(scope);
}
return new TaskResult(exception);
}
/*
* (non-Javadoc)
*
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#taskType()
*/
@Override
public TaskType taskType() {
return taskType;
}
@VisibleForTesting
public ShutdownReason getReason() {
return reason;
}
}

31
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/TaskExecutionListener.java
View file

@ -1,31 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
import software.amazon.kinesis.lifecycle.events.TaskExecutionListenerInput;
/**
* A listener for callbacks on task execution lifecycle for for a shard.
*
* Note: Recommended not to have a blocking implementation since these methods are
* called around the ShardRecordProcessor. A blocking call would result in slowing
* down the ShardConsumer.
*/
public interface TaskExecutionListener {
void beforeTaskExecution(TaskExecutionListenerInput input);
void afterTaskExecution(TaskExecutionListenerInput input);
}

33
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/TaskOutcome.java
View file

@ -1,33 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
/**
* Enumerates types of outcome of tasks executed as part of processing a shard.
*/
public enum TaskOutcome {
/**
* Denotes a successful task outcome.
*/
SUCCESSFUL,
/**
* Denotes that the last record from the shard has been read/consumed.
*/
END_OF_SHARD,
/**
* Denotes a failure or exception during processing of the shard.
*/
FAILURE
}

49
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/TaskType.java
View file

@ -1,49 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle;
/**
* Enumerates types of tasks executed as part of processing a shard.
*/
public enum TaskType {
/**
* Polls and waits until parent shard(s) have been fully processed.
*/
BLOCK_ON_PARENT_SHARDS,
/**
* Initialization of ShardRecordProcessor (and Amazon Kinesis Client Library internal state for a shard).
*/
INITIALIZE,
/**
* Fetching and processing of records.
*/
PROCESS,
/**
* Shutdown of ShardRecordProcessor.
*/
SHUTDOWN,
/**
* Graceful shutdown has been requested, and notification of the record processor will occur.
*/
SHUTDOWN_NOTIFICATION,
/**
* Occurs once the shutdown has been completed
*/
SHUTDOWN_COMPLETE,
/**
* Sync leases/activities corresponding to Kinesis shards.
*/
SHARDSYNC
}

50
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/events/InitializationInput.java
View file

@ -1,50 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle.events;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Container for the parameters to the ShardRecordProcessor
* {@link ShardRecordProcessor#initialize(InitializationInput initializationInput) initialize} method.
*/
@Builder
@Getter
@Accessors(fluent = true)
@EqualsAndHashCode
@ToString
public class InitializationInput {
/**
* The shardId that the record processor is being initialized for.
*/
private final String shardId;
/**
* The last extended sequence number that was successfully checkpointed by the previous record processor.
*/
private final ExtendedSequenceNumber extendedSequenceNumber;
/**
* The pending extended sequence number that may have been started by the previous record processor.
*
* This will only be set if the previous record processor had prepared a checkpoint, but lost its lease before
* completing the checkpoint.
*/
private final ExtendedSequenceNumber pendingCheckpointSequenceNumber;
}

37
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/events/LeaseLostInput.java
View file

@ -1,37 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle.events;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.processor.ShardRecordProcessor;
/**
* Provides data, and interaction about the loss of a lease to a
* {@link ShardRecordProcessor}.
*
* This currently has no members, but exists for forward compatibility reasons.
*/
@Accessors(fluent = true)
@Getter
@Builder
@EqualsAndHashCode
@ToString
public class LeaseLostInput {
}

82
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/events/ProcessRecordsInput.java
View file

@ -1,82 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle.events;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
/**
* Container for the parameters to the ShardRecordProcessor's
* {@link ShardRecordProcessor#processRecords(ProcessRecordsInput processRecordsInput) processRecords} method.
*/
@Builder(toBuilder = true)
@Getter
@Accessors(fluent = true)
@EqualsAndHashCode
@ToString
public class ProcessRecordsInput {
/**
* The time that this batch of records was received by the KCL.
*/
private Instant cacheEntryTime;
/**
* The time that this batch of records was prepared to be provided to the {@link ShardRecordProcessor}
*/
private Instant cacheExitTime;
/**
* Whether this batch of records is at the end of the shard.
*
* {@link ShardRecordProcessor}'s do not need to check this. If this is set the Scheduler will trigger a call to
* {@link ShardRecordProcessor#shardEnded(ShardEndedInput)} after the completion of the current processing call.
*/
private boolean isAtShardEnd;
/**
* The records received from Kinesis. These records may have been de-aggregated if they were published by the KPL.
*/
private List<KinesisClientRecord> records;
/**
* A checkpointer that the {@link ShardRecordProcessor} can use to checkpoint its progress.
*/
private RecordProcessorCheckpointer checkpointer;
/**
* How far behind this batch of records was when received from Kinesis.
*
* This value does not include the {@link #timeSpentInCache()}.
*/
private Long millisBehindLatest;
/**
* How long the records spent waiting to be dispatched to the {@link ShardRecordProcessor}
*
* @return the amount of time that records spent waiting before processing.
*/
public Duration timeSpentInCache() {
if (cacheEntryTime == null || cacheExitTime == null) {
return Duration.ZERO;
}
return Duration.between(cacheEntryTime, cacheExitTime);
}
}

45
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/events/ShardEndedInput.java
View file

@ -1,45 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle.events;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
/**
* Provides a checkpointer that <b>must</b> be used to signal the completion of the shard to the Scheduler.
*/
@Builder
@Accessors(fluent = true)
@Getter
@EqualsAndHashCode
@ToString
public class ShardEndedInput {
/**
* The checkpointer used to record that the record processor has completed the shard.
*
* The record processor <b>must</b> call {@link RecordProcessorCheckpointer#checkpoint()} before returning from
* {@link ShardRecordProcessor#shardEnded(ShardEndedInput)}. Failing to do so will trigger the Scheduler to retry
* shutdown until a successful checkpoint occurs.
*/
private final RecordProcessorCheckpointer checkpointer;
}

41
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/events/ShutdownRequestedInput.java
View file

@ -1,41 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle.events;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.processor.ShardRecordProcessor;
/**
* Provides access to a checkpointer so that {@link ShardRecordProcessor}'s can checkpoint
* before the lease is released during shutdown.
*/
@Builder
@Accessors(fluent = true)
@Getter
@EqualsAndHashCode
@ToString
public class ShutdownRequestedInput {
/**
* Checkpointer used to record the current progress of the
* {@link ShardRecordProcessor}.
*/
private final RecordProcessorCheckpointer checkpointer;
}

48
amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/events/TaskExecutionListenerInput.java
View file

@ -1,48 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.lifecycle.events;
import lombok.Builder;
import lombok.Data;
import lombok.experimental.Accessors;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.lifecycle.TaskOutcome;
import software.amazon.kinesis.lifecycle.TaskType;
import software.amazon.kinesis.lifecycle.TaskExecutionListener;
/**
* Container for the parameters to the TaskExecutionListener's
* {@link TaskExecutionListener#beforeTaskExecution(TaskExecutionListenerInput)} method.
* {@link TaskExecutionListener#afterTaskExecution(TaskExecutionListenerInput)} method.
*/
@Data
@Builder(toBuilder = true)
@Accessors(fluent = true)
public class TaskExecutionListenerInput {
/**
* Detailed information about the shard whose progress is monitored by TaskExecutionListener.
*/
private final ShardInfo shardInfo;
/**
* The type of task being executed for the shard.
*
* This corresponds to the state the shard is in.
*/
private final TaskType taskType;
/**
* Outcome of the task execution for the shard.
*/
private final TaskOutcome taskOutcome;
}

29
amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/AccumulateByNameMetricsScope.java
View file

@ -1,29 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.metrics;
/**
* This is a MetricScope with a KeyType of String. It provides the implementation of
* getting the key based off of the String KeyType.
*/
public abstract class AccumulateByNameMetricsScope extends AccumulatingMetricsScope<String> {
@Override
protected String getKey(String name) {
return name;
}
}

61
amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/CloudWatchMetricKey.java
View file

@ -1,61 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.metrics;
import java.util.List;
import java.util.Objects;
import software.amazon.awssdk.services.cloudwatch.model.Dimension;
import software.amazon.awssdk.services.cloudwatch.model.MetricDatum;
/*
* A representation of a key of a MetricDatum. This class is useful when wanting to compare
* whether 2 keys have the same MetricDatum. This feature will be used in MetricAccumulatingQueue
* where we aggregate metrics across multiple MetricScopes.
*/
public class CloudWatchMetricKey {
private List<Dimension> dimensions;
private String metricName;
/**
* @param datum data point
*/
public CloudWatchMetricKey(MetricDatum datum) {
this.dimensions = datum.dimensions();
this.metricName = datum.metricName();
}
@Override
public int hashCode() {
return Objects.hash(dimensions, metricName);
}
@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
CloudWatchMetricKey other = (CloudWatchMetricKey) obj;
return Objects.equals(other.dimensions, dimensions) && Objects.equals(other.metricName, metricName);
}
}

94
amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/CloudWatchMetricsFactory.java
View file

@ -1,94 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.metrics;
import java.util.Set;
import com.google.common.collect.ImmutableSet;
import lombok.NonNull;
import software.amazon.awssdk.core.exception.AbortedException;
import software.amazon.awssdk.services.cloudwatch.CloudWatchAsyncClient;
/**
* An IMetricsFactory that creates IMetricsScopes that output themselves via CloudWatch. Batches IMetricsScopes together
* to reduce API calls.
*/
public class CloudWatchMetricsFactory implements MetricsFactory {
/**
* If the CloudWatchPublisherRunnable accumulates more than FLUSH_SIZE distinct metrics, it will call CloudWatch
* immediately instead of waiting for the next scheduled call.
*/
private final CloudWatchPublisherRunnable runnable;
private final Thread publicationThread;
/**
* Enabled metrics level. All metrics below this level will be dropped.
*/
private final MetricsLevel metricsLevel;
/**
* List of enabled dimensions for metrics.
*/
private final Set<String> metricsEnabledDimensions;
/**
* Constructor.
*
* @param cloudWatchClient
* Client used to make CloudWatch requests
* @param namespace
* the namespace under which the metrics will appear in the CloudWatch console
* @param bufferTimeMillis
* time to buffer metrics before publishing to CloudWatch
* @param maxQueueSize
* maximum number of metrics that we can have in a queue
* @param metricsLevel
* metrics level to enable
* @param metricsEnabledDimensions
* metrics dimensions to allow
* @param flushSize
* size of batch that can be published
*/
public CloudWatchMetricsFactory(@NonNull final CloudWatchAsyncClient cloudWatchClient,
@NonNull final String namespace, final long bufferTimeMillis, final int maxQueueSize,
@NonNull final MetricsLevel metricsLevel, @NonNull final Set<String> metricsEnabledDimensions,
final int flushSize) {
this.metricsLevel = metricsLevel;
this.metricsEnabledDimensions = (metricsEnabledDimensions == null ? ImmutableSet.of()
: ImmutableSet.copyOf(metricsEnabledDimensions));
runnable = new CloudWatchPublisherRunnable(new CloudWatchMetricsPublisher(cloudWatchClient, namespace),
bufferTimeMillis, maxQueueSize, flushSize);
publicationThread = new Thread(runnable);
publicationThread.setName("cw-metrics-publisher");
publicationThread.start();
}
@Override
public MetricsScope createMetrics() {
return new CloudWatchMetricsScope(runnable, metricsLevel, metricsEnabledDimensions);
}
public void shutdown() {
runnable.shutdown();
try {
publicationThread.join();
} catch (InterruptedException e) {
throw AbortedException.builder().message(e.getMessage()).cause(e).build();
}
}
}

71
amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/CloudWatchMetricsPublisher.java
View file

@ -1,71 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.metrics;
import java.util.ArrayList;
import java.util.List;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.cloudwatch.CloudWatchAsyncClient;
import software.amazon.awssdk.services.cloudwatch.model.CloudWatchException;
import software.amazon.awssdk.services.cloudwatch.model.MetricDatum;
import software.amazon.awssdk.services.cloudwatch.model.PutMetricDataRequest;
/**
* Publisher that contains the logic to publish metrics.
*/
@Slf4j
public class CloudWatchMetricsPublisher {
// CloudWatch API has a limit of 20 MetricDatums per request
private static final int BATCH_SIZE = 20;
private final String namespace;
private final CloudWatchAsyncClient cloudWatchClient;
public CloudWatchMetricsPublisher(CloudWatchAsyncClient cloudWatchClient, String namespace) {
this.cloudWatchClient = cloudWatchClient;
this.namespace = namespace;
}
/**
* Given a list of MetricDatumWithKey, this method extracts the MetricDatum from each
* MetricDatumWithKey and publishes those datums.
*
* @param dataToPublish a list containing all the MetricDatums to publish
*/
public void publishMetrics(List<MetricDatumWithKey<CloudWatchMetricKey>> dataToPublish) {
for (int startIndex = 0; startIndex < dataToPublish.size(); startIndex += BATCH_SIZE) {
int endIndex = Math.min(dataToPublish.size(), startIndex + BATCH_SIZE);
PutMetricDataRequest.Builder request = PutMetricDataRequest.builder();
request = request.namespace(namespace);
List<MetricDatum> metricData = new ArrayList<>();
for (int i = startIndex; i < endIndex; i++) {
metricData.add(dataToPublish.get(i).datum);
}
request = request.metricData(metricData);
try {
cloudWatchClient.putMetricData(request.build());
log.debug("Successfully published {} datums.", endIndex - startIndex);
} catch (CloudWatchException e) {
log.warn("Could not publish {} datums to CloudWatch", endIndex - startIndex, e);
}
}
}
}

59
amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/CloudWatchMetricsScope.java
View file

@ -1,59 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.metrics;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
/**
* Metrics scope for CloudWatch metrics.
*/
public class CloudWatchMetricsScope extends FilteringMetricsScope implements MetricsScope {
private CloudWatchPublisherRunnable publisher;
/**
* Creates a CloudWatch metrics scope with given metrics level and enabled dimensions.
* @param publisher Publisher that emits CloudWatch metrics periodically.
* @param metricsLevel Metrics level to enable. All data with level below this will be dropped.
* @param metricsEnabledDimensions Enabled dimensions for CloudWatch metrics.
*/
public CloudWatchMetricsScope(CloudWatchPublisherRunnable publisher,
MetricsLevel metricsLevel, Set<String> metricsEnabledDimensions) {
super(metricsLevel, metricsEnabledDimensions);
this.publisher = publisher;
}
/**
* Once we call this method, all MetricDatums added to the scope will be enqueued to the publisher runnable.
* We enqueue MetricDatumWithKey because the publisher will aggregate similar metrics (i.e. MetricDatum with the
* same metricName) in the background thread. Hence aggregation using MetricDatumWithKey will be especially useful
* when aggregating across multiple MetricScopes.
*/
@Override
public void end() {
super.end();
final List<MetricDatumWithKey<CloudWatchMetricKey>> dataWithKeys = data.values().stream()
.map(metricDatum -> metricDatum.toBuilder().dimensions(getDimensions()).build())
.map(metricDatum -> new MetricDatumWithKey<>(new CloudWatchMetricKey(metricDatum), metricDatum))
.collect(Collectors.toList());
publisher.enqueue(dataWithKeys);
}
}

53
amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/DimensionTrackingMetricsScope.java
View file

@ -1,53 +0,0 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.metrics;
import software.amazon.awssdk.services.cloudwatch.model.Dimension;
import java.util.HashSet;
import java.util.Set;
/**
* DimensionTrackingMetricsScope is where we provide functionality for dimensions.
* Dimensions allow the user to be able view their metrics based off of the parameters they specify.
*
* The following examples show how to add dimensions if they would like to view their all metrics
* pertaining to a particular stream or for a specific date.
*
* myScope.addDimension("StreamName", "myStreamName");
* myScope.addDimension("Date", "Dec012013");
*
*
*/
public abstract class DimensionTrackingMetricsScope implements MetricsScope {
private Set<Dimension> dimensions = new HashSet<>();
@Override
public void addDimension(String name, String value) {
dimensions.add(Dimension.builder().name(name).value(value).build());
}
/**
* @return a set of dimensions for an IMetricsScope
*/
protected Set<Dimension> getDimensions() {
return dimensions;
}
}

Some files were not shown because too many files have changed in this diff Show more