luciano/amazon-kinesis-client
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Release 2.0.0 of the Amazon Kinesis Client for Java

Browse source 
* Added support for Enhanced Fan Out.
  Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
  * Records are now delivered via streaming, reducing end-to-end latency.
  * The Amazon Kinesis Client will automatically register a new consumer if required.
    When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
  * New configuration options are available to configure Enhanced Fan Out.
  * `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
    The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
    __WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__

  | Name            | Default | Description                                                                                                         |
  |-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
  | consumerArn     | Unset   | The ARN for an already created consumer.  If this is set, the Kinesis Client will not attempt to create a consumer. |
  | streamName      | Unset   | The name of the stream that a consumer should be create for if necessary                                            |
  | consumerName    | Unset   | The name of the consumer to create.  If this is not set the applicationName will be used instead.                   |
  | applicationName | Unset   | The name of the application.  This is used as the name of the consumer unless consumerName is set.                  |

* Modular Configuration of the Kinesis Client
  The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
  Configuration has been split into 7 classes.  Default versions of the configuration can be created from the `ConfigsBuilder`.
  Please see the migration guide for more information
  * `CheckpointConfig`
  * `CoordinatorConfig`
  * `LeaseManagementConfig`
  * `LifecycleConfig`
  * `MetricsConfig`
  * `ProcessorConfig`
  * `RetrievalConfig`

* Upgraded to AWS Java SDK 2.0
  The Kinesis Client now uses the AWS Java SDK 2.0.  The dependency on AWS Java SDK 1.11 has been removed.
  All configurations will only accept 2.0 clients.
  * When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
  * __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
    __When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
    Please see the migration guide for more information

* MultiLangDaemon is now a separate module
  The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`.  To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
This commit is contained in:
Pfifer, Justin 2018-08-02 10:57:11 -07:00
parent f1d60ec1a6
commit 258be9a504
369 changed files with 24673 additions and 23890 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

41
CHANGELOG.md
View file

@ -1,5 +1,46 @@
# Changelog
### Release 2.0.0 (August 02, 2018)
* Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
* Records are now delivered via streaming, reducing end-to-end latency.
* The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
* New configuration options are available to configure Enhanced Fan Out.
* `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
__WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__
| Name | Default | Description |
|-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
| consumerArn | Unset | The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. |
| streamName | Unset | The name of the stream that a consumer should be create for if necessary |
| consumerName | Unset | The name of the consumer to create. If this is not set the applicationName will be used instead. |
| applicationName | Unset | The name of the application. This is used as the name of the consumer unless consumerName is set. |
* Modular Configuration of the Kinesis Client
The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from the `ConfigsBuilder`.
Please [see the migration guide for more information][migration-guide].
* `CheckpointConfig`
* `CoordinatorConfig`
* `LeaseManagementConfig`
* `LifecycleConfig`
* `MetricsConfig`
* `ProcessorConfig`
* `RetrievalConfig`
* Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
All configurations will only accept 2.0 clients.
* When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
* __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
__When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
[Please see the migration guide for more information][migration-guide]
* MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`. To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
## Release 1.9.1 (April 30, 2018)
* Added the ability to create a prepared checkpoint when at `SHARD_END`.
* [PR #301](https://github.com/awslabs/amazon-kinesis-client/pull/301)

6
META-INF/MANIFEST.MF
View file

@ -2,7 +2,7 @@ Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Amazon Kinesis Client Library for Java
Bundle-SymbolicName: com.amazonaws.kinesisclientlibrary;singleton:=true
Bundle-Version: 1.9.1
Bundle-Version: 2.0.0
Bundle-Vendor: Amazon Technologies, Inc
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Require-Bundle: org.apache.commons.codec;bundle-version="1.6",
@ -15,13 +15,13 @@ Require-Bundle: org.apache.commons.codec;bundle-version="1.6",
com.amazonaws.sdk;bundle-version="1.11.319",
Export-Package: com.amazonaws.services.kinesis,
com.amazonaws.services.kinesis.clientlibrary,
com.amazonaws.services.kinesis.clientlibrary.config,
com.amazonaws.services.kinesis.clientlibrary.kinesisClientLibConfiguration,
com.amazonaws.services.kinesis.clientlibrary.exceptions,
com.amazonaws.services.kinesis.clientlibrary.exceptions.internal,
com.amazonaws.services.kinesis.clientlibrary.interfaces,
com.amazonaws.services.kinesis.clientlibrary.lib,
com.amazonaws.services.kinesis.clientlibrary.lib.checkpoint,
com.amazonaws.services.kinesis.clientlibrary.lib.worker,
com.amazonaws.services.kinesis.clientlibrary.lib.scheduler,
com.amazonaws.services.kinesis.clientlibrary.proxies,
com.amazonaws.services.kinesis.clientlibrary.types,
com.amazonaws.services.kinesis.leases,

50
README.md
View file

@ -30,16 +30,46 @@ To make it easier for developers to write record processors in other languages,
## Release Notes
### Latest Release (1.9.1)
* Added the ability to create a prepared checkpoint when at `SHARD_END`.
* [PR #301](https://github.com/awslabs/amazon-kinesis-client/pull/301)
* Added the ability to subscribe to worker state change events.
* [PR #291](https://github.com/awslabs/amazon-kinesis-client/pull/291)
* Added support for custom lease managers.
A custom `LeaseManager` can be provided to `Worker.Builder` that will be used to provide lease services.
This makes it possible to implement custom lease management systems in addition to the default DynamoDB system.
* [PR #297](https://github.com/awslabs/amazon-kinesis-client/pull/297)
* Updated the version of the AWS Java SDK to 1.11.219
### Latest Release (2.0.0)
* Added support for Enhanced Fan Out.
Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.
* Records are now delivered via streaming, reducing end-to-end latency.
* The Amazon Kinesis Client will automatically register a new consumer if required.
When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
* New configuration options are available to configure Enhanced Fan Out.
* `SubscribeToShard` maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
The `KinesisClientUtil` has been added to assist configuring the `maxConcurrency` of the `KinesisAsyncClient`.
__WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the `KinesisAsyncClient` is configured to have a `maxConcurrency` high enough to allow all leases plus additional usages of the `KinesisAsyncClient`.__
| Name | Default | Description |
|-----------------|---------|---------------------------------------------------------------------------------------------------------------------|
| consumerArn | Unset | The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer. |
| streamName | Unset | The name of the stream that a consumer should be create for if necessary |
| consumerName | Unset | The name of the consumer to create. If this is not set the applicationName will be used instead. |
| applicationName | Unset | The name of the application. This is used as the name of the consumer unless consumerName is set. |
* Modular Configuration of the Kinesis Client
The Kinesis Client has migrated to a modular configuration system, and the `KinesisClientLibConfiguration` class has been removed.
Configuration has been split into 7 classes. Default versions of the configuration can be created from the `ConfigsBuilder`.
Please [see the migration guide for more information][migration-guide].
* `CheckpointConfig`
* `CoordinatorConfig`
* `LeaseManagementConfig`
* `LifecycleConfig`
* `MetricsConfig`
* `ProcessorConfig`
* `RetrievalConfig`
* Upgraded to AWS Java SDK 2.0
The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
All configurations will only accept 2.0 clients.
* When configuring the `KinesisAsyncClient` the `KinesisClientUtil#createKinesisAsyncClient` can be used to configure the Kinesis Client
* __If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.__
__When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded__
[Please see the migration guide for more information][migration-guide]
* MultiLangDaemon is now a separate module
The MultiLangDaemon has been separated to its own Maven module and is no longer available in `amazon-kinesis-client`. To include the MultiLangDaemon, add a dependency on `amazon-kinesis-client-multilang`.
### For remaining release notes check **[CHANGELOG.md][changelog-md]**.

130
amazon-kinesis-client-multilang/pom.xml Normal file
View file

@ -0,0 +1,130 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
~
~ Licensed under the Amazon Software License (the "License").
~ You may not use this file except in compliance with the License.
~ A copy of the License is located at
~
~ http://aws.amazon.com/asl/
~
~ or in the "license" file accompanying this file. This file is distributed
~ on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
~ express or implied. See the License for the specific language governing
~ permissions and limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>amazon-kinesis-client-pom</artifactId>
<groupId>software.amazon.kinesis</groupId>
<version>2.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>amazon-kinesis-client-multilang</artifactId>
<dependencies>
<dependency>
<groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.16.20</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.1.7</version>
</dependency>
<!-- Test -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.19</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.3</version>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>disable-java8-doclint</id>
<activation>
<jdk>[1.8,)</jdk>
</activation>
<properties>
<additionalparam>-Xdoclint:none</additionalparam>
</properties>
</profile>
</profiles>
</project>

9
src/main/java/com/amazonaws/services/kinesis/multilang/DrainChildSTDERRTask.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/DrainChildSTDERRTask.java
View file

@ -16,22 +16,19 @@ package com.amazonaws.services.kinesis.multilang;
import java.io.BufferedReader;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import lombok.extern.slf4j.Slf4j;
/**
* Reads lines off the STDERR of the child process and prints them to this process's (the JVM's) STDERR and log.
*/
@Slf4j
class DrainChildSTDERRTask extends LineReaderTask<Boolean> {
private static final Log LOG = LogFactory.getLog(DrainChildSTDERRTask.class);
DrainChildSTDERRTask() {
}
@Override
protected HandleLineResult<Boolean> handleLine(String line) {
LOG.error("Received error line from subprocess [" + line + "] for shard " + getShardId());
log.error("Received error line from subprocess [{}] for shard {}", line, getShardId());
System.err.println(line);
return new HandleLineResult<Boolean>();
}

13
src/main/java/com/amazonaws/services/kinesis/multilang/DrainChildSTDOUTTask.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/DrainChildSTDOUTTask.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -16,8 +16,7 @@ package com.amazonaws.services.kinesis.multilang;
import java.io.BufferedReader;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import lombok.extern.slf4j.Slf4j;
/**
* This class is used to drain the STDOUT of the child process. After the child process has been given a shutdown
@ -36,22 +35,20 @@ import org.apache.commons.logging.LogFactory;
* To prevent the child process from becoming blocked in this way, it is the responsibility of the parent process to
* drain the child process's STDOUT. We reprint each drained line to our log to permit debugging.
*/
@Slf4j
class DrainChildSTDOUTTask extends LineReaderTask<Boolean> {
private static final Log LOG = LogFactory.getLog(DrainChildSTDOUTTask.class);
DrainChildSTDOUTTask() {
}
@Override
protected HandleLineResult<Boolean> handleLine(String line) {
LOG.info("Drained line for shard " + getShardId() + ": " + line);
log.info("Drained line for shard {}: {}", getShardId(), line);
return new HandleLineResult<Boolean>();
}
@Override
protected Boolean returnAfterException(Exception e) {
LOG.info("Encountered exception while draining STDOUT of child process for shard " + getShardId(), e);
log.info("Encountered exception while draining STDOUT of child process for shard {}", getShardId(), e);
return false;
}

14
src/main/java/com/amazonaws/services/kinesis/multilang/GetNextMessageTask.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/GetNextMessageTask.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -17,21 +17,17 @@ package com.amazonaws.services.kinesis.multilang;
import java.io.BufferedReader;
import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.kinesis.multilang.messages.Message;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
/**
* Gets the next message off the STDOUT of the child process. Throws an exception if a message is not found before the
* end of the input stream is reached.
*/
@Slf4j
class GetNextMessageTask extends LineReaderTask<Message> {
private static final Log LOG = LogFactory.getLog(GetNextMessageTask.class);
private ObjectMapper objectMapper;
private static final String EMPTY_LINE = "";
@ -68,7 +64,7 @@ class GetNextMessageTask extends LineReaderTask<Message> {
return new HandleLineResult<Message>(objectMapper.readValue(line, Message.class));
}
} catch (IOException e) {
LOG.info("Skipping unexpected line on STDOUT for shard " + getShardId() + ": " + line);
log.info("Skipping unexpected line on STDOUT for shard {}: {}", getShardId(), line);
}
return new HandleLineResult<Message>();
}

17
src/main/java/com/amazonaws/services/kinesis/multilang/LineReaderTask.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/LineReaderTask.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -20,8 +20,7 @@ import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.concurrent.Callable;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import lombok.extern.slf4j.Slf4j;
/**
* This abstract class captures the process of reading from an input stream. Three methods must be provided for
@ -34,10 +33,8 @@ import org.apache.commons.logging.LogFactory;
*
* @param <T>
*/
@Slf4j
abstract class LineReaderTask<T> implements Callable<T> {
private static final Log LOG = LogFactory.getLog(LineReaderTask.class);
private BufferedReader reader;
private String description;
@ -56,7 +53,7 @@ abstract class LineReaderTask<T> implements Callable<T> {
public T call() throws Exception {
String nextLine = null;
try {
LOG.info("Starting: " + description);
log.info("Starting: {}", description);
while ((nextLine = reader.readLine()) != null) {
HandleLineResult<T> result = handleLine(nextLine);
if (result.hasReturnValue()) {
@ -66,7 +63,7 @@ abstract class LineReaderTask<T> implements Callable<T> {
} catch (IOException e) {
return returnAfterException(e);
}
LOG.info("Stopping: " + description);
log.info("Stopping: {}", description);
return returnAfterEndOfInput();
}
@ -157,8 +154,8 @@ abstract class LineReaderTask<T> implements Callable<T> {
/**
* An initialization method allows us to delay setting the attributes of this class. Some of the attributes, stream
* and shardId, are not known to the {@link MultiLangRecordProcessorFactory} when it constructs a
* {@link MultiLangRecordProcessor} but are later determined when
* {@link MultiLangRecordProcessor#initialize(String)} is called. So we follow a pattern where the attributes are
* {@link MultiLangShardRecordProcessor} but are later determined when
* {@link MultiLangShardRecordProcessor#initialize(String)} is called. So we follow a pattern where the attributes are
* set inside this method instead of the constructor so that this object will be initialized when all its attributes
* are known to the record processor.
*

4
src/main/java/com/amazonaws/services/kinesis/multilang/MessageReader.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/MessageReader.java
View file

@ -85,8 +85,8 @@ class MessageReader {
/**
* An initialization method allows us to delay setting the attributes of this class. Some of the attributes,
* stream and shardId, are not known to the {@link MultiLangRecordProcessorFactory} when it constructs a
* {@link MultiLangRecordProcessor} but are later determined when
* {@link MultiLangRecordProcessor#initialize(String)} is called. So we follow a pattern where the attributes are
* {@link MultiLangShardRecordProcessor} but are later determined when
* {@link MultiLangShardRecordProcessor#initialize(String)} is called. So we follow a pattern where the attributes are
* set inside this method instead of the constructor so that this object will be initialized when all its attributes
* are known to the record processor.
*

31
src/main/java/com/amazonaws/services/kinesis/multilang/MessageWriter.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/MessageWriter.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -22,29 +22,24 @@ import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason;
import com.amazonaws.services.kinesis.multilang.messages.CheckpointMessage;
import com.amazonaws.services.kinesis.multilang.messages.InitializeMessage;
import com.amazonaws.services.kinesis.multilang.messages.Message;
import com.amazonaws.services.kinesis.multilang.messages.ProcessRecordsMessage;
import com.amazonaws.services.kinesis.multilang.messages.ShutdownMessage;
import com.amazonaws.services.kinesis.multilang.messages.ShutdownRequestedMessage;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.ShutdownReason;
/**
* Defines methods for writing {@link Message} objects to the child process's STDIN.
*/
@Slf4j
class MessageWriter {
private static final Log LOG = LogFactory.getLog(MessageWriter.class);
private BufferedWriter writer;
private volatile boolean open = true;
@ -82,7 +77,7 @@ class MessageWriter {
writer.write(System.lineSeparator(), 0, System.lineSeparator().length());
writer.flush();
}
LOG.info("Message size == " + message.getBytes().length + " bytes for shard " + shardId);
log.info("Message size == {} bytes for shard {}", message.getBytes().length, shardId);
} catch (IOException e) {
open = false;
}
@ -94,7 +89,7 @@ class MessageWriter {
return this.executorService.submit(writeMessageToOutputTask);
} else {
String errorMessage = "Cannot write message " + message + " because writer is closed for shard " + shardId;
LOG.info(errorMessage);
log.info(errorMessage);
throw new IllegalStateException(errorMessage);
}
}
@ -106,7 +101,7 @@ class MessageWriter {
* @return
*/
private Future<Boolean> writeMessage(Message message) {
LOG.info("Writing " + message.getClass().getSimpleName() + " to child process for shard " + shardId);
log.info("Writing {} to child process for shard {}", message.getClass().getSimpleName(), shardId);
try {
String jsonText = objectMapper.writeValueAsString(message);
return writeMessageToOutput(jsonText);
@ -114,7 +109,7 @@ class MessageWriter {
String errorMessage =
String.format("Encountered I/O error while writing %s action to subprocess", message.getClass()
.getSimpleName());
LOG.error(errorMessage, e);
log.error(errorMessage, e);
throw new RuntimeException(errorMessage, e);
}
}
@ -187,8 +182,8 @@ class MessageWriter {
/**
* An initialization method allows us to delay setting the attributes of this class. Some of the attributes,
* stream and shardId, are not known to the {@link MultiLangRecordProcessorFactory} when it constructs a
* {@link MultiLangRecordProcessor} but are later determined when
* {@link MultiLangRecordProcessor#initialize(String)} is called. So we follow a pattern where the attributes are
* {@link MultiLangShardRecordProcessor} but are later determined when
* {@link MultiLangShardRecordProcessor (String)} is called. So we follow a pattern where the attributes are
* set inside this method instead of the constructor so that this object will be initialized when all its attributes
* are known to the record processor.
*

44
src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemon.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemon.java
View file

@ -16,7 +16,6 @@ package com.amazonaws.services.kinesis.multilang;
import java.io.IOException;
import java.io.PrintStream;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
@ -24,15 +23,13 @@ import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessorFactory;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.coordinator.Scheduler;
import software.amazon.kinesis.processor.ShardRecordProcessorFactory;
/**
* Main app that launches the worker that runs the multi-language record processor.
* Main app that launches the scheduler that runs the multi-language record processor.
*
* Requires a properties file containing configuration for this daemon and the KCL. A properties file should at minimum
* define these properties:
@ -58,11 +55,9 @@ import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker;
* AWSCredentialsProvider = DefaultAWSCredentialsProviderChain
* </pre>
*/
@Slf4j
public class MultiLangDaemon implements Callable<Integer> {
private static final Log LOG = LogFactory.getLog(MultiLangDaemon.class);
private Worker worker;
private Scheduler scheduler;
/**
* Constructor.
@ -78,18 +73,17 @@ public class MultiLangDaemon implements Callable<Integer> {
this(buildWorker(recordProcessorFactory, configuration, workerThreadPool));
}
private static Worker buildWorker(IRecordProcessorFactory recordProcessorFactory,
private static Scheduler buildWorker(ShardRecordProcessorFactory recordShardRecordProcessorFactory,
KinesisClientLibConfiguration configuration, ExecutorService workerThreadPool) {
return new Worker.Builder().recordProcessorFactory(recordProcessorFactory).config(configuration)
.execService(workerThreadPool).build();
return null;
}
/**
*
* @param worker A worker to use instead of the default worker.
* @param scheduler A scheduler to use instead of the default scheduler.
*/
public MultiLangDaemon(Worker worker) {
this.worker = worker;
public MultiLangDaemon(Scheduler scheduler) {
this.scheduler = scheduler;
}
/**
@ -111,9 +105,9 @@ public class MultiLangDaemon implements Callable<Integer> {
public Integer call() throws Exception {
int exitCode = 0;
try {
worker.run();
scheduler.run();
} catch (Throwable t) {
LOG.error("Caught throwable while processing data.", t);
log.error("Caught throwable while processing data.", t);
exitCode = 1;
}
return exitCode;
@ -152,13 +146,13 @@ public class MultiLangDaemon implements Callable<Integer> {
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
LOG.info("Process terminanted, will initiate shutdown.");
log.info("Process terminanted, will initiate shutdown.");
try {
Future<Void> fut = daemon.worker.requestShutdown();
Future<Void> fut = daemon.scheduler.requestShutdown();
fut.get(shutdownGraceMillis, TimeUnit.MILLISECONDS);
LOG.info("Process shutdown is complete.");
log.info("Process shutdown is complete.");
} catch (InterruptedException | ExecutionException | TimeoutException e) {
LOG.error("Encountered an error during shutdown.", e);
log.error("Encountered an error during shutdown.", e);
}
}
});
@ -167,7 +161,7 @@ public class MultiLangDaemon implements Callable<Integer> {
try {
System.exit(future.get());
} catch (InterruptedException | ExecutionException e) {
LOG.error("Encountered an error while running daemon", e);
log.error("Encountered an error while running daemon", e);
}
System.exit(1);
}

104
src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemonConfig.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemonConfig.java
View file

@ -1,16 +1,9 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Amazon Software License
* (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at
* http://aws.amazon.com/asl/ or in the "license" file accompanying this file. This file is distributed on an "AS IS"
* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
* language governing permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang;
@ -26,20 +19,18 @@ import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.kinesis.clientlibrary.config.KinesisClientLibConfigurator;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfigurator;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.retrieval.RetrievalConfig;
/**
* This class captures the configuration needed to run the MultiLangDaemon.
*/
@Slf4j
public class MultiLangDaemonConfig {
private static final Log LOG = LogFactory.getLog(MultiLangDaemonConfig.class);
private static final String USER_AGENT = "amazon-kinesis-multi-lang-daemon";
private static final String VERSION = "1.0.1";
@ -56,9 +47,12 @@ public class MultiLangDaemonConfig {
/**
* Constructor.
*
* @param propertiesFile The location of the properties file.
* @throws IOException Thrown when the properties file can't be accessed.
* @throws IllegalArgumentException Thrown when the contents of the properties file are not as expected.
* @param propertiesFile
* The location of the properties file.
* @throws IOException
* Thrown when the properties file can't be accessed.
* @throws IllegalArgumentException
* Thrown when the contents of the properties file are not as expected.
*/
public MultiLangDaemonConfig(String propertiesFile) throws IOException, IllegalArgumentException {
this(propertiesFile, Thread.currentThread().getContextClassLoader());
@ -66,33 +60,39 @@ public class MultiLangDaemonConfig {
/**
*
* @param propertiesFile The location of the properties file.
* @param classLoader A classloader, useful if trying to programmatically configure with the daemon, such as in a
* unit test.
* @throws IOException Thrown when the properties file can't be accessed.
* @throws IllegalArgumentException Thrown when the contents of the properties file are not as expected.
* @param propertiesFile
* The location of the properties file.
* @param classLoader
* A classloader, useful if trying to programmatically configure with the daemon, such as in a unit test.
* @throws IOException
* Thrown when the properties file can't be accessed.
* @throws IllegalArgumentException
* Thrown when the contents of the properties file are not as expected.
*/
public MultiLangDaemonConfig(String propertiesFile, ClassLoader classLoader) throws IOException,
IllegalArgumentException {
public MultiLangDaemonConfig(String propertiesFile, ClassLoader classLoader)
throws IOException, IllegalArgumentException {
this(propertiesFile, classLoader, new KinesisClientLibConfigurator());
}
/**
*
* @param propertiesFile The location of the properties file.
* @param classLoader A classloader, useful if trying to programmatically configure with the daemon, such as in a
* unit test.
* @param configurator A configurator to use.
* @throws IOException Thrown when the properties file can't be accessed.
* @throws IllegalArgumentException Thrown when the contents of the properties file are not as expected.
* @param propertiesFile
* The location of the properties file.
* @param classLoader
* A classloader, useful if trying to programmatically configure with the daemon, such as in a unit test.
* @param configurator
* A configurator to use.
* @throws IOException
* Thrown when the properties file can't be accessed.
* @throws IllegalArgumentException
* Thrown when the contents of the properties file are not as expected.
*/
public MultiLangDaemonConfig(String propertiesFile,
ClassLoader classLoader,
public MultiLangDaemonConfig(String propertiesFile, ClassLoader classLoader,
KinesisClientLibConfigurator configurator) throws IOException, IllegalArgumentException {
Properties properties = loadProperties(classLoader, propertiesFile);
if (!validateProperties(properties)) {
throw new IllegalArgumentException("Must provide an executable name in the properties file, "
+ "e.g. executableName = sampleapp.py");
throw new IllegalArgumentException(
"Must provide an executable name in the properties file, " + "e.g. executableName = sampleapp.py");
}
String executableName = properties.getProperty(PROP_EXECUTABLE_NAME);
@ -100,10 +100,11 @@ public class MultiLangDaemonConfig {
kinesisClientLibConfig = configurator.getConfiguration(properties);
executorService = buildExecutorService(properties);
recordProcessorFactory = new MultiLangRecordProcessorFactory(executableName, executorService, kinesisClientLibConfig);
recordProcessorFactory = new MultiLangRecordProcessorFactory(executableName, executorService,
kinesisClientLibConfig);
LOG.info("Running " + kinesisClientLibConfig.getApplicationName() + " to process stream "
+ kinesisClientLibConfig.getStreamName() + " with executable " + executableName);
log.info("Running {} to process stream {} with executable {}", kinesisClientLibConfig.getApplicationName(),
kinesisClientLibConfig.getStreamName(), executableName);
prepare(processingLanguage);
}
@ -111,11 +112,11 @@ public class MultiLangDaemonConfig {
// Ensure the JVM will refresh the cached IP values of AWS resources (e.g. service endpoints).
java.security.Security.setProperty("networkaddress.cache.ttl", "60");
LOG.info("Using workerId: " + kinesisClientLibConfig.getWorkerIdentifier());
LOG.info("Using credentials with access key id: "
+ kinesisClientLibConfig.getKinesisCredentialsProvider().getCredentials().getAWSAccessKeyId());
log.info("Using workerId: {}", kinesisClientLibConfig.getWorkerIdentifier());
log.info("Using credentials with access key id: {}",
kinesisClientLibConfig.getKinesisCredentialsProvider().resolveCredentials().accessKeyId());
StringBuilder userAgent = new StringBuilder(KinesisClientLibConfiguration.KINESIS_CLIENT_LIB_USER_AGENT);
StringBuilder userAgent = new StringBuilder(RetrievalConfig.KINESIS_CLIENT_LIB_USER_AGENT);
userAgent.append(" ");
userAgent.append(USER_AGENT);
userAgent.append("/");
@ -131,8 +132,7 @@ public class MultiLangDaemonConfig {
userAgent.append(recordProcessorFactory.getCommandArray()[0]);
}
LOG.info(String.format("MultiLangDaemon is adding the following fields to the User Agent: %s",
userAgent.toString()));
log.info("MultiLangDaemon is adding the following fields to the User Agent: {}", userAgent.toString());
kinesisClientLibConfig.withUserAgent(userAgent.toString());
}
@ -174,13 +174,13 @@ public class MultiLangDaemonConfig {
private static ExecutorService buildExecutorService(Properties properties) {
int maxActiveThreads = getMaxActiveThreads(properties);
ThreadFactoryBuilder builder = new ThreadFactoryBuilder().setNameFormat("multi-lang-daemon-%04d");
LOG.debug(String.format("Value for %s property is %d", PROP_MAX_ACTIVE_THREADS, maxActiveThreads));
log.debug("Value for {} property is {}", PROP_MAX_ACTIVE_THREADS, maxActiveThreads);
if (maxActiveThreads <= 0) {
LOG.info("Using a cached thread pool.");
log.info("Using a cached thread pool.");
return new ThreadPoolExecutor(0, Integer.MAX_VALUE, 60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>(),
builder.build());
} else {
LOG.info(String.format("Using a fixed thread pool with %d max active threads.", maxActiveThreads));
log.info("Using a fixed thread pool with {} max active threads.", maxActiveThreads);
return new ThreadPoolExecutor(maxActiveThreads, maxActiveThreads, 0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(), builder.build());
}

72
src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangProtocol.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangProtocol.java
View file

@ -14,12 +14,18 @@
*/
package com.amazonaws.services.kinesis.multilang;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
import java.util.Optional;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import com.amazonaws.services.kinesis.multilang.messages.CheckpointMessage;
import com.amazonaws.services.kinesis.multilang.messages.InitializeMessage;
import com.amazonaws.services.kinesis.multilang.messages.Message;
@ -27,18 +33,13 @@ import com.amazonaws.services.kinesis.multilang.messages.ProcessRecordsMessage;
import com.amazonaws.services.kinesis.multilang.messages.ShutdownMessage;
import com.amazonaws.services.kinesis.multilang.messages.ShutdownRequestedMessage;
import com.amazonaws.services.kinesis.multilang.messages.StatusMessage;
import lombok.extern.apachecommons.CommonsLog;
import java.util.Optional;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import lombok.extern.slf4j.Slf4j;
/**
* An implementation of the multi language protocol.
*/
@CommonsLog
@Slf4j
class MultiLangProtocol {
private MessageReader messageReader;
@ -89,7 +90,7 @@ class MultiLangProtocol {
*/
boolean processRecords(ProcessRecordsInput processRecordsInput) {
Future<Boolean> writeFuture = messageWriter.writeProcessRecordsMessage(processRecordsInput);
return waitForStatusMessage(ProcessRecordsMessage.ACTION, processRecordsInput.getCheckpointer(), writeFuture);
return waitForStatusMessage(ProcessRecordsMessage.ACTION, processRecordsInput.checkpointer(), writeFuture);
}
/**
@ -100,7 +101,7 @@ class MultiLangProtocol {
* @param reason Why this processor is being shutdown.
* @return Whether or not this operation succeeded.
*/
boolean shutdown(IRecordProcessorCheckpointer checkpointer, ShutdownReason reason) {
boolean shutdown(RecordProcessorCheckpointer checkpointer, ShutdownReason reason) {
Future<Boolean> writeFuture = messageWriter.writeShutdownMessage(reason);
return waitForStatusMessage(ShutdownMessage.ACTION, checkpointer, writeFuture);
}
@ -112,14 +113,14 @@ class MultiLangProtocol {
* @param checkpointer A checkpointer.
* @return Whether or not this operation succeeded.
*/
boolean shutdownRequested(IRecordProcessorCheckpointer checkpointer) {
boolean shutdownRequested(RecordProcessorCheckpointer checkpointer) {
Future<Boolean> writeFuture = messageWriter.writeShutdownRequestedMessage();
return waitForStatusMessage(ShutdownRequestedMessage.ACTION, checkpointer, writeFuture);
}
/**
* Waits for a {@link StatusMessage} for a particular action. If a {@link CheckpointMessage} is received, then this
* method will attempt to checkpoint with the provided {@link IRecordProcessorCheckpointer}. This method returns
* method will attempt to checkpoint with the provided {@link RecordProcessorCheckpointer}. This method returns
* true if writing to the child process succeeds and the status message received back was for the correct action and
* all communications with the child process regarding checkpointing were successful. Note that whether or not the
* checkpointing itself was successful is not the concern of this method. This method simply cares whether it was
@ -133,7 +134,7 @@ class MultiLangProtocol {
* The writing task.
* @return Whether or not this operation succeeded.
*/
private boolean waitForStatusMessage(String action, IRecordProcessorCheckpointer checkpointer,
private boolean waitForStatusMessage(String action, RecordProcessorCheckpointer checkpointer,
Future<Boolean> writeFuture) {
boolean statusWasCorrect = waitForStatusMessage(action, checkpointer);
@ -142,13 +143,10 @@ class MultiLangProtocol {
boolean writerIsStillOpen = writeFuture.get();
return statusWasCorrect && writerIsStillOpen;
} catch (InterruptedException e) {
log.error(String.format("Interrupted while writing %s message for shard %s", action,
initializationInput.getShardId()));
log.error("Interrupted while writing {} message for shard {}", action, initializationInput.shardId());
return false;
} catch (ExecutionException e) {
log.error(
String.format("Failed to write %s message for shard %s", action, initializationInput.getShardId()),
e);
log.error("Failed to write {} message for shard {}", action, initializationInput.shardId(), e);
return false;
}
}
@ -162,7 +160,7 @@ class MultiLangProtocol {
* the original process records request
* @return Whether or not this operation succeeded.
*/
boolean waitForStatusMessage(String action, IRecordProcessorCheckpointer checkpointer) {
boolean waitForStatusMessage(String action, RecordProcessorCheckpointer checkpointer) {
Optional<StatusMessage> statusMessage = Optional.empty();
while (!statusMessage.isPresent()) {
Future<Message> future = this.messageReader.getNextMessageFromSTDOUT();
@ -196,15 +194,15 @@ class MultiLangProtocol {
try {
return Optional.of(fm.get());
} catch (InterruptedException e) {
log.error(String.format("Interrupted while waiting for %s message for shard %s", action,
initializationInput.getShardId()), e);
log.error("Interrupted while waiting for {} message for shard {}", action,
initializationInput.shardId(), e);
} catch (ExecutionException e) {
log.error(String.format("Failed to get status message for %s action for shard %s", action,
initializationInput.getShardId()), e);
log.error("Failed to get status message for {} action for shard {}", action,
initializationInput.shardId(), e);
} catch (TimeoutException e) {
log.error(String.format("Timedout to get status message for %s action for shard %s. Terminating...",
log.error("Timedout to get status message for {} action for shard {}. Terminating...",
action,
initializationInput.getShardId()),
initializationInput.shardId(),
e);
haltJvm(1);
}
@ -229,24 +227,24 @@ class MultiLangProtocol {
* @return Whether or not this operation succeeded.
*/
private boolean validateStatusMessage(StatusMessage statusMessage, String action) {
log.info("Received response " + statusMessage + " from subprocess while waiting for " + action
+ " while processing shard " + initializationInput.getShardId());
log.info("Received response {} from subprocess while waiting for {}"
+ " while processing shard {}", statusMessage, action, initializationInput.shardId());
return !(statusMessage == null || statusMessage.getResponseFor() == null || !statusMessage.getResponseFor()
.equals(action));
}
/**
* Attempts to checkpoint with the provided {@link IRecordProcessorCheckpointer} at the sequence number in the
* Attempts to checkpoint with the provided {@link RecordProcessorCheckpointer} at the sequence number in the
* provided {@link CheckpointMessage}. If no sequence number is provided, i.e. the sequence number is null, then
* this method will call {@link IRecordProcessorCheckpointer#checkpoint()}. The method returns a future representing
* this method will call {@link RecordProcessorCheckpointer#checkpoint()}. The method returns a future representing
* the attempt to write the result of this checkpoint attempt to the child process.
*
* @param checkpointMessage A checkpoint message.
* @param checkpointer A checkpointer.
* @return Whether or not this operation succeeded.
*/
private Future<Boolean> checkpoint(CheckpointMessage checkpointMessage, IRecordProcessorCheckpointer checkpointer) {
private Future<Boolean> checkpoint(CheckpointMessage checkpointMessage, RecordProcessorCheckpointer checkpointer) {
String sequenceNumber = checkpointMessage.getSequenceNumber();
Long subSequenceNumber = checkpointMessage.getSubSequenceNumber();
try {
@ -265,7 +263,7 @@ class MultiLangProtocol {
} else {
String message =
String.format("Was asked to checkpoint at %s but no checkpointer was provided for shard %s",
sequenceNumber, initializationInput.getShardId());
sequenceNumber, initializationInput.shardId());
log.error(message);
return this.messageWriter.writeCheckpointMessageWithError(sequenceNumber, subSequenceNumber,
new InvalidStateException(
@ -278,7 +276,7 @@ class MultiLangProtocol {
private String logCheckpointMessage(String sequenceNumber, Long subSequenceNumber) {
return String.format("Attempting to checkpoint shard %s @ sequence number %s, and sub sequence number %s",
initializationInput.getShardId(), sequenceNumber, subSequenceNumber);
initializationInput.shardId(), sequenceNumber, subSequenceNumber);
}
}

25
src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangRecordProcessorFactory.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangRecordProcessorFactory.java
View file

@ -16,21 +16,18 @@ package com.amazonaws.services.kinesis.multilang;
import java.util.concurrent.ExecutorService;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessorFactory;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.processor.ShardRecordProcessorFactory;
import software.amazon.kinesis.processor.ShardRecordProcessor;
/**
* Creates {@link MultiLangRecordProcessor}'s.
* Creates {@link MultiLangShardRecordProcessor}'s.
*/
public class MultiLangRecordProcessorFactory implements IRecordProcessorFactory {
private static final Log LOG = LogFactory.getLog(MultiLangRecordProcessorFactory.class);
@Slf4j
public class MultiLangRecordProcessorFactory implements ShardRecordProcessorFactory {
private static final String COMMAND_DELIMETER_REGEX = " +";
private final String command;
@ -66,12 +63,12 @@ public class MultiLangRecordProcessorFactory implements IRecordProcessorFactory
}
@Override
public IRecordProcessor createProcessor() {
LOG.debug(String.format("Creating new record processor for client executable: %s", command));
public ShardRecordProcessor shardRecordProcessor() {
log.debug("Creating new record processor for client executable: {}", command);
/*
* Giving ProcessBuilder the command as an array of Strings allows users to specify command line arguments.
*/
return new MultiLangRecordProcessor(new ProcessBuilder(commandArray), executorService, this.objectMapper,
return new MultiLangShardRecordProcessor(new ProcessBuilder(commandArray), executorService, this.objectMapper,
this.configuration);
}

92
src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangRecordProcessor.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangShardRecordProcessor.java
View file

@ -20,20 +20,19 @@ import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IShutdownNotificationAware;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownInput;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.lifecycle.events.LeaseLostInput;
import software.amazon.kinesis.lifecycle.events.ShardEndedInput;
import software.amazon.kinesis.lifecycle.events.ShutdownRequestedInput;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.ShutdownInput;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
/**
* A record processor that manages creating a child process that implements the multi language protocol and connecting
@ -41,9 +40,8 @@ import com.fasterxml.jackson.databind.ObjectMapper;
* that object when its corresponding {@link #initialize}, {@link #processRecords}, and {@link #shutdown} methods are
* called.
*/
public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNotificationAware {
private static final Log LOG = LogFactory.getLog(MultiLangRecordProcessor.class);
@Slf4j
public class MultiLangShardRecordProcessor implements ShardRecordProcessor {
private static final int EXIT_VALUE = 1;
/** Whether or not record processor initialization is successful. Defaults to false. */
@ -71,7 +69,7 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
@Override
public void initialize(InitializationInput initializationInput) {
try {
this.shardId = initializationInput.getShardId();
this.shardId = initializationInput.shardId();
try {
this.process = startProcess();
} catch (IOException e) {
@ -114,11 +112,33 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
}
@Override
public void shutdown(ShutdownInput shutdownInput) {
public void leaseLost(LeaseLostInput leaseLostInput) {
shutdown(ShutdownInput.builder().shutdownReason(ShutdownReason.LEASE_LOST).build());
}
@Override
public void shardEnded(ShardEndedInput shardEndedInput) {
shutdown(ShutdownInput.builder().shutdownReason(ShutdownReason.SHARD_END).checkpointer(shardEndedInput.checkpointer()).build());
}
@Override
public void shutdownRequested(ShutdownRequestedInput shutdownRequestedInput) {
log.info("Shutdown is requested.");
if (!initialized) {
log.info("Record processor was not initialized so no need to initiate a final checkpoint.");
return;
}
log.info("Requesting a checkpoint on shutdown notification.");
if (!protocol.shutdownRequested(shutdownRequestedInput.checkpointer())) {
log.error("Child process failed to complete shutdown notification.");
}
}
void shutdown(ShutdownInput shutdownInput) {
// In cases where KCL loses lease for the shard after creating record processor instance but before
// record processor initialize() is called, then shutdown() may be called directly before initialize().
if (!initialized) {
LOG.info("Record processor was not initialized and will not have a child process, "
log.info("Record processor was not initialized and will not have a child process, "
+ "so not invoking child process shutdown.");
this.state = ProcessState.SHUTDOWN;
return;
@ -126,13 +146,13 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
try {
if (ProcessState.ACTIVE.equals(this.state)) {
if (!protocol.shutdown(shutdownInput.getCheckpointer(), shutdownInput.getShutdownReason())) {
if (!protocol.shutdown(shutdownInput.checkpointer(), shutdownInput.shutdownReason())) {
throw new RuntimeException("Child process failed to shutdown");
}
childProcessShutdownSequence();
} else {
LOG.warn("Shutdown was called but this processor is already shutdown. Not doing anything.");
log.warn("Shutdown was called but this processor is already shutdown. Not doing anything.");
}
} catch (Throwable t) {
if (ProcessState.ACTIVE.equals(this.state)) {
@ -144,20 +164,6 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
}
}
@Override
public void shutdownRequested(IRecordProcessorCheckpointer checkpointer) {
LOG.info("Shutdown is requested.");
if (!initialized) {
LOG.info("Record processor was not initialized so no need to initiate a final checkpoint.");
return;
}
LOG.info("Requesting a checkpoint on shutdown notification.");
if (!protocol.shutdownRequested(checkpointer)) {
LOG.error("Child process failed to complete shutdown notification.");
}
}
/**
* Used to tell whether the processor has been shutdown already.
*/
@ -175,7 +181,7 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
* @param objectMapper
* An obejct mapper.
*/
MultiLangRecordProcessor(ProcessBuilder processBuilder, ExecutorService executorService,
MultiLangShardRecordProcessor(ProcessBuilder processBuilder, ExecutorService executorService,
ObjectMapper objectMapper, KinesisClientLibConfiguration configuration) {
this(processBuilder, executorService, objectMapper, new MessageWriter(), new MessageReader(),
new DrainChildSTDERRTask(), configuration);
@ -197,7 +203,7 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
* @param readSTDERRTask
* Error reader to read from child process's stderr
*/
MultiLangRecordProcessor(ProcessBuilder processBuilder, ExecutorService executorService, ObjectMapper objectMapper,
MultiLangShardRecordProcessor(ProcessBuilder processBuilder, ExecutorService executorService, ObjectMapper objectMapper,
MessageWriter messageWriter, MessageReader messageReader, DrainChildSTDERRTask readSTDERRTask,
KinesisClientLibConfiguration configuration) {
this.executorService = executorService;
@ -228,7 +234,7 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
messageWriter.close();
}
} catch (IOException e) {
LOG.error("Encountered exception while trying to close output stream.", e);
log.error("Encountered exception while trying to close output stream.", e);
}
// We should drain the STDOUT and STDERR of the child process. If we don't, the child process might remain
@ -245,9 +251,9 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
* sure that it exits before we finish.
*/
try {
LOG.info("Child process exited with value: " + process.waitFor());
log.info("Child process exited with value: {}", process.waitFor());
} catch (InterruptedException e) {
LOG.error("Interrupted before process finished exiting. Attempting to kill process.");
log.error("Interrupted before process finished exiting. Attempting to kill process.");
process.destroy();
}
@ -258,7 +264,7 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
try {
inputStream.close();
} catch (IOException e) {
LOG.error("Encountered exception while trying to close " + name + " stream.", e);
log.error("Encountered exception while trying to close {} stream.", name, e);
}
}
@ -273,7 +279,7 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
try {
future.get();
} catch (InterruptedException | ExecutionException e) {
LOG.error("Encountered error while " + whatThisFutureIsDoing + " for shard " + shardId, e);
log.error("Encountered error while {} for shard {}", whatThisFutureIsDoing, shardId, e);
}
}
@ -286,12 +292,12 @@ public class MultiLangRecordProcessor implements IRecordProcessor, IShutdownNoti
*/
private void stopProcessing(String message, Throwable reason) {
try {
LOG.error(message, reason);
log.error(message, reason);
if (!state.equals(ProcessState.SHUTDOWN)) {
childProcessShutdownSequence();
}
} catch (Throwable t) {
LOG.error("Encountered error while trying to shutdown", t);
log.error("Encountered error while trying to shutdown", t);
}
exit();
}

46
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/AWSCredentialsProviderPropertyValueDecoder.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/AWSCredentialsProviderPropertyValueDecoder.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,23 +12,22 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import java.lang.reflect.Constructor;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.lang.reflect.Constructor;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.AWSCredentialsProviderChain;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
/**
* Get AWSCredentialsProvider property.
*/
class AWSCredentialsProviderPropertyValueDecoder implements IPropertyValueDecoder<AWSCredentialsProvider> {
private static final Log LOG = LogFactory.getLog(AWSCredentialsProviderPropertyValueDecoder.class);
@Slf4j
class AWSCredentialsProviderPropertyValueDecoder implements IPropertyValueDecoder<AwsCredentialsProvider> {
private static final String AUTH_PREFIX = "com.amazonaws.auth.";
private static final String LIST_DELIMITER = ",";
private static final String ARG_DELIMITER = "|";
@ -42,17 +41,18 @@ class AWSCredentialsProviderPropertyValueDecoder implements IPropertyValueDecode
/**
* Get AWSCredentialsProvider property.
*
* @param value property value as String
* @param value
* property value as String
* @return corresponding variable in correct type
*/
@Override
public AWSCredentialsProvider decodeValue(String value) {
public AwsCredentialsProvider decodeValue(String value) {
if (value != null) {
List<String> providerNames = getProviderNames(value);
List<AWSCredentialsProvider> providers = getValidCredentialsProviders(providerNames);
AWSCredentialsProvider[] ps = new AWSCredentialsProvider[providers.size()];
List<AwsCredentialsProvider> providers = getValidCredentialsProviders(providerNames);
AwsCredentialsProvider[] ps = new AwsCredentialsProvider[providers.size()];
providers.toArray(ps);
return new AWSCredentialsProviderChain(ps);
return AwsCredentialsProviderChain.builder().credentialsProviders(ps).build();
} else {
throw new IllegalArgumentException("Property AWSCredentialsProvider is missing.");
}
@ -62,15 +62,15 @@ class AWSCredentialsProviderPropertyValueDecoder implements IPropertyValueDecode
* @return list of supported types
*/
@Override
public List<Class<AWSCredentialsProvider>> getSupportedTypes() {
return Arrays.asList(AWSCredentialsProvider.class);
public List<Class<AwsCredentialsProvider>> getSupportedTypes() {
return Arrays.asList(AwsCredentialsProvider.class);
}
/*
* Convert string list to a list of valid credentials providers.
*/
private static List<AWSCredentialsProvider> getValidCredentialsProviders(List<String> providerNames) {
List<AWSCredentialsProvider> credentialsProviders = new ArrayList<AWSCredentialsProvider>();
private static List<AwsCredentialsProvider> getValidCredentialsProviders(List<String> providerNames) {
List<AwsCredentialsProvider> credentialsProviders = new ArrayList<AwsCredentialsProvider>();
for (String providerName : providerNames) {
if (providerName.contains(ARG_DELIMITER)) {
String[] nameAndArgs = providerName.split("\\" + ARG_DELIMITER);
@ -79,17 +79,17 @@ class AWSCredentialsProviderPropertyValueDecoder implements IPropertyValueDecode
try {
Class<?> className = Class.forName(nameAndArgs[0]);
Constructor<?> c = className.getConstructor(argTypes);
credentialsProviders.add((AWSCredentialsProvider) c.newInstance(
Arrays.copyOfRange(nameAndArgs, 1, nameAndArgs.length)));
credentialsProviders.add((AwsCredentialsProvider) c
.newInstance(Arrays.copyOfRange(nameAndArgs, 1, nameAndArgs.length)));
} catch (Exception e) {
LOG.debug("Can't find any credentials provider matching " + providerName + ".");
log.debug("Can't find any credentials provider matching {}.", providerName);
}
} else {
try {
Class<?> className = Class.forName(providerName);
credentialsProviders.add((AWSCredentialsProvider) className.newInstance());
credentialsProviders.add((AwsCredentialsProvider) className.newInstance());
} catch (Exception e) {
LOG.debug("Can't find any credentials provider matching " + providerName + ".");
log.debug("Can't find any credentials provider matching {}.", providerName);
}
}
}

48
amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/BooleanPropertyValueDecoder.java Normal file
View file

@ -0,0 +1,48 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.config;
import java.util.Arrays;
import java.util.List;
/**
* Provide boolean property.
*/
class BooleanPropertyValueDecoder implements IPropertyValueDecoder<Boolean> {
/**
* Constructor.
*/
BooleanPropertyValueDecoder() {
}
/**
* @param value property value as String
* @return corresponding variable in correct type
*/
@Override
public Boolean decodeValue(String value) {
return Boolean.parseBoolean(value);
}
/**
* @return list of supported types
*/
@Override
public List<Class<Boolean>> getSupportedTypes() {
return Arrays.asList(boolean.class, Boolean.class);
}
}

23
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/BooleanPropertyValueDecoder.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/DatePropertyValueDecoder.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,20 +12,21 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import java.util.Arrays;
import java.util.Date;
import java.util.List;
/**
* Provide boolean property.
* Provide Date property.
*/
class BooleanPropertyValueDecoder implements IPropertyValueDecoder<Boolean> {
public class DatePropertyValueDecoder implements IPropertyValueDecoder<Date> {
/**
* Constructor.
*/
BooleanPropertyValueDecoder() {
DatePropertyValueDecoder() {
}
/**
@ -33,16 +34,20 @@ class BooleanPropertyValueDecoder implements IPropertyValueDecoder<Boolean> {
* @return corresponding variable in correct type
*/
@Override
public Boolean decodeValue(String value) {
return Boolean.parseBoolean(value);
public Date decodeValue(String value) {
try {
return new Date(Long.parseLong(value) * 1000L);
} catch (NumberFormatException e) {
throw new IllegalArgumentException("Date property value must be numeric.");
}
}
/**
* @return list of supported types
*/
@Override
public List<Class<Boolean>> getSupportedTypes() {
return Arrays.asList(boolean.class, Boolean.class);
public List<Class<Date>> getSupportedTypes() {
return Arrays.asList(Date.class);
}
}

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/IPropertyValueDecoder.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/IPropertyValueDecoder.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import java.util.List;

6
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/InitialPositionInStreamPropertyValueDecoder.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/InitialPositionInStreamPropertyValueDecoder.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,12 +12,12 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import java.util.Arrays;
import java.util.List;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
import software.amazon.kinesis.common.InitialPositionInStream;
/**
* Get an InitialiPosition enum property.

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/IntegerPropertyValueDecoder.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/IntegerPropertyValueDecoder.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import java.util.Arrays;
import java.util.List;

67
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/KinesisClientLibConfigurator.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/KinesisClientLibConfigurator.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,11 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import java.io.IOException;
import java.io.InputStream;
@ -28,12 +32,6 @@ import java.util.Properties;
import java.util.Set;
import java.util.UUID;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
/**
* KinesisClientLibConfigurator constructs a KinesisClientLibConfiguration from java properties file. The following
* three properties must be provided. 1) "applicationName" 2) "streamName" 3) "AWSCredentialsProvider"
@ -42,9 +40,8 @@ import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibC
* KinesisClientLibConfiguration and has a corresponding "with{variableName}" setter method, will be read in, and its
* value in properties file will be assigned to corresponding variable in KinesisClientLibConfiguration.
*/
@Slf4j
public class KinesisClientLibConfigurator {
private static final Log LOG = LogFactory.getLog(KinesisClientLibConfigurator.class);
private static final String PREFIX = "with";
// Required properties
@ -70,10 +67,9 @@ public class KinesisClientLibConfigurator {
new AWSCredentialsProviderPropertyValueDecoder(),
new StringPropertyValueDecoder(),
new InitialPositionInStreamPropertyValueDecoder(),
new ClientConfigurationPropertyValueDecoder(),
new SetPropertyValueDecoder());
classToDecoder = new Hashtable<Class<?>, IPropertyValueDecoder<?>>();
classToDecoder = new Hashtable<>();
for (IPropertyValueDecoder<?> getter : getters) {
for (Class<?> clazz : getter.getSupportedTypes()) {
/*
@ -83,10 +79,10 @@ public class KinesisClientLibConfigurator {
classToDecoder.put(clazz, getter);
}
}
nameToMethods = new Hashtable<String, List<Method>>();
nameToMethods = new Hashtable<>();
for (Method method : KinesisClientLibConfiguration.class.getMethods()) {
if (!nameToMethods.containsKey(method.getName())) {
nameToMethods.put(method.getName(), new ArrayList<Method>());
nameToMethods.put(method.getName(), new ArrayList<>());
}
nameToMethods.get(method.getName()).add(method);
}
@ -105,11 +101,11 @@ public class KinesisClientLibConfigurator {
// The three minimum required arguments for constructor are obtained first. They are all mandatory, all of them
// should be provided. If any of these three failed to be set, program will fail.
IPropertyValueDecoder<String> stringValueDecoder = new StringPropertyValueDecoder();
IPropertyValueDecoder<AWSCredentialsProvider> awsCPPropGetter =
IPropertyValueDecoder<AwsCredentialsProvider> awsCPPropGetter =
new AWSCredentialsProviderPropertyValueDecoder();
String applicationName = stringValueDecoder.decodeValue(properties.getProperty(PROP_APP_NAME));
String streamName = stringValueDecoder.decodeValue(properties.getProperty(PROP_STREAM_NAME));
AWSCredentialsProvider provider =
AwsCredentialsProvider provider =
awsCPPropGetter.decodeValue(properties.getProperty(PROP_CREDENTIALS_PROVIDER_KINESIS));
if (applicationName == null || applicationName.isEmpty()) {
@ -120,7 +116,7 @@ public class KinesisClientLibConfigurator {
}
// Decode the DynamoDB credentials provider if it exists. If not use the Kinesis credentials provider.
AWSCredentialsProvider providerDynamoDB;
AwsCredentialsProvider providerDynamoDB;
String propCredentialsProviderDynamoDBValue = properties.getProperty(PROP_CREDENTIALS_PROVIDER_DYNAMODB);
if (propCredentialsProviderDynamoDBValue == null) {
providerDynamoDB = provider;
@ -129,7 +125,7 @@ public class KinesisClientLibConfigurator {
}
// Decode the CloudWatch credentials provider if it exists. If not use the Kinesis credentials provider.
AWSCredentialsProvider providerCloudWatch;
AwsCredentialsProvider providerCloudWatch;
String propCredentialsProviderCloudWatchValue = properties.getProperty(PROP_CREDENTIALS_PROVIDER_CLOUDWATCH);
if (propCredentialsProviderCloudWatchValue == null) {
providerCloudWatch = provider;
@ -141,8 +137,8 @@ public class KinesisClientLibConfigurator {
String workerId = stringValueDecoder.decodeValue(properties.getProperty(PROP_WORKER_ID));
if (workerId == null || workerId.isEmpty()) {
workerId = UUID.randomUUID().toString();
LOG.info("Value of workerId is not provided in the properties. WorkerId is automatically "
+ "assigned as: " + workerId);
log.info("Value of workerId is not provided in the properties. WorkerId is automatically assigned as: {}",
workerId);
}
KinesisClientLibConfiguration config =
@ -203,38 +199,27 @@ public class KinesisClientLibConfigurator {
IPropertyValueDecoder<?> decoder = classToDecoder.get(paramType);
try {
method.invoke(config, decoder.decodeValue(propertyValue));
LOG.info(String.format("Successfully set property %s with value %s",
propertyKey,
propertyValue));
log.info("Successfully set property {} with value {}", propertyKey, propertyValue);
return;
} catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
// At this point, we really thought that we could call this method.
LOG.warn(String.format("Encountered an error while invoking method %s with value %s. "
+ "Exception was %s",
method,
propertyValue,
e));
log.warn("Encountered an error while invoking method %s with value {}. Exception was {}",
method, propertyValue, e);
} catch (UnsupportedOperationException e) {
LOG.warn(String.format("The property %s is not supported as type %s at this time.",
propertyKey,
paramType));
log.warn("The property {} is not supported as type {} at this time.", propertyKey,
paramType);
}
} else {
LOG.debug(String.format("No method for decoding parameters of type %s so method %s could not "
+ "be invoked.",
paramType,
method));
log.debug("No method for decoding parameters of type {} so method {} could not be invoked.",
paramType, method);
}
} else {
LOG.debug(String.format("Method %s doesn't look like it is appropriate for setting property %s. "
+ "Looking for something called %s.",
method,
propertyKey,
targetMethodName));
log.debug("Method {} doesn't look like it is appropriate for setting property {}. Looking for"
+ " something called {}.", method, propertyKey, targetMethodName);
}
}
} else {
LOG.debug(String.format("There was no appropriately named method for setting property %s.", propertyKey));
log.debug(String.format("There was no appropriately named method for setting property %s.", propertyKey));
}
}
}

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/LongPropertyValueDecoder.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/LongPropertyValueDecoder.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import java.util.Arrays;
import java.util.List;

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/SetPropertyValueDecoder.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/SetPropertyValueDecoder.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import java.util.Arrays;
import java.util.HashSet;

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/StringPropertyValueDecoder.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/config/StringPropertyValueDecoder.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.config;
package com.amazonaws.services.kinesis.multilang.config;
import java.util.Arrays;
import java.util.List;

9
src/main/java/com/amazonaws/services/kinesis/multilang/messages/CheckpointMessage.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/CheckpointMessage.java
View file

@ -15,13 +15,16 @@
package com.amazonaws.services.kinesis.multilang.messages;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.experimental.Accessors;
/**
* A checkpoint message is sent by the client's subprocess to indicate to the kcl processor that it should attempt to
* checkpoint. The processor sends back a checkpoint message as an acknowledgement that it attempted to checkpoint along
* with an error message which corresponds to the names of exceptions that a checkpointer can throw.
*/
@NoArgsConstructor
@Getter
@Setter
public class CheckpointMessage extends Message {
@ -41,12 +44,6 @@ public class CheckpointMessage extends Message {
*/
private String error;
/**
* Default constructor.
*/
public CheckpointMessage() {
}
/**
* Convenience constructor.
*

13
src/main/java/com/amazonaws/services/kinesis/multilang/messages/InitializeMessage.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/InitializeMessage.java
View file

@ -14,9 +14,9 @@
*/
package com.amazonaws.services.kinesis.multilang.messages;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import lombok.Getter;
import lombok.Setter;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
/**
* An initialize message is sent to the client's subprocess to indicate that it should perform its initialization steps.
@ -45,18 +45,17 @@ public class InitializeMessage extends Message {
/**
* Convenience constructor.
*
* @param shardId The shard id.
* @param initializationInput {@link InitializationInput}
*/
public InitializeMessage(InitializationInput initializationInput) {
this.shardId = initializationInput.getShardId();
if (initializationInput.getExtendedSequenceNumber() != null) {
this.sequenceNumber = initializationInput.getExtendedSequenceNumber().getSequenceNumber();
this.subSequenceNumber = initializationInput.getExtendedSequenceNumber().getSubSequenceNumber();
this.shardId = initializationInput.shardId();
if (initializationInput.extendedSequenceNumber() != null) {
this.sequenceNumber = initializationInput.extendedSequenceNumber().sequenceNumber();
this.subSequenceNumber = initializationInput.extendedSequenceNumber().subSequenceNumber();
} else {
this.sequenceNumber = null;
this.subSequenceNumber = null;
}
}
}

43
src/main/java/com/amazonaws/services/kinesis/multilang/messages/JsonFriendlyRecord.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/JsonFriendlyRecord.java
View file

@ -14,56 +14,47 @@
*/
package com.amazonaws.services.kinesis.multilang.messages;
import java.util.Date;
import java.time.Instant;
import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
import com.amazonaws.services.kinesis.model.Record;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.AllArgsConstructor;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.NonNull;
import lombok.Setter;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
/**
* Class for encoding Record objects to json. Needed because Records have byte buffers for their data field which causes
* problems for the json library we're using.
*/
@NoArgsConstructor
@AllArgsConstructor
@Getter
@Setter
@EqualsAndHashCode
@ToString
public class JsonFriendlyRecord {
private byte[] data;
private String partitionKey;
private String sequenceNumber;
private Date approximateArrivalTimestamp;
private Instant approximateArrivalTimestamp;
private Long subSequenceNumber;
public static String ACTION = "record";
/**
* Default Constructor.
*/
public JsonFriendlyRecord() {
}
/**
* Convenience constructor.
*
* @param record The record that this message will represent.
*/
public JsonFriendlyRecord(Record record) {
this.data = record.getData() == null ? null : record.getData().array();
this.partitionKey = record.getPartitionKey();
this.sequenceNumber = record.getSequenceNumber();
this.approximateArrivalTimestamp = record.getApproximateArrivalTimestamp();
if (record instanceof UserRecord) {
this.subSequenceNumber = ((UserRecord) record).getSubSequenceNumber();
} else {
this.subSequenceNumber = null;
}
public static JsonFriendlyRecord fromKinesisClientRecord(@NonNull final KinesisClientRecord record) {
byte[] data = record.data() == null ? null : record.data().array();
return new JsonFriendlyRecord(data, record.partitionKey(), record.sequenceNumber(),
record.approximateArrivalTimestamp(), record.subSequenceNumber());
}
@JsonProperty
public String getAction() {
return ACTION;
}
}

2
src/main/java/com/amazonaws/services/kinesis/multilang/messages/Message.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/Message.java
View file

@ -15,8 +15,8 @@
package com.amazonaws.services.kinesis.multilang.messages;
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.annotation.JsonSubTypes.Type;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.databind.ObjectMapper;
/**

12
src/main/java/com/amazonaws/services/kinesis/multilang/messages/ProcessRecordsMessage.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/ProcessRecordsMessage.java
View file

@ -17,10 +17,10 @@ package com.amazonaws.services.kinesis.multilang.messages;
import java.util.ArrayList;
import java.util.List;
import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
import com.amazonaws.services.kinesis.model.Record;
import lombok.Getter;
import lombok.Setter;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
/**
* A message to indicate to the client's process that it should process a list of records.
@ -52,10 +52,10 @@ public class ProcessRecordsMessage extends Message {
* the process records input to be sent to the child
*/
public ProcessRecordsMessage(ProcessRecordsInput processRecordsInput) {
this.millisBehindLatest = processRecordsInput.getMillisBehindLatest();
List<JsonFriendlyRecord> recordMessages = new ArrayList<JsonFriendlyRecord>();
for (Record record : processRecordsInput.getRecords()) {
recordMessages.add(new JsonFriendlyRecord(record));
this.millisBehindLatest = processRecordsInput.millisBehindLatest();
List<JsonFriendlyRecord> recordMessages = new ArrayList<>();
for (KinesisClientRecord record : processRecordsInput.records()) {
recordMessages.add(JsonFriendlyRecord.fromKinesisClientRecord(record));
}
this.setRecords(recordMessages);
}

44
src/main/java/com/amazonaws/services/kinesis/multilang/messages/ShutdownMessage.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/ShutdownMessage.java
View file

@ -14,11 +14,18 @@
*/
package com.amazonaws.services.kinesis.multilang.messages;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.experimental.Accessors;
import software.amazon.kinesis.lifecycle.ShutdownReason;
/**
* A message to indicate to the client's process that it should shutdown and then terminate.
*/
@NoArgsConstructor
@Getter
@Setter
public class ShutdownMessage extends Message {
/**
* The name used for the action field in {@link Message}.
@ -26,40 +33,13 @@ public class ShutdownMessage extends Message {
public static final String ACTION = "shutdown";
/**
* The reason for shutdown, e.g. TERMINATE or ZOMBIE
* The reason for shutdown, e.g. SHARD_END or LEASE_LOST
*/
private String reason;
/**
* Default constructor.
*/
public ShutdownMessage() {
}
/**
* Convenience constructor.
*
* @param reason The reason.
*/
public ShutdownMessage(ShutdownReason reason) {
if (reason == null) {
this.setReason(null);
} else {
this.setReason(String.valueOf(reason));
public ShutdownMessage(final ShutdownReason reason) {
if (reason != null) {
this.reason = String.valueOf(reason);
}
}
/**
* @return reason The reason.
*/
public String getReason() {
return reason;
}
/**
* @param reason The reason.
*/
public void setReason(String reason) {
this.reason = reason;
}
}

9
src/main/java/com/amazonaws/services/kinesis/multilang/messages/ShutdownRequestedMessage.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/ShutdownRequestedMessage.java
View file

@ -14,18 +14,15 @@
*/
package com.amazonaws.services.kinesis.multilang.messages;
import lombok.NoArgsConstructor;
/**
* A message to indicate to the client's process that shutdown is requested.
*/
@NoArgsConstructor
public class ShutdownRequestedMessage extends Message {
/**
* The name used for the action field in {@link Message}.
*/
public static final String ACTION = "shutdownRequested";
/**
* Convenience constructor.
*/
public ShutdownRequestedMessage() {
}
}

41
src/main/java/com/amazonaws/services/kinesis/multilang/messages/StatusMessage.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/messages/StatusMessage.java
View file

@ -14,9 +14,19 @@
*/
package com.amazonaws.services.kinesis.multilang.messages;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.experimental.Accessors;
/**
* A message sent by the client's process to indicate to the record processor that it completed a particular action.
*/
@NoArgsConstructor
@AllArgsConstructor
@Getter
@Setter
public class StatusMessage extends Message {
/**
* The name used for the action field in {@link Message}.
@ -27,35 +37,4 @@ public class StatusMessage extends Message {
* The name of the most recently received action.
*/
private String responseFor;
/**
* Default constructor.
*/
public StatusMessage() {
}
/**
* Convenience constructor.
*
* @param responseFor The response for.
*/
public StatusMessage(String responseFor) {
this.setResponseFor(responseFor);
}
/**
*
* @return The response for.
*/
public String getResponseFor() {
return responseFor;
}
/**
*
* @param responseFor The response for.
*/
public void setResponseFor(String responseFor) {
this.responseFor = responseFor;
}
}

0
src/main/java/com/amazonaws/services/kinesis/multilang/package-info.java → amazon-kinesis-client-multilang/src/main/java/com/amazonaws/services/kinesis/multilang/package-info.java
View file

713
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java → amazon-kinesis-client-multilang/src/main/java/software/amazon/kinesis/coordinator/KinesisClientLibConfiguration.java
View file

File diff suppressed because it is too large Load diff

26
amazon-kinesis-client-multilang/src/main/resources/logback.xml Normal file
View file

@ -0,0 +1,26 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
~ Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
~
~ Licensed under the Amazon Software License (the "License").
~ You may not use this file except in compliance with the License.
~ A copy of the License is located at
~
~ http://aws.amazon.com/asl/
~
~ or in the "license" file accompanying this file. This file is distributed
~ on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
~ express or implied. See the License for the specific language governing
~ permissions and limitations under the License.
-->
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d [%thread] %-5level %logger{36} [%mdc{ShardId:-NONE}] - %msg %n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>

22
src/test/java/com/amazonaws/services/kinesis/multilang/Matchers.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/Matchers.java
View file

@ -21,8 +21,8 @@ import org.hamcrest.Description;
import org.hamcrest.Matcher;
import org.hamcrest.TypeSafeDiagnosingMatcher;
import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
public class Matchers {
@ -36,19 +36,19 @@ public class Matchers {
private final Matcher<ExtendedSequenceNumber> sequenceNumberMatcher;
public InitializationInputMatcher(InitializationInput input) {
shardIdMatcher = equalTo(input.getShardId());
sequenceNumberMatcher = withSequence(input.getExtendedSequenceNumber());
shardIdMatcher = equalTo(input.shardId());
sequenceNumberMatcher = withSequence(input.extendedSequenceNumber());
}
@Override
protected boolean matchesSafely(final InitializationInput item, Description mismatchDescription) {
boolean matches = true;
if (!shardIdMatcher.matches(item.getShardId())) {
if (!shardIdMatcher.matches(item.shardId())) {
matches = false;
shardIdMatcher.describeMismatch(item.getShardId(), mismatchDescription);
shardIdMatcher.describeMismatch(item.shardId(), mismatchDescription);
}
if (!sequenceNumberMatcher.matches(item.getExtendedSequenceNumber())) {
if (!sequenceNumberMatcher.matches(item.extendedSequenceNumber())) {
matches = false;
sequenceNumberMatcher.describeMismatch(item, mismatchDescription);
}
@ -76,19 +76,19 @@ public class Matchers {
private final Matcher<Long> subSequenceNumberMatcher;
public ExtendedSequenceNumberMatcher(ExtendedSequenceNumber extendedSequenceNumber) {
sequenceNumberMatcher = equalTo(extendedSequenceNumber.getSequenceNumber());
subSequenceNumberMatcher = equalTo(extendedSequenceNumber.getSubSequenceNumber());
sequenceNumberMatcher = equalTo(extendedSequenceNumber.sequenceNumber());
subSequenceNumberMatcher = equalTo(extendedSequenceNumber.subSequenceNumber());
}
@Override
protected boolean matchesSafely(ExtendedSequenceNumber item, Description mismatchDescription) {
boolean matches = true;
if (!sequenceNumberMatcher.matches(item.getSequenceNumber())) {
if (!sequenceNumberMatcher.matches(item.sequenceNumber())) {
matches = false;
mismatchDescription.appendDescriptionOf(sequenceNumberMatcher);
}
if (!subSequenceNumberMatcher.matches(item.getSubSequenceNumber())) {
if (!subSequenceNumberMatcher.matches(item.subSequenceNumber())) {
matches = false;
mismatchDescription.appendDescriptionOf(subSequenceNumberMatcher);
}

7
src/test/java/com/amazonaws/services/kinesis/multilang/MessageReaderTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/MessageReaderTest.java
View file

@ -18,13 +18,11 @@ import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.mockito.Mockito;
@ -39,11 +37,6 @@ public class MessageReaderTest {
private static final String shardId = "shard-123";
@Before
public void setup() {
}
/*
* This line is based on the definition of the protocol for communication between the KCL record processor and
* the client's process.

40
src/test/java/com/amazonaws/services/kinesis/multilang/MessageWriterTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/MessageWriterTest.java
View file

@ -17,25 +17,26 @@ package com.amazonaws.services.kinesis.multilang;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.mockito.Mockito;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason;
import com.amazonaws.services.kinesis.model.Record;
import com.amazonaws.services.kinesis.multilang.messages.Message;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
public class MessageWriterTest {
private static final String shardId = "shard-123";
@ -74,7 +75,7 @@ public class MessageWriterTest {
@Test
public void writeInitializeMessageTest() throws IOException, InterruptedException, ExecutionException {
Future<Boolean> future = this.messageWriter.writeInitializeMessage(new InitializationInput().withShardId(shardId));
Future<Boolean> future = this.messageWriter.writeInitializeMessage(InitializationInput.builder().shardId(shardId).build());
future.get();
Mockito.verify(this.stream, Mockito.atLeastOnce()).write(Mockito.any(byte[].class), Mockito.anyInt(),
Mockito.anyInt());
@ -83,19 +84,12 @@ public class MessageWriterTest {
@Test
public void writeProcessRecordsMessageTest() throws IOException, InterruptedException, ExecutionException {
List<Record> records = new ArrayList<Record>() {
{
this.add(new Record() {
{
this.setData(ByteBuffer.wrap("kitten".getBytes()));
this.setPartitionKey("some cats");
this.setSequenceNumber("357234807854789057805");
}
});
this.add(new Record());
}
};
Future<Boolean> future = this.messageWriter.writeProcessRecordsMessage(new ProcessRecordsInput().withRecords(records));
List<KinesisClientRecord> records = Arrays.asList(
KinesisClientRecord.builder().data(ByteBuffer.wrap("kitten".getBytes())).partitionKey("some cats")
.sequenceNumber("357234807854789057805").build(),
KinesisClientRecord.builder().build()
);
Future<Boolean> future = this.messageWriter.writeProcessRecordsMessage(ProcessRecordsInput.builder().records(records).build());
future.get();
Mockito.verify(this.stream, Mockito.atLeastOnce()).write(Mockito.any(byte[].class), Mockito.anyInt(),
@ -105,7 +99,7 @@ public class MessageWriterTest {
@Test
public void writeShutdownMessageTest() throws IOException, InterruptedException, ExecutionException {
Future<Boolean> future = this.messageWriter.writeShutdownMessage(ShutdownReason.TERMINATE);
Future<Boolean> future = this.messageWriter.writeShutdownMessage(ShutdownReason.SHARD_END);
future.get();
Mockito.verify(this.stream, Mockito.atLeastOnce()).write(Mockito.any(byte[].class), Mockito.anyInt(),
@ -126,7 +120,7 @@ public class MessageWriterTest {
@Test
public void streamIOExceptionTest() throws IOException, InterruptedException, ExecutionException {
Mockito.doThrow(IOException.class).when(stream).flush();
Future<Boolean> initializeTask = this.messageWriter.writeInitializeMessage(new InitializationInput().withShardId(shardId));
Future<Boolean> initializeTask = this.messageWriter.writeInitializeMessage(InitializationInput.builder().shardId(shardId).build());
Boolean result = initializeTask.get();
Assert.assertNotNull(result);
Assert.assertFalse(result);
@ -139,7 +133,7 @@ public class MessageWriterTest {
messageWriter = new MessageWriter().initialize(stream, shardId, mapper, Executors.newCachedThreadPool());
try {
messageWriter.writeShutdownMessage(ShutdownReason.ZOMBIE);
messageWriter.writeShutdownMessage(ShutdownReason.LEASE_LOST);
Assert.fail("The mapper failed so no write method should be able to succeed.");
} catch (Exception e) {
// Note that this is different than the stream failing. The stream is expected to fail, so we handle it
@ -156,7 +150,7 @@ public class MessageWriterTest {
Assert.assertFalse(this.messageWriter.isOpen());
try {
// Any message should fail
this.messageWriter.writeInitializeMessage(new InitializationInput().withShardId(shardId));
this.messageWriter.writeInitializeMessage(InitializationInput.builder().shardId(shardId).build());
Assert.fail("MessageWriter should be closed and unable to write.");
} catch (IllegalStateException e) {
// This should happen.

71
src/test/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemonConfigTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemonConfigTest.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -15,71 +15,80 @@
package com.amazonaws.services.kinesis.multilang;
import static org.junit.Assert.assertNotNull;
import static org.mockito.Matchers.any;
import static org.mockito.Mockito.when;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.Properties;
import junit.framework.Assert;
import org.junit.Before;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.runners.MockitoJUnitRunner;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.services.kinesis.clientlibrary.config.KinesisClientLibConfigurator;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfigurator;
import junit.framework.Assert;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
@RunWith(MockitoJUnitRunner.class)
public class MultiLangDaemonConfigTest {
private static String FILENAME = "some.properties";
private KinesisClientLibConfigurator buildMockConfigurator() {
AWSCredentialsProvider credentialsProvider = Mockito.mock(AWSCredentialsProvider.class);
AWSCredentials creds = Mockito.mock(AWSCredentials.class);
Mockito.doReturn(creds).when(credentialsProvider).getCredentials();
Mockito.doReturn("cool-user").when(creds).getAWSAccessKeyId();
KinesisClientLibConfiguration kclConfig =
new KinesisClientLibConfiguration("cool-app", "cool-stream", credentialsProvider, "cool-worker");
KinesisClientLibConfigurator configurator = Mockito.mock(KinesisClientLibConfigurator.class);
Mockito.doReturn(kclConfig).when(configurator).getConfiguration(Mockito.any(Properties.class));
return configurator;
@Mock
private AwsCredentialsProvider credentialsProvider;
@Mock
private AwsCredentials creds;
@Mock
private KinesisClientLibConfigurator configurator;
@Before
public void setup() {
when(credentialsProvider.resolveCredentials()).thenReturn(creds);
when(creds.accessKeyId()).thenReturn("cool-user");
when(configurator.getConfiguration(any(Properties.class))).thenReturn(
new KinesisClientLibConfiguration("cool-app", "cool-stream", credentialsProvider, "cool-worker"));
}
// TODO: Fix test
@Ignore
@Test
public void constructorTest() throws IOException {
String PROPERTIES =
"executableName = randomEXE \n" + "applicationName = testApp \n" + "streamName = fakeStream \n"
+ "AWSCredentialsProvider = DefaultAWSCredentialsProviderChain\n"
String PROPERTIES = "executableName = randomEXE \n" + "applicationName = testApp \n"
+ "streamName = fakeStream \n" + "AWSCredentialsProvider = DefaultAWSCredentialsProviderChain\n"
+ "processingLanguage = malbolge";
ClassLoader classLoader = Mockito.mock(ClassLoader.class);
Mockito.doReturn(new ByteArrayInputStream(PROPERTIES.getBytes()))
.when(classLoader)
Mockito.doReturn(new ByteArrayInputStream(PROPERTIES.getBytes())).when(classLoader)
.getResourceAsStream(FILENAME);
MultiLangDaemonConfig deamonConfig = new MultiLangDaemonConfig(FILENAME, classLoader, buildMockConfigurator());
MultiLangDaemonConfig deamonConfig = new MultiLangDaemonConfig(FILENAME, classLoader, configurator);
assertNotNull(deamonConfig.getExecutorService());
assertNotNull(deamonConfig.getKinesisClientLibConfiguration());
assertNotNull(deamonConfig.getRecordProcessorFactory());
}
// TODO: Fix test
@Ignore
@Test
public void propertyValidation() {
String PROPERTIES_NO_EXECUTABLE_NAME =
"applicationName = testApp \n" + "streamName = fakeStream \n"
+ "AWSCredentialsProvider = DefaultAWSCredentialsProviderChain\n"
+ "processingLanguage = malbolge";
String PROPERTIES_NO_EXECUTABLE_NAME = "applicationName = testApp \n" + "streamName = fakeStream \n"
+ "AWSCredentialsProvider = DefaultAWSCredentialsProviderChain\n" + "processingLanguage = malbolge";
ClassLoader classLoader = Mockito.mock(ClassLoader.class);
Mockito.doReturn(new ByteArrayInputStream(PROPERTIES_NO_EXECUTABLE_NAME.getBytes()))
.when(classLoader)
Mockito.doReturn(new ByteArrayInputStream(PROPERTIES_NO_EXECUTABLE_NAME.getBytes())).when(classLoader)
.getResourceAsStream(FILENAME);
MultiLangDaemonConfig config;
try {
config = new MultiLangDaemonConfig(FILENAME, classLoader, buildMockConfigurator());
config = new MultiLangDaemonConfig(FILENAME, classLoader, configurator);
Assert.fail("Construction of the config should have failed due to property validation failing.");
} catch (IllegalArgumentException e) {
// Good

17
src/test/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemonTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemonTest.java
View file

@ -20,26 +20,23 @@ import java.util.concurrent.Executors;
import org.junit.Test;
import org.mockito.Mockito;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
public class MultiLangDaemonTest {
@Test
public void buildWorkerTest() {
// Mocking Kinesis creds
AWSCredentialsProvider provider = Mockito.mock(AWSCredentialsProvider.class);
Mockito.doReturn(Mockito.mock(AWSCredentials.class)).when(provider).getCredentials();
KinesisClientLibConfiguration configuration = new KinesisClientLibConfiguration( "Derp",
"Blurp",
provider,
AwsCredentialsProvider provider = Mockito.mock(AwsCredentialsProvider.class);
Mockito.doReturn(Mockito.mock(AwsCredentials.class)).when(provider).resolveCredentials();
KinesisClientLibConfiguration configuration = new KinesisClientLibConfiguration("Derp", "Blurp", provider,
"Worker");
MultiLangRecordProcessorFactory factory = Mockito.mock(MultiLangRecordProcessorFactory.class);
Mockito.doReturn(new String[] { "someExecutableName" }).when(factory).getCommandArray();
MultiLangDaemon daemon =
new MultiLangDaemon(configuration, factory, Executors.newCachedThreadPool());
MultiLangDaemon daemon = new MultiLangDaemon(configuration, factory, Executors.newCachedThreadPool());
}
@Test

120
src/test/java/com/amazonaws/services/kinesis/multilang/MultiLangProtocolTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/MultiLangProtocolTest.java
View file

@ -14,40 +14,6 @@
*/
package com.amazonaws.services.kinesis.multilang;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
import com.amazonaws.services.kinesis.model.Record;
import com.amazonaws.services.kinesis.multilang.messages.CheckpointMessage;
import com.amazonaws.services.kinesis.multilang.messages.Message;
import com.amazonaws.services.kinesis.multilang.messages.ProcessRecordsMessage;
import com.amazonaws.services.kinesis.multilang.messages.StatusMessage;
import com.google.common.util.concurrent.SettableFuture;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.runners.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
@ -61,10 +27,46 @@ import static org.mockito.Mockito.timeout;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.runners.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import com.amazonaws.services.kinesis.multilang.messages.CheckpointMessage;
import com.amazonaws.services.kinesis.multilang.messages.Message;
import com.amazonaws.services.kinesis.multilang.messages.ProcessRecordsMessage;
import com.amazonaws.services.kinesis.multilang.messages.StatusMessage;
import com.google.common.util.concurrent.SettableFuture;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
@RunWith(MockitoJUnitRunner.class)
public class MultiLangProtocolTest {
private static final List<KinesisClientRecord> EMPTY_RECORD_LIST = Collections.emptyList();
private static final List<Record> EMPTY_RECORD_LIST = Collections.emptyList();
@Mock
private MultiLangProtocol protocol;
@Mock
@ -73,17 +75,15 @@ public class MultiLangProtocolTest {
private MessageReader messageReader;
private String shardId;
@Mock
private IRecordProcessorCheckpointer checkpointer;
private RecordProcessorCheckpointer checkpointer;
@Mock
private KinesisClientLibConfiguration configuration;
@Before
public void setup() {
this.shardId = "shard-id-123";
protocol = new MultiLangProtocolForTesting(messageReader, messageWriter,
new InitializationInput().withShardId(shardId), configuration);
InitializationInput.builder().shardId(shardId).build(), configuration);
when(configuration.getTimeoutInSeconds()).thenReturn(Optional.empty());
}
@ -103,38 +103,45 @@ public class MultiLangProtocolTest {
@Test
public void initializeTest() throws InterruptedException, ExecutionException {
when(messageWriter
.writeInitializeMessage(argThat(Matchers.withInit(new InitializationInput().withShardId(shardId)))))
.thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(new StatusMessage("initialize"), Message.class));
.writeInitializeMessage(argThat(Matchers.withInit(InitializationInput.builder()
.shardId(shardId).build())))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("initialize"), Message.class));
assertThat(protocol.initialize(), equalTo(true));
}
@Test
public void processRecordsTest() throws InterruptedException, ExecutionException {
when(messageWriter.writeProcessRecordsMessage(any(ProcessRecordsInput.class))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(new StatusMessage("processRecords"), Message.class));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("processRecords"), Message.class));
assertThat(protocol.processRecords(new ProcessRecordsInput().withRecords(EMPTY_RECORD_LIST)), equalTo(true));
assertThat(protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST).build()),
equalTo(true));
}
@Test
public void shutdownTest() throws InterruptedException, ExecutionException {
when(messageWriter.writeShutdownMessage(any(ShutdownReason.class))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(new StatusMessage("shutdown"), Message.class));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("shutdown"), Message.class));
Mockito.doReturn(buildFuture(true)).when(messageWriter)
.writeShutdownMessage(any(ShutdownReason.class));
Mockito.doReturn(buildFuture(new StatusMessage("shutdown"))).when(messageReader).getNextMessageFromSTDOUT();
assertThat(protocol.shutdown(null, ShutdownReason.ZOMBIE), equalTo(true));
Mockito.doReturn(buildFuture(new StatusMessage("shutdown")))
.when(messageReader).getNextMessageFromSTDOUT();
assertThat(protocol.shutdown(null, ShutdownReason.LEASE_LOST), equalTo(true));
}
@Test
public void shutdownRequestedTest() {
when(messageWriter.writeShutdownRequestedMessage()).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(new StatusMessage("shutdownRequested"), Message.class));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("shutdownRequested"), Message.class));
Mockito.doReturn(buildFuture(true)).when(messageWriter)
.writeShutdownRequestedMessage();
Mockito.doReturn(buildFuture(new StatusMessage("shutdownRequested"))).when(messageReader).getNextMessageFromSTDOUT();
Mockito.doReturn(buildFuture(new StatusMessage("shutdownRequested")))
.when(messageReader).getNextMessageFromSTDOUT();
assertThat(protocol.shutdownRequested(null), equalTo(true));
}
@ -180,7 +187,8 @@ public class MultiLangProtocolTest {
}
}));
boolean result = protocol.processRecords(new ProcessRecordsInput().withRecords(EMPTY_RECORD_LIST).withCheckpointer(checkpointer));
boolean result = protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST)
.checkpointer(checkpointer).build());
assertThat(result, equalTo(true));
@ -198,7 +206,8 @@ public class MultiLangProtocolTest {
this.add(new StatusMessage("processRecords"));
}
}));
assertThat(protocol.processRecords(new ProcessRecordsInput().withRecords(EMPTY_RECORD_LIST).withCheckpointer(checkpointer)), equalTo(false));
assertThat(protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST)
.checkpointer(checkpointer).build()), equalTo(false));
}
@Test(expected = NullPointerException.class)
@ -210,19 +219,20 @@ public class MultiLangProtocolTest {
when(future.get(anyInt(), eq(TimeUnit.SECONDS))).thenThrow(TimeoutException.class);
protocol = new MultiLangProtocolForTesting(messageReader,
messageWriter,
new InitializationInput().withShardId(shardId),
InitializationInput.builder().shardId(shardId).build(),
configuration);
protocol.processRecords(new ProcessRecordsInput().withRecords(EMPTY_RECORD_LIST));
protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST).build());
}
@Test
public void waitForStatusMessageSuccessTest() {
when(messageWriter.writeProcessRecordsMessage(any(ProcessRecordsInput.class))).thenReturn(buildFuture(true));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(new StatusMessage("processRecords"), Message.class));
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(buildFuture(
new StatusMessage("processRecords"), Message.class));
when(configuration.getTimeoutInSeconds()).thenReturn(Optional.of(5));
assertTrue(protocol.processRecords(new ProcessRecordsInput().withRecords(EMPTY_RECORD_LIST)));
assertTrue(protocol.processRecords(ProcessRecordsInput.builder().records(EMPTY_RECORD_LIST).build()));
}
private class MultiLangProtocolForTesting extends MultiLangProtocol {

0
src/test/java/com/amazonaws/services/kinesis/multilang/ReadSTDERRTaskTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/ReadSTDERRTaskTest.java
View file

10
src/test/java/com/amazonaws/services/kinesis/multilang/StreamingRecordProcessorFactoryTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/StreamingShardRecordProcessorFactoryTest.java
View file

@ -14,17 +14,17 @@
*/
package com.amazonaws.services.kinesis.multilang;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import org.junit.Assert;
import org.junit.Test;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor;
import software.amazon.kinesis.processor.ShardRecordProcessor;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;
@RunWith(MockitoJUnitRunner.class)
public class StreamingRecordProcessorFactoryTest {
public class StreamingShardRecordProcessorFactoryTest {
@Mock
private KinesisClientLibConfiguration configuration;
@ -32,9 +32,9 @@ public class StreamingRecordProcessorFactoryTest {
@Test
public void createProcessorTest() {
MultiLangRecordProcessorFactory factory = new MultiLangRecordProcessorFactory("somecommand", null, configuration);
IRecordProcessor processor = factory.createProcessor();
ShardRecordProcessor processor = factory.shardRecordProcessor();
Assert.assertEquals("Should have constructed a StreamingRecordProcessor", MultiLangRecordProcessor.class,
Assert.assertEquals("Should have constructed a StreamingRecordProcessor", MultiLangShardRecordProcessor.class,
processor.getClass());
}
}

73
src/test/java/com/amazonaws/services/kinesis/multilang/StreamingRecordProcessorTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/StreamingShardRecordProcessorTest.java
View file

@ -14,18 +14,19 @@
*/
package com.amazonaws.services.kinesis.multilang;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IPreparedCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownInput;
import com.amazonaws.services.kinesis.model.Record;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.awssdk.services.kinesis.model.Record;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.processor.PreparedCheckpointer;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.ShutdownInput;
import com.amazonaws.services.kinesis.multilang.messages.InitializeMessage;
import com.amazonaws.services.kinesis.multilang.messages.Message;
import com.amazonaws.services.kinesis.multilang.messages.ProcessRecordsMessage;
@ -41,11 +42,12 @@ import org.mockito.Mockito;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.runners.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.ExecutionException;
@ -63,7 +65,7 @@ import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@RunWith(MockitoJUnitRunner.class)
public class StreamingRecordProcessorTest {
public class StreamingShardRecordProcessorTest {
private static final String shardId = "shard-123";
@ -72,7 +74,7 @@ public class StreamingRecordProcessorTest {
@Mock
private Future<Message> messageFuture;
private IRecordProcessorCheckpointer unimplementedCheckpointer = new IRecordProcessorCheckpointer() {
private RecordProcessorCheckpointer unimplementedCheckpointer = new RecordProcessorCheckpointer() {
@Override
public void checkpoint() throws KinesisClientLibDependencyException, InvalidStateException,
@ -102,32 +104,37 @@ public class StreamingRecordProcessorTest {
}
@Override
public IPreparedCheckpointer prepareCheckpoint()
public PreparedCheckpointer prepareCheckpoint()
throws KinesisClientLibDependencyException,
InvalidStateException, ThrottlingException, ShutdownException {
throw new UnsupportedOperationException();
}
@Override
public IPreparedCheckpointer prepareCheckpoint(Record record)
public PreparedCheckpointer prepareCheckpoint(Record record)
throws KinesisClientLibDependencyException,
InvalidStateException, ThrottlingException, ShutdownException {
throw new UnsupportedOperationException();
}
@Override
public IPreparedCheckpointer prepareCheckpoint(String sequenceNumber)
public PreparedCheckpointer prepareCheckpoint(String sequenceNumber)
throws KinesisClientLibDependencyException,
InvalidStateException, ThrottlingException, ShutdownException, IllegalArgumentException {
throw new UnsupportedOperationException();
}
@Override
public IPreparedCheckpointer prepareCheckpoint(String sequenceNumber, long subSequenceNumber)
public PreparedCheckpointer prepareCheckpoint(String sequenceNumber, long subSequenceNumber)
throws KinesisClientLibDependencyException,
InvalidStateException, ThrottlingException, ShutdownException, IllegalArgumentException {
throw new UnsupportedOperationException();
}
@Override
public Checkpointer checkpointer() {
throw new UnsupportedOperationException();
}
};
private MessageWriter messageWriter;
@ -136,7 +143,7 @@ public class StreamingRecordProcessorTest {
private MessageReader messageReader;
private MultiLangRecordProcessor recordProcessor;
private MultiLangShardRecordProcessor recordProcessor;
@Mock
private KinesisClientLibConfiguration configuration;
@ -157,7 +164,7 @@ public class StreamingRecordProcessorTest {
when(configuration.getTimeoutInSeconds()).thenReturn(Optional.empty());
recordProcessor =
new MultiLangRecordProcessor(new ProcessBuilder(), executor, new ObjectMapper(), messageWriter,
new MultiLangShardRecordProcessor(new ProcessBuilder(), executor, new ObjectMapper(), messageWriter,
messageReader, errorReader, configuration) {
// Just don't do anything when we exit.
@ -201,12 +208,15 @@ public class StreamingRecordProcessorTest {
when(messageFuture.get()).thenAnswer(answer);
when(messageReader.getNextMessageFromSTDOUT()).thenReturn(messageFuture);
List<Record> testRecords = new ArrayList<Record>();
List<KinesisClientRecord> testRecords = Collections.emptyList();
recordProcessor.initialize(new InitializationInput().withShardId(shardId));
recordProcessor.processRecords(new ProcessRecordsInput().withRecords(testRecords).withCheckpointer(unimplementedCheckpointer));
recordProcessor.processRecords(new ProcessRecordsInput().withRecords(testRecords).withCheckpointer(unimplementedCheckpointer));
recordProcessor.shutdown(new ShutdownInput().withCheckpointer(unimplementedCheckpointer).withShutdownReason(ShutdownReason.ZOMBIE));
recordProcessor.initialize(InitializationInput.builder().shardId(shardId).build());
recordProcessor.processRecords(ProcessRecordsInput.builder().records(testRecords)
.checkpointer(unimplementedCheckpointer).build());
recordProcessor.processRecords(ProcessRecordsInput.builder().records(testRecords)
.checkpointer(unimplementedCheckpointer).build());
recordProcessor.shutdown(ShutdownInput.builder().checkpointer(unimplementedCheckpointer)
.shutdownReason(ShutdownReason.LEASE_LOST).build());
}
@Test
@ -233,9 +243,10 @@ public class StreamingRecordProcessorTest {
phases(answer);
verify(messageWriter)
.writeInitializeMessage(argThat(Matchers.withInit(new InitializationInput().withShardId(shardId))));
.writeInitializeMessage(argThat(Matchers.withInit(
InitializationInput.builder().shardId(shardId).build())));
verify(messageWriter, times(2)).writeProcessRecordsMessage(any(ProcessRecordsInput.class));
verify(messageWriter).writeShutdownMessage(ShutdownReason.ZOMBIE);
verify(messageWriter).writeShutdownMessage(ShutdownReason.LEASE_LOST);
}
@Test
@ -264,10 +275,10 @@ public class StreamingRecordProcessorTest {
phases(answer);
verify(messageWriter).writeInitializeMessage(argThat(Matchers.withInit(new InitializationInput()
.withShardId(shardId))));
verify(messageWriter).writeInitializeMessage(argThat(Matchers.withInit(InitializationInput.builder()
.shardId(shardId).build())));
verify(messageWriter, times(2)).writeProcessRecordsMessage(any(ProcessRecordsInput.class));
verify(messageWriter, never()).writeShutdownMessage(ShutdownReason.ZOMBIE);
verify(messageWriter, never()).writeShutdownMessage(ShutdownReason.LEASE_LOST);
Assert.assertEquals(1, systemExitCount);
}
}

103
amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/config/AWSCredentialsProviderPropertyValueDecoderTest.java Normal file
View file

@ -0,0 +1,103 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.config;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
public class AWSCredentialsProviderPropertyValueDecoderTest {
private static final String TEST_ACCESS_KEY_ID = "123";
private static final String TEST_SECRET_KEY = "456";
private String credentialName1 = "com.amazonaws.services.kinesis.multilang.config.AWSCredentialsProviderPropertyValueDecoderTest$AlwaysSucceedCredentialsProvider";
private String credentialName2 = "com.amazonaws.services.kinesis.multilang.config.AWSCredentialsProviderPropertyValueDecoderTest$ConstructorCredentialsProvider";
private AWSCredentialsProviderPropertyValueDecoder decoder = new AWSCredentialsProviderPropertyValueDecoder();
@Test
public void testSingleProvider() {
AwsCredentialsProvider provider = decoder.decodeValue(credentialName1);
assertEquals(provider.getClass(), AwsCredentialsProviderChain.class);
assertEquals(provider.resolveCredentials().accessKeyId(), TEST_ACCESS_KEY_ID);
assertEquals(provider.resolveCredentials().secretAccessKey(), TEST_SECRET_KEY);
}
@Test
public void testTwoProviders() {
AwsCredentialsProvider provider = decoder.decodeValue(credentialName1 + "," + credentialName1);
assertEquals(provider.getClass(), AwsCredentialsProviderChain.class);
assertEquals(provider.resolveCredentials().accessKeyId(), TEST_ACCESS_KEY_ID);
assertEquals(provider.resolveCredentials().secretAccessKey(), TEST_SECRET_KEY);
}
@Test
public void testProfileProviderWithOneArg() {
AwsCredentialsProvider provider = decoder.decodeValue(credentialName2 + "|arg");
assertEquals(provider.getClass(), AwsCredentialsProviderChain.class);
assertEquals(provider.resolveCredentials().accessKeyId(), "arg");
assertEquals(provider.resolveCredentials().secretAccessKey(), "blank");
}
@Test
public void testProfileProviderWithTwoArgs() {
AwsCredentialsProvider provider = decoder.decodeValue(credentialName2 + "|arg1|arg2");
assertEquals(provider.getClass(), AwsCredentialsProviderChain.class);
assertEquals(provider.resolveCredentials().accessKeyId(), "arg1");
assertEquals(provider.resolveCredentials().secretAccessKey(), "arg2");
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProvider implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create(TEST_ACCESS_KEY_ID, TEST_SECRET_KEY);
}
}
/**
* This credentials provider needs a constructor call to instantiate it
*/
public static class ConstructorCredentialsProvider implements AwsCredentialsProvider {
private String arg1;
private String arg2;
public ConstructorCredentialsProvider(String arg1) {
this.arg1 = arg1;
this.arg2 = "blank";
}
public ConstructorCredentialsProvider(String arg1, String arg2) {
this.arg1 = arg1;
this.arg2 = arg2;
}
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create(arg1, arg2);
}
}
}

419
amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/config/KinesisClientLibConfiguratorTest.java Normal file
View file

@ -0,0 +1,419 @@
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.multilang.config;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.Optional;
import java.util.Set;
import org.apache.commons.lang.StringUtils;
import org.junit.Ignore;
import org.junit.Test;
import com.google.common.collect.ImmutableSet;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.kinesis.common.InitialPositionInStream;
import software.amazon.kinesis.coordinator.KinesisClientLibConfiguration;
import software.amazon.kinesis.metrics.MetricsLevel;
public class KinesisClientLibConfiguratorTest {
private String credentialName1 = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysSucceedCredentialsProvider";
private String credentialName2 = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysFailCredentialsProvider";
private String credentialNameKinesis = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysSucceedCredentialsProviderKinesis";
private String credentialNameDynamoDB = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysSucceedCredentialsProviderDynamoDB";
private String credentialNameCloudWatch = "com.amazonaws.services.kinesis.multilang.config.KinesisClientLibConfiguratorTest$AlwaysSucceedCredentialsProviderCloudWatch";
private KinesisClientLibConfigurator configurator = new KinesisClientLibConfigurator();
@Test
public void testWithBasicSetup() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = " + credentialName1, "workerId = 123" }, '\n'));
assertEquals(config.getApplicationName(), "b");
assertEquals(config.getStreamName(), "a");
assertEquals(config.getWorkerIdentifier(), "123");
assertEquals(config.getMaxGetRecordsThreadPool(), Optional.empty());
assertEquals(config.getRetryGetRecordsInSeconds(), Optional.empty());
}
@Test
public void testWithLongVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "applicationName = app",
"streamName = 123", "AWSCredentialsProvider = " + credentialName1 + ", " + credentialName2,
"workerId = 123", "failoverTimeMillis = 100", "shardSyncIntervalMillis = 500" }, '\n'));
assertEquals(config.getApplicationName(), "app");
assertEquals(config.getStreamName(), "123");
assertEquals(config.getWorkerIdentifier(), "123");
assertEquals(config.getFailoverTimeMillis(), 100);
assertEquals(config.getShardSyncIntervalMillis(), 500);
}
@Test
public void testWithUnsupportedClientConfigurationVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(
new String[] { "AWSCredentialsProvider = " + credentialName1 + ", " + credentialName2, "workerId = id",
"kinesisClientConfig = {}", "streamName = stream", "applicationName = b" },
'\n'));
assertEquals(config.getApplicationName(), "b");
assertEquals(config.getStreamName(), "stream");
assertEquals(config.getWorkerIdentifier(), "id");
// by setting the configuration there is no effect on kinesisClientConfiguration variable.
}
@Test
public void testWithIntVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = kinesis",
"AWSCredentialsProvider = " + credentialName2 + ", " + credentialName1, "workerId = w123",
"maxRecords = 10", "metricsMaxQueueSize = 20", "applicationName = kinesis",
"retryGetRecordsInSeconds = 2", "maxGetRecordsThreadPool = 1" }, '\n'));
assertEquals(config.getApplicationName(), "kinesis");
assertEquals(config.getStreamName(), "kinesis");
assertEquals(config.getWorkerIdentifier(), "w123");
assertEquals(config.getMaxRecords(), 10);
assertEquals(config.getMetricsMaxQueueSize(), 20);
assertEquals(config.getRetryGetRecordsInSeconds(), Optional.of(2));
assertEquals(config.getMaxGetRecordsThreadPool(), Optional.of(1));
}
@Test
public void testWithBooleanVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD, " + credentialName1, "workerId = 0",
"cleanupLeasesUponShardCompletion = false", "validateSequenceNumberBeforeCheckpointing = true" },
'\n'));
assertEquals(config.getApplicationName(), "b");
assertEquals(config.getStreamName(), "a");
assertEquals(config.getWorkerIdentifier(), "0");
assertFalse(config.shouldCleanupLeasesUponShardCompletion());
assertTrue(config.shouldValidateSequenceNumberBeforeCheckpointing());
}
@Test
public void testWithStringVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 1",
"kinesisEndpoint = https://kinesis", "metricsLevel = SUMMARY" }, '\n'));
assertEquals(config.getWorkerIdentifier(), "1");
assertEquals(config.getKinesisEndpoint(), "https://kinesis");
assertEquals(config.getMetricsLevel(), MetricsLevel.SUMMARY);
}
@Test
public void testWithSetVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 1",
"metricsEnabledDimensions = ShardId, WorkerIdentifier" }, '\n'));
Set<String> expectedMetricsEnabledDimensions = ImmutableSet.<String> builder()
.add("ShardId", "WorkerIdentifier")
.addAll(KinesisClientLibConfiguration.METRICS_ALWAYS_ENABLED_DIMENSIONS).build();
assertEquals(config.getMetricsEnabledDimensions(), expectedMetricsEnabledDimensions);
}
@Test
public void testWithInitialPositionInStreamVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 123",
"initialPositionInStream = TriM_Horizon" }, '\n'));
assertEquals(config.getInitialPositionInStream(), InitialPositionInStream.TRIM_HORIZON);
}
@Test
public void testSkippingNonKCLVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 123",
"initialPositionInStream = TriM_Horizon", "abc = 1" }, '\n'));
assertEquals(config.getApplicationName(), "b");
assertEquals(config.getStreamName(), "a");
assertEquals(config.getWorkerIdentifier(), "123");
assertEquals(config.getInitialPositionInStream(), InitialPositionInStream.TRIM_HORIZON);
}
@Test
public void testEmptyOptionalVariables() {
KinesisClientLibConfiguration config = getConfiguration(StringUtils.join(new String[] { "streamName = a",
"applicationName = b", "AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 123",
"initialPositionInStream = TriM_Horizon", "maxGetRecordsThreadPool = 1" }, '\n'));
assertEquals(config.getMaxGetRecordsThreadPool(), Optional.of(1));
assertEquals(config.getRetryGetRecordsInSeconds(), Optional.empty());
}
@Test
public void testWithZeroValue() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = ABCD," + credentialName1, "workerId = 123",
"initialPositionInStream = TriM_Horizon", "maxGetRecordsThreadPool = 0",
"retryGetRecordsInSeconds = 0" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
try {
configurator.getConfiguration(input);
} catch (Exception e) {
fail("Don't expect to fail on invalid variable value");
}
}
@Test
public void testWithInvalidIntValue() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = " + credentialName1, "workerId = 123", "failoverTimeMillis = 100nf" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
try {
configurator.getConfiguration(input);
} catch (Exception e) {
fail("Don't expect to fail on invalid variable value");
}
}
@Test
public void testWithNegativeIntValue() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = " + credentialName1, "workerId = 123", "failoverTimeMillis = -12" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
configurator.getConfiguration(input);
} catch (Exception e) {
fail("Don't expect to fail on invalid variable value");
}
}
@Test
public void testWithMissingCredentialsProvider() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b", "workerId = 123",
"failoverTimeMillis = 100", "shardSyncIntervalMillis = 500" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
configurator.getConfiguration(input);
fail("expect failure with no credentials provider variables");
} catch (Exception e) {
// succeed
}
}
@Test
public void testWithMissingWorkerId() {
String test = StringUtils.join(
new String[] { "streamName = a", "applicationName = b", "AWSCredentialsProvider = " + credentialName1,
"failoverTimeMillis = 100", "shardSyncIntervalMillis = 500" },
'\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
// if workerId is not provided, configurator should assign one for it automatically
assertNotNull(config.getWorkerIdentifier());
assertFalse(config.getWorkerIdentifier().isEmpty());
}
@Test
public void testWithMissingStreamName() {
String test = StringUtils.join(new String[] { "applicationName = b",
"AWSCredentialsProvider = " + credentialName1, "workerId = 123", "failoverTimeMillis = 100" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
configurator.getConfiguration(input);
fail("expect failure with no stream name variables");
} catch (Exception e) {
// succeed
}
}
@Test
public void testWithMissingApplicationName() {
String test = StringUtils.join(new String[] { "streamName = a", "AWSCredentialsProvider = " + credentialName1,
"workerId = 123", "failoverTimeMillis = 100" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
configurator.getConfiguration(input);
fail("expect failure with no application variables");
} catch (Exception e) {
// succeed
}
}
@Test
public void testWithAWSCredentialsFailed() {
String test = StringUtils.join(
new String[] { "streamName = a", "applicationName = b", "AWSCredentialsProvider = " + credentialName2,
"failoverTimeMillis = 100", "shardSyncIntervalMillis = 500" },
'\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
try {
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
config.getKinesisCredentialsProvider().resolveCredentials();
fail("expect failure with wrong credentials provider");
} catch (Exception e) {
// succeed
}
}
// TODO: fix this test
@Test
@Ignore
public void testWithDifferentAWSCredentialsForDynamoDBAndCloudWatch() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = " + credentialNameKinesis,
"AWSCredentialsProviderDynamoDB = " + credentialNameDynamoDB,
"AWSCredentialsProviderCloudWatch = " + credentialNameCloudWatch, "failoverTimeMillis = 100",
"shardSyncIntervalMillis = 500" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
try {
config.getKinesisCredentialsProvider().resolveCredentials();
} catch (Exception e) {
fail("Kinesis credential providers should not fail.");
}
try {
config.getDynamoDBCredentialsProvider().resolveCredentials();
} catch (Exception e) {
fail("DynamoDB credential providers should not fail.");
}
try {
config.getCloudWatchCredentialsProvider().resolveCredentials();
} catch (Exception e) {
fail("CloudWatch credential providers should not fail.");
}
}
// TODO: fix this test
@Test
@Ignore
public void testWithDifferentAWSCredentialsForDynamoDBAndCloudWatchFailed() {
String test = StringUtils.join(new String[] { "streamName = a", "applicationName = b",
"AWSCredentialsProvider = " + credentialNameKinesis,
"AWSCredentialsProviderDynamoDB = " + credentialName1,
"AWSCredentialsProviderCloudWatch = " + credentialName1, "failoverTimeMillis = 100",
"shardSyncIntervalMillis = 500" }, '\n');
InputStream input = new ByteArrayInputStream(test.getBytes());
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
// separate input stream with getConfiguration to explicitly catch exception from the getConfiguration statement
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
try {
config.getKinesisCredentialsProvider().resolveCredentials();
} catch (Exception e) {
fail("Kinesis credential providers should not fail.");
}
try {
config.getDynamoDBCredentialsProvider().resolveCredentials();
fail("DynamoDB credential providers should fail.");
} catch (Exception e) {
// succeed
}
try {
config.getCloudWatchCredentialsProvider().resolveCredentials();
fail("CloudWatch credential providers should fail.");
} catch (Exception e) {
// succeed
}
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProvider implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return null;
}
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProviderKinesis implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create("", "");
}
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProviderDynamoDB implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create("", "");
}
}
/**
* This credentials provider will always succeed
*/
public static class AlwaysSucceedCredentialsProviderCloudWatch implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create("", "");
}
}
/**
* This credentials provider will always fail
*/
public static class AlwaysFailCredentialsProvider implements AwsCredentialsProvider {
@Override
public AwsCredentials resolveCredentials() {
throw new IllegalArgumentException();
}
}
private KinesisClientLibConfiguration getConfiguration(String configString) {
InputStream input = new ByteArrayInputStream(configString.getBytes());
KinesisClientLibConfiguration config = configurator.getConfiguration(input);
return config;
}
}

52
src/test/java/com/amazonaws/services/kinesis/multilang/messages/MessageTest.java → amazon-kinesis-client-multilang/src/test/java/com/amazonaws/services/kinesis/multilang/messages/MessageTest.java
View file

@ -15,46 +15,52 @@
package com.amazonaws.services.kinesis.multilang.messages;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Collections;
import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
import org.junit.Assert;
import org.junit.Test;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason;
import com.amazonaws.services.kinesis.model.Record;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import software.amazon.kinesis.lifecycle.events.InitializationInput;
import software.amazon.kinesis.lifecycle.events.ProcessRecordsInput;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.retrieval.KinesisClientRecord;
public class MessageTest {
@Test
public void toStringTest() {
Message[] messages =
new Message[] { new CheckpointMessage("1234567890", 0L, null), new InitializeMessage(new InitializationInput().withShardId("shard-123")),
new ProcessRecordsMessage(new ProcessRecordsInput().withRecords(new ArrayList<Record>() {
{
this.add(new Record() {
{
this.withData(ByteBuffer.wrap("cat".getBytes()));
this.withPartitionKey("cat");
this.withSequenceNumber("555");
}
});
}
})), new ShutdownMessage(ShutdownReason.ZOMBIE), new StatusMessage("processRecords"),
new InitializeMessage(), new ProcessRecordsMessage(), new ShutdownRequestedMessage() };
Message[] messages = new Message[]{
new CheckpointMessage("1234567890", 0L, null),
new InitializeMessage(InitializationInput.builder().shardId("shard-123").build()),
new ProcessRecordsMessage(ProcessRecordsInput.builder()
.records(Collections.singletonList(
KinesisClientRecord.builder()
.data(ByteBuffer.wrap("cat".getBytes()))
.partitionKey("cat")
.sequenceNumber("555")
.build()))
.build()),
new ShutdownMessage(ShutdownReason.LEASE_LOST),
new StatusMessage("processRecords"),
new InitializeMessage(),
new ProcessRecordsMessage(),
new ShutdownRequestedMessage()
};
// TODO: fix this
for (int i = 0; i < messages.length; i++) {
System.out.println(messages[i].toString());
Assert.assertTrue("Each message should contain the action field", messages[i].toString().contains("action"));
}
// Hit this constructor
JsonFriendlyRecord defaultJsonFriendlyRecord = new JsonFriendlyRecord();
Assert.assertNull(defaultJsonFriendlyRecord.getPartitionKey());
Assert.assertNull(defaultJsonFriendlyRecord.getData());
Assert.assertNull(defaultJsonFriendlyRecord.getSequenceNumber());
KinesisClientRecord defaultJsonFriendlyRecord = KinesisClientRecord.builder().build();
Assert.assertNull(defaultJsonFriendlyRecord.partitionKey());
Assert.assertNull(defaultJsonFriendlyRecord.data());
Assert.assertNull(defaultJsonFriendlyRecord.sequenceNumber());
Assert.assertNull(new ShutdownMessage(null).getReason());
// Hit the bad object mapping path

26
amazon-kinesis-client-multilang/src/test/resources/logback.xml Normal file
View file

@ -0,0 +1,26 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
~ Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
~
~ Licensed under the Amazon Software License (the "License").
~ You may not use this file except in compliance with the License.
~ A copy of the License is located at
~
~ http://aws.amazon.com/asl/
~
~ or in the "license" file accompanying this file. This file is distributed
~ on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
~ express or implied. See the License for the specific language governing
~ permissions and limitations under the License.
-->
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d [%thread] %-5level %logger{36} [%mdc{ShardId:-NONE}] - %msg %n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>

337
amazon-kinesis-client/pom.xml Normal file
View file

@ -0,0 +1,337 @@
<!--
~ Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
~
~ Licensed under the Amazon Software License (the "License").
~ You may not use this file except in compliance with the License.
~ A copy of the License is located at
~
~ http://aws.amazon.com/asl/
~
~ or in the "license" file accompanying this file. This file is distributed
~ on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
~ express or implied. See the License for the specific language governing
~ permissions and limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-client-pom</artifactId>
<version>2.0.0</version>
</parent>
<artifactId>amazon-kinesis-client</artifactId>
<packaging>jar</packaging>
<name>Amazon Kinesis Client Library for Java</name>
<description>The Amazon Kinesis Client Library for Java enables Java developers to easily consume and process data
from Amazon Kinesis.
</description>
<url>https://aws.amazon.com/kinesis</url>
<scm>
<url>https://github.com/awslabs/amazon-kinesis-client.git</url>
</scm>
<licenses>
<license>
<name>Amazon Software License</name>
<url>https://aws.amazon.com/asl</url>
<distribution>repo</distribution>
</license>
</licenses>
<properties>
<aws-java-sdk.version>1.11.272</aws-java-sdk.version>
<awssdk.version>2.0.0</awssdk.version>
<sqlite4java.version>1.0.392</sqlite4java.version>
<sqlite4java.native>libsqlite4java</sqlite4java.native>
<sqlite4java.libpath>${project.build.directory}/test-lib</sqlite4java.libpath>
<slf4j.version>1.7.25</slf4j.version>
</properties>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>kinesis</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>dynamodb</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>cloudwatch</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.6.1</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.7</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>io.reactivex.rxjava2</groupId>
<artifactId>rxjava</artifactId>
<version>2.1.14</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.16.20</version>
<scope>provided</scope>
</dependency>
<!-- Test -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.19</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
<!--<dependency>-->
<!--<groupId>com.amazonaws</groupId>-->
<!--<artifactId>DynamoDBLocal</artifactId>-->
<!--<version>1.11.86</version>-->
<!--<scope>test</scope>-->
<!--</dependency>-->
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.1.7</version>
<scope>test</scope>
</dependency>
</dependencies>
<!--<repositories>-->
<!--<repository>-->
<!--<id>dynamodblocal</id>-->
<!--<name>AWS DynamoDB Local Release Repository</name>-->
<!--<url>https://s3-us-west-2.amazonaws.com/dynamodb-local/release</url>-->
<!--</repository>-->
<!--</repositories>-->
<developers>
<developer>
<id>amazonwebservices</id>
<organization>Amazon Web Services</organization>
<organizationUrl>https://aws.amazon.com</organizationUrl>
<roles>
<role>developer</role>
</roles>
</developer>
</developers>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<excludes>
<exclude>**/*IntegrationTest.java</exclude>
</excludes>
<systemProperties>
<property>
<name>sqlite4java.library.path</name>
<value>${sqlite4java.libpath}</value>
</property>
</systemProperties>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<includes>
<include>**/*IntegrationTest.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy</id>
<phase>test-compile</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<!-- Mac OS X -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>${sqlite4java.native}-osx</artifactId>
<version>${sqlite4java.version}</version>
<type>dylib</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
<!-- Linux -->
<!-- i386 -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>${sqlite4java.native}-linux-i386</artifactId>
<version>${sqlite4java.version}</version>
<type>so</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
<!-- amd64 -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>${sqlite4java.native}-linux-amd64</artifactId>
<version>${sqlite4java.version}</version>
<type>so</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
<!-- Windows -->
<!-- x86 -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>sqlite4java-win32-x86</artifactId>
<version>${sqlite4java.version}</version>
<type>dll</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
<!-- x64 -->
<artifactItem>
<groupId>com.almworks.sqlite4java</groupId>
<artifactId>sqlite4java-win32-x64</artifactId>
<version>${sqlite4java.version}</version>
<type>dll</type>
<overWrite>true</overWrite>
<outputDirectory>${sqlite4java.libpath}</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.3</version>
<configuration>
<excludePackageNames>com.amazonaws.services.kinesis.producer.protobuf</excludePackageNames>
</configuration>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>disable-java8-doclint</id>
<activation>
<jdk>[1.8,)</jdk>
</activation>
<properties>
<additionalparam>-Xdoclint:none</additionalparam>
</properties>
</profile>
</profiles>
</project>

26
amazon-kinesis-client/src/main/java/software/amazon/kinesis/annotations/KinesisClientInternalApi.java Normal file
View file

@ -0,0 +1,26 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.annotations;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
/**
* Any class/method/variable marked with this annotation is subject to breaking changes between minor releases.
*/
@Retention(RetentionPolicy.CLASS)
public @interface KinesisClientInternalApi {
}

43
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/Checkpoint.java Normal file
View file

@ -0,0 +1,43 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import lombok.Data;
import lombok.experimental.Accessors;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* A class encapsulating the 2 pieces of state stored in a checkpoint.
*/
@Data
@Accessors(fluent = true)
public class Checkpoint {
private final ExtendedSequenceNumber checkpoint;
private final ExtendedSequenceNumber pendingCheckpoint;
/**
* Constructor.
*
* @param checkpoint the checkpoint sequence number - cannot be null or empty.
* @param pendingCheckpoint the pending checkpoint sequence number - can be null.
*/
public Checkpoint(final ExtendedSequenceNumber checkpoint, final ExtendedSequenceNumber pendingCheckpoint) {
if (checkpoint == null || checkpoint.sequenceNumber().isEmpty()) {
throw new IllegalArgumentException("Checkpoint cannot be null or empty");
}
this.checkpoint = checkpoint;
this.pendingCheckpoint = pendingCheckpoint;
}
}

16
src/test/java/com/amazonaws/services/dynamodbv2/streamsadapter/AmazonDynamoDBStreamsAdapterClient.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/CheckpointConfig.java
View file

@ -13,14 +13,18 @@
* permissions and limitations under the License.
*/
package com.amazonaws.services.dynamodbv2.streamsadapter;
package software.amazon.kinesis.checkpoint;
import com.amazonaws.services.kinesis.AmazonKinesis;
import com.amazonaws.services.kinesis.AmazonKinesisClient;
import lombok.Data;
import lombok.experimental.Accessors;
import software.amazon.kinesis.checkpoint.dynamodb.DynamoDBCheckpointFactory;
/**
* This class is only used for testing purposes, to make sure that the correct calls are made while using DynamoDB
* streams.
* Used by the KCL to manage checkpointing.
*/
public class AmazonDynamoDBStreamsAdapterClient extends AmazonKinesisClient {
@Data
@Accessors(fluent = true)
public class CheckpointConfig {
private CheckpointFactory checkpointFactory = new DynamoDBCheckpointFactory();
}

27
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/CheckpointFactory.java Normal file
View file

@ -0,0 +1,27 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.processor.Checkpointer;
/**
*
*/
public interface CheckpointFactory {
Checkpointer createCheckpointer(LeaseCoordinator leaseCoordinator, LeaseRefresher leaseRefresher);
}

20
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/DoesNothingPreparedCheckpointer.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/DoesNothingPreparedCheckpointer.java
View file

@ -12,17 +12,17 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.checkpoint;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IPreparedCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.kinesis.processor.PreparedCheckpointer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* A special IPreparedCheckpointer that does nothing, which can be used when preparing a checkpoint at the current
* A special PreparedCheckpointer that does nothing, which can be used when preparing a checkpoint at the current
* checkpoint sequence number where it is never necessary to do another checkpoint.
* This simplifies programming by preventing application developers from having to reason about whether
* their application has processed records before calling prepareCheckpoint
@ -32,7 +32,7 @@ import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber
* initialized, processes 0 records, then calls prepareCheckpoint(). The value in the table is the same, so there's
* no reason to overwrite it with another copy of itself.
*/
public class DoesNothingPreparedCheckpointer implements IPreparedCheckpointer {
public class DoesNothingPreparedCheckpointer implements PreparedCheckpointer {
private final ExtendedSequenceNumber sequenceNumber;
@ -48,7 +48,7 @@ public class DoesNothingPreparedCheckpointer implements IPreparedCheckpointer {
* {@inheritDoc}
*/
@Override
public ExtendedSequenceNumber getPendingCheckpoint() {
public ExtendedSequenceNumber pendingCheckpoint() {
return sequenceNumber;
}

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/checkpoint/SentinelCheckpoint.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/SentinelCheckpoint.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.checkpoint;
package software.amazon.kinesis.checkpoint;
/**
* Enumeration of the sentinel values of checkpoints.

32
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PreparedCheckpointer.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/ShardPreparedCheckpointer.java
View file

@ -12,25 +12,25 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.checkpoint;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IPreparedCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.kinesis.processor.PreparedCheckpointer;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Objects of this class are prepared to checkpoint at a specific sequence number. They use an
* IRecordProcessorCheckpointer to do the actual checkpointing, so their checkpoint is subject to the same 'didn't go
* RecordProcessorCheckpointer to do the actual checkpointing, so their checkpoint is subject to the same 'didn't go
* backwards' validation as a normal checkpoint.
*/
public class PreparedCheckpointer implements IPreparedCheckpointer {
public class ShardPreparedCheckpointer implements PreparedCheckpointer {
private final ExtendedSequenceNumber pendingCheckpointSequenceNumber;
private final IRecordProcessorCheckpointer checkpointer;
private final RecordProcessorCheckpointer checkpointer;
/**
* Constructor.
@ -38,8 +38,8 @@ public class PreparedCheckpointer implements IPreparedCheckpointer {
* @param pendingCheckpointSequenceNumber sequence number to checkpoint at
* @param checkpointer checkpointer to use
*/
public PreparedCheckpointer(ExtendedSequenceNumber pendingCheckpointSequenceNumber,
IRecordProcessorCheckpointer checkpointer) {
public ShardPreparedCheckpointer(ExtendedSequenceNumber pendingCheckpointSequenceNumber,
RecordProcessorCheckpointer checkpointer) {
this.pendingCheckpointSequenceNumber = pendingCheckpointSequenceNumber;
this.checkpointer = checkpointer;
}
@ -48,7 +48,7 @@ public class PreparedCheckpointer implements IPreparedCheckpointer {
* {@inheritDoc}
*/
@Override
public ExtendedSequenceNumber getPendingCheckpoint() {
public ExtendedSequenceNumber pendingCheckpoint() {
return pendingCheckpointSequenceNumber;
}
@ -59,7 +59,7 @@ public class PreparedCheckpointer implements IPreparedCheckpointer {
public void checkpoint()
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException,
IllegalArgumentException {
checkpointer.checkpoint(pendingCheckpointSequenceNumber.getSequenceNumber(),
pendingCheckpointSequenceNumber.getSubSequenceNumber());
checkpointer.checkpoint(pendingCheckpointSequenceNumber.sequenceNumber(),
pendingCheckpointSequenceNumber.subSequenceNumber());
}
}

198
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/RecordProcessorCheckpointer.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/ShardRecordProcessorCheckpointer.java
View file

@ -12,75 +12,56 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.checkpoint;
import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
import com.amazonaws.services.kinesis.metrics.impl.ThreadSafeMetricsDelegatingScope;
import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.ICheckpoint;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IPreparedCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer;
import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
import com.amazonaws.services.kinesis.model.Record;
import lombok.Getter;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.kinesis.model.Record;
import software.amazon.kinesis.exceptions.InvalidStateException;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.KinesisClientLibException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.processor.PreparedCheckpointer;
import software.amazon.kinesis.processor.RecordProcessorCheckpointer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* This class is used to enable RecordProcessors to checkpoint their progress.
* The Amazon Kinesis Client Library will instantiate an object and provide a reference to the application
* RecordProcessor instance. Amazon Kinesis Client Library will create one instance per shard assignment.
* ShardRecordProcessor instance. Amazon Kinesis Client Library will create one instance per shard assignment.
*/
class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
@RequiredArgsConstructor
@Slf4j
public class ShardRecordProcessorCheckpointer implements RecordProcessorCheckpointer {
@NonNull
private final ShardInfo shardInfo;
@NonNull
@Getter @Accessors(fluent = true)
private final Checkpointer checkpointer;
private static final Log LOG = LogFactory.getLog(RecordProcessorCheckpointer.class);
private ICheckpoint checkpoint;
private ExtendedSequenceNumber largestPermittedCheckpointValue;
// Set to the last value set via checkpoint().
// Sample use: verify application shutdown() invoked checkpoint() at the end of a shard.
@Getter @Accessors(fluent = true)
private ExtendedSequenceNumber lastCheckpointValue;
private ShardInfo shardInfo;
private SequenceNumberValidator sequenceNumberValidator;
@Getter @Accessors(fluent = true)
private ExtendedSequenceNumber largestPermittedCheckpointValue;
private ExtendedSequenceNumber sequenceNumberAtShardEnd;
private IMetricsFactory metricsFactory;
/**
* Only has package level access, since only the Amazon Kinesis Client Library should be creating these.
*
* @param checkpoint Used to checkpoint progress of a RecordProcessor
* @param validator Used for validating sequence numbers
*/
RecordProcessorCheckpointer(ShardInfo shardInfo,
ICheckpoint checkpoint,
SequenceNumberValidator validator,
IMetricsFactory metricsFactory) {
this.shardInfo = shardInfo;
this.checkpoint = checkpoint;
this.sequenceNumberValidator = validator;
this.metricsFactory = metricsFactory;
}
/**
* {@inheritDoc}
*/
@Override
public synchronized void checkpoint()
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
if (LOG.isDebugEnabled()) {
LOG.debug("Checkpointing " + shardInfo.getShardId() + ", " + " token " + shardInfo.getConcurrencyToken()
+ " at largest permitted value " + this.largestPermittedCheckpointValue);
if (log.isDebugEnabled()) {
log.debug("Checkpointing {}, token {} at largest permitted value {}", shardInfo.shardId(),
shardInfo.concurrencyToken(), this.largestPermittedCheckpointValue);
}
advancePosition(this.largestPermittedCheckpointValue);
}
@ -92,12 +73,14 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
public synchronized void checkpoint(Record record)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException,
IllegalArgumentException {
// TODO: UserRecord Deprecation
if (record == null) {
throw new IllegalArgumentException("Could not checkpoint a null record");
} else if (record instanceof UserRecord) {
checkpoint(record.getSequenceNumber(), ((UserRecord) record).getSubSequenceNumber());
} else {
checkpoint(record.getSequenceNumber(), 0);
} /* else if (record instanceof UserRecord) {
checkpoint(record.sequenceNumber(), ((UserRecord) record).subSequenceNumber());
} */ else {
checkpoint(record.sequenceNumber(), 0);
}
}
@ -124,12 +107,6 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
+ subSequenceNumber);
}
// throws exception if sequence number shouldn't be checkpointed for this shard
sequenceNumberValidator.validateSequenceNumber(sequenceNumber);
if (LOG.isDebugEnabled()) {
LOG.debug("Validated checkpoint sequence number " + sequenceNumber + " for " + shardInfo.getShardId()
+ ", token " + shardInfo.getConcurrencyToken());
}
/*
* If there isn't a last checkpoint value, we only care about checking the upper bound.
* If there is a last checkpoint value, we want to check both the lower and upper bound.
@ -138,9 +115,9 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
if ((lastCheckpointValue == null || lastCheckpointValue.compareTo(newCheckpoint) <= 0)
&& newCheckpoint.compareTo(largestPermittedCheckpointValue) <= 0) {
if (LOG.isDebugEnabled()) {
LOG.debug("Checkpointing " + shardInfo.getShardId() + ", token " + shardInfo.getConcurrencyToken()
+ " at specific extended sequence number " + newCheckpoint);
if (log.isDebugEnabled()) {
log.debug("Checkpointing {}, token {} at specific extended sequence number {}", shardInfo.shardId(),
shardInfo.concurrencyToken(), newCheckpoint);
}
this.advancePosition(newCheckpoint);
} else {
@ -156,25 +133,28 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
* {@inheritDoc}
*/
@Override
public synchronized IPreparedCheckpointer prepareCheckpoint()
public synchronized PreparedCheckpointer prepareCheckpoint()
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
return this.prepareCheckpoint(
this.largestPermittedCheckpointValue.getSequenceNumber(),
this.largestPermittedCheckpointValue.getSubSequenceNumber());
this.largestPermittedCheckpointValue.sequenceNumber(),
this.largestPermittedCheckpointValue.subSequenceNumber());
}
/**
* {@inheritDoc}
*/
@Override
public synchronized IPreparedCheckpointer prepareCheckpoint(Record record)
public synchronized PreparedCheckpointer prepareCheckpoint(Record record)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
//
// TODO: UserRecord Deprecation
//
if (record == null) {
throw new IllegalArgumentException("Could not prepare checkpoint a null record");
} else if (record instanceof UserRecord) {
return prepareCheckpoint(record.getSequenceNumber(), ((UserRecord) record).getSubSequenceNumber());
} else {
return prepareCheckpoint(record.getSequenceNumber(), 0);
} /*else if (record instanceof UserRecord) {
return prepareCheckpoint(record.sequenceNumber(), ((UserRecord) record).subSequenceNumber());
} */ else {
return prepareCheckpoint(record.sequenceNumber(), 0);
}
}
@ -182,7 +162,7 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
* {@inheritDoc}
*/
@Override
public synchronized IPreparedCheckpointer prepareCheckpoint(String sequenceNumber)
public synchronized PreparedCheckpointer prepareCheckpoint(String sequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
return prepareCheckpoint(sequenceNumber, 0);
}
@ -191,7 +171,7 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
* {@inheritDoc}
*/
@Override
public synchronized IPreparedCheckpointer prepareCheckpoint(String sequenceNumber, long subSequenceNumber)
public synchronized PreparedCheckpointer prepareCheckpoint(String sequenceNumber, long subSequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
if (subSequenceNumber < 0) {
@ -199,12 +179,6 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
+ subSequenceNumber);
}
// throws exception if sequence number shouldn't be checkpointed for this shard
sequenceNumberValidator.validateSequenceNumber(sequenceNumber);
if (LOG.isDebugEnabled()) {
LOG.debug("Validated prepareCheckpoint sequence number " + sequenceNumber + " for " + shardInfo.getShardId()
+ ", token " + shardInfo.getConcurrencyToken());
}
/*
* If there isn't a last checkpoint value, we only care about checking the upper bound.
* If there is a last checkpoint value, we want to check both the lower and upper bound.
@ -213,10 +187,9 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
if ((lastCheckpointValue == null || lastCheckpointValue.compareTo(pendingCheckpoint) <= 0)
&& pendingCheckpoint.compareTo(largestPermittedCheckpointValue) <= 0) {
if (LOG.isDebugEnabled()) {
LOG.debug("Preparing checkpoint " + shardInfo.getShardId()
+ ", token " + shardInfo.getConcurrencyToken()
+ " at specific extended sequence number " + pendingCheckpoint);
if (log.isDebugEnabled()) {
log.debug("Preparing checkpoint {}, token {} at specific extended sequence number {}",
shardInfo.shardId(), shardInfo.concurrencyToken(), pendingCheckpoint);
}
return doPrepareCheckpoint(pendingCheckpoint);
} else {
@ -228,30 +201,14 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
}
}
/**
* @return the lastCheckpointValue
*/
ExtendedSequenceNumber getLastCheckpointValue() {
return lastCheckpointValue;
}
synchronized void setInitialCheckpointValue(ExtendedSequenceNumber initialCheckpoint) {
public synchronized void setInitialCheckpointValue(ExtendedSequenceNumber initialCheckpoint) {
lastCheckpointValue = initialCheckpoint;
}
/**
* Used for testing.
*
* @return the largest permitted checkpoint
*/
synchronized ExtendedSequenceNumber getLargestPermittedCheckpointValue() {
return largestPermittedCheckpointValue;
}
/**
* @param largestPermittedCheckpointValue the largest permitted checkpoint
*/
synchronized void setLargestPermittedCheckpointValue(ExtendedSequenceNumber largestPermittedCheckpointValue) {
public synchronized void largestPermittedCheckpointValue(ExtendedSequenceNumber largestPermittedCheckpointValue) {
this.largestPermittedCheckpointValue = largestPermittedCheckpointValue;
}
@ -262,7 +219,7 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
*
* @param extendedSequenceNumber
*/
synchronized void setSequenceNumberAtShardEnd(ExtendedSequenceNumber extendedSequenceNumber) {
public synchronized void sequenceNumberAtShardEnd(ExtendedSequenceNumber extendedSequenceNumber) {
this.sequenceNumberAtShardEnd = extendedSequenceNumber;
}
@ -291,38 +248,27 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
checkpointToRecord = ExtendedSequenceNumber.SHARD_END;
}
boolean unsetMetrics = false;
// Don't checkpoint a value we already successfully checkpointed
try {
if (!MetricsHelper.isMetricsScopePresent()) {
MetricsHelper.setMetricsScope(new ThreadSafeMetricsDelegatingScope(metricsFactory.createMetrics()));
unsetMetrics = true;
}
if (extendedSequenceNumber != null && !extendedSequenceNumber.equals(lastCheckpointValue)) {
try {
if (LOG.isDebugEnabled()) {
LOG.debug("Setting " + shardInfo.getShardId() + ", token " + shardInfo.getConcurrencyToken()
+ " checkpoint to " + checkpointToRecord);
if (log.isDebugEnabled()) {
log.debug("Setting {}, token {} checkpoint to {}", shardInfo.shardId(),
shardInfo.concurrencyToken(), checkpointToRecord);
}
checkpoint.setCheckpoint(shardInfo.getShardId(), checkpointToRecord, shardInfo.getConcurrencyToken());
checkpointer.setCheckpoint(shardInfo.shardId(), checkpointToRecord, shardInfo.concurrencyToken());
lastCheckpointValue = checkpointToRecord;
} catch (ThrottlingException | ShutdownException | InvalidStateException
| KinesisClientLibDependencyException e) {
throw e;
} catch (KinesisClientLibException e) {
LOG.warn("Caught exception setting checkpoint.", e);
log.warn("Caught exception setting checkpoint.", e);
throw new KinesisClientLibDependencyException("Caught exception while checkpointing", e);
}
}
} finally {
if (unsetMetrics) {
MetricsHelper.unsetMetricsScope();
}
}
}
/**
* This method stores the given sequenceNumber as a pending checkpooint in the lease table without overwriting the
* This method stores the given sequenceNumber as a pending checkpoint in the lease table without overwriting the
* current checkpoint, then returns a PreparedCheckpointer that is ready to checkpoint at the given sequence number.
*
* This method does not advance lastCheckpointValue, but calls to PreparedCheckpointer.checkpoint() on the returned
@ -333,18 +279,18 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
* the prepared checkpoint at snA.
* 2) prepareCheckpoint(snA); prepareCheckpoint(snB). // this works regardless of whether snA or snB is bigger. It
* replaces the preparedCheckpoint at snA with a new one at snB.
* 3) checkpointerA = prepareCheckpoint(snA); checkpointerB = prepareCheckpoint(snB); checkpointerB.checkpoint();
* 3) checkpointA = prepareCheckpoint(snA); checkpointB = prepareCheckpoint(snB); checkpointB.checkpoint();
* checkpointerA.checkpoint(); // This replaces the prepared checkpoint at snA with a new one at snB, then
* checkpoints at snB regardless of whether snA or snB is bigger. The checkpoint at snA only succeeds if snA > snB.
*
* @param extendedSequenceNumber the sequence number for the prepared checkpoint
* @return a prepared checkpointer that is ready to checkpoint at the given sequence number.
* @return a prepared checkpoint that is ready to checkpoint at the given sequence number.
* @throws KinesisClientLibDependencyException
* @throws InvalidStateException
* @throws ThrottlingException
* @throws ShutdownException
*/
private IPreparedCheckpointer doPrepareCheckpoint(ExtendedSequenceNumber extendedSequenceNumber)
private PreparedCheckpointer doPrepareCheckpoint(ExtendedSequenceNumber extendedSequenceNumber)
throws KinesisClientLibDependencyException, InvalidStateException, ThrottlingException, ShutdownException {
ExtendedSequenceNumber newPrepareCheckpoint = extendedSequenceNumber;
@ -362,16 +308,16 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
}
try {
checkpoint.prepareCheckpoint(shardInfo.getShardId(), newPrepareCheckpoint, shardInfo.getConcurrencyToken());
checkpointer.prepareCheckpoint(shardInfo.shardId(), newPrepareCheckpoint, shardInfo.concurrencyToken());
} catch (ThrottlingException | ShutdownException | InvalidStateException
| KinesisClientLibDependencyException e) {
throw e;
} catch (KinesisClientLibException e) {
LOG.warn("Caught exception setting prepareCheckpoint.", e);
log.warn("Caught exception setting prepareCheckpoint.", e);
throw new KinesisClientLibDependencyException("Caught exception while prepareCheckpointing", e);
}
PreparedCheckpointer result = new PreparedCheckpointer(newPrepareCheckpoint, this);
ShardPreparedCheckpointer result = new ShardPreparedCheckpointer(newPrepareCheckpoint, this);
return result;
}
}

35
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/dynamodb/DynamoDBCheckpointFactory.java Normal file
View file

@ -0,0 +1,35 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint.dynamodb;
import lombok.Data;
import software.amazon.kinesis.checkpoint.CheckpointFactory;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.processor.Checkpointer;
/**
*
*/
@Data
public class DynamoDBCheckpointFactory implements CheckpointFactory {
@Override
public Checkpointer createCheckpointer(final LeaseCoordinator leaseLeaseCoordinator,
final LeaseRefresher leaseRefresher) {
return new DynamoDBCheckpointer(leaseLeaseCoordinator, leaseRefresher);
}
}

156
amazon-kinesis-client/src/main/java/software/amazon/kinesis/checkpoint/dynamodb/DynamoDBCheckpointer.java Normal file
View file

@ -0,0 +1,156 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.checkpoint.dynamodb;
import java.util.Objects;
import java.util.UUID;
import com.google.common.annotations.VisibleForTesting;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.checkpoint.Checkpoint;
import software.amazon.kinesis.exceptions.KinesisClientLibDependencyException;
import software.amazon.kinesis.exceptions.KinesisClientLibException;
import software.amazon.kinesis.exceptions.ShutdownException;
import software.amazon.kinesis.exceptions.ThrottlingException;
import software.amazon.kinesis.exceptions.internal.KinesisClientLibIOException;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
*
*/
@RequiredArgsConstructor
@Slf4j
public class DynamoDBCheckpointer implements Checkpointer {
@NonNull
private final LeaseCoordinator leaseCoordinator;
@NonNull
private final LeaseRefresher leaseRefresher;
private String operation;
@Override
public void setCheckpoint(final String shardId, final ExtendedSequenceNumber checkpointValue,
final String concurrencyToken) throws KinesisClientLibException {
try {
boolean wasSuccessful = setCheckpoint(shardId, checkpointValue, UUID.fromString(concurrencyToken));
if (!wasSuccessful) {
throw new ShutdownException("Can't update checkpoint - instance doesn't hold the lease for this shard");
}
} catch (ProvisionedThroughputException e) {
throw new ThrottlingException("Got throttled while updating checkpoint.", e);
} catch (InvalidStateException e) {
String message = "Unable to save checkpoint for shardId " + shardId;
log.error(message, e);
throw new software.amazon.kinesis.exceptions.InvalidStateException(message, e);
} catch (DependencyException e) {
throw new KinesisClientLibDependencyException("Unable to save checkpoint for shardId " + shardId, e);
}
}
@Override
public ExtendedSequenceNumber getCheckpoint(final String shardId) throws KinesisClientLibException {
try {
return leaseRefresher.getLease(shardId).checkpoint();
} catch (DependencyException | InvalidStateException | ProvisionedThroughputException e) {
String message = "Unable to fetch checkpoint for shardId " + shardId;
log.error(message, e);
throw new KinesisClientLibIOException(message, e);
}
}
@Override
public Checkpoint getCheckpointObject(final String shardId) throws KinesisClientLibException {
try {
Lease lease = leaseRefresher.getLease(shardId);
return new Checkpoint(lease.checkpoint(), lease.pendingCheckpoint());
} catch (DependencyException | InvalidStateException | ProvisionedThroughputException e) {
String message = "Unable to fetch checkpoint for shardId " + shardId;
log.error(message, e);
throw new KinesisClientLibIOException(message, e);
}
}
@Override
public void prepareCheckpoint(final String shardId, final ExtendedSequenceNumber pendingCheckpoint,
final String concurrencyToken) throws KinesisClientLibException {
try {
boolean wasSuccessful =
prepareCheckpoint(shardId, pendingCheckpoint, UUID.fromString(concurrencyToken));
if (!wasSuccessful) {
throw new ShutdownException(
"Can't prepare checkpoint - instance doesn't hold the lease for this shard");
}
} catch (ProvisionedThroughputException e) {
throw new ThrottlingException("Got throttled while preparing checkpoint.", e);
} catch (InvalidStateException e) {
String message = "Unable to prepare checkpoint for shardId " + shardId;
log.error(message, e);
throw new software.amazon.kinesis.exceptions.InvalidStateException(message, e);
} catch (DependencyException e) {
throw new KinesisClientLibDependencyException("Unable to prepare checkpoint for shardId " + shardId, e);
}
}
@VisibleForTesting
public boolean setCheckpoint(String shardId, ExtendedSequenceNumber checkpoint, UUID concurrencyToken)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
Lease lease = leaseCoordinator.getCurrentlyHeldLease(shardId);
if (lease == null) {
log.info("Worker {} could not update checkpoint for shard {} because it does not hold the lease",
leaseCoordinator.workerIdentifier(), shardId);
return false;
}
lease.checkpoint(checkpoint);
lease.pendingCheckpoint(null);
lease.ownerSwitchesSinceCheckpoint(0L);
return leaseCoordinator.updateLease(lease, concurrencyToken, operation, shardId);
}
boolean prepareCheckpoint(String shardId, ExtendedSequenceNumber pendingCheckpoint, UUID concurrencyToken)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
Lease lease = leaseCoordinator.getCurrentlyHeldLease(shardId);
if (lease == null) {
log.info("Worker {} could not prepare checkpoint for shard {} because it does not hold the lease",
leaseCoordinator.workerIdentifier(), shardId);
return false;
}
lease.pendingCheckpoint(Objects.requireNonNull(pendingCheckpoint, "pendingCheckpoint should not be null"));
return leaseCoordinator.updateLease(lease, concurrencyToken, operation, shardId);
}
@Override
public void operation(@NonNull final String operation) {
this.operation = operation;
}
@Override
public String operation() {
return operation;
}
}

175
amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/ConfigsBuilder.java Normal file
View file

@ -0,0 +1,175 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.common;
import org.apache.commons.lang.StringUtils;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import software.amazon.awssdk.services.cloudwatch.CloudWatchAsyncClient;
import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.kinesis.checkpoint.CheckpointConfig;
import software.amazon.kinesis.coordinator.CoordinatorConfig;
import software.amazon.kinesis.leases.LeaseManagementConfig;
import software.amazon.kinesis.lifecycle.LifecycleConfig;
import software.amazon.kinesis.metrics.MetricsConfig;
import software.amazon.kinesis.processor.ProcessorConfig;
import software.amazon.kinesis.processor.ShardRecordProcessorFactory;
import software.amazon.kinesis.retrieval.RetrievalConfig;
/**
* This Builder is useful to create all configurations for the KCL with default values.
*/
@Data
@Accessors(fluent = true)
public class ConfigsBuilder {
/**
* Name of the stream to consume records from
*/
@NonNull
private final String streamName;
/**
* Application name for the KCL Worker
*/
@NonNull
private final String applicationName;
/**
* KinesisClient to be used to consumer records from Kinesis
*/
@NonNull
private final KinesisAsyncClient kinesisClient;
/**
* DynamoDBClient to be used to interact with DynamoDB service for lease management and checkpoiniting
*/
@NonNull
private final DynamoDbAsyncClient dynamoDBClient;
/**
* CloudWatchClient to be used to push KCL metrics to CloudWatch service
*/
@NonNull
private final CloudWatchAsyncClient cloudWatchClient;
/**
* KCL worker identifier to distinguish between 2 unique workers
*/
@NonNull
private final String workerIdentifier;
/**
* ShardRecordProcessorFactory to be used to create ShardRecordProcesor for processing records
*/
@NonNull
private final ShardRecordProcessorFactory shardRecordProcessorFactory;
/**
* Lease table name used for lease management and checkpointing.
*/
private String tableName;
/**
* Lease table name used for lease management and checkpointing.
*
* @return DynamoDB table name
*/
public String tableName() {
if (StringUtils.isEmpty(tableName)) {
tableName = applicationName();
}
return tableName;
}
/**
* CloudWatch namespace for KCL metrics.
*/
private String namespace;
/**
* CloudWatch namespace for KCL metrics.
*
* @return CloudWatch namespace
*/
public String namespace() {
if (StringUtils.isEmpty(namespace)) {
namespace = applicationName();
}
return namespace;
}
/**
* Creates a new instance of CheckpointConfig
*
* @return CheckpointConfig
*/
public CheckpointConfig checkpointConfig() {
return new CheckpointConfig();
}
/**
* Creates a new instance of CoordinatorConfig
*
* @return CoordinatorConfig
*/
public CoordinatorConfig coordinatorConfig() {
return new CoordinatorConfig(applicationName());
}
/**
* Creates a new instance of LeaseManagementConfig
*
* @return LeaseManagementConfig
*/
public LeaseManagementConfig leaseManagementConfig() {
return new LeaseManagementConfig(tableName(), dynamoDBClient(), kinesisClient(), streamName(),
workerIdentifier());
}
/**
* Creates a new instance of LifecycleConfig
*
* @return LifecycleConfig
*/
public LifecycleConfig lifecycleConfig() {
return new LifecycleConfig();
}
/**
* Creates a new instance of MetricsConfig
*
* @return MetricsConfig
*/
public MetricsConfig metricsConfig() {
return new MetricsConfig(cloudWatchClient(), namespace());
}
/**
* Creates a new instance of ProcessorConfig
*
* @return ProcessorConfigConfig
*/
public ProcessorConfig processorConfig() {
return new ProcessorConfig(shardRecordProcessorFactory());
}
/**
* Creates a new instance of RetrievalConfig
*
* @return RetrievalConfig
*/
public RetrievalConfig retrievalConfig() {
return new RetrievalConfig(kinesisClient(), streamName(), applicationName());
}
}

36
amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java Normal file
View file

@ -0,0 +1,36 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.common;
/**
* Used to specify the position in the stream where a new application should start from.
* This is used during initial application bootstrap (when a checkpoint doesn't exist for a shard or its parents).
*/
public enum InitialPositionInStream {
/**
* Start after the most recent data record (fetch new data).
*/
LATEST,
/**
* Start from the oldest available data record.
*/
TRIM_HORIZON,
/**
* Start from the record at or after the specified server-side timestamp.
*/
AT_TIMESTAMP
}

14
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/InitialPositionInStreamExtended.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStreamExtended.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.common;
import java.util.Date;
@ -20,7 +20,7 @@ import java.util.Date;
* Class that houses the entities needed to specify the position in the stream from where a new application should
* start.
*/
class InitialPositionInStreamExtended {
public class InitialPositionInStreamExtended {
private final InitialPositionInStream position;
private final Date timestamp;
@ -44,7 +44,7 @@ class InitialPositionInStreamExtended {
*
* @return The initial position in stream.
*/
protected InitialPositionInStream getInitialPositionInStream() {
public InitialPositionInStream getInitialPositionInStream() {
return this.position;
}
@ -54,11 +54,11 @@ class InitialPositionInStreamExtended {
*
* @return The timestamp from where we need to start the application.
*/
protected Date getTimestamp() {
public Date getTimestamp() {
return this.timestamp;
}
protected static InitialPositionInStreamExtended newInitialPosition(final InitialPositionInStream position) {
public static InitialPositionInStreamExtended newInitialPosition(final InitialPositionInStream position) {
switch (position) {
case LATEST:
return new InitialPositionInStreamExtended(InitialPositionInStream.LATEST, null);
@ -69,7 +69,7 @@ class InitialPositionInStreamExtended {
}
}
protected static InitialPositionInStreamExtended newInitialPositionAtTimestamp(final Date timestamp) {
public static InitialPositionInStreamExtended newInitialPositionAtTimestamp(final Date timestamp) {
if (timestamp == null) {
throw new IllegalArgumentException("Timestamp must be specified for InitialPosition AT_TIMESTAMP");
}

37
amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/KinesisClientUtil.java Normal file
View file

@ -0,0 +1,37 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.common;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClientBuilder;
/**
* Utility to setup KinesisAsyncClient to be used with KCL.
*/
public class KinesisClientUtil {
/**
* Creates a client from a builder.
*
* @param clientBuilder
* @return
*/
public static KinesisAsyncClient createKinesisAsyncClient(KinesisAsyncClientBuilder clientBuilder) {
return clientBuilder.httpClient(NettyNioAsyncHttpClient.builder().maxConcurrency(Integer.MAX_VALUE).build())
.build();
}
}

72
amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/KinesisRequestsBuilder.java Normal file
View file

@ -0,0 +1,72 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.common;
import software.amazon.awssdk.awscore.AwsRequest;
import software.amazon.awssdk.awscore.AwsRequestOverrideConfiguration;
import software.amazon.awssdk.core.ApiName;
import software.amazon.awssdk.services.kinesis.model.DescribeStreamConsumerRequest;
import software.amazon.awssdk.services.kinesis.model.DescribeStreamSummaryRequest;
import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
import software.amazon.awssdk.services.kinesis.model.RegisterStreamConsumerRequest;
import software.amazon.awssdk.services.kinesis.model.SubscribeToShardRequest;
import software.amazon.kinesis.retrieval.RetrievalConfig;
/**
*
*/
public class KinesisRequestsBuilder {
public static ListShardsRequest.Builder listShardsRequestBuilder() {
return appendUserAgent(ListShardsRequest.builder());
}
public static SubscribeToShardRequest.Builder subscribeToShardRequestBuilder() {
return appendUserAgent(SubscribeToShardRequest.builder());
}
public static GetRecordsRequest.Builder getRecordsRequestBuilder() {
return appendUserAgent(GetRecordsRequest.builder());
}
public static GetShardIteratorRequest.Builder getShardIteratorRequestBuilder() {
return appendUserAgent(GetShardIteratorRequest.builder());
}
public static DescribeStreamSummaryRequest.Builder describeStreamSummaryRequestBuilder() {
return appendUserAgent(DescribeStreamSummaryRequest.builder());
}
public static RegisterStreamConsumerRequest.Builder registerStreamConsumerRequestBuilder() {
return appendUserAgent(RegisterStreamConsumerRequest.builder());
}
public static DescribeStreamConsumerRequest.Builder describeStreamConsumerRequestBuilder() {
return appendUserAgent(DescribeStreamConsumerRequest.builder());
}
@SuppressWarnings("unchecked")
private static <T extends AwsRequest.Builder> T appendUserAgent(final T builder) {
return (T) builder
.overrideConfiguration(
AwsRequestOverrideConfiguration.builder()
.addApiName(ApiName.builder().name(RetrievalConfig.KINESIS_CLIENT_LIB_USER_AGENT)
.version(RetrievalConfig.KINESIS_CLIENT_LIB_USER_AGENT_VERSION).build())
.build());
}
}

72
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/CoordinatorConfig.java Normal file
View file

@ -0,0 +1,72 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import software.amazon.kinesis.leases.NoOpShardPrioritization;
import software.amazon.kinesis.leases.ShardPrioritization;
/**
* Used by the KCL to configure the coordinator.
*/
@Data
@Accessors(fluent = true)
public class CoordinatorConfig {
/**
* Application name used by checkpointer to checkpoint.
*
* @return String
*/
@NonNull
private final String applicationName;
/**
* Interval in milliseconds between polling to check for parent shard completion.
* Polling frequently will take up more DynamoDB IOPS (when there are leases for shards waiting on
* completion of parent shards).
*
* <p>Default value: 10000L</p>
*/
private long parentShardPollIntervalMillis = 10000L;
/**
* The Scheduler will skip shard sync during initialization if there are one or more leases in the lease table. This
* assumes that the shards and leases are in-sync. This enables customers to choose faster startup times (e.g.
* during incremental deployments of an application).
*
* <p>Default value: false</p>
*/
private boolean skipShardSyncAtWorkerInitializationIfLeasesExist = false;
/**
* The number of milliseconds between polling of the shard consumer for triggering state changes, and health checks.
*
* <p>Default value: 1000 milliseconds</p>
*/
private long shardConsumerDispatchPollIntervalMillis = 1000L;
/**
* Shard prioritization strategy.
*
* <p>Default value: {@link NoOpShardPrioritization}</p>
*/
private ShardPrioritization shardPrioritization = new NoOpShardPrioritization();
private CoordinatorFactory coordinatorFactory = new SchedulerCoordinatorFactory();
}

35
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/CoordinatorFactory.java Normal file
View file

@ -0,0 +1,35 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import java.util.concurrent.ExecutorService;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.processor.Checkpointer;
/**
*
*/
public interface CoordinatorFactory {
ExecutorService createExecutorService();
GracefulShutdownCoordinator createGracefulShutdownCoordinator();
WorkerStateChangeListener createWorkerStateChangeListener();
ShardRecordProcessorCheckpointer createRecordProcessorCheckpointer(ShardInfo shardInfo, Checkpointer checkpoint);
}

8
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/GracefulShutdownContext.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/GracefulShutdownContext.java
View file

@ -12,22 +12,24 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.coordinator;
import lombok.Data;
import lombok.experimental.Accessors;
import java.util.concurrent.CountDownLatch;
@Data
@Accessors(fluent = true)
class GracefulShutdownContext {
private final CountDownLatch shutdownCompleteLatch;
private final CountDownLatch notificationCompleteLatch;
private final Worker worker;
private final Scheduler scheduler;
static GracefulShutdownContext SHUTDOWN_ALREADY_COMPLETED = new GracefulShutdownContext(null, null, null);
boolean isShutdownAlreadyCompleted() {
return shutdownCompleteLatch == null && notificationCompleteLatch == null && worker == null;
return shutdownCompleteLatch == null && notificationCompleteLatch == null && scheduler == null;
}
}

45
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/GracefulShutdownCoordinator.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/GracefulShutdownCoordinator.java
View file

@ -12,15 +12,14 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.coordinator;
import java.util.concurrent.Callable;
import java.util.concurrent.Future;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import lombok.extern.slf4j.Slf4j;
class GracefulShutdownCoordinator {
@ -36,10 +35,8 @@ class GracefulShutdownCoordinator {
return new GracefulShutdownCallable(startWorkerShutdown);
}
@Slf4j
static class GracefulShutdownCallable implements Callable<Boolean> {
private static final Log log = LogFactory.getLog(GracefulShutdownCallable.class);
private final Callable<GracefulShutdownContext> startWorkerShutdown;
GracefulShutdownCallable(Callable<GracefulShutdownContext> startWorkerShutdown) {
@ -47,12 +44,12 @@ class GracefulShutdownCoordinator {
}
private boolean isWorkerShutdownComplete(GracefulShutdownContext context) {
return context.getWorker().isShutdownComplete() || context.getWorker().getShardInfoShardConsumerMap().isEmpty();
return context.scheduler().shutdownComplete() || context.scheduler().shardInfoShardConsumerMap().isEmpty();
}
private String awaitingLogMessage(GracefulShutdownContext context) {
long awaitingNotification = context.getNotificationCompleteLatch().getCount();
long awaitingFinalShutdown = context.getShutdownCompleteLatch().getCount();
long awaitingNotification = context.notificationCompleteLatch().getCount();
long awaitingFinalShutdown = context.shutdownCompleteLatch().getCount();
return String.format(
"Waiting for %d record process to complete shutdown notification, and %d record processor to complete final shutdown ",
@ -60,7 +57,7 @@ class GracefulShutdownCoordinator {
}
private String awaitingFinalShutdownMessage(GracefulShutdownContext context) {
long outstanding = context.getShutdownCompleteLatch().getCount();
long outstanding = context.shutdownCompleteLatch().getCount();
return String.format("Waiting for %d record processors to complete final shutdown", outstanding);
}
@ -73,18 +70,18 @@ class GracefulShutdownCoordinator {
// ShardConsumer would start the lease loss shutdown, and may never call the notification methods.
//
try {
while (!context.getNotificationCompleteLatch().await(1, TimeUnit.SECONDS)) {
while (!context.notificationCompleteLatch().await(1, TimeUnit.SECONDS)) {
if (Thread.interrupted()) {
throw new InterruptedException();
}
log.info(awaitingLogMessage(context));
if (workerShutdownWithRemaining(context.getShutdownCompleteLatch().getCount(), context)) {
if (workerShutdownWithRemaining(context.shutdownCompleteLatch().getCount(), context)) {
return false;
}
}
} catch (InterruptedException ie) {
log.warn("Interrupted while waiting for notification complete, terminating shutdown. "
+ awaitingLogMessage(context));
log.warn("Interrupted while waiting for notification complete, terminating shutdown. {}",
awaitingLogMessage(context));
return false;
}
@ -97,7 +94,7 @@ class GracefulShutdownCoordinator {
// Once all record processors have been notified of the shutdown it is safe to allow the worker to
// start its shutdown behavior. Once shutdown starts it will stop renewer, and drop any remaining leases.
//
context.getWorker().shutdown();
context.scheduler().shutdown();
if (Thread.interrupted()) {
log.warn("Interrupted after worker shutdown, terminating shutdown");
@ -105,23 +102,23 @@ class GracefulShutdownCoordinator {
}
//
// Want to wait for all the remaining ShardConsumers/RecordProcessor's to complete their final shutdown
// Want to wait for all the remaining ShardConsumers/ShardRecordProcessor's to complete their final shutdown
// processing. This should really be a no-op since as part of the notification completion the lease for
// ShardConsumer is terminated.
//
try {
while (!context.getShutdownCompleteLatch().await(1, TimeUnit.SECONDS)) {
while (!context.shutdownCompleteLatch().await(1, TimeUnit.SECONDS)) {
if (Thread.interrupted()) {
throw new InterruptedException();
}
log.info(awaitingFinalShutdownMessage(context));
if (workerShutdownWithRemaining(context.getShutdownCompleteLatch().getCount(), context)) {
if (workerShutdownWithRemaining(context.shutdownCompleteLatch().getCount(), context)) {
return false;
}
}
} catch (InterruptedException ie) {
log.warn("Interrupted while waiting for shutdown completion, terminating shutdown. "
+ awaitingFinalShutdownMessage(context));
log.warn("Interrupted while waiting for shutdown completion, terminating shutdown. {}",
awaitingFinalShutdownMessage(context));
return false;
}
return true;
@ -138,10 +135,10 @@ class GracefulShutdownCoordinator {
private boolean workerShutdownWithRemaining(long outstanding, GracefulShutdownContext context) {
if (isWorkerShutdownComplete(context)) {
if (outstanding != 0) {
log.info("Shutdown completed, but shutdownCompleteLatch still had outstanding " + outstanding
+ " with a current value of " + context.getShutdownCompleteLatch().getCount() + ". shutdownComplete: "
+ context.getWorker().isShutdownComplete() + " -- Consumer Map: "
+ context.getWorker().getShardInfoShardConsumerMap().size());
log.info("Shutdown completed, but shutdownCompleteLatch still had outstanding {} with a current"
+ " value of {}. shutdownComplete: {} -- Consumer Map: {}", outstanding,
context.shutdownCompleteLatch().getCount(), context.scheduler().shutdownComplete(),
context.scheduler().shardInfoShardConsumerMap().size());
return true;
}
}

30
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/NoOpWorkerStateChangeListener.java Normal file
View file

@ -0,0 +1,30 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
public class NoOpWorkerStateChangeListener implements WorkerStateChangeListener {
/**
* Empty constructor for NoOp Worker State Change Listener
*/
public NoOpWorkerStateChangeListener() {
}
@Override
public void onWorkerStateChange(WorkerState newState) {
}
}

633
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/Scheduler.java Normal file
View file

@ -0,0 +1,633 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import com.google.common.annotations.VisibleForTesting;
import lombok.AccessLevel;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.NonNull;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.checkpoint.CheckpointConfig;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.leases.Lease;
import software.amazon.kinesis.leases.LeaseCoordinator;
import software.amazon.kinesis.leases.LeaseManagementConfig;
import software.amazon.kinesis.leases.LeaseRefresher;
import software.amazon.kinesis.leases.ShardDetector;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.leases.ShardPrioritization;
import software.amazon.kinesis.leases.ShardSyncTask;
import software.amazon.kinesis.leases.ShardSyncTaskManager;
import software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator;
import software.amazon.kinesis.leases.exceptions.LeasingException;
import software.amazon.kinesis.lifecycle.LifecycleConfig;
import software.amazon.kinesis.lifecycle.ShardConsumer;
import software.amazon.kinesis.lifecycle.ShardConsumerArgument;
import software.amazon.kinesis.lifecycle.ShardConsumerShutdownNotification;
import software.amazon.kinesis.lifecycle.ShutdownNotification;
import software.amazon.kinesis.lifecycle.ShutdownReason;
import software.amazon.kinesis.lifecycle.TaskResult;
import software.amazon.kinesis.metrics.CloudWatchMetricsFactory;
import software.amazon.kinesis.metrics.MetricsCollectingTaskDecorator;
import software.amazon.kinesis.metrics.MetricsConfig;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.processor.Checkpointer;
import software.amazon.kinesis.processor.ProcessorConfig;
import software.amazon.kinesis.processor.ShardRecordProcessorFactory;
import software.amazon.kinesis.processor.ShutdownNotificationAware;
import software.amazon.kinesis.retrieval.AggregatorUtil;
import software.amazon.kinesis.retrieval.RecordsPublisher;
import software.amazon.kinesis.retrieval.RetrievalConfig;
/**
*
*/
@Getter
@Accessors(fluent = true)
@Slf4j
public class Scheduler implements Runnable {
static final int MAX_INITIALIZATION_ATTEMPTS = 20;
private SchedulerLog slog = new SchedulerLog();
private final CheckpointConfig checkpointConfig;
private final CoordinatorConfig coordinatorConfig;
private final LeaseManagementConfig leaseManagementConfig;
private final LifecycleConfig lifecycleConfig;
private final MetricsConfig metricsConfig;
private final ProcessorConfig processorConfig;
private final RetrievalConfig retrievalConfig;
private final String applicationName;
private final Checkpointer checkpoint;
private final long shardConsumerDispatchPollIntervalMillis;
// Backoff time when polling to check if application has finished processing
// parent shards
private final long parentShardPollIntervalMillis;
private final ExecutorService executorService;
// private final GetRecordsRetrievalStrategy getRecordsRetrievalStrategy;
private final LeaseCoordinator leaseCoordinator;
private final ShardSyncTaskManager shardSyncTaskManager;
private final ShardPrioritization shardPrioritization;
private final boolean cleanupLeasesUponShardCompletion;
private final boolean skipShardSyncAtWorkerInitializationIfLeasesExist;
private final GracefulShutdownCoordinator gracefulShutdownCoordinator;
private final WorkerStateChangeListener workerStateChangeListener;
private final InitialPositionInStreamExtended initialPosition;
private final MetricsFactory metricsFactory;
private final long failoverTimeMillis;
private final long taskBackoffTimeMillis;
private final String streamName;
private final long listShardsBackoffTimeMillis;
private final int maxListShardsRetryAttempts;
private final LeaseRefresher leaseRefresher;
private final ShardDetector shardDetector;
private final boolean ignoreUnexpetedChildShards;
private final AggregatorUtil aggregatorUtil;
// Holds consumers for shards the worker is currently tracking. Key is shard
// info, value is ShardConsumer.
private ConcurrentMap<ShardInfo, ShardConsumer> shardInfoShardConsumerMap = new ConcurrentHashMap<>();
private volatile boolean shutdown;
private volatile long shutdownStartTimeMillis;
private volatile boolean shutdownComplete = false;
/**
* Used to ensure that only one requestedShutdown is in progress at a time.
*/
private Future<Boolean> gracefulShutdownFuture;
@VisibleForTesting
protected boolean gracefuleShutdownStarted = false;
public Scheduler(@NonNull final CheckpointConfig checkpointConfig,
@NonNull final CoordinatorConfig coordinatorConfig,
@NonNull final LeaseManagementConfig leaseManagementConfig,
@NonNull final LifecycleConfig lifecycleConfig,
@NonNull final MetricsConfig metricsConfig,
@NonNull final ProcessorConfig processorConfig,
@NonNull final RetrievalConfig retrievalConfig) {
this.checkpointConfig = checkpointConfig;
this.coordinatorConfig = coordinatorConfig;
this.leaseManagementConfig = leaseManagementConfig;
this.lifecycleConfig = lifecycleConfig;
this.metricsConfig = metricsConfig;
this.processorConfig = processorConfig;
this.retrievalConfig = retrievalConfig;
this.applicationName = this.coordinatorConfig.applicationName();
this.metricsFactory = this.metricsConfig.metricsFactory();
this.leaseCoordinator = this.leaseManagementConfig.leaseManagementFactory()
.createLeaseCoordinator(this.metricsFactory);
this.leaseRefresher = this.leaseCoordinator.leaseRefresher();
//
// TODO: Figure out what to do with lease manage <=> checkpoint relationship
//
this.checkpoint = this.checkpointConfig.checkpointFactory().createCheckpointer(this.leaseCoordinator,
this.leaseRefresher);
//
// TODO: Move this configuration to lifecycle
//
this.shardConsumerDispatchPollIntervalMillis = this.coordinatorConfig.shardConsumerDispatchPollIntervalMillis();
this.parentShardPollIntervalMillis = this.coordinatorConfig.parentShardPollIntervalMillis();
this.executorService = this.coordinatorConfig.coordinatorFactory().createExecutorService();
this.shardSyncTaskManager = this.leaseManagementConfig.leaseManagementFactory()
.createShardSyncTaskManager(this.metricsFactory);
this.shardPrioritization = this.coordinatorConfig.shardPrioritization();
this.cleanupLeasesUponShardCompletion = this.leaseManagementConfig.cleanupLeasesUponShardCompletion();
this.skipShardSyncAtWorkerInitializationIfLeasesExist =
this.coordinatorConfig.skipShardSyncAtWorkerInitializationIfLeasesExist();
this.gracefulShutdownCoordinator =
this.coordinatorConfig.coordinatorFactory().createGracefulShutdownCoordinator();
this.workerStateChangeListener = this.coordinatorConfig.coordinatorFactory().createWorkerStateChangeListener();
this.initialPosition = retrievalConfig.initialPositionInStreamExtended();
this.failoverTimeMillis = this.leaseManagementConfig.failoverTimeMillis();
this.taskBackoffTimeMillis = this.lifecycleConfig.taskBackoffTimeMillis();
// this.retryGetRecordsInSeconds = this.retrievalConfig.retryGetRecordsInSeconds();
// this.maxGetRecordsThreadPool = this.retrievalConfig.maxGetRecordsThreadPool();
this.streamName = this.retrievalConfig.streamName();
this.listShardsBackoffTimeMillis = this.retrievalConfig.listShardsBackoffTimeInMillis();
this.maxListShardsRetryAttempts = this.retrievalConfig.maxListShardsRetryAttempts();
this.shardDetector = this.shardSyncTaskManager.shardDetector();
this.ignoreUnexpetedChildShards = this.leaseManagementConfig.ignoreUnexpectedChildShards();
this.aggregatorUtil = this.lifecycleConfig.aggregatorUtil();
}
/**
* Start consuming data from the stream, and pass it to the application record processors.
*/
@Override
public void run() {
if (shutdown) {
return;
}
try {
initialize();
log.info("Initialization complete. Starting worker loop.");
} catch (RuntimeException e) {
log.error("Unable to initialize after {} attempts. Shutting down.", MAX_INITIALIZATION_ATTEMPTS, e);
shutdown();
}
while (!shouldShutdown()) {
runProcessLoop();
}
finalShutdown();
log.info("Worker loop is complete. Exiting from worker.");
}
private void initialize() {
workerStateChangeListener.onWorkerStateChange(WorkerStateChangeListener.WorkerState.INITIALIZING);
boolean isDone = false;
Exception lastException = null;
for (int i = 0; (!isDone) && (i < MAX_INITIALIZATION_ATTEMPTS); i++) {
try {
log.info("Initialization attempt {}", (i + 1));
log.info("Initializing LeaseCoordinator");
leaseCoordinator.initialize();
TaskResult result = null;
if (!skipShardSyncAtWorkerInitializationIfLeasesExist || leaseRefresher.isLeaseTableEmpty()) {
log.info("Syncing Kinesis shard info");
ShardSyncTask shardSyncTask = new ShardSyncTask(shardDetector, leaseRefresher, initialPosition,
cleanupLeasesUponShardCompletion, ignoreUnexpetedChildShards, 0L, metricsFactory);
result = new MetricsCollectingTaskDecorator(shardSyncTask, metricsFactory).call();
} else {
log.info("Skipping shard sync per configuration setting (and lease table is not empty)");
}
if (result == null || result.getException() == null) {
if (!leaseCoordinator.isRunning()) {
log.info("Starting LeaseCoordinator");
leaseCoordinator.start();
} else {
log.info("LeaseCoordinator is already running. No need to start it.");
}
isDone = true;
} else {
lastException = result.getException();
}
} catch (LeasingException e) {
log.error("Caught exception when initializing LeaseCoordinator", e);
lastException = e;
} catch (Exception e) {
lastException = e;
}
try {
Thread.sleep(parentShardPollIntervalMillis);
} catch (InterruptedException e) {
log.debug("Sleep interrupted while initializing worker.");
}
}
if (!isDone) {
throw new RuntimeException(lastException);
}
workerStateChangeListener.onWorkerStateChange(WorkerStateChangeListener.WorkerState.STARTED);
}
@VisibleForTesting
void runProcessLoop() {
try {
boolean foundCompletedShard = false;
Set<ShardInfo> assignedShards = new HashSet<>();
for (ShardInfo shardInfo : getShardInfoForAssignments()) {
ShardConsumer shardConsumer = createOrGetShardConsumer(shardInfo,
processorConfig.shardRecordProcessorFactory());
if (shardConsumer.isShutdown() && shardConsumer.shutdownReason().equals(ShutdownReason.SHARD_END)) {
foundCompletedShard = true;
} else {
shardConsumer.executeLifecycle();
}
assignedShards.add(shardInfo);
}
if (foundCompletedShard) {
shardSyncTaskManager.syncShardAndLeaseInfo();
}
// clean up shard consumers for unassigned shards
cleanupShardConsumers(assignedShards);
slog.info("Sleeping ...");
Thread.sleep(shardConsumerDispatchPollIntervalMillis);
} catch (Exception e) {
log.error("Worker.run caught exception, sleeping for {} milli seconds!",
String.valueOf(shardConsumerDispatchPollIntervalMillis), e);
try {
Thread.sleep(shardConsumerDispatchPollIntervalMillis);
} catch (InterruptedException ex) {
log.info("Worker: sleep interrupted after catching exception ", ex);
}
}
slog.resetInfoLogging();
}
/**
* Returns whether worker can shutdown immediately. Note that this method is called from Worker's {{@link #run()}
* method before every loop run, so method must do minimum amount of work to not impact shard processing timings.
*
* @return Whether worker should shutdown immediately.
*/
@VisibleForTesting
boolean shouldShutdown() {
if (executorService.isShutdown()) {
log.error("Worker executor service has been shutdown, so record processors cannot be shutdown.");
return true;
}
if (shutdown) {
if (shardInfoShardConsumerMap.isEmpty()) {
log.info("All record processors have been shutdown successfully.");
return true;
}
if ((System.currentTimeMillis() - shutdownStartTimeMillis) >= failoverTimeMillis) {
log.info("Lease failover time is reached, so forcing shutdown.");
return true;
}
}
return false;
}
/**
* Requests a graceful shutdown of the worker, notifying record processors, that implement
* {@link ShutdownNotificationAware}, of the impending shutdown. This gives the record processor a final chance to
* checkpoint.
*
* This will only create a single shutdown future. Additional attempts to start a graceful shutdown will return the
* previous future.
*
* <b>It's possible that a record processor won't be notify before being shutdown. This can occur if the lease is
* lost after requesting shutdown, but before the notification is dispatched.</b>
*
* <h2>Requested Shutdown Process</h2> When a shutdown process is requested it operates slightly differently to
* allow the record processors a chance to checkpoint a final time.
* <ol>
* <li>Call to request shutdown invoked.</li>
* <li>Worker stops attempting to acquire new leases</li>
* <li>Record Processor Shutdown Begins
* <ol>
* <li>Record processor is notified of the impending shutdown, and given a final chance to checkpoint</li>
* <li>The lease for the record processor is then dropped.</li>
* <li>The record processor enters into an idle state waiting for the worker to complete final termination</li>
* <li>The worker will detect a record processor that has lost it's lease, and will terminate the record processor
* with {@link ShutdownReason#LEASE_LOST}</li>
* </ol>
* </li>
* <li>The worker will shutdown all record processors.</li>
* <li>Once all record processors have been terminated, the worker will terminate all owned resources.</li>
* <li>Once the worker shutdown is complete, the returned future is completed.</li>
* </ol>
*
* @return a future that will be set once the shutdown has completed. True indicates that the graceful shutdown
* completed successfully. A false value indicates that a non-exception case caused the shutdown process to
* terminate early.
*/
public Future<Boolean> startGracefulShutdown() {
synchronized (this) {
if (gracefulShutdownFuture == null) {
gracefulShutdownFuture = gracefulShutdownCoordinator
.startGracefulShutdown(createGracefulShutdownCallable());
}
}
return gracefulShutdownFuture;
}
/**
* Creates a callable that will execute the graceful shutdown process. This callable can be used to execute graceful
* shutdowns in your own executor, or execute the shutdown synchronously.
*
* @return a callable that run the graceful shutdown process. This may return a callable that return true if the
* graceful shutdown has already been completed.
* @throws IllegalStateException
* thrown by the callable if another callable has already started the shutdown process.
*/
public Callable<Boolean> createGracefulShutdownCallable() {
if (shutdownComplete()) {
return () -> true;
}
Callable<GracefulShutdownContext> startShutdown = createWorkerShutdownCallable();
return gracefulShutdownCoordinator.createGracefulShutdownCallable(startShutdown);
}
public boolean hasGracefulShutdownStarted() {
return gracefuleShutdownStarted;
}
@VisibleForTesting
Callable<GracefulShutdownContext> createWorkerShutdownCallable() {
return () -> {
synchronized (this) {
if (this.gracefuleShutdownStarted) {
throw new IllegalStateException("Requested shutdown has already been started");
}
this.gracefuleShutdownStarted = true;
}
//
// Stop accepting new leases. Once we do this we can be sure that
// no more leases will be acquired.
//
leaseCoordinator.stopLeaseTaker();
Collection<Lease> leases = leaseCoordinator.getAssignments();
if (leases == null || leases.isEmpty()) {
//
// If there are no leases notification is already completed, but we still need to shutdown the worker.
//
this.shutdown();
return GracefulShutdownContext.SHUTDOWN_ALREADY_COMPLETED;
}
CountDownLatch shutdownCompleteLatch = new CountDownLatch(leases.size());
CountDownLatch notificationCompleteLatch = new CountDownLatch(leases.size());
for (Lease lease : leases) {
ShutdownNotification shutdownNotification = new ShardConsumerShutdownNotification(leaseCoordinator,
lease, notificationCompleteLatch, shutdownCompleteLatch);
ShardInfo shardInfo = DynamoDBLeaseCoordinator.convertLeaseToAssignment(lease);
ShardConsumer consumer = shardInfoShardConsumerMap.get(shardInfo);
if (consumer != null) {
consumer.gracefulShutdown(shutdownNotification);
} else {
//
// There is a race condition between retrieving the current assignments, and creating the
// notification. If the a lease is lost in between these two points, we explicitly decrement the
// notification latches to clear the shutdown.
//
notificationCompleteLatch.countDown();
shutdownCompleteLatch.countDown();
}
}
return new GracefulShutdownContext(shutdownCompleteLatch, notificationCompleteLatch, this);
};
}
/**
* Signals worker to shutdown. Worker will try initiating shutdown of all record processors. Note that if executor
* services were passed to the worker by the user, worker will not attempt to shutdown those resources.
*
* <h2>Shutdown Process</h2> When called this will start shutdown of the record processor, and eventually shutdown
* the worker itself.
* <ol>
* <li>Call to start shutdown invoked</li>
* <li>Lease coordinator told to stop taking leases, and to drop existing leases.</li>
* <li>Worker discovers record processors that no longer have leases.</li>
* <li>Worker triggers shutdown with state {@link ShutdownReason#LEASE_LOST}.</li>
* <li>Once all record processors are shutdown, worker terminates owned resources.</li>
* <li>Shutdown complete.</li>
* </ol>
*/
public void shutdown() {
if (shutdown) {
log.warn("Shutdown requested a second time.");
return;
}
log.info("Worker shutdown requested.");
// Set shutdown flag, so Worker.run can start shutdown process.
shutdown = true;
shutdownStartTimeMillis = System.currentTimeMillis();
// Stop lease coordinator, so leases are not renewed or stolen from other workers.
// Lost leases will force Worker to begin shutdown process for all shard consumers in
// Worker.run().
leaseCoordinator.stop();
workerStateChangeListener.onWorkerStateChange(WorkerStateChangeListener.WorkerState.SHUT_DOWN);
}
/**
* Perform final shutdown related tasks for the worker including shutting down worker owned executor services,
* threads, etc.
*/
private void finalShutdown() {
log.info("Starting worker's final shutdown.");
if (executorService instanceof SchedulerCoordinatorFactory.SchedulerThreadPoolExecutor) {
// This should interrupt all active record processor tasks.
executorService.shutdownNow();
}
if (metricsFactory instanceof CloudWatchMetricsFactory) {
((CloudWatchMetricsFactory) metricsFactory).shutdown();
}
shutdownComplete = true;
}
private List<ShardInfo> getShardInfoForAssignments() {
List<ShardInfo> assignedStreamShards = leaseCoordinator.getCurrentAssignments();
List<ShardInfo> prioritizedShards = shardPrioritization.prioritize(assignedStreamShards);
if ((prioritizedShards != null) && (!prioritizedShards.isEmpty())) {
if (slog.isInfoEnabled()) {
StringBuilder builder = new StringBuilder();
boolean firstItem = true;
for (ShardInfo shardInfo : prioritizedShards) {
if (!firstItem) {
builder.append(", ");
}
builder.append(shardInfo.shardId());
firstItem = false;
}
slog.info("Current stream shard assignments: " + builder.toString());
}
} else {
slog.info("No activities assigned");
}
return prioritizedShards;
}
/**
* NOTE: This method is internal/private to the Worker class. It has package access solely for testing.
*
* @param shardInfo
* Kinesis shard info
* @return ShardConsumer for the shard
*/
ShardConsumer createOrGetShardConsumer(@NonNull final ShardInfo shardInfo,
@NonNull final ShardRecordProcessorFactory shardRecordProcessorFactory) {
ShardConsumer consumer = shardInfoShardConsumerMap.get(shardInfo);
// Instantiate a new consumer if we don't have one, or the one we
// had was from an earlier
// lease instance (and was shutdown). Don't need to create another
// one if the shard has been
// completely processed (shutdown reason terminate).
if ((consumer == null)
|| (consumer.isShutdown() && consumer.shutdownReason().equals(ShutdownReason.LEASE_LOST))) {
consumer = buildConsumer(shardInfo, shardRecordProcessorFactory);
shardInfoShardConsumerMap.put(shardInfo, consumer);
slog.infoForce("Created new shardConsumer for : " + shardInfo);
}
return consumer;
}
protected ShardConsumer buildConsumer(@NonNull final ShardInfo shardInfo,
@NonNull final ShardRecordProcessorFactory shardRecordProcessorFactory) {
RecordsPublisher cache = retrievalConfig.retrievalFactory().createGetRecordsCache(shardInfo, metricsFactory);
ShardRecordProcessorCheckpointer checkpointer = coordinatorConfig.coordinatorFactory().createRecordProcessorCheckpointer(shardInfo,
checkpoint);
ShardConsumerArgument argument = new ShardConsumerArgument(shardInfo,
streamName,
leaseRefresher,
executorService,
cache,
shardRecordProcessorFactory.shardRecordProcessor(),
checkpoint,
checkpointer,
parentShardPollIntervalMillis,
taskBackoffTimeMillis,
skipShardSyncAtWorkerInitializationIfLeasesExist,
listShardsBackoffTimeMillis,
maxListShardsRetryAttempts,
processorConfig.callProcessRecordsEvenForEmptyRecordList(),
shardConsumerDispatchPollIntervalMillis,
initialPosition,
cleanupLeasesUponShardCompletion,
ignoreUnexpetedChildShards,
shardDetector,
metricsFactory,
aggregatorUtil);
return new ShardConsumer(cache, executorService, shardInfo, lifecycleConfig.logWarningForTaskAfterMillis(), argument);
}
/**
* NOTE: This method is internal/private to the Worker class. It has package access solely for testing.
*
* This method relies on ShardInfo.equals() method returning true for ShardInfo objects which may have been
* instantiated with parentShardIds in a different order (and rest of the fields being the equal). For example
* shardInfo1.equals(shardInfo2) should return true with shardInfo1 and shardInfo2 defined as follows. ShardInfo
* shardInfo1 = new ShardInfo(shardId1, concurrencyToken1, Arrays.asList("parent1", "parent2")); ShardInfo
* shardInfo2 = new ShardInfo(shardId1, concurrencyToken1, Arrays.asList("parent2", "parent1"));
*/
void cleanupShardConsumers(Set<ShardInfo> assignedShards) {
for (ShardInfo shard : shardInfoShardConsumerMap.keySet()) {
if (!assignedShards.contains(shard)) {
// Shutdown the consumer since we are no longer responsible for
// the shard.
ShardConsumer consumer = shardInfoShardConsumerMap.get(shard);
if (consumer.leaseLost()) {
shardInfoShardConsumerMap.remove(shard);
log.debug("Removed consumer for {} as lease has been lost", shard.shardId());
} else {
consumer.executeLifecycle();
}
}
}
}
/**
* Logger for suppressing too much INFO logging. To avoid too much logging information Worker will output logging at
* INFO level for a single pass through the main loop every minute. At DEBUG level it will output all INFO logs on
* every pass.
*/
@NoArgsConstructor(access = AccessLevel.PRIVATE)
private static class SchedulerLog {
private long reportIntervalMillis = TimeUnit.MINUTES.toMillis(1);
private long nextReportTime = System.currentTimeMillis() + reportIntervalMillis;
private boolean infoReporting;
void info(Object message) {
if (this.isInfoEnabled()) {
log.info("{}", message);
}
}
void infoForce(Object message) {
log.info("{}", message);
}
private boolean isInfoEnabled() {
return infoReporting;
}
private void resetInfoLogging() {
if (infoReporting) {
// We just logged at INFO level for a pass through worker loop
if (log.isInfoEnabled()) {
infoReporting = false;
nextReportTime = System.currentTimeMillis() + reportIntervalMillis;
} // else is DEBUG or TRACE so leave reporting true
} else if (nextReportTime <= System.currentTimeMillis()) {
infoReporting = true;
}
}
}
@Deprecated
public Future<Void> requestShutdown() {
return null;
}
}

66
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/SchedulerCoordinatorFactory.java Normal file
View file

@ -0,0 +1,66 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import lombok.Data;
import lombok.NonNull;
import software.amazon.kinesis.checkpoint.ShardRecordProcessorCheckpointer;
import software.amazon.kinesis.leases.ShardInfo;
import software.amazon.kinesis.processor.Checkpointer;
/**
*
*/
@Data
public class SchedulerCoordinatorFactory implements CoordinatorFactory {
@Override
public ExecutorService createExecutorService() {
return new SchedulerThreadPoolExecutor(
new ThreadFactoryBuilder().setNameFormat("ShardRecordProcessor-%04d").build());
}
@Override
public GracefulShutdownCoordinator createGracefulShutdownCoordinator() {
return new GracefulShutdownCoordinator();
}
@Override
public WorkerStateChangeListener createWorkerStateChangeListener() {
return new NoOpWorkerStateChangeListener();
}
static class SchedulerThreadPoolExecutor extends ThreadPoolExecutor {
private static final long DEFAULT_KEEP_ALIVE = 60L;
SchedulerThreadPoolExecutor(ThreadFactory threadFactory) {
super(0, Integer.MAX_VALUE, DEFAULT_KEEP_ALIVE, TimeUnit.SECONDS, new SynchronousQueue<>(),
threadFactory);
}
}
@Override
public ShardRecordProcessorCheckpointer createRecordProcessorCheckpointer(@NonNull final ShardInfo shardInfo,
@NonNull final Checkpointer checkpoint) {
return new ShardRecordProcessorCheckpointer(shardInfo, checkpoint);
}
}

30
amazon-kinesis-client/src/main/java/software/amazon/kinesis/coordinator/WorkerStateChangeListener.java Normal file
View file

@ -0,0 +1,30 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.coordinator;
/**
* A listener for callbacks on changes worker state
*/
@FunctionalInterface
public interface WorkerStateChangeListener {
enum WorkerState {
CREATED,
INITIALIZING,
STARTED,
SHUT_DOWN
}
void onWorkerStateChange(WorkerState newState);
}

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/exceptions/InvalidStateException.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/InvalidStateException.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.exceptions;
package software.amazon.kinesis.exceptions;
/**
* This is thrown when the Amazon Kinesis Client Library encounters issues with its internal state (e.g. DynamoDB table

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/exceptions/KinesisClientLibDependencyException.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/KinesisClientLibDependencyException.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.exceptions;
package software.amazon.kinesis.exceptions;
/**
* This is thrown when the Amazon Kinesis Client Library encounters issues talking to its dependencies

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/exceptions/KinesisClientLibException.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/KinesisClientLibException.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.exceptions;
package software.amazon.kinesis.exceptions;
/**
* Abstract class for exceptions of the Amazon Kinesis Client Library.

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/exceptions/KinesisClientLibNonRetryableException.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/KinesisClientLibNonRetryableException.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.exceptions;
package software.amazon.kinesis.exceptions;
/**
* Non-retryable exceptions. Simply retrying the same request/operation is not expected to succeed.

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/exceptions/KinesisClientLibRetryableException.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/KinesisClientLibRetryableException.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.exceptions;
package software.amazon.kinesis.exceptions;
/**
* Retryable exceptions (e.g. transient errors). The request/operation is expected to succeed upon (back off and) retry.

39
amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/ShutdownException.java Normal file
View file

@ -0,0 +1,39 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.exceptions;
/**
* The ShardRecordProcessor instance has been shutdown (e.g. and attempts a checkpoint).
*/
public class ShutdownException extends KinesisClientLibNonRetryableException {
private static final long serialVersionUID = 1L;
/**
* @param message provides more details about the cause and potential ways to debug/address.
*/
public ShutdownException(String message) {
super(message);
}
/**
* @param message provides more details about the cause and potential ways to debug/address.
* @param e Cause of the exception
*/
public ShutdownException(String message, Exception e) {
super(message, e);
}
}

4
src/main/java/com/amazonaws/services/kinesis/clientlibrary/exceptions/ThrottlingException.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/ThrottlingException.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.exceptions;
package software.amazon.kinesis.exceptions;
/**
* Thrown when requests are throttled by a service (e.g. DynamoDB when storing a checkpoint).

6
src/main/java/com/amazonaws/services/kinesis/clientlibrary/exceptions/internal/BlockedOnParentShardException.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/internal/BlockedOnParentShardException.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -13,9 +13,9 @@
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.exceptions.internal;
package software.amazon.kinesis.exceptions.internal;
import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibRetryableException;
import software.amazon.kinesis.exceptions.KinesisClientLibRetryableException;
/**
* Used internally in the Amazon Kinesis Client Library. Indicates that we cannot start processing data for a shard

44
amazon-kinesis-client/src/main/java/software/amazon/kinesis/exceptions/internal/KinesisClientLibIOException.java Normal file
View file

@ -0,0 +1,44 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.exceptions.internal;
import software.amazon.kinesis.exceptions.KinesisClientLibRetryableException;
/**
* Thrown when we encounter issues when reading/writing information (e.g. shard information from Kinesis may not be
* current/complete).
*/
public class KinesisClientLibIOException extends KinesisClientLibRetryableException {
private static final long serialVersionUID = 1L;
/**
* Constructor.
*
* @param message Error message.
*/
public KinesisClientLibIOException(String message) {
super(message);
}
/**
* Constructor.
*
* @param message Error message.
* @param e Cause.
*/
public KinesisClientLibIOException(String message, Exception e) {
super(message, e);
}
}

20
src/main/java/com/amazonaws/services/kinesis/leases/util/DynamoUtils.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/DynamoUtils.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,15 +12,15 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.leases.util;
package software.amazon.kinesis.leases;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
/**
* Static utility functions used by our LeaseSerializers.
*/
@ -31,7 +31,7 @@ public class DynamoUtils {
throw new IllegalArgumentException("Collection attributeValues cannot be null or empty.");
}
return new AttributeValue().withSS(collectionValue);
return AttributeValue.builder().ss(collectionValue).build();
}
public static AttributeValue createAttributeValue(String stringValue) {
@ -39,7 +39,7 @@ public class DynamoUtils {
throw new IllegalArgumentException("String attributeValues cannot be null or empty.");
}
return new AttributeValue().withS(stringValue);
return AttributeValue.builder().s(stringValue).build();
}
public static AttributeValue createAttributeValue(Long longValue) {
@ -47,7 +47,7 @@ public class DynamoUtils {
throw new IllegalArgumentException("Number AttributeValues cannot be null.");
}
return new AttributeValue().withN(longValue.toString());
return AttributeValue.builder().n(longValue.toString()).build();
}
public static Long safeGetLong(Map<String, AttributeValue> dynamoRecord, String key) {
@ -55,7 +55,7 @@ public class DynamoUtils {
if (av == null) {
return null;
} else {
return new Long(av.getN());
return new Long(av.n());
}
}
@ -64,7 +64,7 @@ public class DynamoUtils {
if (av == null) {
return null;
} else {
return av.getS();
return av.s();
}
}
@ -74,7 +74,7 @@ public class DynamoUtils {
if (av == null) {
return new ArrayList<String>();
} else {
return av.getSS();
return av.ss();
}
}

217
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/KinesisShardDetector.java Normal file
View file

@ -0,0 +1,217 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.time.Duration;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Function;
import java.util.stream.Collectors;
import org.apache.commons.lang.StringUtils;
import lombok.AccessLevel;
import lombok.Getter;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.Synchronized;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.awssdk.services.kinesis.model.KinesisException;
import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
import software.amazon.awssdk.services.kinesis.model.ResourceInUseException;
import software.amazon.awssdk.services.kinesis.model.Shard;
import software.amazon.awssdk.utils.CollectionUtils;
import software.amazon.kinesis.common.KinesisRequestsBuilder;
import software.amazon.kinesis.retrieval.AWSExceptionManager;
/**
*
*/
@RequiredArgsConstructor
@Slf4j
@Accessors(fluent = true)
public class KinesisShardDetector implements ShardDetector {
@NonNull
private final KinesisAsyncClient kinesisClient;
@NonNull
private final String streamName;
private final long listShardsBackoffTimeInMillis;
private final int maxListShardsRetryAttempts;
private final long listShardsCacheAllowedAgeInSeconds;
private final int maxCacheMissesBeforeReload;
private final int cacheMissWarningModulus;
private volatile Map<String, Shard> cachedShardMap = null;
private volatile Instant lastCacheUpdateTime;
@Getter(AccessLevel.PACKAGE)
private AtomicInteger cacheMisses = new AtomicInteger(0);
@Override
public Shard shard(@NonNull final String shardId) {
if (CollectionUtils.isNullOrEmpty(this.cachedShardMap)) {
synchronized (this) {
if (CollectionUtils.isNullOrEmpty(this.cachedShardMap)) {
listShards();
}
}
}
Shard shard = cachedShardMap.get(shardId);
if (shard == null) {
if (cacheMisses.incrementAndGet() > maxCacheMissesBeforeReload || shouldRefreshCache()) {
synchronized (this) {
shard = cachedShardMap.get(shardId);
if (shard == null) {
log.info("Too many shard map cache misses or cache is out of date -- forcing a refresh");
listShards();
shard = cachedShardMap.get(shardId);
if (shard == null) {
log.warn("Even after cache refresh shard '{}' wasn't found. This could indicate a bigger"
+ " problem.", shardId);
}
cacheMisses.set(0);
} else {
//
// If the shardmap got updated, go ahead and set cache misses to 0
//
cacheMisses.set(0);
}
}
}
}
if (shard == null) {
final String message = String.format("Cannot find the shard given the shardId %s. Cache misses: %s",
shardId, cacheMisses);
if (cacheMisses.get() % cacheMissWarningModulus == 0) {
log.warn(message);
} else {
log.debug(message);
}
}
return shard;
}
@Override
@Synchronized
public List<Shard> listShards() {
final List<Shard> shards = new ArrayList<>();
ListShardsResponse result;
String nextToken = null;
do {
result = listShards(nextToken);
if (result == null) {
/*
* If listShards ever returns null, we should bail and return null. This indicates the stream is not
* in ACTIVE or UPDATING state and we may not have accurate/consistent information about the stream.
*/
return null;
} else {
shards.addAll(result.shards());
nextToken = result.nextToken();
}
} while (StringUtils.isNotEmpty(result.nextToken()));
cachedShardMap(shards);
return shards;
}
private ListShardsResponse listShards(final String nextToken) {
final AWSExceptionManager exceptionManager = new AWSExceptionManager();
exceptionManager.add(LimitExceededException.class, t -> t);
exceptionManager.add(ResourceInUseException.class, t -> t);
exceptionManager.add(KinesisException.class, t -> t);
ListShardsRequest.Builder request = KinesisRequestsBuilder.listShardsRequestBuilder();
if (StringUtils.isEmpty(nextToken)) {
request = request.streamName(streamName);
} else {
request = request.nextToken(nextToken);
}
ListShardsResponse result = null;
LimitExceededException lastException = null;
int remainingRetries = maxListShardsRetryAttempts;
while (result == null) {
try {
try {
result = kinesisClient.listShards(request.build()).get();
} catch (ExecutionException e) {
throw exceptionManager.apply(e.getCause());
} catch (InterruptedException e) {
// TODO: check if this is the correct behavior for Interrupted Exception
log.debug("Interrupted exception caught, shutdown initiated, returning null");
return null;
}
} catch (ResourceInUseException e) {
log.info("Stream is not in Active/Updating status, returning null (wait until stream is in"
+ " Active or Updating)");
return null;
} catch (LimitExceededException e) {
log.info("Got LimitExceededException when listing shards {}. Backing off for {} millis.", streamName,
listShardsBackoffTimeInMillis);
try {
Thread.sleep(listShardsBackoffTimeInMillis);
} catch (InterruptedException ie) {
log.debug("Stream {} : Sleep was interrupted ", streamName, ie);
}
lastException = e;
}
remainingRetries--;
if (remainingRetries <= 0 && result == null) {
if (lastException != null) {
throw lastException;
}
throw new IllegalStateException("Received null from ListShards call.");
}
}
return result;
}
void cachedShardMap(final List<Shard> shards) {
cachedShardMap = shards.stream().collect(Collectors.toMap(Shard::shardId, Function.identity()));
lastCacheUpdateTime = Instant.now();
}
private boolean shouldRefreshCache() {
final Duration secondsSinceLastUpdate = Duration.between(lastCacheUpdateTime, Instant.now());
final String message = String.format("Shard map cache is %d seconds old", secondsSinceLastUpdate.getSeconds());
if (secondsSinceLastUpdate.compareTo(Duration.of(listShardsCacheAllowedAgeInSeconds, ChronoUnit.SECONDS)) > 0) {
log.info("{}. Age exceeds limit of {} seconds -- Refreshing.", message, listShardsCacheAllowedAgeInSeconds);
return true;
}
log.debug("{}. Age doesn't exceed limit of {} seconds.", message, listShardsCacheAllowedAgeInSeconds);
return false;
}
}

221
src/main/java/com/amazonaws/services/kinesis/leases/impl/Lease.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/Lease.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,12 +12,23 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.leases.impl;
package software.amazon.kinesis.leases;
import java.util.Collection;
import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
import com.amazonaws.util.json.Jackson;
import com.google.common.collect.Collections2;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.NonNull;
import lombok.ToString;
import lombok.experimental.Accessors;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* This class contains data pertaining to a Lease. Distributed systems may use leases to partition work across a
@ -26,6 +37,11 @@ import com.amazonaws.util.json.Jackson;
* processing the corresponding unit of work, or until it fails. When the worker stops holding the lease, another worker will
* take and hold the lease.
*/
@NoArgsConstructor
@Getter
@Accessors(fluent = true)
@EqualsAndHashCode(exclude = {"concurrencyToken", "lastCounterIncrementNanos"})
@ToString
public class Lease {
/*
* See javadoc for System.nanoTime - summary:
@ -35,8 +51,17 @@ public class Lease {
*/
private static final long MAX_ABS_AGE_NANOS = TimeUnit.DAYS.toNanos(365);
/**
* @return leaseKey - identifies the unit of work associated with this lease.
*/
private String leaseKey;
/**
* @return current owner of the lease, may be null.
*/
private String leaseOwner;
/**
* @return leaseCounter is incremented periodically by the holder of the lease. Used for optimistic locking.
*/
private Long leaseCounter = 0L;
/*
@ -50,12 +75,20 @@ public class Lease {
* deliberately not persisted in DynamoDB and excluded from hashCode and equals.
*/
private Long lastCounterIncrementNanos;
/**
* Constructor.
* @return most recently application-supplied checkpoint value. During fail over, the new worker will pick up after
* the old worker's last checkpoint.
*/
public Lease() {
}
private ExtendedSequenceNumber checkpoint;
/**
* @return pending checkpoint, possibly null.
*/
private ExtendedSequenceNumber pendingCheckpoint;
/**
* @return count of distinct lease holders between checkpoints.
*/
private Long ownerSwitchesSinceCheckpoint = 0L;
private Set<String> parentShardIds = new HashSet<>();
/**
* Copy constructor, used by clone().
@ -63,62 +96,46 @@ public class Lease {
* @param lease lease to copy
*/
protected Lease(Lease lease) {
this(lease.getLeaseKey(), lease.getLeaseOwner(), lease.getLeaseCounter(), lease.getConcurrencyToken(),
lease.getLastCounterIncrementNanos());
this(lease.leaseKey(), lease.leaseOwner(), lease.leaseCounter(), lease.concurrencyToken(),
lease.lastCounterIncrementNanos(), lease.checkpoint(), lease.pendingCheckpoint(),
lease.ownerSwitchesSinceCheckpoint(), lease.parentShardIds());
}
protected Lease(String leaseKey, String leaseOwner, Long leaseCounter, UUID concurrencyToken,
Long lastCounterIncrementNanos) {
public Lease(final String leaseKey, final String leaseOwner, final Long leaseCounter,
final UUID concurrencyToken, final Long lastCounterIncrementNanos,
final ExtendedSequenceNumber checkpoint, final ExtendedSequenceNumber pendingCheckpoint,
final Long ownerSwitchesSinceCheckpoint, final Set<String> parentShardIds) {
this.leaseKey = leaseKey;
this.leaseOwner = leaseOwner;
this.leaseCounter = leaseCounter;
this.concurrencyToken = concurrencyToken;
this.lastCounterIncrementNanos = lastCounterIncrementNanos;
this.checkpoint = checkpoint;
this.pendingCheckpoint = pendingCheckpoint;
this.ownerSwitchesSinceCheckpoint = ownerSwitchesSinceCheckpoint;
if (parentShardIds != null) {
this.parentShardIds.addAll(parentShardIds);
}
}
/**
* @return shardIds that parent this lease. Used for resharding.
*/
public Set<String> parentShardIds() {
return new HashSet<>(parentShardIds);
}
/**
* Updates this Lease's mutable, application-specific fields based on the passed-in lease object. Does not update
* fields that are internal to the leasing library (leaseKey, leaseOwner, leaseCounter).
*
* @param other
* @param lease
*/
public <T extends Lease> void update(T other) {
// The default implementation (no application-specific fields) has nothing to do.
}
/**
* @return leaseKey - identifies the unit of work associated with this lease.
*/
public String getLeaseKey() {
return leaseKey;
}
/**
* @return leaseCounter is incremented periodically by the holder of the lease. Used for optimistic locking.
*/
public Long getLeaseCounter() {
return leaseCounter;
}
/**
* @return current owner of the lease, may be null.
*/
public String getLeaseOwner() {
return leaseOwner;
}
/**
* @return concurrency token
*/
public UUID getConcurrencyToken() {
return concurrencyToken;
}
/**
* @return last update in nanoseconds since the epoch
*/
public Long getLastCounterIncrementNanos() {
return lastCounterIncrementNanos;
public void update(final Lease lease) {
ownerSwitchesSinceCheckpoint(lease.ownerSwitchesSinceCheckpoint());
checkpoint(lease.checkpoint);
pendingCheckpoint(lease.pendingCheckpoint);
parentShardIds(lease.parentShardIds);
}
/**
@ -145,7 +162,7 @@ public class Lease {
*
* @param lastCounterIncrementNanos last renewal in nanoseconds since the epoch
*/
public void setLastCounterIncrementNanos(Long lastCounterIncrementNanos) {
public void lastCounterIncrementNanos(Long lastCounterIncrementNanos) {
this.lastCounterIncrementNanos = lastCounterIncrementNanos;
}
@ -154,8 +171,7 @@ public class Lease {
*
* @param concurrencyToken may not be null
*/
public void setConcurrencyToken(UUID concurrencyToken) {
verifyNotNull(concurrencyToken, "concurencyToken cannot be null");
public void concurrencyToken(@NonNull final UUID concurrencyToken) {
this.concurrencyToken = concurrencyToken;
}
@ -164,12 +180,10 @@ public class Lease {
*
* @param leaseKey may not be null.
*/
public void setLeaseKey(String leaseKey) {
public void leaseKey(@NonNull final String leaseKey) {
if (this.leaseKey != null) {
throw new IllegalArgumentException("LeaseKey is immutable once set");
}
verifyNotNull(leaseKey, "LeaseKey cannot be set to null");
this.leaseKey = leaseKey;
}
@ -178,77 +192,62 @@ public class Lease {
*
* @param leaseCounter may not be null
*/
public void setLeaseCounter(Long leaseCounter) {
verifyNotNull(leaseCounter, "leaseCounter must not be null");
public void leaseCounter(@NonNull final Long leaseCounter) {
this.leaseCounter = leaseCounter;
}
/**
* Sets checkpoint.
*
* @param checkpoint may not be null
*/
public void checkpoint(@NonNull final ExtendedSequenceNumber checkpoint) {
this.checkpoint = checkpoint;
}
/**
* Sets pending checkpoint.
*
* @param pendingCheckpoint can be null
*/
public void pendingCheckpoint(ExtendedSequenceNumber pendingCheckpoint) {
this.pendingCheckpoint = pendingCheckpoint;
}
/**
* Sets ownerSwitchesSinceCheckpoint.
*
* @param ownerSwitchesSinceCheckpoint may not be null
*/
public void ownerSwitchesSinceCheckpoint(@NonNull final Long ownerSwitchesSinceCheckpoint) {
this.ownerSwitchesSinceCheckpoint = ownerSwitchesSinceCheckpoint;
}
/**
* Sets parentShardIds.
*
* @param parentShardIds may not be null
*/
public void parentShardIds(@NonNull final Collection<String> parentShardIds) {
this.parentShardIds.clear();
this.parentShardIds.addAll(parentShardIds);
}
/**
* Sets leaseOwner.
*
* @param leaseOwner may be null.
*/
public void setLeaseOwner(String leaseOwner) {
public void leaseOwner(String leaseOwner) {
this.leaseOwner = leaseOwner;
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((leaseCounter == null) ? 0 : leaseCounter.hashCode());
result = prime * result + ((leaseOwner == null) ? 0 : leaseOwner.hashCode());
result = prime * result + ((leaseKey == null) ? 0 : leaseKey.hashCode());
return result;
}
@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
Lease other = (Lease) obj;
if (leaseCounter == null) {
if (other.leaseCounter != null)
return false;
} else if (!leaseCounter.equals(other.leaseCounter))
return false;
if (leaseOwner == null) {
if (other.leaseOwner != null)
return false;
} else if (!leaseOwner.equals(other.leaseOwner))
return false;
if (leaseKey == null) {
if (other.leaseKey != null)
return false;
} else if (!leaseKey.equals(other.leaseKey))
return false;
return true;
}
@Override
public String toString() {
return Jackson.toJsonPrettyString(this);
}
/**
* Returns a deep copy of this object. Type-unsafe - there aren't good mechanisms for copy-constructing generics.
*
* @return A deep copy of this object.
*/
@SuppressWarnings("unchecked")
public <T extends Lease> T copy() {
return (T) new Lease(this);
}
private void verifyNotNull(Object object, String message) {
if (object == null) {
throw new IllegalArgumentException(message);
public Lease copy() {
return new Lease(this);
}
}
}

140
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseCoordinator.java Normal file
View file

@ -0,0 +1,140 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.Collection;
import java.util.List;
import java.util.UUID;
import software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
/**
*
*/
public interface LeaseCoordinator {
/**
* Initialize the lease coordinator (create the lease table if needed).
* @throws DependencyException
* @throws ProvisionedThroughputException
*/
void initialize() throws ProvisionedThroughputException, DependencyException, IllegalStateException;
/**
* Start background LeaseHolder and LeaseTaker threads.
* @throws ProvisionedThroughputException If we can't talk to DynamoDB due to insufficient capacity.
* @throws InvalidStateException If the lease table doesn't exist
* @throws DependencyException If we encountered exception taking to DynamoDB
*/
void start() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Runs a single iteration of the lease taker - used by integration tests.
*
* @throws InvalidStateException
* @throws DependencyException
*/
void runLeaseTaker() throws DependencyException, InvalidStateException;
/**
* Runs a single iteration of the lease renewer - used by integration tests.
*
* @throws InvalidStateException
* @throws DependencyException
*/
void runLeaseRenewer() throws DependencyException, InvalidStateException;
/**
* @return true if this LeaseCoordinator is running
*/
boolean isRunning();
/**
* @return workerIdentifier
*/
String workerIdentifier();
/**
* @return {@link LeaseRefresher}
*/
LeaseRefresher leaseRefresher();
/**
* @return currently held leases
*/
Collection<Lease> getAssignments();
/**
* @param leaseKey lease key to fetch currently held lease for
*
* @return deep copy of currently held Lease for given key, or null if we don't hold the lease for that key
*/
Lease getCurrentlyHeldLease(String leaseKey);
/**
* Updates application-specific lease values in DynamoDB.
*
* @param lease lease object containing updated values
* @param concurrencyToken obtained by calling Lease.concurrencyToken for a currently held lease
*
* @return true if update succeeded, false otherwise
*
* @throws InvalidStateException if lease table does not exist
* @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
* @throws DependencyException if DynamoDB update fails in an unexpected way
*/
boolean updateLease(Lease lease, UUID concurrencyToken, String operation, String shardId)
throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Requests the cancellation of the lease taker.
*/
void stopLeaseTaker();
/**
* Requests that renewals for the given lease are stopped.
*
* @param lease the lease to stop renewing.
*/
void dropLease(Lease lease);
/**
* Stops background threads and waits for specific amount of time for all background tasks to complete.
* If tasks are not completed after this time, method will shutdown thread pool forcefully and return.
*/
void stop();
/**
* @return Current shard/lease assignments
*/
List<ShardInfo> getCurrentAssignments();
/**
* @param writeCapacity The DynamoDB table used for tracking leases will be provisioned with the specified initial
* write capacity
* @return LeaseCoordinator
*/
DynamoDBLeaseCoordinator initialLeaseTableWriteCapacity(long writeCapacity);
/**
* @param readCapacity The DynamoDB table used for tracking leases will be provisioned with the specified initial
* read capacity
* @return LeaseCoordinator
*/
DynamoDBLeaseCoordinator initialLeaseTableReadCapacity(long readCapacity);
}

224
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseManagementConfig.java Normal file
View file

@ -0,0 +1,224 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
import software.amazon.kinesis.common.InitialPositionInStream;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;
import software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseManagementFactory;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.NullMetricsFactory;
/**
* Used by the KCL to configure lease management.
*/
@Data
@Accessors(fluent = true)
public class LeaseManagementConfig {
/**
* Name of the table to use in DynamoDB
*
* @return String
*/
@NonNull
private final String tableName;
/**
* Client to be used to access DynamoDB service.
*
* @return {@link DynamoDbAsyncClient}
*/
@NonNull
private final DynamoDbAsyncClient dynamoDBClient;
/**
* Client to be used to access Kinesis Data Streams service.
*
* @return {@link KinesisAsyncClient}
*/
@NonNull
private final KinesisAsyncClient kinesisClient;
/**
* Name of the Kinesis Data Stream to read records from.
*/
@NonNull
private final String streamName;
/**
* Used to distinguish different workers/processes of a KCL application.
*
* @return String
*/
@NonNull
private final String workerIdentifier;
/**
* Fail over time in milliseconds. A worker which does not renew it's lease within this time interval
* will be regarded as having problems and it's shards will be assigned to other workers.
* For applications that have a large number of shards, this may be set to a higher number to reduce
* the number of DynamoDB IOPS required for tracking leases.
*
* <p>Default value: 10000L</p>
*/
private long failoverTimeMillis = 10000L;
/**
* Shard sync interval in milliseconds - e.g. wait for this long between shard sync tasks.
*
* <p>Default value: 60000L</p>
*/
private long shardSyncIntervalMillis = 60000L;
/**
* Cleanup leases upon shards completion (don't wait until they expire in Kinesis).
* Keeping leases takes some tracking/resources (e.g. they need to be renewed, assigned), so by default we try
* to delete the ones we don't need any longer.
*
* <p>Default value: true</p>
*/
private boolean cleanupLeasesUponShardCompletion = true;
/**
* The max number of leases (shards) this worker should process.
* This can be useful to avoid overloading (and thrashing) a worker when a host has resource constraints
* or during deployment.
*
* <p>NOTE: Setting this to a low value can cause data loss if workers are not able to pick up all shards in the
* stream due to the max limit.</p>
*
* <p>Default value: {@link Integer#MAX_VALUE}</p>
*/
private int maxLeasesForWorker = Integer.MAX_VALUE;;
/**
* Max leases to steal from another worker at one time (for load balancing).
* Setting this to a higher number can allow for faster load convergence (e.g. during deployments, cold starts),
* but can cause higher churn in the system.
*
* <p>Default value: 1</p>
*/
private int maxLeasesToStealAtOneTime = 1;
/**
* The Amazon DynamoDB table used for tracking leases will be provisioned with this read capacity.
*
* <p>Default value: 10</p>
*/
private int initialLeaseTableReadCapacity = 10;
/**
* The Amazon DynamoDB table used for tracking leases will be provisioned with this write capacity.
*
* <p>Default value: 10</p>
*/
private int initialLeaseTableWriteCapacity = 10;
/**
* The size of the thread pool to create for the lease renewer to use.
*
* <p>Default value: 20</p>
*/
private int maxLeaseRenewalThreads = 20;
/**
*
*/
private boolean ignoreUnexpectedChildShards = false;
/**
*
*/
private boolean consistentReads = false;
private long listShardsBackoffTimeInMillis = 1500L;
private int maxListShardsRetryAttempts = 50;
public long epsilonMillis = 25L;
/**
* The initial position for getting records from Kinesis streams.
*
* <p>Default value: {@link InitialPositionInStream#TRIM_HORIZON}</p>
*/
private InitialPositionInStreamExtended initialPositionInStream =
InitialPositionInStreamExtended.newInitialPosition(InitialPositionInStream.TRIM_HORIZON);
private int maxCacheMissesBeforeReload = 1000;
private long listShardsCacheAllowedAgeInSeconds = 30;
private int cacheMissWarningModulus = 250;
/**
*
*/
private MetricsFactory metricsFactory = new NullMetricsFactory();
/**
* The {@link ExecutorService} to be used by {@link ShardSyncTaskManager}.
*
* <p>Default value: {@link LeaseManagementThreadPool}</p>
*/
private ExecutorService executorService = new LeaseManagementThreadPool(
new ThreadFactoryBuilder().setNameFormat("ShardSyncTaskManager-%04d").build());
static class LeaseManagementThreadPool extends ThreadPoolExecutor {
private static final long DEFAULT_KEEP_ALIVE_TIME = 60L;
LeaseManagementThreadPool(ThreadFactory threadFactory) {
super(0, Integer.MAX_VALUE, DEFAULT_KEEP_ALIVE_TIME, TimeUnit.SECONDS, new SynchronousQueue<>(),
threadFactory);
}
};
private LeaseManagementFactory leaseManagementFactory;
public LeaseManagementFactory leaseManagementFactory() {
if (leaseManagementFactory == null) {
leaseManagementFactory = new DynamoDBLeaseManagementFactory(kinesisClient(),
streamName(),
dynamoDBClient(),
tableName(),
workerIdentifier(),
executorService(),
initialPositionInStream(),
failoverTimeMillis(),
epsilonMillis(),
maxLeasesForWorker(),
maxLeasesToStealAtOneTime(),
maxLeaseRenewalThreads(),
cleanupLeasesUponShardCompletion(),
ignoreUnexpectedChildShards(),
shardSyncIntervalMillis(),
consistentReads(),
listShardsBackoffTimeInMillis(),
maxListShardsRetryAttempts(),
maxCacheMissesBeforeReload(),
listShardsCacheAllowedAgeInSeconds(),
cacheMissWarningModulus());
}
return leaseManagementFactory;
}
}

32
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseManagementFactory.java Normal file
View file

@ -0,0 +1,32 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseRefresher;
import software.amazon.kinesis.metrics.MetricsFactory;
/**
*
*/
public interface LeaseManagementFactory {
LeaseCoordinator createLeaseCoordinator(MetricsFactory metricsFactory);
ShardSyncTaskManager createShardSyncTaskManager(MetricsFactory metricsFactory);
DynamoDBLeaseRefresher createLeaseRefresher();
ShardDetector createShardDetector();
}

56
src/main/java/com/amazonaws/services/kinesis/leases/interfaces/ILeaseManager.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseRefresher.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,21 +12,19 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.leases.interfaces;
package software.amazon.kinesis.leases;
import java.util.List;
import com.amazonaws.services.kinesis.leases.exceptions.DependencyException;
import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.leases.exceptions.ProvisionedThroughputException;
import com.amazonaws.services.kinesis.leases.impl.Lease;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Supports basic CRUD operations for Leases.
*
* @param <T> Lease subclass, possibly Lease itself.
*/
public interface ILeaseManager<T extends Lease> {
public interface LeaseRefresher {
/**
* Creates the table that will store leases. Succeeds if table already exists.
@ -40,7 +38,7 @@ public interface ILeaseManager<T extends Lease> {
* restrictions.
* @throws DependencyException if DynamoDB createTable fails in an unexpected way
*/
public boolean createLeaseTableIfNotExists(Long readCapacity, Long writeCapacity)
boolean createLeaseTableIfNotExists(Long readCapacity, Long writeCapacity)
throws ProvisionedThroughputException, DependencyException;
/**
@ -48,7 +46,7 @@ public interface ILeaseManager<T extends Lease> {
*
* @throws DependencyException if DynamoDB describeTable fails in an unexpected way
*/
public boolean leaseTableExists() throws DependencyException;
boolean leaseTableExists() throws DependencyException;
/**
* Blocks until the lease table exists by polling leaseTableExists.
@ -60,7 +58,7 @@ public interface ILeaseManager<T extends Lease> {
*
* @throws DependencyException if DynamoDB describeTable fails in an unexpected way
*/
public boolean waitUntilLeaseTableExists(long secondsBetweenPolls, long timeoutSeconds) throws DependencyException;
boolean waitUntilLeaseTableExists(long secondsBetweenPolls, long timeoutSeconds) throws DependencyException;
/**
* List all objects in table synchronously.
@ -71,7 +69,7 @@ public interface ILeaseManager<T extends Lease> {
*
* @return list of leases
*/
public List<T> listLeases() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
List<Lease> listLeases() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Create a new lease. Conditional on a lease not already existing with this shardId.
@ -84,7 +82,7 @@ public interface ILeaseManager<T extends Lease> {
* @throws InvalidStateException if lease table does not exist
* @throws ProvisionedThroughputException if DynamoDB put fails due to lack of capacity
*/
public boolean createLeaseIfNotExists(T lease)
boolean createLeaseIfNotExists(Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
@ -96,7 +94,7 @@ public interface ILeaseManager<T extends Lease> {
*
* @return lease for the specified shardId, or null if one doesn't exist
*/
public T getLease(String shardId) throws DependencyException, InvalidStateException, ProvisionedThroughputException;
Lease getLease(String shardId) throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Renew a lease by incrementing the lease counter. Conditional on the leaseCounter in DynamoDB matching the leaseCounter
@ -110,7 +108,7 @@ public interface ILeaseManager<T extends Lease> {
* @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
* @throws DependencyException if DynamoDB update fails in an unexpected way
*/
public boolean renewLease(T lease)
boolean renewLease(Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
@ -127,7 +125,7 @@ public interface ILeaseManager<T extends Lease> {
* @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
* @throws DependencyException if DynamoDB update fails in an unexpected way
*/
public boolean takeLease(T lease, String owner)
boolean takeLease(Lease lease, String owner)
throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
@ -142,7 +140,7 @@ public interface ILeaseManager<T extends Lease> {
* @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
* @throws DependencyException if DynamoDB update fails in an unexpected way
*/
public boolean evictLease(T lease)
boolean evictLease(Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
@ -154,7 +152,7 @@ public interface ILeaseManager<T extends Lease> {
* @throws ProvisionedThroughputException if DynamoDB delete fails due to lack of capacity
* @throws DependencyException if DynamoDB delete fails in an unexpected way
*/
public void deleteLease(T lease) throws DependencyException, InvalidStateException, ProvisionedThroughputException;
void deleteLease(Lease lease) throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Delete all leases from DynamoDB. Useful for tools/utils and testing.
@ -163,7 +161,7 @@ public interface ILeaseManager<T extends Lease> {
* @throws ProvisionedThroughputException if DynamoDB scan or delete fail due to lack of capacity
* @throws DependencyException if DynamoDB scan or delete fail in an unexpected way
*/
public void deleteAll() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
void deleteAll() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Update application-specific fields of the given lease in DynamoDB. Does not update fields managed by the leasing
@ -177,7 +175,7 @@ public interface ILeaseManager<T extends Lease> {
* @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
* @throws DependencyException if DynamoDB update fails in an unexpected way
*/
public boolean updateLease(T lease)
boolean updateLease(Lease lease)
throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
@ -189,6 +187,20 @@ public interface ILeaseManager<T extends Lease> {
* @throws InvalidStateException if lease table does not exist
* @throws ProvisionedThroughputException if DynamoDB scan fails due to lack of capacity
*/
public boolean isLeaseTableEmpty() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
boolean isLeaseTableEmpty() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Gets the current checkpoint of the shard. This is useful in the resharding use case
* where we will wait for the parent shard to complete before starting on the records from a child shard.
*
* @param shardId Checkpoint of this shard will be returned
* @return Checkpoint of this shard, or null if the shard record doesn't exist.
*
* @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
* @throws InvalidStateException if lease table does not exist
* @throws DependencyException if DynamoDB update fails in an unexpected way
*/
ExtendedSequenceNumber getCheckpoint(String shardId)
throws ProvisionedThroughputException, InvalidStateException, DependencyException;
}

33
src/main/java/com/amazonaws/services/kinesis/leases/interfaces/ILeaseRenewer.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseRenewer.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,31 +12,30 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.leases.interfaces;
package software.amazon.kinesis.leases;
import java.util.Collection;
import java.util.Map;
import java.util.UUID;
import com.amazonaws.services.kinesis.leases.exceptions.DependencyException;
import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.leases.exceptions.ProvisionedThroughputException;
import com.amazonaws.services.kinesis.leases.impl.Lease;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
/**
* ILeaseRenewer objects are used by LeaseCoordinator to renew leases held by the LeaseCoordinator. Each
* LeaseCoordinator instance corresponds to one worker, and uses exactly one ILeaseRenewer to manage lease renewal for
* that worker.
*/
public interface ILeaseRenewer<T extends Lease> {
public interface LeaseRenewer {
/**
* Bootstrap initial set of leases from the LeaseManager (e.g. upon process restart, pick up leases we own)
* Bootstrap initial set of leases from the {@link LeaseRefresher} (e.g. upon process restart, pick up leases we own)
* @throws DependencyException on unexpected DynamoDB failures
* @throws InvalidStateException if lease table doesn't exist
* @throws ProvisionedThroughputException if DynamoDB reads fail due to insufficient capacity
*/
public void initialize() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
void initialize() throws DependencyException, InvalidStateException, ProvisionedThroughputException;
/**
* Attempt to renew all currently held leases.
@ -44,21 +43,21 @@ public interface ILeaseRenewer<T extends Lease> {
* @throws DependencyException on unexpected DynamoDB failures
* @throws InvalidStateException if lease table does not exist
*/
public void renewLeases() throws DependencyException, InvalidStateException;
void renewLeases() throws DependencyException, InvalidStateException;
/**
* @return currently held leases. Key is shardId, value is corresponding Lease object. A lease is currently held if
* we successfully renewed it on the last run of renewLeases(). Lease objects returned are deep copies -
* their lease counters will not tick.
*/
public Map<String, T> getCurrentlyHeldLeases();
Map<String, Lease> getCurrentlyHeldLeases();
/**
* @param leaseKey key of the lease to retrieve
*
* @return a deep copy of a currently held lease, or null if we don't hold the lease
*/
public T getCurrentlyHeldLease(String leaseKey);
Lease getCurrentlyHeldLease(String leaseKey);
/**
* Adds leases to this LeaseRenewer's set of currently held leases. Leases must have lastRenewalNanos set to the
@ -66,19 +65,19 @@ public interface ILeaseRenewer<T extends Lease> {
*
* @param newLeases new leases.
*/
public void addLeasesToRenew(Collection<T> newLeases);
void addLeasesToRenew(Collection<Lease> newLeases);
/**
* Clears this LeaseRenewer's set of currently held leases.
*/
public void clearCurrentlyHeldLeases();
void clearCurrentlyHeldLeases();
/**
* Stops the lease renewer from continunig to maintain the given lease.
*
* @param lease the lease to drop.
*/
void dropLease(T lease);
void dropLease(Lease lease);
/**
* Update application-specific fields in a currently held lease. Cannot be used to update internal fields such as
@ -86,7 +85,7 @@ public interface ILeaseRenewer<T extends Lease> {
* the concurrency token on the internal authoritative copy of the lease (ie, if we lost and re-acquired the lease).
*
* @param lease lease object containing updated data
* @param concurrencyToken obtained by calling Lease.getConcurrencyToken for a currently held lease
* @param concurrencyToken obtained by calling Lease.concurrencyToken for a currently held lease
*
* @return true if update succeeds, false otherwise
*
@ -94,7 +93,7 @@ public interface ILeaseRenewer<T extends Lease> {
* @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
* @throws DependencyException if DynamoDB update fails in an unexpected way
*/
boolean updateLease(T lease, UUID concurrencyToken)
boolean updateLease(Lease lease, UUID concurrencyToken, String operation, String shardId)
throws DependencyException, InvalidStateException, ProvisionedThroughputException;
}

115
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseSerializer.java Normal file
View file

@ -0,0 +1,115 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.Collection;
import java.util.Map;
import software.amazon.awssdk.services.dynamodb.model.AttributeDefinition;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.AttributeValueUpdate;
import software.amazon.awssdk.services.dynamodb.model.ExpectedAttributeValue;
import software.amazon.awssdk.services.dynamodb.model.KeySchemaElement;
import software.amazon.kinesis.leases.Lease;
/**
* Utility class that manages the mapping of Lease objects/operations to records in DynamoDB.
*/
public interface LeaseSerializer {
/**
* Construct a DynamoDB record out of a Lease object
*
* @param lease lease object to serialize
* @return an attribute value map representing the lease object
*/
Map<String, AttributeValue> toDynamoRecord(Lease lease);
/**
* Construct a Lease object out of a DynamoDB record.
*
* @param dynamoRecord attribute value map from DynamoDB
* @return a deserialized lease object representing the attribute value map
*/
Lease fromDynamoRecord(Map<String, AttributeValue> dynamoRecord);
/**
* @param lease
* @return the attribute value map representing a Lease's hash key given a Lease object.
*/
Map<String, AttributeValue> getDynamoHashKey(Lease lease);
/**
* Special getDynamoHashKey implementation used by {@link LeaseRefresher#getLease(String)}.
*
* @param leaseKey
* @return the attribute value map representing a Lease's hash key given a string.
*/
Map<String, AttributeValue> getDynamoHashKey(String leaseKey);
/**
* @param lease
* @return the attribute value map asserting that a lease counter is what we expect.
*/
Map<String, ExpectedAttributeValue> getDynamoLeaseCounterExpectation(Lease lease);
/**
* @param lease
* @return the attribute value map asserting that the lease owner is what we expect.
*/
Map<String, ExpectedAttributeValue> getDynamoLeaseOwnerExpectation(Lease lease);
/**
* @return the attribute value map asserting that a lease does not exist.
*/
Map<String, ExpectedAttributeValue> getDynamoNonexistantExpectation();
/**
* @param lease
* @return the attribute value map that increments a lease counter
*/
Map<String, AttributeValueUpdate> getDynamoLeaseCounterUpdate(Lease lease);
/**
* @param lease
* @param newOwner
* @return the attribute value map that takes a lease for a new owner
*/
Map<String, AttributeValueUpdate> getDynamoTakeLeaseUpdate(Lease lease, String newOwner);
/**
* @param lease
* @return the attribute value map that voids a lease
*/
Map<String, AttributeValueUpdate> getDynamoEvictLeaseUpdate(Lease lease);
/**
* @param lease
* @return the attribute value map that updates application-specific data for a lease and increments the lease
* counter
*/
Map<String, AttributeValueUpdate> getDynamoUpdateLeaseUpdate(Lease lease);
/**
* @return the key schema for creating a DynamoDB table to store leases
*/
Collection<KeySchemaElement> getKeySchema();
/**
* @return attribute definitions for creating a DynamoDB table to store leases
*/
Collection<AttributeDefinition> getAttributeDefinitions();
}

15
src/main/java/com/amazonaws/services/kinesis/leases/interfaces/ILeaseTaker.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseTaker.java
View file

@ -1,5 +1,5 @@
/*
* Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
@ -12,19 +12,18 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.leases.interfaces;
package software.amazon.kinesis.leases;
import java.util.Map;
import com.amazonaws.services.kinesis.leases.exceptions.DependencyException;
import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
import com.amazonaws.services.kinesis.leases.impl.Lease;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
/**
* ILeaseTaker is used by LeaseCoordinator to take new leases, or leases that other workers fail to renew. Each
* LeaseCoordinator instance corresponds to one worker and uses exactly one ILeaseTaker to take leases for that worker.
*/
public interface ILeaseTaker<T extends Lease> {
public interface LeaseTaker {
/**
* Compute the set of leases available to be taken and attempt to take them. Lease taking rules are:
@ -39,11 +38,11 @@ public interface ILeaseTaker<T extends Lease> {
* @throws DependencyException on unexpected DynamoDB failures
* @throws InvalidStateException if lease table does not exist
*/
public abstract Map<String, T> takeLeases() throws DependencyException, InvalidStateException;
Map<String, Lease> takeLeases() throws DependencyException, InvalidStateException;
/**
* @return workerIdentifier for this LeaseTaker
*/
public abstract String getWorkerIdentifier();
String getWorkerIdentifier();
}

2
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/NoOpShardPrioritization.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/NoOpShardPrioritization.java
View file

@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.leases;
import java.util.List;

8
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ParentsFirstShardPrioritization.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ParentsFirstShardPrioritization.java
View file

@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.leases;
import java.util.ArrayList;
import java.util.Collections;
@ -51,14 +51,14 @@ public class ParentsFirstShardPrioritization implements
public List<ShardInfo> prioritize(List<ShardInfo> original) {
Map<String, ShardInfo> shards = new HashMap<>();
for (ShardInfo shardInfo : original) {
shards.put(shardInfo.getShardId(),
shards.put(shardInfo.shardId(),
shardInfo);
}
Map<String, SortingNode> processedNodes = new HashMap<>();
for (ShardInfo shardInfo : original) {
populateDepth(shardInfo.getShardId(),
populateDepth(shardInfo.shardId(),
shards,
processedNodes);
}
@ -104,7 +104,7 @@ public class ParentsFirstShardPrioritization implements
processedNodes.put(shardId, PROCESSING_NODE);
int maxParentDepth = 0;
for (String parentId : shardInfo.getParentShardIds()) {
for (String parentId : shardInfo.parentShardIds()) {
maxParentDepth = Math.max(maxParentDepth,
populateDepth(parentId,
shards,

15
src/test/java/com/amazonaws/services/dynamodbv2/streamsadapter/AmazonDynamoDBStreamsAdapterClientChild.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardDetector.java
View file

@ -13,11 +13,18 @@
* permissions and limitations under the License.
*/
package com.amazonaws.services.dynamodbv2.streamsadapter;
package software.amazon.kinesis.leases;
import software.amazon.awssdk.services.kinesis.model.Shard;
import java.util.List;
/**
* This class is only used for testing purposes, to make sure that the correct calls are made while using DynamoDB
* streams.
*
*/
public class AmazonDynamoDBStreamsAdapterClientChild extends AmazonDynamoDBStreamsAdapterClient {
public interface ShardDetector {
Shard shard(String shardId);
List<Shard> listShards();
}

55
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardInfo.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardInfo.java
View file

@ -12,21 +12,28 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.leases;
import java.util.Collection;
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
import lombok.Getter;
import lombok.NonNull;
import lombok.ToString;
import lombok.experimental.Accessors;
import org.apache.commons.lang.builder.EqualsBuilder;
import org.apache.commons.lang.builder.HashCodeBuilder;
import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Used to pass shard related info among different classes and as a key to the map of shard consumers.
*/
@Getter
@Accessors(fluent = true)
@ToString
public class ShardInfo {
private final String shardId;
@ -47,13 +54,14 @@ public class ShardInfo {
* @param checkpoint
* the latest checkpoint from lease
*/
public ShardInfo(String shardId,
String concurrencyToken,
Collection<String> parentShardIds,
ExtendedSequenceNumber checkpoint) {
// TODO: check what values can be null
public ShardInfo(@NonNull final String shardId,
final String concurrencyToken,
final Collection<String> parentShardIds,
final ExtendedSequenceNumber checkpoint) {
this.shardId = shardId;
this.concurrencyToken = concurrencyToken;
this.parentShardIds = new LinkedList<String>();
this.parentShardIds = new LinkedList<>();
if (parentShardIds != null) {
this.parentShardIds.addAll(parentShardIds);
}
@ -63,31 +71,13 @@ public class ShardInfo {
this.checkpoint = checkpoint;
}
/**
* The shardId that this ShardInfo contains data about
*
* @return the shardId
*/
public String getShardId() {
return shardId;
}
/**
* Concurrency token for the lease that this shard is part of
*
* @return the concurrencyToken
*/
public String getConcurrencyToken() {
return concurrencyToken;
}
/**
* A list of shards that are parents of this shard. This may be empty if the shard has no parents.
*
* @return a list of shardId's that are parents of this shard, or empty if the shard has no parents.
*/
protected List<String> getParentShardIds() {
return new LinkedList<String>(parentShardIds);
public List<String> parentShardIds() {
return new LinkedList<>(parentShardIds);
}
/**
@ -95,7 +85,7 @@ public class ShardInfo {
*
* @return completion status of the shard
*/
protected boolean isCompleted() {
public boolean isCompleted() {
return ExtendedSequenceNumber.SHARD_END.equals(checkpoint);
}
@ -132,13 +122,4 @@ public class ShardInfo {
}
@Override
public String toString() {
return "ShardInfo [shardId=" + shardId + ", concurrencyToken=" + concurrencyToken + ", parentShardIds="
+ parentShardIds + ", checkpoint=" + checkpoint + "]";
}
}

2
src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardPrioritization.java → amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardPrioritization.java
View file

@ -12,7 +12,7 @@
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
package software.amazon.kinesis.leases;
import java.util.List;

87
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardSyncTask.java Normal file
View file

@ -0,0 +1,87 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.lifecycle.ConsumerTask;
import software.amazon.kinesis.lifecycle.TaskResult;
import software.amazon.kinesis.lifecycle.TaskType;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsUtil;
/**
* This task syncs leases/activies with shards of the stream.
* It will create new leases/activites when it discovers new shards (e.g. setup/resharding).
* It will clean up leases/activities for shards that have been completely processed (if
* cleanupLeasesUponShardCompletion is true).
*/
@RequiredArgsConstructor
@Slf4j
public class ShardSyncTask implements ConsumerTask {
private final String SHARD_SYNC_TASK_OPERATION = "ShardSyncTask";
@NonNull
private final ShardDetector shardDetector;
@NonNull
private final LeaseRefresher leaseRefresher;
@NonNull
private final InitialPositionInStreamExtended initialPosition;
private final boolean cleanupLeasesUponShardCompletion;
private final boolean ignoreUnexpectedChildShards;
private final long shardSyncTaskIdleTimeMillis;
@NonNull
private final MetricsFactory metricsFactory;
private final TaskType taskType = TaskType.SHARDSYNC;
/*
* (non-Javadoc)
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#call()
*/
@Override
public TaskResult call() {
Exception exception = null;
final MetricsScope scope = MetricsUtil.createMetricsWithOperation(metricsFactory, SHARD_SYNC_TASK_OPERATION);
try {
ShardSyncer.checkAndCreateLeasesForNewShards(shardDetector, leaseRefresher, initialPosition,
cleanupLeasesUponShardCompletion, ignoreUnexpectedChildShards, scope);
if (shardSyncTaskIdleTimeMillis > 0) {
Thread.sleep(shardSyncTaskIdleTimeMillis);
}
} catch (Exception e) {
log.error("Caught exception while sync'ing Kinesis shards and leases", e);
exception = e;
} finally {
MetricsUtil.endScope(scope);
}
return new TaskResult(exception);
}
/*
* (non-Javadoc)
* @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.ConsumerTask#taskType()
*/
@Override
public TaskType taskType() {
return taskType;
}
}

101
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardSyncTaskManager.java Normal file
View file

@ -0,0 +1,101 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import lombok.Data;
import lombok.NonNull;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;
import software.amazon.kinesis.lifecycle.ConsumerTask;
import software.amazon.kinesis.lifecycle.TaskResult;
import software.amazon.kinesis.metrics.MetricsFactory;
import software.amazon.kinesis.metrics.MetricsCollectingTaskDecorator;
/**
* The ShardSyncTaskManager is used to track the task to sync shards with leases (create leases for new
* Kinesis shards, remove obsolete leases). We'll have at most one outstanding sync task at any time.
* Worker will use this class to kick off a sync task when it finds shards which have been completely processed.
*/
@Data
@Accessors(fluent = true)
@Slf4j
public class ShardSyncTaskManager {
@NonNull
private final ShardDetector shardDetector;
@NonNull
private final LeaseRefresher leaseRefresher;
@NonNull
private final InitialPositionInStreamExtended initialPositionInStream;
private final boolean cleanupLeasesUponShardCompletion;
private final boolean ignoreUnexpectedChildShards;
private final long shardSyncIdleTimeMillis;
@NonNull
private final ExecutorService executorService;
@NonNull
private final MetricsFactory metricsFactory;
private ConsumerTask currentTask;
private Future<TaskResult> future;
public synchronized boolean syncShardAndLeaseInfo() {
return checkAndSubmitNextTask();
}
private synchronized boolean checkAndSubmitNextTask() {
boolean submittedNewTask = false;
if ((future == null) || future.isCancelled() || future.isDone()) {
if ((future != null) && future.isDone()) {
try {
TaskResult result = future.get();
if (result.getException() != null) {
log.error("Caught exception running {} task: ", currentTask.taskType(),
result.getException());
}
} catch (InterruptedException | ExecutionException e) {
log.warn("{} task encountered exception.", currentTask.taskType(), e);
}
}
currentTask =
new MetricsCollectingTaskDecorator(
new ShardSyncTask(shardDetector,
leaseRefresher,
initialPositionInStream,
cleanupLeasesUponShardCompletion,
ignoreUnexpectedChildShards,
shardSyncIdleTimeMillis,
metricsFactory),
metricsFactory);
future = executorService.submit(currentTask);
submittedNewTask = true;
if (log.isDebugEnabled()) {
log.debug("Submitted new {} task.", currentTask.taskType());
}
} else {
if (log.isDebugEnabled()) {
log.debug("Previous {} task still pending. Not submitting new task.", currentTask.taskType());
}
}
return submittedNewTask;
}
}

754
amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/ShardSyncer.java Normal file
View file

@ -0,0 +1,754 @@
/*
* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package software.amazon.kinesis.leases;
import java.io.Serializable;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Comparator;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.function.Function;
import java.util.stream.Collectors;
import org.apache.commons.lang.StringUtils;
import lombok.AccessLevel;
import lombok.NoArgsConstructor;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import software.amazon.awssdk.services.kinesis.model.Shard;
import software.amazon.awssdk.utils.CollectionUtils;
import software.amazon.kinesis.common.InitialPositionInStream;
import software.amazon.kinesis.common.InitialPositionInStreamExtended;
import software.amazon.kinesis.exceptions.internal.KinesisClientLibIOException;
import software.amazon.kinesis.leases.exceptions.DependencyException;
import software.amazon.kinesis.leases.exceptions.InvalidStateException;
import software.amazon.kinesis.leases.exceptions.ProvisionedThroughputException;
import software.amazon.kinesis.metrics.MetricsScope;
import software.amazon.kinesis.metrics.MetricsLevel;
import software.amazon.kinesis.metrics.MetricsUtil;
import software.amazon.kinesis.retrieval.kpl.ExtendedSequenceNumber;
/**
* Helper class to sync leases with shards of the Kinesis stream.
* It will create new leases/activities when it discovers new Kinesis shards (bootstrap/resharding).
* It deletes leases for shards that have been trimmed from Kinesis, or if we've completed processing it
* and begun processing it's child shards.
*/
@NoArgsConstructor(access = AccessLevel.PRIVATE)
@Slf4j
public class ShardSyncer {
/**
* Check and create leases for any new shards (e.g. following a reshard operation). Sync leases with Kinesis shards
* (e.g. at startup, or when we reach end of a shard).
*
* @param leaseRefresher
* @param initialPosition
* @param cleanupLeasesOfCompletedShards
* @param ignoreUnexpectedChildShards
* @throws DependencyException
* @throws InvalidStateException
* @throws ProvisionedThroughputException
* @throws KinesisClientLibIOException
*/
// CHECKSTYLE:OFF CyclomaticComplexity
public static synchronized void checkAndCreateLeasesForNewShards(@NonNull final ShardDetector shardDetector,
final LeaseRefresher leaseRefresher, final InitialPositionInStreamExtended initialPosition,
final boolean cleanupLeasesOfCompletedShards, final boolean ignoreUnexpectedChildShards,
final MetricsScope scope) throws DependencyException, InvalidStateException,
ProvisionedThroughputException, KinesisClientLibIOException {
final List<Shard> shards = getShardList(shardDetector);
log.debug("Num shards: {}", shards.size());
final Map<String, Shard> shardIdToShardMap = constructShardIdToShardMap(shards);
final Map<String, Set<String>> shardIdToChildShardIdsMap = constructShardIdToChildShardIdsMap(
shardIdToShardMap);
final Set<String> inconsistentShardIds = findInconsistentShardIds(shardIdToChildShardIdsMap, shardIdToShardMap);
if (!ignoreUnexpectedChildShards) {
assertAllParentShardsAreClosed(inconsistentShardIds);
}
final List<Lease> currentLeases = leaseRefresher.listLeases();
final List<Lease> newLeasesToCreate = determineNewLeasesToCreate(shards, currentLeases, initialPosition,
inconsistentShardIds);
log.debug("Num new leases to create: {}", newLeasesToCreate.size());
for (Lease lease : newLeasesToCreate) {
long startTime = System.currentTimeMillis();
boolean success = false;
try {
leaseRefresher.createLeaseIfNotExists(lease);
success = true;
} finally {
MetricsUtil.addSuccessAndLatency(scope, "CreateLease", success, startTime, MetricsLevel.DETAILED);
}
}
final List<Lease> trackedLeases = new ArrayList<>(currentLeases);
trackedLeases.addAll(newLeasesToCreate);
cleanupGarbageLeases(shardDetector, shards, trackedLeases, leaseRefresher);
if (cleanupLeasesOfCompletedShards) {
cleanupLeasesOfFinishedShards(currentLeases, shardIdToShardMap, shardIdToChildShardIdsMap, trackedLeases,
leaseRefresher);
}
}
// CHECKSTYLE:ON CyclomaticComplexity
/** Helper method to detect a race condition between fetching the shards via paginated DescribeStream calls
* and a reshard operation.
* @param inconsistentShardIds
* @throws KinesisClientLibIOException
*/
private static void assertAllParentShardsAreClosed(final Set<String> inconsistentShardIds)
throws KinesisClientLibIOException {
if (!CollectionUtils.isNullOrEmpty(inconsistentShardIds)) {
final String ids = StringUtils.join(inconsistentShardIds, ' ');
throw new KinesisClientLibIOException(String.format(
"%d open child shards (%s) are inconsistent. This can happen due to a race condition between describeStream and a reshard operation.",
inconsistentShardIds.size(), ids));
}
}
/**
* Helper method to construct the list of inconsistent shards, which are open shards with non-closed ancestor
* parent(s).
* @param shardIdToChildShardIdsMap
* @param shardIdToShardMap
* @return Set of inconsistent open shard ids for shards having open parents.
*/
private static Set<String> findInconsistentShardIds(final Map<String, Set<String>> shardIdToChildShardIdsMap,
final Map<String, Shard> shardIdToShardMap) {
return shardIdToChildShardIdsMap.entrySet().stream()
.filter(entry -> entry.getKey() == null
|| shardIdToShardMap.get(entry.getKey()).sequenceNumberRange().endingSequenceNumber() == null)
.flatMap(entry -> shardIdToChildShardIdsMap.get(entry.getKey()).stream()).collect(Collectors.toSet());
}
/**
* Note: this has package level access for testing purposes.
* Useful for asserting that we don't have an incomplete shard list following a reshard operation.
* We verify that if the shard is present in the shard list, it is closed and its hash key range
* is covered by its child shards.
* @param shardIdsOfClosedShards Id of the shard which is expected to be closed
* @return ShardIds of child shards (children of the expectedClosedShard)
* @throws KinesisClientLibIOException
*/
static synchronized void assertClosedShardsAreCoveredOrAbsent(final Map<String, Shard> shardIdToShardMap,
final Map<String, Set<String>> shardIdToChildShardIdsMap, final Set<String> shardIdsOfClosedShards)
throws KinesisClientLibIOException {
final String exceptionMessageSuffix = "This can happen if we constructed the list of shards "
+ " while a reshard operation was in progress.";
for (String shardId : shardIdsOfClosedShards) {
final Shard shard = shardIdToShardMap.get(shardId);
if (shard == null) {
log.info("Shard {} is not present in Kinesis anymore.", shardId);
continue;
}
final String endingSequenceNumber = shard.sequenceNumberRange().endingSequenceNumber();
if (endingSequenceNumber == null) {
throw new KinesisClientLibIOException("Shard " + shardIdsOfClosedShards
+ " is not closed. " + exceptionMessageSuffix);
}
final Set<String> childShardIds = shardIdToChildShardIdsMap.get(shardId);
if (childShardIds == null) {
throw new KinesisClientLibIOException("Incomplete shard list: Closed shard " + shardId
+ " has no children." + exceptionMessageSuffix);
}
assertHashRangeOfClosedShardIsCovered(shard, shardIdToShardMap, childShardIds);
}
}
private static synchronized void assertHashRangeOfClosedShardIsCovered(final Shard closedShard,
final Map<String, Shard> shardIdToShardMap, final Set<String> childShardIds)
throws KinesisClientLibIOException {
BigInteger minStartingHashKeyOfChildren = null;
BigInteger maxEndingHashKeyOfChildren = null;
final BigInteger startingHashKeyOfClosedShard = new BigInteger(closedShard.hashKeyRange().startingHashKey());
final BigInteger endingHashKeyOfClosedShard = new BigInteger(closedShard.hashKeyRange().endingHashKey());
for (String childShardId : childShardIds) {
final Shard childShard = shardIdToShardMap.get(childShardId);
final BigInteger startingHashKey = new BigInteger(childShard.hashKeyRange().startingHashKey());
if (minStartingHashKeyOfChildren == null || startingHashKey.compareTo(minStartingHashKeyOfChildren) < 0) {
minStartingHashKeyOfChildren = startingHashKey;
}
final BigInteger endingHashKey = new BigInteger(childShard.hashKeyRange().endingHashKey());
if (maxEndingHashKeyOfChildren == null || endingHashKey.compareTo(maxEndingHashKeyOfChildren) > 0) {
maxEndingHashKeyOfChildren = endingHashKey;
}
}
if (minStartingHashKeyOfChildren == null || maxEndingHashKeyOfChildren == null
|| minStartingHashKeyOfChildren.compareTo(startingHashKeyOfClosedShard) > 0
|| maxEndingHashKeyOfChildren.compareTo(endingHashKeyOfClosedShard) < 0) {
throw new KinesisClientLibIOException(String.format(
"Incomplete shard list: hash key range of shard %s is not covered by its child shards.",
closedShard.shardId()));
}
}
/**
* Helper method to construct shardId->setOfChildShardIds map.
* Note: This has package access for testing purposes only.
* @param shardIdToShardMap
* @return
*/
static Map<String, Set<String>> constructShardIdToChildShardIdsMap(final Map<String, Shard> shardIdToShardMap) {
final Map<String, Set<String>> shardIdToChildShardIdsMap = new HashMap<>();
for (final Map.Entry<String, Shard> entry : shardIdToShardMap.entrySet()) {
final String shardId = entry.getKey();
final Shard shard = entry.getValue();
final String parentShardId = shard.parentShardId();
if (parentShardId != null && shardIdToShardMap.containsKey(parentShardId)) {
final Set<String> childShardIds = shardIdToChildShardIdsMap.computeIfAbsent(parentShardId,
key -> new HashSet<>());
childShardIds.add(shardId);
}
final String adjacentParentShardId = shard.adjacentParentShardId();
if (adjacentParentShardId != null && shardIdToShardMap.containsKey(adjacentParentShardId)) {
final Set<String> childShardIds = shardIdToChildShardIdsMap.computeIfAbsent(adjacentParentShardId,
key -> new HashSet<>());
childShardIds.add(shardId);
}
}
return shardIdToChildShardIdsMap;
}
private static List<Shard> getShardList(@NonNull final ShardDetector shardDetector) throws KinesisClientLibIOException {
final List<Shard> shards = shardDetector.listShards();
if (shards == null) {
throw new KinesisClientLibIOException(
"Stream is not in ACTIVE OR UPDATING state - will retry getting the shard list.");
}
return shards;
}
/**
* Determine new leases to create and their initial checkpoint.
* Note: Package level access only for testing purposes.
*
* For each open (no ending sequence number) shard without open parents that doesn't already have a lease,
* determine if it is a descendent of any shard which is or will be processed (e.g. for which a lease exists):
* If so, set checkpoint of the shard to TrimHorizon and also create leases for ancestors if needed.
* If not, set checkpoint of the shard to the initial position specified by the client.
* To check if we need to create leases for ancestors, we use the following rules:
* * If we began (or will begin) processing data for a shard, then we must reach end of that shard before
* we begin processing data from any of its descendants.
* * A shard does not start processing data until data from all its parents has been processed.
* Note, if the initial position is LATEST and a shard has two parents and only one is a descendant - we'll create
* leases corresponding to both the parents - the parent shard which is not a descendant will have
* its checkpoint set to Latest.
*
* We assume that if there is an existing lease for a shard, then either:
* * we have previously created a lease for its parent (if it was needed), or
* * the parent shard has expired.
*
* For example:
* Shard structure (each level depicts a stream segment):
* 0 1 2 3 4 5 - shards till epoch 102
* \ / \ / | |
* 6 7 4 5 - shards from epoch 103 - 205
* \ / | / \
* 8 4 9 10 - shards from epoch 206 (open - no ending sequenceNumber)
* Current leases: (3, 4, 5)
* New leases to create: (2, 6, 7, 8, 9, 10)
*
* The leases returned are sorted by the starting sequence number - following the same order
* when persisting the leases in DynamoDB will ensure that we recover gracefully if we fail
* before creating all the leases.
*
* If a shard has no existing lease, is open, and is a descendant of a parent which is still open, we ignore it
* here; this happens when the list of shards is inconsistent, which could be due to pagination delay for very
* high shard count streams (i.e., dynamodb streams for tables with thousands of partitions). This can only
* currently happen here if ignoreUnexpectedChildShards was true in syncShardleases.
*
*
* @param shards List of all shards in Kinesis (we'll create new leases based on this set)
* @param currentLeases List of current leases
* @param initialPosition One of LATEST, TRIM_HORIZON, or AT_TIMESTAMP. We'll start fetching records from that
* location in the shard (when an application starts up for the first time - and there are no checkpoints).
* @param inconsistentShardIds Set of child shard ids having open parents.
* @return List of new leases to create sorted by starting sequenceNumber of the corresponding shard
*/
static List<Lease> determineNewLeasesToCreate(final List<Shard> shards, final List<Lease> currentLeases,
final InitialPositionInStreamExtended initialPosition, final Set<String> inconsistentShardIds) {
final Map<String, Lease> shardIdToNewLeaseMap = new HashMap<>();
final Map<String, Shard> shardIdToShardMapOfAllKinesisShards = constructShardIdToShardMap(shards);
final Set<String> shardIdsOfCurrentLeases = currentLeases.stream()
.peek(lease -> log.debug("Existing lease: {}", lease)).map(Lease::leaseKey).collect(Collectors.toSet());
final List<Shard> openShards = getOpenShards(shards);
final Map<String, Boolean> memoizationContext = new HashMap<>();
// Iterate over the open shards and find those that don't have any lease entries.
for (Shard shard : openShards) {
final String shardId = shard.shardId();
log.debug("Evaluating leases for open shard {} and its ancestors.", shardId);
if (shardIdsOfCurrentLeases.contains(shardId)) {
log.debug("Lease for shardId {} already exists. Not creating a lease", shardId);
} else if (inconsistentShardIds.contains(shardId)) {
log.info("shardId {} is an inconsistent child. Not creating a lease", shardId);
} else {
log.debug("Need to create a lease for shardId {}", shardId);
final Lease newLease = newKCLLease(shard);
final boolean isDescendant = checkIfDescendantAndAddNewLeasesForAncestors(shardId, initialPosition,
shardIdsOfCurrentLeases, shardIdToShardMapOfAllKinesisShards, shardIdToNewLeaseMap,
memoizationContext);
/**
* If the shard is a descendant and the specified initial position is AT_TIMESTAMP, then the
* checkpoint should be set to AT_TIMESTAMP, else to TRIM_HORIZON. For AT_TIMESTAMP, we will add a
* lease just like we do for TRIM_HORIZON. However we will only return back records with server-side
* timestamp at or after the specified initial position timestamp.
*
* Shard structure (each level depicts a stream segment):
* 0 1 2 3 4 5 - shards till epoch 102
* \ / \ / | |
* 6 7 4 5 - shards from epoch 103 - 205
* \ / | /\
* 8 4 9 10 - shards from epoch 206 (open - no ending sequenceNumber)
*
* Current leases: empty set
*
* For the above example, suppose the initial position in stream is set to AT_TIMESTAMP with
* timestamp value 206. We will then create new leases for all the shards (with checkpoint set to
* AT_TIMESTAMP), including the ancestor shards with epoch less than 206. However as we begin
* processing the ancestor shards, their checkpoints would be updated to SHARD_END and their leases
* would then be deleted since they won't have records with server-side timestamp at/after 206. And
* after that we will begin processing the descendant shards with epoch at/after 206 and we will
* return the records that meet the timestamp requirement for these shards.
*/
if (isDescendant
&& !initialPosition.getInitialPositionInStream().equals(InitialPositionInStream.AT_TIMESTAMP)) {
newLease.checkpoint(ExtendedSequenceNumber.TRIM_HORIZON);
} else {
newLease.checkpoint(convertToCheckpoint(initialPosition));
}
log.debug("Set checkpoint of {} to {}", newLease.leaseKey(), newLease.checkpoint());
shardIdToNewLeaseMap.put(shardId, newLease);
}
}
final List<Lease> newLeasesToCreate = new ArrayList<>(shardIdToNewLeaseMap.values());
final Comparator<Lease> startingSequenceNumberComparator = new StartingSequenceNumberAndShardIdBasedComparator(
shardIdToShardMapOfAllKinesisShards);
newLeasesToCreate.sort(startingSequenceNumberComparator);
return newLeasesToCreate;
}
/**
* Determine new leases to create and their initial checkpoint.
* Note: Package level access only for testing purposes.
*/
static List<Lease> determineNewLeasesToCreate(final List<Shard> shards, final List<Lease> currentLeases,
final InitialPositionInStreamExtended initialPosition) {
final Set<String> inconsistentShardIds = new HashSet<>();
return determineNewLeasesToCreate(shards, currentLeases, initialPosition, inconsistentShardIds);
}
/**
* Note: Package level access for testing purposes only.
* Check if this shard is a descendant of a shard that is (or will be) processed.
* Create leases for the ancestors of this shard as required.
* See javadoc of determineNewLeasesToCreate() for rules and example.
*
* @param shardId The shardId to check.
* @param initialPosition One of LATEST, TRIM_HORIZON, or AT_TIMESTAMP. We'll start fetching records from that
* location in the shard (when an application starts up for the first time - and there are no checkpoints).
* @param shardIdsOfCurrentLeases The shardIds for the current leases.
* @param shardIdToShardMapOfAllKinesisShards ShardId->Shard map containing all shards obtained via DescribeStream.
* @param shardIdToLeaseMapOfNewShards Add lease POJOs corresponding to ancestors to this map.
* @param memoizationContext Memoization of shards that have been evaluated as part of the evaluation
* @return true if the shard is a descendant of any current shard (lease already exists)
*/
// CHECKSTYLE:OFF CyclomaticComplexity
static boolean checkIfDescendantAndAddNewLeasesForAncestors(final String shardId,
final InitialPositionInStreamExtended initialPosition, final Set<String> shardIdsOfCurrentLeases,
final Map<String, Shard> shardIdToShardMapOfAllKinesisShards,
final Map<String, Lease> shardIdToLeaseMapOfNewShards, final Map<String, Boolean> memoizationContext) {
final Boolean previousValue = memoizationContext.get(shardId);
if (previousValue != null) {
return previousValue;
}
boolean isDescendant = false;
final Set<String> descendantParentShardIds = new HashSet<>();
if (shardId != null && shardIdToShardMapOfAllKinesisShards.containsKey(shardId)) {
if (shardIdsOfCurrentLeases.contains(shardId)) {
// This shard is a descendant of a current shard.
isDescendant = true;
// We don't need to add leases of its ancestors,
// because we'd have done it when creating a lease for this shard.
} else {
final Shard shard = shardIdToShardMapOfAllKinesisShards.get(shardId);
final Set<String> parentShardIds = getParentShardIds(shard, shardIdToShardMapOfAllKinesisShards);
for (String parentShardId : parentShardIds) {
// Check if the parent is a descendant, and include its ancestors.
if (checkIfDescendantAndAddNewLeasesForAncestors(parentShardId, initialPosition,
shardIdsOfCurrentLeases, shardIdToShardMapOfAllKinesisShards, shardIdToLeaseMapOfNewShards,
memoizationContext)) {
isDescendant = true;
descendantParentShardIds.add(parentShardId);
log.debug("Parent shard {} is a descendant.", parentShardId);
} else {
log.debug("Parent shard {} is NOT a descendant.", parentShardId);
}
}
// If this is a descendant, create leases for its parent shards (if they don't exist)
if (isDescendant) {
for (String parentShardId : parentShardIds) {
if (!shardIdsOfCurrentLeases.contains(parentShardId)) {
log.debug("Need to create a lease for shardId {}", parentShardId);
Lease lease = shardIdToLeaseMapOfNewShards.get(parentShardId);
if (lease == null) {
lease = newKCLLease(shardIdToShardMapOfAllKinesisShards.get(parentShardId));
shardIdToLeaseMapOfNewShards.put(parentShardId, lease);
}
if (descendantParentShardIds.contains(parentShardId)
&& !initialPosition.getInitialPositionInStream()
.equals(InitialPositionInStream.AT_TIMESTAMP)) {
lease.checkpoint(ExtendedSequenceNumber.TRIM_HORIZON);
} else {
lease.checkpoint(convertToCheckpoint(initialPosition));
}
}
}
} else {
// This shard should be included, if the customer wants to process all records in the stream or
// if the initial position is AT_TIMESTAMP. For AT_TIMESTAMP, we will add a lease just like we do
// for TRIM_HORIZON. However we will only return back records with server-side timestamp at or
// after the specified initial position timestamp.
if (initialPosition.getInitialPositionInStream().equals(InitialPositionInStream.TRIM_HORIZON)
|| initialPosition.getInitialPositionInStream()
.equals(InitialPositionInStream.AT_TIMESTAMP)) {
isDescendant = true;
}
}
}
}
memoizationContext.put(shardId, isDescendant);
return isDescendant;
}
// CHECKSTYLE:ON CyclomaticComplexity
/**
* Helper method to get parent shardIds of the current shard - includes the parent shardIds if:
* a/ they are not null
* b/ if they exist in the current shard map (i.e. haven't expired)
*
* @param shard Will return parents of this shard
* @param shardIdToShardMapOfAllKinesisShards ShardId->Shard map containing all shards obtained via DescribeStream.
* @return Set of parentShardIds
*/
static Set<String> getParentShardIds(final Shard shard,
final Map<String, Shard> shardIdToShardMapOfAllKinesisShards) {
final Set<String> parentShardIds = new HashSet<>(2);
final String parentShardId = shard.parentShardId();
if (parentShardId != null && shardIdToShardMapOfAllKinesisShards.containsKey(parentShardId)) {
parentShardIds.add(parentShardId);
}
final String adjacentParentShardId = shard.adjacentParentShardId();
if (adjacentParentShardId != null && shardIdToShardMapOfAllKinesisShards.containsKey(adjacentParentShardId)) {
parentShardIds.add(adjacentParentShardId);
}
return parentShardIds;
}
/**
* Delete leases corresponding to shards that no longer exist in the stream. Current scheme: Delete a lease if:
* <ul>
* <li>The corresponding shard is not present in the list of Kinesis shards</li>
* <li>The parentShardIds listed in the lease are also not present in the list of Kinesis shards.</li>
* </ul>
*
* @param shards
* List of all Kinesis shards (assumed to be a consistent snapshot - when stream is in Active state).
* @param trackedLeases
* List of
* @param leaseRefresher
* @throws KinesisClientLibIOException
* Thrown if we couldn't get a fresh shard list from Kinesis.
* @throws ProvisionedThroughputException
* @throws InvalidStateException
* @throws DependencyException
*/
private static void cleanupGarbageLeases(@NonNull final ShardDetector shardDetector, final List<Shard> shards,
final List<Lease> trackedLeases, final LeaseRefresher leaseRefresher) throws KinesisClientLibIOException,
DependencyException, InvalidStateException, ProvisionedThroughputException {
final Set<String> kinesisShards = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
// Check if there are leases for non-existent shards
final List<Lease> garbageLeases = trackedLeases.stream()
.filter(lease -> isCandidateForCleanup(lease, kinesisShards)).collect(Collectors.toList());
if (!CollectionUtils.isNullOrEmpty(garbageLeases)) {
log.info("Found {} candidate leases for cleanup. Refreshing list of"
+ " Kinesis shards to pick up recent/latest shards", garbageLeases.size());
final Set<String> currentKinesisShardIds = getShardList(shardDetector).stream().map(Shard::shardId)
.collect(Collectors.toSet());
for (Lease lease : garbageLeases) {
if (isCandidateForCleanup(lease, currentKinesisShardIds)) {
log.info("Deleting lease for shard {} as it is not present in Kinesis stream.", lease.leaseKey());
leaseRefresher.deleteLease(lease);
}
}
}
}
/**
* Note: This method has package level access, solely for testing purposes.
*
* @param lease Candidate shard we are considering for deletion.
* @param currentKinesisShardIds
* @return true if neither the shard (corresponding to the lease), nor its parents are present in
* currentKinesisShardIds
* @throws KinesisClientLibIOException Thrown if currentKinesisShardIds contains a parent shard but not the child
* shard (we are evaluating for deletion).
*/
static boolean isCandidateForCleanup(final Lease lease, final Set<String> currentKinesisShardIds)
throws KinesisClientLibIOException {
boolean isCandidateForCleanup = true;
if (currentKinesisShardIds.contains(lease.leaseKey())) {
isCandidateForCleanup = false;
} else {
log.info("Found lease for non-existent shard: {}. Checking its parent shards", lease.leaseKey());
final Set<String> parentShardIds = lease.parentShardIds();
for (String parentShardId : parentShardIds) {
// Throw an exception if the parent shard exists (but the child does not).
// This may be a (rare) race condition between fetching the shard list and Kinesis expiring shards.
if (currentKinesisShardIds.contains(parentShardId)) {
final String message = String.format("Parent shard %s exists but not the child shard %s",
parentShardId, lease.leaseKey());
log.info(message);
throw new KinesisClientLibIOException(message);
}
}
}
return isCandidateForCleanup;
}
/**
* Private helper method.
* Clean up leases for shards that meet the following criteria:
* a/ the shard has been fully processed (checkpoint is set to SHARD_END)
* b/ we've begun processing all the child shards: we have leases for all child shards and their checkpoint is not
* TRIM_HORIZON.
*
* @param currentLeases List of leases we evaluate for clean up
* @param shardIdToShardMap Map of shardId->Shard (assumed to include all Kinesis shards)
* @param shardIdToChildShardIdsMap Map of shardId->childShardIds (assumed to include all Kinesis shards)
* @param trackedLeases List of all leases we are tracking.
* @param leaseRefresher Lease refresher (will be used to delete leases)
* @throws DependencyException
* @throws InvalidStateException
* @throws ProvisionedThroughputException
* @throws KinesisClientLibIOException
*/
private static synchronized void cleanupLeasesOfFinishedShards(final Collection<Lease> currentLeases,
final Map<String, Shard> shardIdToShardMap, final Map<String, Set<String>> shardIdToChildShardIdsMap,
final List<Lease> trackedLeases, final LeaseRefresher leaseRefresher) throws DependencyException,
InvalidStateException, ProvisionedThroughputException, KinesisClientLibIOException {
final List<Lease> leasesOfClosedShards = currentLeases.stream()
.filter(lease -> lease.checkpoint().equals(ExtendedSequenceNumber.SHARD_END))
.collect(Collectors.toList());
final Set<String> shardIdsOfClosedShards = leasesOfClosedShards.stream().map(Lease::leaseKey)
.collect(Collectors.toSet());
if (!CollectionUtils.isNullOrEmpty(leasesOfClosedShards)) {
assertClosedShardsAreCoveredOrAbsent(shardIdToShardMap, shardIdToChildShardIdsMap, shardIdsOfClosedShards);
Comparator<? super Lease> startingSequenceNumberComparator = new StartingSequenceNumberAndShardIdBasedComparator(
shardIdToShardMap);
leasesOfClosedShards.sort(startingSequenceNumberComparator);
final Map<String, Lease> trackedLeaseMap = trackedLeases.stream()
.collect(Collectors.toMap(Lease::leaseKey, Function.identity()));
for (Lease leaseOfClosedShard : leasesOfClosedShards) {
final String closedShardId = leaseOfClosedShard.leaseKey();
final Set<String> childShardIds = shardIdToChildShardIdsMap.get(closedShardId);
if (closedShardId != null && !CollectionUtils.isNullOrEmpty(childShardIds)) {
cleanupLeaseForClosedShard(closedShardId, childShardIds, trackedLeaseMap, leaseRefresher);
}
}
}
}
/**
* Delete lease for the closed shard. Rules for deletion are:
* a/ the checkpoint for the closed shard is SHARD_END,
* b/ there are leases for all the childShardIds and their checkpoint is NOT TRIM_HORIZON
* Note: This method has package level access solely for testing purposes.
*
* @param closedShardId Identifies the closed shard
* @param childShardIds ShardIds of children of the closed shard
* @param trackedLeases shardId->Lease map with all leases we are tracking (should not be null)
* @param leaseRefresher
* @throws ProvisionedThroughputException
* @throws InvalidStateException
* @throws DependencyException
*/
static synchronized void cleanupLeaseForClosedShard(final String closedShardId, final Set<String> childShardIds,
final Map<String, Lease> trackedLeases, final LeaseRefresher leaseRefresher)
throws DependencyException, InvalidStateException, ProvisionedThroughputException {
final Lease leaseForClosedShard = trackedLeases.get(closedShardId);
final List<Lease> childShardLeases = childShardIds.stream().map(trackedLeases::get).filter(Objects::nonNull)
.collect(Collectors.toList());
if (leaseForClosedShard != null && leaseForClosedShard.checkpoint().equals(ExtendedSequenceNumber.SHARD_END)
&& childShardLeases.size() == childShardIds.size()) {
boolean okayToDelete = true;
for (Lease lease : childShardLeases) {
if (lease.checkpoint().equals(ExtendedSequenceNumber.TRIM_HORIZON)) {
okayToDelete = false;
break;
}
}
if (okayToDelete) {
log.info("Deleting lease for shard {} as it has been completely processed and processing of child "
+ "shards has begun.", leaseForClosedShard.leaseKey());
leaseRefresher.deleteLease(leaseForClosedShard);
}
}
}
/**
* Helper method to create a new Lease POJO for a shard.
* Note: Package level access only for testing purposes
*
* @param shard
* @return
*/
private static Lease newKCLLease(final Shard shard) {
Lease newLease = new Lease();
newLease.leaseKey(shard.shardId());
List<String> parentShardIds = new ArrayList<>(2);
if (shard.parentShardId() != null) {
parentShardIds.add(shard.parentShardId());
}
if (shard.adjacentParentShardId() != null) {
parentShardIds.add(shard.adjacentParentShardId());
}
newLease.parentShardIds(parentShardIds);
newLease.ownerSwitchesSinceCheckpoint(0L);
return newLease;
}
/**
* Helper method to construct a shardId->Shard map for the specified list of shards.
*
* @param shards List of shards
* @return ShardId->Shard map
*/
static Map<String, Shard> constructShardIdToShardMap(final List<Shard> shards) {
return shards.stream().collect(Collectors.toMap(Shard::shardId, Function.identity()));
}
/**
* Helper method to return all the open shards for a stream.
* Note: Package level access only for testing purposes.
*
* @param allShards All shards returved via DescribeStream. We assume this to represent a consistent shard list.
* @return List of open shards (shards at the tip of the stream) - may include shards that are not yet active.
*/
static List<Shard> getOpenShards(final List<Shard> allShards) {
return allShards.stream().filter(shard -> shard.sequenceNumberRange().endingSequenceNumber() == null)
.peek(shard -> log.debug("Found open shard: {}", shard.shardId())).collect(Collectors.toList());
}
private static ExtendedSequenceNumber convertToCheckpoint(final InitialPositionInStreamExtended position) {
ExtendedSequenceNumber checkpoint = null;
if (position.getInitialPositionInStream().equals(InitialPositionInStream.TRIM_HORIZON)) {
checkpoint = ExtendedSequenceNumber.TRIM_HORIZON;
} else if (position.getInitialPositionInStream().equals(InitialPositionInStream.LATEST)) {
checkpoint = ExtendedSequenceNumber.LATEST;
} else if (position.getInitialPositionInStream().equals(InitialPositionInStream.AT_TIMESTAMP)) {
checkpoint = ExtendedSequenceNumber.AT_TIMESTAMP;
}
return checkpoint;
}
/** Helper class to compare leases based on starting sequence number of the corresponding shards.
*
*/
@RequiredArgsConstructor
private static class StartingSequenceNumberAndShardIdBasedComparator implements Comparator<Lease>, Serializable {
private static final long serialVersionUID = 1L;
private final Map<String, Shard> shardIdToShardMap;
/**
* Compares two leases based on the starting sequence number of corresponding shards.
* If shards are not found in the shardId->shard map supplied, we do a string comparison on the shardIds.
* We assume that lease1 and lease2 are:
* a/ not null,
* b/ shards (if found) have non-null starting sequence numbers
*
* {@inheritDoc}
*/
@Override
public int compare(final Lease lease1, final Lease lease2) {
int result = 0;
final String shardId1 = lease1.leaseKey();
final String shardId2 = lease2.leaseKey();
final Shard shard1 = shardIdToShardMap.get(shardId1);
final Shard shard2 = shardIdToShardMap.get(shardId2);
// If we found shards for the two leases, use comparison of the starting sequence numbers
if (shard1 != null && shard2 != null) {
BigInteger sequenceNumber1 = new BigInteger(shard1.sequenceNumberRange().startingSequenceNumber());
BigInteger sequenceNumber2 = new BigInteger(shard2.sequenceNumberRange().startingSequenceNumber());
result = sequenceNumber1.compareTo(sequenceNumber2);
}
if (result == 0) {
result = shardId1.compareTo(shardId2);
}
return result;
}
}
}

Some files were not shown because too many files have changed in this diff Show more