Merge remote-tracking branch 'upstream/master'

META-INF/MANIFEST.MF

@@ -2,9 +2,9 @@ Manifest-Version: 1.0
 Bundle-ManifestVersion: 2
 Bundle-Name: Amazon Kinesis Client Library for Java
 Bundle-SymbolicName: com.amazonaws.kinesisclientlibrary;singleton:=true
-Bundle-Version: 1.8.1
+Bundle-Version: 1.8.9
 Bundle-Vendor: Amazon Technologies, Inc
-Bundle-RequiredExecutionEnvironment: JavaSE-1.7
+Bundle-RequiredExecutionEnvironment: JavaSE-1.8
 Require-Bundle: org.apache.commons.codec;bundle-version="1.6",
  org.apache.commons.logging;bundle-version="1.1.3";visibility:=reexport,
  com.fasterxml.jackson.core.jackson-databind;bundle-version="2.5.3",

README.md

@@ -15,7 +15,7 @@ The **Amazon Kinesis Client Library for Java** (Amazon KCL) enables Java develop
 
 1. **Sign up for AWS** — Before you begin, you need an AWS account. For more information about creating an AWS account and retrieving your AWS credentials, see [AWS Account and Credentials][docs-signup] in the AWS SDK for Java Developer Guide.
 1. **Sign up for Amazon Kinesis** — Go to the Amazon Kinesis console to sign up for the service and create an Amazon Kinesis stream. For more information, see [Create an Amazon Kinesis Stream][kinesis-guide-create] in the Amazon Kinesis Developer Guide.
-1. **Minimum requirements** — To use the Amazon Kinesis Client Library, you'll need **Java 1.7+**. For more information about Amazon Kinesis Client Library requirements, see [Before You Begin][kinesis-guide-begin] in the Amazon Kinesis Developer Guide.
+1. **Minimum requirements** — To use the Amazon Kinesis Client Library, you'll need **Java 1.8+**. For more information about Amazon Kinesis Client Library requirements, see [Before You Begin][kinesis-guide-begin] in the Amazon Kinesis Developer Guide.
 1. **Using the Amazon Kinesis Client Library** — The best way to get familiar with the Amazon Kinesis Client Library is to read [Developing Record Consumer Applications][kinesis-guide-applications] in the Amazon Kinesis Developer Guide.
 
 ## Building from Source
@@ -29,6 +29,89 @@ For producer-side developers using the **[Kinesis Producer Library (KPL)][kinesi
 To make it easier for developers to write record processors in other languages, we have implemented a Java based daemon, called MultiLangDaemon that does all the heavy lifting. Our approach has the daemon spawn a sub-process, which in turn runs the record processor, which can be written in any language. The MultiLangDaemon process and the record processor sub-process communicate with each other over [STDIN and STDOUT using a defined protocol][multi-lang-protocol]. There will be a one to one correspondence amongst record processors, child processes, and shards. For Python developers specifically, we have abstracted these implementation details away and [expose an interface][kclpy] that enables you to focus on writing record processing logic in Python. This approach enables KCL to be language agnostic, while providing identical features and similar parallel processing model across all languages.
 
 ## Release Notes
+
+### Release 1.8.9
+* Allow disabling check for the case where a child shard has an open parent shard.  
+  There is a race condition where it's possible for the a parent shard to appear open, while having child shards. This check can now be disabled by setting [`ignoreUnexpectedChildShards`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1037) to true.
+  * [PR #240](https://github.com/awslabs/amazon-kinesis-client/pull/240)
+  * [Issue #210](https://github.com/awslabs/amazon-kinesis-client/issues/210)
+* Upgraded the AWS SDK for Java to 1.11.261
+  * [PR #281](https://github.com/awslabs/amazon-kinesis-client/pull/281)
+
+### Release 1.8.8
+* Fixed issues with leases losses due to `ExpiredIteratorException` in `PrefetchGetRecordsCache` and `AsynchronousFetchingStrategy`.  
+  PrefetchGetRecordsCache will request for a new iterator and start fetching data again.  
+  * [PR#263](https://github.com/awslabs/amazon-kinesis-client/pull/263)
+* Added warning message for long running tasks.  
+  Logging long running tasks can be enabled by setting the following configuration property:
+  
+  | Name | Default | Description |
+  | ---- | ------- | ----------- |
+  | [`logWarningForTaskAfterMillis`](https://github.com/awslabs/amazon-kinesis-client/blob/3de901ea9327370ed732af86c4d4999c8d99541c/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1367) | Not set | Milliseconds after which the logger will log a warning message for the long running task |
+  
+  * [PR#259](https://github.com/awslabs/amazon-kinesis-client/pull/259)
+* Handling spurious lease renewal failures gracefully.  
+  Added better handling of DynamoDB failures when updating leases.  These failures would occur when a request to DynamoDB appeared to fail, but was actually successful.  
+  * [PR#247](https://github.com/awslabs/amazon-kinesis-client/pull/247)
+* ShutdownTask gets retried if the previous attempt on the ShutdownTask fails.
+  * [PR#267](https://github.com/awslabs/amazon-kinesis-client/pull/267)
+* Fix for using maxRecords from `KinesisClientLibConfiguration` in `GetRecordsCache` for fetching records.
+  * [PR#264](https://github.com/awslabs/amazon-kinesis-client/pull/264)
+
+### Release 1.8.7
+* Don't add a delay for synchronous requests to Kinesis  
+  Removes a delay that had been added for synchronous `GetRecords` calls to Kinesis. 
+  * [PR #256](https://github.com/awslabs/amazon-kinesis-client/pull/256)
+
+### Release 1.8.6
+* Add prefetching of records from Kinesis  
+  Prefetching will retrieve and queue additional records from Kinesis while the application is processing existing records.  
+  Prefetching can be enabled by setting [`dataFetchingStrategy`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1317) to `PREFETCH_CACHED`. Once enabled an additional fetching thread will be started to retrieve records from Kinesis. Retrieved records will be held in a queue until the application is ready to process them.  
+  Pre-fetching supports the following configuration values:  
+  
+  | Name | Default | Description |
+  | ---- | ------- | ----------- |
+  | [`dataFetchingStrategy`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1317) | `DEFAULT` | Which data fetching strategy to use |
+  | [`maxPendingProcessRecordsInput`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1296) | 3 | The maximum number of process records input that can be queued |
+  | [`maxCacheByteSize`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1307) | 8 MiB | The maximum number of bytes that can be queued |
+  | [`maxRecordsCount`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1326) | 30,000 | The maximum number of records that can be queued |
+  | [`idleMillisBetweenCalls`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L1353) | 1,500 ms | The amount of time to wait between calls to Kinesis |
+  
+  * [PR #246](https://github.com/awslabs/amazon-kinesis-client/pull/246)
+
+### Release 1.8.5 (September 26, 2017)
+* Only advance the shard iterator for the accepted response.  
+  This fixes a race condition in the `KinesisDataFetcher` when it's being used to make asynchronous requests.  The shard iterator is now only advanced when the retriever calls `DataFetcherResult#accept()`.
+  * [PR #230](https://github.com/awslabs/amazon-kinesis-client/pull/230)
+  * [Issue #231](https://github.com/awslabs/amazon-kinesis-client/issues/231)
+
+### Release 1.8.4 (September 22, 2017)
+* Create a new completion service for each request.  
+  This ensures that canceled tasks are discarded.  This will prevent a cancellation exception causing issues processing records.
+  * [PR #227](https://github.com/awslabs/amazon-kinesis-client/pull/227)
+  * [Issue #226](https://github.com/awslabs/amazon-kinesis-client/issues/226)
+
+### Release 1.8.3 (September 22, 2017)
+* Call shutdown on the retriever when the record processor is being shutdown  
+  This fixes a bug that could leak threads if using the [`AsynchronousGetRecordsRetrievalStrategy`](https://github.com/awslabs/amazon-kinesis-client/blob/9a82b6bd05b3c9c5f8581af007141fa6d5f0fc4e/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/AsynchronousGetRecordsRetrievalStrategy.java#L42) is being used.  
+  The asynchronous retriever is only used when [`KinesisClientLibConfiguration#retryGetRecordsInSeconds`](https://github.com/awslabs/amazon-kinesis-client/blob/01d2688bc6e68fd3fe5cb698cb65299d67ac930d/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L227), and [`KinesisClientLibConfiguration#maxGetRecordsThreadPool`](https://github.com/awslabs/amazon-kinesis-client/blob/01d2688bc6e68fd3fe5cb698cb65299d67ac930d/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java#L230) are set.
+  * [PR #222](https://github.com/awslabs/amazon-kinesis-client/pull/222)
+
+### Release 1.8.2 (September 20, 2017)
+* Add support for two phase checkpoints  
+  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.  
+  Should the checkpoint fail the attempted sequence number is provided in the [`InitializationInput#getPendingCheckpointSequenceNumber`](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/types/InitializationInput.java#L81) otherwise the value will be null.
+  * [PR #188](https://github.com/awslabs/amazon-kinesis-client/pull/188)
+* Support timeouts, and retry for GetRecords calls.  
+  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
+  * [PR #214](https://github.com/awslabs/amazon-kinesis-client/pull/214)
+* Notification when the lease table is throttled  
+  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
+  * [PR #212](https://github.com/awslabs/amazon-kinesis-client/pull/212)
+* Support configuring the graceful shutdown timeout for MultiLang Clients  
+  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding `shutdownGraceMillis` to the properties file set to the number of milliseconds to wait.
+  * [PR #204](https://github.com/awslabs/amazon-kinesis-client/pull/204)
+
 ### Release 1.8.1 (August 2, 2017)
 * Support timeouts for calls to the MultiLang Daemon
   This adds support for setting a timeout when dispatching records to the client record processor. If the record processor doesn't respond within the timeout the parent Java process will be terminated. This is a temporary fix to handle cases where the KCL becomes blocked while waiting for a client record processor.

pom.xml

@@ -6,7 +6,7 @@
   <artifactId>amazon-kinesis-client</artifactId>
   <packaging>jar</packaging>
   <name>Amazon Kinesis Client Library for Java</name>
-  <version>1.8.1</version>
+  <version>1.8.10-SNAPSHOT</version>
   <description>The Amazon Kinesis Client Library for Java enables Java developers to easily consume and process data
     from Amazon Kinesis.
   </description>
@@ -25,7 +25,7 @@
   </licenses>
 
   <properties>
-    <aws-java-sdk.version>1.11.171</aws-java-sdk.version>
+    <aws-java-sdk.version>1.11.261</aws-java-sdk.version>
     <sqlite4java.version>1.0.392</sqlite4java.version>
     <sqlite4java.native>libsqlite4java</sqlite4java.native>
     <sqlite4java.libpath>${project.build.directory}/test-lib</sqlite4java.libpath>

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/AsynchronousGetRecordsRetrievalStrategy.java

@@ -0,0 +1,154 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletionService;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
+import com.amazonaws.services.kinesis.metrics.impl.ThreadSafeMetricsDelegatingScope;
+import com.amazonaws.services.kinesis.model.ExpiredIteratorException;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import lombok.NonNull;
+import lombok.extern.apachecommons.CommonsLog;
+
+/**
+ *
+ */
+@CommonsLog
+public class AsynchronousGetRecordsRetrievalStrategy implements GetRecordsRetrievalStrategy {
+    private static final int TIME_TO_KEEP_ALIVE = 5;
+    private static final int CORE_THREAD_POOL_COUNT = 1;
+
+    private final KinesisDataFetcher dataFetcher;
+    private final ExecutorService executorService;
+    private final int retryGetRecordsInSeconds;
+    private final String shardId;
+    final Supplier<CompletionService<DataFetcherResult>> completionServiceSupplier;
+
+    public AsynchronousGetRecordsRetrievalStrategy(@NonNull final KinesisDataFetcher dataFetcher,
+            final int retryGetRecordsInSeconds, final int maxGetRecordsThreadPool, String shardId) {
+        this(dataFetcher, buildExector(maxGetRecordsThreadPool, shardId), retryGetRecordsInSeconds, shardId);
+    }
+
+    public AsynchronousGetRecordsRetrievalStrategy(final KinesisDataFetcher dataFetcher,
+            final ExecutorService executorService, final int retryGetRecordsInSeconds, String shardId) {
+        this(dataFetcher, executorService, retryGetRecordsInSeconds, () -> new ExecutorCompletionService<>(executorService),
+                shardId);
+    }
+
+    AsynchronousGetRecordsRetrievalStrategy(KinesisDataFetcher dataFetcher, ExecutorService executorService,
+            int retryGetRecordsInSeconds, Supplier<CompletionService<DataFetcherResult>> completionServiceSupplier,
+            String shardId) {
+        this.dataFetcher = dataFetcher;
+        this.executorService = executorService;
+        this.retryGetRecordsInSeconds = retryGetRecordsInSeconds;
+        this.completionServiceSupplier = completionServiceSupplier;
+        this.shardId = shardId;
+    }
+
+    @Override
+    public GetRecordsResult getRecords(final int maxRecords) {
+        if (executorService.isShutdown()) {
+            throw new IllegalStateException("Strategy has been shutdown");
+        }
+        GetRecordsResult result = null;
+        CompletionService<DataFetcherResult> completionService = completionServiceSupplier.get();
+        Set<Future<DataFetcherResult>> futures = new HashSet<>();
+        Callable<DataFetcherResult> retrieverCall = createRetrieverCallable(maxRecords);
+        try {
+            while (true) {
+                try {
+                    futures.add(completionService.submit(retrieverCall));
+                } catch (RejectedExecutionException e) {
+                    log.warn("Out of resources, unable to start additional requests.");
+                }
+
+                try {
+                    Future<DataFetcherResult> resultFuture = completionService.poll(retryGetRecordsInSeconds,
+                            TimeUnit.SECONDS);
+                    if (resultFuture != null) {
+                        //
+                        // Fix to ensure that we only let the shard iterator advance when we intend to return the result
+                        // to the caller.  This ensures that the shard iterator is consistently advance in step with
+                        // what the caller sees.
+                        //
+                        result = resultFuture.get().accept();
+                        break;
+                    }
+                } catch (ExecutionException e) {
+                    if (e.getCause() instanceof ExpiredIteratorException) {
+                        throw (ExpiredIteratorException) e.getCause();
+                    }
+                    log.error("ExecutionException thrown while trying to get records", e);
+                } catch (InterruptedException e) {
+                    log.error("Thread was interrupted", e);
+                    break;
+                }
+            }
+        } finally {
+            futures.forEach(f -> f.cancel(true));
+        }
+        return result;
+    }
+
+    private Callable<DataFetcherResult> createRetrieverCallable(int maxRecords) {
+        ThreadSafeMetricsDelegatingScope metricsScope = new ThreadSafeMetricsDelegatingScope(MetricsHelper.getMetricsScope());
+        return () -> {
+            try {
+                MetricsHelper.setMetricsScope(metricsScope);
+                return dataFetcher.getRecords(maxRecords);
+            } finally {
+                MetricsHelper.unsetMetricsScope();
+            }
+        };
+    }
+
+    @Override
+    public void shutdown() {
+        executorService.shutdownNow();
+    }
+
+    @Override
+    public boolean isShutdown() {
+        return executorService.isShutdown();
+    }
+
+    private static ExecutorService buildExector(int maxGetRecordsThreadPool, String shardId) {
+        String threadNameFormat = "get-records-worker-" + shardId + "-%d";
+        return new ThreadPoolExecutor(CORE_THREAD_POOL_COUNT, maxGetRecordsThreadPool, TIME_TO_KEEP_ALIVE,
+                TimeUnit.SECONDS, new LinkedBlockingQueue<>(1),
+                new ThreadFactoryBuilder().setDaemon(true).setNameFormat(threadNameFormat).build(),
+                new ThreadPoolExecutor.AbortPolicy());
+    }
+
+    @Override
+    public KinesisDataFetcher getDataFetcher() {
+        return dataFetcher;
+    }
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/BlockingGetRecordsCache.java

@@ -0,0 +1,65 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License. 
+ */
+
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import java.time.Duration;
+import java.time.Instant;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+
+import lombok.extern.apachecommons.CommonsLog;
+
+/**
+ * This is the BlockingGetRecordsCache class. This class blocks any calls to the getRecords on the
+ * GetRecordsRetrievalStrategy class.
+ */
+@CommonsLog
+public class BlockingGetRecordsCache implements GetRecordsCache {
+    private final int maxRecordsPerCall;
+    private final GetRecordsRetrievalStrategy getRecordsRetrievalStrategy;
+
+    public BlockingGetRecordsCache(final int maxRecordsPerCall,
+                                   final GetRecordsRetrievalStrategy getRecordsRetrievalStrategy) {
+        this.maxRecordsPerCall = maxRecordsPerCall;
+        this.getRecordsRetrievalStrategy = getRecordsRetrievalStrategy;
+    }
+
+    @Override
+    public void start() {
+        //
+        // Nothing to do here
+        //
+    }
+
+    @Override
+    public ProcessRecordsInput getNextResult() {
+        GetRecordsResult getRecordsResult = getRecordsRetrievalStrategy.getRecords(maxRecordsPerCall);
+        return new ProcessRecordsInput()
+                .withRecords(getRecordsResult.getRecords())
+                .withMillisBehindLatest(getRecordsResult.getMillisBehindLatest());
+    }
+    
+    @Override
+    public GetRecordsRetrievalStrategy getGetRecordsRetrievalStrategy() {
+        return getRecordsRetrievalStrategy;
+    }
+
+    @Override
+    public void shutdown() {
+        getRecordsRetrievalStrategy.shutdown();
+    }
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ConsumerStates.java

@@ -251,9 +251,14 @@ class ConsumerStates {
 
         @Override
         public ITask createTask(ShardConsumer consumer) {
-            return new InitializeTask(consumer.getShardInfo(), consumer.getRecordProcessor(), consumer.getCheckpoint(),
+            return new InitializeTask(consumer.getShardInfo(),
-                    consumer.getRecordProcessorCheckpointer(), consumer.getDataFetcher(),
+                    consumer.getRecordProcessor(),
-                    consumer.getTaskBackoffTimeMillis(), consumer.getStreamConfig());
+                    consumer.getCheckpoint(),
+                    consumer.getRecordProcessorCheckpointer(),
+                    consumer.getDataFetcher(),
+                    consumer.getTaskBackoffTimeMillis(),
+                    consumer.getStreamConfig(),
+                    consumer.getGetRecordsCache());
         }
 
         @Override
@@ -307,9 +312,14 @@ class ConsumerStates {
 
         @Override
         public ITask createTask(ShardConsumer consumer) {
-            return new ProcessTask(consumer.getShardInfo(), consumer.getStreamConfig(), consumer.getRecordProcessor(),
+            return new ProcessTask(consumer.getShardInfo(),
-                    consumer.getRecordProcessorCheckpointer(), consumer.getDataFetcher(),
+                    consumer.getStreamConfig(),
-                    consumer.getTaskBackoffTimeMillis(), consumer.isSkipShardSyncAtWorkerInitializationIfLeasesExist());
+                    consumer.getRecordProcessor(),
+                    consumer.getRecordProcessorCheckpointer(),
+                    consumer.getDataFetcher(),
+                    consumer.getTaskBackoffTimeMillis(),
+                    consumer.isSkipShardSyncAtWorkerInitializationIfLeasesExist(),
+                    consumer.getGetRecordsCache());
         }
 
         @Override
@@ -368,8 +378,10 @@ class ConsumerStates {
 
         @Override
         public ITask createTask(ShardConsumer consumer) {
-            return new ShutdownNotificationTask(consumer.getRecordProcessor(), consumer.getRecordProcessorCheckpointer(),
+            return new ShutdownNotificationTask(consumer.getRecordProcessor(),
-                    consumer.getShutdownNotification(), consumer.getShardInfo());
+                    consumer.getRecordProcessorCheckpointer(),
+                    consumer.getShutdownNotification(),
+                    consumer.getShardInfo());
         }
 
         @Override
@@ -508,12 +520,17 @@ class ConsumerStates {
 
         @Override
         public ITask createTask(ShardConsumer consumer) {
-            return new ShutdownTask(consumer.getShardInfo(), consumer.getRecordProcessor(),
+            return new ShutdownTask(consumer.getShardInfo(),
-                    consumer.getRecordProcessorCheckpointer(), consumer.getShutdownReason(),
+                    consumer.getRecordProcessor(),
+                    consumer.getRecordProcessorCheckpointer(),
+                    consumer.getShutdownReason(),
                     consumer.getStreamConfig().getStreamProxy(),
                     consumer.getStreamConfig().getInitialPositionInStream(),
-                    consumer.isCleanupLeasesOfCompletedShards(), consumer.getLeaseManager(),
+                    consumer.isCleanupLeasesOfCompletedShards(),
-                    consumer.getTaskBackoffTimeMillis());
+                    consumer.isIgnoreUnexpectedChildShards(),
+                    consumer.getLeaseManager(),
+                    consumer.getTaskBackoffTimeMillis(),
+                    consumer.getGetRecordsCache());
         }
 
         @Override

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/DataFetcherResult.java

@@ -0,0 +1,37 @@
+/*
+ * Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Amazon Software License
+ * (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at
+ * http://aws.amazon.com/asl/ or in the "license" file accompanying this file. This file is distributed on an "AS IS"
+ * BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
+ * language governing permissions and limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+
+/**
+ * Represents the result from the DataFetcher, and allows the receiver to accept a result
+ */
+public interface DataFetcherResult {
+    /**
+     * The result of the request to Kinesis
+     * 
+     * @return The result of the request, this can be null if the request failed.
+     */
+    GetRecordsResult getResult();
+
+    /**
+     * Accepts the result, and advances the shard iterator. A result from the data fetcher must be accepted before any
+     * further progress can be made.
+     * 
+     * @return the result of the request, this can be null if the request failed.
+     */
+    GetRecordsResult accept();
+
+    /**
+     * Indicates whether this result is at the end of the shard or not
+     * 
+     * @return true if the result is at the end of a shard, false otherwise
+     */
+    boolean isShardEnd();
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/DataFetchingStrategy.java

@@ -0,0 +1,8 @@
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+/**
+ *
+ */
+public enum DataFetchingStrategy {
+    DEFAULT, PREFETCH_CACHED;
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/DoesNothingPreparedCheckpointer.java

@@ -1,3 +1,17 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
 import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/GetRecordsCache.java

@@ -0,0 +1,43 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License. 
+ */
+
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
+
+/**
+ * This class is used as a cache for Prefetching data from Kinesis.
+ */
+public interface GetRecordsCache {
+    /**
+     * This method calls the start behavior on the cache, if available.
+     */
+    void start();
+    
+    /**
+     * This method returns the next set of records from the Cache if present, or blocks the request till it gets the
+     * next set of records back from Kinesis.
+     * 
+     * @return The next set of records.
+     */
+    ProcessRecordsInput getNextResult();
+    
+    GetRecordsRetrievalStrategy getGetRecordsRetrievalStrategy();
+
+    /**
+     * This method calls the shutdown behavior on the cache, if available.
+     */
+    void shutdown();
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/GetRecordsRetrievalStrategy.java

@@ -0,0 +1,54 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+
+/**
+ * Represents a strategy to retrieve records from Kinesis. Allows for variations on how records are retrieved from
+ * Kinesis.
+ */
+public interface GetRecordsRetrievalStrategy {
+    /**
+     * Gets a set of records from Kinesis.
+     *
+     * @param maxRecords
+     *            passed to Kinesis, and can be used to restrict the number of records returned from Kinesis.
+     * @return the resulting records.
+     * @throws IllegalStateException
+     *             if the strategy has been shutdown.
+     */
+    GetRecordsResult getRecords(int maxRecords);
+
+    /**
+     * Releases any resources used by the strategy. Once the strategy is shutdown it is no longer safe to call
+     * {@link #getRecords(int)}.
+     */
+    void shutdown();
+
+    /**
+     * Returns whether this strategy has been shutdown.
+     * 
+     * @return true if the strategy has been shutdown, false otherwise.
+     */
+    boolean isShutdown();
+
+    /**
+     * Returns the KinesisDataFetcher used to getRecords from Kinesis.
+     * 
+     * @return KinesisDataFetcher
+     */
+    KinesisDataFetcher getDataFetcher();
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/GetRecordsRetriever.java

@@ -0,0 +1,12 @@
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+
+import java.util.concurrent.Callable;
+
+/**
+ * This class uses the GetRecordsRetrievalStrategy class to retrieve the next set of records and update the cache. 
+ */
+public interface GetRecordsRetriever {
+    GetRecordsResult getNextRecords(int maxRecords);
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/InitializeTask.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
@@ -43,17 +43,19 @@ class InitializeTask implements ITask {
     // Back off for this interval if we encounter a problem (exception)
     private final long backoffTimeMillis;
     private final StreamConfig streamConfig;
+    private final GetRecordsCache getRecordsCache;
 
     /**
      * Constructor.
      */
     InitializeTask(ShardInfo shardInfo,
-            IRecordProcessor recordProcessor,
+                   IRecordProcessor recordProcessor,
-            ICheckpoint checkpoint,
+                   ICheckpoint checkpoint,
-            RecordProcessorCheckpointer recordProcessorCheckpointer,
+                   RecordProcessorCheckpointer recordProcessorCheckpointer,
-            KinesisDataFetcher dataFetcher,
+                   KinesisDataFetcher dataFetcher,
-            long backoffTimeMillis,
+                   long backoffTimeMillis,
-            StreamConfig streamConfig) {
+                   StreamConfig streamConfig,
+                   GetRecordsCache getRecordsCache) {
         this.shardInfo = shardInfo;
         this.recordProcessor = recordProcessor;
         this.checkpoint = checkpoint;
@@ -61,6 +63,7 @@ class InitializeTask implements ITask {
         this.dataFetcher = dataFetcher;
         this.backoffTimeMillis = backoffTimeMillis;
         this.streamConfig = streamConfig;
+        this.getRecordsCache = getRecordsCache;
     }
 
     /*
@@ -80,6 +83,7 @@ class InitializeTask implements ITask {
             ExtendedSequenceNumber initialCheckpoint = initialCheckpointObject.getCheckpoint();
 
             dataFetcher.initialize(initialCheckpoint.getSequenceNumber(), streamConfig.getInitialPositionInStream());
+            getRecordsCache.start();
             recordProcessorCheckpointer.setLargestPermittedCheckpointValue(initialCheckpoint);
             recordProcessorCheckpointer.setInitialCheckpointValue(initialCheckpoint);
 

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
@@ -126,7 +126,7 @@ public class KinesisClientLibConfiguration {
     /**
      * User agent set when Amazon Kinesis Client Library makes AWS requests.
      */
-    public static final String KINESIS_CLIENT_LIB_USER_AGENT = "amazon-kinesis-client-library-java-1.8.1";
+    public static final String KINESIS_CLIENT_LIB_USER_AGENT = "amazon-kinesis-client-library-java-1.8.10";
 
     /**
      * KCL will validate client provided sequence numbers with a call to Amazon Kinesis before checkpointing for calls
@@ -172,6 +172,11 @@ public class KinesisClientLibConfiguration {
      */
     public static final ShardPrioritization DEFAULT_SHARD_PRIORITIZATION = new NoOpShardPrioritization();
 
+    /**
+     * The amount of milliseconds to wait before graceful shutdown forcefully terminates.
+     */
+    public static final long DEFAULT_SHUTDOWN_GRACE_MILLIS = 5000L;
+
     /**
      * The size of the thread pool to create for the lease renewer to use.
      */
@@ -195,6 +200,7 @@ public class KinesisClientLibConfiguration {
     private boolean callProcessRecordsEvenForEmptyRecordList;
     private long parentShardPollIntervalMillis;
     private boolean cleanupLeasesUponShardCompletion;
+    private boolean ignoreUnexpectedChildShards;
     private ClientConfiguration kinesisClientConfig;
     private ClientConfiguration dynamoDBClientConfig;
     private ClientConfiguration cloudWatchClientConfig;
@@ -213,13 +219,26 @@ public class KinesisClientLibConfiguration {
     // This is useful for optimizing deployments to large fleets working on a stable stream.
     private boolean skipShardSyncAtWorkerInitializationIfLeasesExist;
     private ShardPrioritization shardPrioritization;
+    private long shutdownGraceMillis;
 
     @Getter
     private Optional<Integer> timeoutInSeconds = Optional.empty();
 
+    @Getter
+    private Optional<Integer> retryGetRecordsInSeconds = Optional.empty();
+
+    @Getter
+    private Optional<Integer> maxGetRecordsThreadPool = Optional.empty();
+
     @Getter
     private int maxLeaseRenewalThreads = DEFAULT_MAX_LEASE_RENEWAL_THREADS;
 
+    @Getter
+    private RecordsFetcherFactory recordsFetcherFactory;
+    
+    @Getter
+    private Optional<Long> logWarningForTaskAfterMillis = Optional.empty();
+
     /**
      * Constructor.
      *
@@ -255,14 +274,31 @@ public class KinesisClientLibConfiguration {
             AWSCredentialsProvider dynamoDBCredentialsProvider,
             AWSCredentialsProvider cloudWatchCredentialsProvider,
             String workerId) {
-        this(applicationName, streamName, null, null, DEFAULT_INITIAL_POSITION_IN_STREAM, kinesisCredentialsProvider,
+        this(applicationName,
-                dynamoDBCredentialsProvider, cloudWatchCredentialsProvider, DEFAULT_FAILOVER_TIME_MILLIS, workerId,
+                streamName,
-                DEFAULT_MAX_RECORDS, DEFAULT_IDLETIME_BETWEEN_READS_MILLIS,
+                null,
-                DEFAULT_DONT_CALL_PROCESS_RECORDS_FOR_EMPTY_RECORD_LIST, DEFAULT_PARENT_SHARD_POLL_INTERVAL_MILLIS,
+                null,
-                DEFAULT_SHARD_SYNC_INTERVAL_MILLIS, DEFAULT_CLEANUP_LEASES_UPON_SHARDS_COMPLETION,
+                DEFAULT_INITIAL_POSITION_IN_STREAM,
-                new ClientConfiguration(), new ClientConfiguration(), new ClientConfiguration(),
+                kinesisCredentialsProvider,
-                DEFAULT_TASK_BACKOFF_TIME_MILLIS, DEFAULT_METRICS_BUFFER_TIME_MILLIS, DEFAULT_METRICS_MAX_QUEUE_SIZE,
+                dynamoDBCredentialsProvider,
-                DEFAULT_VALIDATE_SEQUENCE_NUMBER_BEFORE_CHECKPOINTING, null);
+                cloudWatchCredentialsProvider,
+                DEFAULT_FAILOVER_TIME_MILLIS,
+                workerId,
+                DEFAULT_MAX_RECORDS,
+                DEFAULT_IDLETIME_BETWEEN_READS_MILLIS,
+                DEFAULT_DONT_CALL_PROCESS_RECORDS_FOR_EMPTY_RECORD_LIST,
+                DEFAULT_PARENT_SHARD_POLL_INTERVAL_MILLIS,
+                DEFAULT_SHARD_SYNC_INTERVAL_MILLIS,
+                DEFAULT_CLEANUP_LEASES_UPON_SHARDS_COMPLETION,
+                new ClientConfiguration(),
+                new ClientConfiguration(),
+                new ClientConfiguration(),
+                DEFAULT_TASK_BACKOFF_TIME_MILLIS,
+                DEFAULT_METRICS_BUFFER_TIME_MILLIS,
+                DEFAULT_METRICS_MAX_QUEUE_SIZE,
+                DEFAULT_VALIDATE_SEQUENCE_NUMBER_BEFORE_CHECKPOINTING,
+                null,
+                DEFAULT_SHUTDOWN_GRACE_MILLIS);
     }
 
     /**
@@ -297,32 +333,34 @@ public class KinesisClientLibConfiguration {
      *        with a call to Amazon Kinesis before checkpointing for calls to
      *        {@link RecordProcessorCheckpointer#checkpoint(String)}
      * @param regionName The region name for the service
+     * @param shutdownGraceMillis The number of milliseconds before graceful shutdown terminates forcefully
      */
     // CHECKSTYLE:IGNORE HiddenFieldCheck FOR NEXT 26 LINES
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 26 LINES
     public KinesisClientLibConfiguration(String applicationName,
-            String streamName,
+                                         String streamName,
-            String kinesisEndpoint,
+                                         String kinesisEndpoint,
-            InitialPositionInStream initialPositionInStream,
+                                         InitialPositionInStream initialPositionInStream,
-            AWSCredentialsProvider kinesisCredentialsProvider,
+                                         AWSCredentialsProvider kinesisCredentialsProvider,
-            AWSCredentialsProvider dynamoDBCredentialsProvider,
+                                         AWSCredentialsProvider dynamoDBCredentialsProvider,
-            AWSCredentialsProvider cloudWatchCredentialsProvider,
+                                         AWSCredentialsProvider cloudWatchCredentialsProvider,
-            long failoverTimeMillis,
+                                         long failoverTimeMillis,
-            String workerId,
+                                         String workerId,
-            int maxRecords,
+                                         int maxRecords,
-            long idleTimeBetweenReadsInMillis,
+                                         long idleTimeBetweenReadsInMillis,
-            boolean callProcessRecordsEvenForEmptyRecordList,
+                                         boolean callProcessRecordsEvenForEmptyRecordList,
-            long parentShardPollIntervalMillis,
+                                         long parentShardPollIntervalMillis,
-            long shardSyncIntervalMillis,
+                                         long shardSyncIntervalMillis,
-            boolean cleanupTerminatedShardsBeforeExpiry,
+                                         boolean cleanupTerminatedShardsBeforeExpiry,
-            ClientConfiguration kinesisClientConfig,
+                                         ClientConfiguration kinesisClientConfig,
-            ClientConfiguration dynamoDBClientConfig,
+                                         ClientConfiguration dynamoDBClientConfig,
-            ClientConfiguration cloudWatchClientConfig,
+                                         ClientConfiguration cloudWatchClientConfig,
-            long taskBackoffTimeMillis,
+                                         long taskBackoffTimeMillis,
-            long metricsBufferTimeMillis,
+                                         long metricsBufferTimeMillis,
-            int metricsMaxQueueSize,
+                                         int metricsMaxQueueSize,
-            boolean validateSequenceNumberBeforeCheckpointing,
+                                         boolean validateSequenceNumberBeforeCheckpointing,
-            String regionName) {
+                                         String regionName,
+                                         long shutdownGraceMillis) {
         this(applicationName, streamName, kinesisEndpoint, null, initialPositionInStream, kinesisCredentialsProvider,
                 dynamoDBCredentialsProvider, cloudWatchCredentialsProvider, failoverTimeMillis, workerId,
                 maxRecords, idleTimeBetweenReadsInMillis,
@@ -330,7 +368,117 @@ public class KinesisClientLibConfiguration {
                 shardSyncIntervalMillis, cleanupTerminatedShardsBeforeExpiry,
                 kinesisClientConfig, dynamoDBClientConfig, cloudWatchClientConfig,
                 taskBackoffTimeMillis, metricsBufferTimeMillis, metricsMaxQueueSize,
-                validateSequenceNumberBeforeCheckpointing, regionName);
+                validateSequenceNumberBeforeCheckpointing, regionName, shutdownGraceMillis);
+    }
+
+    /**
+     * @param applicationName Name of the Kinesis application
+     *        By default the application name is included in the user agent string used to make AWS requests. This
+     *        can assist with troubleshooting (e.g. distinguish requests made by separate applications).
+     * @param streamName Name of the Kinesis stream
+     * @param kinesisEndpoint Kinesis endpoint
+     * @param dynamoDBEndpoint DynamoDB endpoint
+     * @param initialPositionInStream One of LATEST or TRIM_HORIZON. The KinesisClientLibrary will start fetching
+     *        records from that location in the stream when an application starts up for the first time and there
+     *        are no checkpoints. If there are checkpoints, then we start from the checkpoint position.
+     * @param kinesisCredentialsProvider Provides credentials used to access Kinesis
+     * @param dynamoDBCredentialsProvider Provides credentials used to access DynamoDB
+     * @param cloudWatchCredentialsProvider Provides credentials used to access CloudWatch
+     * @param failoverTimeMillis Lease duration (leases not renewed within this period will be claimed by others)
+     * @param workerId Used to distinguish different workers/processes of a Kinesis application
+     * @param maxRecords Max records to read per Kinesis getRecords() call
+     * @param idleTimeBetweenReadsInMillis Idle time between calls to fetch data from Kinesis
+     * @param callProcessRecordsEvenForEmptyRecordList Call the IRecordProcessor::processRecords() API even if
+     *        GetRecords returned an empty record list.
+     * @param parentShardPollIntervalMillis Wait for this long between polls to check if parent shards are done
+     * @param shardSyncIntervalMillis Time between tasks to sync leases and Kinesis shards
+     * @param cleanupTerminatedShardsBeforeExpiry Clean up shards we've finished processing (don't wait for expiration
+     *        in Kinesis)
+     * @param kinesisClientConfig Client Configuration used by Kinesis client
+     * @param dynamoDBClientConfig Client Configuration used by DynamoDB client
+     * @param cloudWatchClientConfig Client Configuration used by CloudWatch client
+     * @param taskBackoffTimeMillis Backoff period when tasks encounter an exception
+     * @param metricsBufferTimeMillis Metrics are buffered for at most this long before publishing to CloudWatch
+     * @param metricsMaxQueueSize Max number of metrics to buffer before publishing to CloudWatch
+     * @param validateSequenceNumberBeforeCheckpointing whether KCL should validate client provided sequence numbers
+     *        with a call to Amazon Kinesis before checkpointing for calls to
+     *        {@link RecordProcessorCheckpointer#checkpoint(String)}
+     * @param regionName The region name for the service
+     */
+    // CHECKSTYLE:IGNORE HiddenFieldCheck FOR NEXT 26 LINES
+    // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 26 LINES
+    public KinesisClientLibConfiguration(String applicationName,
+                                         String streamName,
+                                         String kinesisEndpoint,
+                                         String dynamoDBEndpoint,
+                                         InitialPositionInStream initialPositionInStream,
+                                         AWSCredentialsProvider kinesisCredentialsProvider,
+                                         AWSCredentialsProvider dynamoDBCredentialsProvider,
+                                         AWSCredentialsProvider cloudWatchCredentialsProvider,
+                                         long failoverTimeMillis,
+                                         String workerId,
+                                         int maxRecords,
+                                         long idleTimeBetweenReadsInMillis,
+                                         boolean callProcessRecordsEvenForEmptyRecordList,
+                                         long parentShardPollIntervalMillis,
+                                         long shardSyncIntervalMillis,
+                                         boolean cleanupTerminatedShardsBeforeExpiry,
+                                         ClientConfiguration kinesisClientConfig,
+                                         ClientConfiguration dynamoDBClientConfig,
+                                         ClientConfiguration cloudWatchClientConfig,
+                                         long taskBackoffTimeMillis,
+                                         long metricsBufferTimeMillis,
+                                         int metricsMaxQueueSize,
+                                         boolean validateSequenceNumberBeforeCheckpointing,
+                                         String regionName,
+                                         long shutdownGraceMillis) {
+        // Check following values are greater than zero
+        checkIsValuePositive("FailoverTimeMillis", failoverTimeMillis);
+        checkIsValuePositive("IdleTimeBetweenReadsInMillis", idleTimeBetweenReadsInMillis);
+        checkIsValuePositive("ParentShardPollIntervalMillis", parentShardPollIntervalMillis);
+        checkIsValuePositive("ShardSyncIntervalMillis", shardSyncIntervalMillis);
+        checkIsValuePositive("MaxRecords", (long) maxRecords);
+        checkIsValuePositive("TaskBackoffTimeMillis", taskBackoffTimeMillis);
+        checkIsValuePositive("MetricsBufferTimeMills", metricsBufferTimeMillis);
+        checkIsValuePositive("MetricsMaxQueueSize", (long) metricsMaxQueueSize);
+        checkIsValuePositive("ShutdownGraceMillis", shutdownGraceMillis);
+        checkIsRegionNameValid(regionName);
+        this.applicationName = applicationName;
+        this.tableName = applicationName;
+        this.streamName = streamName;
+        this.kinesisEndpoint = kinesisEndpoint;
+        this.dynamoDBEndpoint = dynamoDBEndpoint;
+        this.initialPositionInStream = initialPositionInStream;
+        this.kinesisCredentialsProvider = kinesisCredentialsProvider;
+        this.dynamoDBCredentialsProvider = dynamoDBCredentialsProvider;
+        this.cloudWatchCredentialsProvider = cloudWatchCredentialsProvider;
+        this.failoverTimeMillis = failoverTimeMillis;
+        this.maxRecords = maxRecords;
+        this.idleTimeBetweenReadsInMillis = idleTimeBetweenReadsInMillis;
+        this.callProcessRecordsEvenForEmptyRecordList = callProcessRecordsEvenForEmptyRecordList;
+        this.parentShardPollIntervalMillis = parentShardPollIntervalMillis;
+        this.shardSyncIntervalMillis = shardSyncIntervalMillis;
+        this.cleanupLeasesUponShardCompletion = cleanupTerminatedShardsBeforeExpiry;
+        this.workerIdentifier = workerId;
+        this.kinesisClientConfig = checkAndAppendKinesisClientLibUserAgent(kinesisClientConfig);
+        this.dynamoDBClientConfig = checkAndAppendKinesisClientLibUserAgent(dynamoDBClientConfig);
+        this.cloudWatchClientConfig = checkAndAppendKinesisClientLibUserAgent(cloudWatchClientConfig);
+        this.taskBackoffTimeMillis = taskBackoffTimeMillis;
+        this.metricsBufferTimeMillis = metricsBufferTimeMillis;
+        this.metricsMaxQueueSize = metricsMaxQueueSize;
+        this.metricsLevel = DEFAULT_METRICS_LEVEL;
+        this.metricsEnabledDimensions = DEFAULT_METRICS_ENABLED_DIMENSIONS;
+        this.validateSequenceNumberBeforeCheckpointing = validateSequenceNumberBeforeCheckpointing;
+        this.regionName = regionName;
+        this.maxLeasesForWorker = DEFAULT_MAX_LEASES_FOR_WORKER;
+        this.maxLeasesToStealAtOneTime = DEFAULT_MAX_LEASES_TO_STEAL_AT_ONE_TIME;
+        this.initialLeaseTableReadCapacity = DEFAULT_INITIAL_LEASE_TABLE_READ_CAPACITY;
+        this.initialLeaseTableWriteCapacity = DEFAULT_INITIAL_LEASE_TABLE_WRITE_CAPACITY;
+        this.initialPositionInStreamExtended =
+                InitialPositionInStreamExtended.newInitialPosition(initialPositionInStream);
+        this.skipShardSyncAtWorkerInitializationIfLeasesExist = DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST;
+        this.shardPrioritization = DEFAULT_SHARD_PRIORITIZATION;
+        this.recordsFetcherFactory = new SimpleRecordsFetcherFactory();
     }
 
     /**
@@ -392,7 +540,8 @@ public class KinesisClientLibConfiguration {
             long metricsBufferTimeMillis,
             int metricsMaxQueueSize,
             boolean validateSequenceNumberBeforeCheckpointing,
-            String regionName) {
+            String regionName,
+            RecordsFetcherFactory recordsFetcherFactory) {
         // Check following values are greater than zero
         checkIsValuePositive("FailoverTimeMillis", failoverTimeMillis);
         checkIsValuePositive("IdleTimeBetweenReadsInMillis", idleTimeBetweenReadsInMillis);
@@ -438,6 +587,8 @@ public class KinesisClientLibConfiguration {
                 InitialPositionInStreamExtended.newInitialPosition(initialPositionInStream);
         this.skipShardSyncAtWorkerInitializationIfLeasesExist = DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST;
         this.shardPrioritization = DEFAULT_SHARD_PRIORITIZATION;
+        this.recordsFetcherFactory = recordsFetcherFactory;
+        this.shutdownGraceMillis = shutdownGraceMillis;
     }
 
     // Check if value is positive, otherwise throw an exception
@@ -652,6 +803,13 @@ public class KinesisClientLibConfiguration {
         return cleanupLeasesUponShardCompletion;
     }
 
+    /**
+     * @return true if we should ignore child shards which have open parents
+     */
+    public boolean shouldIgnoreUnexpectedChildShards() {
+        return ignoreUnexpectedChildShards;
+    }
+
     /**
      * @return true if KCL should validate client provided sequence numbers with a call to Amazon Kinesis before
      *         checkpointing for calls to {@link RecordProcessorCheckpointer#checkpoint(String)}
@@ -725,6 +883,14 @@ public class KinesisClientLibConfiguration {
         return shardPrioritization;
     }
 
+    /**
+     * @return Graceful shutdown timeout
+     */
+    public long getShutdownGraceMillis() {
+        return shutdownGraceMillis;
+    }
+
+    /*
     // CHECKSTYLE:IGNORE HiddenFieldCheck FOR NEXT 190 LINES
     /**
      * @param tableName name of the lease table in DynamoDB
@@ -864,6 +1030,16 @@ public class KinesisClientLibConfiguration {
         return this;
     }
 
+    /**
+     * @param ignoreUnexpectedChildShards Ignore child shards with open parents.
+     * @return KinesisClientLibConfiguration
+     */
+    public KinesisClientLibConfiguration withIgnoreUnexpectedChildShards(
+            boolean ignoreUnexpectedChildShards) {
+        this.ignoreUnexpectedChildShards = ignoreUnexpectedChildShards;
+        return this;
+    }
+
     /**
      * @param clientConfig Common client configuration used by Kinesis/DynamoDB/CloudWatch client
      * @return KinesisClientLibConfiguration
@@ -1111,6 +1287,69 @@ public class KinesisClientLibConfiguration {
         return this;
     }
 
+
+    /**
+     * @param retryGetRecordsInSeconds the time in seconds to wait before the worker retries to get a record.
+     * @return this configuration object.
+     */
+    public KinesisClientLibConfiguration withRetryGetRecordsInSeconds(final int retryGetRecordsInSeconds) {
+        checkIsValuePositive("retryGetRecordsInSeconds", retryGetRecordsInSeconds);
+        this.retryGetRecordsInSeconds = Optional.of(retryGetRecordsInSeconds);
+        return this;
+    }
+
+    /**
+     *@param maxGetRecordsThreadPool the max number of threads in the getRecords thread pool.
+     *@return this configuration object
+     */
+    public KinesisClientLibConfiguration withMaxGetRecordsThreadPool(final int maxGetRecordsThreadPool) {
+        checkIsValuePositive("maxGetRecordsThreadPool", maxGetRecordsThreadPool);
+        this.maxGetRecordsThreadPool = Optional.of(maxGetRecordsThreadPool);
+        return this;
+    }
+
+    /**
+     *
+     * @param maxPendingProcessRecordsInput The max number of ProcessRecordsInput that can be stored in the cache before
+     *                                      blocking
+     * @return this configuration object
+     */
+    public KinesisClientLibConfiguration withMaxPendingProcessRecordsInput(final int maxPendingProcessRecordsInput) {
+        checkIsValuePositive("maxPendingProcessRecordsInput", maxPendingProcessRecordsInput);
+        this.recordsFetcherFactory.setMaxPendingProcessRecordsInput(maxPendingProcessRecordsInput);
+        return this;
+    }
+
+    /**
+     * @param maxCacheByteSize Max byte size for the cache at any given point of time. After this threshold is crossed
+     *                         the KinesisDataFetcher will be blocked until the cache has more space available.
+     * @return KinesisClientLibConfiguration
+     */
+    public KinesisClientLibConfiguration withMaxCacheByteSize(final int maxCacheByteSize) {
+        checkIsValuePositive("maxCacheByteSize", maxCacheByteSize);
+        this.recordsFetcherFactory.setMaxByteSize(maxCacheByteSize);
+        return this;
+    }
+
+    /**
+     * @param dataFetchingStrategy The strategy for fetching data from kinesis.
+     * @return KinesisClientLibConfiguration
+     */
+    public KinesisClientLibConfiguration withDataFetchingStrategy(String dataFetchingStrategy) {
+        this.recordsFetcherFactory.setDataFetchingStrategy(DataFetchingStrategy.valueOf(dataFetchingStrategy.toUpperCase()));
+        return this;
+    }
+
+    /**
+     * @param maxRecordsCount The maximum number of records in the cache, accross all ProcessRecordInput objects
+     * @return KinesisClientLibConfiguration
+     */
+    public KinesisClientLibConfiguration withMaxRecordsCount(final int maxRecordsCount) {
+        checkIsValuePositive("maxRecordsCount", maxRecordsCount);
+        this.recordsFetcherFactory.setMaxRecordsCount(maxRecordsCount);
+        return this;
+    }
+
     /**
      * @param timeoutInSeconds The timeout in seconds to wait for the MultiLangProtocol to wait for
      */
@@ -1118,4 +1357,34 @@ public class KinesisClientLibConfiguration {
         this.timeoutInSeconds = Optional.of(timeoutInSeconds);
     }
 
+    /**
+     * @param shutdownGraceMillis Time before gracefully shutdown forcefully terminates
+     * @return KinesisClientLibConfiguration
+     */
+    public KinesisClientLibConfiguration withShutdownGraceMillis(long shutdownGraceMillis) {
+        checkIsValuePositive("ShutdownGraceMillis", shutdownGraceMillis);
+        this.shutdownGraceMillis = shutdownGraceMillis;
+        return this;
+    }
+
+    /** 
+     * @param idleMillisBetweenCalls Idle time between 2 getcalls from the data fetcher.
+     * @return KinesisClientLibConfiguration
+     */
+    public KinesisClientLibConfiguration withIdleMillisBetweenCalls(long idleMillisBetweenCalls) {
+        checkIsValuePositive("IdleMillisBetweenCalls", idleMillisBetweenCalls);
+        this.recordsFetcherFactory.setIdleMillisBetweenCalls(idleMillisBetweenCalls);
+        return this;
+    }
+
+    /**
+     * @param logWarningForTaskAfterMillis Logs warn message if as task is held in  a task for more than the set
+     *                                          time.
+     * @return KinesisClientLibConfiguration
+     */
+    public KinesisClientLibConfiguration withLogWarningForTaskAfterMillis(long logWarningForTaskAfterMillis) {
+        checkIsValuePositive("LogProcessTaskStatusAfterInMillis", logWarningForTaskAfterMillis);
+        this.logWarningForTaskAfterMillis = Optional.of(logWarningForTaskAfterMillis);
+        return this;
+    }
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisDataFetcher.java

@@ -1,31 +1,37 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import java.util.Collections;
+import java.util.Date;
+
+import org.apache.commons.lang.StringUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 
-import com.amazonaws.services.kinesis.model.GetRecordsResult;
-import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
-import com.amazonaws.services.kinesis.model.ShardIteratorType;
 import com.amazonaws.services.kinesis.clientlibrary.lib.checkpoint.SentinelCheckpoint;
 import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
 import com.amazonaws.services.kinesis.clientlibrary.proxies.MetricsCollectingKinesisProxyDecorator;
 import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
+import com.amazonaws.services.kinesis.model.ShardIteratorType;
+import com.amazonaws.util.CollectionUtils;
+import com.google.common.collect.Iterables;
 
-import java.util.Date;
+import lombok.Data;
 
 /**
  * Used to get data from Amazon Kinesis. Tracks iterator state internally.
@@ -39,6 +45,8 @@ class KinesisDataFetcher {
     private final String shardId;
     private boolean isShardEndReached;
     private boolean isInitialized;
+    private String lastKnownSequenceNumber;
+    private InitialPositionInStreamExtended initialPositionInStream;
 
     /**
      *
@@ -47,8 +55,7 @@ class KinesisDataFetcher {
      */
     public KinesisDataFetcher(IKinesisProxy kinesisProxy, ShardInfo shardInfo) {
         this.shardId = shardInfo.getShardId();
-        this.kinesisProxy =
+        this.kinesisProxy = new MetricsCollectingKinesisProxyDecorator("KinesisDataFetcher", kinesisProxy, this.shardId);
-                new MetricsCollectingKinesisProxyDecorator("KinesisDataFetcher", kinesisProxy, this.shardId);
     }
 
     /**
@@ -57,28 +64,68 @@ class KinesisDataFetcher {
      * @param maxRecords Max records to fetch
      * @return list of records of up to maxRecords size
      */
-    public GetRecordsResult getRecords(int maxRecords) {
+    public DataFetcherResult getRecords(int maxRecords) {
         if (!isInitialized) {
             throw new IllegalArgumentException("KinesisDataFetcher.getRecords called before initialization.");
         }
-
+        
-        GetRecordsResult response = null;
         if (nextIterator != null) {
             try {
-                response = kinesisProxy.get(nextIterator, maxRecords);
+                return new AdvancingResult(kinesisProxy.get(nextIterator, maxRecords));
-                nextIterator = response.getNextShardIterator();
             } catch (ResourceNotFoundException e) {
                 LOG.info("Caught ResourceNotFoundException when fetching records for shard " + shardId);
-                nextIterator = null;
+                return TERMINAL_RESULT;
+            }
+        } else {
+            return TERMINAL_RESULT;
+        }
+    }
+
+    final DataFetcherResult TERMINAL_RESULT = new DataFetcherResult() {
+        @Override
+        public GetRecordsResult getResult() {
+            return new GetRecordsResult().withMillisBehindLatest(null).withRecords(Collections.emptyList())
+                    .withNextShardIterator(null);
+        }
+
+        @Override
+        public GetRecordsResult accept() {
+            isShardEndReached = true;
+            return getResult();
+        }
+
+        @Override
+        public boolean isShardEnd() {
+            return isShardEndReached;
+        }
+    };
+
+    @Data
+    class AdvancingResult implements DataFetcherResult {
+
+        final GetRecordsResult result;
+
+        @Override
+        public GetRecordsResult getResult() {
+            return result;
+        }
+
+        @Override
+        public GetRecordsResult accept() {
+            nextIterator = result.getNextShardIterator();
+            if (!CollectionUtils.isNullOrEmpty(result.getRecords())) {
+                lastKnownSequenceNumber = Iterables.getLast(result.getRecords()).getSequenceNumber();
             }
             if (nextIterator == null) {
                 isShardEndReached = true;
             }
-        } else {
+            return getResult();
-            isShardEndReached = true;
         }
 
-        return response;
+        @Override
+        public boolean isShardEnd() {
+            return isShardEndReached;
+        }
     }
 
     /**
@@ -122,6 +169,8 @@ class KinesisDataFetcher {
         if (nextIterator == null) {
             isShardEndReached = true;
         }
+        this.lastKnownSequenceNumber = sequenceNumber;
+        this.initialPositionInStream = initialPositionInStream;
     }
 
     /**
@@ -178,6 +227,17 @@ class KinesisDataFetcher {
         return iterator;
     }
 
+    /**
+     * Gets a new iterator from the last known sequence number i.e. the sequence number of the last record from the last
+     * getRecords call.
+     */
+    public void restartIterator() {
+        if (StringUtils.isEmpty(lastKnownSequenceNumber) || initialPositionInStream == null) {
+            throw new IllegalStateException("Make sure to initialize the KinesisDataFetcher before restarting the iterator.");
+        }
+        advanceIteratorTo(lastKnownSequenceNumber, initialPositionInStream);
+    }
+
     /**
      * @return the shardEndReached
      */

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PrefetchGetRecordsCache.java

@@ -0,0 +1,268 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License. 
+ */
+
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import org.apache.commons.lang.Validate;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.cloudwatch.model.StandardUnit;
+import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
+import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
+import com.amazonaws.services.kinesis.metrics.impl.ThreadSafeMetricsDelegatingFactory;
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+import com.amazonaws.services.kinesis.metrics.interfaces.MetricsLevel;
+import com.amazonaws.services.kinesis.model.ExpiredIteratorException;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+
+import lombok.NonNull;
+import lombok.extern.apachecommons.CommonsLog;
+
+/**
+ * This is the prefetch caching class, this class spins up a thread if prefetching is enabled. That thread fetches the
+ * next set of records and stores it in the cache. The size of the cache is limited by setting
+ * maxPendingProcessRecordsInput i.e. the maximum number of GetRecordsResult that the cache can store, maxByteSize
+ * i.e. the byte size of the records stored in the cache and maxRecordsCount i.e. the max number of records that should
+ * be present in the cache across multiple GetRecordsResult object. If no data is available in the cache, the call from
+ * the record processor is blocked till records are retrieved from Kinesis.
+ */
+@CommonsLog
+public class PrefetchGetRecordsCache implements GetRecordsCache {
+    private static final String EXPIRED_ITERATOR_METRIC = "ExpiredIterator";
+    LinkedBlockingQueue<ProcessRecordsInput> getRecordsResultQueue;
+    private int maxPendingProcessRecordsInput;
+    private int maxByteSize;
+    private int maxRecordsCount;
+    private final int maxRecordsPerCall;
+    private final GetRecordsRetrievalStrategy getRecordsRetrievalStrategy;
+    private final ExecutorService executorService;
+    private final IMetricsFactory metricsFactory;
+    private final long idleMillisBetweenCalls;
+    private Instant lastSuccessfulCall;
+    private final DefaultGetRecordsCacheDaemon defaultGetRecordsCacheDaemon;
+    private PrefetchCounters prefetchCounters;
+    private boolean started = false;
+    private final String operation;
+    private final KinesisDataFetcher dataFetcher;
+    private final String shardId;
+
+    /**
+     * Constructor for the PrefetchGetRecordsCache. This cache prefetches records from Kinesis and stores them in a
+     * LinkedBlockingQueue.
+     * 
+     * @see com.amazonaws.services.kinesis.clientlibrary.lib.worker.PrefetchGetRecordsCache
+     * 
+     * @param maxPendingProcessRecordsInput Max number of ProcessRecordsInput that can be held in the cache before
+     *                                     blocking
+     * @param maxByteSize Max byte size of the queue before blocking next get records call
+     * @param maxRecordsCount Max number of records in the queue across all ProcessRecordInput objects
+     * @param maxRecordsPerCall Max records to be returned per call
+     * @param getRecordsRetrievalStrategy Retrieval strategy for the get records call
+     * @param executorService Executor service for the cache
+     * @param idleMillisBetweenCalls maximum time to wait before dispatching the next get records call
+     */
+    public PrefetchGetRecordsCache(final int maxPendingProcessRecordsInput, final int maxByteSize, final int maxRecordsCount,
+                                   final int maxRecordsPerCall,
+                                   @NonNull final GetRecordsRetrievalStrategy getRecordsRetrievalStrategy,
+                                   @NonNull final ExecutorService executorService,
+                                   final long idleMillisBetweenCalls,
+                                   @NonNull final IMetricsFactory metricsFactory,
+                                   @NonNull final String operation,
+                                   @NonNull final String shardId) {
+        this.getRecordsRetrievalStrategy = getRecordsRetrievalStrategy;
+        this.maxRecordsPerCall = maxRecordsPerCall;
+        this.maxPendingProcessRecordsInput = maxPendingProcessRecordsInput;
+        this.maxByteSize = maxByteSize;
+        this.maxRecordsCount = maxRecordsCount;
+        this.getRecordsResultQueue = new LinkedBlockingQueue<>(this.maxPendingProcessRecordsInput);
+        this.prefetchCounters = new PrefetchCounters();
+        this.executorService = executorService;
+        this.metricsFactory = new ThreadSafeMetricsDelegatingFactory(metricsFactory);
+        this.idleMillisBetweenCalls = idleMillisBetweenCalls;
+        this.defaultGetRecordsCacheDaemon = new DefaultGetRecordsCacheDaemon();
+        Validate.notEmpty(operation, "Operation cannot be empty");
+        this.operation = operation;
+        this.dataFetcher = this.getRecordsRetrievalStrategy.getDataFetcher();
+        this.shardId = shardId;
+    }
+
+    @Override
+    public void start() {
+        if (executorService.isShutdown()) {
+            throw new IllegalStateException("ExecutorService has been shutdown.");
+        }
+        
+        if (!started) {
+            log.info("Starting prefetching thread.");
+            executorService.execute(defaultGetRecordsCacheDaemon);
+        }
+        started = true;
+    }
+
+    @Override
+    public ProcessRecordsInput getNextResult() {
+        if (executorService.isShutdown()) {
+            throw new IllegalStateException("Shutdown has been called on the cache, can't accept new requests.");
+        }
+        
+        if (!started) {
+            throw new IllegalStateException("Cache has not been initialized, make sure to call start.");
+        }
+        ProcessRecordsInput result = null;
+        try {
+            result = getRecordsResultQueue.take().withCacheExitTime(Instant.now());
+            prefetchCounters.removed(result);
+        } catch (InterruptedException e) {
+            log.error("Interrupted while getting records from the cache", e);
+        }
+        return result;
+    }
+
+    @Override
+    public GetRecordsRetrievalStrategy getGetRecordsRetrievalStrategy() {
+        return getRecordsRetrievalStrategy;
+    }
+
+    @Override
+    public void shutdown() {
+        defaultGetRecordsCacheDaemon.isShutdown = true;
+        executorService.shutdownNow();
+        started = false;
+    }
+
+    private class DefaultGetRecordsCacheDaemon implements Runnable {
+        volatile boolean isShutdown = false;
+        
+        @Override
+        public void run() {
+            while (!isShutdown) {
+                if (Thread.currentThread().isInterrupted()) {
+                    log.warn("Prefetch thread was interrupted.");
+                    break;
+                }
+                MetricsHelper.startScope(metricsFactory, operation);
+                if (prefetchCounters.shouldGetNewRecords()) {
+                    try {
+                        sleepBeforeNextCall();
+                        GetRecordsResult getRecordsResult = getRecordsRetrievalStrategy.getRecords(maxRecordsPerCall);
+                        lastSuccessfulCall = Instant.now();
+                        ProcessRecordsInput processRecordsInput = new ProcessRecordsInput()
+                                .withRecords(getRecordsResult.getRecords())
+                                .withMillisBehindLatest(getRecordsResult.getMillisBehindLatest())
+                                .withCacheEntryTime(lastSuccessfulCall);
+                        getRecordsResultQueue.put(processRecordsInput);
+                        prefetchCounters.added(processRecordsInput);
+                    } catch (InterruptedException e) {
+                        log.info("Thread was interrupted, indicating shutdown was called on the cache.");
+                    } catch (ExpiredIteratorException e) {
+                        log.info(String.format("ShardId %s: getRecords threw ExpiredIteratorException - restarting"
+                                + " after greatest seqNum passed to customer", shardId), e);
+                        
+                        MetricsHelper.getMetricsScope().addData(EXPIRED_ITERATOR_METRIC, 1, StandardUnit.Count,
+                                MetricsLevel.SUMMARY);
+                        
+                        dataFetcher.restartIterator();
+                    } catch (SdkClientException e) {
+                        log.error("Exception thrown while fetching records from Kinesis", e);
+                    } catch (Throwable e) {
+                        log.error("Unexpected exception was thrown. This could probably be an issue or a bug." +
+                                " Please search for the exception/error online to check what is going on. If the " +
+                                "issue persists or is a recurring problem, feel free to open an issue on, " +
+                                "https://github.com/awslabs/amazon-kinesis-client.", e);
+                    } finally {
+                        MetricsHelper.endScope();
+                    }
+                } else {
+                    //
+                    // Consumer isn't ready to receive new records will allow prefetch counters to pause
+                    //
+                    try {
+                        prefetchCounters.waitForConsumer();
+                    } catch (InterruptedException ie) {
+                        log.info("Thread was interrupted while waiting for the consumer.  " +
+                                "Shutdown has probably been started");
+                    }
+                }
+            }
+            callShutdownOnStrategy();
+        }
+        
+        private void callShutdownOnStrategy() {
+            if (!getRecordsRetrievalStrategy.isShutdown()) {
+                getRecordsRetrievalStrategy.shutdown();
+            }
+        }
+        
+        private void sleepBeforeNextCall() throws InterruptedException {
+            if (lastSuccessfulCall == null) {
+                return;
+            }
+            long timeSinceLastCall = Duration.between(lastSuccessfulCall, Instant.now()).abs().toMillis();
+            if (timeSinceLastCall < idleMillisBetweenCalls) {
+                Thread.sleep(idleMillisBetweenCalls - timeSinceLastCall);
+            }
+        }
+    }
+
+    private class PrefetchCounters {
+        private long size = 0;
+        private long byteSize = 0;
+
+        public synchronized void added(final ProcessRecordsInput result) {
+            size += getSize(result);
+            byteSize += getByteSize(result);
+        }
+
+        public synchronized void removed(final ProcessRecordsInput result) {
+            size -= getSize(result);
+            byteSize -= getByteSize(result);
+            this.notifyAll();
+        }
+
+        private long getSize(final ProcessRecordsInput result) {
+            return result.getRecords().size();
+        }
+
+        private long getByteSize(final ProcessRecordsInput result) {
+            return result.getRecords().stream().mapToLong(record -> record.getData().array().length).sum();
+        }
+
+        public synchronized void waitForConsumer() throws InterruptedException {
+            if (!shouldGetNewRecords()) {
+                log.debug("Queue is full waiting for consumer for " + idleMillisBetweenCalls + " ms");
+                this.wait(idleMillisBetweenCalls);
+            }
+        }
+        
+        public synchronized boolean shouldGetNewRecords() {
+            if (log.isDebugEnabled()) {
+                log.debug("Current Prefetch Counter States: " + this.toString());
+            }
+            return size < maxRecordsCount && byteSize < maxByteSize;
+        }
+
+        @Override
+        public String toString() {
+            return String.format("{ Requests: %d, Records: %d, Bytes: %d }", getRecordsResultQueue.size(), size,
+                    byteSize);
+        }
+    }
+
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PreparedCheckpointer.java

@@ -1,3 +1,17 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
 import com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException;

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ProcessTask.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
@@ -62,6 +62,8 @@ class ProcessTask implements ITask {
     private final Shard shard;
     private final ThrottlingReporter throttlingReporter;
 
+    private final GetRecordsCache getRecordsCache;
+
     /**
      * @param shardInfo
      *            contains information about the shard
@@ -75,13 +77,17 @@ class ProcessTask implements ITask {
      *            Kinesis data fetcher (used to fetch records from Kinesis)
      * @param backoffTimeMillis
      *            backoff time when catching exceptions
+     * @param getRecordsCache
+     *            The retrieval strategy for fetching records from kinesis
      */
     public ProcessTask(ShardInfo shardInfo, StreamConfig streamConfig, IRecordProcessor recordProcessor,
-            RecordProcessorCheckpointer recordProcessorCheckpointer, KinesisDataFetcher dataFetcher,
+                       RecordProcessorCheckpointer recordProcessorCheckpointer, KinesisDataFetcher dataFetcher,
-            long backoffTimeMillis, boolean skipShardSyncAtWorkerInitializationIfLeasesExist) {
+                       long backoffTimeMillis, boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
+                       GetRecordsCache getRecordsCache) {
         this(shardInfo, streamConfig, recordProcessor, recordProcessorCheckpointer, dataFetcher, backoffTimeMillis,
                 skipShardSyncAtWorkerInitializationIfLeasesExist,
-                new ThrottlingReporter(MAX_CONSECUTIVE_THROTTLES, shardInfo.getShardId()));
+                new ThrottlingReporter(MAX_CONSECUTIVE_THROTTLES, shardInfo.getShardId()),
+                getRecordsCache);
     }
 
     /**
@@ -101,9 +107,9 @@ class ProcessTask implements ITask {
      *            determines how throttling events should be reported in the log.
      */
     public ProcessTask(ShardInfo shardInfo, StreamConfig streamConfig, IRecordProcessor recordProcessor,
-            RecordProcessorCheckpointer recordProcessorCheckpointer, KinesisDataFetcher dataFetcher,
+                       RecordProcessorCheckpointer recordProcessorCheckpointer, KinesisDataFetcher dataFetcher,
-            long backoffTimeMillis, boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
+                       long backoffTimeMillis, boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
-            ThrottlingReporter throttlingReporter) {
+                       ThrottlingReporter throttlingReporter, GetRecordsCache getRecordsCache) {
         super();
         this.shardInfo = shardInfo;
         this.recordProcessor = recordProcessor;
@@ -113,6 +119,7 @@ class ProcessTask implements ITask {
         this.backoffTimeMillis = backoffTimeMillis;
         this.throttlingReporter = throttlingReporter;
         IKinesisProxy kinesisProxy = this.streamConfig.getStreamProxy();
+        this.getRecordsCache = getRecordsCache;
         // If skipShardSyncAtWorkerInitializationIfLeasesExist is set, we will not get the shard for
         // this ProcessTask. In this case, duplicate KPL user records in the event of resharding will
         // not be dropped during deaggregation of Amazon Kinesis records. This is only applicable if
@@ -141,7 +148,6 @@ class ProcessTask implements ITask {
         scope.addDimension(MetricsHelper.SHARD_ID_DIMENSION_NAME, shardInfo.getShardId());
         scope.addData(RECORDS_PROCESSED_METRIC, 0, StandardUnit.Count, MetricsLevel.SUMMARY);
         scope.addData(DATA_BYTES_PROCESSED_METRIC, 0, StandardUnit.Bytes, MetricsLevel.SUMMARY);
-
         Exception exception = null;
 
         try {
@@ -150,10 +156,10 @@ class ProcessTask implements ITask {
                 return new TaskResult(null, true);
             }
 
-            final GetRecordsResult getRecordsResult = getRecordsResult();
+            final ProcessRecordsInput processRecordsInput = getRecordsResult();
             throttlingReporter.success();
-            List<Record> records = getRecordsResult.getRecords();
+            List<Record> records = processRecordsInput.getRecords();
-            
+
             if (!records.isEmpty()) {
                 scope.addData(RECORDS_PROCESSED_METRIC, records.size(), StandardUnit.Count, MetricsLevel.SUMMARY);
             } else {
@@ -167,7 +173,7 @@ class ProcessTask implements ITask {
                             recordProcessorCheckpointer.getLargestPermittedCheckpointValue()));
 
             if (shouldCallProcessRecords(records)) {
-                callProcessRecords(getRecordsResult, records);
+                callProcessRecords(processRecordsInput, records);
             }
         } catch (ProvisionedThroughputExceededException pte) {
             throttlingReporter.throttled();
@@ -197,18 +203,18 @@ class ProcessTask implements ITask {
 
     /**
      * Dispatches a batch of records to the record processor, and handles any fallout from that.
-     * 
+     *
-     * @param getRecordsResult
+     * @param input
      *            the result of the last call to Kinesis
      * @param records
      *            the records to be dispatched. It's possible the records have been adjusted by KPL deaggregation.
      */
-    private void callProcessRecords(GetRecordsResult getRecordsResult, List<Record> records) {
+    private void callProcessRecords(ProcessRecordsInput input, List<Record> records) {
         LOG.debug("Calling application processRecords() with " + records.size() + " records from "
                 + shardInfo.getShardId());
         final ProcessRecordsInput processRecordsInput = new ProcessRecordsInput().withRecords(records)
                 .withCheckpointer(recordProcessorCheckpointer)
-                .withMillisBehindLatest(getRecordsResult.getMillisBehindLatest());
+                .withMillisBehindLatest(input.getMillisBehindLatest());
 
         final long recordProcessorStartTimeMillis = System.currentTimeMillis();
         try {
@@ -225,7 +231,7 @@ class ProcessTask implements ITask {
 
     /**
      * Whether we should call process records or not
-     * 
+     *
      * @param records
      *            the records returned from the call to Kinesis, and/or deaggregation
      * @return true if the set of records should be dispatched to the record process, false if they should not.
@@ -236,7 +242,7 @@ class ProcessTask implements ITask {
 
     /**
      * Determines whether to deaggregate the given records, and if they are KPL records dispatches them to deaggregation
-     * 
+     *
      * @param records
      *            the records to deaggregate is deaggregation is required.
      * @return returns either the deaggregated records, or the original records
@@ -259,7 +265,7 @@ class ProcessTask implements ITask {
 
     /**
      * Emits metrics, and sleeps if there are no records available
-     * 
+     *
      * @param startTimeMillis
      *            the time when the task started
      */
@@ -296,8 +302,8 @@ class ProcessTask implements ITask {
      * @return the largest extended sequence number among the retained records
      */
     private ExtendedSequenceNumber filterAndGetMaxExtendedSequenceNumber(IMetricsScope scope, List<Record> records,
-            final ExtendedSequenceNumber lastCheckpointValue,
+                                                                         final ExtendedSequenceNumber lastCheckpointValue,
-            final ExtendedSequenceNumber lastLargestPermittedCheckpointValue) {
+                                                                         final ExtendedSequenceNumber lastLargestPermittedCheckpointValue) {
         ExtendedSequenceNumber largestExtendedSequenceNumber = lastLargestPermittedCheckpointValue;
         ListIterator<Record> recordIterator = records.listIterator();
         while (recordIterator.hasNext()) {
@@ -331,7 +337,7 @@ class ProcessTask implements ITask {
      *
      * @return list of data records from Kinesis
      */
-    private GetRecordsResult getRecordsResult() {
+    private ProcessRecordsInput getRecordsResult() {
         try {
             return getRecordsResultAndRecordMillisBehindLatest();
         } catch (ExpiredIteratorException e) {
@@ -367,22 +373,17 @@ class ProcessTask implements ITask {
      *
      * @return list of data records from Kinesis
      */
-    private GetRecordsResult getRecordsResultAndRecordMillisBehindLatest() {
+    private ProcessRecordsInput getRecordsResultAndRecordMillisBehindLatest() {
-        final GetRecordsResult getRecordsResult = dataFetcher.getRecords(streamConfig.getMaxRecords());
+        final ProcessRecordsInput processRecordsInput = getRecordsCache.getNextResult();
 
-        if (getRecordsResult == null) {
+        if (processRecordsInput.getMillisBehindLatest() != null) {
-            // Stream no longer exists
-            return new GetRecordsResult().withRecords(Collections.<Record>emptyList());
-        }
-
-        if (getRecordsResult.getMillisBehindLatest() != null) {
             MetricsHelper.getMetricsScope().addData(MILLIS_BEHIND_LATEST_METRIC,
-                    getRecordsResult.getMillisBehindLatest(),
+                    processRecordsInput.getMillisBehindLatest(),
                     StandardUnit.Milliseconds,
                     MetricsLevel.SUMMARY);
         }
 
-        return getRecordsResult;
+        return processRecordsInput;
     }
 
-}
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/RecordProcessorCheckpointer.java

@@ -1,5 +1,5 @@
 /*
- * Copyright 2012-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
  * Licensed under the Amazon Software License (the "License").
  * You may not use this file except in compliance with the License.
@@ -14,6 +14,9 @@
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
+import com.amazonaws.services.kinesis.metrics.impl.ThreadSafeMetricsDelegatingScope;
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 
@@ -50,6 +53,8 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
     private SequenceNumberValidator sequenceNumberValidator;
 
     private ExtendedSequenceNumber sequenceNumberAtShardEnd;
+    
+    private IMetricsFactory metricsFactory;
 
     /**
      * Only has package level access, since only the Amazon Kinesis Client Library should be creating these.
@@ -59,10 +64,12 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
      */
     RecordProcessorCheckpointer(ShardInfo shardInfo,
             ICheckpoint checkpoint,
-            SequenceNumberValidator validator) {
+            SequenceNumberValidator validator,
+            IMetricsFactory metricsFactory) {
         this.shardInfo = shardInfo;
         this.checkpoint = checkpoint;
         this.sequenceNumberValidator = validator;
+        this.metricsFactory = metricsFactory;
     }
 
     /**
@@ -283,21 +290,33 @@ class RecordProcessorCheckpointer implements IRecordProcessorCheckpointer {
             // just checkpoint at SHARD_END
             checkpointToRecord = ExtendedSequenceNumber.SHARD_END;
         }
+        
+        boolean unsetMetrics = false;
         // Don't checkpoint a value we already successfully checkpointed
-        if (extendedSequenceNumber != null && !extendedSequenceNumber.equals(lastCheckpointValue)) {
+        try {
-            try {
+            if (!MetricsHelper.isMetricsScopePresent()) {
-                if (LOG.isDebugEnabled()) {
+                MetricsHelper.setMetricsScope(new ThreadSafeMetricsDelegatingScope(metricsFactory.createMetrics()));
-                    LOG.debug("Setting " + shardInfo.getShardId() + ", token " + shardInfo.getConcurrencyToken()
+                unsetMetrics = true;
-                            + " checkpoint to " + checkpointToRecord);
+            }
+            if (extendedSequenceNumber != null && !extendedSequenceNumber.equals(lastCheckpointValue)) {
+                try {
+                    if (LOG.isDebugEnabled()) {
+                        LOG.debug("Setting " + shardInfo.getShardId() + ", token " + shardInfo.getConcurrencyToken()
+                                + " checkpoint to " + checkpointToRecord);
+                    }
+                    checkpoint.setCheckpoint(shardInfo.getShardId(), checkpointToRecord, shardInfo.getConcurrencyToken());
+                    lastCheckpointValue = checkpointToRecord;
+                } catch (ThrottlingException | ShutdownException | InvalidStateException
+                        | KinesisClientLibDependencyException e) {
+                    throw e;
+                } catch (KinesisClientLibException e) {
+                    LOG.warn("Caught exception setting checkpoint.", e);
+                    throw new KinesisClientLibDependencyException("Caught exception while checkpointing", e);
                 }
-                checkpoint.setCheckpoint(shardInfo.getShardId(), checkpointToRecord, shardInfo.getConcurrencyToken());
+            }
-                lastCheckpointValue = checkpointToRecord;
+        } finally {
-            } catch (ThrottlingException | ShutdownException | InvalidStateException
+            if (unsetMetrics) {
-                    | KinesisClientLibDependencyException e) {
+                MetricsHelper.unsetMetricsScope();
-                throw e;
-            } catch (KinesisClientLibException e) {
-                LOG.warn("Caught exception setting checkpoint.", e);
-                throw new KinesisClientLibDependencyException("Caught exception while checkpointing", e);
             }
         }
     }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/RecordsFetcherFactory.java

@@ -0,0 +1,76 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+
+/**
+ * This factory is used to create the records fetcher to retrieve data from Kinesis for a given shard.
+ */
+public interface RecordsFetcherFactory {
+    /**
+     * Returns a GetRecordsCache to be used for retrieving records for a given shard.
+     *
+     * @param getRecordsRetrievalStrategy GetRecordsRetrievalStrategy to be used with the GetRecordsCache
+     * @param shardId ShardId of the shard that the fetcher will retrieve records for
+     * @param metricsFactory MetricsFactory used to create metricScope
+     * @param maxRecords Max number of records to be returned in a single get call
+     *                
+     * @return GetRecordsCache used to get records from Kinesis.
+     */
+    GetRecordsCache createRecordsFetcher(GetRecordsRetrievalStrategy getRecordsRetrievalStrategy, String shardId,
+                                         IMetricsFactory metricsFactory, int maxRecords);
+
+    /**
+     * Sets the maximum number of ProcessRecordsInput objects the GetRecordsCache can hold, before further requests are
+     * blocked.
+     * 
+     * @param maxPendingProcessRecordsInput The maximum number of ProcessRecordsInput objects that the cache will accept
+     *                                     before blocking.
+     */
+    void setMaxPendingProcessRecordsInput(int maxPendingProcessRecordsInput);
+
+    /**
+     * Sets the max byte size for the GetRecordsCache, before further requests are blocked. The byte size of the cache
+     * is the sum of byte size of all the ProcessRecordsInput objects in the cache at any point of time.
+     * 
+     * @param maxByteSize The maximum byte size for the cache before blocking.
+     */
+    void setMaxByteSize(int maxByteSize);
+
+    /**
+     * Sets the max number of records for the GetRecordsCache can hold, before further requests are blocked. The records
+     * count is the sum of all records present in across all the ProcessRecordsInput objects in the cache at any point
+     * of time.
+     * 
+     * @param maxRecordsCount The mximum number of records in the cache before blocking.
+     */
+    void setMaxRecordsCount(int maxRecordsCount);
+
+    /**
+     * Sets the dataFetchingStrategy to determine the type of GetRecordsCache to be used.
+     * 
+     * @param dataFetchingStrategy Fetching strategy to be used
+     */
+    void setDataFetchingStrategy(DataFetchingStrategy dataFetchingStrategy);
+
+    /**
+     * Sets the maximum idle time between two get calls.
+     * 
+     * @param idleMillisBetweenCalls Sleep millis between calls.
+     */
+    void setIdleMillisBetweenCalls(long idleMillisBetweenCalls);
+
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardConsumer.java

@@ -1,20 +1,21 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
 
+import java.util.Optional;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Future;
 import java.util.concurrent.RejectedExecutionException;
@@ -30,6 +31,8 @@ import com.amazonaws.services.kinesis.leases.interfaces.ILeaseManager;
 import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
 import com.google.common.annotations.VisibleForTesting;
 
+import lombok.Getter;
+
 /**
  * Responsible for consuming data records of a (specified) shard.
  * The instance should be shutdown when we lose the primary responsibility for a shard.
@@ -41,6 +44,7 @@ class ShardConsumer {
 
     private final StreamConfig streamConfig;
     private final IRecordProcessor recordProcessor;
+    private final KinesisClientLibConfiguration config;
     private final RecordProcessorCheckpointer recordProcessorCheckpointer;
     private final ExecutorService executorService;
     private final ShardInfo shardInfo;
@@ -57,6 +61,20 @@ class ShardConsumer {
     private ITask currentTask;
     private long currentTaskSubmitTime;
     private Future<TaskResult> future;
+    
+    @Getter
+    private final GetRecordsCache getRecordsCache;
+
+    private static final GetRecordsRetrievalStrategy makeStrategy(KinesisDataFetcher dataFetcher,
+                                                                  Optional<Integer> retryGetRecordsInSeconds,
+                                                                  Optional<Integer> maxGetRecordsThreadPool,
+                                                                  ShardInfo shardInfo) {
+        Optional<GetRecordsRetrievalStrategy> getRecordsRetrievalStrategy = retryGetRecordsInSeconds.flatMap(retry ->
+                maxGetRecordsThreadPool.map(max ->
+                        new AsynchronousGetRecordsRetrievalStrategy(dataFetcher, retry, max, shardInfo.getShardId())));
+
+        return getRecordsRetrievalStrategy.orElse(new SynchronousGetRecordsRetrievalStrategy(dataFetcher));
+    }
 
     /*
      * Tracks current state. It is only updated via the consumeStream/shutdown APIs. Therefore we don't do
@@ -75,6 +93,7 @@ class ShardConsumer {
      * @param streamConfig Stream configuration to use
      * @param checkpoint Checkpoint tracker
      * @param recordProcessor Record processor used to process the data records for the shard
+     * @param config Kinesis library configuration
      * @param leaseManager Used to create leases for new shards
      * @param parentShardPollIntervalMillis Wait for this long if parent shards are not done (or we get an exception)
      * @param executorService ExecutorService used to execute process tasks for this shard
@@ -83,34 +102,141 @@ class ShardConsumer {
      */
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 10 LINES
     ShardConsumer(ShardInfo shardInfo,
-            StreamConfig streamConfig,
+                  StreamConfig streamConfig,
-            ICheckpoint checkpoint,
+                  ICheckpoint checkpoint,
-            IRecordProcessor recordProcessor,
+                  IRecordProcessor recordProcessor,
-            ILeaseManager<KinesisClientLease> leaseManager,
+                  ILeaseManager<KinesisClientLease> leaseManager,
-            long parentShardPollIntervalMillis,
+                  long parentShardPollIntervalMillis,
-            boolean cleanupLeasesOfCompletedShards,
+                  boolean cleanupLeasesOfCompletedShards,
-            ExecutorService executorService,
+                  ExecutorService executorService,
-            IMetricsFactory metricsFactory,
+                  IMetricsFactory metricsFactory,
-            long backoffTimeMillis,
+                  long backoffTimeMillis,
-            boolean skipShardSyncAtWorkerInitializationIfLeasesExist) {
+                  boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
-        this.streamConfig = streamConfig;
+                  KinesisClientLibConfiguration config) {
-        this.recordProcessor = recordProcessor;
+        this(shardInfo,
-        this.executorService = executorService;
+                streamConfig,
-        this.shardInfo = shardInfo;
+                checkpoint,
-        this.checkpoint = checkpoint;
+                recordProcessor,
-        this.recordProcessorCheckpointer =
+                leaseManager,
-                new RecordProcessorCheckpointer(shardInfo,
+                parentShardPollIntervalMillis,
+                cleanupLeasesOfCompletedShards,
+                executorService,
+                metricsFactory,
+                backoffTimeMillis,
+                skipShardSyncAtWorkerInitializationIfLeasesExist,
+                Optional.empty(),
+                Optional.empty(),
+                config);
+    }
+
+    /**
+     * @param shardInfo Shard information
+     * @param streamConfig Stream configuration to use
+     * @param checkpoint Checkpoint tracker
+     * @param recordProcessor Record processor used to process the data records for the shard
+     * @param leaseManager Used to create leases for new shards
+     * @param parentShardPollIntervalMillis Wait for this long if parent shards are not done (or we get an exception)
+     * @param executorService ExecutorService used to execute process tasks for this shard
+     * @param metricsFactory IMetricsFactory used to construct IMetricsScopes for this shard
+     * @param backoffTimeMillis backoff interval when we encounter exceptions
+     * @param retryGetRecordsInSeconds time in seconds to wait before the worker retries to get a record.
+     * @param maxGetRecordsThreadPool max number of threads in the getRecords thread pool.
+     * @param config Kinesis library configuration
+     */
+    // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 10 LINES
+    ShardConsumer(ShardInfo shardInfo,
+                  StreamConfig streamConfig,
+                  ICheckpoint checkpoint,
+                  IRecordProcessor recordProcessor,
+                  ILeaseManager<KinesisClientLease> leaseManager,
+                  long parentShardPollIntervalMillis,
+                  boolean cleanupLeasesOfCompletedShards,
+                  ExecutorService executorService,
+                  IMetricsFactory metricsFactory,
+                  long backoffTimeMillis,
+                  boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
+                  Optional<Integer> retryGetRecordsInSeconds,
+                  Optional<Integer> maxGetRecordsThreadPool,
+                  KinesisClientLibConfiguration config) {
+        
+        this(
+                shardInfo,
+                streamConfig,
+                checkpoint,
+                recordProcessor,
+                new RecordProcessorCheckpointer(
+                        shardInfo,
                         checkpoint,
-                        new SequenceNumberValidator(streamConfig.getStreamProxy(),
+                        new SequenceNumberValidator(
+                                streamConfig.getStreamProxy(),
                                 shardInfo.getShardId(),
-                                streamConfig.shouldValidateSequenceNumberBeforeCheckpointing()));
+                                streamConfig.shouldValidateSequenceNumberBeforeCheckpointing()),
-        this.dataFetcher = new KinesisDataFetcher(streamConfig.getStreamProxy(), shardInfo);
+                        metricsFactory),
+                leaseManager,
+                parentShardPollIntervalMillis,
+                cleanupLeasesOfCompletedShards,
+                executorService,
+                metricsFactory,
+                backoffTimeMillis,
+                skipShardSyncAtWorkerInitializationIfLeasesExist,
+                new KinesisDataFetcher(streamConfig.getStreamProxy(), shardInfo),
+                retryGetRecordsInSeconds,
+                maxGetRecordsThreadPool,
+                config
+        );
+    }
+
+    /**
+     * @param shardInfo Shard information
+     * @param streamConfig Stream Config to use
+     * @param checkpoint Checkpoint tracker
+     * @param recordProcessor Record processor used to process the data records for the shard
+     * @param recordProcessorCheckpointer RecordProcessorCheckpointer to use to checkpoint progress
+     * @param leaseManager Used to create leases for new shards
+     * @param parentShardPollIntervalMillis Wait for this long if parent shards are not done (or we get an exception)
+     * @param cleanupLeasesOfCompletedShards  clean up the leases of completed shards
+     * @param executorService ExecutorService used to execute process tasks for this shard
+     * @param metricsFactory IMetricsFactory used to construct IMetricsScopes for this shard
+     * @param backoffTimeMillis backoff interval when we encounter exceptions
+     * @param skipShardSyncAtWorkerInitializationIfLeasesExist Skip sync at init if lease exists
+     * @param kinesisDataFetcher KinesisDataFetcher to fetch data from Kinesis streams.
+     * @param retryGetRecordsInSeconds time in seconds to wait before the worker retries to get a record
+     * @param maxGetRecordsThreadPool max number of threads in the getRecords thread pool
+     * @param config Kinesis library configuration
+     */
+    ShardConsumer(ShardInfo shardInfo,
+                  StreamConfig streamConfig,
+                  ICheckpoint checkpoint,
+                  IRecordProcessor recordProcessor,
+                  RecordProcessorCheckpointer recordProcessorCheckpointer,
+                  ILeaseManager<KinesisClientLease> leaseManager,
+                  long parentShardPollIntervalMillis,
+                  boolean cleanupLeasesOfCompletedShards,
+                  ExecutorService executorService,
+                  IMetricsFactory metricsFactory,
+                  long backoffTimeMillis,
+                  boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
+                  KinesisDataFetcher kinesisDataFetcher,
+                  Optional<Integer> retryGetRecordsInSeconds,
+                  Optional<Integer> maxGetRecordsThreadPool,
+                  KinesisClientLibConfiguration config) {
+        this.shardInfo = shardInfo;
+        this.streamConfig = streamConfig;
+        this.checkpoint = checkpoint;
+        this.recordProcessor = recordProcessor;
+        this.recordProcessorCheckpointer = recordProcessorCheckpointer;
         this.leaseManager = leaseManager;
-        this.metricsFactory = metricsFactory;
         this.parentShardPollIntervalMillis = parentShardPollIntervalMillis;
         this.cleanupLeasesOfCompletedShards = cleanupLeasesOfCompletedShards;
+        this.executorService = executorService;
+        this.metricsFactory = metricsFactory;
         this.taskBackoffTimeMillis = backoffTimeMillis;
         this.skipShardSyncAtWorkerInitializationIfLeasesExist = skipShardSyncAtWorkerInitializationIfLeasesExist;
+        this.config = config;
+        this.dataFetcher = kinesisDataFetcher;
+        this.getRecordsCache = config.getRecordsFetcherFactory().createRecordsFetcher(
+                makeStrategy(this.dataFetcher, retryGetRecordsInSeconds, maxGetRecordsThreadPool, this.shardInfo),
+                this.getShardInfo().getShardId(), this.metricsFactory, this.config.getMaxRecords());
     }
 
     /**
@@ -158,11 +284,17 @@ class ShardConsumer {
                 }
             }
         } else {
+            final long timeElapsed = System.currentTimeMillis() - currentTaskSubmitTime;
+            final String commonMessage = String.format("Previous %s task still pending for shard %s since %d ms ago. ",
+                    currentTask.getTaskType(), shardInfo.getShardId(), timeElapsed);
             if (LOG.isDebugEnabled()) {
-                LOG.debug("Previous " + currentTask.getTaskType() + " task still pending for shard "
+                LOG.debug(commonMessage + "Not submitting new task.");
-                        + shardInfo.getShardId() + " since " + (System.currentTimeMillis() - currentTaskSubmitTime)
-                        + " ms ago" + ".  Not submitting new task.");
             }
+            config.getLogWarningForTaskAfterMillis().ifPresent(value -> {
+                if (timeElapsed > value) {
+                    LOG.warn(commonMessage);
+                }
+            });
         }
 
         return submittedNewTask;
@@ -281,7 +413,7 @@ class ShardConsumer {
         if (taskOutcome == TaskOutcome.END_OF_SHARD) {
             markForShutdown(ShutdownReason.TERMINATE);
         }
-        if (isShutdownRequested()) {
+        if (isShutdownRequested() && taskOutcome != TaskOutcome.FAILURE) {
             currentState = currentState.shutdownTransition(shutdownReason);
         } else if (taskOutcome == TaskOutcome.SUCCESSFUL) {
             if (currentState.getTaskType() == currentTask.getTaskType()) {
@@ -353,6 +485,10 @@ class ShardConsumer {
         return cleanupLeasesOfCompletedShards;
     }
 
+    boolean isIgnoreUnexpectedChildShards() {
+        return config.shouldIgnoreUnexpectedChildShards();
+    }
+
     long getTaskBackoffTimeMillis() {
         return taskBackoffTimeMillis;
     }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardInfo.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardSyncTask.java

@@ -35,6 +35,7 @@ class ShardSyncTask implements ITask {
     private final ILeaseManager<KinesisClientLease> leaseManager;
     private InitialPositionInStreamExtended initialPosition;
     private final boolean cleanupLeasesUponShardCompletion;
+    private final boolean ignoreUnexpectedChildShards;
     private final long shardSyncTaskIdleTimeMillis;
     private final TaskType taskType = TaskType.SHARDSYNC;
 
@@ -49,11 +50,13 @@ class ShardSyncTask implements ITask {
             ILeaseManager<KinesisClientLease> leaseManager,
             InitialPositionInStreamExtended initialPositionInStream,
             boolean cleanupLeasesUponShardCompletion,
+            boolean ignoreUnexpectedChildShards,
             long shardSyncTaskIdleTimeMillis) {
         this.kinesisProxy = kinesisProxy;
         this.leaseManager = leaseManager;
         this.initialPosition = initialPositionInStream;
         this.cleanupLeasesUponShardCompletion = cleanupLeasesUponShardCompletion;
+        this.ignoreUnexpectedChildShards = ignoreUnexpectedChildShards;
         this.shardSyncTaskIdleTimeMillis = shardSyncTaskIdleTimeMillis;
     }
 
@@ -68,7 +71,8 @@ class ShardSyncTask implements ITask {
             ShardSyncer.checkAndCreateLeasesForNewShards(kinesisProxy,
                     leaseManager,
                     initialPosition,
-                    cleanupLeasesUponShardCompletion);
+                    cleanupLeasesUponShardCompletion,
+                    ignoreUnexpectedChildShards);
             if (shardSyncTaskIdleTimeMillis > 0) {
                 Thread.sleep(shardSyncTaskIdleTimeMillis);
             }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardSyncTaskManager.java

@@ -44,6 +44,7 @@ class ShardSyncTaskManager {
     private final ExecutorService executorService;
     private final InitialPositionInStreamExtended initialPositionInStream;
     private boolean cleanupLeasesUponShardCompletion;
+    private boolean ignoreUnexpectedChildShards;
     private final long shardSyncIdleTimeMillis;
 
 
@@ -55,6 +56,7 @@ class ShardSyncTaskManager {
      * @param initialPositionInStream Initial position in stream
      * @param cleanupLeasesUponShardCompletion Clean up leases for shards that we've finished processing (don't wait
      *        until they expire)
+     * @param ignoreUnexpectedChildShards Ignore child shards with open parents
      * @param shardSyncIdleTimeMillis Time between tasks to sync leases and Kinesis shards
      * @param metricsFactory Metrics factory
      * @param executorService ExecutorService to execute the shard sync tasks
@@ -63,6 +65,7 @@ class ShardSyncTaskManager {
             final ILeaseManager<KinesisClientLease> leaseManager,
             final InitialPositionInStreamExtended initialPositionInStream,
             final boolean cleanupLeasesUponShardCompletion,
+            final boolean ignoreUnexpectedChildShards,
             final long shardSyncIdleTimeMillis,
             final IMetricsFactory metricsFactory,
             ExecutorService executorService) {
@@ -70,6 +73,7 @@ class ShardSyncTaskManager {
         this.leaseManager = leaseManager;
         this.metricsFactory = metricsFactory;
         this.cleanupLeasesUponShardCompletion = cleanupLeasesUponShardCompletion;
+        this.ignoreUnexpectedChildShards = ignoreUnexpectedChildShards;
         this.shardSyncIdleTimeMillis = shardSyncIdleTimeMillis;
         this.executorService = executorService;
         this.initialPositionInStream = initialPositionInStream;
@@ -99,6 +103,7 @@ class ShardSyncTaskManager {
                             leaseManager,
                             initialPositionInStream,
                             cleanupLeasesUponShardCompletion,
+                            ignoreUnexpectedChildShards,
                             shardSyncIdleTimeMillis), metricsFactory);
             future = executorService.submit(currentTask);
             submittedNewTask = true;

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardSyncer.java

@@ -28,6 +28,7 @@ import java.util.Set;
 
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
+import org.apache.commons.lang.StringUtils;
 
 import com.amazonaws.services.kinesis.clientlibrary.exceptions.internal.KinesisClientLibIOException;
 import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
@@ -60,9 +61,11 @@ class ShardSyncer {
     static synchronized void bootstrapShardLeases(IKinesisProxy kinesisProxy,
             ILeaseManager<KinesisClientLease> leaseManager,
             InitialPositionInStreamExtended initialPositionInStream,
-            boolean cleanupLeasesOfCompletedShards)
+            boolean cleanupLeasesOfCompletedShards,
+            boolean ignoreUnexpectedChildShards)
         throws DependencyException, InvalidStateException, ProvisionedThroughputException, KinesisClientLibIOException {
-        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream, cleanupLeasesOfCompletedShards);
+        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream, cleanupLeasesOfCompletedShards,
+                        ignoreUnexpectedChildShards);
     }
 
     /**
@@ -71,21 +74,28 @@ class ShardSyncer {
      * @param kinesisProxy
      * @param leaseManager
      * @param initialPositionInStream
-     * @param expectedClosedShardId If this is not null, we will assert that the shard list we get from Kinesis
+     * @param cleanupLeasesOfCompletedShards
-     *        shows this shard to be closed (e.g. parent shard must be closed after a reshard operation).
+     * @param ignoreUnexpectedChildShards
-     *        If it is open, we assume this is an race condition around a reshard event and throw
-     *        a KinesisClientLibIOException so client can backoff and retry later.
      * @throws DependencyException
      * @throws InvalidStateException
      * @throws ProvisionedThroughputException
      * @throws KinesisClientLibIOException
      */
+    static synchronized void checkAndCreateLeasesForNewShards(IKinesisProxy kinesisProxy,
+            ILeaseManager<KinesisClientLease> leaseManager,
+            InitialPositionInStreamExtended initialPositionInStream,
+            boolean cleanupLeasesOfCompletedShards,
+            boolean ignoreUnexpectedChildShards)
+        throws DependencyException, InvalidStateException, ProvisionedThroughputException, KinesisClientLibIOException {
+        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream, cleanupLeasesOfCompletedShards, ignoreUnexpectedChildShards);
+    }
+
     static synchronized void checkAndCreateLeasesForNewShards(IKinesisProxy kinesisProxy,
             ILeaseManager<KinesisClientLease> leaseManager,
             InitialPositionInStreamExtended initialPositionInStream,
             boolean cleanupLeasesOfCompletedShards)
         throws DependencyException, InvalidStateException, ProvisionedThroughputException, KinesisClientLibIOException {
-        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream, cleanupLeasesOfCompletedShards);
+        checkAndCreateLeasesForNewShards(kinesisProxy, leaseManager, initialPositionInStream, cleanupLeasesOfCompletedShards, false);
     }
 
     /**
@@ -93,11 +103,9 @@ class ShardSyncer {
      * 
      * @param kinesisProxy
      * @param leaseManager
-     * @param expectedClosedShardId If this is not null, we will assert that the shard list we get from Kinesis
+     * @param initialPosition
-     *        does not show this shard to be open (e.g. parent shard must be closed after a reshard operation).
+     * @param cleanupLeasesOfCompletedShards
-     *        If it is still open, we assume this is a race condition around a reshard event and
+     * @param ignoreUnexpectedChildShards
-     *        throw a KinesisClientLibIOException so client can backoff and retry later. If the shard doesn't exist in
-     *        Kinesis at all, we assume this is an old/expired shard and continue with the sync operation.
      * @throws DependencyException
      * @throws InvalidStateException
      * @throws ProvisionedThroughputException
@@ -107,18 +115,23 @@ class ShardSyncer {
     private static synchronized void syncShardLeases(IKinesisProxy kinesisProxy,
             ILeaseManager<KinesisClientLease> leaseManager,
             InitialPositionInStreamExtended initialPosition,
-            boolean cleanupLeasesOfCompletedShards)
+            boolean cleanupLeasesOfCompletedShards,
+            boolean ignoreUnexpectedChildShards)
         throws DependencyException, InvalidStateException, ProvisionedThroughputException, KinesisClientLibIOException {
         List<Shard> shards = getShardList(kinesisProxy);
         LOG.debug("Num shards: " + shards.size());
 
         Map<String, Shard> shardIdToShardMap = constructShardIdToShardMap(shards);
         Map<String, Set<String>> shardIdToChildShardIdsMap = constructShardIdToChildShardIdsMap(shardIdToShardMap);
-        assertAllParentShardsAreClosed(shardIdToChildShardIdsMap, shardIdToShardMap);
+        Set<String> inconsistentShardIds = findInconsistentShardIds(shardIdToChildShardIdsMap, shardIdToShardMap);
-        
+        if (!ignoreUnexpectedChildShards) {
+            assertAllParentShardsAreClosed(inconsistentShardIds);
+        }
+
         List<KinesisClientLease> currentLeases = leaseManager.listLeases();
-        
+
-        List<KinesisClientLease> newLeasesToCreate = determineNewLeasesToCreate(shards, currentLeases, initialPosition);
+        List<KinesisClientLease> newLeasesToCreate = determineNewLeasesToCreate(shards, currentLeases, initialPosition,
+                                                                                inconsistentShardIds);
         LOG.debug("Num new leases to create: " + newLeasesToCreate.size());
         for (KinesisClientLease lease : newLeasesToCreate) {
             long startTimeMillis = System.currentTimeMillis();
@@ -149,19 +162,37 @@ class ShardSyncer {
 
     /** Helper method to detect a race condition between fetching the shards via paginated DescribeStream calls
      * and a reshard operation.
-     * @param shardIdToChildShardIdsMap
+     * @param inconsistentShardIds
-     * @param shardIdToShardMap
      * @throws KinesisClientLibIOException
      */
-    private static void assertAllParentShardsAreClosed(Map<String, Set<String>> shardIdToChildShardIdsMap,
+    private static void assertAllParentShardsAreClosed(Set<String> inconsistentShardIds)
-            Map<String, Shard> shardIdToShardMap) throws KinesisClientLibIOException {
+        throws KinesisClientLibIOException {
+        if (!inconsistentShardIds.isEmpty()) {
+            String ids = StringUtils.join(inconsistentShardIds, ' ');
+            throw new KinesisClientLibIOException(String.format("%d open child shards (%s) are inconsistent. "
+                                                                + "This can happen due to a race condition between describeStream and a reshard operation.",
+                                                                inconsistentShardIds.size(), ids));
+        }
+    }
+
+    /**
+     * Helper method to construct the list of inconsistent shards, which are open shards with non-closed ancestor
+     * parent(s).
+     * @param shardIdToChildShardIdsMap
+     * @param shardIdToShardMap
+     * @return Set of inconsistent open shard ids for shards having open parents.
+     */
+    private static Set<String> findInconsistentShardIds(Map<String, Set<String>> shardIdToChildShardIdsMap,
+            Map<String, Shard> shardIdToShardMap) {
+        Set<String> result = new HashSet<String>();
         for (String parentShardId : shardIdToChildShardIdsMap.keySet()) {
             Shard parentShard = shardIdToShardMap.get(parentShardId);
             if ((parentShardId == null) || (parentShard.getSequenceNumberRange().getEndingSequenceNumber() == null)) {
-                throw new KinesisClientLibIOException("Parent shardId " + parentShardId + " is not closed. " 
+                Set<String> childShardIdsMap = shardIdToChildShardIdsMap.get(parentShardId);
-                        + "This can happen due to a race condition between describeStream and a reshard operation.");
+                result.addAll(childShardIdsMap);
             }
         }
+        return result;
     }
 
     /**
@@ -296,8 +327,8 @@ class ShardSyncer {
     /**
      * Determine new leases to create and their initial checkpoint.
      * Note: Package level access only for testing purposes.
-     * 
+     *
-     * For each open (no ending sequence number) shard that doesn't already have a lease,
+     * For each open (no ending sequence number) shard without open parents that doesn't already have a lease,
      * determine if it is a descendent of any shard which is or will be processed (e.g. for which a lease exists):
      * If so, set checkpoint of the shard to TrimHorizon and also create leases for ancestors if needed.
      * If not, set checkpoint of the shard to the initial position specified by the client.
@@ -315,27 +346,35 @@ class ShardSyncer {
      * 
      * For example:
      * Shard structure (each level depicts a stream segment):
-     * 0 1 2 3 4 5- shards till epoch 102
+     * 0 1 2 3 4   5   - shards till epoch 102
-     * \ / \ / | |
+     * \ / \ / |   |
-     * 6 7 4 5- shards from epoch 103 - 205
+     *  6   7  4   5   - shards from epoch 103 - 205
-     * \ / | /\
+     *   \ /   |  / \
-     * 8 4 9 10 - shards from epoch 206 (open - no ending sequenceNumber)
+     *    8    4 9  10 - shards from epoch 206 (open - no ending sequenceNumber)
      * Current leases: (3, 4, 5)
      * New leases to create: (2, 6, 7, 8, 9, 10)
      * 
      * The leases returned are sorted by the starting sequence number - following the same order
      * when persisting the leases in DynamoDB will ensure that we recover gracefully if we fail
      * before creating all the leases.
+     *
+     * If a shard has no existing lease, is open, and is a descendant of a parent which is still open, we ignore it
+     * here; this happens when the list of shards is inconsistent, which could be due to pagination delay for very
+     * high shard count streams (i.e., dynamodb streams for tables with thousands of partitions).  This can only
+     * currently happen here if ignoreUnexpectedChildShards was true in syncShardleases.
+     *
      * 
      * @param shards List of all shards in Kinesis (we'll create new leases based on this set)
      * @param currentLeases List of current leases
      * @param initialPosition One of LATEST, TRIM_HORIZON, or AT_TIMESTAMP. We'll start fetching records from that
      *        location in the shard (when an application starts up for the first time - and there are no checkpoints).
+     * @param inconsistentShardIds Set of child shard ids having open parents.
      * @return List of new leases to create sorted by starting sequenceNumber of the corresponding shard
      */
     static List<KinesisClientLease> determineNewLeasesToCreate(List<Shard> shards,
             List<KinesisClientLease> currentLeases,
-            InitialPositionInStreamExtended initialPosition) {
+            InitialPositionInStreamExtended initialPosition,
+            Set<String> inconsistentShardIds) {
         Map<String, KinesisClientLease> shardIdToNewLeaseMap = new HashMap<String, KinesisClientLease>();
         Map<String, Shard> shardIdToShardMapOfAllKinesisShards = constructShardIdToShardMap(shards);
 
@@ -354,6 +393,8 @@ class ShardSyncer {
             LOG.debug("Evaluating leases for open shard " + shardId + " and its ancestors.");
             if (shardIdsOfCurrentLeases.contains(shardId)) {
                 LOG.debug("Lease for shardId " + shardId + " already exists. Not creating a lease");
+            } else if (inconsistentShardIds.contains(shardId)) {
+                LOG.info("shardId " + shardId + " is an inconsistent child.  Not creating a lease");
             } else {
                 LOG.debug("Need to create a lease for shardId " + shardId);
                 KinesisClientLease newLease = newKCLLease(shard);
@@ -407,6 +448,17 @@ class ShardSyncer {
         return newLeasesToCreate;
     }
 
+    /**
+     * Determine new leases to create and their initial checkpoint.
+     * Note: Package level access only for testing purposes.
+     */
+    static List<KinesisClientLease> determineNewLeasesToCreate(List<Shard> shards,
+            List<KinesisClientLease> currentLeases,
+            InitialPositionInStreamExtended initialPosition) {
+        Set<String> inconsistentShardIds = new HashSet<String>();
+        return determineNewLeasesToCreate(shards, currentLeases, initialPosition, inconsistentShardIds);
+    }
+
     /**
      * Note: Package level access for testing purposes only.
      * Check if this shard is a descendant of a shard that is (or will be) processed.

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShutdownReason.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShutdownTask.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
@@ -44,22 +44,26 @@ class ShutdownTask implements ITask {
     private final ILeaseManager<KinesisClientLease> leaseManager;
     private final InitialPositionInStreamExtended initialPositionInStream;
     private final boolean cleanupLeasesOfCompletedShards;
+    private final boolean ignoreUnexpectedChildShards;
     private final TaskType taskType = TaskType.SHUTDOWN;
     private final long backoffTimeMillis;
+    private final GetRecordsCache getRecordsCache;
 
     /**
      * Constructor.
      */
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 10 LINES
     ShutdownTask(ShardInfo shardInfo,
-            IRecordProcessor recordProcessor,
+                 IRecordProcessor recordProcessor,
-            RecordProcessorCheckpointer recordProcessorCheckpointer,
+                 RecordProcessorCheckpointer recordProcessorCheckpointer,
-            ShutdownReason reason,
+                 ShutdownReason reason,
-            IKinesisProxy kinesisProxy,
+                 IKinesisProxy kinesisProxy,
-            InitialPositionInStreamExtended initialPositionInStream,
+                 InitialPositionInStreamExtended initialPositionInStream,
-            boolean cleanupLeasesOfCompletedShards,
+                 boolean cleanupLeasesOfCompletedShards,
-            ILeaseManager<KinesisClientLease> leaseManager,
+                 boolean ignoreUnexpectedChildShards,
-            long backoffTimeMillis) {
+                 ILeaseManager<KinesisClientLease> leaseManager,
+                 long backoffTimeMillis, 
+                 GetRecordsCache getRecordsCache) {
         this.shardInfo = shardInfo;
         this.recordProcessor = recordProcessor;
         this.recordProcessorCheckpointer = recordProcessorCheckpointer;
@@ -67,8 +71,10 @@ class ShutdownTask implements ITask {
         this.kinesisProxy = kinesisProxy;
         this.initialPositionInStream = initialPositionInStream;
         this.cleanupLeasesOfCompletedShards = cleanupLeasesOfCompletedShards;
+        this.ignoreUnexpectedChildShards = ignoreUnexpectedChildShards;
         this.leaseManager = leaseManager;
         this.backoffTimeMillis = backoffTimeMillis;
+        this.getRecordsCache = getRecordsCache;
     }
 
     /*
@@ -79,7 +85,7 @@ class ShutdownTask implements ITask {
      */
     @Override
     public TaskResult call() {
-        Exception exception = null;
+        Exception exception;
         boolean applicationException = false;
 
         try {
@@ -107,6 +113,8 @@ class ShutdownTask implements ITask {
                                 + shardInfo.getShardId());
                     }
                 }
+                LOG.debug("Shutting down retrieval strategy.");
+                getRecordsCache.shutdown();
                 LOG.debug("Record processor completed shutdown() for shard " + shardInfo.getShardId());
             } catch (Exception e) {
                 applicationException = true;
@@ -122,7 +130,8 @@ class ShutdownTask implements ITask {
                 ShardSyncer.checkAndCreateLeasesForNewShards(kinesisProxy,
                         leaseManager,
                         initialPositionInStream,
-                        cleanupLeasesOfCompletedShards);
+                        cleanupLeasesOfCompletedShards,
+                        ignoreUnexpectedChildShards);
                 LOG.debug("Finished checking for child shards of shard " + shardInfo.getShardId());
             }
 

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/SimpleRecordsFetcherFactory.java

@@ -0,0 +1,74 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import java.util.concurrent.Executors;
+
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import lombok.extern.apachecommons.CommonsLog;
+
+@CommonsLog
+public class SimpleRecordsFetcherFactory implements RecordsFetcherFactory {
+    private int maxPendingProcessRecordsInput = 3;
+    private int maxByteSize = 8 * 1024 * 1024;
+    private int maxRecordsCount = 30000;
+    private long idleMillisBetweenCalls = 1500L;
+    private DataFetchingStrategy dataFetchingStrategy = DataFetchingStrategy.DEFAULT;
+    
+    @Override
+    public GetRecordsCache createRecordsFetcher(GetRecordsRetrievalStrategy getRecordsRetrievalStrategy, String shardId,
+                                                IMetricsFactory metricsFactory, int maxRecords) {
+        if(dataFetchingStrategy.equals(DataFetchingStrategy.DEFAULT)) {
+            return new BlockingGetRecordsCache(maxRecords, getRecordsRetrievalStrategy);
+        } else {
+            return new PrefetchGetRecordsCache(maxPendingProcessRecordsInput, maxByteSize, maxRecordsCount, maxRecords,
+                    getRecordsRetrievalStrategy,
+                    Executors.newFixedThreadPool(1, new ThreadFactoryBuilder()
+                            .setDaemon(true)
+                            .setNameFormat("prefetch-cache-" + shardId + "-%04d")
+                            .build()),
+                            idleMillisBetweenCalls,
+                    metricsFactory,
+                    "ProcessTask",
+                    shardId);
+        }
+    }
+
+    @Override
+    public void setMaxPendingProcessRecordsInput(int maxPendingProcessRecordsInput){
+        this.maxPendingProcessRecordsInput = maxPendingProcessRecordsInput;
+    }
+
+    @Override
+    public void setMaxByteSize(int maxByteSize){
+        this.maxByteSize = maxByteSize;
+    }
+
+    @Override
+    public void setMaxRecordsCount(int maxRecordsCount) {
+        this.maxRecordsCount = maxRecordsCount;
+    }
+
+    @Override
+    public void setDataFetchingStrategy(DataFetchingStrategy dataFetchingStrategy){
+        this.dataFetchingStrategy = dataFetchingStrategy;
+    }
+
+    public void setIdleMillisBetweenCalls(final long idleMillisBetweenCalls) {
+        this.idleMillisBetweenCalls = idleMillisBetweenCalls;
+    }
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/SynchronousGetRecordsRetrievalStrategy.java

@@ -0,0 +1,50 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import lombok.Data;
+import lombok.NonNull;
+
+/**
+ *
+ */
+@Data
+public class SynchronousGetRecordsRetrievalStrategy implements GetRecordsRetrievalStrategy {
+    @NonNull
+    private final KinesisDataFetcher dataFetcher;
+
+    @Override
+    public GetRecordsResult getRecords(final int maxRecords) {
+        return dataFetcher.getRecords(maxRecords).accept();
+    }
+
+    @Override
+    public void shutdown() {
+        //
+        // Does nothing as this retriever doesn't manage any resources
+        //
+    }
+
+    @Override
+    public boolean isShutdown() {
+        return false;
+    }
+    
+    @Override
+    public KinesisDataFetcher getDataFetcher() {
+        return dataFetcher;
+    }
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java

@@ -17,6 +17,7 @@ package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 import java.util.Collection;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Optional;
 import java.util.Set;
 import java.util.concurrent.Callable;
 import java.util.concurrent.ConcurrentHashMap;
@@ -31,6 +32,7 @@ import java.util.concurrent.ThreadPoolExecutor;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 
+import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 
@@ -72,6 +74,7 @@ public class Worker implements Runnable {
 
     private final String applicationName;
     private final IRecordProcessorFactory recordProcessorFactory;
+    private final KinesisClientLibConfiguration config;
     private final StreamConfig streamConfig;
     private final InitialPositionInStreamExtended initialPosition;
     private final ICheckpoint checkpointTracker;
@@ -85,6 +88,9 @@ public class Worker implements Runnable {
     private final long taskBackoffTimeMillis;
     private final long failoverTimeMillis;
 
+    private final Optional<Integer> retryGetRecordsInSeconds;
+    private final Optional<Integer> maxGetRecordsThreadPool;
+
     // private final KinesisClientLeaseManager leaseManager;
     private final KinesisClientLibLeaseCoordinator leaseCoordinator;
     private final ShardSyncTaskManager controlServer;
@@ -241,6 +247,7 @@ public class Worker implements Runnable {
             KinesisClientLibConfiguration config, AmazonKinesis kinesisClient, AmazonDynamoDB dynamoDBClient,
             IMetricsFactory metricsFactory, ExecutorService execService) {
         this(config.getApplicationName(), new V1ToV2RecordProcessorFactoryAdapter(recordProcessorFactory),
+                config,
                 new StreamConfig(
                         new KinesisProxyFactory(config.getKinesisCredentialsProvider(), kinesisClient)
                                 .getProxy(config.getStreamName()),
@@ -266,7 +273,9 @@ public class Worker implements Runnable {
                 config.getTaskBackoffTimeMillis(),
                 config.getFailoverTimeMillis(),
                 config.getSkipShardSyncAtWorkerInitializationIfLeasesExist(),
-                config.getShardPrioritizationStrategy());
+                config.getShardPrioritizationStrategy(),
+                config.getRetryGetRecordsInSeconds(),
+                config.getMaxGetRecordsThreadPool());
 
         // If a region name was explicitly specified, use it as the region for Amazon Kinesis and Amazon DynamoDB.
         if (config.getRegionName() != null) {
@@ -300,6 +309,8 @@ public class Worker implements Runnable {
      *            Name of the Kinesis application
      * @param recordProcessorFactory
      *            Used to get record processor instances for processing data from shards
+     * @paran config
+     *            Kinesis Library configuration
      * @param streamConfig
      *            Stream configuration
      * @param initialPositionInStream
@@ -327,14 +338,66 @@ public class Worker implements Runnable {
      */
     // NOTE: This has package level access solely for testing
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 10 LINES
-    Worker(String applicationName, IRecordProcessorFactory recordProcessorFactory, StreamConfig streamConfig,
+    Worker(String applicationName, IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config,
-            InitialPositionInStreamExtended initialPositionInStream, long parentShardPollIntervalMillis,
+           StreamConfig streamConfig, InitialPositionInStreamExtended initialPositionInStream, long parentShardPollIntervalMillis,
             long shardSyncIdleTimeMillis, boolean cleanupLeasesUponShardCompletion, ICheckpoint checkpoint,
             KinesisClientLibLeaseCoordinator leaseCoordinator, ExecutorService execService,
             IMetricsFactory metricsFactory, long taskBackoffTimeMillis, long failoverTimeMillis,
             boolean skipShardSyncAtWorkerInitializationIfLeasesExist, ShardPrioritization shardPrioritization) {
+        this(applicationName, recordProcessorFactory, config, streamConfig, initialPositionInStream, parentShardPollIntervalMillis,
+                shardSyncIdleTimeMillis, cleanupLeasesUponShardCompletion, checkpoint, leaseCoordinator, execService,
+                metricsFactory, taskBackoffTimeMillis, failoverTimeMillis, skipShardSyncAtWorkerInitializationIfLeasesExist,
+                shardPrioritization, Optional.empty(), Optional.empty());
+    }
+
+    /**
+     * @param applicationName
+     *            Name of the Kinesis application
+     * @param recordProcessorFactory
+     *            Used to get record processor instances for processing data from shards
+     * @param config
+     *            Kinesis Library Configuration
+     * @param streamConfig
+     *            Stream configuration
+     * @param initialPositionInStream
+     *            One of LATEST, TRIM_HORIZON, or AT_TIMESTAMP. The KinesisClientLibrary will start fetching data from
+     *            this location in the stream when an application starts up for the first time and there are no
+     *            checkpoints. If there are checkpoints, we start from the checkpoint position.
+     * @param parentShardPollIntervalMillis
+     *            Wait for this long between polls to check if parent shards are done
+     * @param shardSyncIdleTimeMillis
+     *            Time between tasks to sync leases and Kinesis shards
+     * @param cleanupLeasesUponShardCompletion
+     *            Clean up shards we've finished processing (don't wait till they expire in Kinesis)
+     * @param checkpoint
+     *            Used to get/set checkpoints
+     * @param leaseCoordinator
+     *            Lease coordinator (coordinates currently owned leases)
+     * @param execService
+     *            ExecutorService to use for processing records (support for multi-threaded consumption)
+     * @param metricsFactory
+     *            Metrics factory used to emit metrics
+     * @param taskBackoffTimeMillis
+     *            Backoff period when tasks encounter an exception
+     * @param shardPrioritization
+     *            Provides prioritization logic to decide which available shards process first
+     * @param retryGetRecordsInSeconds
+     *            Time in seconds to wait before the worker retries to get a record.
+     * @param maxGetRecordsThreadPool
+     *            Max number of threads in the getRecords thread pool.
+     */
+    // NOTE: This has package level access solely for testing
+    // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 10 LINES
+    Worker(String applicationName, IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, StreamConfig streamConfig,
+           InitialPositionInStreamExtended initialPositionInStream, long parentShardPollIntervalMillis,
+           long shardSyncIdleTimeMillis, boolean cleanupLeasesUponShardCompletion, ICheckpoint checkpoint,
+           KinesisClientLibLeaseCoordinator leaseCoordinator, ExecutorService execService,
+           IMetricsFactory metricsFactory, long taskBackoffTimeMillis, long failoverTimeMillis,
+           boolean skipShardSyncAtWorkerInitializationIfLeasesExist, ShardPrioritization shardPrioritization,
+           Optional<Integer> retryGetRecordsInSeconds, Optional<Integer> maxGetRecordsThreadPool) {
         this.applicationName = applicationName;
         this.recordProcessorFactory = recordProcessorFactory;
+        this.config = config;
         this.streamConfig = streamConfig;
         this.initialPosition = initialPositionInStream;
         this.parentShardPollIntervalMillis = parentShardPollIntervalMillis;
@@ -345,12 +408,14 @@ public class Worker implements Runnable {
         this.leaseCoordinator = leaseCoordinator;
         this.metricsFactory = metricsFactory;
         this.controlServer = new ShardSyncTaskManager(streamConfig.getStreamProxy(), leaseCoordinator.getLeaseManager(),
-                initialPositionInStream, cleanupLeasesUponShardCompletion, shardSyncIdleTimeMillis, metricsFactory,
+                initialPositionInStream, cleanupLeasesUponShardCompletion, config.shouldIgnoreUnexpectedChildShards(),
-                executorService);
+                shardSyncIdleTimeMillis, metricsFactory, executorService);
         this.taskBackoffTimeMillis = taskBackoffTimeMillis;
         this.failoverTimeMillis = failoverTimeMillis;
         this.skipShardSyncAtWorkerInitializationIfLeasesExist = skipShardSyncAtWorkerInitializationIfLeasesExist;
         this.shardPrioritization = shardPrioritization;
+        this.retryGetRecordsInSeconds = retryGetRecordsInSeconds;
+        this.maxGetRecordsThreadPool = maxGetRecordsThreadPool;
     }
 
     /**
@@ -435,7 +500,8 @@ public class Worker implements Runnable {
                         || leaseCoordinator.getLeaseManager().isLeaseTableEmpty()) {
                     LOG.info("Syncing Kinesis shard info");
                     ShardSyncTask shardSyncTask = new ShardSyncTask(streamConfig.getStreamProxy(),
-                            leaseCoordinator.getLeaseManager(), initialPosition, cleanupLeasesUponShardCompletion, 0L);
+                            leaseCoordinator.getLeaseManager(), initialPosition, cleanupLeasesUponShardCompletion,
+                            config.shouldIgnoreUnexpectedChildShards(), 0L);
                     result = new MetricsCollectingTaskDecorator(shardSyncTask, metricsFactory).call();
                 } else {
                     LOG.info("Skipping shard sync per config setting (and lease table is not empty)");
@@ -760,11 +826,11 @@ public class Worker implements Runnable {
      *
      * @param shardInfo
      *            Kinesis shard info
-     * @param factory
+     * @param processorFactory
      *            RecordProcessor factory
      * @return ShardConsumer for the shard
      */
-    ShardConsumer createOrGetShardConsumer(ShardInfo shardInfo, IRecordProcessorFactory factory) {
+    ShardConsumer createOrGetShardConsumer(ShardInfo shardInfo, IRecordProcessorFactory processorFactory) {
         ShardConsumer consumer = shardInfoShardConsumerMap.get(shardInfo);
         // Instantiate a new consumer if we don't have one, or the one we
         // had was from an earlier
@@ -773,20 +839,30 @@ public class Worker implements Runnable {
         // completely processed (shutdown reason terminate).
         if ((consumer == null)
                 || (consumer.isShutdown() && consumer.getShutdownReason().equals(ShutdownReason.ZOMBIE))) {
-            consumer = buildConsumer(shardInfo, factory);
+            consumer = buildConsumer(shardInfo, processorFactory);
             shardInfoShardConsumerMap.put(shardInfo, consumer);
             wlog.infoForce("Created new shardConsumer for : " + shardInfo);
         }
         return consumer;
     }
 
-    protected ShardConsumer buildConsumer(ShardInfo shardInfo, IRecordProcessorFactory factory) {
+    protected ShardConsumer buildConsumer(ShardInfo shardInfo, IRecordProcessorFactory processorFactory) {
-        IRecordProcessor recordProcessor = factory.createProcessor();
+        IRecordProcessor recordProcessor = processorFactory.createProcessor();
 
-        return new ShardConsumer(shardInfo, streamConfig, checkpointTracker, recordProcessor,
+        return new ShardConsumer(shardInfo,
-                leaseCoordinator.getLeaseManager(), parentShardPollIntervalMillis, cleanupLeasesUponShardCompletion,
+                streamConfig,
-                executorService, metricsFactory, taskBackoffTimeMillis,
+                checkpointTracker,
-                skipShardSyncAtWorkerInitializationIfLeasesExist);
+                recordProcessor,
+                leaseCoordinator.getLeaseManager(),
+                parentShardPollIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                executorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                skipShardSyncAtWorkerInitializationIfLeasesExist,
+                retryGetRecordsInSeconds,
+                maxGetRecordsThreadPool,
+                config);
 
     }
 
@@ -921,6 +997,11 @@ public class Worker implements Runnable {
                 metricsFactory, execService);
     }
 
+    @VisibleForTesting
+    StreamConfig getStreamConfig() {
+        return streamConfig;
+    }
+
     /**
      * Given configuration, returns appropriate metrics factory.
      * 
@@ -990,6 +1071,7 @@ public class Worker implements Runnable {
     public static class Builder {
 
         private IRecordProcessorFactory recordProcessorFactory;
+        private RecordsFetcherFactory recordsFetcherFactory;
         private KinesisClientLibConfiguration config;
         private AmazonKinesis kinesisClient;
         private AmazonDynamoDB dynamoDBClient;
@@ -997,6 +1079,7 @@ public class Worker implements Runnable {
         private IMetricsFactory metricsFactory;
         private ExecutorService execService;
         private ShardPrioritization shardPrioritization;
+        private IKinesisProxy kinesisProxy;
 
         /**
          * Default constructor.
@@ -1116,6 +1199,19 @@ public class Worker implements Runnable {
             return this;
         }
 
+        /**
+         * Set KinesisProxy for the worker.
+         *
+         * @param kinesisProxy
+         *            Sets an implementation of IKinesisProxy.
+         *
+         * @return A reference to this updated object so that method calls can be chained together.
+         */
+        public Builder kinesisProxy(IKinesisProxy kinesisProxy) {
+            this.kinesisProxy = kinesisProxy;
+            return this;
+        }
+
         /**
          * Build the Worker instance.
          *
@@ -1181,12 +1277,15 @@ public class Worker implements Runnable {
             if (shardPrioritization == null) {
                 shardPrioritization = new ParentsFirstShardPrioritization(1);
             }
-
+            if (kinesisProxy == null) {
+                kinesisProxy = new KinesisProxyFactory(config.getKinesisCredentialsProvider(), kinesisClient)
+                    .getProxy(config.getStreamName());
+            }
 
             return new Worker(config.getApplicationName(),
                     recordProcessorFactory,
-                    new StreamConfig(new KinesisProxyFactory(config.getKinesisCredentialsProvider(),
+                    config,
-                            kinesisClient).getProxy(config.getStreamName()),
+                    new StreamConfig(kinesisProxy,
                             config.getMaxRecords(),
                             config.getIdleTimeBetweenReadsInMillis(),
                             config.shouldCallProcessRecordsEvenForEmptyRecordList(),
@@ -1213,7 +1312,9 @@ public class Worker implements Runnable {
                     config.getTaskBackoffTimeMillis(),
                     config.getFailoverTimeMillis(),
                     config.getSkipShardSyncAtWorkerInitializationIfLeasesExist(),
-                    shardPrioritization);
+                    shardPrioritization,
+                    config.getRetryGetRecordsInSeconds(),
+                    config.getMaxGetRecordsThreadPool());
 
         }
 

src/main/java/com/amazonaws/services/kinesis/clientlibrary/proxies/KinesisProxy.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.proxies;
 

src/main/java/com/amazonaws/services/kinesis/clientlibrary/types/InitializationInput.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.types;
 

src/main/java/com/amazonaws/services/kinesis/clientlibrary/types/ProcessRecordsInput.java

@@ -1,30 +1,38 @@
 /*
- * Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.types;
 
+import java.time.Duration;
+import java.time.Instant;
 import java.util.List;
+
 import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer;
 import com.amazonaws.services.kinesis.model.Record;
 
+import lombok.Getter;
+
 /**
  * Container for the parameters to the IRecordProcessor's
  * {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor#processRecords(
  * ProcessRecordsInput processRecordsInput) processRecords} method.
  */
 public class ProcessRecordsInput {
-
+    @Getter
+    private Instant cacheEntryTime;
+    @Getter
+    private Instant cacheExitTime;
     private List<Record> records;
     private IRecordProcessorCheckpointer checkpointer;
     private Long millisBehindLatest;
@@ -96,4 +104,21 @@ public class ProcessRecordsInput {
         this.millisBehindLatest = millisBehindLatest;
         return this;
     }
+    
+    public ProcessRecordsInput withCacheEntryTime(Instant cacheEntryTime) {
+        this.cacheEntryTime = cacheEntryTime;
+        return this;
+    }
+    
+    public ProcessRecordsInput withCacheExitTime(Instant cacheExitTime) {
+        this.cacheExitTime = cacheExitTime;
+        return this;
+    }
+    
+    public Duration getTimeSpentInCache() {
+        if (cacheEntryTime == null || cacheExitTime == null) {
+            return Duration.ZERO;
+        }
+        return Duration.between(cacheEntryTime, cacheExitTime);
+    }
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/types/ShutdownInput.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.types;
 

src/main/java/com/amazonaws/services/kinesis/leases/impl/KinesisClientLease.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.leases.impl;
 

src/main/java/com/amazonaws/services/kinesis/leases/impl/KinesisClientLeaseSerializer.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.leases.impl;
 

src/main/java/com/amazonaws/services/kinesis/leases/impl/LeaseManager.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.leases.impl;
 
@@ -19,6 +19,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.concurrent.TimeUnit;
 
+import com.amazonaws.services.kinesis.leases.util.DynamoUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 
@@ -386,7 +387,19 @@ public class LeaseManager<T extends Lease> implements ILeaseManager<T> {
                         + " because the lease counter was not " + lease.getLeaseCounter());
             }
 
-            return false;
+            // If we had a spurious retry during the Dynamo update, then this conditional PUT failure
+            // might be incorrect. So, we get the item straight away and check if the lease owner + lease counter
+            // are what we expected.
+            String expectedOwner = lease.getLeaseOwner();
+            Long expectedCounter = lease.getLeaseCounter() + 1;
+            T updatedLease = getLease(lease.getLeaseKey());
+            if (updatedLease == null || !expectedOwner.equals(updatedLease.getLeaseOwner()) ||
+                    !expectedCounter.equals(updatedLease.getLeaseCounter())) {
+                return false;
+            }
+
+            LOG.info("Detected spurious renewal failure for lease with key " + lease.getLeaseKey()
+                    + ", but recovered");
         } catch (AmazonClientException e) {
             throw convertAndRethrowExceptions("renew", lease.getLeaseKey(), e);
         }
@@ -564,6 +577,7 @@ public class LeaseManager<T extends Lease> implements ILeaseManager<T> {
     protected DependencyException convertAndRethrowExceptions(String operation, String leaseKey, AmazonClientException e)
         throws ProvisionedThroughputException, InvalidStateException {
         if (e instanceof ProvisionedThroughputExceededException) {
+            LOG.warn("Provisioned Throughput on the lease table has been exceeded. It's recommended that you increase the IOPs on the table. Failure to increase the IOPs may cause the application to not make progress.");
             throw new ProvisionedThroughputException(e);
         } else if (e instanceof ResourceNotFoundException) {
             // @formatter:on

src/main/java/com/amazonaws/services/kinesis/metrics/impl/MetricsHelper.java

@@ -72,13 +72,22 @@ public class MetricsHelper {
      * @param scope
      */
     public static void setMetricsScope(IMetricsScope scope) {
-        if (currentScope.get() != null) {
+        if (isMetricsScopePresent()) {
             throw new RuntimeException(String.format(
                     "Metrics scope is already set for the current thread %s", Thread.currentThread().getName()));
         }
         currentScope.set(scope);
     }
 
+    /**
+     * Checks if current metricsscope is present or not.
+     * 
+     * @return true if metrics scope is present, else returns false
+     */
+    public static boolean isMetricsScopePresent() {
+        return currentScope.get() != null;
+    }
+
     /**
      * Unsets the metrics scope for the current thread.
      */

src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemon.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 
@@ -148,13 +148,14 @@ public class MultiLangDaemon implements Callable<Integer> {
                 config.getRecordProcessorFactory(),
                 executorService);
 
+        final long shutdownGraceMillis = config.getKinesisClientLibConfiguration().getShutdownGraceMillis();
         Runtime.getRuntime().addShutdownHook(new Thread() {
             @Override
             public void run() {
                 LOG.info("Process terminanted, will initiate shutdown.");
                 try {
                     Future<Void> fut = daemon.worker.requestShutdown();
-                    fut.get(5000, TimeUnit.MILLISECONDS);
+                    fut.get(shutdownGraceMillis, TimeUnit.MILLISECONDS);
                     LOG.info("Process shutdown is complete.");
                 } catch (InterruptedException | ExecutionException | TimeoutException e) {
                     LOG.error("Encountered an error during shutdown.", e);

src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangDaemonConfig.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 

src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangProtocol.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 

src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangRecordProcessor.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 

src/main/java/com/amazonaws/services/kinesis/multilang/MultiLangRecordProcessorFactory.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 

src/main/java/com/amazonaws/services/kinesis/multilang/messages/CheckpointMessage.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang.messages;
 

src/main/java/com/amazonaws/services/kinesis/multilang/messages/InitializeMessage.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang.messages;
 

src/main/java/com/amazonaws/services/kinesis/multilang/messages/JsonFriendlyRecord.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang.messages;
 

src/main/java/com/amazonaws/services/kinesis/multilang/messages/Message.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang.messages;
 

src/main/java/com/amazonaws/services/kinesis/multilang/messages/ProcessRecordsMessage.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang.messages;
 

src/main/java/com/amazonaws/services/kinesis/multilang/messages/ShutdownRequestedMessage.java

@@ -1,3 +1,17 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
 package com.amazonaws.services.kinesis.multilang.messages;
 
 /**

src/test/java/com/amazonaws/services/kinesis/clientlibrary/config/KinesisClientLibConfiguratorTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.config;
 
@@ -22,6 +22,7 @@ import static org.junit.Assert.fail;
 
 import java.io.ByteArrayInputStream;
 import java.io.InputStream;
+import java.util.Optional;
 import java.util.Set;
 
 import org.apache.commons.lang.StringUtils;
@@ -60,6 +61,8 @@ public class KinesisClientLibConfiguratorTest {
         assertEquals(config.getApplicationName(), "b");
         assertEquals(config.getStreamName(), "a");
         assertEquals(config.getWorkerIdentifier(), "123");
+        assertEquals(config.getMaxGetRecordsThreadPool(), Optional.empty());
+        assertEquals(config.getRetryGetRecordsInSeconds(), Optional.empty());
     }
 
     @Test
@@ -107,7 +110,9 @@ public class KinesisClientLibConfiguratorTest {
                         "workerId = w123",
                         "maxRecords = 10",
                         "metricsMaxQueueSize = 20",
-                        "applicationName = kinesis"
+                        "applicationName = kinesis",
+                        "retryGetRecordsInSeconds = 2",
+                        "maxGetRecordsThreadPool = 1"
                 }, '\n'));
 
         assertEquals(config.getApplicationName(), "kinesis");
@@ -115,6 +120,8 @@ public class KinesisClientLibConfiguratorTest {
         assertEquals(config.getWorkerIdentifier(), "w123");
         assertEquals(config.getMaxRecords(), 10);
         assertEquals(config.getMetricsMaxQueueSize(), 20);
+        assertEquals(config.getRetryGetRecordsInSeconds(), Optional.of(2));
+        assertEquals(config.getMaxGetRecordsThreadPool(), Optional.of(1));
     }
 
     @Test
@@ -202,6 +209,42 @@ public class KinesisClientLibConfiguratorTest {
         assertEquals(config.getInitialPositionInStream(), InitialPositionInStream.TRIM_HORIZON);
     }
 
+    @Test
+    public void testEmptyOptionalVariables() {
+        KinesisClientLibConfiguration config =
+                getConfiguration(StringUtils.join(new String[] {
+                        "streamName = a",
+                        "applicationName = b",
+                        "AWSCredentialsProvider = ABCD," + credentialName1,
+                        "workerId = 123",
+                        "initialPositionInStream = TriM_Horizon",
+                        "maxGetRecordsThreadPool = 1"
+                }, '\n'));
+        assertEquals(config.getMaxGetRecordsThreadPool(), Optional.of(1));
+        assertEquals(config.getRetryGetRecordsInSeconds(), Optional.empty());
+    }
+
+    @Test
+    public void testWithZeroValue() {
+        String test = StringUtils.join(new String[]{
+                        "streamName = a",
+                        "applicationName = b",
+                        "AWSCredentialsProvider = ABCD," + credentialName1,
+                        "workerId = 123",
+                        "initialPositionInStream = TriM_Horizon",
+                        "maxGetRecordsThreadPool = 0",
+                        "retryGetRecordsInSeconds = 0"
+                }, '\n');
+        InputStream input = new ByteArrayInputStream(test.getBytes());
+
+        try {
+            configurator.getConfiguration(input);
+        } catch (Exception e) {
+            fail("Don't expect to fail on invalid variable value");
+
+        }
+    }
+
     @Test
     public void testWithInvalidIntValue() {
         String test = StringUtils.join(new String[] {

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/AsynchronousGetRecordsRetrievalStrategyIntegrationTest.java

@@ -0,0 +1,193 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import static org.hamcrest.CoreMatchers.nullValue;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.IsEqual.equalTo;
+import static org.junit.Assert.assertNull;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.atLeast;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.CompletionService;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+import com.amazonaws.services.kinesis.model.ExpiredIteratorException;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.mockito.stubbing.Answer;
+
+@RunWith(MockitoJUnitRunner.class)
+public class AsynchronousGetRecordsRetrievalStrategyIntegrationTest {
+
+    private static final int CORE_POOL_SIZE = 1;
+    private static final int MAX_POOL_SIZE = 2;
+    private static final int TIME_TO_LIVE = 5;
+    private static final int RETRY_GET_RECORDS_IN_SECONDS = 2;
+    private static final int SLEEP_GET_RECORDS_IN_SECONDS = 10;
+
+    @Mock
+    private IKinesisProxy mockKinesisProxy;
+    @Mock
+    private ShardInfo mockShardInfo;
+    @Mock
+    private Supplier<CompletionService<DataFetcherResult>> completionServiceSupplier;
+    @Mock
+    private DataFetcherResult result;
+    @Mock
+    private GetRecordsResult recordsResult;
+
+    private CompletionService<DataFetcherResult> completionService;
+
+    private AsynchronousGetRecordsRetrievalStrategy getRecordsRetrivalStrategy;
+    private KinesisDataFetcher dataFetcher;
+    private ExecutorService executorService;
+    private RejectedExecutionHandler rejectedExecutionHandler;
+    private int numberOfRecords = 10;
+
+
+    @Before
+    public void setup() {
+        dataFetcher = spy(new KinesisDataFetcherForTests(mockKinesisProxy, mockShardInfo));
+        rejectedExecutionHandler = spy(new ThreadPoolExecutor.AbortPolicy());
+        executorService = spy(new ThreadPoolExecutor(
+                CORE_POOL_SIZE,
+                MAX_POOL_SIZE,
+                TIME_TO_LIVE,
+                TimeUnit.SECONDS,
+                new LinkedBlockingQueue<>(1),
+                new ThreadFactoryBuilder().setDaemon(true).setNameFormat("getrecords-worker-%d").build(),
+                rejectedExecutionHandler));
+        completionService = spy(new ExecutorCompletionService<DataFetcherResult>(executorService));
+        when(completionServiceSupplier.get()).thenReturn(completionService);
+        getRecordsRetrivalStrategy = new AsynchronousGetRecordsRetrievalStrategy(dataFetcher, executorService, RETRY_GET_RECORDS_IN_SECONDS, completionServiceSupplier, "shardId-0001");
+        when(result.accept()).thenReturn(recordsResult);
+    }
+
+    @Test
+    public void oneRequestMultithreadTest() {
+        when(result.accept()).thenReturn(null);
+        GetRecordsResult getRecordsResult = getRecordsRetrivalStrategy.getRecords(numberOfRecords);
+        verify(dataFetcher, atLeast(getLeastNumberOfCalls())).getRecords(eq(numberOfRecords));
+        verify(executorService, atLeast(getLeastNumberOfCalls())).execute(any());
+        assertNull(getRecordsResult);
+    }
+
+    @Test
+    public void multiRequestTest() {
+        ExecutorCompletionService<DataFetcherResult> completionService1 = spy(new ExecutorCompletionService<DataFetcherResult>(executorService));
+        when(completionServiceSupplier.get()).thenReturn(completionService1);
+        GetRecordsResult getRecordsResult = getRecordsRetrivalStrategy.getRecords(numberOfRecords);
+        verify(dataFetcher, atLeast(getLeastNumberOfCalls())).getRecords(numberOfRecords);
+        verify(executorService, atLeast(getLeastNumberOfCalls())).execute(any());
+        assertThat(getRecordsResult, equalTo(recordsResult));
+
+        when(result.accept()).thenReturn(null);
+        ExecutorCompletionService<DataFetcherResult> completionService2 = spy(new ExecutorCompletionService<DataFetcherResult>(executorService));
+        when(completionServiceSupplier.get()).thenReturn(completionService2);
+        getRecordsResult = getRecordsRetrivalStrategy.getRecords(numberOfRecords);
+        assertThat(getRecordsResult, nullValue(GetRecordsResult.class));
+    }
+
+    @Test
+    @Ignore
+    public void testInterrupted() throws InterruptedException, ExecutionException {
+        Future<DataFetcherResult> mockFuture = mock(Future.class);
+        when(completionService.submit(any())).thenReturn(mockFuture);
+        when(completionService.poll()).thenReturn(mockFuture);
+        doThrow(InterruptedException.class).when(mockFuture).get();
+        GetRecordsResult getRecordsResult = getRecordsRetrivalStrategy.getRecords(numberOfRecords);
+        verify(mockFuture).get();
+        assertNull(getRecordsResult);
+    }
+    
+    @Test (expected = ExpiredIteratorException.class)
+    public void testExpiredIteratorExcpetion() throws InterruptedException {
+        when(dataFetcher.getRecords(eq(numberOfRecords))).thenAnswer(new Answer<DataFetcherResult>() {
+            @Override
+            public DataFetcherResult answer(final InvocationOnMock invocationOnMock) throws Throwable {
+                Thread.sleep(SLEEP_GET_RECORDS_IN_SECONDS * 1000);
+                throw new ExpiredIteratorException("ExpiredIterator");
+            }
+        });
+        
+        try {
+            getRecordsRetrivalStrategy.getRecords(numberOfRecords);
+        } finally {
+            verify(dataFetcher, atLeast(getLeastNumberOfCalls())).getRecords(eq(numberOfRecords));
+            verify(executorService, atLeast(getLeastNumberOfCalls())).execute(any());
+        }
+    }
+
+    private int getLeastNumberOfCalls() {
+        int leastNumberOfCalls = 0;
+        for (int i = MAX_POOL_SIZE; i > 0; i--) {
+            if (i * RETRY_GET_RECORDS_IN_SECONDS <= SLEEP_GET_RECORDS_IN_SECONDS) {
+                leastNumberOfCalls = i;
+                break;
+            }
+        }
+        return leastNumberOfCalls;
+    }
+
+    @After
+    public void shutdown() {
+        getRecordsRetrivalStrategy.shutdown();
+        verify(executorService).shutdownNow();
+    }
+
+    private class KinesisDataFetcherForTests extends KinesisDataFetcher {
+        public KinesisDataFetcherForTests(final IKinesisProxy kinesisProxy, final ShardInfo shardInfo) {
+            super(kinesisProxy, shardInfo);
+        }
+
+        @Override
+        public DataFetcherResult getRecords(final int maxRecords) {
+            try {
+                Thread.sleep(SLEEP_GET_RECORDS_IN_SECONDS * 1000);
+            } catch (InterruptedException e) {
+                // Do nothing
+            }
+            
+            return result;
+        }
+    }
+
+}

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/AsynchronousGetRecordsRetrievalStrategyTest.java

@@ -0,0 +1,184 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyBoolean;
+import static org.mockito.Matchers.anyLong;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.CompletionService;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+import com.amazonaws.services.kinesis.model.ExpiredIteratorException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+
+/**
+ *
+ */
+@RunWith(MockitoJUnitRunner.class)
+public class AsynchronousGetRecordsRetrievalStrategyTest {
+
+    private static final long RETRY_GET_RECORDS_IN_SECONDS = 5;
+    private static final String SHARD_ID = "ShardId-0001";
+    @Mock
+    private KinesisDataFetcher dataFetcher;
+    @Mock
+    private ExecutorService executorService;
+    @Mock
+    private Supplier<CompletionService<DataFetcherResult>> completionServiceSupplier;
+    @Mock
+    private CompletionService<DataFetcherResult> completionService;
+    @Mock
+    private Future<DataFetcherResult> successfulFuture;
+    @Mock
+    private Future<DataFetcherResult> blockedFuture;
+    @Mock
+    private DataFetcherResult dataFetcherResult;
+    @Mock
+    private GetRecordsResult expectedResults;
+
+    @Before
+    public void before() {
+        when(completionServiceSupplier.get()).thenReturn(completionService);
+        when(dataFetcherResult.getResult()).thenReturn(expectedResults);
+        when(dataFetcherResult.accept()).thenReturn(expectedResults);
+    }
+
+    @Test
+    public void testSingleSuccessfulRequestFuture() throws Exception {
+        AsynchronousGetRecordsRetrievalStrategy strategy = new AsynchronousGetRecordsRetrievalStrategy(dataFetcher,
+                executorService, (int) RETRY_GET_RECORDS_IN_SECONDS, completionServiceSupplier, SHARD_ID);
+
+        when(executorService.isShutdown()).thenReturn(false);
+        when(completionService.submit(any())).thenReturn(successfulFuture);
+        when(completionService.poll(anyLong(), any())).thenReturn(successfulFuture);
+        when(successfulFuture.get()).thenReturn(dataFetcherResult);
+
+        GetRecordsResult result = strategy.getRecords(10);
+
+        verify(executorService).isShutdown();
+        verify(completionService).submit(any());
+        verify(completionService).poll(eq(RETRY_GET_RECORDS_IN_SECONDS), eq(TimeUnit.SECONDS));
+        verify(successfulFuture).get();
+        verify(successfulFuture).cancel(eq(true));
+
+        assertThat(result, equalTo(expectedResults));
+    }
+
+    @Test
+    public void testBlockedAndSuccessfulFuture() throws Exception {
+        AsynchronousGetRecordsRetrievalStrategy strategy = new AsynchronousGetRecordsRetrievalStrategy(dataFetcher,
+                executorService, (int) RETRY_GET_RECORDS_IN_SECONDS, completionServiceSupplier, SHARD_ID);
+
+        when(executorService.isShutdown()).thenReturn(false);
+        when(completionService.submit(any())).thenReturn(blockedFuture).thenReturn(successfulFuture);
+        when(completionService.poll(anyLong(), any())).thenReturn(null).thenReturn(successfulFuture);
+        when(successfulFuture.get()).thenReturn(dataFetcherResult);
+        when(successfulFuture.cancel(anyBoolean())).thenReturn(false);
+        when(blockedFuture.cancel(anyBoolean())).thenReturn(true);
+        when(successfulFuture.isCancelled()).thenReturn(false);
+        when(blockedFuture.isCancelled()).thenReturn(true);
+
+        GetRecordsResult actualResults = strategy.getRecords(10);
+
+        verify(completionService, times(2)).submit(any());
+        verify(completionService, times(2)).poll(eq(RETRY_GET_RECORDS_IN_SECONDS), eq(TimeUnit.SECONDS));
+        verify(successfulFuture).get();
+        verify(blockedFuture, never()).get();
+        verify(successfulFuture).cancel(eq(true));
+        verify(blockedFuture).cancel(eq(true));
+
+        assertThat(actualResults, equalTo(expectedResults));
+    }
+
+    @Test(expected = IllegalStateException.class)
+    public void testStrategyIsShutdown() throws Exception {
+        AsynchronousGetRecordsRetrievalStrategy strategy = new AsynchronousGetRecordsRetrievalStrategy(dataFetcher,
+                executorService, (int) RETRY_GET_RECORDS_IN_SECONDS, completionServiceSupplier, SHARD_ID);
+
+        when(executorService.isShutdown()).thenReturn(true);
+
+        strategy.getRecords(10);
+    }
+
+    @Test
+    public void testPoolOutOfResources() throws Exception {
+        AsynchronousGetRecordsRetrievalStrategy strategy = new AsynchronousGetRecordsRetrievalStrategy(dataFetcher,
+                executorService, (int) RETRY_GET_RECORDS_IN_SECONDS, completionServiceSupplier, SHARD_ID);
+
+        when(executorService.isShutdown()).thenReturn(false);
+        when(completionService.submit(any())).thenReturn(blockedFuture).thenThrow(new RejectedExecutionException("Rejected!")).thenReturn(successfulFuture);
+        when(completionService.poll(anyLong(), any())).thenReturn(null).thenReturn(null).thenReturn(successfulFuture);
+        when(successfulFuture.get()).thenReturn(dataFetcherResult);
+        when(successfulFuture.cancel(anyBoolean())).thenReturn(false);
+        when(blockedFuture.cancel(anyBoolean())).thenReturn(true);
+        when(successfulFuture.isCancelled()).thenReturn(false);
+        when(blockedFuture.isCancelled()).thenReturn(true);
+
+        GetRecordsResult actualResult = strategy.getRecords(10);
+
+        verify(completionService, times(3)).submit(any());
+        verify(completionService, times(3)).poll(eq(RETRY_GET_RECORDS_IN_SECONDS), eq(TimeUnit.SECONDS));
+        verify(successfulFuture).cancel(eq(true));
+        verify(blockedFuture).cancel(eq(true));
+
+
+        assertThat(actualResult, equalTo(expectedResults));
+    }
+    
+    @Test (expected = ExpiredIteratorException.class)
+    public void testExpiredIteratorExceptionCase() throws Exception {
+        AsynchronousGetRecordsRetrievalStrategy strategy = new AsynchronousGetRecordsRetrievalStrategy(dataFetcher,
+                executorService, (int) RETRY_GET_RECORDS_IN_SECONDS, completionServiceSupplier, SHARD_ID);
+        Future<DataFetcherResult> successfulFuture2 = mock(Future.class);
+
+        when(executorService.isShutdown()).thenReturn(false);
+        when(completionService.submit(any())).thenReturn(successfulFuture, successfulFuture2);
+        when(completionService.poll(anyLong(), any())).thenReturn(null).thenReturn(successfulFuture);
+        when(successfulFuture.get()).thenThrow(new ExecutionException(new ExpiredIteratorException("ExpiredException")));
+        
+        try {
+            strategy.getRecords(10);
+        } finally {
+            verify(executorService).isShutdown();
+            verify(completionService, times(2)).submit(any());
+            verify(completionService, times(2)).poll(eq(RETRY_GET_RECORDS_IN_SECONDS), eq(TimeUnit.SECONDS));
+            verify(successfulFuture).cancel(eq(true));
+            verify(successfulFuture2).cancel(eq(true));
+        }
+    }
+
+}

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/BlockingGetRecordsCacheTest.java

@@ -0,0 +1,83 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License. 
+ */
+
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.when;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import com.amazonaws.services.kinesis.model.Record;
+
+/**
+ * Test class for the BlockingGetRecordsCache class.
+ */
+@RunWith(MockitoJUnitRunner.class)
+public class BlockingGetRecordsCacheTest {
+    private static final int MAX_RECORDS_PER_COUNT = 10_000;
+
+    @Mock
+    private GetRecordsRetrievalStrategy getRecordsRetrievalStrategy;
+    @Mock
+    private GetRecordsResult getRecordsResult;
+    
+    private List<Record> records;
+    private BlockingGetRecordsCache blockingGetRecordsCache;
+
+    @Before
+    public void setup() {
+        records = new ArrayList<>();
+        blockingGetRecordsCache = new BlockingGetRecordsCache(MAX_RECORDS_PER_COUNT, getRecordsRetrievalStrategy);
+
+        when(getRecordsRetrievalStrategy.getRecords(eq(MAX_RECORDS_PER_COUNT))).thenReturn(getRecordsResult);
+        when(getRecordsResult.getRecords()).thenReturn(records);
+    }
+
+    @Test
+    public void testGetNextRecordsWithNoRecords() {
+        ProcessRecordsInput result = blockingGetRecordsCache.getNextResult();
+
+        assertEquals(result.getRecords(), records);
+        assertNull(result.getCacheEntryTime());
+        assertNull(result.getCacheExitTime());
+        assertEquals(result.getTimeSpentInCache(), Duration.ZERO);
+    }
+    
+    @Test
+    public void testGetNextRecordsWithRecords() {
+        Record record = new Record();
+        records.add(record);
+        records.add(record);
+        records.add(record);
+        records.add(record);
+        
+        ProcessRecordsInput result = blockingGetRecordsCache.getNextResult();
+        
+        assertEquals(result.getRecords(), records);
+    }
+}

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ConsumerStatesTest.java

@@ -55,6 +55,8 @@ public class ConsumerStatesTest {
     @Mock
     private IRecordProcessor recordProcessor;
     @Mock
+    private KinesisClientLibConfiguration config;
+    @Mock
     private RecordProcessorCheckpointer recordProcessorCheckpointer;
     @Mock
     private ExecutorService executorService;
@@ -74,6 +76,8 @@ public class ConsumerStatesTest {
     private IKinesisProxy kinesisProxy;
     @Mock
     private InitialPositionInStreamExtended initialPositionInStream;
+    @Mock
+    private GetRecordsCache getRecordsCache;
 
     private long parentShardPollIntervalMillis = 0xCAFE;
     private boolean cleanupLeasesOfCompletedShards = true;
@@ -96,7 +100,7 @@ public class ConsumerStatesTest {
         when(consumer.isCleanupLeasesOfCompletedShards()).thenReturn(cleanupLeasesOfCompletedShards);
         when(consumer.getTaskBackoffTimeMillis()).thenReturn(taskBackoffTimeMillis);
         when(consumer.getShutdownReason()).thenReturn(reason);
-
+        when(consumer.getGetRecordsCache()).thenReturn(getRecordsCache);
     }
 
     private static final Class<ILeaseManager<KinesisClientLease>> LEASE_MANAGER_CLASS = (Class<ILeaseManager<KinesisClientLease>>) (Class<?>) ILeaseManager.class;
@@ -152,7 +156,7 @@ public class ConsumerStatesTest {
     }
 
     @Test
-    public void processingStateTest() {
+    public void processingStateTestSynchronous() {
         ConsumerState state = ShardConsumerState.PROCESSING.getConsumerState();
         ITask task = state.createTask(consumer);
 
@@ -178,6 +182,60 @@ public class ConsumerStatesTest {
 
     }
 
+    @Test
+    public void processingStateTestAsynchronous() {
+        ConsumerState state = ShardConsumerState.PROCESSING.getConsumerState();
+        ITask task = state.createTask(consumer);
+
+        assertThat(task, procTask(ShardInfo.class, "shardInfo", equalTo(shardInfo)));
+        assertThat(task, procTask(IRecordProcessor.class, "recordProcessor", equalTo(recordProcessor)));
+        assertThat(task, procTask(RecordProcessorCheckpointer.class, "recordProcessorCheckpointer",
+                equalTo(recordProcessorCheckpointer)));
+        assertThat(task, procTask(KinesisDataFetcher.class, "dataFetcher", equalTo(dataFetcher)));
+        assertThat(task, procTask(StreamConfig.class, "streamConfig", equalTo(streamConfig)));
+        assertThat(task, procTask(Long.class, "backoffTimeMillis", equalTo(taskBackoffTimeMillis)));
+
+        assertThat(state.successTransition(), equalTo(ShardConsumerState.PROCESSING.getConsumerState()));
+
+        assertThat(state.shutdownTransition(ShutdownReason.ZOMBIE),
+                equalTo(ShardConsumerState.SHUTTING_DOWN.getConsumerState()));
+        assertThat(state.shutdownTransition(ShutdownReason.TERMINATE),
+                equalTo(ShardConsumerState.SHUTTING_DOWN.getConsumerState()));
+        assertThat(state.shutdownTransition(ShutdownReason.REQUESTED),
+                equalTo(ShardConsumerState.SHUTDOWN_REQUESTED.getConsumerState()));
+
+        assertThat(state.getState(), equalTo(ShardConsumerState.PROCESSING));
+        assertThat(state.getTaskType(), equalTo(TaskType.PROCESS));
+
+    }
+
+    @Test
+    public void processingStateRecordsFetcher() {
+
+        ConsumerState state = ShardConsumerState.PROCESSING.getConsumerState();
+        ITask task = state.createTask(consumer);
+
+        assertThat(task, procTask(ShardInfo.class, "shardInfo", equalTo(shardInfo)));
+        assertThat(task, procTask(IRecordProcessor.class, "recordProcessor", equalTo(recordProcessor)));
+        assertThat(task, procTask(RecordProcessorCheckpointer.class, "recordProcessorCheckpointer",
+                equalTo(recordProcessorCheckpointer)));
+        assertThat(task, procTask(KinesisDataFetcher.class, "dataFetcher", equalTo(dataFetcher)));
+        assertThat(task, procTask(StreamConfig.class, "streamConfig", equalTo(streamConfig)));
+        assertThat(task, procTask(Long.class, "backoffTimeMillis", equalTo(taskBackoffTimeMillis)));
+
+        assertThat(state.successTransition(), equalTo(ShardConsumerState.PROCESSING.getConsumerState()));
+
+        assertThat(state.shutdownTransition(ShutdownReason.ZOMBIE),
+                equalTo(ShardConsumerState.SHUTTING_DOWN.getConsumerState()));
+        assertThat(state.shutdownTransition(ShutdownReason.TERMINATE),
+                equalTo(ShardConsumerState.SHUTTING_DOWN.getConsumerState()));
+        assertThat(state.shutdownTransition(ShutdownReason.REQUESTED),
+                equalTo(ShardConsumerState.SHUTDOWN_REQUESTED.getConsumerState()));
+
+        assertThat(state.getState(), equalTo(ShardConsumerState.PROCESSING));
+        assertThat(state.getTaskType(), equalTo(TaskType.PROCESS));
+    }
+
     @Test
     public void shutdownRequestState() {
         ConsumerState state = ShardConsumerState.SHUTDOWN_REQUESTED.getConsumerState();
@@ -284,7 +342,7 @@ public class ConsumerStatesTest {
     }
 
     static <ValueType> ReflectionPropertyMatcher<ShutdownTask, ValueType> shutdownTask(Class<ValueType> valueTypeClass,
-            String propertyName, Matcher<ValueType> matcher) {
+                                                                                       String propertyName, Matcher<ValueType> matcher) {
         return taskWith(ShutdownTask.class, valueTypeClass, propertyName, matcher);
     }
 
@@ -294,17 +352,17 @@ public class ConsumerStatesTest {
     }
 
     static <ValueType> ReflectionPropertyMatcher<ProcessTask, ValueType> procTask(Class<ValueType> valueTypeClass,
-            String propertyName, Matcher<ValueType> matcher) {
+                                                                                  String propertyName, Matcher<ValueType> matcher) {
         return taskWith(ProcessTask.class, valueTypeClass, propertyName, matcher);
     }
 
     static <ValueType> ReflectionPropertyMatcher<InitializeTask, ValueType> initTask(Class<ValueType> valueTypeClass,
-            String propertyName, Matcher<ValueType> matcher) {
+                                                                                     String propertyName, Matcher<ValueType> matcher) {
         return taskWith(InitializeTask.class, valueTypeClass, propertyName, matcher);
     }
 
     static <TaskType, ValueType> ReflectionPropertyMatcher<TaskType, ValueType> taskWith(Class<TaskType> taskTypeClass,
-            Class<ValueType> valueTypeClass, String propertyName, Matcher<ValueType> matcher) {
+                                                                                         Class<ValueType> valueTypeClass, String propertyName, Matcher<ValueType> matcher) {
         return new ReflectionPropertyMatcher<>(taskTypeClass, valueTypeClass, matcher, propertyName);
     }
 
@@ -317,7 +375,7 @@ public class ConsumerStatesTest {
         private final Field matchingField;
 
         private ReflectionPropertyMatcher(Class<TaskType> taskTypeClass, Class<ValueType> valueTypeClass,
-                Matcher<ValueType> matcher, String propertyName) {
+                                          Matcher<ValueType> matcher, String propertyName) {
             this.taskTypeClass = taskTypeClass;
             this.valueTypeClazz = valueTypeClass;
             this.matcher = matcher;

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfigurationTest.java

@@ -1,25 +1,26 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.fail;
 
-import junit.framework.Assert;
+import java.util.Date;
 
 import org.junit.Test;
 import org.mockito.Mockito;
@@ -35,7 +36,7 @@ import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorF
 import com.amazonaws.services.kinesis.metrics.interfaces.MetricsLevel;
 import com.google.common.collect.ImmutableSet;
 
-import java.util.Date;
+import junit.framework.Assert;
 
 public class KinesisClientLibConfigurationTest {
     private static final long INVALID_LONG = 0L;
@@ -84,7 +85,8 @@ public class KinesisClientLibConfigurationTest {
                         TEST_VALUE_LONG,
                         TEST_VALUE_INT,
                         skipCheckpointValidationValue,
-                        null);
+                        null,
+                        TEST_VALUE_LONG);
     }
 
     @Test
@@ -94,7 +96,8 @@ public class KinesisClientLibConfigurationTest {
         // Try each argument at one time.
         KinesisClientLibConfiguration config = null;
         long[] longValues =
-        { TEST_VALUE_LONG, TEST_VALUE_LONG, TEST_VALUE_LONG, TEST_VALUE_LONG, TEST_VALUE_LONG, TEST_VALUE_LONG };
+                { TEST_VALUE_LONG, TEST_VALUE_LONG, TEST_VALUE_LONG, TEST_VALUE_LONG, TEST_VALUE_LONG, TEST_VALUE_LONG,
+                        TEST_VALUE_LONG };
         for (int i = 0; i < PARAMETER_COUNT; i++) {
             longValues[i] = INVALID_LONG;
             try {
@@ -122,7 +125,8 @@ public class KinesisClientLibConfigurationTest {
                                 longValues[5],
                                 TEST_VALUE_INT,
                                 skipCheckpointValidationValue,
-                                null);
+                                null,
+                                longValues[6]);
             } catch (IllegalArgumentException e) {
                 System.out.println(e.getMessage());
             }
@@ -156,7 +160,8 @@ public class KinesisClientLibConfigurationTest {
                                 TEST_VALUE_LONG,
                                 intValues[1],
                                 skipCheckpointValidationValue,
-                                null);
+                                null,
+                                TEST_VALUE_LONG);
             } catch (IllegalArgumentException e) {
                 System.out.println(e.getMessage());
             }
@@ -297,29 +302,30 @@ public class KinesisClientLibConfigurationTest {
         Mockito.mock(AWSCredentialsProvider.class);
         try {
             new KinesisClientLibConfiguration(TEST_STRING,
-                        TEST_STRING,
+                    TEST_STRING,
-                        TEST_STRING,
+                    TEST_STRING,
-                        TEST_STRING,
+                    TEST_STRING,
-                        null,
+                    null,
-                        null,
+                    null,
-                        null,
+                    null,
-                        null,
+                    null,
-                        TEST_VALUE_LONG,
+                    TEST_VALUE_LONG,
-                        TEST_STRING,
+                    TEST_STRING,
-                        3,
+                    3,
-                        TEST_VALUE_LONG,
+                    TEST_VALUE_LONG,
-                        false,
+                    false,
-                        TEST_VALUE_LONG,
+                    TEST_VALUE_LONG,
-                        TEST_VALUE_LONG,
+                    TEST_VALUE_LONG,
-                        true,
+                    true,
-                        new ClientConfiguration(),
+                    new ClientConfiguration(),
-                        new ClientConfiguration(),
+                    new ClientConfiguration(),
-                        new ClientConfiguration(),
+                    new ClientConfiguration(),
-                        TEST_VALUE_LONG,
+                    TEST_VALUE_LONG,
-                        TEST_VALUE_LONG,
+                    TEST_VALUE_LONG,
-                        1,
+                    1,
-                        skipCheckpointValidationValue,
+                    skipCheckpointValidationValue,
-                        "abcd");
+                    "abcd",
+                    TEST_VALUE_LONG);
             Assert.fail("No expected Exception is thrown.");
         } catch(IllegalArgumentException e) {
             System.out.println(e.getMessage());
@@ -402,4 +408,14 @@ public class KinesisClientLibConfigurationTest {
             fail("Should not have thrown");
         }
     }
+
+    @Test
+    public void testKCLConfigurationWithIgnoreUnexpectedChildShards() {
+        KinesisClientLibConfiguration config =
+                new KinesisClientLibConfiguration("TestApplication", "TestStream", null, "TestWorker");
+        // By default, unexpected child shards should not be ignored.
+        assertFalse(config.shouldIgnoreUnexpectedChildShards());
+        config = config.withIgnoreUnexpectedChildShards(true);
+        assertTrue(config.shouldIgnoreUnexpectedChildShards());
+    }
 }

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisDataFetcherTest.java

@@ -1,36 +1,48 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.notNullValue;
+import static org.hamcrest.CoreMatchers.nullValue;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.collection.IsEmptyCollection.empty;
+import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
 import static org.mockito.Mockito.doReturn;
 import static org.mockito.Mockito.doThrow;
 import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.reset;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
 
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.Date;
 import java.util.List;
 
 import org.junit.Assert;
 import org.junit.BeforeClass;
 import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
 
-import com.amazonaws.services.kinesis.model.GetRecordsResult;
-import com.amazonaws.services.kinesis.model.Record;
-import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
-import com.amazonaws.services.kinesis.model.ShardIteratorType;
 import com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibException;
 import com.amazonaws.services.kinesis.clientlibrary.interfaces.ICheckpoint;
 import com.amazonaws.services.kinesis.clientlibrary.lib.checkpoint.SentinelCheckpoint;
@@ -39,12 +51,20 @@ import com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisProxy;
 import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
 import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
 import com.amazonaws.services.kinesis.metrics.impl.NullMetricsFactory;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import com.amazonaws.services.kinesis.model.Record;
+import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
+import com.amazonaws.services.kinesis.model.ShardIteratorType;
 
 /**
  * Unit tests for KinesisDataFetcher.
  */
+@RunWith(MockitoJUnitRunner.class)
 public class KinesisDataFetcherTest {
 
+    @Mock
+    private KinesisProxy kinesisProxy;
+
     private static final int MAX_RECORDS = 1;
     private static final String SHARD_ID = "shardId-1";
     private static final String AT_SEQUENCE_NUMBER = ShardIteratorType.AT_SEQUENCE_NUMBER.toString();
@@ -117,6 +137,7 @@ public class KinesisDataFetcherTest {
         ICheckpoint checkpoint = mock(ICheckpoint.class);
 
         KinesisDataFetcher fetcher = new KinesisDataFetcher(kinesis, SHARD_INFO);
+        GetRecordsRetrievalStrategy getRecordsRetrievalStrategy = new SynchronousGetRecordsRetrievalStrategy(fetcher);
 
         String iteratorA = "foo";
         String iteratorB = "bar";
@@ -138,10 +159,10 @@ public class KinesisDataFetcherTest {
         fetcher.initialize(seqA, null);
 
         fetcher.advanceIteratorTo(seqA, null);
-        Assert.assertEquals(recordsA, fetcher.getRecords(MAX_RECORDS).getRecords());
+        Assert.assertEquals(recordsA, getRecordsRetrievalStrategy.getRecords(MAX_RECORDS).getRecords());
 
         fetcher.advanceIteratorTo(seqB, null);
-        Assert.assertEquals(recordsB, fetcher.getRecords(MAX_RECORDS).getRecords());
+        Assert.assertEquals(recordsB, getRecordsRetrievalStrategy.getRecords(MAX_RECORDS).getRecords());
     }
 
     @Test
@@ -181,12 +202,149 @@ public class KinesisDataFetcherTest {
         // Create data fectcher and initialize it with latest type checkpoint
         KinesisDataFetcher dataFetcher = new KinesisDataFetcher(mockProxy, SHARD_INFO);
         dataFetcher.initialize(SentinelCheckpoint.LATEST.toString(), INITIAL_POSITION_LATEST);
+        GetRecordsRetrievalStrategy getRecordsRetrievalStrategy = new SynchronousGetRecordsRetrievalStrategy(dataFetcher);
         // Call getRecords of dataFetcher which will throw an exception
-        dataFetcher.getRecords(maxRecords);
+        getRecordsRetrievalStrategy.getRecords(maxRecords);
 
         // Test shard has reached the end
         Assert.assertTrue("Shard should reach the end", dataFetcher.isShardEndReached());
     }
+    
+    @Test
+    public void testNonNullGetRecords() {
+        String nextIterator = "TestIterator";
+        int maxRecords = 100;
+        
+        KinesisProxy mockProxy = mock(KinesisProxy.class);
+        doThrow(new ResourceNotFoundException("Test Exception")).when(mockProxy).get(nextIterator, maxRecords);
+
+        KinesisDataFetcher dataFetcher = new KinesisDataFetcher(mockProxy, SHARD_INFO);
+        dataFetcher.initialize(SentinelCheckpoint.LATEST.toString(), INITIAL_POSITION_LATEST);
+        
+        DataFetcherResult dataFetcherResult = dataFetcher.getRecords(maxRecords);
+        
+        assertThat(dataFetcherResult, notNullValue());
+    }
+
+    @Test
+    public void testFetcherDoesNotAdvanceWithoutAccept() {
+        final String INITIAL_ITERATOR = "InitialIterator";
+        final String NEXT_ITERATOR_ONE = "NextIteratorOne";
+        final String NEXT_ITERATOR_TWO = "NextIteratorTwo";
+        when(kinesisProxy.getIterator(anyString(), anyString())).thenReturn(INITIAL_ITERATOR);
+        GetRecordsResult iteratorOneResults = mock(GetRecordsResult.class);
+        when(iteratorOneResults.getNextShardIterator()).thenReturn(NEXT_ITERATOR_ONE);
+        when(kinesisProxy.get(eq(INITIAL_ITERATOR), anyInt())).thenReturn(iteratorOneResults);
+
+        GetRecordsResult iteratorTwoResults = mock(GetRecordsResult.class);
+        when(kinesisProxy.get(eq(NEXT_ITERATOR_ONE), anyInt())).thenReturn(iteratorTwoResults);
+        when(iteratorTwoResults.getNextShardIterator()).thenReturn(NEXT_ITERATOR_TWO);
+
+        GetRecordsResult finalResult = mock(GetRecordsResult.class);
+        when(kinesisProxy.get(eq(NEXT_ITERATOR_TWO), anyInt())).thenReturn(finalResult);
+        when(finalResult.getNextShardIterator()).thenReturn(null);
+
+        KinesisDataFetcher dataFetcher = new KinesisDataFetcher(kinesisProxy, SHARD_INFO);
+        dataFetcher.initialize("TRIM_HORIZON",
+                InitialPositionInStreamExtended.newInitialPosition(InitialPositionInStream.TRIM_HORIZON));
+
+        assertNoAdvance(dataFetcher, iteratorOneResults, INITIAL_ITERATOR);
+        assertAdvanced(dataFetcher, iteratorOneResults, INITIAL_ITERATOR, NEXT_ITERATOR_ONE);
+
+        assertNoAdvance(dataFetcher, iteratorTwoResults, NEXT_ITERATOR_ONE);
+        assertAdvanced(dataFetcher, iteratorTwoResults, NEXT_ITERATOR_ONE, NEXT_ITERATOR_TWO);
+
+        assertNoAdvance(dataFetcher, finalResult, NEXT_ITERATOR_TWO);
+        assertAdvanced(dataFetcher, finalResult, NEXT_ITERATOR_TWO, null);
+
+        verify(kinesisProxy, times(2)).get(eq(INITIAL_ITERATOR), anyInt());
+        verify(kinesisProxy, times(2)).get(eq(NEXT_ITERATOR_ONE), anyInt());
+        verify(kinesisProxy, times(2)).get(eq(NEXT_ITERATOR_TWO), anyInt());
+
+        reset(kinesisProxy);
+
+        DataFetcherResult terminal = dataFetcher.getRecords(100);
+        assertThat(terminal.isShardEnd(), equalTo(true));
+        assertThat(terminal.getResult(), notNullValue());
+        GetRecordsResult terminalResult = terminal.getResult();
+        assertThat(terminalResult.getRecords(), notNullValue());
+        assertThat(terminalResult.getRecords(), empty());
+        assertThat(terminalResult.getNextShardIterator(), nullValue());
+        assertThat(terminal, equalTo(dataFetcher.TERMINAL_RESULT));
+
+        verify(kinesisProxy, never()).get(anyString(), anyInt());
+    }
+    
+    @Test
+    public void testRestartIterator() {
+        GetRecordsResult getRecordsResult = mock(GetRecordsResult.class);
+        GetRecordsResult restartGetRecordsResult = new GetRecordsResult();
+        Record record = mock(Record.class);
+        final String initialIterator = "InitialIterator";
+        final String nextShardIterator = "NextShardIterator";
+        final String restartShardIterator = "RestartIterator";
+        final String sequenceNumber = "SequenceNumber";
+        final String iteratorType = "AT_SEQUENCE_NUMBER";
+        KinesisProxy kinesisProxy = mock(KinesisProxy.class);
+        KinesisDataFetcher fetcher = new KinesisDataFetcher(kinesisProxy, SHARD_INFO);
+        
+        when(kinesisProxy.getIterator(eq(SHARD_ID), eq(InitialPositionInStream.LATEST.toString()))).thenReturn(initialIterator);
+        when(kinesisProxy.get(eq(initialIterator), eq(10))).thenReturn(getRecordsResult);
+        when(getRecordsResult.getRecords()).thenReturn(Collections.singletonList(record));
+        when(getRecordsResult.getNextShardIterator()).thenReturn(nextShardIterator);
+        when(record.getSequenceNumber()).thenReturn(sequenceNumber);
+
+        fetcher.initialize(InitialPositionInStream.LATEST.toString(), INITIAL_POSITION_LATEST);
+        verify(kinesisProxy).getIterator(eq(SHARD_ID), eq(InitialPositionInStream.LATEST.toString()));
+        Assert.assertEquals(getRecordsResult, fetcher.getRecords(10).accept());
+        verify(kinesisProxy).get(eq(initialIterator), eq(10));
+
+        when(kinesisProxy.getIterator(eq(SHARD_ID), eq(iteratorType), eq(sequenceNumber))).thenReturn(restartShardIterator);
+        when(kinesisProxy.get(eq(restartShardIterator), eq(10))).thenReturn(restartGetRecordsResult);
+
+        fetcher.restartIterator();
+        Assert.assertEquals(restartGetRecordsResult, fetcher.getRecords(10).accept());
+        verify(kinesisProxy).getIterator(eq(SHARD_ID), eq(iteratorType), eq(sequenceNumber));
+        verify(kinesisProxy).get(eq(restartShardIterator), eq(10));
+    }
+    
+    @Test (expected = IllegalStateException.class)
+    public void testRestartIteratorNotInitialized() {
+        KinesisDataFetcher dataFetcher = new KinesisDataFetcher(kinesisProxy, SHARD_INFO);
+        dataFetcher.restartIterator();
+    }
+
+    private DataFetcherResult assertAdvanced(KinesisDataFetcher dataFetcher, GetRecordsResult expectedResult,
+            String previousValue, String nextValue) {
+        DataFetcherResult acceptResult = dataFetcher.getRecords(100);
+        assertThat(acceptResult.getResult(), equalTo(expectedResult));
+
+        assertThat(dataFetcher.getNextIterator(), equalTo(previousValue));
+        assertThat(dataFetcher.isShardEndReached(), equalTo(false));
+
+        assertThat(acceptResult.accept(), equalTo(expectedResult));
+        assertThat(dataFetcher.getNextIterator(), equalTo(nextValue));
+        if (nextValue == null) {
+            assertThat(dataFetcher.isShardEndReached(), equalTo(true));
+        }
+
+        verify(kinesisProxy, times(2)).get(eq(previousValue), anyInt());
+
+        return acceptResult;
+    }
+
+    private DataFetcherResult assertNoAdvance(KinesisDataFetcher dataFetcher, GetRecordsResult expectedResult,
+            String previousValue) {
+        assertThat(dataFetcher.getNextIterator(), equalTo(previousValue));
+        DataFetcherResult noAcceptResult = dataFetcher.getRecords(100);
+        assertThat(noAcceptResult.getResult(), equalTo(expectedResult));
+
+        assertThat(dataFetcher.getNextIterator(), equalTo(previousValue));
+
+        verify(kinesisProxy).get(eq(previousValue), anyInt());
+
+        return noAcceptResult;
+    }
 
     private void testInitializeAndFetch(String iteratorType,
             String seqNo,
@@ -206,8 +364,9 @@ public class KinesisDataFetcherTest {
         when(checkpoint.getCheckpoint(SHARD_ID)).thenReturn(new ExtendedSequenceNumber(seqNo));
 
         KinesisDataFetcher fetcher = new KinesisDataFetcher(kinesis, SHARD_INFO);
+        GetRecordsRetrievalStrategy getRecordsRetrievalStrategy = new SynchronousGetRecordsRetrievalStrategy(fetcher);
         fetcher.initialize(seqNo, initialPositionInStream);
-        List<Record> actualRecords = fetcher.getRecords(MAX_RECORDS).getRecords();
+        List<Record> actualRecords = getRecordsRetrievalStrategy.getRecords(MAX_RECORDS).getRecords();
 
         Assert.assertEquals(expectedRecords, actualRecords);
     }

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PrefetchGetRecordsCacheIntegrationTest.java

@@ -0,0 +1,222 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License. 
+ */
+
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.runners.MockitoJUnitRunner;
+import org.mockito.stubbing.Answer;
+
+import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
+import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
+import com.amazonaws.services.kinesis.metrics.impl.NullMetricsFactory;
+import com.amazonaws.services.kinesis.model.ExpiredIteratorException;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import com.amazonaws.services.kinesis.model.Record;
+
+import lombok.extern.apachecommons.CommonsLog;
+
+/**
+ * These are the integration tests for the PrefetchGetRecordsCache class. 
+ */
+@RunWith(MockitoJUnitRunner.class)
+@CommonsLog
+public class PrefetchGetRecordsCacheIntegrationTest {
+    private static final int MAX_SIZE = 3;
+    private static final int MAX_BYTE_SIZE = 5 * 1024 * 1024;
+    private static final int MAX_RECORDS_COUNT = 30_000;
+    private static final int MAX_RECORDS_PER_CALL = 10_000;
+    private static final long IDLE_MILLIS_BETWEEN_CALLS = 500L;
+    
+    private PrefetchGetRecordsCache getRecordsCache;
+    private GetRecordsRetrievalStrategy getRecordsRetrievalStrategy;
+    private KinesisDataFetcher dataFetcher;
+    private ExecutorService executorService;
+    private List<Record> records;
+    private String operation = "ProcessTask";
+    
+    @Mock
+    private IKinesisProxy proxy;
+    @Mock
+    private ShardInfo shardInfo;
+    
+    @Before
+    public void setup() {
+        records = new ArrayList<>();
+        dataFetcher = spy(new KinesisDataFetcherForTest(proxy, shardInfo));
+        getRecordsRetrievalStrategy = spy(new SynchronousGetRecordsRetrievalStrategy(dataFetcher));
+        executorService = spy(Executors.newFixedThreadPool(1));
+        
+        getRecordsCache = new PrefetchGetRecordsCache(MAX_SIZE,
+                MAX_BYTE_SIZE,
+                MAX_RECORDS_COUNT,
+                MAX_RECORDS_PER_CALL,
+                getRecordsRetrievalStrategy,
+                executorService,
+                IDLE_MILLIS_BETWEEN_CALLS,
+                new NullMetricsFactory(),
+                operation,
+                "test-shard");
+    }
+    
+    @Test
+    public void testRollingCache() {
+        getRecordsCache.start();
+        sleep(IDLE_MILLIS_BETWEEN_CALLS);
+        
+        ProcessRecordsInput processRecordsInput1 = getRecordsCache.getNextResult();
+        
+        assertTrue(processRecordsInput1.getRecords().isEmpty());
+        assertEquals(processRecordsInput1.getMillisBehindLatest(), new Long(1000));
+        assertNotNull(processRecordsInput1.getCacheEntryTime());
+        
+        ProcessRecordsInput processRecordsInput2 = getRecordsCache.getNextResult();
+        
+        assertNotEquals(processRecordsInput1, processRecordsInput2);
+    }
+    
+    @Test
+    public void testFullCache() {
+        getRecordsCache.start();
+        sleep(MAX_SIZE * IDLE_MILLIS_BETWEEN_CALLS);
+        
+        assertEquals(getRecordsCache.getRecordsResultQueue.size(), MAX_SIZE);
+        
+        ProcessRecordsInput processRecordsInput1 = getRecordsCache.getNextResult();
+        ProcessRecordsInput processRecordsInput2 = getRecordsCache.getNextResult();
+        
+        assertNotEquals(processRecordsInput1, processRecordsInput2);
+    }
+    
+    @Test
+    public void testDifferentShardCaches() {
+        ExecutorService executorService2 = spy(Executors.newFixedThreadPool(1));
+        KinesisDataFetcher kinesisDataFetcher = spy(new KinesisDataFetcherForTest(proxy, shardInfo));
+        GetRecordsRetrievalStrategy getRecordsRetrievalStrategy2 = spy(new AsynchronousGetRecordsRetrievalStrategy(kinesisDataFetcher, 5 , 5, "Test-shard"));
+        GetRecordsCache getRecordsCache2 = new PrefetchGetRecordsCache(
+                MAX_SIZE,
+                MAX_BYTE_SIZE,
+                MAX_RECORDS_COUNT,
+                MAX_RECORDS_PER_CALL,
+                getRecordsRetrievalStrategy2,
+                executorService2,
+                IDLE_MILLIS_BETWEEN_CALLS,
+                new NullMetricsFactory(),
+                operation,
+                "test-shard-2");
+        
+        getRecordsCache.start();
+        sleep(IDLE_MILLIS_BETWEEN_CALLS);
+        
+        Record record = mock(Record.class);
+        ByteBuffer byteBuffer = ByteBuffer.allocate(512 * 1024);
+        when(record.getData()).thenReturn(byteBuffer);
+        
+        records.add(record);
+        records.add(record);
+        records.add(record);
+        records.add(record);
+        getRecordsCache2.start();
+        
+        sleep(IDLE_MILLIS_BETWEEN_CALLS);
+        
+        ProcessRecordsInput p1 = getRecordsCache.getNextResult();
+        
+        ProcessRecordsInput p2 = getRecordsCache2.getNextResult();
+        
+        assertNotEquals(p1, p2);
+        assertTrue(p1.getRecords().isEmpty());
+        assertFalse(p2.getRecords().isEmpty());
+        assertEquals(p2.getRecords().size(), records.size());
+
+        getRecordsCache2.shutdown();
+        sleep(100L);
+        verify(executorService2).shutdownNow();
+        verify(getRecordsRetrievalStrategy2).shutdown();
+    }
+    
+    @Test
+    public void testExpiredIteratorException() {
+        when(dataFetcher.getRecords(eq(MAX_RECORDS_PER_CALL))).thenAnswer(new Answer<DataFetcherResult>() {
+            @Override
+            public DataFetcherResult answer(final InvocationOnMock invocationOnMock) throws Throwable {
+                throw new ExpiredIteratorException("ExpiredIterator");
+            }
+        }).thenCallRealMethod();
+        doNothing().when(dataFetcher).restartIterator();
+        
+        getRecordsCache.start();
+        sleep(IDLE_MILLIS_BETWEEN_CALLS);
+
+        ProcessRecordsInput processRecordsInput = getRecordsCache.getNextResult();
+        
+        assertNotNull(processRecordsInput);
+        assertTrue(processRecordsInput.getRecords().isEmpty());
+        verify(dataFetcher).restartIterator();
+    }
+    
+    @After
+    public void shutdown() {
+        getRecordsCache.shutdown();
+        sleep(100L);
+        verify(executorService).shutdownNow();
+        verify(getRecordsRetrievalStrategy).shutdown();
+    }
+    
+    private void sleep(long millis) {
+        try {
+            Thread.sleep(millis);
+        } catch (InterruptedException e) {}
+    }
+    
+    private class KinesisDataFetcherForTest extends KinesisDataFetcher {
+        public KinesisDataFetcherForTest(final IKinesisProxy kinesisProxy,
+                                         final ShardInfo shardInfo) {
+            super(kinesisProxy, shardInfo);
+        }
+
+        @Override
+        public DataFetcherResult getRecords(final int maxRecords) {
+            GetRecordsResult getRecordsResult = new GetRecordsResult();
+            getRecordsResult.setRecords(new ArrayList<>(records));
+            getRecordsResult.setMillisBehindLatest(1000L);
+
+            return new AdvancingResult(getRecordsResult);
+        }
+    }
+}

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PrefetchGetRecordsCacheTest.java

@@ -0,0 +1,236 @@
+/*
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Amazon Software License (the "License").
+ *  You may not use this file except in compliance with the License.
+ *  A copy of the License is located at
+ *
+ *  http://aws.amazon.com/asl/
+ *
+ *  or in the "license" file accompanying this file. This file is distributed
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  express or implied. See the License for the specific language governing
+ *  permissions and limitations under the License. 
+ */
+
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.atLeast;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.stream.IntStream;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
+import com.amazonaws.services.kinesis.metrics.impl.NullMetricsFactory;
+import com.amazonaws.services.kinesis.model.ExpiredIteratorException;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import com.amazonaws.services.kinesis.model.Record;
+
+/**
+ * Test class for the PrefetchGetRecordsCache class.
+ */
+@RunWith(MockitoJUnitRunner.class)
+public class PrefetchGetRecordsCacheTest {
+    private static final int SIZE_512_KB = 512 * 1024;
+    private static final int SIZE_1_MB = 2 * SIZE_512_KB;
+    private static final int MAX_RECORDS_PER_CALL = 10000;
+    private static final int MAX_SIZE = 5;
+    private static final int MAX_RECORDS_COUNT = 15000;
+    private static final long IDLE_MILLIS_BETWEEN_CALLS = 0L;
+
+    @Mock
+    private GetRecordsRetrievalStrategy getRecordsRetrievalStrategy;
+    @Mock
+    private GetRecordsResult getRecordsResult;
+    @Mock
+    private Record record;
+    @Mock
+    private KinesisDataFetcher dataFetcher;
+
+    private List<Record> records;
+    private ExecutorService executorService;
+    private LinkedBlockingQueue<ProcessRecordsInput> spyQueue;
+    private PrefetchGetRecordsCache getRecordsCache;
+    private String operation = "ProcessTask";
+
+    @Before
+    public void setup() {
+        when(getRecordsRetrievalStrategy.getDataFetcher()).thenReturn(dataFetcher);
+        
+        executorService = spy(Executors.newFixedThreadPool(1));
+        getRecordsCache = new PrefetchGetRecordsCache(
+                MAX_SIZE,
+                3 * SIZE_1_MB,
+                MAX_RECORDS_COUNT,
+                MAX_RECORDS_PER_CALL,
+                getRecordsRetrievalStrategy,
+                executorService,
+                IDLE_MILLIS_BETWEEN_CALLS,
+                new NullMetricsFactory(),
+                operation,
+                "shardId");
+        spyQueue = spy(getRecordsCache.getRecordsResultQueue);
+        records = spy(new ArrayList<>());
+
+        when(getRecordsRetrievalStrategy.getRecords(eq(MAX_RECORDS_PER_CALL))).thenReturn(getRecordsResult);
+        when(getRecordsResult.getRecords()).thenReturn(records);
+    }
+
+    @Test
+    public void testGetRecords() {
+        when(records.size()).thenReturn(1000);
+        when(record.getData()).thenReturn(createByteBufferWithSize(SIZE_512_KB));
+
+        records.add(record);
+        records.add(record);
+        records.add(record);
+        records.add(record);
+        records.add(record);
+
+        getRecordsCache.start();
+        ProcessRecordsInput result = getRecordsCache.getNextResult();
+
+        assertEquals(result.getRecords(), records);
+
+        verify(executorService).execute(any());
+        verify(getRecordsRetrievalStrategy, atLeast(1)).getRecords(eq(MAX_RECORDS_PER_CALL));
+    }
+
+    @Test
+    public void testFullCacheByteSize() {
+        when(records.size()).thenReturn(500);
+        when(record.getData()).thenReturn(createByteBufferWithSize(SIZE_1_MB));
+
+        records.add(record);
+
+        getRecordsCache.start();
+
+        // Sleep for a few seconds for the cache to fill up.
+        sleep(2000);
+
+        verify(getRecordsRetrievalStrategy, times(3)).getRecords(eq(MAX_RECORDS_PER_CALL));
+        assertEquals(spyQueue.size(), 3);
+    }
+
+    @Test
+    public void testFullCacheRecordsCount() {
+        int recordsSize = 4500;
+        when(records.size()).thenReturn(recordsSize);
+
+        getRecordsCache.start();
+
+        sleep(2000);
+
+        int callRate = (int) Math.ceil((double) MAX_RECORDS_COUNT/recordsSize);
+        verify(getRecordsRetrievalStrategy, times(callRate)).getRecords(MAX_RECORDS_PER_CALL);
+        assertEquals(spyQueue.size(), callRate);
+        assertTrue(callRate < MAX_SIZE);
+    }
+
+    @Test
+    public void testFullCacheSize() {
+        int recordsSize = 200;
+        when(records.size()).thenReturn(recordsSize);
+
+        getRecordsCache.start();
+
+        // Sleep for a few seconds for the cache to fill up.
+        sleep(2000);
+
+        verify(getRecordsRetrievalStrategy, times(MAX_SIZE + 1)).getRecords(eq(MAX_RECORDS_PER_CALL));
+        assertEquals(spyQueue.size(), MAX_SIZE);
+    }
+
+    @Test
+    public void testMultipleCacheCalls() {
+        int recordsSize = 20;
+        when(record.getData()).thenReturn(createByteBufferWithSize(1024));
+
+        IntStream.range(0, recordsSize).forEach(i -> records.add(record));
+
+        getRecordsCache.start();
+        ProcessRecordsInput processRecordsInput = getRecordsCache.getNextResult();
+
+        verify(executorService).execute(any());
+        assertEquals(processRecordsInput.getRecords(), records);
+        assertNotNull(processRecordsInput.getCacheEntryTime());
+        assertNotNull(processRecordsInput.getCacheExitTime());
+
+        sleep(2000);
+
+        ProcessRecordsInput processRecordsInput2 = getRecordsCache.getNextResult();
+        assertNotEquals(processRecordsInput, processRecordsInput2);
+        assertEquals(processRecordsInput2.getRecords(), records);
+        assertNotEquals(processRecordsInput2.getTimeSpentInCache(), Duration.ZERO);
+
+        assertTrue(spyQueue.size() <= MAX_SIZE);
+    }
+    
+    @Test(expected = IllegalStateException.class)
+    public void testGetNextRecordsWithoutStarting() {
+        verify(executorService, times(0)).execute(any());
+        getRecordsCache.getNextResult();
+    }
+    
+    @Test(expected = IllegalStateException.class)
+    public void testCallAfterShutdown() {
+        when(executorService.isShutdown()).thenReturn(true);
+        getRecordsCache.getNextResult();
+    }
+    
+    @Test
+    public void testExpiredIteratorException() {
+        getRecordsCache.start();
+        
+        when(getRecordsRetrievalStrategy.getRecords(MAX_RECORDS_PER_CALL)).thenThrow(ExpiredIteratorException.class).thenReturn(getRecordsResult);
+        doNothing().when(dataFetcher).restartIterator();
+        
+        getRecordsCache.getNextResult();
+        
+        sleep(1000);
+        
+        verify(dataFetcher).restartIterator();
+    }
+
+    @After
+    public void shutdown() {
+        getRecordsCache.shutdown();
+        verify(executorService).shutdownNow();
+    }
+
+    private void sleep(long millis) {
+        try {
+            Thread.sleep(millis);
+        } catch (InterruptedException e) {}
+    }
+
+    private ByteBuffer createByteBufferWithSize(int size) {
+        ByteBuffer byteBuffer = ByteBuffer.allocate(size);
+        byteBuffer.put(new byte[size]);
+        return byteBuffer;
+    }
+}

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ProcessTaskTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
@@ -18,8 +18,7 @@ import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertSame;
 import static org.junit.Assert.assertTrue;
-import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.any;
-import static org.mockito.Matchers.anyString;
 import static org.mockito.Mockito.doReturn;
 import static org.mockito.Mockito.doThrow;
 import static org.mockito.Mockito.never;
@@ -49,7 +48,6 @@ import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber
 import com.amazonaws.services.kinesis.clientlibrary.types.Messages.AggregatedRecord;
 import com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput;
 import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
-import com.amazonaws.services.kinesis.model.GetRecordsResult;
 import com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException;
 import com.amazonaws.services.kinesis.model.Record;
 import com.google.protobuf.ByteString;
@@ -76,6 +74,8 @@ public class ProcessTaskTest {
     private @Mock RecordProcessorCheckpointer mockCheckpointer;
     @Mock
     private ThrottlingReporter throttlingReporter;
+    @Mock
+    private GetRecordsCache getRecordsCache;
 
     private List<Record> processedRecords;
     private ExtendedSequenceNumber newLargestPermittedCheckpointValue;
@@ -93,30 +93,39 @@ public class ProcessTaskTest {
                         INITIAL_POSITION_LATEST);
         final ShardInfo shardInfo = new ShardInfo(shardId, null, null, null);
         processTask = new ProcessTask(
-                shardInfo, config, mockRecordProcessor, mockCheckpointer, mockDataFetcher, taskBackoffTimeMillis,
+                shardInfo,
-                KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST, throttlingReporter);
+                config,
+                mockRecordProcessor,
+                mockCheckpointer,
+                mockDataFetcher,
+                taskBackoffTimeMillis,
+                KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                throttlingReporter,
+                getRecordsCache);
     }
 
     @Test
     public void testProcessTaskWithProvisionedThroughputExceededException() {
         // Set data fetcher to throw exception
         doReturn(false).when(mockDataFetcher).isShardEndReached();
-        doThrow(new ProvisionedThroughputExceededException("Test Exception")).when(mockDataFetcher)
+        doThrow(new ProvisionedThroughputExceededException("Test Exception")).when(getRecordsCache)
-                .getRecords(maxRecords);
+                .getNextResult();
 
         TaskResult result = processTask.call();
         verify(throttlingReporter).throttled();
         verify(throttlingReporter, never()).success();
+        verify(getRecordsCache).getNextResult();
         assertTrue("Result should contain ProvisionedThroughputExceededException",
                 result.getException() instanceof ProvisionedThroughputExceededException);
     }
 
     @Test
     public void testProcessTaskWithNonExistentStream() {
-        // Data fetcher returns a null Result when the stream does not exist
+        // Data fetcher returns a null Result ` the stream does not exist
-        doReturn(null).when(mockDataFetcher).getRecords(maxRecords);
+        doReturn(new ProcessRecordsInput().withRecords(Collections.emptyList()).withMillisBehindLatest((long) 0)).when(getRecordsCache).getNextResult();
 
         TaskResult result = processTask.call();
+        verify(getRecordsCache).getNextResult();
         assertNull("Task should not throw an exception", result.getException());
     }
 
@@ -300,14 +309,13 @@ public class ProcessTaskTest {
     private void testWithRecords(List<Record> records,
             ExtendedSequenceNumber lastCheckpointValue,
             ExtendedSequenceNumber largestPermittedCheckpointValue) {
-        when(mockDataFetcher.getRecords(anyInt())).thenReturn(
+        when(getRecordsCache.getNextResult()).thenReturn(new ProcessRecordsInput().withRecords(records).withMillisBehindLatest((long) 1000 * 50));
-                new GetRecordsResult().withRecords(records));
         when(mockCheckpointer.getLastCheckpointValue()).thenReturn(lastCheckpointValue);
         when(mockCheckpointer.getLargestPermittedCheckpointValue()).thenReturn(largestPermittedCheckpointValue);
         processTask.call();
         verify(throttlingReporter).success();
         verify(throttlingReporter, never()).throttled();
-
+        verify(getRecordsCache).getNextResult();
         ArgumentCaptor<ProcessRecordsInput> priCaptor = ArgumentCaptor.forClass(ProcessRecordsInput.class);
         verify(mockRecordProcessor).processRecords(priCaptor.capture());
         processedRecords = priCaptor.getValue().getRecords();

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/RecordProcessorCheckpointerTest.java

@@ -14,6 +14,13 @@
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+
 import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.List;
@@ -23,7 +30,10 @@ import org.junit.After;
 import org.junit.Assert;
 import org.junit.Before;
 import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
 import org.mockito.Mockito;
+import org.mockito.runners.MockitoJUnitRunner;
 
 import com.amazonaws.services.kinesis.clientlibrary.interfaces.ICheckpoint;
 import com.amazonaws.services.kinesis.clientlibrary.interfaces.IPreparedCheckpointer;
@@ -31,15 +41,15 @@ import com.amazonaws.services.kinesis.clientlibrary.lib.checkpoint.InMemoryCheck
 import com.amazonaws.services.kinesis.clientlibrary.lib.checkpoint.SentinelCheckpoint;
 import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
 import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
+import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
+import com.amazonaws.services.kinesis.metrics.impl.NullMetricsScope;
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
 import com.amazonaws.services.kinesis.model.Record;
 
-import static org.junit.Assert.fail;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Matchers.anyString;
-
 /**
  *
  */
+@RunWith(MockitoJUnitRunner.class)
 public class RecordProcessorCheckpointerTest {
     private String startingSequenceNumber = "13";
     private ExtendedSequenceNumber startingExtendedSequenceNumber = new ExtendedSequenceNumber(startingSequenceNumber);
@@ -48,6 +58,9 @@ public class RecordProcessorCheckpointerTest {
     private ShardInfo shardInfo;
     private SequenceNumberValidator sequenceNumberValidator;
     private String shardId = "shardId-123";
+    
+    @Mock
+    IMetricsFactory metricsFactory;
 
     /**
      * @throws java.lang.Exception
@@ -78,7 +91,7 @@ public class RecordProcessorCheckpointerTest {
     public final void testCheckpoint() throws Exception {
         // First call to checkpoint
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, null);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, null, metricsFactory);
         processingCheckpointer.setLargestPermittedCheckpointValue(startingExtendedSequenceNumber);
         processingCheckpointer.checkpoint();
         Assert.assertEquals(startingExtendedSequenceNumber, checkpoint.getCheckpoint(shardId));
@@ -98,7 +111,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testCheckpointRecord() throws Exception {
     	RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
     	processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
     	ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber("5025");
     	Record record = new Record().withSequenceNumber("5025");
@@ -114,7 +127,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testCheckpointSubRecord() throws Exception {
     	RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
     	processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
     	ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber("5030");
     	Record record = new Record().withSequenceNumber("5030");
@@ -131,7 +144,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testCheckpointSequenceNumber() throws Exception {
     	RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
     	processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
     	ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber("5035");
     	processingCheckpointer.setLargestPermittedCheckpointValue(extendedSequenceNumber);
@@ -146,7 +159,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testCheckpointExtendedSequenceNumber() throws Exception {
     	RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
     	processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
     	ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber("5040");
     	processingCheckpointer.setLargestPermittedCheckpointValue(extendedSequenceNumber);
@@ -162,7 +175,7 @@ public class RecordProcessorCheckpointerTest {
     public final void testPrepareCheckpoint() throws Exception {
         // First call to checkpoint
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
         processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
 
         ExtendedSequenceNumber sequenceNumber1 = new ExtendedSequenceNumber("5001");
@@ -193,7 +206,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testPrepareCheckpointRecord() throws Exception {
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
         processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
         ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber("5025");
         Record record = new Record().withSequenceNumber("5025");
@@ -218,7 +231,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testPrepareCheckpointSubRecord() throws Exception {
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
         processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
         ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber("5030");
         Record record = new Record().withSequenceNumber("5030");
@@ -244,7 +257,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testPrepareCheckpointSequenceNumber() throws Exception {
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
         processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
         ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber("5035");
         processingCheckpointer.setLargestPermittedCheckpointValue(extendedSequenceNumber);
@@ -268,7 +281,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testPrepareCheckpointExtendedSequenceNumber() throws Exception {
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
         processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
         ExtendedSequenceNumber extendedSequenceNumber = new ExtendedSequenceNumber("5040");
         processingCheckpointer.setLargestPermittedCheckpointValue(extendedSequenceNumber);
@@ -291,7 +304,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testMultipleOutstandingCheckpointersHappyCase() throws Exception {
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
         processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
         processingCheckpointer.setLargestPermittedCheckpointValue(new ExtendedSequenceNumber("6040"));
 
@@ -323,7 +336,7 @@ public class RecordProcessorCheckpointerTest {
     @Test
     public final void testMultipleOutstandingCheckpointersOutOfOrder() throws Exception {
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, sequenceNumberValidator, metricsFactory);
         processingCheckpointer.setInitialCheckpointValue(startingExtendedSequenceNumber);
         processingCheckpointer.setLargestPermittedCheckpointValue(new ExtendedSequenceNumber("7040"));
 
@@ -358,7 +371,7 @@ public class RecordProcessorCheckpointerTest {
      */
     @Test
     public final void testUpdate() throws Exception {
-        RecordProcessorCheckpointer checkpointer = new RecordProcessorCheckpointer(shardInfo, checkpoint, null);
+        RecordProcessorCheckpointer checkpointer = new RecordProcessorCheckpointer(shardInfo, checkpoint, null, metricsFactory);
 
         ExtendedSequenceNumber sequenceNumber = new ExtendedSequenceNumber("10");
         checkpointer.setLargestPermittedCheckpointValue(sequenceNumber);
@@ -379,7 +392,7 @@ public class RecordProcessorCheckpointerTest {
         SequenceNumberValidator validator = mock(SequenceNumberValidator.class);
         Mockito.doNothing().when(validator).validateSequenceNumber(anyString());
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, validator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, validator, metricsFactory);
 
         // Several checkpoints we're gonna hit
         ExtendedSequenceNumber tooSmall = new ExtendedSequenceNumber("2");
@@ -467,7 +480,7 @@ public class RecordProcessorCheckpointerTest {
         SequenceNumberValidator validator = mock(SequenceNumberValidator.class);
         Mockito.doNothing().when(validator).validateSequenceNumber(anyString());
         RecordProcessorCheckpointer processingCheckpointer =
-                new RecordProcessorCheckpointer(shardInfo, checkpoint, validator);
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, validator, metricsFactory);
 
         // Several checkpoints we're gonna hit
         ExtendedSequenceNumber tooSmall = new ExtendedSequenceNumber("2");
@@ -595,7 +608,7 @@ public class RecordProcessorCheckpointerTest {
 
         for (LinkedHashMap<String, CheckpointAction> testPlan : getMixedCallsTestPlan()) {
             RecordProcessorCheckpointer processingCheckpointer =
-                    new RecordProcessorCheckpointer(shardInfo, checkpoint, validator);
+                    new RecordProcessorCheckpointer(shardInfo, checkpoint, validator, metricsFactory);
             testMixedCheckpointCalls(processingCheckpointer, testPlan, CheckpointerType.CHECKPOINTER);
         }
     }
@@ -615,7 +628,7 @@ public class RecordProcessorCheckpointerTest {
 
         for (LinkedHashMap<String, CheckpointAction> testPlan : getMixedCallsTestPlan()) {
             RecordProcessorCheckpointer processingCheckpointer =
-                    new RecordProcessorCheckpointer(shardInfo, checkpoint, validator);
+                    new RecordProcessorCheckpointer(shardInfo, checkpoint, validator, metricsFactory);
             testMixedCheckpointCalls(processingCheckpointer, testPlan, CheckpointerType.PREPARED_CHECKPOINTER);
         }
     }
@@ -636,7 +649,7 @@ public class RecordProcessorCheckpointerTest {
 
         for (LinkedHashMap<String, CheckpointAction> testPlan : getMixedCallsTestPlan()) {
             RecordProcessorCheckpointer processingCheckpointer =
-                    new RecordProcessorCheckpointer(shardInfo, checkpoint, validator);
+                    new RecordProcessorCheckpointer(shardInfo, checkpoint, validator, metricsFactory);
             testMixedCheckpointCalls(processingCheckpointer, testPlan, CheckpointerType.PREPARE_THEN_CHECKPOINTER);
         }
     }
@@ -785,4 +798,34 @@ public class RecordProcessorCheckpointerTest {
             Assert.assertEquals(null, checkpoint.getCheckpointObject(shardId).getPendingCheckpoint());
         }
     }
+
+    @Test
+    public final void testUnsetMetricsScopeDuringCheckpointing() throws Exception {
+        // First call to checkpoint
+        RecordProcessorCheckpointer processingCheckpointer =
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, null, metricsFactory);
+        ExtendedSequenceNumber sequenceNumber = new ExtendedSequenceNumber("5019");
+        processingCheckpointer.setLargestPermittedCheckpointValue(sequenceNumber);
+        processingCheckpointer.checkpoint();
+        Assert.assertEquals(sequenceNumber, checkpoint.getCheckpoint(shardId));
+        verify(metricsFactory).createMetrics();
+        Assert.assertFalse(MetricsHelper.isMetricsScopePresent());
+    }
+
+    @Test
+    public final void testSetMetricsScopeDuringCheckpointing() throws Exception {
+        // First call to checkpoint
+        RecordProcessorCheckpointer processingCheckpointer =
+                new RecordProcessorCheckpointer(shardInfo, checkpoint, null, metricsFactory);
+        NullMetricsScope scope = new NullMetricsScope();
+        MetricsHelper.setMetricsScope(scope);
+        ExtendedSequenceNumber sequenceNumber = new ExtendedSequenceNumber("5019");
+        processingCheckpointer.setLargestPermittedCheckpointValue(sequenceNumber);
+        processingCheckpointer.checkpoint();
+        Assert.assertEquals(sequenceNumber, checkpoint.getCheckpoint(shardId));
+        verify(metricsFactory, never()).createMetrics();
+        Assert.assertTrue(MetricsHelper.isMetricsScopePresent());
+        assertEquals(scope, MetricsHelper.getMetricsScope());
+        MetricsHelper.unsetMetricsScope();
+    }
 }

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/RecordsFetcherFactoryTest.java

@@ -0,0 +1,43 @@
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import static org.hamcrest.CoreMatchers.instanceOf;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+
+public class RecordsFetcherFactoryTest {
+    private String shardId = "TestShard";
+    private RecordsFetcherFactory recordsFetcherFactory;
+
+    @Mock
+    private GetRecordsRetrievalStrategy getRecordsRetrievalStrategy;
+    @Mock
+    private IMetricsFactory metricsFactory;
+
+    @Before
+    public void setUp() {
+        MockitoAnnotations.initMocks(this);
+        recordsFetcherFactory = new SimpleRecordsFetcherFactory();
+    }
+
+    @Test
+    public void createDefaultRecordsFetcherTest() {
+        GetRecordsCache recordsCache = recordsFetcherFactory.createRecordsFetcher(getRecordsRetrievalStrategy, shardId,
+                metricsFactory, 1);
+        assertThat(recordsCache, instanceOf(BlockingGetRecordsCache.class));
+    }
+
+    @Test
+    public void createPrefetchRecordsFetcherTest() {
+        recordsFetcherFactory.setDataFetchingStrategy(DataFetchingStrategy.PREFETCH_CACHED);
+        GetRecordsCache recordsCache = recordsFetcherFactory.createRecordsFetcher(getRecordsRetrievalStrategy, shardId,
+                metricsFactory, 1);
+        assertThat(recordsCache, instanceOf(PrefetchGetRecordsCache.class));
+    }
+
+}

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardConsumerTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
@@ -22,11 +22,13 @@ import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.fail;
 import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyInt;
 import static org.mockito.Matchers.anyString;
 import static org.mockito.Matchers.argThat;
 import static org.mockito.Mockito.atLeastOnce;
 import static org.mockito.Mockito.doNothing;
 import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.spy;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
@@ -39,9 +41,12 @@ import java.util.Date;
 import java.util.List;
 import java.util.ListIterator;
 import java.util.Objects;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
 import java.util.concurrent.RejectedExecutionException;
 import java.util.concurrent.TimeUnit;
 
@@ -50,6 +55,7 @@ import org.apache.commons.logging.LogFactory;
 import org.hamcrest.Description;
 import org.hamcrest.Matcher;
 import org.hamcrest.TypeSafeMatcher;
+import org.junit.Before;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.mockito.Mock;
@@ -64,12 +70,14 @@ import com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisLocalFileProx
 import com.amazonaws.services.kinesis.clientlibrary.proxies.util.KinesisLocalFileDataCreator;
 import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
 import com.amazonaws.services.kinesis.clientlibrary.types.InitializationInput;
+import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownInput;
 import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
 import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
 import com.amazonaws.services.kinesis.leases.interfaces.ILeaseManager;
 import com.amazonaws.services.kinesis.metrics.impl.NullMetricsFactory;
 import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
 import com.amazonaws.services.kinesis.model.Record;
+import com.amazonaws.services.kinesis.model.Shard;
 import com.amazonaws.services.kinesis.model.ShardIteratorType;
 
 /**
@@ -93,10 +101,17 @@ public class ShardConsumerTest {
     // Use Executors.newFixedThreadPool since it returns ThreadPoolExecutor, which is
     // ... a non-final public class, and so can be mocked and spied.
     private final ExecutorService executorService = Executors.newFixedThreadPool(1);
-
+    private RecordsFetcherFactory recordsFetcherFactory;
+    
+    private GetRecordsCache getRecordsCache;
+    
+    private KinesisDataFetcher dataFetcher;
+    
     @Mock
     private IRecordProcessor processor;
     @Mock
+    private KinesisClientLibConfiguration config;
+    @Mock
     private IKinesisProxy streamProxy;
     @Mock
     private ILeaseManager<KinesisClientLease> leaseManager;
@@ -105,6 +120,16 @@ public class ShardConsumerTest {
     @Mock
     private ShutdownNotification shutdownNotification;
 
+    @Before
+    public void setup() {
+        getRecordsCache = null;
+        dataFetcher = null;
+        
+        recordsFetcherFactory = spy(new SimpleRecordsFetcherFactory());
+        when(config.getRecordsFetcherFactory()).thenReturn(recordsFetcherFactory);
+        when(config.getLogWarningForTaskAfterMillis()).thenReturn(Optional.empty());
+    }
+    
     /**
      * Test method to verify consumer stays in INITIALIZING state when InitializationTask fails.
      */
@@ -135,8 +160,9 @@ public class ShardConsumerTest {
                         executorService,
                         metricsFactory,
                         taskBackoffTimeMillis,
-                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST);
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
-
+                        config);
+        
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.WAITING_ON_PARENT_SHARDS)));
         consumer.consumeShard(); // initialize
         Thread.sleep(50L);
@@ -152,7 +178,6 @@ public class ShardConsumerTest {
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.INITIALIZING)));
     }
 
-
     /**
      * Test method to verify consumer stays in INITIALIZING state when InitializationTask fails.
      */
@@ -183,7 +208,8 @@ public class ShardConsumerTest {
                         spyExecutorService,
                         metricsFactory,
                         taskBackoffTimeMillis,
-                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST);
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        config);
 
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.WAITING_ON_PARENT_SHARDS)));
         consumer.consumeShard(); // initialize
@@ -224,7 +250,8 @@ public class ShardConsumerTest {
                         executorService,
                         metricsFactory,
                         taskBackoffTimeMillis,
-                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST);
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        config);
 
         final ExtendedSequenceNumber checkpointSequenceNumber = new ExtendedSequenceNumber("123");
         final ExtendedSequenceNumber pendingCheckpointSequenceNumber = null;
@@ -297,7 +324,6 @@ public class ShardConsumerTest {
         ICheckpoint checkpoint = new InMemoryCheckpointImpl(startSeqNum.toString());
         checkpoint.setCheckpoint(streamShardId, ExtendedSequenceNumber.TRIM_HORIZON, testConcurrencyToken);
         when(leaseManager.getLease(anyString())).thenReturn(null);
-
         TestStreamlet processor = new TestStreamlet();
 
         StreamConfig streamConfig =
@@ -308,18 +334,43 @@ public class ShardConsumerTest {
                         skipCheckpointValidationValue, INITIAL_POSITION_LATEST);
 
         ShardInfo shardInfo = new ShardInfo(streamShardId, testConcurrencyToken, null, null);
+
+        RecordProcessorCheckpointer recordProcessorCheckpointer = new RecordProcessorCheckpointer(
+                shardInfo,
+                checkpoint,
+                new SequenceNumberValidator(
+                        streamConfig.getStreamProxy(),
+                        shardInfo.getShardId(),
+                        streamConfig.shouldValidateSequenceNumberBeforeCheckpointing()
+                ),
+                metricsFactory
+        );
+
+        dataFetcher = new KinesisDataFetcher(streamConfig.getStreamProxy(), shardInfo);
+
+        getRecordsCache = spy(new BlockingGetRecordsCache(maxRecords,
+                new SynchronousGetRecordsRetrievalStrategy(dataFetcher)));
+        when(recordsFetcherFactory.createRecordsFetcher(any(GetRecordsRetrievalStrategy.class), anyString(),
+                any(IMetricsFactory.class), anyInt()))
+                .thenReturn(getRecordsCache);
+        
         ShardConsumer consumer =
                 new ShardConsumer(shardInfo,
                         streamConfig,
                         checkpoint,
                         processor,
+                        recordProcessorCheckpointer,
                         leaseManager,
                         parentShardPollIntervalMillis,
                         cleanupLeasesOfCompletedShards,
                         executorService,
                         metricsFactory,
                         taskBackoffTimeMillis,
-                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST);
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        dataFetcher,
+                        Optional.empty(),
+                        Optional.empty(),
+                        config);
 
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.WAITING_ON_PARENT_SHARDS)));
         consumer.consumeShard(); // check on parent shards
@@ -328,6 +379,7 @@ public class ShardConsumerTest {
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.INITIALIZING)));
         consumer.consumeShard(); // initialize
         processor.getInitializeLatch().await(5, TimeUnit.SECONDS);
+        verify(getRecordsCache).start();
 
         // We expect to process all records in numRecs calls
         for (int i = 0; i < numRecs;) {
@@ -340,6 +392,8 @@ public class ShardConsumerTest {
             }
             Thread.sleep(50L);
         }
+        
+        verify(getRecordsCache, times(5)).getNextResult();
 
         assertThat(processor.getShutdownReason(), nullValue());
         consumer.notifyShutdownRequested(shutdownNotification);
@@ -363,6 +417,157 @@ public class ShardConsumerTest {
         verify(shutdownNotification, atLeastOnce()).shutdownComplete();
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.SHUTDOWN_COMPLETE)));
         assertThat(processor.getShutdownReason(), is(equalTo(ShutdownReason.ZOMBIE)));
+        
+        verify(getRecordsCache).shutdown();
+
+        executorService.shutdown();
+        executorService.awaitTermination(60, TimeUnit.SECONDS);
+
+        String iterator = fileBasedProxy.getIterator(streamShardId, ShardIteratorType.TRIM_HORIZON.toString());
+        List<Record> expectedRecords = toUserRecords(fileBasedProxy.get(iterator, numRecs).getRecords());
+        verifyConsumedRecords(expectedRecords, processor.getProcessedRecords());
+        file.delete();
+    }
+
+    private static final class TransientShutdownErrorTestStreamlet extends TestStreamlet {
+        private final CountDownLatch errorShutdownLatch = new CountDownLatch(1);
+
+        @Override
+        public void shutdown(ShutdownInput input) {
+            ShutdownReason reason = input.getShutdownReason();
+            if (reason.equals(ShutdownReason.TERMINATE) && errorShutdownLatch.getCount() > 0) {
+                errorShutdownLatch.countDown();
+                throw new RuntimeException("test");
+            } else {
+                super.shutdown(input);
+            }
+        }
+    }
+
+    /**
+     * Test method for {@link ShardConsumer#consumeShard()} that ensures a transient error thrown from the record
+     * processor's shutdown method with reason terminate will be retried.
+     */
+    @Test
+    public final void testConsumeShardWithTransientTerminateError() throws Exception {
+        int numRecs = 10;
+        BigInteger startSeqNum = BigInteger.ONE;
+        String streamShardId = "kinesis-0-0";
+        String testConcurrencyToken = "testToken";
+        List<Shard> shardList = KinesisLocalFileDataCreator.createShardList(1, "kinesis-0-", startSeqNum);
+        // Close the shard so that shutdown is called with reason terminate
+        shardList.get(0).getSequenceNumberRange().setEndingSequenceNumber(
+                KinesisLocalFileProxy.MAX_SEQUENCE_NUMBER.subtract(BigInteger.ONE).toString());
+        File file = KinesisLocalFileDataCreator.generateTempDataFile(shardList, numRecs, "unitTestSCT002");
+
+        IKinesisProxy fileBasedProxy = new KinesisLocalFileProxy(file.getAbsolutePath());
+
+        final int maxRecords = 2;
+        final int idleTimeMS = 0; // keep unit tests fast
+        ICheckpoint checkpoint = new InMemoryCheckpointImpl(startSeqNum.toString());
+        checkpoint.setCheckpoint(streamShardId, ExtendedSequenceNumber.TRIM_HORIZON, testConcurrencyToken);
+        when(leaseManager.getLease(anyString())).thenReturn(null);
+
+        TransientShutdownErrorTestStreamlet processor = new TransientShutdownErrorTestStreamlet();
+
+        StreamConfig streamConfig =
+                new StreamConfig(fileBasedProxy,
+                        maxRecords,
+                        idleTimeMS,
+                        callProcessRecordsForEmptyRecordList,
+                        skipCheckpointValidationValue, INITIAL_POSITION_LATEST);
+
+        ShardInfo shardInfo = new ShardInfo(streamShardId, testConcurrencyToken, null, null);
+
+        dataFetcher = new KinesisDataFetcher(streamConfig.getStreamProxy(), shardInfo);
+
+        getRecordsCache = spy(new BlockingGetRecordsCache(maxRecords,
+                new SynchronousGetRecordsRetrievalStrategy(dataFetcher)));
+        when(recordsFetcherFactory.createRecordsFetcher(any(GetRecordsRetrievalStrategy.class), anyString(),
+                any(IMetricsFactory.class), anyInt()))
+                .thenReturn(getRecordsCache);
+
+        RecordProcessorCheckpointer recordProcessorCheckpointer = new RecordProcessorCheckpointer(
+                shardInfo,
+                checkpoint,
+                new SequenceNumberValidator(
+                        streamConfig.getStreamProxy(),
+                        shardInfo.getShardId(),
+                        streamConfig.shouldValidateSequenceNumberBeforeCheckpointing()
+                ),
+                metricsFactory
+        );
+        
+        ShardConsumer consumer =
+                new ShardConsumer(shardInfo,
+                        streamConfig,
+                        checkpoint,
+                        processor,
+                        recordProcessorCheckpointer,
+                        leaseManager,
+                        parentShardPollIntervalMillis,
+                        cleanupLeasesOfCompletedShards,
+                        executorService,
+                        metricsFactory,
+                        taskBackoffTimeMillis,
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        dataFetcher,
+                        Optional.empty(),
+                        Optional.empty(),
+                        config);
+
+        assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.WAITING_ON_PARENT_SHARDS)));
+        consumer.consumeShard(); // check on parent shards
+        Thread.sleep(50L);
+        consumer.consumeShard(); // start initialization
+        assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.INITIALIZING)));
+        consumer.consumeShard(); // initialize
+        processor.getInitializeLatch().await(5, TimeUnit.SECONDS);
+        verify(getRecordsCache).start();
+
+        // We expect to process all records in numRecs calls
+        for (int i = 0; i < numRecs;) {
+            boolean newTaskSubmitted = consumer.consumeShard();
+            if (newTaskSubmitted) {
+                LOG.debug("New processing task was submitted, call # " + i);
+                assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.PROCESSING)));
+                // CHECKSTYLE:IGNORE ModifiedControlVariable FOR NEXT 1 LINES
+                i += maxRecords;
+            }
+            Thread.sleep(50L);
+        }
+
+        // Consume shards until shutdown terminate is called and it has thrown an exception
+        for (int i = 0; i < 100; i++) {
+            consumer.consumeShard();
+            if (processor.errorShutdownLatch.await(50, TimeUnit.MILLISECONDS)) {
+                break;
+            }
+        }
+        assertEquals(0, processor.errorShutdownLatch.getCount());
+
+        // Wait for a retry of shutdown terminate that should succeed
+        for (int i = 0; i < 100; i++) {
+            consumer.consumeShard();
+            if (processor.getShutdownLatch().await(50, TimeUnit.MILLISECONDS)) {
+                break;
+            }
+        }
+        assertEquals(0, processor.getShutdownLatch().getCount());
+
+        // Wait for shutdown complete now that terminate shutdown is successful
+        for (int i = 0; i < 100; i++) {
+            consumer.consumeShard();
+            if (consumer.getCurrentState() == ConsumerStates.ShardConsumerState.SHUTDOWN_COMPLETE) {
+                break;
+            }
+            Thread.sleep(50L);
+        }
+        assertThat(consumer.getCurrentState(), equalTo(ConsumerStates.ShardConsumerState.SHUTDOWN_COMPLETE));
+
+        assertThat(processor.getShutdownReason(), is(equalTo(ShutdownReason.TERMINATE)));
+
+        verify(getRecordsCache).shutdown();
 
         executorService.shutdown();
         executorService.awaitTermination(60, TimeUnit.SECONDS);
@@ -399,7 +604,6 @@ public class ShardConsumerTest {
         ICheckpoint checkpoint = new InMemoryCheckpointImpl(startSeqNum.toString());
         checkpoint.setCheckpoint(streamShardId, ExtendedSequenceNumber.AT_TIMESTAMP, testConcurrencyToken);
         when(leaseManager.getLease(anyString())).thenReturn(null);
-
         TestStreamlet processor = new TestStreamlet();
 
         StreamConfig streamConfig =
@@ -411,18 +615,43 @@ public class ShardConsumerTest {
                         atTimestamp);
 
         ShardInfo shardInfo = new ShardInfo(streamShardId, testConcurrencyToken, null, ExtendedSequenceNumber.TRIM_HORIZON);
+        
+        RecordProcessorCheckpointer recordProcessorCheckpointer = new RecordProcessorCheckpointer(
+                shardInfo,
+                checkpoint,
+                new SequenceNumberValidator(
+                        streamConfig.getStreamProxy(),
+                        shardInfo.getShardId(),
+                        streamConfig.shouldValidateSequenceNumberBeforeCheckpointing()
+                ),
+                metricsFactory
+        );
+
+        dataFetcher = new KinesisDataFetcher(streamConfig.getStreamProxy(), shardInfo);
+        
+        getRecordsCache = spy(new BlockingGetRecordsCache(maxRecords,
+                new SynchronousGetRecordsRetrievalStrategy(dataFetcher)));
+        when(recordsFetcherFactory.createRecordsFetcher(any(GetRecordsRetrievalStrategy.class), anyString(),
+                any(IMetricsFactory.class), anyInt()))
+                .thenReturn(getRecordsCache);
+
         ShardConsumer consumer =
                 new ShardConsumer(shardInfo,
                         streamConfig,
                         checkpoint,
                         processor,
+                        recordProcessorCheckpointer,
                         leaseManager,
                         parentShardPollIntervalMillis,
                         cleanupLeasesOfCompletedShards,
                         executorService,
                         metricsFactory,
                         taskBackoffTimeMillis,
-                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST);
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        dataFetcher,
+                        Optional.empty(),
+                        Optional.empty(),
+                        config);
 
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.WAITING_ON_PARENT_SHARDS)));
         consumer.consumeShard(); // check on parent shards
@@ -431,6 +660,8 @@ public class ShardConsumerTest {
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.INITIALIZING)));
         consumer.consumeShard(); // initialize
         Thread.sleep(50L);
+        
+        verify(getRecordsCache).start();
 
         // We expect to process all records in numRecs calls
         for (int i = 0; i < numRecs;) {
@@ -443,6 +674,8 @@ public class ShardConsumerTest {
             }
             Thread.sleep(50L);
         }
+        
+        verify(getRecordsCache, times(4)).getNextResult();
 
         assertThat(processor.getShutdownReason(), nullValue());
         consumer.beginShutdown();
@@ -455,8 +688,11 @@ public class ShardConsumerTest {
         executorService.shutdown();
         executorService.awaitTermination(60, TimeUnit.SECONDS);
 
+        verify(getRecordsCache).shutdown();
+
         String iterator = fileBasedProxy.getIterator(streamShardId, timestamp);
         List<Record> expectedRecords = toUserRecords(fileBasedProxy.get(iterator, numRecs).getRecords());
+        
         verifyConsumedRecords(expectedRecords, processor.getProcessedRecords());
         assertEquals(4, processor.getProcessedRecords().size());
         file.delete();
@@ -484,11 +720,15 @@ public class ShardConsumerTest {
                         executorService,
                         metricsFactory,
                         taskBackoffTimeMillis,
-                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST);
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        config);
+
+        GetRecordsCache getRecordsCache = spy(consumer.getGetRecordsCache());
 
         final ExtendedSequenceNumber checkpointSequenceNumber = new ExtendedSequenceNumber("123");
         final ExtendedSequenceNumber pendingCheckpointSequenceNumber = new ExtendedSequenceNumber("999");
         when(leaseManager.getLease(anyString())).thenReturn(null);
+        when(config.getRecordsFetcherFactory()).thenReturn(new SimpleRecordsFetcherFactory());
         when(checkpoint.getCheckpointObject(anyString())).thenReturn(
                 new Checkpoint(checkpointSequenceNumber, pendingCheckpointSequenceNumber));
 
@@ -509,6 +749,112 @@ public class ShardConsumerTest {
         Thread.sleep(50L);
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.PROCESSING)));
     }
+    
+    @Test
+    public void testCreateSynchronousGetRecordsRetrieval() {
+        ShardInfo shardInfo = new ShardInfo("s-0-0", "testToken", null, ExtendedSequenceNumber.TRIM_HORIZON);
+        StreamConfig streamConfig =
+                new StreamConfig(streamProxy,
+                        1,
+                        10,
+                        callProcessRecordsForEmptyRecordList,
+                        skipCheckpointValidationValue, INITIAL_POSITION_LATEST);
+        
+        ShardConsumer shardConsumer =
+                new ShardConsumer(shardInfo,
+                        streamConfig,
+                        checkpoint,
+                        processor,
+                        null,
+                        parentShardPollIntervalMillis,
+                        cleanupLeasesOfCompletedShards,
+                        executorService,
+                        metricsFactory,
+                        taskBackoffTimeMillis,
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        Optional.empty(),
+                        Optional.empty(),
+                        config);
+        
+        assertEquals(shardConsumer.getGetRecordsCache().getGetRecordsRetrievalStrategy().getClass(),
+                SynchronousGetRecordsRetrievalStrategy.class);
+    }
+
+    @Test
+    public void testCreateAsynchronousGetRecordsRetrieval() {
+        ShardInfo shardInfo = new ShardInfo("s-0-0", "testToken", null, ExtendedSequenceNumber.TRIM_HORIZON);
+        StreamConfig streamConfig =
+                new StreamConfig(streamProxy,
+                        1,
+                        10,
+                        callProcessRecordsForEmptyRecordList,
+                        skipCheckpointValidationValue, INITIAL_POSITION_LATEST);
+
+        ShardConsumer shardConsumer =
+                new ShardConsumer(shardInfo,
+                        streamConfig,
+                        checkpoint,
+                        processor,
+                        null,
+                        parentShardPollIntervalMillis,
+                        cleanupLeasesOfCompletedShards,
+                        executorService,
+                        metricsFactory,
+                        taskBackoffTimeMillis,
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        Optional.of(1),
+                        Optional.of(2),
+                        config);
+
+        assertEquals(shardConsumer.getGetRecordsCache().getGetRecordsRetrievalStrategy().getClass(),
+                AsynchronousGetRecordsRetrievalStrategy.class);
+    }
+    
+    @SuppressWarnings("unchecked")
+    @Test
+    public void testLongRunningTasks() throws InterruptedException {
+        final long sleepTime = 1000L;
+        ExecutorService mockExecutorService = mock(ExecutorService.class);
+        Future<TaskResult> mockFuture = mock(Future.class);
+        
+        when(mockExecutorService.submit(any(ITask.class))).thenReturn(mockFuture);
+        when(mockFuture.isDone()).thenReturn(false);
+        when(mockFuture.isCancelled()).thenReturn(false);
+        when(config.getLogWarningForTaskAfterMillis()).thenReturn(Optional.of(sleepTime));
+        
+        ShardInfo shardInfo = new ShardInfo("s-0-0", "testToken", null, ExtendedSequenceNumber.LATEST);
+        StreamConfig streamConfig = new StreamConfig(
+                streamProxy,
+                1,
+                10,
+                callProcessRecordsForEmptyRecordList,
+                skipCheckpointValidationValue,
+                INITIAL_POSITION_LATEST);
+        
+        ShardConsumer shardConsumer = new ShardConsumer(
+                shardInfo,
+                streamConfig,
+                checkpoint,
+                processor,
+                null,
+                parentShardPollIntervalMillis,
+                cleanupLeasesOfCompletedShards,
+                mockExecutorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                config);
+        
+        shardConsumer.consumeShard();
+
+        Thread.sleep(sleepTime);
+        
+        shardConsumer.consumeShard();
+        
+        verify(config).getLogWarningForTaskAfterMillis();
+        verify(mockFuture).isDone();
+        verify(mockFuture).isCancelled();
+    }
 
     //@formatter:off (gets the formatting wrong)
     private void verifyConsumedRecords(List<Record> expectedRecords,

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardSyncTaskIntegrationTest.java

@@ -124,6 +124,7 @@ public class ShardSyncTaskIntegrationTest {
                 leaseManager,
                 InitialPositionInStreamExtended.newInitialPosition(InitialPositionInStream.LATEST),
                 false,
+                false,
                 0L);
         syncTask.call();
         List<KinesisClientLease> leases = leaseManager.listLeases();

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardSyncerTest.java

@@ -146,6 +146,39 @@ public class ShardSyncerTest {
         }
     }
 
+    /**
+     * Test determineNewLeasesToCreate() where there are no leases and no resharding operations have been performed, but one of
+     * the shards was marked as inconsistent.
+     */
+    @Test
+    public final void testDetermineNewLeasesToCreate0Leases0Reshards1Inconsistent() {
+        List<Shard> shards = new ArrayList<Shard>();
+        List<KinesisClientLease> currentLeases = new ArrayList<KinesisClientLease>();
+        SequenceNumberRange sequenceRange = ShardObjectHelper.newSequenceNumberRange("342980", null);
+
+        String shardId0 = "shardId-0";
+        shards.add(ShardObjectHelper.newShard(shardId0, null, null, sequenceRange));
+
+        String shardId1 = "shardId-1";
+        shards.add(ShardObjectHelper.newShard(shardId1, null, null, sequenceRange));
+
+        String shardId2 = "shardId-2";
+        shards.add(ShardObjectHelper.newShard(shardId2, shardId1, null, sequenceRange));
+
+        Set<String> inconsistentShardIds = new HashSet<String>();
+        inconsistentShardIds.add(shardId2);
+
+        List<KinesisClientLease> newLeases =
+            ShardSyncer.determineNewLeasesToCreate(shards, currentLeases, INITIAL_POSITION_LATEST, inconsistentShardIds);
+        Assert.assertEquals(2, newLeases.size());
+        Set<String> expectedLeaseShardIds = new HashSet<String>();
+        expectedLeaseShardIds.add(shardId0);
+        expectedLeaseShardIds.add(shardId1);
+        for (KinesisClientLease lease : newLeases) {
+            Assert.assertTrue(expectedLeaseShardIds.contains(lease.getLeaseKey()));
+        }
+    }
+
     /**
      * Test bootstrapShardLeases() starting at TRIM_HORIZON ("beginning" of stream)
      * 
@@ -296,6 +329,41 @@ public class ShardSyncerTest {
         dataFile.delete();
     }
 
+    /**
+     * Test checkAndCreateLeasesForNewShards() when a parent is open and children of open parents are being ignored.
+     */
+    @Test
+    public final void testCheckAndCreateLeasesForNewShardsWhenParentIsOpenAndIgnoringInconsistentChildren()
+        throws KinesisClientLibIOException, DependencyException, InvalidStateException, ProvisionedThroughputException,
+        IOException {
+        List<Shard> shards = constructShardListForGraphA();
+        Shard shard = shards.get(5);
+        Assert.assertEquals("shardId-5", shard.getShardId());
+        SequenceNumberRange range = shard.getSequenceNumberRange();
+        // shardId-5 in graph A has two children (shardId-9 and shardId-10).  if shardId-5
+        // is not closed, those children should be ignored when syncing shards, no leases
+        // should be obtained for them, and we should obtain a lease on the still-open
+        // parent.
+        range.setEndingSequenceNumber(null);
+        shard.setSequenceNumberRange(range);
+        File dataFile = KinesisLocalFileDataCreator.generateTempDataFile(shards, 2, "testBootstrap1");
+        dataFile.deleteOnExit();
+        IKinesisProxy kinesisProxy = new KinesisLocalFileProxy(dataFile.getAbsolutePath());
+        ShardSyncer.checkAndCreateLeasesForNewShards(kinesisProxy, leaseManager, INITIAL_POSITION_LATEST,
+                                                     cleanupLeasesOfCompletedShards, true);
+        List<KinesisClientLease> newLeases = leaseManager.listLeases();
+        Set<String> expectedLeaseShardIds = new HashSet<String>();
+        expectedLeaseShardIds.add("shardId-4");
+        expectedLeaseShardIds.add("shardId-5");
+        expectedLeaseShardIds.add("shardId-8");
+        Assert.assertEquals(expectedLeaseShardIds.size(), newLeases.size());
+        for (KinesisClientLease lease1 : newLeases) {
+            Assert.assertTrue(expectedLeaseShardIds.contains(lease1.getLeaseKey()));
+            Assert.assertEquals(ExtendedSequenceNumber.LATEST, lease1.getCheckpoint());
+        }
+        dataFile.delete();
+    }
+
     /**
      * @throws KinesisClientLibIOException
      * @throws DependencyException
@@ -586,7 +654,8 @@ public class ShardSyncerTest {
         dataFile.deleteOnExit();
         IKinesisProxy kinesisProxy = new KinesisLocalFileProxy(dataFile.getAbsolutePath());
 
-        ShardSyncer.bootstrapShardLeases(kinesisProxy, leaseManager, initialPosition, cleanupLeasesOfCompletedShards);
+        ShardSyncer.bootstrapShardLeases(kinesisProxy, leaseManager, initialPosition, cleanupLeasesOfCompletedShards,
+                                         false);
         List<KinesisClientLease> newLeases = leaseManager.listLeases();
         Assert.assertEquals(2, newLeases.size());
         Set<String> expectedLeaseShardIds = new HashSet<String>();

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShutdownTaskTest.java

@@ -1,20 +1,22 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import static org.mockito.Mockito.doNothing;
 import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
 
 import java.util.HashSet;
@@ -34,10 +36,14 @@ import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber
 import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
 import com.amazonaws.services.kinesis.leases.impl.KinesisClientLeaseManager;
 import com.amazonaws.services.kinesis.leases.interfaces.ILeaseManager;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
 
 /**
  *
  */
+@RunWith(MockitoJUnitRunner.class)
 public class ShutdownTaskTest {
     private static final long TASK_BACKOFF_TIME_MILLIS = 1L;
     private static final InitialPositionInStreamExtended INITIAL_POSITION_TRIM_HORIZON =
@@ -51,6 +57,9 @@ public class ShutdownTaskTest {
             defaultParentShardIds,
             ExtendedSequenceNumber.LATEST);
     IRecordProcessor defaultRecordProcessor = new TestStreamlet();
+    
+    @Mock
+    private GetRecordsCache getRecordsCache;
 
     /**
      * @throws java.lang.Exception
@@ -71,6 +80,7 @@ public class ShutdownTaskTest {
      */
     @Before
     public void setUp() throws Exception {
+        doNothing().when(getRecordsCache).shutdown();
     }
 
     /**
@@ -90,6 +100,7 @@ public class ShutdownTaskTest {
         IKinesisProxy kinesisProxy = mock(IKinesisProxy.class);
         ILeaseManager<KinesisClientLease> leaseManager = mock(KinesisClientLeaseManager.class);
         boolean cleanupLeasesOfCompletedShards = false;
+        boolean ignoreUnexpectedChildShards = false;
         ShutdownTask task = new ShutdownTask(defaultShardInfo,
                 defaultRecordProcessor,
                 checkpointer,
@@ -97,8 +108,10 @@ public class ShutdownTaskTest {
                 kinesisProxy,
                 INITIAL_POSITION_TRIM_HORIZON,
                 cleanupLeasesOfCompletedShards,
+                ignoreUnexpectedChildShards,
                 leaseManager,
-                TASK_BACKOFF_TIME_MILLIS);
+                TASK_BACKOFF_TIME_MILLIS,
+                getRecordsCache);
         TaskResult result = task.call();
         Assert.assertNotNull(result.getException());
         Assert.assertTrue(result.getException() instanceof IllegalArgumentException);
@@ -115,6 +128,7 @@ public class ShutdownTaskTest {
         when(kinesisProxy.getShardList()).thenReturn(null);
         ILeaseManager<KinesisClientLease> leaseManager = mock(KinesisClientLeaseManager.class);
         boolean cleanupLeasesOfCompletedShards = false;
+        boolean ignoreUnexpectedChildShards = false;
         ShutdownTask task = new ShutdownTask(defaultShardInfo,
                 defaultRecordProcessor,
                 checkpointer,
@@ -122,11 +136,14 @@ public class ShutdownTaskTest {
                 kinesisProxy,
                 INITIAL_POSITION_TRIM_HORIZON,
                 cleanupLeasesOfCompletedShards,
+                ignoreUnexpectedChildShards,
                 leaseManager,
-                TASK_BACKOFF_TIME_MILLIS);
+                TASK_BACKOFF_TIME_MILLIS,
+                getRecordsCache);
         TaskResult result = task.call();
         Assert.assertNotNull(result.getException());
         Assert.assertTrue(result.getException() instanceof KinesisClientLibIOException);
+        verify(getRecordsCache).shutdown();
     }
 
     /**
@@ -134,7 +151,7 @@ public class ShutdownTaskTest {
      */
     @Test
     public final void testGetTaskType() {
-        ShutdownTask task = new ShutdownTask(null, null, null, null, null, null, false, null, 0);
+        ShutdownTask task = new ShutdownTask(null, null, null, null, null, null, false, false, null, 0, getRecordsCache);
         Assert.assertEquals(TaskType.SHUTDOWN, task.getTaskType());
     }
 

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/WorkerTest.java

@@ -21,10 +21,21 @@ import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.Matchers.greaterThanOrEqualTo;
 import static org.junit.Assert.fail;
 import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.anyString;
 import static org.mockito.Matchers.argThat;
 import static org.mockito.Matchers.eq;
 import static org.mockito.Matchers.same;
-import static org.mockito.Mockito.*;
+import static org.mockito.Mockito.atLeast;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
 
 import java.io.File;
 import java.lang.Thread.State;
@@ -60,6 +71,7 @@ import org.hamcrest.Matcher;
 import org.hamcrest.TypeSafeDiagnosingMatcher;
 import org.hamcrest.TypeSafeMatcher;
 import org.junit.Assert;
+import org.junit.Before;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.mockito.Matchers;
@@ -78,6 +90,7 @@ import com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcess
 import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.WorkerCWMetricsFactory;
 import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.WorkerThreadPoolExecutor;
 import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
+import com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisProxy;
 import com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisLocalFileProxy;
 import com.amazonaws.services.kinesis.clientlibrary.proxies.util.KinesisLocalFileDataCreator;
 import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
@@ -129,6 +142,9 @@ public class WorkerTest {
 
     private static final String KINESIS_SHARD_ID_FORMAT = "kinesis-0-0-%d";
     private static final String CONCURRENCY_TOKEN_FORMAT = "testToken-%d";
+    
+    private RecordsFetcherFactory recordsFetcherFactory;
+    private KinesisClientLibConfiguration config;
 
     @Mock
     private KinesisClientLibLeaseCoordinator leaseCoordinator;
@@ -154,6 +170,13 @@ public class WorkerTest {
     private Future<TaskResult> taskFuture;
     @Mock
     private TaskResult taskResult;
+    
+    @Before
+    public void setup() {
+        config = spy(new KinesisClientLibConfiguration("app", null, null, null));
+        recordsFetcherFactory = spy(new SimpleRecordsFetcherFactory());
+        when(config.getRecordsFetcherFactory()).thenReturn(recordsFetcherFactory);
+    }
 
     // CHECKSTYLE:IGNORE AnonInnerLengthCheck FOR NEXT 50 LINES
     private static final com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory SAMPLE_RECORD_PROCESSOR_FACTORY = 
@@ -195,14 +218,13 @@ public class WorkerTest {
 
 
     /**
-     * Test method for {@link com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker#getApplicationName()}.
+     * Test method for {@link Worker#getApplicationName()}.
      */
     @Test
     public final void testGetStageName() {
         final String stageName = "testStageName";
-        final KinesisClientLibConfiguration clientConfig =
+        config = new KinesisClientLibConfiguration(stageName, null, null, null);
-                new KinesisClientLibConfiguration(stageName, null, null, null);
+        Worker worker = new Worker(v1RecordProcessorFactory, config);
-        Worker worker = new Worker(v1RecordProcessorFactory, clientConfig);
         Assert.assertEquals(stageName, worker.getApplicationName());
     }
 
@@ -210,6 +232,7 @@ public class WorkerTest {
     public final void testCreateOrGetShardConsumer() {
         final String stageName = "testStageName";
         IRecordProcessorFactory streamletFactory = SAMPLE_RECORD_PROCESSOR_FACTORY_V2;
+        config = new KinesisClientLibConfiguration(stageName, null, null, null);
         IKinesisProxy proxy = null;
         ICheckpoint checkpoint = null;
         int maxRecords = 1;
@@ -228,7 +251,9 @@ public class WorkerTest {
 
         Worker worker =
                 new Worker(stageName,
-                        streamletFactory, streamConfig, INITIAL_POSITION_LATEST,
+                        streamletFactory,
+                        config,
+                        streamConfig, INITIAL_POSITION_LATEST,
                         parentShardPollIntervalMillis,
                         shardSyncIntervalMillis,
                         cleanupLeasesUponShardCompletion,
@@ -275,10 +300,22 @@ public class WorkerTest {
         when(leaseCoordinator.getCurrentAssignments()).thenReturn(initialState).thenReturn(firstCheckpoint)
                 .thenReturn(secondCheckpoint);
 
-        Worker worker = new Worker(stageName, streamletFactory, streamConfig, INITIAL_POSITION_LATEST,
+        Worker worker = new Worker(stageName,
-                parentShardPollIntervalMillis, shardSyncIntervalMillis, cleanupLeasesUponShardCompletion, checkpoint,
+                streamletFactory,
-                leaseCoordinator, execService, nullMetricsFactory, taskBackoffTimeMillis, failoverTimeMillis,
+                config,
-                KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST, shardPrioritization);
+                streamConfig,
+                INITIAL_POSITION_LATEST,
+                parentShardPollIntervalMillis,
+                shardSyncIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                checkpoint,
+                leaseCoordinator,
+                execService,
+                nullMetricsFactory,
+                taskBackoffTimeMillis,
+                failoverTimeMillis,
+                KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                shardPrioritization);
 
         Worker workerSpy = spy(worker);
 
@@ -314,6 +351,7 @@ public class WorkerTest {
     public final void testCleanupShardConsumers() {
         final String stageName = "testStageName";
         IRecordProcessorFactory streamletFactory = SAMPLE_RECORD_PROCESSOR_FACTORY_V2;
+        config = new KinesisClientLibConfiguration(stageName, null, null, null);
         IKinesisProxy proxy = null;
         ICheckpoint checkpoint = null;
         int maxRecords = 1;
@@ -332,7 +370,9 @@ public class WorkerTest {
 
         Worker worker =
                 new Worker(stageName,
-                        streamletFactory, streamConfig, INITIAL_POSITION_LATEST,
+                        streamletFactory,
+                        config,
+                        streamConfig, INITIAL_POSITION_LATEST,
                         parentShardPollIntervalMillis,
                         shardSyncIntervalMillis,
                         cleanupLeasesUponShardCompletion,
@@ -371,6 +411,7 @@ public class WorkerTest {
     public final void testInitializationFailureWithRetries() {
         String stageName = "testInitializationWorker";
         IRecordProcessorFactory recordProcessorFactory = new TestStreamletFactory(null, null);
+        config = new KinesisClientLibConfiguration(stageName, null, null, null);
         int count = 0;
         when(proxy.getShardList()).thenThrow(new RuntimeException(Integer.toString(count++)));
         int maxRecords = 2;
@@ -386,6 +427,7 @@ public class WorkerTest {
         Worker worker =
                 new Worker(stageName,
                         recordProcessorFactory,
+                        config,
                         streamConfig, INITIAL_POSITION_TRIM_HORIZON,
                         shardPollInterval,
                         shardSyncIntervalMillis,
@@ -437,7 +479,7 @@ public class WorkerTest {
 
     /**
      * Runs worker with threadPoolSize < numShards
-     * Test method for {@link com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker#run()}.
+     * Test method for {@link Worker#run()}.
      */
     @Test
     public final void testOneSplitShard2Threads() throws Exception {
@@ -448,12 +490,12 @@ public class WorkerTest {
         KinesisClientLease lease = ShardSyncer.newKCLLease(shardList.get(0));
         lease.setCheckpoint(new ExtendedSequenceNumber("2"));
         initialLeases.add(lease);
-        runAndTestWorker(shardList, threadPoolSize, initialLeases, callProcessRecordsForEmptyRecordList, numberOfRecordsPerShard);
+        runAndTestWorker(shardList, threadPoolSize, initialLeases, callProcessRecordsForEmptyRecordList, numberOfRecordsPerShard, config);
     }
 
     /**
      * Runs worker with threadPoolSize < numShards
-     * Test method for {@link com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker#run()}.
+     * Test method for {@link Worker#run()}.
      */
     @Test
     public final void testOneSplitShard2ThreadsWithCallsForEmptyRecords() throws Exception {
@@ -465,7 +507,10 @@ public class WorkerTest {
         lease.setCheckpoint(new ExtendedSequenceNumber("2"));
         initialLeases.add(lease);
         boolean callProcessRecordsForEmptyRecordList = true;
-        runAndTestWorker(shardList, threadPoolSize, initialLeases, callProcessRecordsForEmptyRecordList, numberOfRecordsPerShard);
+        RecordsFetcherFactory recordsFetcherFactory = new SimpleRecordsFetcherFactory();
+        recordsFetcherFactory.setIdleMillisBetweenCalls(0L);
+        when(config.getRecordsFetcherFactory()).thenReturn(recordsFetcherFactory);
+        runAndTestWorker(shardList, threadPoolSize, initialLeases, callProcessRecordsForEmptyRecordList, numberOfRecordsPerShard, config);
     }
 
     @Test
@@ -490,7 +535,8 @@ public class WorkerTest {
                 10,
                 kinesisProxy, v2RecordProcessorFactory,
                 executorService,
-                cwMetricsFactory);
+                cwMetricsFactory,
+                config);
 
         // Give some time for thread to run.
         workerStarted.await();
@@ -526,7 +572,8 @@ public class WorkerTest {
                 10,
                 kinesisProxy, v2RecordProcessorFactory,
                 executorService,
-                cwMetricsFactory);
+                cwMetricsFactory,
+                config);
 
         // Give some time for thread to run.
         workerStarted.await();
@@ -572,6 +619,14 @@ public class WorkerTest {
                 return null;
             }
         }).when(v2RecordProcessor).processRecords(any(ProcessRecordsInput.class));
+        
+        RecordsFetcherFactory recordsFetcherFactory = mock(RecordsFetcherFactory.class);
+        GetRecordsCache getRecordsCache = mock(GetRecordsCache.class);
+        when(config.getRecordsFetcherFactory()).thenReturn(recordsFetcherFactory);
+        when(recordsFetcherFactory.createRecordsFetcher(any(GetRecordsRetrievalStrategy.class), anyString(),
+                any(IMetricsFactory.class), anyInt()))
+                .thenReturn(getRecordsCache);
+        when(getRecordsCache.getNextResult()).thenReturn(new ProcessRecordsInput().withRecords(Collections.emptyList()).withMillisBehindLatest(0L));
 
         WorkerThread workerThread = runWorker(shardList,
                 initialLeases,
@@ -581,7 +636,8 @@ public class WorkerTest {
                 fileBasedProxy,
                 v2RecordProcessorFactory,
                 executorService,
-                nullMetricsFactory);
+                nullMetricsFactory,
+                config);
 
         // Only sleep for time that is required.
         processRecordsLatch.await();
@@ -672,7 +728,8 @@ public class WorkerTest {
                 fileBasedProxy,
                 v2RecordProcessorFactory,
                 executorService,
-                nullMetricsFactory);
+                nullMetricsFactory,
+                config);
 
         // Only sleep for time that is required.
         processRecordsLatch.await();
@@ -742,10 +799,22 @@ public class WorkerTest {
         when(recordProcessorFactory.createProcessor()).thenReturn(processor);
 
 
-        Worker worker = new Worker("testRequestShutdown", recordProcessorFactory, streamConfig,
+        Worker worker = new Worker("testRequestShutdown",
-                INITIAL_POSITION_TRIM_HORIZON, parentShardPollIntervalMillis, shardSyncIntervalMillis,
+                recordProcessorFactory,
-                cleanupLeasesUponShardCompletion, leaseCoordinator, leaseCoordinator, executorService, metricsFactory,
+                config,
-                taskBackoffTimeMillis, failoverTimeMillis, false, shardPrioritization);
+                streamConfig,
+                INITIAL_POSITION_TRIM_HORIZON,
+                parentShardPollIntervalMillis,
+                shardSyncIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                leaseCoordinator,
+                leaseCoordinator,
+                executorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                failoverTimeMillis,
+                false,
+                shardPrioritization);
 
         when(executorService.submit(Matchers.<Callable<TaskResult>> any()))
                 .thenAnswer(new ShutdownHandlingAnswer(taskFuture));
@@ -816,7 +885,7 @@ public class WorkerTest {
         IRecordProcessor processor = mock(IRecordProcessor.class);
         when(recordProcessorFactory.createProcessor()).thenReturn(processor);
 
-        Worker worker = new InjectableWorker("testRequestShutdown", recordProcessorFactory, streamConfig,
+        Worker worker = new InjectableWorker("testRequestShutdown", recordProcessorFactory, config, streamConfig,
                 INITIAL_POSITION_TRIM_HORIZON, parentShardPollIntervalMillis, shardSyncIntervalMillis,
                 cleanupLeasesUponShardCompletion, leaseCoordinator, leaseCoordinator, executorService, metricsFactory,
                 taskBackoffTimeMillis, failoverTimeMillis, false, shardPrioritization) {
@@ -888,10 +957,22 @@ public class WorkerTest {
 
         when(coordinator.startGracefulShutdown(any(Callable.class))).thenReturn(gracefulShutdownFuture);
 
-        Worker worker = new InjectableWorker("testRequestShutdown", recordProcessorFactory, streamConfig,
+        Worker worker = new InjectableWorker("testRequestShutdown",
-                INITIAL_POSITION_TRIM_HORIZON, parentShardPollIntervalMillis, shardSyncIntervalMillis,
+                recordProcessorFactory,
-                cleanupLeasesUponShardCompletion, leaseCoordinator, leaseCoordinator, executorService, metricsFactory,
+                config,
-                taskBackoffTimeMillis, failoverTimeMillis, false, shardPrioritization) {
+                streamConfig,
+                INITIAL_POSITION_TRIM_HORIZON,
+                parentShardPollIntervalMillis,
+                shardSyncIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                leaseCoordinator,
+                leaseCoordinator,
+                executorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                failoverTimeMillis,
+                false,
+                shardPrioritization) {
             @Override
             void postConstruct() {
                 this.gracefulShutdownCoordinator = coordinator;
@@ -950,10 +1031,22 @@ public class WorkerTest {
         when(recordProcessorFactory.createProcessor()).thenReturn(processor);
 
 
-        Worker worker = new Worker("testRequestShutdown", recordProcessorFactory, streamConfig,
+        Worker worker = new Worker("testRequestShutdown",
-                INITIAL_POSITION_TRIM_HORIZON, parentShardPollIntervalMillis, shardSyncIntervalMillis,
+                recordProcessorFactory,
-                cleanupLeasesUponShardCompletion, leaseCoordinator, leaseCoordinator, executorService, metricsFactory,
+                config,
-                taskBackoffTimeMillis, failoverTimeMillis, false, shardPrioritization);
+                streamConfig,
+                INITIAL_POSITION_TRIM_HORIZON,
+                parentShardPollIntervalMillis,
+                shardSyncIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                leaseCoordinator,
+                leaseCoordinator,
+                executorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                failoverTimeMillis,
+                false,
+                shardPrioritization);
 
         when(executorService.submit(Matchers.<Callable<TaskResult>> any()))
                 .thenAnswer(new ShutdownHandlingAnswer(taskFuture));
@@ -1020,10 +1113,22 @@ public class WorkerTest {
         IRecordProcessor processor = mock(IRecordProcessor.class);
         when(recordProcessorFactory.createProcessor()).thenReturn(processor);
 
-        Worker worker = new Worker("testRequestShutdown", recordProcessorFactory, streamConfig,
+        Worker worker = new Worker("testRequestShutdown",
-                INITIAL_POSITION_TRIM_HORIZON, parentShardPollIntervalMillis, shardSyncIntervalMillis,
+                recordProcessorFactory,
-                cleanupLeasesUponShardCompletion, leaseCoordinator, leaseCoordinator, executorService, metricsFactory,
+                config,
-                taskBackoffTimeMillis, failoverTimeMillis, false, shardPrioritization);
+                streamConfig,
+                INITIAL_POSITION_TRIM_HORIZON,
+                parentShardPollIntervalMillis,
+                shardSyncIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                leaseCoordinator,
+                leaseCoordinator,
+                executorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                failoverTimeMillis,
+                false,
+                shardPrioritization);
 
         when(executorService.submit(Matchers.<Callable<TaskResult>> any()))
                 .thenAnswer(new ShutdownHandlingAnswer(taskFuture));
@@ -1121,10 +1226,22 @@ public class WorkerTest {
         IRecordProcessor processor = mock(IRecordProcessor.class);
         when(recordProcessorFactory.createProcessor()).thenReturn(processor);
 
-        Worker worker = new Worker("testRequestShutdown", recordProcessorFactory, streamConfig,
+        Worker worker = new Worker("testRequestShutdown",
-                INITIAL_POSITION_TRIM_HORIZON, parentShardPollIntervalMillis, shardSyncIntervalMillis,
+                recordProcessorFactory,
-                cleanupLeasesUponShardCompletion, leaseCoordinator, leaseCoordinator, executorService, metricsFactory,
+                config,
-                taskBackoffTimeMillis, failoverTimeMillis, false, shardPrioritization);
+                streamConfig,
+                INITIAL_POSITION_TRIM_HORIZON,
+                parentShardPollIntervalMillis,
+                shardSyncIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                leaseCoordinator,
+                leaseCoordinator,
+                executorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                failoverTimeMillis,
+                false,
+                shardPrioritization);
 
         when(executorService.submit(Matchers.<Callable<TaskResult>> any()))
                 .thenAnswer(new ShutdownHandlingAnswer(taskFuture));
@@ -1226,10 +1343,22 @@ public class WorkerTest {
         IRecordProcessor processor = mock(IRecordProcessor.class);
         when(recordProcessorFactory.createProcessor()).thenReturn(processor);
 
-        Worker worker = new Worker("testRequestShutdown", recordProcessorFactory, streamConfig,
+        Worker worker = new Worker("testRequestShutdown",
-                INITIAL_POSITION_TRIM_HORIZON, parentShardPollIntervalMillis, shardSyncIntervalMillis,
+                recordProcessorFactory,
-                cleanupLeasesUponShardCompletion, leaseCoordinator, leaseCoordinator, executorService, metricsFactory,
+                config,
-                taskBackoffTimeMillis, failoverTimeMillis, false, shardPrioritization);
+                streamConfig,
+                INITIAL_POSITION_TRIM_HORIZON,
+                parentShardPollIntervalMillis,
+                shardSyncIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                leaseCoordinator,
+                leaseCoordinator,
+                executorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                failoverTimeMillis, 
+                false,
+                shardPrioritization);
 
         when(executorService.submit(Matchers.<Callable<TaskResult>> any()))
                 .thenAnswer(new ShutdownHandlingAnswer(taskFuture));
@@ -1298,10 +1427,22 @@ public class WorkerTest {
         IRecordProcessor processor = mock(IRecordProcessor.class);
         when(recordProcessorFactory.createProcessor()).thenReturn(processor);
 
-        Worker worker = new Worker("testRequestShutdown", recordProcessorFactory, streamConfig,
+        Worker worker = new Worker("testRequestShutdown",
-                INITIAL_POSITION_TRIM_HORIZON, parentShardPollIntervalMillis, shardSyncIntervalMillis,
+                recordProcessorFactory,
-                cleanupLeasesUponShardCompletion, leaseCoordinator, leaseCoordinator, executorService, metricsFactory,
+                config,
-                taskBackoffTimeMillis, failoverTimeMillis, false, shardPrioritization);
+                streamConfig,
+                INITIAL_POSITION_TRIM_HORIZON,
+                parentShardPollIntervalMillis, 
+                shardSyncIntervalMillis,
+                cleanupLeasesUponShardCompletion,
+                leaseCoordinator,
+                leaseCoordinator,
+                executorService,
+                metricsFactory,
+                taskBackoffTimeMillis,
+                failoverTimeMillis,
+                false,
+                shardPrioritization);
 
         when(executorService.submit(Matchers.<Callable<TaskResult>> any()))
                 .thenAnswer(new ShutdownHandlingAnswer(taskFuture));
@@ -1334,18 +1475,56 @@ public class WorkerTest {
 
     }
 
+    @Test
+    public void testBuilderWithDefaultKinesisProxy() {
+        IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
+        Worker worker = new Worker.Builder()
+            .recordProcessorFactory(recordProcessorFactory)
+            .config(config)
+            .build();
+        Assert.assertNotNull(worker.getStreamConfig().getStreamProxy());
+        Assert.assertTrue(worker.getStreamConfig().getStreamProxy() instanceof KinesisProxy);
+    }
+
+    @Test
+    public void testBuilderWhenKinesisProxyIsSet() {
+        IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
+        // Create an instance of KinesisLocalFileProxy for injection and validation
+        IKinesisProxy kinesisProxy = mock(KinesisLocalFileProxy.class);
+        Worker worker = new Worker.Builder()
+            .recordProcessorFactory(recordProcessorFactory)
+            .config(config)
+            .kinesisProxy(kinesisProxy)
+            .build();
+        Assert.assertNotNull(worker.getStreamConfig().getStreamProxy());
+        Assert.assertTrue(worker.getStreamConfig().getStreamProxy() instanceof KinesisLocalFileProxy);
+    }
+
     private abstract class InjectableWorker extends Worker {
         InjectableWorker(String applicationName, IRecordProcessorFactory recordProcessorFactory,
-                StreamConfig streamConfig, InitialPositionInStreamExtended initialPositionInStream,
+                KinesisClientLibConfiguration config, StreamConfig streamConfig,
+                InitialPositionInStreamExtended initialPositionInStream,
                 long parentShardPollIntervalMillis, long shardSyncIdleTimeMillis,
                 boolean cleanupLeasesUponShardCompletion, ICheckpoint checkpoint,
                 KinesisClientLibLeaseCoordinator leaseCoordinator, ExecutorService execService,
                 IMetricsFactory metricsFactory, long taskBackoffTimeMillis, long failoverTimeMillis,
                 boolean skipShardSyncAtWorkerInitializationIfLeasesExist, ShardPrioritization shardPrioritization) {
-            super(applicationName, recordProcessorFactory, streamConfig, initialPositionInStream,
+            super(applicationName,
-                    parentShardPollIntervalMillis, shardSyncIdleTimeMillis, cleanupLeasesUponShardCompletion,
+                    recordProcessorFactory,
-                    checkpoint, leaseCoordinator, execService, metricsFactory, taskBackoffTimeMillis,
+                    config,
-                    failoverTimeMillis, skipShardSyncAtWorkerInitializationIfLeasesExist, shardPrioritization);
+                    streamConfig,
+                    initialPositionInStream,
+                    parentShardPollIntervalMillis,
+                    shardSyncIdleTimeMillis,
+                    cleanupLeasesUponShardCompletion,
+                    checkpoint,
+                    leaseCoordinator,
+                    execService,
+                    metricsFactory,
+                    taskBackoffTimeMillis,
+                    failoverTimeMillis,
+                    skipShardSyncAtWorkerInitializationIfLeasesExist,
+                    shardPrioritization);
             postConstruct();
         }
 
@@ -1578,14 +1757,15 @@ public class WorkerTest {
             lease.setCheckpoint(ExtendedSequenceNumber.AT_TIMESTAMP);
             initialLeases.add(lease);
         }
-        runAndTestWorker(shardList, threadPoolSize, initialLeases, callProcessRecordsForEmptyRecordList, numberOfRecordsPerShard);
+        runAndTestWorker(shardList, threadPoolSize, initialLeases, callProcessRecordsForEmptyRecordList, numberOfRecordsPerShard, config);
     }
 
     private void runAndTestWorker(List<Shard> shardList,
-            int threadPoolSize,
+                                  int threadPoolSize,
-            List<KinesisClientLease> initialLeases,
+                                  List<KinesisClientLease> initialLeases,
-            boolean callProcessRecordsForEmptyRecordList,
+                                  boolean callProcessRecordsForEmptyRecordList,
-            int numberOfRecordsPerShard) throws Exception {
+                                  int numberOfRecordsPerShard,
+                                  KinesisClientLibConfiguration clientConfig) throws Exception {
         File file = KinesisLocalFileDataCreator.generateTempDataFile(shardList, numberOfRecordsPerShard, "unitTestWT001");
         IKinesisProxy fileBasedProxy = new KinesisLocalFileProxy(file.getAbsolutePath());
 
@@ -1594,10 +1774,10 @@ public class WorkerTest {
         TestStreamletFactory recordProcessorFactory = new TestStreamletFactory(recordCounter, shardSequenceVerifier);
 
         ExecutorService executorService = Executors.newFixedThreadPool(threadPoolSize);
-
+        
         WorkerThread workerThread = runWorker(
                 shardList, initialLeases, callProcessRecordsForEmptyRecordList, failoverTimeMillis,
-                numberOfRecordsPerShard, fileBasedProxy, recordProcessorFactory, executorService, nullMetricsFactory);
+                numberOfRecordsPerShard, fileBasedProxy, recordProcessorFactory, executorService, nullMetricsFactory, clientConfig);
 
         // TestStreamlet will release the semaphore once for every record it processes
         recordCounter.acquire(numberOfRecordsPerShard * shardList.size());
@@ -1614,14 +1794,15 @@ public class WorkerTest {
     }
 
     private WorkerThread runWorker(List<Shard> shardList,
-            List<KinesisClientLease> initialLeases,
+                                   List<KinesisClientLease> initialLeases,
-            boolean callProcessRecordsForEmptyRecordList,
+                                   boolean callProcessRecordsForEmptyRecordList,
-            long failoverTimeMillis,
+                                   long failoverTimeMillis,
-            int numberOfRecordsPerShard,
+                                   int numberOfRecordsPerShard,
-            IKinesisProxy kinesisProxy,
+                                   IKinesisProxy kinesisProxy,
-            IRecordProcessorFactory recordProcessorFactory,
+                                   IRecordProcessorFactory recordProcessorFactory,
-            ExecutorService executorService,
+                                   ExecutorService executorService,
-            IMetricsFactory metricsFactory) throws Exception {
+                                   IMetricsFactory metricsFactory,
+                                   KinesisClientLibConfiguration clientConfig) throws Exception {
         final String stageName = "testStageName";
         final int maxRecords = 2;
 
@@ -1649,10 +1830,11 @@ public class WorkerTest {
                 idleTimeInMilliseconds,
                 callProcessRecordsForEmptyRecordList,
                 skipCheckpointValidationValue, InitialPositionInStreamExtended.newInitialPositionAtTimestamp(timestamp));
-
+        
         Worker worker =
                 new Worker(stageName,
                         recordProcessorFactory,
+                        clientConfig,
                         streamConfig, INITIAL_POSITION_TRIM_HORIZON,
                         parentShardPollIntervalMillis,
                         shardSyncIntervalMillis,

src/test/java/com/amazonaws/services/kinesis/leases/impl/LeaseIntegrationTest.java

@@ -1,21 +1,22 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.leases.impl;
 
 import java.util.logging.Logger;
 
+import com.amazonaws.services.kinesis.leases.exceptions.LeasingException;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.junit.Ignore;

src/test/java/com/amazonaws/services/kinesis/leases/impl/LeaseManagerIntegrationTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.leases.impl;
 
@@ -108,7 +108,8 @@ public class LeaseManagerIntegrationTest extends LeaseIntegrationTest {
 
         KinesisClientLease leaseCopy = leaseManager.getLease(lease.getLeaseKey());
 
-        leaseManager.renewLease(lease);
+        // lose lease
+        leaseManager.takeLease(lease, "bar");
 
         Assert.assertFalse(leaseManager.renewLease(leaseCopy));
     }

src/test/java/com/amazonaws/services/kinesis/leases/impl/LeaseRenewerIntegrationTest.java

@@ -1,30 +1,29 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.leases.impl;
 
-import java.util.Collections;
-import java.util.Map;
-import java.util.concurrent.Executors;
-
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.Test;
-
 import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
 import com.amazonaws.services.kinesis.leases.exceptions.LeasingException;
 import com.amazonaws.services.kinesis.leases.interfaces.ILeaseRenewer;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.Executors;
 
 public class LeaseRenewerIntegrationTest extends LeaseIntegrationTest {
 
@@ -58,7 +57,9 @@ public class LeaseRenewerIntegrationTest extends LeaseIntegrationTest {
         builder.addLeasesToRenew(renewer, "1", "2");
         KinesisClientLease renewedLease = builder.renewMutateAssert(renewer, "1", "2").get("2");
 
-        leaseManager.updateLease(renewedLease);
+        // lose lease 2
+        leaseManager.takeLease(renewedLease, "bar");
+
         builder.renewMutateAssert(renewer, "1");
     }
 
@@ -96,9 +97,9 @@ public class LeaseRenewerIntegrationTest extends LeaseIntegrationTest {
     public void testGetCurrentlyHeldLeases() throws LeasingException {
         TestHarnessBuilder builder = new TestHarnessBuilder(leaseManager);
 
-        KinesisClientLease lease2 = builder.withLease("1", "foo").withLease("2", "foo").build().get("2");
+        builder.withLease("1", "foo").withLease("2", "foo").build();
         builder.addLeasesToRenew(renewer, "1", "2");
-        builder.renewMutateAssert(renewer, "1", "2");
+        KinesisClientLease lease2 =  builder.renewMutateAssert(renewer, "1", "2").get("2");
 
         // This should be a copy that doesn't get updated
         Map<String, KinesisClientLease> heldLeases = renewer.getCurrentlyHeldLeases();
@@ -106,7 +107,9 @@ public class LeaseRenewerIntegrationTest extends LeaseIntegrationTest {
         Assert.assertEquals((Long) 1L, heldLeases.get("1").getLeaseCounter());
         Assert.assertEquals((Long) 1L, heldLeases.get("2").getLeaseCounter());
 
-        leaseManager.updateLease(lease2); // lose lease 2
+        // lose lease 2
+        leaseManager.takeLease(lease2, "bar");
+
         // Do another renewal and make sure the copy doesn't change
         builder.renewMutateAssert(renewer, "1");
 
@@ -176,7 +179,7 @@ public class LeaseRenewerIntegrationTest extends LeaseIntegrationTest {
         KinesisClientLease lease = renewer.getCurrentlyHeldLease("1");
 
         // cause lease loss such that the renewer knows the lease has been lost when update is called
-        leaseManager.renewLease(lease);
+        leaseManager.takeLease(lease, "bar");
         builder.renewMutateAssert(renewer);
 
         lease.setCheckpoint(new ExtendedSequenceNumber("new checkpoint"));
@@ -195,7 +198,7 @@ public class LeaseRenewerIntegrationTest extends LeaseIntegrationTest {
         KinesisClientLease lease = renewer.getCurrentlyHeldLease("1");
 
         // cause lease loss such that the renewer knows the lease has been lost when update is called
-        leaseManager.renewLease(lease);
+        leaseManager.takeLease(lease, "bar");
         builder.renewMutateAssert(renewer);
 
         // regain the lease

src/test/java/com/amazonaws/services/kinesis/leases/impl/TestHarnessBuilder.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License. 
  */
 package com.amazonaws.services.kinesis.leases.impl;
 
@@ -35,6 +35,7 @@ public class TestHarnessBuilder {
 
     private Map<String, KinesisClientLease> leases = new HashMap<String, KinesisClientLease>();
     private KinesisClientLeaseManager leaseManager;
+    private Map<String, KinesisClientLease> originalLeases = new HashMap<>();
 
     private Callable<Long> timeProvider = new Callable<Long>() {
 
@@ -54,6 +55,15 @@ public class TestHarnessBuilder {
     }
 
     public TestHarnessBuilder withLease(String shardId, String owner) {
+        KinesisClientLease lease = createLease(shardId, owner);
+        KinesisClientLease originalLease = createLease(shardId, owner);
+
+        leases.put(shardId, lease);
+        originalLeases.put(shardId, originalLease);
+        return this;
+    }
+
+    private KinesisClientLease createLease(String shardId, String owner) {
         KinesisClientLease lease = new KinesisClientLease();
         lease.setCheckpoint(new ExtendedSequenceNumber("checkpoint"));
         lease.setOwnerSwitchesSinceCheckpoint(0L);
@@ -62,8 +72,7 @@ public class TestHarnessBuilder {
         lease.setParentShardIds(Collections.singleton("parentShardId"));
         lease.setLeaseKey(shardId);
 
-        leases.put(shardId, lease);
+        return lease;
-        return this;
     }
 
     public Map<String, KinesisClientLease> build() throws LeasingException {
@@ -147,7 +156,7 @@ public class TestHarnessBuilder {
         Assert.assertEquals(renewedShardIds.length, heldLeases.size());
 
         for (String shardId : renewedShardIds) {
-            KinesisClientLease original = leases.get(shardId);
+            KinesisClientLease original = originalLeases.get(shardId);
             Assert.assertNotNull(original);
 
             KinesisClientLease actual = heldLeases.get(shardId);

src/test/java/com/amazonaws/services/kinesis/multilang/MessageWriterTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 

src/test/java/com/amazonaws/services/kinesis/multilang/MultiLangProtocolTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 

src/test/java/com/amazonaws/services/kinesis/multilang/StreamingRecordProcessorFactoryTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 

src/test/java/com/amazonaws/services/kinesis/multilang/StreamingRecordProcessorTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang;
 

src/test/java/com/amazonaws/services/kinesis/multilang/messages/MessageTest.java

@@ -1,16 +1,16 @@
 /*
- * Copyright 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  *
- * Licensed under the Amazon Software License (the "License").
+ *  Licensed under the Amazon Software License (the "License").
- * You may not use this file except in compliance with the License.
+ *  You may not use this file except in compliance with the License.
- * A copy of the License is located at
+ *  A copy of the License is located at
  *
- * http://aws.amazon.com/asl/
+ *  http://aws.amazon.com/asl/
  *
- * or in the "license" file accompanying this file. This file is distributed
+ *  or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+ *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
+ *  express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
+ *  permissions and limitations under the License.
  */
 package com.amazonaws.services.kinesis.multilang.messages;