Merge pull request #735 from ashwing/v1.x-ltr-release

Merging LTR changes to public KCL 1.x branch. Preparing for KCL 1.14.0 release

CHANGELOG.md

@@ -1,5 +1,52 @@
 # Changelog
-## Latest Release (1.13.3 March 2, 2020)
+
+## Latest Release (1.14.0 - August 17, 2020)
+
+* [Milestone#50](https://github.com/awslabs/amazon-kinesis-client/milestone/50)
+
+* Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.
+* When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.
+	* Currently, the following shard filters are supported:
+		* `AT_TRIM_HORIZON` - the response includes all the shards that were open at `TRIM_HORIZON`.
+		* `AT_LATEST` - the response includes only the currently open shards of the data stream.
+	    * `AT_TIMESTAMP` - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
+	* `ShardFilter` is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at `KinesisClientLibConfiguration#initialPositionInStreamExtended`.
+	* For more information about ShardFilter, see the [official AWS documentation on ShardFilter](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ShardFilter.html).
+
+* Introducing support for the `ChildShards` response of the `GetRecords` API to perform lease/shard synchronization that happens at `SHARD_END` for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.
+	* For KCL 1.x applications, this uses the `ChildShards` response of the `GetRecords` API.
+	* For more information, see the official AWS Documentation on [GetRecords](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) and [ChildShard](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ChildShard.html).
+
+* KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. When `KinesisClientLibConfiguration#shardSyncStrategyType` is set to `ShardSyncStrategyType.SHARD_END`, `PeriodicShardSyncManager#leasesRecoveryAuditorInconsistencyConfidenceThreshold` will be used to determine the threshold for number of consecutive scans containing holes in the lease table after which to enforce a shard sync. When `KinesisClientLibConfiguration#shardSyncStrategyType` is set to `ShardSyncStrategyType.PERIODIC`, `leasesRecoveryAuditorInconsistencyConfidenceThreshold` is ignored.
+	* New configuration options are available to configure `PeriodicShardSyncManager` in `KinesisClientLibConfiguration`
+
+	| Name                                                  | Default | Description                                                                                                                                                                                                                                                                                       |
+	| ----------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+	| leasesRecoveryAuditorInconsistencyConfidenceThreshold | 3       | Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync. Only used for `ShardSyncStrategyType.SHARD_END`. |
+
+	* New CloudWatch metrics are also now emitted to monitor the health of `PeriodicShardSyncManager`:
+
+	| Name                        | Description                                            |
+	| --------------------------- | ------------------------------------------------------ |
+	| NumStreamsWithPartialLeases | Number of streams that had holes in their hash ranges. |
+	| NumStreamsToSync            | Number of streams which underwent a full shard sync.   |
+
+* Introducing deferred lease cleanup. Leases will be deleted asynchronously by `LeaseCleanupManager` upon reaching `SHARD_END`, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.
+	* New configuration options are available to configure `LeaseCleanupManager`.
+
+	| Name                                | Default    | Description                                                                                               |
+	| ----------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------- |
+	| leaseCleanupIntervalMillis          | 1 minute   | Interval at which to run lease cleanup thread.                                                            |
+	| completedLeaseCleanupIntervalMillis | 5 minutes  | Interval at which to check if a lease is completed or not.                                                |
+	| garbageLeaseCleanupIntervalMillis   | 30 minutes | Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not. |
+
+* Including an optimization to `KinesisShardSyncer` to only create leases for one layer of shards.
+* Changing default shard prioritization strategy to be `NoOpShardPrioritization` to allow prioritization of completed shards. Customers who are upgrading to this version and are reading from `TRIM_HORIZON` should continue using `ParentsFirstShardPrioritization` while upgrading.
+* Upgrading version of AWS SDK to 1.11.844.
+* [#719](https://github.com/awslabs/amazon-kinesis-client/pull/719) Upgrading version of Google Protobuf to 3.11.4.
+* [#712](https://github.com/awslabs/amazon-kinesis-client/pull/712) Allowing KCL to consider lease tables in `UPDATING` healthy.
+
+## Release 1.13.3 (1.13.3 March 2, 2020)
 [Milestone#49] (https://github.com/awslabs/amazon-kinesis-client/milestone/49）
 * Refactoring shard closure verification performed by ShutdownTask.
   * [PR #684] (https://github.com/awslabs/amazon-kinesis-client/pull/684)

README.md

@@ -31,28 +31,51 @@ To make it easier for developers to write record processors in other languages,
 
 ## Release Notes
 
-#### Latest Release (1.13.3 March 2, 2020)
+### Latest Release (1.14.0 - August 17, 2020)
-* Refactoring shard closure verification performed by ShutdownTask.
-  * [PR #684] (https://github.com/awslabs/amazon-kinesis-client/pull/684)
-* Fixing the bug in ShardSyncTaskManager to resolve the issue of new shards not being processed after resharding.
-  * [PR #694] (https://github.com/awslabs/amazon-kinesis-client/pull/694)
 
-#### Release (1.13.2 Janurary 13, 2020)
+* [Milestone#50](https://github.com/awslabs/amazon-kinesis-client/milestone/50)
-* Adding backward compatible constructors that use the default DDB Billing Mode (#673)
-  * [PR #673](https://github.com/awslabs/amazon-kinesis-client/pull/673)
 
-#### Release (1.13.1 December 30, 2019)
+* Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.
-* Adding BillingMode Support to KCL 1.x. This enables the customer to specify if they want provisioned capacity for DDB, or pay per request.
+* When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.
-  * [PR #656](https://github.com/awslabs/amazon-kinesis-client/pull/656)
+	* Currently, the following shard filters are supported:
-* Ensure ShardSyncTask invocation from ShardSyncTaskManager for pending ShardEnd events.
+		* `AT_TRIM_HORIZON` - the response includes all the shards that were open at `TRIM_HORIZON`.
-  * [PR #659](https://github.com/awslabs/amazon-kinesis-client/pull/659)
+		* `AT_LATEST` - the response includes only the currently open shards of the data stream.
-* Fix the LeaseManagementIntegrationTest failure.
+	    * `AT_TIMESTAMP` - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
-  * [PR #670](https://github.com/awslabs/amazon-kinesis-client/pull/670)
+	* `ShardFilter` is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at `KinesisClientLibConfiguration#initialPositionInStreamExtended`.
+	* For more information about ShardFilter, see the [official AWS documentation on ShardFilter](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ShardFilter.html).
 
-#### Release (1.13.0 November 5, 2019)
+* Introducing support for the `ChildShards` response of the `GetRecords` API to perform lease/shard synchronization that happens at `SHARD_END` for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.
-* Handling completed and blocked tasks better during graceful shutdown
+	* For KCL 1.x applications, this uses the `ChildShards` response of the `GetRecords` API.
-  * [PR #640](https://github.com/awslabs/amazon-kinesis-client/pull/640)
+	* For more information, see the official AWS Documentation on [GetRecords](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) and [ChildShard](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ChildShard.html).
 
+* KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. When `KinesisClientLibConfiguration#shardSyncStrategyType` is set to `ShardSyncStrategyType.SHARD_END`, `PeriodicShardSyncManager#leasesRecoveryAuditorInconsistencyConfidenceThreshold` will be used to determine the threshold for number of consecutive scans containing holes in the lease table after which to enforce a shard sync. When `KinesisClientLibConfiguration#shardSyncStrategyType` is set to `ShardSyncStrategyType.PERIODIC`, `leasesRecoveryAuditorInconsistencyConfidenceThreshold` is ignored.
+	* New configuration options are available to configure `PeriodicShardSyncManager` in `KinesisClientLibConfiguration`
+
+	| Name                                                  | Default | Description                                                                                                                                                                                                                                                                                       |
+	| ----------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+	| leasesRecoveryAuditorInconsistencyConfidenceThreshold | 3       | Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync. Only used for `ShardSyncStrategyType.SHARD_END`. |
+
+	* New CloudWatch metrics are also now emitted to monitor the health of `PeriodicShardSyncManager`:
+
+	| Name                        | Description                                            |
+	| --------------------------- | ------------------------------------------------------ |
+	| NumStreamsWithPartialLeases | Number of streams that had holes in their hash ranges. |
+	| NumStreamsToSync            | Number of streams which underwent a full shard sync.   |
+
+* Introducing deferred lease cleanup. Leases will be deleted asynchronously by `LeaseCleanupManager` upon reaching `SHARD_END`, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.
+	* New configuration options are available to configure `LeaseCleanupManager`.
+
+	| Name                                | Default    | Description                                                                                               |
+	| ----------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------- |
+	| leaseCleanupIntervalMillis          | 1 minute   | Interval at which to run lease cleanup thread.                                                            |
+	| completedLeaseCleanupIntervalMillis | 5 minutes  | Interval at which to check if a lease is completed or not.                                                |
+	| garbageLeaseCleanupIntervalMillis   | 30 minutes | Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not. |
+
+* Including an optimization to `KinesisShardSyncer` to only create leases for one layer of shards.
+* Changing default shard prioritization strategy to be `NoOpShardPrioritization` to allow prioritization of completed shards. Customers who are upgrading to this version and are reading from `TRIM_HORIZON` should continue using `ParentsFirstShardPrioritization` while upgrading.
+* Upgrading version of AWS SDK to 1.11.844.
+* [#719](https://github.com/awslabs/amazon-kinesis-client/pull/719) Upgrading version of Google Protobuf to 3.11.4.
+* [#712](https://github.com/awslabs/amazon-kinesis-client/pull/712) Allowing KCL to consider lease tables in `UPDATING` healthy.
 
 ###### For remaining release notes check **[CHANGELOG.md][changelog-md]**.
 

pom.xml

@@ -6,7 +6,7 @@
   <artifactId>amazon-kinesis-client</artifactId>
   <packaging>jar</packaging>
   <name>Amazon Kinesis Client Library for Java</name>
-  <version>1.13.4-SNAPSHOT</version>
+  <version>1.14.0</version>
   <description>The Amazon Kinesis Client Library for Java enables Java developers to easily consume and process data
     from Amazon Kinesis.
   </description>
@@ -25,7 +25,7 @@
   </licenses>
 
   <properties>
-    <aws-java-sdk.version>1.11.728</aws-java-sdk.version>
+    <aws-java-sdk.version>1.11.844</aws-java-sdk.version>
     <sqlite4java.version>1.0.392</sqlite4java.version>
     <sqlite4java.native>libsqlite4java</sqlite4java.native>
     <sqlite4java.libpath>${project.build.directory}/test-lib</sqlite4java.libpath>

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ConsumerStates.java

@@ -199,7 +199,7 @@ class ConsumerStates {
 
         @Override
         public ConsumerState shutdownTransition(ShutdownReason shutdownReason) {
-            return ShardConsumerState.SHUTDOWN_COMPLETE.getConsumerState();
+            return ShardConsumerState.SHUTTING_DOWN.getConsumerState();
         }
 
         @Override
@@ -530,7 +530,9 @@ class ConsumerStates {
                     consumer.isIgnoreUnexpectedChildShards(),
                     consumer.getLeaseCoordinator(),
                     consumer.getTaskBackoffTimeMillis(),
-                    consumer.getGetRecordsCache(), consumer.getShardSyncer(), consumer.getShardSyncStrategy());
+                    consumer.getGetRecordsCache(), consumer.getShardSyncer(),
+                    consumer.getShardSyncStrategy(), consumer.getChildShards(),
+                    consumer.getLeaseCleanupManager());
         }
 
         @Override

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/EmptyLeaseTableSynchronizer.java

@@ -0,0 +1,98 @@
+/*
+ * Copyright 2019 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
+import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
+import com.amazonaws.services.kinesis.leases.impl.Lease;
+import com.amazonaws.services.kinesis.model.Shard;
+import lombok.AllArgsConstructor;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisShardSyncer.StartingSequenceNumberAndShardIdBasedComparator;
+
+/**
+ * Class to help create leases when the table is initially empty.
+ */
+@AllArgsConstructor
+class EmptyLeaseTableSynchronizer implements LeaseSynchronizer {
+
+    private static final Log LOG = LogFactory.getLog(EmptyLeaseTableSynchronizer.class);
+
+    /**
+     * Determines how to create leases when the lease table is initially empty. For this, we read all shards where
+     * the KCL is reading from. For any shards which are closed, we will discover their child shards through GetRecords
+     * child shard information.
+     *
+     * @param shards
+     * @param currentLeases
+     * @param initialPosition
+     * @param inconsistentShardIds
+     * @return
+     */
+    @Override
+    public List<KinesisClientLease> determineNewLeasesToCreate(List<Shard> shards,
+                                                               List<KinesisClientLease> currentLeases,
+                                                               InitialPositionInStreamExtended initialPosition,
+                                                               Set<String> inconsistentShardIds) {
+
+        final Map<String, Shard> shardIdToShardMapOfAllKinesisShards =
+                KinesisShardSyncer.constructShardIdToShardMap(shards);
+
+        currentLeases.forEach(lease -> LOG.debug("Existing lease: " + lease.getLeaseKey()));
+
+        final List<KinesisClientLease> newLeasesToCreate =
+                getLeasesToCreateForOpenAndClosedShards(initialPosition, shards);
+
+        final Comparator<KinesisClientLease> startingSequenceNumberComparator =
+                new StartingSequenceNumberAndShardIdBasedComparator(shardIdToShardMapOfAllKinesisShards);
+
+        newLeasesToCreate.sort(startingSequenceNumberComparator);
+        return newLeasesToCreate;
+    }
+
+    /**
+     * Helper method to create leases. For an empty lease table, we will be creating leases for all shards
+     * regardless of if they are open or closed. Closed shards will be unblocked via child shard information upon
+     * reaching SHARD_END.
+     */
+    private List<KinesisClientLease> getLeasesToCreateForOpenAndClosedShards(
+            InitialPositionInStreamExtended initialPosition,
+            List<Shard> shards) {
+
+        final Map<String, Lease> shardIdToNewLeaseMap = new HashMap<>();
+
+        for (Shard shard : shards) {
+            final String shardId = shard.getShardId();
+            final KinesisClientLease lease = KinesisShardSyncer.newKCLLease(shard);
+
+            final ExtendedSequenceNumber checkpoint = KinesisShardSyncer.convertToCheckpoint(initialPosition);
+            lease.setCheckpoint(checkpoint);
+
+            LOG.debug("Need to create a lease for shard with shardId " + shardId);
+            shardIdToNewLeaseMap.put(shardId, lease);
+        }
+
+        return new ArrayList(shardIdToNewLeaseMap.values());
+    }
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java

@@ -14,6 +14,7 @@
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import java.time.Duration;
 import java.util.Date;
 import java.util.Optional;
 import java.util.Set;
@@ -89,6 +90,23 @@ public class KinesisClientLibConfiguration {
      */
     public static final boolean DEFAULT_CLEANUP_LEASES_UPON_SHARDS_COMPLETION = true;
 
+    /**
+     * Interval to run lease cleanup thread in {@link LeaseCleanupManager}.
+     */
+    private static final long DEFAULT_LEASE_CLEANUP_INTERVAL_MILLIS = Duration.ofMinutes(1).toMillis();
+
+    /**
+     * Threshold in millis at which to check if there are any completed leases (leases for shards which have been
+     * closed as a result of a resharding operation) that need to be cleaned up.
+     */
+    private static final long DEFAULT_COMPLETED_LEASE_CLEANUP_THRESHOLD_MILLIS = Duration.ofMinutes(5).toMillis();
+
+    /**
+     * Threshold in millis at which to check if there are any garbage leases (leases for shards which no longer exist
+     * in the stream) that need to be cleaned up.
+     */
+    private static final long DEFAULT_GARBAGE_LEASE_CLEANUP_THRESHOLD_MILLIS = Duration.ofMinutes(30).toMillis();
+
     /**
      * Backoff time in milliseconds for Amazon Kinesis Client Library tasks (in the event of failures).
      */
@@ -129,7 +147,7 @@ public class KinesisClientLibConfiguration {
     /**
      * User agent set when Amazon Kinesis Client Library makes AWS requests.
      */
-    public static final String KINESIS_CLIENT_LIB_USER_AGENT = "amazon-kinesis-client-library-java-1.13.4-SNAPSHOT";
+    public static final String KINESIS_CLIENT_LIB_USER_AGENT = "amazon-kinesis-client-library-java-1.14.0";
 
     /**
      * KCL will validate client provided sequence numbers with a call to Amazon Kinesis before checkpointing for calls
@@ -175,6 +193,16 @@ public class KinesisClientLibConfiguration {
      */
     public static final ShardSyncStrategyType DEFAULT_SHARD_SYNC_STRATEGY_TYPE = ShardSyncStrategyType.SHARD_END;
 
+    /**
+     * Default Lease Recovery Auditor execution period for SHARD_END ShardSyncStrategyType.
+     */
+    public static final long LEASES_RECOVERY_AUDITOR_EXECUTION_FREQUENCY_MILLIS = 2 * 60 * 1000L;
+
+    /**
+     * Default Lease Recovery Auditor inconsistency confidence threshold for running full shard sync for SHARD_END ShardSyncStrategyType.
+     */
+    public static final int LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD = 3;
+
     /**
      * Default Shard prioritization strategy.
      */
@@ -200,6 +228,11 @@ public class KinesisClientLibConfiguration {
      */
     public static final int DEFAULT_MAX_LIST_SHARDS_RETRY_ATTEMPTS = 50;
 
+    /**
+     * The number of times the {@link Worker} will try to initialize before giving up.
+     */
+    public static final int DEFAULT_MAX_INITIALIZATION_ATTEMPTS = 20;
+
     @Getter
     private BillingMode billingMode;
     private String applicationName;
@@ -241,6 +274,11 @@ public class KinesisClientLibConfiguration {
     private ShardPrioritization shardPrioritization;
     private long shutdownGraceMillis;
     private ShardSyncStrategyType shardSyncStrategyType;
+    private long leaseCleanupIntervalMillis;
+    private long completedLeaseCleanupThresholdMillis;
+    private long garbageLeaseCleanupThresholdMillis;
+    private long leasesRecoveryAuditorExecutionFrequencyMillis;
+    private int leasesRecoveryAuditorInconsistencyConfidenceThreshold;
 
     @Getter
     private Optional<Integer> timeoutInSeconds = Optional.empty();
@@ -266,6 +304,9 @@ public class KinesisClientLibConfiguration {
     @Getter
     private int maxListShardsRetryAttempts = DEFAULT_MAX_LIST_SHARDS_RETRY_ATTEMPTS;
 
+    @Getter
+    private int maxInitializationAttempts = DEFAULT_MAX_INITIALIZATION_ATTEMPTS;
+
     /**
      * Constructor.
      *
@@ -276,6 +317,7 @@ public class KinesisClientLibConfiguration {
      * @param credentialsProvider Provides credentials used to sign AWS requests
      * @param workerId Used to distinguish different workers/processes of a Kinesis application
      */
+    @Deprecated
     public KinesisClientLibConfiguration(String applicationName,
             String streamName,
             AWSCredentialsProvider credentialsProvider,
@@ -295,6 +337,7 @@ public class KinesisClientLibConfiguration {
      * @param cloudWatchCredentialsProvider Provides credentials used to access CloudWatch
      * @param workerId Used to distinguish different workers/processes of a Kinesis application
      */
+    @Deprecated
     public KinesisClientLibConfiguration(String applicationName,
             String streamName,
             AWSCredentialsProvider kinesisCredentialsProvider,
@@ -365,6 +408,7 @@ public class KinesisClientLibConfiguration {
      */
     // CHECKSTYLE:IGNORE HiddenFieldCheck FOR NEXT 26 LINES
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 26 LINES
+    @Deprecated
     public KinesisClientLibConfiguration(String applicationName,
                                          String streamName,
                                          String kinesisEndpoint,
@@ -436,6 +480,7 @@ public class KinesisClientLibConfiguration {
      */
     // CHECKSTYLE:IGNORE HiddenFieldCheck FOR NEXT 26 LINES
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 26 LINES
+    @Deprecated
     public KinesisClientLibConfiguration(String applicationName,
                                          String streamName,
                                          String kinesisEndpoint,
@@ -462,54 +507,14 @@ public class KinesisClientLibConfiguration {
                                          String regionName,
                                          long shutdownGraceMillis,
                                          BillingMode billingMode) {
-        // Check following values are greater than zero
+
-        checkIsValuePositive("FailoverTimeMillis", failoverTimeMillis);
+        this(applicationName, streamName, kinesisEndpoint, dynamoDBEndpoint, initialPositionInStream, kinesisCredentialsProvider,
-        checkIsValuePositive("IdleTimeBetweenReadsInMillis", idleTimeBetweenReadsInMillis);
+                dynamoDBCredentialsProvider, cloudWatchCredentialsProvider, failoverTimeMillis, workerId, maxRecords, idleTimeBetweenReadsInMillis,
-        checkIsValuePositive("ParentShardPollIntervalMillis", parentShardPollIntervalMillis);
+                callProcessRecordsEvenForEmptyRecordList, parentShardPollIntervalMillis, shardSyncIntervalMillis, cleanupTerminatedShardsBeforeExpiry,
-        checkIsValuePositive("ShardSyncIntervalMillis", shardSyncIntervalMillis);
+                kinesisClientConfig, dynamoDBClientConfig, cloudWatchClientConfig, taskBackoffTimeMillis, metricsBufferTimeMillis,
-        checkIsValuePositive("MaxRecords", (long) maxRecords);
+                metricsMaxQueueSize, validateSequenceNumberBeforeCheckpointing, regionName, shutdownGraceMillis, billingMode,
-        checkIsValuePositive("TaskBackoffTimeMillis", taskBackoffTimeMillis);
+                new SimpleRecordsFetcherFactory(), DEFAULT_LEASE_CLEANUP_INTERVAL_MILLIS, DEFAULT_COMPLETED_LEASE_CLEANUP_THRESHOLD_MILLIS,
-        checkIsValuePositive("MetricsBufferTimeMills", metricsBufferTimeMillis);
+                DEFAULT_GARBAGE_LEASE_CLEANUP_THRESHOLD_MILLIS);
-        checkIsValuePositive("MetricsMaxQueueSize", (long) metricsMaxQueueSize);
-        checkIsValuePositive("ShutdownGraceMillis", shutdownGraceMillis);
-        this.applicationName = applicationName;
-        this.tableName = applicationName;
-        this.streamName = streamName;
-        this.kinesisEndpoint = kinesisEndpoint;
-        this.dynamoDBEndpoint = dynamoDBEndpoint;
-        this.initialPositionInStream = initialPositionInStream;
-        this.kinesisCredentialsProvider = kinesisCredentialsProvider;
-        this.dynamoDBCredentialsProvider = dynamoDBCredentialsProvider;
-        this.cloudWatchCredentialsProvider = cloudWatchCredentialsProvider;
-        this.failoverTimeMillis = failoverTimeMillis;
-        this.maxRecords = maxRecords;
-        this.idleTimeBetweenReadsInMillis = idleTimeBetweenReadsInMillis;
-        this.callProcessRecordsEvenForEmptyRecordList = callProcessRecordsEvenForEmptyRecordList;
-        this.parentShardPollIntervalMillis = parentShardPollIntervalMillis;
-        this.shardSyncIntervalMillis = shardSyncIntervalMillis;
-        this.cleanupLeasesUponShardCompletion = cleanupTerminatedShardsBeforeExpiry;
-        this.workerIdentifier = workerId;
-        this.kinesisClientConfig = checkAndAppendKinesisClientLibUserAgent(kinesisClientConfig);
-        this.dynamoDBClientConfig = checkAndAppendKinesisClientLibUserAgent(dynamoDBClientConfig);
-        this.cloudWatchClientConfig = checkAndAppendKinesisClientLibUserAgent(cloudWatchClientConfig);
-        this.taskBackoffTimeMillis = taskBackoffTimeMillis;
-        this.metricsBufferTimeMillis = metricsBufferTimeMillis;
-        this.metricsMaxQueueSize = metricsMaxQueueSize;
-        this.metricsLevel = DEFAULT_METRICS_LEVEL;
-        this.metricsEnabledDimensions = DEFAULT_METRICS_ENABLED_DIMENSIONS;
-        this.validateSequenceNumberBeforeCheckpointing = validateSequenceNumberBeforeCheckpointing;
-        this.regionName = regionName;
-        this.maxLeasesForWorker = DEFAULT_MAX_LEASES_FOR_WORKER;
-        this.maxLeasesToStealAtOneTime = DEFAULT_MAX_LEASES_TO_STEAL_AT_ONE_TIME;
-        this.initialLeaseTableReadCapacity = DEFAULT_INITIAL_LEASE_TABLE_READ_CAPACITY;
-        this.initialLeaseTableWriteCapacity = DEFAULT_INITIAL_LEASE_TABLE_WRITE_CAPACITY;
-        this.initialPositionInStreamExtended =
-                InitialPositionInStreamExtended.newInitialPosition(initialPositionInStream);
-        this.skipShardSyncAtWorkerInitializationIfLeasesExist = DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST;
-        this.shardSyncStrategyType = DEFAULT_SHARD_SYNC_STRATEGY_TYPE;
-        this.shardPrioritization = DEFAULT_SHARD_PRIORITIZATION;
-        this.recordsFetcherFactory = new SimpleRecordsFetcherFactory();
-        this.billingMode = billingMode;
     }
 
     /**
@@ -548,6 +553,7 @@ public class KinesisClientLibConfiguration {
      */
     // CHECKSTYLE:IGNORE HiddenFieldCheck FOR NEXT 26 LINES
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 26 LINES
+    @Deprecated
     public KinesisClientLibConfiguration(String applicationName,
             String streamName,
             String kinesisEndpoint,
@@ -573,6 +579,91 @@ public class KinesisClientLibConfiguration {
             boolean validateSequenceNumberBeforeCheckpointing,
             String regionName,
             RecordsFetcherFactory recordsFetcherFactory) {
+
+
+        this(applicationName, streamName, kinesisEndpoint, dynamoDBEndpoint, initialPositionInStream, kinesisCredentialsProvider,
+                dynamoDBCredentialsProvider, cloudWatchCredentialsProvider, failoverTimeMillis, workerId, maxRecords, idleTimeBetweenReadsInMillis,
+                callProcessRecordsEvenForEmptyRecordList, parentShardPollIntervalMillis, shardSyncIntervalMillis, cleanupTerminatedShardsBeforeExpiry,
+                kinesisClientConfig, dynamoDBClientConfig, cloudWatchClientConfig, taskBackoffTimeMillis, metricsBufferTimeMillis,
+                metricsMaxQueueSize, validateSequenceNumberBeforeCheckpointing, regionName, 0, DEFAULT_DDB_BILLING_MODE,
+                recordsFetcherFactory, DEFAULT_LEASE_CLEANUP_INTERVAL_MILLIS, DEFAULT_COMPLETED_LEASE_CLEANUP_THRESHOLD_MILLIS,
+                DEFAULT_GARBAGE_LEASE_CLEANUP_THRESHOLD_MILLIS);
+    }
+
+    /**
+     * @param applicationName Name of the Kinesis application
+     *        By default the application name is included in the user agent string used to make AWS requests. This
+     *        can assist with troubleshooting (e.g. distinguish requests made by separate applications).
+     * @param streamName Name of the Kinesis stream
+     * @param kinesisEndpoint Kinesis endpoint
+     * @param dynamoDBEndpoint DynamoDB endpoint
+     * @param initialPositionInStream One of LATEST or TRIM_HORIZON. The KinesisClientLibrary will start fetching
+     *        records from that location in the stream when an application starts up for the first time and there
+     *        are no checkpoints. If there are checkpoints, then we start from the checkpoint position.
+     * @param kinesisCredentialsProvider Provides credentials used to access Kinesis
+     * @param dynamoDBCredentialsProvider Provides credentials used to access DynamoDB
+     * @param cloudWatchCredentialsProvider Provides credentials used to access CloudWatch
+     * @param failoverTimeMillis Lease duration (leases not renewed within this period will be claimed by others)
+     * @param workerId Used to distinguish different workers/processes of a Kinesis application
+     * @param maxRecords Max records to read per Kinesis getRecords() call
+     * @param idleTimeBetweenReadsInMillis Idle time between calls to fetch data from Kinesis
+     * @param callProcessRecordsEvenForEmptyRecordList Call the IRecordProcessor::processRecords() API even if
+     *        GetRecords returned an empty record list.
+     * @param parentShardPollIntervalMillis Wait for this long between polls to check if parent shards are done
+     * @param shardSyncIntervalMillis Time between tasks to sync leases and Kinesis shards
+     * @param cleanupTerminatedShardsBeforeExpiry Clean up shards we've finished processing (don't wait for expiration
+     *        in Kinesis)
+     * @param kinesisClientConfig Client Configuration used by Kinesis client
+     * @param dynamoDBClientConfig Client Configuration used by DynamoDB client
+     * @param cloudWatchClientConfig Client Configuration used by CloudWatch client
+     * @param taskBackoffTimeMillis Backoff period when tasks encounter an exception
+     * @param metricsBufferTimeMillis Metrics are buffered for at most this long before publishing to CloudWatch
+     * @param metricsMaxQueueSize Max number of metrics to buffer before publishing to CloudWatch
+     * @param validateSequenceNumberBeforeCheckpointing whether KCL should validate client provided sequence numbers
+     *        with a call to Amazon Kinesis before checkpointing for calls to
+     *        {@link RecordProcessorCheckpointer#checkpoint(String)}
+     * @param regionName The region name for the service
+     * @param shutdownGraceMillis Time before gracefully shutdown forcefully terminates
+     * @param billingMode The DDB Billing mode to set for lease table creation.
+     * @param recordsFetcherFactory Factory to create the records fetcher to retrieve data from Kinesis for a given shard.
+     * @param leaseCleanupIntervalMillis Rate at which to run lease cleanup thread in
+     *        {@link com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager}
+     * @param completedLeaseCleanupThresholdMillis Threshold in millis at which to check if there are any completed leases
+     *        (leases for shards which have been closed as a result of a resharding operation) that need to be cleaned up.
+     * @param garbageLeaseCleanupThresholdMillis Threshold in millis at which to check if there are any garbage leases
+     *        (leases for shards which no longer exist in the stream) that need to be cleaned up.
+     */
+    public KinesisClientLibConfiguration(String applicationName,
+                                         String streamName,
+                                         String kinesisEndpoint,
+                                         String dynamoDBEndpoint,
+                                         InitialPositionInStream initialPositionInStream,
+                                         AWSCredentialsProvider kinesisCredentialsProvider,
+                                         AWSCredentialsProvider dynamoDBCredentialsProvider,
+                                         AWSCredentialsProvider cloudWatchCredentialsProvider,
+                                         long failoverTimeMillis,
+                                         String workerId,
+                                         int maxRecords,
+                                         long idleTimeBetweenReadsInMillis,
+                                         boolean callProcessRecordsEvenForEmptyRecordList,
+                                         long parentShardPollIntervalMillis,
+                                         long shardSyncIntervalMillis,
+                                         boolean cleanupTerminatedShardsBeforeExpiry,
+                                         ClientConfiguration kinesisClientConfig,
+                                         ClientConfiguration dynamoDBClientConfig,
+                                         ClientConfiguration cloudWatchClientConfig,
+                                         long taskBackoffTimeMillis,
+                                         long metricsBufferTimeMillis,
+                                         int metricsMaxQueueSize,
+                                         boolean validateSequenceNumberBeforeCheckpointing,
+                                         String regionName,
+                                         long shutdownGraceMillis,
+                                         BillingMode billingMode,
+                                         RecordsFetcherFactory recordsFetcherFactory,
+                                         long leaseCleanupIntervalMillis,
+                                         long completedLeaseCleanupThresholdMillis,
+                                         long garbageLeaseCleanupThresholdMillis) {
+
         // Check following values are greater than zero
         checkIsValuePositive("FailoverTimeMillis", failoverTimeMillis);
         checkIsValuePositive("IdleTimeBetweenReadsInMillis", idleTimeBetweenReadsInMillis);
@@ -617,9 +708,15 @@ public class KinesisClientLibConfiguration {
                 InitialPositionInStreamExtended.newInitialPosition(initialPositionInStream);
         this.skipShardSyncAtWorkerInitializationIfLeasesExist = DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST;
         this.shardSyncStrategyType = DEFAULT_SHARD_SYNC_STRATEGY_TYPE;
+        this.leasesRecoveryAuditorExecutionFrequencyMillis = LEASES_RECOVERY_AUDITOR_EXECUTION_FREQUENCY_MILLIS;
+        this.leasesRecoveryAuditorInconsistencyConfidenceThreshold = LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD;
         this.shardPrioritization = DEFAULT_SHARD_PRIORITIZATION;
         this.recordsFetcherFactory = recordsFetcherFactory;
+        this.leaseCleanupIntervalMillis = leaseCleanupIntervalMillis;
+        this.completedLeaseCleanupThresholdMillis = completedLeaseCleanupThresholdMillis;
+        this.garbageLeaseCleanupThresholdMillis = garbageLeaseCleanupThresholdMillis;
         this.shutdownGraceMillis = shutdownGraceMillis;
+        this.billingMode = billingMode;
     }
 
     // Check if value is positive, otherwise throw an exception
@@ -828,6 +925,29 @@ public class KinesisClientLibConfiguration {
         return cleanupLeasesUponShardCompletion;
     }
 
+    /**
+     * @return Interval in millis at which to run lease cleanup thread in {@link com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager}
+     */
+    public long leaseCleanupIntervalMillis() {
+        return leaseCleanupIntervalMillis;
+    }
+
+    /**
+     * @return Interval in millis at which to check if there are any completed leases (leases for shards which have been
+     * closed as a result of a resharding operation) that need to be cleaned up.
+     */
+    public long completedLeaseCleanupThresholdMillis() {
+        return completedLeaseCleanupThresholdMillis;
+    }
+
+    /**
+     * @return Interval in millis at which to check if there are any garbage leases (leases for shards which no longer
+     * exist in the stream) that need to be cleaned up.
+     */
+    public long garbageLeaseCleanupThresholdMillis() {
+        return garbageLeaseCleanupThresholdMillis;
+    }
+
     /**
      * @return true if we should ignore child shards which have open parents
      */
@@ -864,6 +984,20 @@ public class KinesisClientLibConfiguration {
         return shardSyncStrategyType;
     }
 
+    /**
+     * @return leasesRecoveryAuditorExecutionFrequencyMillis to be used by SHARD_END ShardSyncStrategyType.
+     */
+    public long getLeasesRecoveryAuditorExecutionFrequencyMillis() {
+        return leasesRecoveryAuditorExecutionFrequencyMillis;
+    }
+
+    /**
+     * @return leasesRecoveryAuditorInconsistencyConfidenceThreshold to be used by SHARD_END ShardSyncStrategyType.
+     */
+    public int getLeasesRecoveryAuditorInconsistencyConfidenceThreshold() {
+        return leasesRecoveryAuditorInconsistencyConfidenceThreshold;
+    }
+
     /**
      * @return Max leases this Worker can handle at a time
      */
@@ -1241,6 +1375,24 @@ public class KinesisClientLibConfiguration {
         return this;
     }
 
+    /**
+     * @param leasesRecoveryAuditorExecutionFrequencyMillis Leases Recovery Auditor Execution period.
+     * @return {@link KinesisClientLibConfiguration}
+     */
+    public KinesisClientLibConfiguration withLeasesRecoveryAuditorExecutionFrequencyMillis(long leasesRecoveryAuditorExecutionFrequencyMillis) {
+        this.leasesRecoveryAuditorExecutionFrequencyMillis = leasesRecoveryAuditorExecutionFrequencyMillis;
+        return this;
+    }
+
+    /**
+     * @param leasesRecoveryAuditorInconsistencyConfidenceThreshold Leases Recovery Auditor Execution inconsistency confidence threshold.
+     * @return {@link KinesisClientLibConfiguration}
+     */
+    public KinesisClientLibConfiguration withLeasesRecoveryAuditorInconsistencyConfidenceThreshold(int leasesRecoveryAuditorInconsistencyConfidenceThreshold) {
+        this.leasesRecoveryAuditorInconsistencyConfidenceThreshold = leasesRecoveryAuditorInconsistencyConfidenceThreshold;
+        return this;
+    }
+
     /**
      *
      * @param regionName The region name for the service
@@ -1458,4 +1610,49 @@ public class KinesisClientLibConfiguration {
         this.maxListShardsRetryAttempts = maxListShardsRetryAttempts;
         return this;
     }
+
+    /**
+     * @param maxInitializationAttempts Max number of Worker initialization attempts before giving up
+     * @return
+     */
+    public KinesisClientLibConfiguration withMaxInitializationAttempts(int maxInitializationAttempts) {
+        checkIsValuePositive("maxInitializationAttempts", maxInitializationAttempts);
+        this.maxInitializationAttempts = maxInitializationAttempts;
+        return this;
+    }
+
+    /**
+     * @param leaseCleanupIntervalMillis Rate at which to run lease cleanup thread in
+     * {@link com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager}
+     * @return
+     */
+    public KinesisClientLibConfiguration withLeaseCleanupIntervalMillis(long leaseCleanupIntervalMillis) {
+        checkIsValuePositive("leaseCleanupIntervalMillis", leaseCleanupIntervalMillis);
+        this.leaseCleanupIntervalMillis = leaseCleanupIntervalMillis;
+        return this;
+    }
+
+    /**
+     * Threshold in millis at which to check if there are any completed leases (leases for shards which have been
+     * closed as a result of a resharding operation) that need to be cleaned up.
+     * @param completedLeaseCleanupThresholdMillis
+     * @return
+     */
+    public KinesisClientLibConfiguration withCompletedLeaseCleanupThresholdMillis(long completedLeaseCleanupThresholdMillis) {
+        checkIsValuePositive("completedLeaseCleanupThresholdMillis", completedLeaseCleanupThresholdMillis);
+        this.completedLeaseCleanupThresholdMillis = completedLeaseCleanupThresholdMillis;
+        return this;
+    }
+
+    /**
+     * Threshold in millis at which to check if there are any garbage leases (leases for shards which no longer exist
+     * in the stream) that need to be cleaned up.
+     * @param garbageLeaseCleanupThresholdMillis
+     * @return
+     */
+    public KinesisClientLibConfiguration withGarbageLeaseCleanupThresholdMillis(long  garbageLeaseCleanupThresholdMillis) {
+        checkIsValuePositive("garbageLeaseCleanupThresholdMillis",  garbageLeaseCleanupThresholdMillis);
+        this.garbageLeaseCleanupThresholdMillis =  garbageLeaseCleanupThresholdMillis;
+        return this;
+    }
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisDataFetcher.java

@@ -16,7 +16,10 @@ package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
 import java.util.Collections;
 import java.util.Date;
+import java.util.List;
 
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.kinesis.model.ChildShard;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
@@ -47,6 +50,7 @@ class KinesisDataFetcher {
     private boolean isInitialized;
     private String lastKnownSequenceNumber;
     private InitialPositionInStreamExtended initialPositionInStream;
+    private List<ChildShard> childShards = Collections.emptyList();
 
     /**
      *
@@ -85,8 +89,11 @@ class KinesisDataFetcher {
     final DataFetcherResult TERMINAL_RESULT = new DataFetcherResult() {
         @Override
         public GetRecordsResult getResult() {
-            return new GetRecordsResult().withMillisBehindLatest(null).withRecords(Collections.emptyList())
+            return new GetRecordsResult()
-                    .withNextShardIterator(null);
+                    .withMillisBehindLatest(null)
+                    .withRecords(Collections.emptyList())
+                    .withNextShardIterator(null)
+                    .withChildShards(Collections.emptyList());
         }
 
         @Override
@@ -113,12 +120,20 @@ class KinesisDataFetcher {
 
         @Override
         public GetRecordsResult accept() {
+            if (!isValidResult(result)) {
+                // Throwing SDK exception when the GetRecords result is not valid. This will allow PrefetchGetRecordsCache to retry the GetRecords call.
+                throw new SdkClientException("Shard " + shardId +": GetRecordsResult is not valid. NextShardIterator: " + result.getNextShardIterator()
+                                                     + ". ChildShards: " + result.getChildShards());
+            }
             nextIterator = result.getNextShardIterator();
             if (!CollectionUtils.isNullOrEmpty(result.getRecords())) {
                 lastKnownSequenceNumber = Iterables.getLast(result.getRecords()).getSequenceNumber();
             }
             if (nextIterator == null) {
-                LOG.info("Reached shard end: nextIterator is null in AdvancingResult.accept for shard " + shardId);
+                LOG.info("Reached shard end: nextIterator is null in AdvancingResult.accept for shard " + shardId + ". childShards: " + result.getChildShards());
+                if (!CollectionUtils.isNullOrEmpty(result.getChildShards())) {
+                    childShards = result.getChildShards();
+                }
                 isShardEndReached = true;
             }
             return getResult();
@@ -130,6 +145,23 @@ class KinesisDataFetcher {
         }
     }
 
+    private boolean isValidResult(GetRecordsResult getRecordsResult) {
+        // GetRecords result should contain childShard information. There are two valid combination for the nextShardIterator and childShards
+        // If the GetRecords call does not reach the shard end, getRecords result should contain a non-null nextShardIterator and an empty list of childShards.
+        // If the GetRecords call reaches the shard end, getRecords result should contain a null nextShardIterator and a non-empty list of childShards.
+        // All other combinations are invalid and indicating an issue with GetRecords result from Kinesis service.
+        if (getRecordsResult.getNextShardIterator() == null && CollectionUtils.isNullOrEmpty(getRecordsResult.getChildShards()) ||
+                getRecordsResult.getNextShardIterator() != null && !CollectionUtils.isNullOrEmpty(getRecordsResult.getChildShards())) {
+            return false;
+        }
+        for (ChildShard childShard : getRecordsResult.getChildShards()) {
+            if (CollectionUtils.isNullOrEmpty(childShard.getParentShards())) {
+                return false;
+            }
+        }
+        return true;
+    }
+
     /**
      * Initializes this KinesisDataFetcher's iterator based on the checkpointed sequence number.
      * @param initialCheckpoint Current checkpoint sequence number for this shard.
@@ -141,8 +173,7 @@ class KinesisDataFetcher {
         isInitialized = true;
     }
 
-    public void initialize(ExtendedSequenceNumber initialCheckpoint,
+    public void initialize(ExtendedSequenceNumber initialCheckpoint, InitialPositionInStreamExtended initialPositionInStream) {
-            InitialPositionInStreamExtended initialPositionInStream) {
         LOG.info("Initializing shard " + shardId + " with " + initialCheckpoint.getSequenceNumber());
         advanceIteratorTo(initialCheckpoint.getSequenceNumber(), initialPositionInStream);
         isInitialized = true;
@@ -171,6 +202,7 @@ class KinesisDataFetcher {
         if (nextIterator == null) {
             LOG.info("Reached shard end: cannot advance iterator for shard " + shardId);
             isShardEndReached = true;
+            // TODO: transition to ShuttingDown state on shardend instead to shutdown state for enqueueing this for cleanup
         }
         this.lastKnownSequenceNumber = sequenceNumber;
         this.initialPositionInStream = initialPositionInStream;
@@ -248,6 +280,10 @@ class KinesisDataFetcher {
         return isShardEndReached;
     }
 
+    protected List<ChildShard> getChildShards() {
+        return childShards;
+    }
+
     /** Note: This method has package level access for testing purposes.
      * @return nextIterator
      */

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisLeaseCleanupValidator.java

@@ -10,6 +10,7 @@ import java.util.Set;
 /**
  * Represents the class that decides if a lease is eligible for cleanup.
  */
+@Deprecated
 class KinesisLeaseCleanupValidator implements LeaseCleanupValidator {
 
     private static final Log LOG = LogFactory.getLog(KinesisLeaseCleanupValidator.class);

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisShardSyncer.java

@@ -17,8 +17,6 @@ package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 import java.io.Serializable;
 import java.math.BigInteger;
 import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
 import java.util.Comparator;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -26,7 +24,11 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
+import com.amazonaws.services.kinesis.model.ChildShard;
+import com.amazonaws.services.kinesis.model.ShardFilter;
+import com.amazonaws.services.kinesis.model.ShardFilterType;
 import com.amazonaws.util.CollectionUtils;
+import lombok.NoArgsConstructor;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.apache.commons.lang3.StringUtils;
@@ -59,11 +61,10 @@ class KinesisShardSyncer implements ShardSyncer {
     }
 
     synchronized void bootstrapShardLeases(IKinesisProxy kinesisProxy, ILeaseManager<KinesisClientLease> leaseManager,
-            InitialPositionInStreamExtended initialPositionInStream, boolean cleanupLeasesOfCompletedShards,
+                                           InitialPositionInStreamExtended initialPositionInStream,
                                            boolean ignoreUnexpectedChildShards)
-            throws DependencyException, InvalidStateException, ProvisionedThroughputException,
+            throws DependencyException, InvalidStateException, ProvisionedThroughputException, KinesisClientLibIOException {
-            KinesisClientLibIOException {
+        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream,
-        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream, cleanupLeasesOfCompletedShards,
                 ignoreUnexpectedChildShards);
     }
 
@@ -86,7 +87,7 @@ class KinesisShardSyncer implements ShardSyncer {
                                           boolean ignoreUnexpectedChildShards)
             throws DependencyException, InvalidStateException, ProvisionedThroughputException,
             KinesisClientLibIOException {
-        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream, cleanupLeasesOfCompletedShards, ignoreUnexpectedChildShards);
+        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream, ignoreUnexpectedChildShards);
     }
 
     /**
@@ -109,7 +110,8 @@ class KinesisShardSyncer implements ShardSyncer {
             boolean cleanupLeasesOfCompletedShards, boolean ignoreUnexpectedChildShards, List<Shard> latestShards)
             throws DependencyException, InvalidStateException, ProvisionedThroughputException,
             KinesisClientLibIOException {
-        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream, cleanupLeasesOfCompletedShards, ignoreUnexpectedChildShards, latestShards);
+        syncShardLeases(kinesisProxy, leaseManager, initialPositionInStream,
+                ignoreUnexpectedChildShards, latestShards, leaseManager.isLeaseTableEmpty());
     }
 
     /**
@@ -118,7 +120,6 @@ class KinesisShardSyncer implements ShardSyncer {
      * @param kinesisProxy
      * @param leaseManager
      * @param initialPosition
-     * @param cleanupLeasesOfCompletedShards
      * @param ignoreUnexpectedChildShards
      * @throws DependencyException
      * @throws InvalidStateException
@@ -126,12 +127,21 @@ class KinesisShardSyncer implements ShardSyncer {
      * @throws KinesisClientLibIOException
      */
     private synchronized void syncShardLeases(IKinesisProxy kinesisProxy,
-            ILeaseManager<KinesisClientLease> leaseManager, InitialPositionInStreamExtended initialPosition,
+                                              ILeaseManager<KinesisClientLease> leaseManager,
-            boolean cleanupLeasesOfCompletedShards, boolean ignoreUnexpectedChildShards)
+                                              InitialPositionInStreamExtended initialPosition,
-            throws DependencyException, InvalidStateException, ProvisionedThroughputException,
+                                              boolean ignoreUnexpectedChildShards)
-            KinesisClientLibIOException {
+            throws DependencyException, InvalidStateException, ProvisionedThroughputException, KinesisClientLibIOException {
-        List<Shard> latestShards = getShardList(kinesisProxy);
+
-        syncShardLeases(kinesisProxy, leaseManager, initialPosition, cleanupLeasesOfCompletedShards, ignoreUnexpectedChildShards, latestShards);
+        // In the case where the lease table is empty, we want to synchronize the minimal amount of shards possible
+        // based on the given initial position.
+        // TODO: Implement shard list filtering on non-empty lease table case
+        final boolean isLeaseTableEmpty = leaseManager.isLeaseTableEmpty();
+        final List<Shard> latestShards = isLeaseTableEmpty
+                ? getShardListAtInitialPosition(kinesisProxy, initialPosition)
+                : getCompleteShardList(kinesisProxy);
+
+        syncShardLeases(kinesisProxy, leaseManager, initialPosition,
+                ignoreUnexpectedChildShards, latestShards, isLeaseTableEmpty);
     }
 
     /**
@@ -140,7 +150,6 @@ class KinesisShardSyncer implements ShardSyncer {
      * @param kinesisProxy
      * @param leaseManager
      * @param initialPosition
-     * @param cleanupLeasesOfCompletedShards
      * @param ignoreUnexpectedChildShards
      * @param latestShards latest snapshot of shards to reuse
      * @throws DependencyException
@@ -150,13 +159,17 @@ class KinesisShardSyncer implements ShardSyncer {
      */
     // CHECKSTYLE:OFF CyclomaticComplexity
     private synchronized void syncShardLeases(IKinesisProxy kinesisProxy,
-            ILeaseManager<KinesisClientLease> leaseManager, InitialPositionInStreamExtended initialPosition,
+                                              ILeaseManager<KinesisClientLease> leaseManager,
-            boolean cleanupLeasesOfCompletedShards, boolean ignoreUnexpectedChildShards, List<Shard> latestShards)
+                                              InitialPositionInStreamExtended initialPosition,
+                                              boolean ignoreUnexpectedChildShards,
+                                              List<Shard> latestShards,
+                                              boolean isLeaseTableEmpty)
             throws DependencyException, InvalidStateException, ProvisionedThroughputException,
             KinesisClientLibIOException {
+
         List<Shard> shards;
         if(CollectionUtils.isNullOrEmpty(latestShards)) {
-            shards = getShardList(kinesisProxy);
+            shards = isLeaseTableEmpty ? getShardListAtInitialPosition(kinesisProxy, initialPosition) : getCompleteShardList(kinesisProxy);
         } else {
             shards = latestShards;
         }
@@ -169,11 +182,16 @@ class KinesisShardSyncer implements ShardSyncer {
             assertAllParentShardsAreClosed(inconsistentShardIds);
         }
 
-        List<KinesisClientLease> currentLeases = leaseManager.listLeases();
+        // Determine which lease sync strategy to use based on the state of the lease table
+        final LeaseSynchronizer leaseSynchronizer = isLeaseTableEmpty
+                ? new EmptyLeaseTableSynchronizer()
+                : new NonEmptyLeaseTableSynchronizer(shardIdToShardMap, shardIdToChildShardIdsMap);
 
-        List<KinesisClientLease> newLeasesToCreate = determineNewLeasesToCreate(shards, currentLeases, initialPosition,
+        final List<KinesisClientLease> currentLeases = leaseManager.listLeases();
-                inconsistentShardIds);
+        final List<KinesisClientLease> newLeasesToCreate = determineNewLeasesToCreate(leaseSynchronizer, shards,
+                currentLeases, initialPosition, inconsistentShardIds);
         LOG.debug("Num new leases to create: " + newLeasesToCreate.size());
+
         for (KinesisClientLease lease : newLeasesToCreate) {
             long startTimeMillis = System.currentTimeMillis();
             boolean success = false;
@@ -190,11 +208,6 @@ class KinesisShardSyncer implements ShardSyncer {
             trackedLeases.addAll(currentLeases);
         }
         trackedLeases.addAll(newLeasesToCreate);
-        cleanupGarbageLeases(shards, trackedLeases, kinesisProxy, leaseManager);
-        if (cleanupLeasesOfCompletedShards) {
-            cleanupLeasesOfFinishedShards(currentLeases, shardIdToShardMap, shardIdToChildShardIdsMap, trackedLeases,
-                    leaseManager);
-        }
     }
     // CHECKSTYLE:ON CyclomaticComplexity
 
@@ -317,7 +330,7 @@ class KinesisShardSyncer implements ShardSyncer {
      * @param shardIdToShardMap
      * @return
      */
-    Map<String, Set<String>> constructShardIdToChildShardIdsMap(Map<String, Shard> shardIdToShardMap) {
+    static Map<String, Set<String>> constructShardIdToChildShardIdsMap(Map<String, Shard> shardIdToShardMap) {
         Map<String, Set<String>> shardIdToChildShardIdsMap = new HashMap<>();
         for (Map.Entry<String, Shard> entry : shardIdToShardMap.entrySet()) {
             String shardId = entry.getKey();
@@ -345,7 +358,7 @@ class KinesisShardSyncer implements ShardSyncer {
         return shardIdToChildShardIdsMap;
     }
 
-    private List<Shard> getShardList(IKinesisProxy kinesisProxy) throws KinesisClientLibIOException {
+    private List<Shard> getCompleteShardList(IKinesisProxy kinesisProxy) throws KinesisClientLibIOException {
         List<Shard> shards = kinesisProxy.getShardList();
         if (shards == null) {
             throw new KinesisClientLibIOException(
@@ -354,46 +367,50 @@ class KinesisShardSyncer implements ShardSyncer {
         return shards;
     }
 
+    private List<Shard> getShardListAtInitialPosition(IKinesisProxy kinesisProxy,
+                                                      InitialPositionInStreamExtended initialPosition)
+            throws KinesisClientLibIOException {
+
+        final ShardFilter shardFilter = getShardFilterAtInitialPosition(initialPosition);
+        final List<Shard> shards = kinesisProxy.getShardListWithFilter(shardFilter);
+
+        if (shards == null) {
+            throw new KinesisClientLibIOException(
+                    "Stream is not in ACTIVE OR UPDATING state - will retry getting the shard list.");
+        }
+
+        return shards;
+    }
+
+    private static ShardFilter getShardFilterAtInitialPosition(InitialPositionInStreamExtended initialPosition) {
+        ShardFilter shardFilter = new ShardFilter();
+
+        switch (initialPosition.getInitialPositionInStream()) {
+            case LATEST:
+                shardFilter = shardFilter.withType(ShardFilterType.AT_LATEST);
+                break;
+            case TRIM_HORIZON:
+                shardFilter = shardFilter.withType(ShardFilterType.AT_TRIM_HORIZON);
+                break;
+            case AT_TIMESTAMP:
+                shardFilter = shardFilter.withType(ShardFilterType.AT_TIMESTAMP)
+                        .withTimestamp(initialPosition.getTimestamp());
+                break;
+            default:
+                throw new IllegalArgumentException(initialPosition.getInitialPositionInStream()
+                        + " is not a supported initial position in a Kinesis stream. Supported initial positions are"
+                        + " AT_LATEST, AT_TRIM_HORIZON, and AT_TIMESTAMP.");
+        }
+
+        return shardFilter;
+    }
+
     /**
      * Determine new leases to create and their initial checkpoint.
      * Note: Package level access only for testing purposes.
      *
-     * For each open (no ending sequence number) shard without open parents that doesn't already have a lease,
+     * @param leaseSynchronizer determines the strategy to use when updating leases based on the current state of
-     * determine if it is a descendent of any shard which is or will be processed (e.g. for which a lease exists):
+     *        the lease table (empty vs. non-empty)
-     * If so, set checkpoint of the shard to TrimHorizon and also create leases for ancestors if needed.
-     * If not, set checkpoint of the shard to the initial position specified by the client.
-     * To check if we need to create leases for ancestors, we use the following rules:
-     *   * If we began (or will begin) processing data for a shard, then we must reach end of that shard before
-     *         we begin processing data from any of its descendants.
-     *   * A shard does not start processing data until data from all its parents has been processed.
-     * Note, if the initial position is LATEST and a shard has two parents and only one is a descendant - we'll create
-     * leases corresponding to both the parents - the parent shard which is not a descendant will have
-     * its checkpoint set to Latest.
-     *
-     * We assume that if there is an existing lease for a shard, then either:
-     *   * we have previously created a lease for its parent (if it was needed), or
-     *   * the parent shard has expired.
-     *
-     * For example:
-     * Shard structure (each level depicts a stream segment):
-     * 0 1 2 3 4   5   - shards till epoch 102
-     * \ / \ / |   |
-     *  6   7  4   5   - shards from epoch 103 - 205
-     *   \ /   |  / \
-     *    8    4 9  10 - shards from epoch 206 (open - no ending sequenceNumber)
-     * Current leases: (3, 4, 5)
-     * New leases to create: (2, 6, 7, 8, 9, 10)
-     *
-     * The leases returned are sorted by the starting sequence number - following the same order
-     * when persisting the leases in DynamoDB will ensure that we recover gracefully if we fail
-     * before creating all the leases.
-     *
-     * If a shard has no existing lease, is open, and is a descendant of a parent which is still open, we ignore it
-     * here; this happens when the list of shards is inconsistent, which could be due to pagination delay for very
-     * high shard count streams (i.e., dynamodb streams for tables with thousands of partitions).  This can only
-     * currently happen here if ignoreUnexpectedChildShards was true in syncShardleases.
-     *
-     *
      * @param shards List of all shards in Kinesis (we'll create new leases based on this set)
      * @param currentLeases List of current leases
      * @param initialPosition One of LATEST, TRIM_HORIZON, or AT_TIMESTAMP. We'll start fetching records from that
@@ -401,91 +418,33 @@ class KinesisShardSyncer implements ShardSyncer {
      * @param inconsistentShardIds Set of child shard ids having open parents.
      * @return List of new leases to create sorted by starting sequenceNumber of the corresponding shard
      */
-    List<KinesisClientLease> determineNewLeasesToCreate(List<Shard> shards, List<KinesisClientLease> currentLeases,
+    List<KinesisClientLease> determineNewLeasesToCreate(LeaseSynchronizer leaseSynchronizer,
-            InitialPositionInStreamExtended initialPosition, Set<String> inconsistentShardIds) {
+                                                        List<Shard> shards,
-        Map<String, KinesisClientLease> shardIdToNewLeaseMap = new HashMap<String, KinesisClientLease>();
+                                                        List<KinesisClientLease> currentLeases,
-        Map<String, Shard> shardIdToShardMapOfAllKinesisShards = constructShardIdToShardMap(shards);
+                                                        InitialPositionInStreamExtended initialPosition,
+                                                        Set<String> inconsistentShardIds) {
 
-        Set<String> shardIdsOfCurrentLeases = new HashSet<String>();
+        return leaseSynchronizer.determineNewLeasesToCreate(shards, currentLeases, initialPosition,
-        for (KinesisClientLease lease : currentLeases) {
+                inconsistentShardIds);
-            shardIdsOfCurrentLeases.add(lease.getLeaseKey());
-            LOG.debug("Existing lease: " + lease);
-        }
-
-        List<Shard> openShards = getOpenShards(shards);
-        Map<String, Boolean> memoizationContext = new HashMap<>();
-
-        // Iterate over the open shards and find those that don't have any lease entries.
-        for (Shard shard : openShards) {
-            String shardId = shard.getShardId();
-            LOG.debug("Evaluating leases for open shard " + shardId + " and its ancestors.");
-            if (shardIdsOfCurrentLeases.contains(shardId)) {
-                LOG.debug("Lease for shardId " + shardId + " already exists. Not creating a lease");
-            } else if (inconsistentShardIds.contains(shardId)) {
-                LOG.info("shardId " + shardId + " is an inconsistent child.  Not creating a lease");
-            } else {
-                LOG.debug("Need to create a lease for shardId " + shardId);
-                KinesisClientLease newLease = newKCLLease(shard);
-                boolean isDescendant = checkIfDescendantAndAddNewLeasesForAncestors(shardId, initialPosition,
-                        shardIdsOfCurrentLeases, shardIdToShardMapOfAllKinesisShards, shardIdToNewLeaseMap,
-                        memoizationContext);
-
-                /**
-                 * If the shard is a descendant and the specified initial position is AT_TIMESTAMP, then the
-                 * checkpoint should be set to AT_TIMESTAMP, else to TRIM_HORIZON. For AT_TIMESTAMP, we will add a
-                 * lease just like we do for TRIM_HORIZON. However we will only return back records with server-side
-                 * timestamp at or after the specified initial position timestamp.
-                 *
-                 * Shard structure (each level depicts a stream segment):
-                 * 0 1 2 3 4   5   - shards till epoch 102
-                 * \ / \ / |   |
-                 *  6   7  4   5   - shards from epoch 103 - 205
-                 *   \ /   |  /\
-                 *    8    4 9  10 - shards from epoch 206 (open - no ending sequenceNumber)
-                 *
-                 * Current leases: empty set
-                 *
-                 * For the above example, suppose the initial position in stream is set to AT_TIMESTAMP with
-                 * timestamp value 206. We will then create new leases for all the shards (with checkpoint set to
-                 * AT_TIMESTAMP), including the ancestor shards with epoch less than 206. However as we begin
-                 * processing the ancestor shards, their checkpoints would be updated to SHARD_END and their leases
-                 * would then be deleted since they won't have records with server-side timestamp at/after 206. And
-                 * after that we will begin processing the descendant shards with epoch at/after 206 and we will
-                 * return the records that meet the timestamp requirement for these shards.
-                 */
-                if (isDescendant && !initialPosition.getInitialPositionInStream()
-                        .equals(InitialPositionInStream.AT_TIMESTAMP)) {
-                    newLease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
-                } else {
-                    newLease.setCheckpoint(convertToCheckpoint(initialPosition));
-                }
-                LOG.debug("Set checkpoint of " + newLease.getLeaseKey() + " to " + newLease.getCheckpoint());
-                shardIdToNewLeaseMap.put(shardId, newLease);
-            }
-        }
-
-        List<KinesisClientLease> newLeasesToCreate = new ArrayList<KinesisClientLease>();
-        newLeasesToCreate.addAll(shardIdToNewLeaseMap.values());
-        Comparator<? super KinesisClientLease> startingSequenceNumberComparator = new StartingSequenceNumberAndShardIdBasedComparator(
-                shardIdToShardMapOfAllKinesisShards);
-        Collections.sort(newLeasesToCreate, startingSequenceNumberComparator);
-        return newLeasesToCreate;
     }
 
     /**
      * Determine new leases to create and their initial checkpoint.
      * Note: Package level access only for testing purposes.
      */
-    List<KinesisClientLease> determineNewLeasesToCreate(List<Shard> shards, List<KinesisClientLease> currentLeases,
+    List<KinesisClientLease> determineNewLeasesToCreate(LeaseSynchronizer leaseSynchronizer,
+                                                        List<Shard> shards,
+                                                        List<KinesisClientLease> currentLeases,
                                                         InitialPositionInStreamExtended initialPosition) {
+
         Set<String> inconsistentShardIds = new HashSet<String>();
-        return determineNewLeasesToCreate(shards, currentLeases, initialPosition, inconsistentShardIds);
+        return determineNewLeasesToCreate(leaseSynchronizer, shards, currentLeases, initialPosition, inconsistentShardIds);
     }
 
     /**
      * Note: Package level access for testing purposes only.
      * Check if this shard is a descendant of a shard that is (or will be) processed.
-     * Create leases for the ancestors of this shard as required.
+     * Create leases for the first ancestor of this shard that needs to be processed, as required.
      * See javadoc of determineNewLeasesToCreate() for rules and example.
      *
      * @param shardId The shardId to check.
@@ -498,12 +457,13 @@ class KinesisShardSyncer implements ShardSyncer {
      * @return true if the shard is a descendant of any current shard (lease already exists)
      */
     // CHECKSTYLE:OFF CyclomaticComplexity
-    boolean checkIfDescendantAndAddNewLeasesForAncestors(String shardId,
+    static boolean checkIfDescendantAndAddNewLeasesForAncestors(String shardId,
             InitialPositionInStreamExtended initialPosition, Set<String> shardIdsOfCurrentLeases,
             Map<String, Shard> shardIdToShardMapOfAllKinesisShards,
-            Map<String, KinesisClientLease> shardIdToLeaseMapOfNewShards, Map<String, Boolean> memoizationContext) {
+            Map<String, KinesisClientLease> shardIdToLeaseMapOfNewShards, MemoizationContext memoizationContext) {
+
+        final Boolean previousValue = memoizationContext.isDescendant(shardId);
 
-        Boolean previousValue = memoizationContext.get(shardId);
         if (previousValue != null) {
             return previousValue;
         }
@@ -523,10 +483,13 @@ class KinesisShardSyncer implements ShardSyncer {
                 shard = shardIdToShardMapOfAllKinesisShards.get(shardId);
                 parentShardIds = getParentShardIds(shard, shardIdToShardMapOfAllKinesisShards);
                 for (String parentShardId : parentShardIds) {
-                    // Check if the parent is a descendant, and include its ancestors.
+                    // Check if the parent is a descendant, and include its ancestors. Or, if the parent is NOT a
-                    if (checkIfDescendantAndAddNewLeasesForAncestors(parentShardId, initialPosition,
+                    // descendant but we should create a lease for it anyway (e.g. to include in processing from
-                            shardIdsOfCurrentLeases, shardIdToShardMapOfAllKinesisShards, shardIdToLeaseMapOfNewShards,
+                    // TRIM_HORIZON or AT_TIMESTAMP). If either is true, then we mark the current shard as a descendant.
-                            memoizationContext)) {
+                    final boolean isParentDescendant = checkIfDescendantAndAddNewLeasesForAncestors(parentShardId,
+                            initialPosition, shardIdsOfCurrentLeases, shardIdToShardMapOfAllKinesisShards,
+                            shardIdToLeaseMapOfNewShards, memoizationContext);
+                    if (isParentDescendant || memoizationContext.shouldCreateLease(parentShardId)) {
                         isDescendant = true;
                         descendantParentShardIds.add(parentShardId);
                         LOG.debug("Parent shard " + parentShardId + " is a descendant.");
@@ -539,13 +502,51 @@ class KinesisShardSyncer implements ShardSyncer {
                 if (isDescendant) {
                     for (String parentShardId : parentShardIds) {
                         if (!shardIdsOfCurrentLeases.contains(parentShardId)) {
-                            LOG.debug("Need to create a lease for shardId " + parentShardId);
                             KinesisClientLease lease = shardIdToLeaseMapOfNewShards.get(parentShardId);
+
+                            // If the lease for the parent shard does not already exist, there are two cases in which we
+                            // would want to create it:
+                            // - If we have already marked the parentShardId for lease creation in a prior recursive
+                            //   call. This could happen if we are trying to process from TRIM_HORIZON or AT_TIMESTAMP.
+                            // - If the parent shard is not a descendant but the current shard is a descendant, then
+                            //   the parent shard is the oldest shard in the shard hierarchy that does not have an
+                            //   ancestor in the lease table (the adjacent parent is necessarily a descendant, and
+                            //   therefore covered in the lease table). So we should create a lease for the parent.
+
                             if (lease == null) {
+                                if (memoizationContext.shouldCreateLease(parentShardId) ||
+                                        !descendantParentShardIds.contains(parentShardId)) {
+                                    LOG.debug("Need to create a lease for shardId " + parentShardId);
                                     lease = newKCLLease(shardIdToShardMapOfAllKinesisShards.get(parentShardId));
                                     shardIdToLeaseMapOfNewShards.put(parentShardId, lease);
                                 }
+                            }
 
+                            /**
+                             * If the shard is a descendant and the specified initial position is AT_TIMESTAMP, then the
+                             * checkpoint should be set to AT_TIMESTAMP, else to TRIM_HORIZON. For AT_TIMESTAMP, we will
+                             * add a lease just like we do for TRIM_HORIZON. However we will only return back records
+                             * with server-side timestamp at or after the specified initial position timestamp.
+                             *
+                             * Shard structure (each level depicts a stream segment):
+                             * 0 1 2 3 4   5   - shards till epoch 102
+                             * \ / \ / |   |
+                             *  6   7  4   5   - shards from epoch 103 - 205
+                             *   \ /   |  /\
+                             *    8    4 9  10 - shards from epoch 206 (open - no ending sequenceNumber)
+                             *
+                             * Current leases: (4, 5, 7)
+                             *
+                             * For the above example, suppose the initial position in stream is set to AT_TIMESTAMP with
+                             * timestamp value 206. We will then create new leases for all the shards 0 and 1 (with
+                             * checkpoint set AT_TIMESTAMP), even though these ancestor shards have an epoch less than
+                             * 206. However as we begin processing the ancestor shards, their checkpoints would be
+                             * updated to SHARD_END and their leases would then be deleted since they won't have records
+                             * with server-side timestamp at/after 206. And after that we will begin processing the
+                             * descendant shards with epoch at/after 206 and we will return the records that meet the
+                             * timestamp requirement for these shards.
+                             */
+                            if (lease != null) {
                                 if (descendantParentShardIds.contains(parentShardId) && !initialPosition
                                         .getInitialPositionInStream().equals(InitialPositionInStream.AT_TIMESTAMP)) {
                                     lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
@@ -554,22 +555,23 @@ class KinesisShardSyncer implements ShardSyncer {
                                 }
                             }
                         }
+                    }
                 } else {
-                    // This shard should be included, if the customer wants to process all records in the stream or
+                    // This shard is not a descendant, but should still be included if the customer wants to process all
-                    // if the initial position is AT_TIMESTAMP. For AT_TIMESTAMP, we will add a lease just like we do
+                    // records in the stream or if the initial position is AT_TIMESTAMP. For AT_TIMESTAMP, we will add a
-                    // for TRIM_HORIZON. However we will only return back records with server-side timestamp at or
+                    // lease just like we do for TRIM_HORIZON. However we will only return back records with server-side
-                    // after the specified initial position timestamp.
+                    // timestamp at or after the specified initial position timestamp.
                     if (initialPosition.getInitialPositionInStream().equals(InitialPositionInStream.TRIM_HORIZON)
                             || initialPosition.getInitialPositionInStream()
                             .equals(InitialPositionInStream.AT_TIMESTAMP)) {
-                        isDescendant = true;
+                        memoizationContext.setShouldCreateLease(shardId, true);
                     }
                 }
 
             }
         }
 
-        memoizationContext.put(shardId, isDescendant);
+        memoizationContext.setIsDescendant(shardId, isDescendant);
         return isDescendant;
     }
     // CHECKSTYLE:ON CyclomaticComplexity
@@ -583,7 +585,7 @@ class KinesisShardSyncer implements ShardSyncer {
      * @param shardIdToShardMapOfAllKinesisShards ShardId->Shard map containing all shards obtained via DescribeStream.
      * @return Set of parentShardIds
      */
-    Set<String> getParentShardIds(Shard shard, Map<String, Shard> shardIdToShardMapOfAllKinesisShards) {
+    static Set<String> getParentShardIds(Shard shard, Map<String, Shard> shardIdToShardMapOfAllKinesisShards) {
         Set<String> parentShardIds = new HashSet<String>(2);
         String parentShardId = shard.getParentShardId();
         if ((parentShardId != null) && shardIdToShardMapOfAllKinesisShards.containsKey(parentShardId)) {
@@ -596,150 +598,6 @@ class KinesisShardSyncer implements ShardSyncer {
         return parentShardIds;
     }
 
-    /**
-     * Delete leases corresponding to shards that no longer exist in the stream.
-     * Current scheme: Delete a lease if:
-     *   * the corresponding shard is not present in the list of Kinesis shards, AND
-     *   * the parentShardIds listed in the lease are also not present in the list of Kinesis shards.
-     * @param shards List of all Kinesis shards (assumed to be a consistent snapshot - when stream is in Active state).
-     * @param trackedLeases List of
-     * @param kinesisProxy Kinesis proxy (used to get shard list)
-     * @param leaseManager
-     * @throws KinesisClientLibIOException Thrown if we couldn't get a fresh shard list from Kinesis.
-     * @throws ProvisionedThroughputException
-     * @throws InvalidStateException
-     * @throws DependencyException
-     */
-    private void cleanupGarbageLeases(List<Shard> shards, List<KinesisClientLease> trackedLeases,
-            IKinesisProxy kinesisProxy, ILeaseManager<KinesisClientLease> leaseManager)
-            throws KinesisClientLibIOException, DependencyException, InvalidStateException,
-            ProvisionedThroughputException {
-        Set<String> kinesisShards = new HashSet<>();
-        for (Shard shard : shards) {
-            kinesisShards.add(shard.getShardId());
-        }
-
-        // Check if there are leases for non-existent shards
-        List<KinesisClientLease> garbageLeases = new ArrayList<>();
-        for (KinesisClientLease lease : trackedLeases) {
-            if (leaseCleanupValidator.isCandidateForCleanup(lease, kinesisShards)) {
-                garbageLeases.add(lease);
-            }
-        }
-
-        if (!garbageLeases.isEmpty()) {
-            LOG.info("Found " + garbageLeases.size() + " candidate leases for cleanup. Refreshing list of"
-                    + " Kinesis shards to pick up recent/latest shards");
-            List<Shard> currentShardList = getShardList(kinesisProxy);
-            Set<String> currentKinesisShardIds = new HashSet<>();
-            for (Shard shard : currentShardList) {
-                currentKinesisShardIds.add(shard.getShardId());
-            }
-
-            for (KinesisClientLease lease : garbageLeases) {
-                if (leaseCleanupValidator.isCandidateForCleanup(lease, currentKinesisShardIds)) {
-                    LOG.info("Deleting lease for shard " + lease.getLeaseKey()
-                            + " as it is not present in Kinesis stream.");
-                    leaseManager.deleteLease(lease);
-                }
-            }
-        }
-
-    }
-
-    /**
-     * Private helper method.
-     * Clean up leases for shards that meet the following criteria:
-     * a/ the shard has been fully processed (checkpoint is set to SHARD_END)
-     * b/ we've begun processing all the child shards: we have leases for all child shards and their checkpoint is not
-     *      TRIM_HORIZON.
-     *
-     * @param currentLeases List of leases we evaluate for clean up
-     * @param shardIdToShardMap Map of shardId->Shard (assumed to include all Kinesis shards)
-     * @param shardIdToChildShardIdsMap Map of shardId->childShardIds (assumed to include all Kinesis shards)
-     * @param trackedLeases List of all leases we are tracking.
-     * @param leaseManager Lease manager (will be used to delete leases)
-     * @throws DependencyException
-     * @throws InvalidStateException
-     * @throws ProvisionedThroughputException
-     * @throws KinesisClientLibIOException
-     */
-    private synchronized void cleanupLeasesOfFinishedShards(Collection<KinesisClientLease> currentLeases,
-            Map<String, Shard> shardIdToShardMap, Map<String, Set<String>> shardIdToChildShardIdsMap,
-            List<KinesisClientLease> trackedLeases, ILeaseManager<KinesisClientLease> leaseManager)
-            throws DependencyException, InvalidStateException, ProvisionedThroughputException,
-            KinesisClientLibIOException {
-        Set<String> shardIdsOfClosedShards = new HashSet<>();
-        List<KinesisClientLease> leasesOfClosedShards = new ArrayList<>();
-        for (KinesisClientLease lease : currentLeases) {
-            if (lease.getCheckpoint().equals(ExtendedSequenceNumber.SHARD_END)) {
-                shardIdsOfClosedShards.add(lease.getLeaseKey());
-                leasesOfClosedShards.add(lease);
-            }
-        }
-
-        if (!leasesOfClosedShards.isEmpty()) {
-            assertClosedShardsAreCoveredOrAbsent(shardIdToShardMap, shardIdToChildShardIdsMap, shardIdsOfClosedShards);
-            Comparator<? super KinesisClientLease> startingSequenceNumberComparator = new StartingSequenceNumberAndShardIdBasedComparator(
-                    shardIdToShardMap);
-            Collections.sort(leasesOfClosedShards, startingSequenceNumberComparator);
-            Map<String, KinesisClientLease> trackedLeaseMap = constructShardIdToKCLLeaseMap(trackedLeases);
-
-            for (KinesisClientLease leaseOfClosedShard : leasesOfClosedShards) {
-                String closedShardId = leaseOfClosedShard.getLeaseKey();
-                Set<String> childShardIds = shardIdToChildShardIdsMap.get(closedShardId);
-                if ((closedShardId != null) && (childShardIds != null) && (!childShardIds.isEmpty())) {
-                    cleanupLeaseForClosedShard(closedShardId, childShardIds, trackedLeaseMap, leaseManager);
-                }
-            }
-        }
-    }
-
-    /**
-     * Delete lease for the closed shard. Rules for deletion are:
-     * a/ the checkpoint for the closed shard is SHARD_END,
-     * b/ there are leases for all the childShardIds and their checkpoint is NOT TRIM_HORIZON
-     * Note: This method has package level access solely for testing purposes.
-     *
-     * @param closedShardId Identifies the closed shard
-     * @param childShardIds ShardIds of children of the closed shard
-     * @param trackedLeases shardId->KinesisClientLease map with all leases we are tracking (should not be null)
-     * @param leaseManager
-     * @throws ProvisionedThroughputException
-     * @throws InvalidStateException
-     * @throws DependencyException
-     */
-    synchronized void cleanupLeaseForClosedShard(String closedShardId, Set<String> childShardIds,
-            Map<String, KinesisClientLease> trackedLeases, ILeaseManager<KinesisClientLease> leaseManager)
-            throws DependencyException, InvalidStateException, ProvisionedThroughputException {
-        KinesisClientLease leaseForClosedShard = trackedLeases.get(closedShardId);
-        List<KinesisClientLease> childShardLeases = new ArrayList<>();
-
-        for (String childShardId : childShardIds) {
-            KinesisClientLease childLease = trackedLeases.get(childShardId);
-            if (childLease != null) {
-                childShardLeases.add(childLease);
-            }
-        }
-
-        if ((leaseForClosedShard != null) && (leaseForClosedShard.getCheckpoint()
-                .equals(ExtendedSequenceNumber.SHARD_END)) && (childShardLeases.size() == childShardIds.size())) {
-            boolean okayToDelete = true;
-            for (KinesisClientLease lease : childShardLeases) {
-                if (lease.getCheckpoint().equals(ExtendedSequenceNumber.TRIM_HORIZON)) {
-                    okayToDelete = false;
-                    break;
-                }
-            }
-
-            if (okayToDelete) {
-                LOG.info("Deleting lease for shard " + leaseForClosedShard.getLeaseKey()
-                        + " as it has been completely processed and processing of child shards has begun.");
-                leaseManager.deleteLease(leaseForClosedShard);
-            }
-        }
-    }
-
     /**
      * Helper method to create a new KinesisClientLease POJO for a shard.
      * Note: Package level access only for testing purposes
@@ -747,7 +605,7 @@ class KinesisShardSyncer implements ShardSyncer {
      * @param shard
      * @return
      */
-    KinesisClientLease newKCLLease(Shard shard) {
+    static KinesisClientLease newKCLLease(Shard shard) {
         KinesisClientLease newLease = new KinesisClientLease();
         newLease.setLeaseKey(shard.getShardId());
         List<String> parentShardIds = new ArrayList<String>(2);
@@ -763,13 +621,36 @@ class KinesisShardSyncer implements ShardSyncer {
         return newLease;
     }
 
+    /**
+     * Helper method to create a new KinesisClientLease POJO for a ChildShard.
+     * Note: Package level access only for testing purposes
+     *
+     * @param childShard
+     * @return
+     */
+    static KinesisClientLease newKCLLeaseForChildShard(ChildShard childShard) throws InvalidStateException {
+        final KinesisClientLease newLease = new KinesisClientLease();
+        newLease.setLeaseKey(childShard.getShardId());
+        final List<String> parentShardIds = new ArrayList<>();
+        if (!CollectionUtils.isNullOrEmpty(childShard.getParentShards())) {
+            parentShardIds.addAll(childShard.getParentShards());
+        } else {
+            throw new InvalidStateException("Unable to populate new lease for child shard " + childShard.getShardId()
+            + " because parent shards cannot be found.");
+        }
+        newLease.setParentShardIds(parentShardIds);
+        newLease.setOwnerSwitchesSinceCheckpoint(0L);
+        newLease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+        return newLease;
+    }
+
     /**
      * Helper method to construct a shardId->Shard map for the specified list of shards.
      *
      * @param shards List of shards
      * @return ShardId->Shard map
      */
-    Map<String, Shard> constructShardIdToShardMap(List<Shard> shards) {
+    static Map<String, Shard> constructShardIdToShardMap(List<Shard> shards) {
         Map<String, Shard> shardIdToShardMap = new HashMap<String, Shard>();
         for (Shard shard : shards) {
             shardIdToShardMap.put(shard.getShardId(), shard);
@@ -784,7 +665,7 @@ class KinesisShardSyncer implements ShardSyncer {
      * @param allShards All shards returved via DescribeStream. We assume this to represent a consistent shard list.
      * @return List of open shards (shards at the tip of the stream) - may include shards that are not yet active.
      */
-    List<Shard> getOpenShards(List<Shard> allShards) {
+    static List<Shard> getOpenShards(List<Shard> allShards) {
         List<Shard> openShards = new ArrayList<Shard>();
         for (Shard shard : allShards) {
             String endingSequenceNumber = shard.getSequenceNumberRange().getEndingSequenceNumber();
@@ -796,7 +677,7 @@ class KinesisShardSyncer implements ShardSyncer {
         return openShards;
     }
 
-    private ExtendedSequenceNumber convertToCheckpoint(InitialPositionInStreamExtended position) {
+    static ExtendedSequenceNumber convertToCheckpoint(InitialPositionInStreamExtended position) {
         ExtendedSequenceNumber checkpoint = null;
 
         if (position.getInitialPositionInStream().equals(InitialPositionInStream.TRIM_HORIZON)) {
@@ -813,7 +694,7 @@ class KinesisShardSyncer implements ShardSyncer {
     /** Helper class to compare leases based on starting sequence number of the corresponding shards.
      *
      */
-    private static class StartingSequenceNumberAndShardIdBasedComparator implements Comparator<KinesisClientLease>,
+    static class StartingSequenceNumberAndShardIdBasedComparator implements Comparator<KinesisClientLease>,
             Serializable {
 
         private static final long serialVersionUID = 1L;
@@ -862,4 +743,28 @@ class KinesisShardSyncer implements ShardSyncer {
 
     }
 
+    /**
+     * Helper class to pass around state between recursive traversals of shard hierarchy.
+     */
+    @NoArgsConstructor
+    static class MemoizationContext {
+        private Map<String, Boolean> isDescendantMap = new HashMap<>();
+        private Map<String, Boolean> shouldCreateLeaseMap = new HashMap<>();
+
+        Boolean isDescendant(String shardId) {
+            return isDescendantMap.get(shardId);
+        }
+
+        void setIsDescendant(String shardId, Boolean isDescendant) {
+            isDescendantMap.put(shardId, isDescendant);
+        }
+
+        Boolean shouldCreateLease(String shardId) {
+            return shouldCreateLeaseMap.computeIfAbsent(shardId, x -> Boolean.FALSE);
+        }
+
+        void setShouldCreateLease(String shardId, Boolean shouldCreateLease) {
+            shouldCreateLeaseMap.put(shardId, shouldCreateLease);
+        }
+    }
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/LeaseCleanupValidator.java

@@ -8,6 +8,7 @@ import java.util.Set;
 /**
  * Represents the class that decides if a lease is eligible for cleanup.
  */
+@Deprecated
 public interface LeaseCleanupValidator {
 
     /**

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/LeaseSynchronizer.java

@@ -0,0 +1,41 @@
+/*
+ * Copyright 2019 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
+import com.amazonaws.services.kinesis.model.Shard;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Interface used by {@link KinesisShardSyncer} to determine how to create new leases based on the current state
+ * of the lease table (i.e. whether the lease table is empty or non-empty).
+ */
+interface LeaseSynchronizer {
+
+    /**
+     * Determines how to create leases.
+     * @param shards
+     * @param currentLeases
+     * @param initialPosition
+     * @param inconsistentShardIds
+     * @return
+     */
+    List<KinesisClientLease> determineNewLeasesToCreate(List<Shard> shards,
+                                                        List<KinesisClientLease> currentLeases,
+                                                        InitialPositionInStreamExtended initialPosition,
+                                                        Set<String> inconsistentShardIds);
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/NonEmptyLeaseTableSynchronizer.java

@@ -0,0 +1,162 @@
+/*
+ * Copyright 2019 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
+import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
+import com.amazonaws.services.kinesis.leases.impl.Lease;
+import com.amazonaws.services.kinesis.model.Shard;
+import lombok.AllArgsConstructor;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * TODO - non-empty lease table sync story
+ */
+@AllArgsConstructor
+class NonEmptyLeaseTableSynchronizer implements LeaseSynchronizer {
+
+    private static final Log LOG = LogFactory.getLog(NonEmptyLeaseTableSynchronizer.class);
+
+    private final Map<String, Shard> shardIdToShardMap;
+    private final Map<String, Set<String>> shardIdToChildShardIdsMap;
+
+    /**
+     * Determine new leases to create and their initial checkpoint.
+     * Note: Package level access only for testing purposes.
+     *
+     * For each open (no ending sequence number) shard without open parents that doesn't already have a lease,
+     * determine if it is a descendant of any shard which is or will be processed (e.g. for which a lease exists):
+     * If so, create a lease for the first ancestor that needs to be processed (if needed). We will create leases
+     * for no more than one level in the ancestry tree. Once we find the first ancestor that needs to be processed,
+     * we will avoid creating leases for further descendants of that ancestor.
+     * If not, set checkpoint of the shard to the initial position specified by the client.
+     * To check if we need to create leases for ancestors, we use the following rules:
+     *   * If we began (or will begin) processing data for a shard, then we must reach end of that shard before
+     *         we begin processing data from any of its descendants.
+     *   * A shard does not start processing data until data from all its parents has been processed.
+     * Note, if the initial position is LATEST and a shard has two parents and only one is a descendant - we'll create
+     * leases corresponding to both the parents - the parent shard which is not a descendant will have
+     * its checkpoint set to Latest.
+     *
+     * We assume that if there is an existing lease for a shard, then either:
+     *   * we have previously created a lease for its parent (if it was needed), or
+     *   * the parent shard has expired.
+     *
+     * For example:
+     * Shard structure (each level depicts a stream segment):
+     * 0 1 2 3 4   5   - shards till epoch 102
+     * \ / \ / |   |
+     *  6   7  4   5   - shards from epoch 103 - 205
+     *  \  /   |  / \
+     *   8     4 9  10 - shards from epoch 206 (open - no ending sequenceNumber)
+     *
+     * Current leases: (4, 5, 7)
+     *
+     * If initial position is LATEST:
+     *   - New leases to create: (6)
+     * If initial position is TRIM_HORIZON:
+     *   - New leases to create: (0, 1)
+     * If initial position is AT_TIMESTAMP(epoch=200):
+     *   - New leases to create: (0, 1)
+     *
+     * The leases returned are sorted by the starting sequence number - following the same order
+     * when persisting the leases in DynamoDB will ensure that we recover gracefully if we fail
+     * before creating all the leases.
+     *
+     * If a shard has no existing lease, is open, and is a descendant of a parent which is still open, we ignore it
+     * here; this happens when the list of shards is inconsistent, which could be due to pagination delay for very
+     * high shard count streams (i.e., dynamodb streams for tables with thousands of partitions).  This can only
+     * currently happen here if ignoreUnexpectedChildShards was true in syncShardleases.
+     *
+     * @param shards List of all shards in Kinesis (we'll create new leases based on this set)
+     * @param currentLeases List of current leases
+     * @param initialPosition One of LATEST, TRIM_HORIZON, or AT_TIMESTAMP. We'll start fetching records from that
+     *        location in the shard (when an application starts up for the first time - and there are no checkpoints).
+     * @param inconsistentShardIds Set of child shard ids having open parents.
+     * @return List of new leases to create sorted by starting sequenceNumber of the corresponding shard
+     */
+    @Override
+    public List<KinesisClientLease> determineNewLeasesToCreate(List<Shard> shards,
+                                                               List<KinesisClientLease> currentLeases,
+                                                               InitialPositionInStreamExtended initialPosition,
+                                                               Set<String> inconsistentShardIds) {
+
+        Map<String, KinesisClientLease> shardIdToNewLeaseMap = new HashMap<>();
+        Map<String, Shard> shardIdToShardMapOfAllKinesisShards = KinesisShardSyncer.constructShardIdToShardMap(shards);
+
+        Set<String> shardIdsOfCurrentLeases = new HashSet<String>();
+        for (Lease lease : currentLeases) {
+            shardIdsOfCurrentLeases.add(lease.getLeaseKey());
+            LOG.debug("Existing lease: " + lease);
+        }
+
+        List<Shard> openShards = KinesisShardSyncer.getOpenShards(shards);
+        final KinesisShardSyncer.MemoizationContext memoizationContext = new KinesisShardSyncer.MemoizationContext();
+
+
+        // Iterate over the open shards and find those that don't have any lease entries.
+        for (Shard shard : openShards) {
+            String shardId = shard.getShardId();
+            LOG.debug("Evaluating leases for open shard " + shardId + " and its ancestors.");
+            if (shardIdsOfCurrentLeases.contains(shardId)) {
+                LOG.debug("Lease for shardId " + shardId + " already exists. Not creating a lease");
+            } else if (inconsistentShardIds.contains(shardId)) {
+                LOG.info("shardId " + shardId + " is an inconsistent child.  Not creating a lease");
+            } else {
+                LOG.debug("Beginning traversal of ancestry tree for shardId " + shardId);
+
+                // A shard is a descendant if at least one if its ancestors exists in the lease table.
+                // We will create leases for only one level in the ancestry tree. Once we find the first ancestor
+                // that needs to be processed in order to complete the hash range, we will not create leases for
+                // further descendants of that ancestor.
+                boolean isDescendant = KinesisShardSyncer.checkIfDescendantAndAddNewLeasesForAncestors(shardId,
+                        initialPosition, shardIdsOfCurrentLeases, shardIdToShardMapOfAllKinesisShards,
+                        shardIdToNewLeaseMap, memoizationContext);
+
+                // If shard is a descendant, the leases for its ancestors were already created above. Open shards
+                // that are NOT descendants will not have leases yet, so we create them here. We will not create
+                // leases for open shards that ARE descendants yet - leases for these shards will be created upon
+                // SHARD_END of their parents.
+                if (!isDescendant) {
+                    LOG.debug("ShardId " + shardId + " has no ancestors. Creating a lease.");
+                    final KinesisClientLease newLease = KinesisShardSyncer.newKCLLease(shard);
+                    newLease.setCheckpoint(KinesisShardSyncer.convertToCheckpoint(initialPosition));
+                    LOG.debug("Set checkpoint of " + newLease.getLeaseKey() + " to " + newLease.getCheckpoint());
+                    shardIdToNewLeaseMap.put(shardId, newLease);
+                } else {
+                    LOG.debug("ShardId " + shardId + " is a descendant whose ancestors should already have leases. " +
+                            "Not creating a lease.");
+                }
+            }
+        }
+
+        List<KinesisClientLease> newLeasesToCreate = new ArrayList<>();
+        newLeasesToCreate.addAll(shardIdToNewLeaseMap.values());
+        Comparator<? super KinesisClientLease> startingSequenceNumberComparator = new KinesisShardSyncer.StartingSequenceNumberAndShardIdBasedComparator(
+                shardIdToShardMapOfAllKinesisShards);
+        Collections.sort(newLeasesToCreate, startingSequenceNumberComparator);
+        return newLeasesToCreate;
+    }
+}

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ParentsFirstShardPrioritization.java

@@ -25,6 +25,7 @@ import java.util.Map;
  * It also limits number of shards that will be available for initialization based on their depth.
  * It doesn't make a lot of sense to work on a shard that has too many unfinished parents.
  */
+@Deprecated
 public class ParentsFirstShardPrioritization implements
         ShardPrioritization {
     private static final SortingNode PROCESSING_NODE = new SortingNode(null, Integer.MIN_VALUE);

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PeriodicShardSyncManager.java

@@ -14,39 +14,105 @@
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import java.io.Serializable;
+import java.math.BigInteger;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
 
+import com.amazonaws.services.cloudwatch.model.StandardUnit;
+import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
+import com.amazonaws.services.kinesis.leases.exceptions.DependencyException;
+import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
+import com.amazonaws.services.kinesis.leases.exceptions.ProvisionedThroughputException;
+import com.amazonaws.services.kinesis.leases.impl.HashKeyRangeForLease;
+import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
+import com.amazonaws.services.kinesis.leases.impl.UpdateField;
+import com.amazonaws.services.kinesis.leases.interfaces.ILeaseManager;
+import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
 import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+import com.amazonaws.services.kinesis.metrics.interfaces.MetricsLevel;
+import com.amazonaws.services.kinesis.model.Shard;
+import com.amazonaws.util.CollectionUtils;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ComparisonChain;
 import lombok.EqualsAndHashCode;
 import lombok.Getter;
+import lombok.NonNull;
+import lombok.Value;
+import lombok.experimental.Accessors;
 import org.apache.commons.lang3.Validate;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 
+import static com.amazonaws.services.kinesis.leases.impl.HashKeyRangeForLease.fromHashKeyRange;
+
 /**
- * The top level orchestrator for coordinating the periodic shard sync related
+ * The top level orchestrator for coordinating the periodic shard sync related activities. If the configured
- * activities.
+ * {@link ShardSyncStrategyType} is PERIODIC, this class will be the main shard sync orchestrator. For non-PERIODIC
+ * strategies, this class will serve as an internal auditor that periodically checks if the full hash range is covered
+ * by currently held leases, and initiates a recovery shard sync if not.
  */
 @Getter
 @EqualsAndHashCode
 class PeriodicShardSyncManager {
     private static final Log LOG = LogFactory.getLog(PeriodicShardSyncManager.class);
     private static final long INITIAL_DELAY = 0;
-    private static final long PERIODIC_SHARD_SYNC_INTERVAL_MILLIS = 1000;
+
+    /** DEFAULT interval is used for PERIODIC {@link ShardSyncStrategyType}. */
+    private static final long DEFAULT_PERIODIC_SHARD_SYNC_INTERVAL_MILLIS = 1000L;
+
+    /** Parameters for validating hash range completeness when running in auditor mode. */
+    @VisibleForTesting
+    static final BigInteger MIN_HASH_KEY = BigInteger.ZERO;
+    @VisibleForTesting
+    static final BigInteger MAX_HASH_KEY = new BigInteger("2").pow(128).subtract(BigInteger.ONE);
+    static final String PERIODIC_SHARD_SYNC_MANAGER = "PeriodicShardSyncManager";
+    private final HashRangeHoleTracker hashRangeHoleTracker = new HashRangeHoleTracker();
 
     private final String workerId;
     private final LeaderDecider leaderDecider;
     private final ITask metricsEmittingShardSyncTask;
     private final ScheduledExecutorService shardSyncThreadPool;
+    private final ILeaseManager<KinesisClientLease> leaseManager;
+    private final IKinesisProxy kinesisProxy;
+    private final boolean isAuditorMode;
+    private final long periodicShardSyncIntervalMillis;
     private boolean isRunning;
+    private final IMetricsFactory metricsFactory;
+    private final int leasesRecoveryAuditorInconsistencyConfidenceThreshold;
 
-    PeriodicShardSyncManager(String workerId, LeaderDecider leaderDecider, ShardSyncTask shardSyncTask, IMetricsFactory metricsFactory) {
+
-       this(workerId, leaderDecider, shardSyncTask, Executors.newSingleThreadScheduledExecutor(), metricsFactory);
+    PeriodicShardSyncManager(String workerId,
+                             LeaderDecider leaderDecider,
+                             ShardSyncTask shardSyncTask,
+                             IMetricsFactory metricsFactory,
+                             ILeaseManager<KinesisClientLease> leaseManager,
+                             IKinesisProxy kinesisProxy,
+                             boolean isAuditorMode,
+                             long leasesRecoveryAuditorExecutionFrequencyMillis,
+                             int leasesRecoveryAuditorInconsistencyConfidenceThreshold) {
+       this(workerId, leaderDecider, shardSyncTask, Executors.newSingleThreadScheduledExecutor(), metricsFactory,
+            leaseManager, kinesisProxy, isAuditorMode, leasesRecoveryAuditorExecutionFrequencyMillis,
+            leasesRecoveryAuditorInconsistencyConfidenceThreshold);
     }
 
-    PeriodicShardSyncManager(String workerId, LeaderDecider leaderDecider, ShardSyncTask shardSyncTask, ScheduledExecutorService shardSyncThreadPool, IMetricsFactory metricsFactory) {
+    PeriodicShardSyncManager(String workerId,
+                             LeaderDecider leaderDecider,
+                             ShardSyncTask shardSyncTask,
+                             ScheduledExecutorService shardSyncThreadPool,
+                             IMetricsFactory metricsFactory,
+                             ILeaseManager<KinesisClientLease> leaseManager,
+                             IKinesisProxy kinesisProxy,
+                             boolean isAuditorMode,
+                             long leasesRecoveryAuditorExecutionFrequencyMillis,
+                             int leasesRecoveryAuditorInconsistencyConfidenceThreshold) {
         Validate.notBlank(workerId, "WorkerID is required to initialize PeriodicShardSyncManager.");
         Validate.notNull(leaderDecider, "LeaderDecider is required to initialize PeriodicShardSyncManager.");
         Validate.notNull(shardSyncTask, "ShardSyncTask is required to initialize PeriodicShardSyncManager.");
@@ -54,18 +120,47 @@ class PeriodicShardSyncManager {
         this.leaderDecider = leaderDecider;
         this.metricsEmittingShardSyncTask = new MetricsCollectingTaskDecorator(shardSyncTask, metricsFactory);
         this.shardSyncThreadPool = shardSyncThreadPool;
+        this.leaseManager = leaseManager;
+        this.kinesisProxy = kinesisProxy;
+        this.metricsFactory = metricsFactory;
+        this.isAuditorMode = isAuditorMode;
+        this.leasesRecoveryAuditorInconsistencyConfidenceThreshold = leasesRecoveryAuditorInconsistencyConfidenceThreshold;
+        if (isAuditorMode) {
+            Validate.notNull(this.leaseManager, "LeaseManager is required for non-PERIODIC shard sync strategies.");
+            Validate.notNull(this.kinesisProxy, "KinesisProxy is required for non-PERIODIC shard sync strategies.");
+            this.periodicShardSyncIntervalMillis = leasesRecoveryAuditorExecutionFrequencyMillis;
+        } else {
+            this.periodicShardSyncIntervalMillis = DEFAULT_PERIODIC_SHARD_SYNC_INTERVAL_MILLIS;
+        }
     }
 
     public synchronized TaskResult start() {
         if (!isRunning) {
+            final Runnable periodicShardSyncer = () -> {
+                try {
+                    runShardSync();
+                } catch (Throwable t) {
+                    LOG.error("Error running shard sync.", t);
+                }
+            };
+
             shardSyncThreadPool
-                    .scheduleWithFixedDelay(this::runShardSync, INITIAL_DELAY, PERIODIC_SHARD_SYNC_INTERVAL_MILLIS,
+                    .scheduleWithFixedDelay(periodicShardSyncer, INITIAL_DELAY, periodicShardSyncIntervalMillis,
                             TimeUnit.MILLISECONDS);
             isRunning = true;
         }
         return new TaskResult(null);
     }
 
+    /**
+     * Runs ShardSync once, without scheduling further periodic ShardSyncs.
+     * @return TaskResult from shard sync
+     */
+    public synchronized TaskResult syncShardsOnce() {
+        LOG.info("Syncing shards once from worker " + workerId);
+        return metricsEmittingShardSyncTask.call();
+    }
+
     public void stop() {
         if (isRunning) {
             LOG.info(String.format("Shutting down leader decider on worker %s", workerId));
@@ -77,15 +172,239 @@ class PeriodicShardSyncManager {
     }
 
     private void runShardSync() {
-        try {
         if (leaderDecider.isLeader(workerId)) {
-                LOG.debug(String.format("WorkerId %s is a leader, running the shard sync task", workerId));
+            LOG.debug("WorkerId " + workerId + " is a leader, running the shard sync task");
+
+            MetricsHelper.startScope(metricsFactory, PERIODIC_SHARD_SYNC_MANAGER);
+            boolean isRunSuccess = false;
+            final long runStartMillis = System.currentTimeMillis();
+
+            try {
+                final ShardSyncResponse shardSyncResponse = checkForShardSync();
+                MetricsHelper.getMetricsScope().addData("NumStreamsToSync", shardSyncResponse.shouldDoShardSync() ? 1 : 0, StandardUnit.Count, MetricsLevel.SUMMARY);
+                MetricsHelper.getMetricsScope().addData("NumStreamsWithPartialLeases", shardSyncResponse.isHoleDetected() ? 1 : 0, StandardUnit.Count, MetricsLevel.SUMMARY);
+                if (shardSyncResponse.shouldDoShardSync()) {
+                    LOG.info("Periodic shard syncer initiating shard sync due to the reason - " +
+                            shardSyncResponse.reasonForDecision());
                     metricsEmittingShardSyncTask.call();
                 } else {
-                LOG.debug(String.format("WorkerId %s is not a leader, not running the shard sync task", workerId));
+                    LOG.info("Skipping shard sync due to the reason - " + shardSyncResponse.reasonForDecision());
                 }
-        } catch (Throwable t) {
+                isRunSuccess = true;
-            LOG.error("Error during runShardSync.", t);
+            } catch (Exception e) {
+                LOG.error("Caught exception while running periodic shard syncer.", e);
+            } finally {
+                MetricsHelper.addSuccessAndLatency(runStartMillis, isRunSuccess, MetricsLevel.SUMMARY);
+                MetricsHelper.endScope();
+            }
+        } else {
+            LOG.debug("WorkerId " + workerId + " is not a leader, not running the shard sync task");
+        }
+    }
+
+    @VisibleForTesting
+    ShardSyncResponse checkForShardSync() throws DependencyException, InvalidStateException,
+            ProvisionedThroughputException {
+
+        if (!isAuditorMode) {
+            // If we are running with PERIODIC shard sync strategy, we should sync every time.
+            return new ShardSyncResponse(true, false, "Syncing every time with PERIODIC shard sync strategy.");
+        }
+
+        // Get current leases from DynamoDB.
+        final List<KinesisClientLease> currentLeases = leaseManager.listLeases();
+
+        if (CollectionUtils.isNullOrEmpty(currentLeases)) {
+            // If the current leases are null or empty, then we need to initiate a shard sync.
+            LOG.info("No leases found. Will trigger a shard sync.");
+            return new ShardSyncResponse(true, false, "No leases found.");
+        }
+
+        // Check if there are any holes in the hash range covered by current leases. Return the first hole if present.
+        Optional<HashRangeHole> hashRangeHoleOpt = hasHoleInLeases(currentLeases);
+        if (hashRangeHoleOpt.isPresent()) {
+            // If hole is present, check if the hole is detected consecutively in previous occurrences. If hole is
+            // determined with high confidence, return true; return false otherwise. We use the high confidence factor
+            // to avoid shard sync on any holes during resharding and lease cleanups, or other intermittent issues.
+            final boolean hasHoleWithHighConfidence =
+                    hashRangeHoleTracker.hashHighConfidenceOfHoleWith(hashRangeHoleOpt.get());
+
+            return new ShardSyncResponse(hasHoleWithHighConfidence, true,
+                    "Detected the same hole for " + hashRangeHoleTracker.getNumConsecutiveHoles() + " times. " +
+                    "Will initiate shard sync after reaching threshold: " + leasesRecoveryAuditorInconsistencyConfidenceThreshold);
+        } else {
+            // If hole is not present, clear any previous hole tracking and return false.
+            hashRangeHoleTracker.reset();
+            return new ShardSyncResponse(false, false, "Hash range is complete.");
+        }
+    }
+
+    @VisibleForTesting
+    Optional<HashRangeHole> hasHoleInLeases(List<KinesisClientLease> leases) {
+        // Filter out any leases with checkpoints other than SHARD_END
+        final List<KinesisClientLease> activeLeases = leases.stream()
+                .filter(lease -> lease.getCheckpoint() != null && !lease.getCheckpoint().isShardEnd())
+                .collect(Collectors.toList());
+
+        final List<KinesisClientLease> activeLeasesWithHashRanges = fillWithHashRangesIfRequired(activeLeases);
+        return checkForHoleInHashKeyRanges(activeLeasesWithHashRanges);
+    }
+
+    private List<KinesisClientLease> fillWithHashRangesIfRequired(List<KinesisClientLease> activeLeases) {
+        final List<KinesisClientLease> activeLeasesWithNoHashRanges = activeLeases.stream()
+                .filter(lease -> lease.getHashKeyRange() == null).collect(Collectors.toList());
+
+        if (activeLeasesWithNoHashRanges.isEmpty()) {
+            return activeLeases;
+        }
+
+        // Fetch shards from Kinesis to fill in the in-memory hash ranges
+        final Map<String, Shard> kinesisShards = kinesisProxy.getShardList().stream()
+                .collect(Collectors.toMap(Shard::getShardId, shard -> shard));
+
+        return activeLeases.stream().map(lease -> {
+            if (lease.getHashKeyRange() == null) {
+                final String shardId = lease.getLeaseKey();
+                final Shard shard = kinesisShards.get(shardId);
+                if (shard == null) {
+                    return lease;
+                }
+                lease.setHashKeyRange(fromHashKeyRange(shard.getHashKeyRange()));
+
+                try {
+                    leaseManager.updateLeaseWithMetaInfo(lease, UpdateField.HASH_KEY_RANGE);
+                } catch (Exception e) {
+                    LOG.warn("Unable to update hash range information for lease " + lease.getLeaseKey() +
+                            ". This may result in explicit lease sync.");
+                }
+            }
+            return lease;
+        }).filter(lease -> lease.getHashKeyRange() != null).collect(Collectors.toList());
+    }
+
+    @VisibleForTesting
+    static Optional<HashRangeHole> checkForHoleInHashKeyRanges(List<KinesisClientLease> leasesWithHashKeyRanges) {
+        // Sort the hash ranges by starting hash key
+        final List<KinesisClientLease> sortedLeasesWithHashKeyRanges = sortLeasesByHashRange(leasesWithHashKeyRanges);
+        if (sortedLeasesWithHashKeyRanges.isEmpty()) {
+            LOG.error("No leases with valid hash ranges found.");
+            return Optional.of(new HashRangeHole());
+        }
+
+        // Validate the hash range bounds
+        final KinesisClientLease minHashKeyLease = sortedLeasesWithHashKeyRanges.get(0);
+        final KinesisClientLease maxHashKeyLease =
+                sortedLeasesWithHashKeyRanges.get(sortedLeasesWithHashKeyRanges.size() - 1);
+        if (!minHashKeyLease.getHashKeyRange().startingHashKey().equals(MIN_HASH_KEY) ||
+            !maxHashKeyLease.getHashKeyRange().endingHashKey().equals(MAX_HASH_KEY)) {
+            LOG.error("Incomplete hash range found between " + minHashKeyLease + " and " + maxHashKeyLease);
+            return Optional.of(new HashRangeHole(minHashKeyLease.getHashKeyRange(), maxHashKeyLease.getHashKeyRange()));
+        }
+
+        // Check for any holes in the sorted hash range intervals
+        if (sortedLeasesWithHashKeyRanges.size() > 1) {
+            KinesisClientLease leftmostLeaseToReportInCaseOfHole = minHashKeyLease;
+            HashKeyRangeForLease leftLeaseHashRange = leftmostLeaseToReportInCaseOfHole.getHashKeyRange();
+
+            for (int i = 1; i < sortedLeasesWithHashKeyRanges.size(); i++) {
+                final KinesisClientLease rightLease = sortedLeasesWithHashKeyRanges.get(i);
+                final HashKeyRangeForLease rightLeaseHashRange = rightLease.getHashKeyRange();
+                final BigInteger rangeDiff =
+                        rightLeaseHashRange.startingHashKey().subtract(leftLeaseHashRange.endingHashKey());
+                // We have overlapping leases when rangeDiff is 0 or negative.
+                // signum() will be -1 for negative and 0 if value is 0.
+                // Merge the ranges for further tracking.
+                if (rangeDiff.signum() <= 0) {
+                    leftLeaseHashRange = new HashKeyRangeForLease(leftLeaseHashRange.startingHashKey(),
+                            leftLeaseHashRange.endingHashKey().max(rightLeaseHashRange.endingHashKey()));
+                } else {
+                    // We have non-overlapping leases when rangeDiff is positive. signum() will be 1 in this case.
+                    // If rangeDiff is 1, then it is a continuous hash range. If not, there is a hole.
+                    if (!rangeDiff.equals(BigInteger.ONE)) {
+                        LOG.error("Incomplete hash range found between " + leftmostLeaseToReportInCaseOfHole +
+                                " and " + rightLease);
+                        return Optional.of(new HashRangeHole(leftmostLeaseToReportInCaseOfHole.getHashKeyRange(),
+                                rightLease.getHashKeyRange()));
+                    }
+
+                    leftmostLeaseToReportInCaseOfHole = rightLease;
+                    leftLeaseHashRange = rightLeaseHashRange;
+                }
+            }
+        }
+
+        return Optional.empty();
+    }
+
+    @VisibleForTesting
+    static List<KinesisClientLease> sortLeasesByHashRange(List<KinesisClientLease> leasesWithHashKeyRanges) {
+        if (leasesWithHashKeyRanges.size() == 0 || leasesWithHashKeyRanges.size() == 1) {
+            return leasesWithHashKeyRanges;
+        }
+        Collections.sort(leasesWithHashKeyRanges, new HashKeyRangeComparator());
+        return leasesWithHashKeyRanges;
+    }
+
+    @Value
+    @Accessors(fluent = true)
+    @VisibleForTesting
+    static class ShardSyncResponse {
+        private final boolean shouldDoShardSync;
+        private final boolean isHoleDetected;
+        private final String reasonForDecision;
+    }
+
+    @Value
+    private static class HashRangeHole {
+        private final HashKeyRangeForLease hashRangeAtStartOfPossibleHole;
+        private final HashKeyRangeForLease hashRangeAtEndOfPossibleHole;
+
+        HashRangeHole() {
+            hashRangeAtStartOfPossibleHole = hashRangeAtEndOfPossibleHole = null;
+        }
+
+        HashRangeHole(HashKeyRangeForLease hashRangeAtStartOfPossibleHole,
+                      HashKeyRangeForLease hashRangeAtEndOfPossibleHole) {
+            this.hashRangeAtStartOfPossibleHole = hashRangeAtStartOfPossibleHole;
+            this.hashRangeAtEndOfPossibleHole = hashRangeAtEndOfPossibleHole;
+        }
+    }
+
+    private class HashRangeHoleTracker {
+        private HashRangeHole hashRangeHole;
+        @Getter
+        private Integer numConsecutiveHoles;
+
+        public boolean hashHighConfidenceOfHoleWith(@NonNull HashRangeHole hashRangeHole) {
+            if (hashRangeHole.equals(this.hashRangeHole)) {
+                ++this.numConsecutiveHoles;
+            } else {
+                this.hashRangeHole = hashRangeHole;
+                this.numConsecutiveHoles = 1;
+            }
+
+            return numConsecutiveHoles >= leasesRecoveryAuditorInconsistencyConfidenceThreshold;
+        }
+
+        public void reset() {
+            this.hashRangeHole = null;
+            this.numConsecutiveHoles = 0;
+        }
+    }
+
+    private static class HashKeyRangeComparator implements Comparator<KinesisClientLease>, Serializable {
+        private static final long serialVersionUID = 1L;
+
+        @Override
+        public int compare(KinesisClientLease lease, KinesisClientLease otherLease) {
+            Validate.notNull(lease);
+            Validate.notNull(otherLease);
+            Validate.notNull(lease.getHashKeyRange());
+            Validate.notNull(otherLease.getHashKeyRange());
+            return ComparisonChain.start()
+                    .compare(lease.getHashKeyRange().startingHashKey(), otherLease.getHashKeyRange().startingHashKey())
+                    .compare(lease.getHashKeyRange().endingHashKey(), otherLease.getHashKeyRange().endingHashKey())
+                    .result();
         }
     }
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PrefetchGetRecordsCache.java

@@ -129,6 +129,7 @@ public class PrefetchGetRecordsCache implements GetRecordsCache {
         try {
             result = getRecordsResultQueue.take().withCacheExitTime(Instant.now());
             prefetchCounters.removed(result);
+            log.info("Shard " + shardId + ": Number of records remaining in queue is " + getRecordsResultQueue.size());
         } catch (InterruptedException e) {
             log.error("Interrupted while getting records from the cache", e);
         }
@@ -177,7 +178,6 @@ public class PrefetchGetRecordsCache implements GetRecordsCache {
                         
                         MetricsHelper.getMetricsScope().addData(EXPIRED_ITERATOR_METRIC, 1, StandardUnit.Count,
                                 MetricsLevel.SUMMARY);
-                        
                         dataFetcher.restartIterator();
                     } catch (SdkClientException e) {
                         log.error("Exception thrown while fetching records from Kinesis", e);

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ProcessTask.java

@@ -152,8 +152,8 @@ class ProcessTask implements ITask {
 
         try {
             if (dataFetcher.isShardEndReached()) {
-                LOG.info("Reached end of shard " + shardInfo.getShardId());
+                LOG.info("Reached end of shard " + shardInfo.getShardId() + ". Found childShards: " + dataFetcher.getChildShards());
-                return new TaskResult(null, true);
+                return new TaskResult(null, true, dataFetcher.getChildShards());
             }
 
             final ProcessRecordsInput processRecordsInput = getRecordsResult();

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardConsumer.java

@@ -15,11 +15,16 @@
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
 
+import java.util.List;
 import java.util.Optional;
 import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 import java.util.concurrent.RejectedExecutionException;
 
+import com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager;
+import com.amazonaws.services.kinesis.model.ChildShard;
+import com.amazonaws.util.CollectionUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 
@@ -52,6 +57,7 @@ class ShardConsumer {
     private final IMetricsFactory metricsFactory;
     private final KinesisClientLibLeaseCoordinator leaseCoordinator;
     private ICheckpoint checkpoint;
+    private LeaseCleanupManager leaseCleanupManager;
     // Backoff time when polling to check if application has finished processing parent shards
     private final long parentShardPollIntervalMillis;
     private final boolean cleanupLeasesOfCompletedShards;
@@ -66,6 +72,9 @@ class ShardConsumer {
     private Future<TaskResult> future;
     private ShardSyncStrategy shardSyncStrategy;
 
+    @Getter
+    private List<ChildShard> childShards;
+
     @Getter
     private final GetRecordsCache getRecordsCache;
 
@@ -106,6 +115,7 @@ class ShardConsumer {
      * @param shardSyncer shardSyncer instance used to check and create new leases
      */
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 10 LINES
+    @Deprecated
     ShardConsumer(ShardInfo shardInfo,
             StreamConfig streamConfig,
             ICheckpoint checkpoint,
@@ -118,6 +128,7 @@ class ShardConsumer {
             long backoffTimeMillis,
             boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
             KinesisClientLibConfiguration config, ShardSyncer shardSyncer, ShardSyncStrategy shardSyncStrategy) {
+
         this(shardInfo,
                 streamConfig,
                 checkpoint,
@@ -150,6 +161,7 @@ class ShardConsumer {
      * @param shardSyncer shardSyncer instance used to check and create new leases
      */
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 10 LINES
+    @Deprecated
     ShardConsumer(ShardInfo shardInfo,
             StreamConfig streamConfig,
             ICheckpoint checkpoint,
@@ -210,6 +222,7 @@ class ShardConsumer {
      * @param config Kinesis library configuration
      * @param shardSyncer shardSyncer instance used to check and create new leases
      */
+    @Deprecated
     ShardConsumer(ShardInfo shardInfo,
             StreamConfig streamConfig,
             ICheckpoint checkpoint,
@@ -226,6 +239,53 @@ class ShardConsumer {
             Optional<Integer> retryGetRecordsInSeconds,
             Optional<Integer> maxGetRecordsThreadPool,
             KinesisClientLibConfiguration config, ShardSyncer shardSyncer, ShardSyncStrategy shardSyncStrategy) {
+
+        this(shardInfo, streamConfig, checkpoint, recordProcessor, recordProcessorCheckpointer, leaseCoordinator,
+        parentShardPollIntervalMillis, cleanupLeasesOfCompletedShards, executorService, metricsFactory,
+        backoffTimeMillis, skipShardSyncAtWorkerInitializationIfLeasesExist, kinesisDataFetcher, retryGetRecordsInSeconds,
+        maxGetRecordsThreadPool, config, shardSyncer, shardSyncStrategy, LeaseCleanupManager.createOrGetInstance(streamConfig.getStreamProxy(), leaseCoordinator.getLeaseManager(),
+                Executors.newSingleThreadScheduledExecutor(), metricsFactory, config.shouldCleanupLeasesUponShardCompletion(),
+                config.leaseCleanupIntervalMillis(), config.completedLeaseCleanupThresholdMillis(),
+                config.garbageLeaseCleanupThresholdMillis(), config.getMaxRecords()));
+    }
+
+    /**
+     * @param shardInfo Shard information
+     * @param streamConfig Stream Config to use
+     * @param checkpoint Checkpoint tracker
+     * @param recordProcessor Record processor used to process the data records for the shard
+     * @param recordProcessorCheckpointer RecordProcessorCheckpointer to use to checkpoint progress
+     * @param leaseCoordinator Used to manage leases for current worker
+     * @param parentShardPollIntervalMillis Wait for this long if parent shards are not done (or we get an exception)
+     * @param cleanupLeasesOfCompletedShards  clean up the leases of completed shards
+     * @param executorService ExecutorService used to execute process tasks for this shard
+     * @param metricsFactory IMetricsFactory used to construct IMetricsScopes for this shard
+     * @param backoffTimeMillis backoff interval when we encounter exceptions
+     * @param skipShardSyncAtWorkerInitializationIfLeasesExist Skip sync at init if lease exists
+     * @param kinesisDataFetcher KinesisDataFetcher to fetch data from Kinesis streams.
+     * @param retryGetRecordsInSeconds time in seconds to wait before the worker retries to get a record
+     * @param maxGetRecordsThreadPool max number of threads in the getRecords thread pool
+     * @param config Kinesis library configuration
+     * @param shardSyncer shardSyncer instance used to check and create new leases
+     * @param leaseCleanupManager used to clean up leases in lease table.
+     */
+    ShardConsumer(ShardInfo shardInfo,
+                  StreamConfig streamConfig,
+                  ICheckpoint checkpoint,
+                  IRecordProcessor recordProcessor,
+                  RecordProcessorCheckpointer recordProcessorCheckpointer,
+                  KinesisClientLibLeaseCoordinator leaseCoordinator,
+                  long parentShardPollIntervalMillis,
+                  boolean cleanupLeasesOfCompletedShards,
+                  ExecutorService executorService,
+                  IMetricsFactory metricsFactory,
+                  long backoffTimeMillis,
+                  boolean skipShardSyncAtWorkerInitializationIfLeasesExist,
+                  KinesisDataFetcher kinesisDataFetcher,
+                  Optional<Integer> retryGetRecordsInSeconds,
+                  Optional<Integer> maxGetRecordsThreadPool,
+                  KinesisClientLibConfiguration config, ShardSyncer shardSyncer, ShardSyncStrategy shardSyncStrategy,
+                  LeaseCleanupManager leaseCleanupManager) {
         this.shardInfo = shardInfo;
         this.streamConfig = streamConfig;
         this.checkpoint = checkpoint;
@@ -245,6 +305,7 @@ class ShardConsumer {
                 this.getShardInfo().getShardId(), this.metricsFactory, this.config.getMaxRecords());
         this.shardSyncer = shardSyncer;
         this.shardSyncStrategy = shardSyncStrategy;
+        this.leaseCleanupManager = leaseCleanupManager;
     }
 
     /**
@@ -321,6 +382,10 @@ class ShardConsumer {
             TaskResult result = future.get();
             if (result.getException() == null) {
                 if (result.isShardEndReached()) {
+                    if (!CollectionUtils.isNullOrEmpty(result.getChildShards())) {
+                        childShards = result.getChildShards();
+                        LOG.info("Shard " + shardInfo.getShardId() + ": Setting childShards in ShardConsumer: " + childShards);
+                    }
                     return TaskOutcome.END_OF_SHARD;
                 }
                 return TaskOutcome.SUCCESSFUL;
@@ -420,6 +485,7 @@ class ShardConsumer {
     void updateState(TaskOutcome taskOutcome) {
         if (taskOutcome == TaskOutcome.END_OF_SHARD) {
             markForShutdown(ShutdownReason.TERMINATE);
+            LOG.info("Shard " + shardInfo.getShardId() + ": Mark for shutdown with reason TERMINATE");
         }
         if (isShutdownRequested() && taskOutcome != TaskOutcome.FAILURE) {
             currentState = currentState.shutdownTransition(shutdownReason);
@@ -518,4 +584,8 @@ class ShardConsumer {
     ShardSyncStrategy getShardSyncStrategy() {
         return shardSyncStrategy;
     }
+
+    LeaseCleanupManager getLeaseCleanupManager() {
+        return leaseCleanupManager;
+    }
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardEndShardSyncStrategy.java

@@ -16,8 +16,13 @@ class ShardEndShardSyncStrategy implements ShardSyncStrategy {
     private static final Log LOG = LogFactory.getLog(Worker.class);
     private ShardSyncTaskManager shardSyncTaskManager;
 
-    ShardEndShardSyncStrategy(ShardSyncTaskManager shardSyncTaskManager) {
+    /** Runs periodic shard sync jobs in the background as an auditor process for shard-end syncs. */
+    private PeriodicShardSyncManager periodicShardSyncManager;
+
+    ShardEndShardSyncStrategy(ShardSyncTaskManager shardSyncTaskManager,
+                              PeriodicShardSyncManager periodicShardSyncManager) {
         this.shardSyncTaskManager = shardSyncTaskManager;
+        this.periodicShardSyncManager = periodicShardSyncManager;
     }
 
     @Override
@@ -42,8 +47,8 @@ class ShardEndShardSyncStrategy implements ShardSyncStrategy {
 
     @Override
     public TaskResult onWorkerInitialization() {
-        LOG.debug(String.format("onWorkerInitialization is NoOp for ShardSyncStrategyType %s", getStrategyType().toString()));
+        LOG.info("Starting periodic shard sync background process for SHARD_END shard sync strategy.");
-        return new TaskResult(null);
+        return periodicShardSyncManager.start();
     }
 
     @Override
@@ -65,6 +70,7 @@ class ShardEndShardSyncStrategy implements ShardSyncStrategy {
 
     @Override
     public void onWorkerShutDown() {
-        LOG.debug(String.format("Stop is NoOp for ShardSyncStrategyType %s", getStrategyType().toString()));
+        LOG.info("Stopping periodic shard sync background process for SHARD_END shard sync strategy.");
+        periodicShardSyncManager.stop();
     }
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardInfo.java

@@ -86,7 +86,7 @@ public class ShardInfo {
      * 
      * @return a list of shardId's that are parents of this shard, or empty if the shard has no parents.
      */
-    protected List<String> getParentShardIds() {
+    public List<String> getParentShardIds() {
         return new LinkedList<String>(parentShardIds);
     }
 

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShutdownTask.java

@@ -14,9 +14,16 @@
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
-import com.amazonaws.services.kinesis.clientlibrary.proxies.ShardClosureVerificationResponse;
+import com.amazonaws.services.kinesis.leases.LeasePendingDeletion;
-import com.amazonaws.services.kinesis.clientlibrary.proxies.ShardListWrappingShardClosureVerificationResponse;
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.internal.BlockedOnParentShardException;
-import com.amazonaws.services.kinesis.model.Shard;
+import com.amazonaws.services.kinesis.leases.exceptions.CustomerApplicationException;
+import com.amazonaws.services.kinesis.leases.exceptions.DependencyException;
+import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
+import com.amazonaws.services.kinesis.leases.exceptions.ProvisionedThroughputException;
+import com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager;
+import com.amazonaws.services.kinesis.leases.impl.UpdateField;
+import com.amazonaws.services.kinesis.model.ChildShard;
+import com.amazonaws.util.CollectionUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 
@@ -25,11 +32,14 @@ import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
 import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
 import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownInput;
 import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
-import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
-import com.amazonaws.services.kinesis.metrics.interfaces.MetricsLevel;
 import com.google.common.annotations.VisibleForTesting;
 
 import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 /**
  * Task for invoking the RecordProcessor shutdown() callback.
@@ -38,7 +48,8 @@ class ShutdownTask implements ITask {
 
     private static final Log LOG = LogFactory.getLog(ShutdownTask.class);
 
-    private static final String RECORD_PROCESSOR_SHUTDOWN_METRIC = "RecordProcessor.shutdown";
+    @VisibleForTesting
+    static final int RETRY_RANDOM_MAX_RANGE = 50;
 
     private final ShardInfo shardInfo;
     private final IRecordProcessor recordProcessor;
@@ -54,6 +65,8 @@ class ShutdownTask implements ITask {
     private final GetRecordsCache getRecordsCache;
     private final ShardSyncer shardSyncer;
     private final ShardSyncStrategy shardSyncStrategy;
+    private final List<ChildShard> childShards;
+    private final LeaseCleanupManager leaseCleanupManager;
 
     /**
      * Constructor.
@@ -69,7 +82,9 @@ class ShutdownTask implements ITask {
             boolean ignoreUnexpectedChildShards,
             KinesisClientLibLeaseCoordinator leaseCoordinator,
             long backoffTimeMillis,
-            GetRecordsCache getRecordsCache, ShardSyncer shardSyncer, ShardSyncStrategy shardSyncStrategy) {
+            GetRecordsCache getRecordsCache, ShardSyncer shardSyncer,
+            ShardSyncStrategy shardSyncStrategy, List<ChildShard> childShards,
+            LeaseCleanupManager leaseCleanupManager) {
         this.shardInfo = shardInfo;
         this.recordProcessor = recordProcessor;
         this.recordProcessorCheckpointer = recordProcessorCheckpointer;
@@ -83,6 +98,8 @@ class ShutdownTask implements ITask {
         this.getRecordsCache = getRecordsCache;
         this.shardSyncer = shardSyncer;
         this.shardSyncStrategy = shardSyncStrategy;
+        this.childShards = childShards;
+        this.leaseCleanupManager = leaseCleanupManager;
     }
 
     /*
@@ -94,87 +111,40 @@ class ShutdownTask implements ITask {
     @Override
     public TaskResult call() {
         Exception exception;
-        boolean applicationException = false;
+
+        LOG.info("Invoking shutdown() for shard " + shardInfo.getShardId() + ", concurrencyToken: "
+                         + shardInfo.getConcurrencyToken() + ", original Shutdown reason: " + reason + ". childShards:" + childShards);
 
         try {
-            ShutdownReason localReason = reason;
+            final KinesisClientLease currentShardLease = leaseCoordinator.getCurrentlyHeldLease(shardInfo.getShardId());
-            List<Shard> latestShards = null;
+            final Runnable leaseLostAction = () -> takeLeaseLostAction();
-            /*
-             * Revalidate if the current shard is closed before shutting down the shard consumer with reason SHARD_END
-             * If current shard is not closed, shut down the shard consumer with reason LEASE_LOST that allows active
-             * workers to contend for the lease of this shard.
-             */
-            if(localReason == ShutdownReason.TERMINATE) {
-                ShardClosureVerificationResponse shardClosureVerificationResponse = kinesisProxy.verifyShardClosure(shardInfo.getShardId());
-                if (shardClosureVerificationResponse instanceof ShardListWrappingShardClosureVerificationResponse) {
-                    latestShards = ((ShardListWrappingShardClosureVerificationResponse)shardClosureVerificationResponse).getLatestShards();
-                }
 
-                // If shard in context is not closed yet we should shut down the ShardConsumer with Zombie state
+            if (reason == ShutdownReason.TERMINATE) {
-                // which avoids checkpoint-ing with SHARD_END sequence number.
-                if(!shardClosureVerificationResponse.isShardClosed()) {
-                    localReason = ShutdownReason.ZOMBIE;
-                    dropLease();
-                    LOG.info("Forcing the lease to be lost before shutting down the consumer for Shard: " + shardInfo.getShardId());
-                }
-            }
-
-
-            // If we reached end of the shard, set sequence number to SHARD_END.
-            if (localReason == ShutdownReason.TERMINATE) {
-                recordProcessorCheckpointer.setSequenceNumberAtShardEnd(
-                        recordProcessorCheckpointer.getLargestPermittedCheckpointValue());
-                recordProcessorCheckpointer.setLargestPermittedCheckpointValue(ExtendedSequenceNumber.SHARD_END);
-            }
-
-            LOG.debug("Invoking shutdown() for shard " + shardInfo.getShardId() + ", concurrencyToken "
-                    + shardInfo.getConcurrencyToken() + ". Shutdown reason: " + localReason);
-            final ShutdownInput shutdownInput = new ShutdownInput()
-                    .withShutdownReason(localReason)
-                    .withCheckpointer(recordProcessorCheckpointer);
-            final long recordProcessorStartTimeMillis = System.currentTimeMillis();
                 try {
-                recordProcessor.shutdown(shutdownInput);
+                    takeShardEndAction(currentShardLease);
-                ExtendedSequenceNumber lastCheckpointValue = recordProcessorCheckpointer.getLastCheckpointValue();
+                } catch (InvalidStateException e) {
+                    // If InvalidStateException happens, it indicates we have a non recoverable error in short term.
+                    // In this scenario, we should shutdown the shardConsumer with ZOMBIE reason to allow other worker to take the lease and retry shutting down.
+                    LOG.warn("Lease " + shardInfo.getShardId() + ": Invalid state encountered while shutting down shardConsumer with TERMINATE reason. " +
+                                     "Dropping the lease and shutting down shardConsumer using ZOMBIE reason. ", e);
+                    dropLease(currentShardLease);
+                    throwOnApplicationException(leaseLostAction);
+                }
+            } else {
+                throwOnApplicationException(leaseLostAction);
+            }
 
-                if (localReason == ShutdownReason.TERMINATE) {
-                    if ((lastCheckpointValue == null)
-                            || (!lastCheckpointValue.equals(ExtendedSequenceNumber.SHARD_END))) {
-                        throw new IllegalArgumentException("Application didn't checkpoint at end of shard "
-                                + shardInfo.getShardId() + ". Application must checkpoint upon shutdown. " +
-                                "See IRecordProcessor.shutdown javadocs for more information.");
-                    }
-                }
             LOG.debug("Shutting down retrieval strategy.");
             getRecordsCache.shutdown();
             LOG.debug("Record processor completed shutdown() for shard " + shardInfo.getShardId());
-            } catch (Exception e) {
-                applicationException = true;
-                throw e;
-            } finally {
-                MetricsHelper.addLatency(RECORD_PROCESSOR_SHUTDOWN_METRIC, recordProcessorStartTimeMillis,
-                        MetricsLevel.SUMMARY);
-            }
-
-            if (localReason == ShutdownReason.TERMINATE) {
-                LOG.debug("Looking for child shards of shard " + shardInfo.getShardId());
-                // create leases for the child shards
-                TaskResult result = shardSyncStrategy.onShardConsumerShutDown(latestShards);
-                if (result.getException() != null) {
-                    LOG.debug("Exception while trying to sync shards on the shutdown of shard: " + shardInfo
-                            .getShardId());
-                    throw result.getException();
-                }
-                LOG.debug("Finished checking for child shards of shard " + shardInfo.getShardId());
-            }
-
             return new TaskResult(null);
         } catch (Exception e) {
-            if (applicationException) {
+            if (e instanceof CustomerApplicationException) {
-                LOG.error("Application exception. ", e);
+                LOG.error("Shard " + shardInfo.getShardId() + ": Application exception: ", e);
             } else {
-                LOG.error("Caught exception: ", e);
+                LOG.error("Shard " + shardInfo.getShardId() + ": Caught exception: ", e);
             }
+
             exception = e;
             // backoff if we encounter an exception.
             try {
@@ -187,6 +157,143 @@ class ShutdownTask implements ITask {
         return new TaskResult(exception);
     }
 
+    // Involves persisting child shard info, attempt to checkpoint and enqueueing lease for cleanup.
+    private void takeShardEndAction(KinesisClientLease currentShardLease)
+        throws InvalidStateException, DependencyException, ProvisionedThroughputException, CustomerApplicationException {
+        // Create new lease for the child shards if they don't exist.
+        // We have one valid scenario that shutdown task got created with SHARD_END reason and an empty list of childShards.
+        // This would happen when KinesisDataFetcher catches ResourceNotFound exception.
+        // In this case, KinesisDataFetcher will send out SHARD_END signal to trigger a shutdown task with empty list of childShards.
+        // This scenario could happen when customer deletes the stream while leaving the KCL application running.
+        if (currentShardLease == null) {
+            throw new InvalidStateException("Shard " + shardInfo.getShardId() + ": Lease not owned by the current worker. Leaving ShardEnd handling to new owner.");
+        }
+        if (!CollectionUtils.isNullOrEmpty(childShards)) {
+            // If childShards is not empty, create new leases for the childShards and update the current lease with the childShards lease information.
+            createLeasesForChildShardsIfNotExist();
+            updateCurrentLeaseWithChildShards(currentShardLease);
+        } else {
+            LOG.warn("Shard " + shardInfo.getShardId()
+                             + ": Shutting down consumer with SHARD_END reason without creating leases for child shards.");
+        }
+        // Checkpoint with SHARD_END sequence number.
+        final LeasePendingDeletion leasePendingDeletion = new LeasePendingDeletion(currentShardLease, shardInfo);
+        if (!leaseCleanupManager.isEnqueuedForDeletion(leasePendingDeletion)) {
+            boolean isSuccess = false;
+            try {
+                isSuccess = attemptShardEndCheckpointing();
+            } finally {
+                // Check if either the shard end ddb persist is successful or
+                // if childshards is empty. When child shards is empty then either it is due to
+                // completed shard being reprocessed or we got RNF from service.
+                // For these cases enqueue the lease for deletion.
+                if (isSuccess || CollectionUtils.isNullOrEmpty(childShards)) {
+                    leaseCleanupManager.enqueueForDeletion(leasePendingDeletion);
+                }
+            }
+        }
+    }
+
+    private void takeLeaseLostAction() {
+        final ShutdownInput leaseLostShutdownInput = new ShutdownInput()
+                .withShutdownReason(ShutdownReason.ZOMBIE)
+                .withCheckpointer(recordProcessorCheckpointer);
+        recordProcessor.shutdown(leaseLostShutdownInput);
+    }
+
+    private boolean attemptShardEndCheckpointing()
+        throws DependencyException, ProvisionedThroughputException, InvalidStateException, CustomerApplicationException {
+        final KinesisClientLease leaseFromDdb = Optional.ofNullable(leaseCoordinator.getLeaseManager().getLease(shardInfo.getShardId()))
+                .orElseThrow(() -> new InvalidStateException("Lease for shard " + shardInfo.getShardId() + " does not exist."));
+        if (!leaseFromDdb.getCheckpoint().equals(ExtendedSequenceNumber.SHARD_END)) {
+            // Call the recordProcessor to checkpoint with SHARD_END sequence number.
+            // The recordProcessor.shutdown is implemented by customer. We should validate if the SHARD_END checkpointing is successful after calling recordProcessor.shutdown.
+            throwOnApplicationException(() -> applicationCheckpointAndVerification());
+        }
+        return true;
+    }
+
+    private void applicationCheckpointAndVerification() {
+        recordProcessorCheckpointer.setSequenceNumberAtShardEnd(
+                recordProcessorCheckpointer.getLargestPermittedCheckpointValue());
+        recordProcessorCheckpointer.setLargestPermittedCheckpointValue(ExtendedSequenceNumber.SHARD_END);
+        final ShutdownInput shardEndShutdownInput = new ShutdownInput()
+                .withShutdownReason(ShutdownReason.TERMINATE)
+                .withCheckpointer(recordProcessorCheckpointer);
+        recordProcessor.shutdown(shardEndShutdownInput);
+
+        final ExtendedSequenceNumber lastCheckpointValue = recordProcessorCheckpointer.getLastCheckpointValue();
+
+        final boolean successfullyCheckpointedShardEnd = lastCheckpointValue.equals(ExtendedSequenceNumber.SHARD_END);
+
+        if ((lastCheckpointValue == null) || (!successfullyCheckpointedShardEnd)) {
+            throw new IllegalArgumentException("Application didn't checkpoint at end of shard "
+                                                       + shardInfo.getShardId() + ". Application must checkpoint upon shutdown. " +
+                                                       "See IRecordProcessor.shutdown javadocs for more information.");
+        }
+    }
+
+    private void throwOnApplicationException(Runnable action) throws CustomerApplicationException {
+        try {
+            action.run();
+        } catch (Exception e) {
+            throw new CustomerApplicationException("Customer application throws exception for shard " + shardInfo.getShardId(), e);
+        }
+    }
+
+    private void createLeasesForChildShardsIfNotExist() throws InvalidStateException, DependencyException, ProvisionedThroughputException {
+        // For child shard resulted from merge of two parent shards, verify if both the parents are either present or
+        // not present in the lease table before creating the lease entry.
+        if (!CollectionUtils.isNullOrEmpty(childShards) && childShards.size() == 1) {
+            final ChildShard childShard = childShards.get(0);
+            final List<String> parentLeaseKeys = childShard.getParentShards();
+
+            if (parentLeaseKeys.size() != 2) {
+                throw new InvalidStateException("Shard " + shardInfo.getShardId()+ "'s only child shard " + childShard
+                                                        + " does not contain other parent information.");
+            } else {
+                boolean isValidLeaseTableState = Objects.isNull(leaseCoordinator.getLeaseManager().getLease(parentLeaseKeys.get(0))) ==
+                                                 Objects.isNull(leaseCoordinator.getLeaseManager().getLease(parentLeaseKeys.get(1)));
+                if (!isValidLeaseTableState) {
+                    if(!isOneInNProbability(RETRY_RANDOM_MAX_RANGE)) {
+                        throw new BlockedOnParentShardException(
+                                "Shard " + shardInfo.getShardId() + "'s only child shard " + childShard
+                                        + " has partial parent information in lease table. Hence deferring lease creation of child shard.");
+                    } else {
+                        throw new InvalidStateException("Shard " + shardInfo.getShardId() + "'s only child shard " + childShard
+                                                                + " has partial parent information in lease table.");
+                    }
+                }
+            }
+        }
+        // Attempt create leases for child shards.
+        for (ChildShard childShard : childShards) {
+            final String leaseKey = childShard.getShardId();
+            if (leaseCoordinator.getLeaseManager().getLease(leaseKey) == null) {
+                final KinesisClientLease leaseToCreate = KinesisShardSyncer.newKCLLeaseForChildShard(childShard);
+                leaseCoordinator.getLeaseManager().createLeaseIfNotExists(leaseToCreate);
+                LOG.info("Shard " + shardInfo.getShardId() + " : Created child shard lease: " + leaseToCreate.getLeaseKey());
+            }
+        }
+    }
+
+    /**
+     * Returns true for 1 in N probability.
+     */
+    @VisibleForTesting
+    boolean isOneInNProbability(int n) {
+        Random r = new Random();
+        return 1 == r.nextInt((n - 1) + 1) + 1;
+    }
+
+    private void updateCurrentLeaseWithChildShards(KinesisClientLease currentLease) throws DependencyException, InvalidStateException, ProvisionedThroughputException {
+        final Set<String> childShardIds = childShards.stream().map(ChildShard::getShardId).collect(Collectors.toSet());
+        currentLease.setChildShardIds(childShardIds);
+        leaseCoordinator.getLeaseManager().updateLeaseWithMetaInfo(currentLease, UpdateField.CHILD_SHARDS);
+        LOG.info("Shard " + shardInfo.getShardId() + ": Updated current lease with child shard information: " + currentLease.getLeaseKey());
+    }
+
+
     /*
      * (non-Javadoc)
      * 
@@ -202,9 +309,12 @@ class ShutdownTask implements ITask {
         return reason;
     }
 
-    private void dropLease() {
+    private void dropLease(KinesisClientLease currentShardLease) {
-        KinesisClientLease lease = leaseCoordinator.getCurrentlyHeldLease(shardInfo.getShardId());
+        if (currentShardLease == null) {
-        leaseCoordinator.dropLease(lease);
+            LOG.warn("Shard " + shardInfo.getShardId() + ": Unable to find the lease for shard. Will shutdown the shardConsumer directly.");
-        LOG.warn("Dropped lease for shutting down ShardConsumer: " + lease.getLeaseKey());
+            return;
+        }
+        leaseCoordinator.dropLease(currentShardLease);
+        LOG.warn("Dropped lease for shutting down ShardConsumer: " + currentShardLease.getLeaseKey());
     }
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/TaskResult.java

@@ -14,6 +14,10 @@
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import com.amazonaws.services.kinesis.model.ChildShard;
+
+import java.util.List;
+
 /**
  * Used to capture information from a task that we want to communicate back to the higher layer.
  * E.g. exception thrown when executing the task, if we reach end of a shard.
@@ -26,6 +30,9 @@ class TaskResult {
     // Any exception caught while executing the task.
     private Exception exception;
 
+    // List of childShards of the current shard. This field is only required for the task result when we reach end of a shard.
+    private List<ChildShard> childShards;
+
     /**
      * @return the shardEndReached
      */
@@ -33,6 +40,11 @@ class TaskResult {
         return shardEndReached;
     }
 
+    /**
+     * @return the list of childShards.
+     */
+    protected List<ChildShard> getChildShards() { return childShards; }
+
     /**
      * @param shardEndReached the shardEndReached to set
      */
@@ -40,6 +52,11 @@ class TaskResult {
         this.shardEndReached = shardEndReached;
     }
 
+    /**
+     * @param childShards the list of childShards to set
+     */
+    protected void setChildShards(List<ChildShard> childShards) { this.childShards = childShards; }
+
     /**
      * @return the exception
      */
@@ -70,4 +87,10 @@ class TaskResult {
         this.shardEndReached = isShardEndReached;
     }
 
+    TaskResult(Exception e, boolean isShardEndReached, List<ChildShard> childShards) {
+        this.exception = e;
+        this.shardEndReached = isShardEndReached;
+        this.childShards = childShards;
+    }
+
 }

src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java

@@ -29,12 +29,17 @@ import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 import java.util.concurrent.SynchronousQueue;
 import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.ThreadLocalRandom;
 import java.util.concurrent.ThreadPoolExecutor;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.function.Consumer;
 
+import com.amazonaws.services.kinesis.leases.exceptions.DependencyException;
+import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
+import com.amazonaws.services.kinesis.leases.exceptions.ProvisionedThroughputException;
 import com.amazonaws.services.kinesis.leases.impl.GenericLeaseSelector;
+import com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager;
 import com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator;
 import com.amazonaws.services.kinesis.leases.impl.LeaseRenewer;
 import com.amazonaws.services.kinesis.leases.impl.LeaseTaker;
@@ -88,9 +93,13 @@ public class Worker implements Runnable {
 
     private static final Log LOG = LogFactory.getLog(Worker.class);
 
+    // Default configs for periodic shard sync
     private static final int SHARD_SYNC_SLEEP_FOR_PERIODIC_SHARD_SYNC = 0;
-    private static final int MAX_INITIALIZATION_ATTEMPTS = 20;
     private static final int PERIODIC_SHARD_SYNC_MAX_WORKERS_DEFAULT = 1; //Default for KCL.
+    static final long LEASE_TABLE_CHECK_FREQUENCY_MILLIS = 3 * 1000L;
+    static final long MIN_WAIT_TIME_FOR_LEASE_TABLE_CHECK_MILLIS = 1 * 1000L;
+    static final long MAX_WAIT_TIME_FOR_LEASE_TABLE_CHECK_MILLIS = 30 * 1000L;
+
     private static final WorkerStateChangeListener DEFAULT_WORKER_STATE_CHANGE_LISTENER = new NoOpWorkerStateChangeListener();
     private static final LeaseCleanupValidator DEFAULT_LEASE_CLEANUP_VALIDATOR = new KinesisLeaseCleanupValidator();
     private static final LeaseSelector<KinesisClientLease> DEFAULT_LEASE_SELECTOR = new GenericLeaseSelector<KinesisClientLease>();
@@ -117,7 +126,7 @@ public class Worker implements Runnable {
     private final Optional<Integer> maxGetRecordsThreadPool;
 
     private final KinesisClientLibLeaseCoordinator leaseCoordinator;
-    private final ShardSyncTaskManager controlServer;
+    private final ShardSyncTaskManager shardSyncTaskManager;
 
     private final ShardPrioritization shardPrioritization;
 
@@ -147,6 +156,9 @@ public class Worker implements Runnable {
     // Periodic Shard Sync related fields
     private LeaderDecider leaderDecider;
     private ShardSyncStrategy shardSyncStrategy;
+    private PeriodicShardSyncManager leaderElectedPeriodicShardSyncManager;
+
+    private final LeaseCleanupManager leaseCleanupManager;
 
     /**
      * Constructor.
@@ -406,7 +418,7 @@ public class Worker implements Runnable {
                 config.getShardPrioritizationStrategy(),
                 config.getRetryGetRecordsInSeconds(),
                 config.getMaxGetRecordsThreadPool(),
-                DEFAULT_WORKER_STATE_CHANGE_LISTENER, DEFAULT_LEASE_CLEANUP_VALIDATOR, null /* leaderDecider */);
+                DEFAULT_WORKER_STATE_CHANGE_LISTENER, DEFAULT_LEASE_CLEANUP_VALIDATOR, null, null);
 
         // If a region name was explicitly specified, use it as the region for Amazon Kinesis and Amazon DynamoDB.
         if (config.getRegionName() != null) {
@@ -467,7 +479,7 @@ public class Worker implements Runnable {
                 shardSyncIdleTimeMillis, cleanupLeasesUponShardCompletion, checkpoint, leaseCoordinator, execService,
                 metricsFactory, taskBackoffTimeMillis, failoverTimeMillis, skipShardSyncAtWorkerInitializationIfLeasesExist,
                 shardPrioritization, Optional.empty(), Optional.empty(), DEFAULT_WORKER_STATE_CHANGE_LISTENER,
-                DEFAULT_LEASE_CLEANUP_VALIDATOR, null);
+                DEFAULT_LEASE_CLEANUP_VALIDATOR, null, null);
     }
 
     /**
@@ -507,6 +519,10 @@ public class Worker implements Runnable {
      *            Max number of threads in the getRecords thread pool.
      * @param leaseCleanupValidator
      *            leaseCleanupValidator instance used to validate leases
+     * @param leaderDecider
+     *            leaderDecider instance used elect shard sync leaders
+     * @param periodicShardSyncManager
+     *            manages periodic shard sync tasks
      */
     // NOTE: This has package level access solely for testing
     // CHECKSTYLE:IGNORE ParameterNumber FOR NEXT 10 LINES
@@ -517,13 +533,13 @@ public class Worker implements Runnable {
             IMetricsFactory metricsFactory, long taskBackoffTimeMillis, long failoverTimeMillis,
             boolean skipShardSyncAtWorkerInitializationIfLeasesExist, ShardPrioritization shardPrioritization,
             Optional<Integer> retryGetRecordsInSeconds, Optional<Integer> maxGetRecordsThreadPool, WorkerStateChangeListener workerStateChangeListener,
-            LeaseCleanupValidator leaseCleanupValidator, LeaderDecider leaderDecider) {
+            LeaseCleanupValidator leaseCleanupValidator, LeaderDecider leaderDecider, PeriodicShardSyncManager periodicShardSyncManager) {
         this(applicationName, recordProcessorFactory, config, streamConfig, initialPositionInStream,
                 parentShardPollIntervalMillis, shardSyncIdleTimeMillis, cleanupLeasesUponShardCompletion, checkpoint,
                 leaseCoordinator, execService, metricsFactory, taskBackoffTimeMillis, failoverTimeMillis,
                 skipShardSyncAtWorkerInitializationIfLeasesExist, shardPrioritization, retryGetRecordsInSeconds,
                 maxGetRecordsThreadPool, workerStateChangeListener, new KinesisShardSyncer(leaseCleanupValidator),
-                leaderDecider);
+                leaderDecider, periodicShardSyncManager);
     }
 
     Worker(String applicationName, IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config,
@@ -533,7 +549,8 @@ public class Worker implements Runnable {
             IMetricsFactory metricsFactory, long taskBackoffTimeMillis, long failoverTimeMillis,
             boolean skipShardSyncAtWorkerInitializationIfLeasesExist, ShardPrioritization shardPrioritization,
             Optional<Integer> retryGetRecordsInSeconds, Optional<Integer> maxGetRecordsThreadPool,
-            WorkerStateChangeListener workerStateChangeListener, ShardSyncer shardSyncer, LeaderDecider leaderDecider) {
+            WorkerStateChangeListener workerStateChangeListener, ShardSyncer shardSyncer, LeaderDecider leaderDecider,
+            PeriodicShardSyncManager periodicShardSyncManager) {
         this.applicationName = applicationName;
         this.recordProcessorFactory = recordProcessorFactory;
         this.config = config;
@@ -547,7 +564,7 @@ public class Worker implements Runnable {
         this.leaseCoordinator = leaseCoordinator;
         this.metricsFactory = metricsFactory;
         this.shardSyncer = shardSyncer;
-        this.controlServer = new ShardSyncTaskManager(streamConfig.getStreamProxy(), leaseCoordinator.getLeaseManager(),
+        this.shardSyncTaskManager = new ShardSyncTaskManager(streamConfig.getStreamProxy(), leaseCoordinator.getLeaseManager(),
                 initialPositionInStream, cleanupLeasesUponShardCompletion, config.shouldIgnoreUnexpectedChildShards(),
                 shardSyncIdleTimeMillis, metricsFactory, executorService, shardSyncer);
         this.taskBackoffTimeMillis = taskBackoffTimeMillis;
@@ -558,19 +575,42 @@ public class Worker implements Runnable {
         this.maxGetRecordsThreadPool = maxGetRecordsThreadPool;
         this.workerStateChangeListener = workerStateChangeListener;
         workerStateChangeListener.onWorkerStateChange(WorkerStateChangeListener.WorkerState.CREATED);
-        this.leaderDecider = leaderDecider;
+        createShardSyncStrategy(config.getShardSyncStrategyType(), leaderDecider, periodicShardSyncManager);
-        this.shardSyncStrategy = createShardSyncStrategy(config.getShardSyncStrategyType());
+        this.leaseCleanupManager = LeaseCleanupManager.createOrGetInstance(streamConfig.getStreamProxy(), leaseCoordinator.getLeaseManager(),
-        LOG.info(String.format("Shard sync strategy determined as %s.", shardSyncStrategy.getStrategyType().toString()));
+                Executors.newSingleThreadScheduledExecutor(), metricsFactory, cleanupLeasesUponShardCompletion,
+                config.leaseCleanupIntervalMillis(), config.completedLeaseCleanupThresholdMillis(),
+                config.garbageLeaseCleanupThresholdMillis(), config.getMaxRecords());
     }
 
-    private ShardSyncStrategy createShardSyncStrategy(ShardSyncStrategyType strategyType) {
+    /**
+     * Create shard sync strategy and corresponding {@link LeaderDecider} based on provided configs. PERIODIC
+     * {@link ShardSyncStrategyType} honors custom leaderDeciders for leader election strategy, and does not permit
+     * skipping shard syncs if the hash range is complete. All other {@link ShardSyncStrategyType}s permit only a
+     * default single-leader strategy, and will skip shard syncs unless a hole in the hash range is detected.
+     */
+    private void createShardSyncStrategy(ShardSyncStrategyType strategyType,
+                                         LeaderDecider leaderDecider,
+                                         PeriodicShardSyncManager periodicShardSyncManager) {
         switch (strategyType) {
             case PERIODIC:
-               return createPeriodicShardSyncStrategy(streamConfig.getStreamProxy(), leaseCoordinator.getLeaseManager());
+                this.leaderDecider = getOrCreateLeaderDecider(leaderDecider);
+                this.leaderElectedPeriodicShardSyncManager =
+                        getOrCreatePeriodicShardSyncManager(periodicShardSyncManager, false);
+                this.shardSyncStrategy = createPeriodicShardSyncStrategy();
+                break;
             case SHARD_END:
             default:
-                return createShardEndShardSyncStrategy(controlServer);
+                if (leaderDecider != null) {
+                    LOG.warn("LeaderDecider cannot be customized with non-PERIODIC shard sync strategy type. Using " +
+                             "default LeaderDecider.");
                 }
+                this.leaderDecider = getOrCreateLeaderDecider(null);
+                this.leaderElectedPeriodicShardSyncManager =
+                        getOrCreatePeriodicShardSyncManager(periodicShardSyncManager, true);
+                this.shardSyncStrategy = createShardEndShardSyncStrategy();
+        }
+
+        LOG.info("Shard sync strategy determined as " + shardSyncStrategy.getStrategyType().toString());
     }
 
     private static KinesisClientLibLeaseCoordinator getLeaseCoordinator(KinesisClientLibConfiguration config,
@@ -602,6 +642,20 @@ public class Worker implements Runnable {
         return leaseCoordinator;
     }
 
+    /**
+     * @return the leaderDecider
+     */
+    LeaderDecider getLeaderDecider() {
+        return leaderDecider;
+    }
+
+    /**
+     * @return the leaderElectedPeriodicShardSyncManager
+     */
+    PeriodicShardSyncManager getPeriodicShardSyncManager() {
+        return leaderElectedPeriodicShardSyncManager;
+    }
+
     /**
      * Start consuming data from the stream, and pass it to the application record processors.
      */
@@ -614,7 +668,8 @@ public class Worker implements Runnable {
             initialize();
             LOG.info("Initialization complete. Starting worker loop.");
         } catch (RuntimeException e1) {
-            LOG.error("Unable to initialize after " + MAX_INITIALIZATION_ATTEMPTS + " attempts. Shutting down.", e1);
+            LOG.error("Unable to initialize after " + config.getMaxInitializationAttempts() + " attempts. " +
+                    "Shutting down.", e1);
             shutdown();
         }
 
@@ -641,10 +696,6 @@ public class Worker implements Runnable {
                 assignedShards.add(shardInfo);
             }
 
-            if (foundCompletedShard) {
-                shardSyncStrategy.onFoundCompletedShard();
-            }
-
             // clean up shard consumers for unassigned shards
             cleanupShardConsumers(assignedShards);
 
@@ -667,36 +718,38 @@ public class Worker implements Runnable {
         boolean isDone = false;
         Exception lastException = null;
 
-        for (int i = 0; (!isDone) && (i < MAX_INITIALIZATION_ATTEMPTS); i++) {
+        for (int i = 0; (!isDone) && (i < config.getMaxInitializationAttempts()); i++) {
             try {
                 LOG.info("Initialization attempt " + (i + 1));
                 LOG.info("Initializing LeaseCoordinator");
                 leaseCoordinator.initialize();
 
-                TaskResult result = null;
+                // Perform initial lease sync if configs allow it, with jitter.
-                if (!skipShardSyncAtWorkerInitializationIfLeasesExist
+                if (shouldInitiateLeaseSync()) {
-                        || leaseCoordinator.getLeaseManager().isLeaseTableEmpty()) {
+                    LOG.info(config.getWorkerIdentifier() + " worker is beginning initial lease sync.");
-                    LOG.info("Syncing Kinesis shard info");
+                    TaskResult result = leaderElectedPeriodicShardSyncManager.syncShardsOnce();
-                    ShardSyncTask shardSyncTask = new ShardSyncTask(streamConfig.getStreamProxy(),
+                    if (result.getException() != null) {
-                            leaseCoordinator.getLeaseManager(), initialPosition, cleanupLeasesUponShardCompletion,
+                        throw result.getException();
-                            config.shouldIgnoreUnexpectedChildShards(), 0L, shardSyncer, null);
+                    }
-                    result = new MetricsCollectingTaskDecorator(shardSyncTask, metricsFactory).call();
-                } else {
-                    LOG.info("Skipping shard sync per config setting (and lease table is not empty)");
                 }
 
-                if (result == null || result.getException() == null) {
+                leaseCleanupManager.start();
+
+                // If we reach this point, then we either skipped the lease sync or did not have any exception for the
+                // shard sync in the previous attempt.
                 if (!leaseCoordinator.isRunning()) {
                     LOG.info("Starting LeaseCoordinator");
                     leaseCoordinator.start();
                 } else {
                     LOG.info("LeaseCoordinator is already running. No need to start it.");
                 }
+
+                // All shard sync strategies' initialization handlers should begin a periodic shard sync. For
+                // PeriodicShardSync strategy, this is the main shard sync loop. For ShardEndShardSync and other
+                // shard sync strategies, this serves as an auditor background process.
                 shardSyncStrategy.onWorkerInitialization();
                 isDone = true;
-                } else {
+
-                    lastException = result.getException();
-                }
             } catch (LeasingException e) {
                 LOG.error("Caught exception when initializing LeaseCoordinator", e);
                 lastException = e;
@@ -712,11 +765,39 @@ public class Worker implements Runnable {
         }
 
         if (!isDone) {
+            leaderElectedPeriodicShardSyncManager.stop();
             throw new RuntimeException(lastException);
+
         }
         workerStateChangeListener.onWorkerStateChange(WorkerStateChangeListener.WorkerState.STARTED);
     }
 
+    @VisibleForTesting
+    boolean shouldInitiateLeaseSync() throws InterruptedException, DependencyException, InvalidStateException,
+            ProvisionedThroughputException {
+
+        final ILeaseManager leaseManager = leaseCoordinator.getLeaseManager();
+        if (skipShardSyncAtWorkerInitializationIfLeasesExist && !leaseManager.isLeaseTableEmpty()) {
+            LOG.info("Skipping shard sync because getSkipShardSyncAtWorkerInitializationIfLeasesExist config is set " +
+                    "to TRUE and lease table is not empty.");
+            return false;
+        }
+
+        final long waitTime = ThreadLocalRandom.current().nextLong(MIN_WAIT_TIME_FOR_LEASE_TABLE_CHECK_MILLIS,
+                MAX_WAIT_TIME_FOR_LEASE_TABLE_CHECK_MILLIS);
+        final long waitUntil = System.currentTimeMillis() + waitTime;
+
+        boolean shouldInitiateLeaseSync = true;
+        while (System.currentTimeMillis() < waitUntil && (shouldInitiateLeaseSync = leaseManager.isLeaseTableEmpty())) {
+            // Check every 3 seconds if lease table is still empty, to minimize contention between all workers
+            // bootstrapping from empty lease table at the same time.
+            LOG.info("Lease table is still empty. Checking again in " + LEASE_TABLE_CHECK_FREQUENCY_MILLIS + " ms.");
+            Thread.sleep(LEASE_TABLE_CHECK_FREQUENCY_MILLIS);
+        }
+
+        return shouldInitiateLeaseSync;
+    }
+
     /**
      * NOTE: This method is internal/private to the Worker class. It has package access solely for testing.
      *
@@ -1039,12 +1120,21 @@ public class Worker implements Runnable {
     }
 
     protected ShardConsumer buildConsumer(ShardInfo shardInfo, IRecordProcessorFactory processorFactory) {
-        IRecordProcessor recordProcessor = processorFactory.createProcessor();
+        final IRecordProcessor recordProcessor = processorFactory.createProcessor();
+        final RecordProcessorCheckpointer recordProcessorCheckpointer = new RecordProcessorCheckpointer(
+                shardInfo,
+                checkpointTracker,
+                new SequenceNumberValidator(
+                        streamConfig.getStreamProxy(),
+                        shardInfo.getShardId(),
+                        streamConfig.shouldValidateSequenceNumberBeforeCheckpointing()),
+                metricsFactory);
 
         return new ShardConsumer(shardInfo,
                 streamConfig,
                 checkpointTracker,
                 recordProcessor,
+                recordProcessorCheckpointer,
                 leaseCoordinator,
                 parentShardPollIntervalMillis,
                 cleanupLeasesUponShardCompletion,
@@ -1052,9 +1142,11 @@ public class Worker implements Runnable {
                 metricsFactory,
                 taskBackoffTimeMillis,
                 skipShardSyncAtWorkerInitializationIfLeasesExist,
+                new KinesisDataFetcher(streamConfig.getStreamProxy(), shardInfo),
                 retryGetRecordsInSeconds,
                 maxGetRecordsThreadPool,
-                config, shardSyncer, shardSyncStrategy);
+                config, shardSyncer, shardSyncStrategy,
+                leaseCleanupManager);
     }
 
     /**
@@ -1163,18 +1255,47 @@ public class Worker implements Runnable {
         }
     }
 
-    private PeriodicShardSyncStrategy createPeriodicShardSyncStrategy(IKinesisProxy kinesisProxy,
+    private PeriodicShardSyncStrategy createPeriodicShardSyncStrategy() {
-            ILeaseManager<KinesisClientLease> leaseManager) {
+        return new PeriodicShardSyncStrategy(leaderElectedPeriodicShardSyncManager);
-        return new PeriodicShardSyncStrategy(
-                new PeriodicShardSyncManager(config.getWorkerIdentifier(), leaderDecider,
-                        new ShardSyncTask(kinesisProxy, leaseManager, config.getInitialPositionInStreamExtended(),
-                                config.shouldCleanupLeasesUponShardCompletion(),
-                                config.shouldIgnoreUnexpectedChildShards(), SHARD_SYNC_SLEEP_FOR_PERIODIC_SHARD_SYNC,
-                                shardSyncer, null), metricsFactory));
     }
 
-    private ShardEndShardSyncStrategy createShardEndShardSyncStrategy(ShardSyncTaskManager shardSyncTaskManager) {
+    private ShardEndShardSyncStrategy createShardEndShardSyncStrategy() {
-        return new ShardEndShardSyncStrategy(shardSyncTaskManager);
+        return new ShardEndShardSyncStrategy(shardSyncTaskManager, leaderElectedPeriodicShardSyncManager);
+    }
+
+    private LeaderDecider getOrCreateLeaderDecider(LeaderDecider leaderDecider) {
+        if (leaderDecider != null) {
+            return leaderDecider;
+        }
+
+        return new DeterministicShuffleShardSyncLeaderDecider(leaseCoordinator.getLeaseManager(),
+                Executors.newSingleThreadScheduledExecutor(), PERIODIC_SHARD_SYNC_MAX_WORKERS_DEFAULT);
+    }
+
+    /** A non-null PeriodicShardSyncManager can only provided from unit tests. Any application code will create the
+     * PeriodicShardSyncManager for the first time here. */
+    private PeriodicShardSyncManager getOrCreatePeriodicShardSyncManager(PeriodicShardSyncManager periodicShardSyncManager,
+                                                                         boolean isAuditorMode) {
+        if (periodicShardSyncManager != null) {
+            return periodicShardSyncManager;
+        }
+
+        return new PeriodicShardSyncManager(config.getWorkerIdentifier(),
+                                            leaderDecider,
+                                            new ShardSyncTask(streamConfig.getStreamProxy(),
+                                                    leaseCoordinator.getLeaseManager(),
+                                                    config.getInitialPositionInStreamExtended(),
+                                                    config.shouldCleanupLeasesUponShardCompletion(),
+                                                    config.shouldIgnoreUnexpectedChildShards(),
+                                                    SHARD_SYNC_SLEEP_FOR_PERIODIC_SHARD_SYNC,
+                                                    shardSyncer,
+                                                    null),
+                                            metricsFactory,
+                                            leaseCoordinator.getLeaseManager(),
+                                            streamConfig.getStreamProxy(),
+                                            isAuditorMode,
+                                            config.getLeasesRecoveryAuditorExecutionFrequencyMillis(),
+                                            config.getLeasesRecoveryAuditorInconsistencyConfidenceThreshold());
     }
 
     /**
@@ -1242,7 +1363,6 @@ public class Worker implements Runnable {
         @Setter @Accessors(fluent = true)
         private ShardSyncer shardSyncer;
 
-
         @VisibleForTesting
         AmazonKinesis getKinesisClient() {
             return kinesisClient;
@@ -1349,7 +1469,7 @@ public class Worker implements Runnable {
             }
 
             if (shardPrioritization == null) {
-                shardPrioritization = new ParentsFirstShardPrioritization(1);
+                shardPrioritization = new NoOpShardPrioritization();
             }
 
             if (kinesisProxy == null) {
@@ -1379,7 +1499,7 @@ public class Worker implements Runnable {
             }
 
             // We expect users to either inject both LeaseRenewer and the corresponding thread-pool, or neither of them (DEFAULT).
-           if (leaseRenewer == null){
+           if (leaseRenewer == null) {
                 ExecutorService leaseRenewerThreadPool = LeaseCoordinator.getDefaultLeaseRenewalExecutorService(config.getMaxLeaseRenewalThreads());
                 leaseRenewer = new LeaseRenewer<>(leaseManager, config.getWorkerIdentifier(), config.getFailoverTimeMillis(), leaseRenewerThreadPool);
             }
@@ -1419,7 +1539,10 @@ public class Worker implements Runnable {
                     shardPrioritization,
                     config.getRetryGetRecordsInSeconds(),
                     config.getMaxGetRecordsThreadPool(),
-                    workerStateChangeListener, shardSyncer, leaderDecider);
+                    workerStateChangeListener,
+                    shardSyncer,
+                    leaderDecider,
+                    null /* PeriodicShardSyncManager */);
         }
 
         <R, T extends AwsClientBuilder<T, R>> R createClient(final T builder,

src/main/java/com/amazonaws/services/kinesis/clientlibrary/proxies/IKinesisProxy.java

@@ -26,6 +26,7 @@ import com.amazonaws.services.kinesis.model.InvalidArgumentException;
 import com.amazonaws.services.kinesis.model.PutRecordResult;
 import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
 import com.amazonaws.services.kinesis.model.Shard;
+import com.amazonaws.services.kinesis.model.ShardFilter;
 
 /**
  * Kinesis proxy interface. Operates on a single stream (set up at initialization).
@@ -78,6 +79,17 @@ public interface IKinesisProxy {
      */
     List<Shard> getShardList() throws ResourceNotFoundException;
 
+    /**
+     * Fetch a subset shards defined for the stream using a filter on the ListShards API. This can be used to
+     * discover new shards and consume data from them, while limiting the total number of shards returned for
+     * performance or efficiency reasons.
+     *
+     * @param shardFilter currently supported filter types are AT_LATEST, AT_TRIM_HORIZON, AT_TIMESTAMP.
+     * @return List of all shards in the Kinesis stream.
+     * @throws ResourceNotFoundException The Kinesis stream was not found.
+     */
+    List<Shard> getShardListWithFilter(ShardFilter shardFilter) throws ResourceNotFoundException;
+
     /**
      * Used to verify during ShardConsumer shutdown if the provided shardId is for a shard that has been closed.
      * @param shardId Id of the shard that needs to be verified.

src/main/java/com/amazonaws/services/kinesis/clientlibrary/proxies/KinesisProxy.java

@@ -29,6 +29,7 @@ import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.Function;
 import java.util.stream.Collectors;
 
+import com.amazonaws.services.kinesis.model.ShardFilter;
 import com.amazonaws.util.CollectionUtils;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.logging.Log;
@@ -309,7 +310,7 @@ public class KinesisProxy implements IKinesisProxyExtended {
         }
     }
     
-    private ListShardsResult listShards(final String nextToken) {
+    private ListShardsResult listShards(final ShardFilter shardFilter, final String nextToken) {
         final ListShardsRequest request = new ListShardsRequest();
         request.setRequestCredentials(credentialsProvider.getCredentials());
         if (StringUtils.isEmpty(nextToken)) {
@@ -317,6 +318,11 @@ public class KinesisProxy implements IKinesisProxyExtended {
         } else {
             request.setNextToken(nextToken);
         }
+
+        if (shardFilter != null) {
+            request.setShardFilter(shardFilter);
+        }
+
         ListShardsResult result = null;
         LimitExceededException lastException = null;
         int remainingRetries = this.maxListShardsRetryAttempts;
@@ -429,6 +435,14 @@ public class KinesisProxy implements IKinesisProxyExtended {
      */
     @Override
     public synchronized List<Shard> getShardList() {
+        return getShardListWithFilter(null);
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    @Override
+    public synchronized List<Shard> getShardListWithFilter(ShardFilter shardFilter) {
         if (shardIterationState == null) {
             shardIterationState = new ShardIterationState();
         }
@@ -438,7 +452,7 @@ public class KinesisProxy implements IKinesisProxyExtended {
             String nextToken = null;
 
             do {
-                result = listShards(nextToken);
+                result = listShards(shardFilter, nextToken);
 
                 if (result == null) {
                     /*

src/main/java/com/amazonaws/services/kinesis/clientlibrary/proxies/MetricsCollectingKinesisProxyDecorator.java

@@ -28,6 +28,7 @@ import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
 import com.amazonaws.services.kinesis.model.Shard;
 import com.amazonaws.services.kinesis.metrics.impl.MetricsHelper;
 import com.amazonaws.services.kinesis.metrics.interfaces.MetricsLevel;
+import com.amazonaws.services.kinesis.model.ShardFilter;
 
 /**
  * IKinesisProxy implementation that wraps another implementation and collects metrics.
@@ -179,6 +180,22 @@ public class MetricsCollectingKinesisProxyDecorator implements IKinesisProxy {
         }
     }
 
+    /**
+     * {@inheritDoc}
+     */
+    @Override
+    public List<Shard> getShardListWithFilter(ShardFilter shardFilter) throws ResourceNotFoundException {
+        long startTime = System.currentTimeMillis();
+        boolean success = false;
+        try {
+            List<Shard> response = other.getShardListWithFilter(shardFilter);
+            success = true;
+            return response;
+        } finally {
+            MetricsHelper.addSuccessAndLatency(getShardListMetric, startTime, success, MetricsLevel.DETAILED);
+        }
+    }
+
     /**
      * {@inheritDoc}
      */

src/main/java/com/amazonaws/services/kinesis/clientlibrary/types/ExtendedSequenceNumber.java

@@ -141,6 +141,10 @@ public class ExtendedSequenceNumber implements Comparable<ExtendedSequenceNumber
         return subSequenceNumber;
     }
 
+    public boolean isShardEnd() {
+        return sequenceNumber.equals(SentinelCheckpoint.SHARD_END.toString());
+    }
+
     @Override
     public String toString() {
         StringBuilder sb = new StringBuilder();

src/main/java/com/amazonaws/services/kinesis/leases/LeasePendingDeletion.java

@@ -0,0 +1,31 @@
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.amazonaws.services.kinesis.leases;
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardInfo;
+import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
+import lombok.Value;
+import lombok.experimental.Accessors;
+
+/**
+ * Helper class for cleaning up leases.
+ */
+@Accessors(fluent=true)
+@Value
+public class LeasePendingDeletion {
+    private final KinesisClientLease lease;
+    private final ShardInfo shardInfo;
+}
+

src/main/java/com/amazonaws/services/kinesis/leases/exceptions/CustomerApplicationException.java

@@ -0,0 +1,24 @@
+/*
+ * Copyright 2019 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.amazonaws.services.kinesis.leases.exceptions;
+
+public class CustomerApplicationException extends Exception {
+    public CustomerApplicationException(Throwable t) {super(t);}
+
+    public CustomerApplicationException(String message, Throwable t) {super(message, t);}
+
+    public CustomerApplicationException(String message) {super(message);}
+}

src/main/java/com/amazonaws/services/kinesis/leases/impl/HashKeyRangeForLease.java

@@ -0,0 +1,79 @@
+/*
+ * Copyright 2019 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.amazonaws.services.kinesis.leases.impl;
+
+import com.amazonaws.services.kinesis.model.HashKeyRange;
+import com.fasterxml.jackson.annotation.JsonAutoDetect;
+import lombok.NonNull;
+import lombok.Value;
+import lombok.experimental.Accessors;
+import org.apache.commons.lang3.Validate;
+
+import java.math.BigInteger;
+
+@Value
+@Accessors(fluent = true)
+@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
+/**
+ * Lease POJO to hold the starting hashkey range and ending hashkey range of kinesis shards.
+ */
+public class HashKeyRangeForLease {
+
+    private final BigInteger startingHashKey;
+    private final BigInteger endingHashKey;
+
+    /**
+     * Serialize the startingHashKey for persisting in external storage
+     *
+     * @return Serialized startingHashKey
+     */
+    public String serializedStartingHashKey() {
+        return startingHashKey.toString();
+    }
+
+    /**
+     * Serialize the endingHashKey for persisting in external storage
+     *
+     * @return Serialized endingHashKey
+     */
+    public String serializedEndingHashKey() {
+        return endingHashKey.toString();
+    }
+
+    /**
+     * Deserialize from serialized hashKeyRange string from external storage.
+     *
+     * @param startingHashKeyStr
+     * @param endingHashKeyStr
+     * @return HashKeyRangeForLease
+     */
+    public static HashKeyRangeForLease deserialize(@NonNull String startingHashKeyStr, @NonNull String endingHashKeyStr) {
+        final BigInteger startingHashKey = new BigInteger(startingHashKeyStr);
+        final BigInteger endingHashKey = new BigInteger(endingHashKeyStr);
+        Validate.isTrue(startingHashKey.compareTo(endingHashKey) < 0,
+                "StartingHashKey %s must be less than EndingHashKey %s ", startingHashKeyStr, endingHashKeyStr);
+        return new HashKeyRangeForLease(startingHashKey, endingHashKey);
+    }
+
+    /**
+     * Construct HashKeyRangeForLease from Kinesis HashKeyRange
+     *
+     * @param hashKeyRange
+     * @return HashKeyRangeForLease
+     */
+    public static HashKeyRangeForLease fromHashKeyRange(HashKeyRange hashKeyRange) {
+        return deserialize(hashKeyRange.getStartingHashKey(), hashKeyRange.getEndingHashKey());
+    }
+}

src/main/java/com/amazonaws/services/kinesis/leases/impl/KinesisClientLease.java

@@ -30,6 +30,9 @@ public class KinesisClientLease extends Lease {
     private ExtendedSequenceNumber pendingCheckpoint;
     private Long ownerSwitchesSinceCheckpoint = 0L;
     private Set<String> parentShardIds = new HashSet<String>();
+    private Set<String> childShardIds = new HashSet<>();
+    private HashKeyRangeForLease hashKeyRangeForLease;
+
 
     public KinesisClientLease() {
 
@@ -41,17 +44,22 @@ public class KinesisClientLease extends Lease {
         this.pendingCheckpoint = other.getPendingCheckpoint();
         this.ownerSwitchesSinceCheckpoint = other.getOwnerSwitchesSinceCheckpoint();
         this.parentShardIds.addAll(other.getParentShardIds());
+        this.childShardIds.addAll(other.getChildShardIds());
+        this.hashKeyRangeForLease = other.getHashKeyRange();
     }
 
     KinesisClientLease(String leaseKey, String leaseOwner, Long leaseCounter, UUID concurrencyToken,
             Long lastCounterIncrementNanos, ExtendedSequenceNumber checkpoint, ExtendedSequenceNumber pendingCheckpoint,
-            Long ownerSwitchesSinceCheckpoint, Set<String> parentShardIds) {
+            Long ownerSwitchesSinceCheckpoint, Set<String> parentShardIds, Set<String> childShardIds,
+            HashKeyRangeForLease hashKeyRangeForLease) {
         super(leaseKey, leaseOwner, leaseCounter, concurrencyToken, lastCounterIncrementNanos);
 
         this.checkpoint = checkpoint;
         this.pendingCheckpoint = pendingCheckpoint;
         this.ownerSwitchesSinceCheckpoint = ownerSwitchesSinceCheckpoint;
         this.parentShardIds.addAll(parentShardIds);
+        this.childShardIds.addAll(childShardIds);
+        this.hashKeyRangeForLease = hashKeyRangeForLease;
     }
 
     /**
@@ -69,6 +77,7 @@ public class KinesisClientLease extends Lease {
         setCheckpoint(casted.checkpoint);
         setPendingCheckpoint(casted.pendingCheckpoint);
         setParentShardIds(casted.parentShardIds);
+        setChildShardIds(casted.childShardIds);
     }
 
     /**
@@ -100,6 +109,20 @@ public class KinesisClientLease extends Lease {
         return new HashSet<String>(parentShardIds);
     }
 
+    /**
+     * @return shardIds for the child shards of the current shard. Used for resharding.
+     */
+    public Set<String> getChildShardIds() {
+        return new HashSet<String>(childShardIds);
+    }
+
+    /**
+     * @return hash key range that this lease covers.
+     */
+    public HashKeyRangeForLease getHashKeyRange() {
+        return hashKeyRangeForLease;
+    }
+
     /**
      * Sets checkpoint.
      * 
@@ -143,6 +166,26 @@ public class KinesisClientLease extends Lease {
         this.parentShardIds.addAll(parentShardIds);
     }
 
+    /**
+     * Sets childShardIds.
+     *
+     * @param childShardIds may not be null
+     */
+    public void setChildShardIds(Collection<String> childShardIds) {
+        this.childShardIds.addAll(childShardIds);
+    }
+
+    /**
+     * Sets hashKeyRangeForLease.
+     *
+     * @param hashKeyRangeForLease may not be null
+     */
+    public void setHashKeyRange(HashKeyRangeForLease hashKeyRangeForLease) {
+        verifyNotNull(hashKeyRangeForLease, "hashKeyRangeForLease should not be null");
+
+        this.hashKeyRangeForLease = hashKeyRangeForLease;
+    }
+
     private void verifyNotNull(Object object, String message) {
         if (object == null) {
             throw new IllegalArgumentException(message);

src/main/java/com/amazonaws/services/kinesis/leases/impl/KinesisClientLeaseSerializer.java

@@ -15,6 +15,7 @@
 package com.amazonaws.services.kinesis.leases.impl;
 
 import java.util.Collection;
+import java.util.HashMap;
 import java.util.Map;
 
 import com.amazonaws.services.dynamodbv2.model.AttributeAction;
@@ -26,8 +27,11 @@ import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
 import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
 import com.amazonaws.services.kinesis.leases.interfaces.ILeaseSerializer;
 import com.amazonaws.services.kinesis.leases.util.DynamoUtils;
+import com.amazonaws.util.CollectionUtils;
 import com.google.common.base.Strings;
 
+import static com.amazonaws.services.kinesis.leases.impl.UpdateField.HASH_KEY_RANGE;
+
 /**
  * An implementation of ILeaseSerializer for KinesisClientLease objects.
  */
@@ -39,6 +43,9 @@ public class KinesisClientLeaseSerializer implements ILeaseSerializer<KinesisCli
     private static final String PENDING_CHECKPOINT_SEQUENCE_KEY = "pendingCheckpoint";
     private static final String PENDING_CHECKPOINT_SUBSEQUENCE_KEY = "pendingCheckpointSubSequenceNumber";
     public final String PARENT_SHARD_ID_KEY = "parentShardId";
+    public final String CHILD_SHARD_IDS_KEY = "childShardIds";
+    private static final String STARTING_HASH_KEY = "startingHashKey";
+    private static final String ENDING_HASH_KEY = "endingHashKey";
 
     private final LeaseSerializer baseSerializer = new LeaseSerializer(KinesisClientLease.class);
 
@@ -49,9 +56,12 @@ public class KinesisClientLeaseSerializer implements ILeaseSerializer<KinesisCli
         result.put(OWNER_SWITCHES_KEY, DynamoUtils.createAttributeValue(lease.getOwnerSwitchesSinceCheckpoint()));
         result.put(CHECKPOINT_SEQUENCE_NUMBER_KEY, DynamoUtils.createAttributeValue(lease.getCheckpoint().getSequenceNumber()));
         result.put(CHECKPOINT_SUBSEQUENCE_NUMBER_KEY, DynamoUtils.createAttributeValue(lease.getCheckpoint().getSubSequenceNumber()));
-        if (lease.getParentShardIds() != null && !lease.getParentShardIds().isEmpty()) {
+        if (!CollectionUtils.isNullOrEmpty(lease.getParentShardIds())) {
             result.put(PARENT_SHARD_ID_KEY, DynamoUtils.createAttributeValue(lease.getParentShardIds()));
         }
+        if (!CollectionUtils.isNullOrEmpty(lease.getChildShardIds())) {
+            result.put(CHILD_SHARD_IDS_KEY, DynamoUtils.createAttributeValue(lease.getChildShardIds()));
+        }
 
         if (lease.getPendingCheckpoint() != null && !lease.getPendingCheckpoint().getSequenceNumber().isEmpty()) {
             result.put(PENDING_CHECKPOINT_SEQUENCE_KEY, DynamoUtils.createAttributeValue(lease.getPendingCheckpoint().getSequenceNumber()));
@@ -72,6 +82,7 @@ public class KinesisClientLeaseSerializer implements ILeaseSerializer<KinesisCli
                 DynamoUtils.safeGetLong(dynamoRecord, CHECKPOINT_SUBSEQUENCE_NUMBER_KEY))
         );
         result.setParentShardIds(DynamoUtils.safeGetSS(dynamoRecord, PARENT_SHARD_ID_KEY));
+        result.setChildShardIds(DynamoUtils.safeGetSS(dynamoRecord, CHILD_SHARD_IDS_KEY));
 
         if (!Strings.isNullOrEmpty(DynamoUtils.safeGetString(dynamoRecord, PENDING_CHECKPOINT_SEQUENCE_KEY))) {
             result.setPendingCheckpoint(
@@ -109,6 +120,11 @@ public class KinesisClientLeaseSerializer implements ILeaseSerializer<KinesisCli
         return baseSerializer.getDynamoNonexistantExpectation();
     }
 
+    @Override
+    public Map<String, ExpectedAttributeValue> getDynamoExistentExpectation(final String leaseKey) {
+        return baseSerializer.getDynamoExistentExpectation(leaseKey);
+    }
+
     @Override
     public Map<String, AttributeValueUpdate> getDynamoLeaseCounterUpdate(KinesisClientLease lease) {
         return baseSerializer.getDynamoLeaseCounterUpdate(lease);
@@ -143,6 +159,9 @@ public class KinesisClientLeaseSerializer implements ILeaseSerializer<KinesisCli
         result.put(OWNER_SWITCHES_KEY,
                 new AttributeValueUpdate(DynamoUtils.createAttributeValue(lease.getOwnerSwitchesSinceCheckpoint()),
                         AttributeAction.PUT));
+        if (!CollectionUtils.isNullOrEmpty(lease.getChildShardIds())) {
+            result.put(CHILD_SHARD_IDS_KEY, new AttributeValueUpdate(DynamoUtils.createAttributeValue(lease.getChildShardIds()), AttributeAction.PUT));
+        }
 
         if (lease.getPendingCheckpoint() != null && !lease.getPendingCheckpoint().getSequenceNumber().isEmpty()) {
             result.put(PENDING_CHECKPOINT_SEQUENCE_KEY, new AttributeValueUpdate(DynamoUtils.createAttributeValue(lease.getPendingCheckpoint().getSequenceNumber()), AttributeAction.PUT));
@@ -155,6 +174,28 @@ public class KinesisClientLeaseSerializer implements ILeaseSerializer<KinesisCli
         return result;
     }
 
+    @Override
+    public Map<String, AttributeValueUpdate> getDynamoUpdateLeaseUpdate(KinesisClientLease lease,
+                                                                        UpdateField updateField) {
+        Map<String, AttributeValueUpdate> result = new HashMap<>();
+
+        switch (updateField) {
+            case CHILD_SHARDS:
+                // TODO: Implement update fields for child shards
+                break;
+            case HASH_KEY_RANGE:
+                if (lease.getHashKeyRange() != null) {
+                    result.put(STARTING_HASH_KEY, new AttributeValueUpdate(DynamoUtils.createAttributeValue(
+                            lease.getHashKeyRange().serializedStartingHashKey()), AttributeAction.PUT));
+                    result.put(ENDING_HASH_KEY, new AttributeValueUpdate(DynamoUtils.createAttributeValue(
+                            lease.getHashKeyRange().serializedEndingHashKey()), AttributeAction.PUT));
+                }
+                break;
+        }
+
+        return result;
+    }
+
     @Override
     public Collection<KeySchemaElement> getKeySchema() {
         return baseSerializer.getKeySchema();

src/main/java/com/amazonaws/services/kinesis/leases/impl/LeaseCleanupManager.java

@@ -0,0 +1,372 @@
+package com.amazonaws.services.kinesis.leases.impl;
+
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardInfo;
+import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
+import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
+import com.amazonaws.services.kinesis.leases.LeasePendingDeletion;
+import com.amazonaws.services.kinesis.leases.exceptions.DependencyException;
+import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
+import com.amazonaws.services.kinesis.leases.exceptions.ProvisionedThroughputException;
+import com.amazonaws.services.kinesis.leases.interfaces.ILeaseManager;
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
+import com.amazonaws.services.kinesis.model.ShardIteratorType;
+import com.amazonaws.util.CollectionUtils;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Stopwatch;
+import lombok.AccessLevel;
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.NonNull;
+import lombok.RequiredArgsConstructor;
+import lombok.Value;
+import lombok.experimental.Accessors;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * Helper class to cleanup of any expired/closed shard leases. It will cleanup leases periodically as defined by
+ * {@link KinesisClientLibConfiguration#leaseCleanupIntervalMillis()} upon worker shutdown, following a re-shard event or
+ * a shard expiring from the service.
+ */
+@RequiredArgsConstructor(access= AccessLevel.PACKAGE)
+@EqualsAndHashCode
+public class LeaseCleanupManager {
+    @NonNull
+    private IKinesisProxy kinesisProxy;
+    @NonNull
+    private final ILeaseManager<KinesisClientLease> leaseManager;
+    @NonNull
+    private final ScheduledExecutorService deletionThreadPool;
+    @NonNull
+    private final IMetricsFactory metricsFactory;
+    private final boolean cleanupLeasesUponShardCompletion;
+    private final long leaseCleanupIntervalMillis;
+    private final long completedLeaseCleanupIntervalMillis;
+    private final long garbageLeaseCleanupIntervalMillis;
+    private final int maxRecords;
+
+    private final Stopwatch completedLeaseStopwatch = Stopwatch.createUnstarted();
+    private final Stopwatch garbageLeaseStopwatch = Stopwatch.createUnstarted();
+    private final Queue<LeasePendingDeletion> deletionQueue = new ConcurrentLinkedQueue<>();
+
+    private static final long INITIAL_DELAY = 0L;
+    private static final Log LOG = LogFactory.getLog(LeaseCleanupManager.class);
+
+    @Getter
+    private volatile boolean isRunning = false;
+
+    private static LeaseCleanupManager instance;
+
+    /**
+     * Factory method to return a singleton instance of {@link LeaseCleanupManager}.
+     * @param kinesisProxy
+     * @param leaseManager
+     * @param deletionThreadPool
+     * @param metricsFactory
+     * @param cleanupLeasesUponShardCompletion
+     * @param leaseCleanupIntervalMillis
+     * @param completedLeaseCleanupIntervalMillis
+     * @param garbageLeaseCleanupIntervalMillis
+     * @param maxRecords
+     * @return
+     */
+    public static LeaseCleanupManager createOrGetInstance(IKinesisProxy kinesisProxy, ILeaseManager leaseManager,
+                                                          ScheduledExecutorService deletionThreadPool, IMetricsFactory metricsFactory,
+                                                          boolean cleanupLeasesUponShardCompletion, long leaseCleanupIntervalMillis,
+                                                          long completedLeaseCleanupIntervalMillis, long garbageLeaseCleanupIntervalMillis,
+                                                          int maxRecords) {
+        if (instance == null) {
+            instance = new LeaseCleanupManager(kinesisProxy, leaseManager, deletionThreadPool, metricsFactory, cleanupLeasesUponShardCompletion,
+                    leaseCleanupIntervalMillis, completedLeaseCleanupIntervalMillis, garbageLeaseCleanupIntervalMillis, maxRecords);
+        }
+
+        return instance;
+    }
+
+    /**
+     * Starts the lease cleanup thread, which is scheduled periodically as specified by
+     * {@link LeaseCleanupManager#leaseCleanupIntervalMillis}
+     */
+    public void start() {
+        if (!isRunning) {
+            LOG.info("Starting lease cleanup thread.");
+            completedLeaseStopwatch.start();
+            garbageLeaseStopwatch.start();
+            deletionThreadPool.scheduleAtFixedRate(new LeaseCleanupThread(), INITIAL_DELAY, leaseCleanupIntervalMillis,
+                                                   TimeUnit.MILLISECONDS);
+            isRunning = true;
+        } else {
+            LOG.info("Lease cleanup thread already running, no need to start.");
+        }
+    }
+
+    /**
+     * Enqueues a lease for deletion without check for duplicate entry. Use {@link #isEnqueuedForDeletion}
+     * for checking the duplicate entries.
+     * @param leasePendingDeletion
+     */
+    public void enqueueForDeletion(LeasePendingDeletion leasePendingDeletion) {
+        final KinesisClientLease lease = leasePendingDeletion.lease();
+        if (lease == null) {
+            LOG.warn("Cannot enqueue lease " + lease.getLeaseKey() + " for deferred deletion - instance doesn't hold " +
+                    "the lease for that shard.");
+        } else {
+            LOG.debug("Enqueuing lease " + lease.getLeaseKey() + " for deferred deletion.");
+            if (!deletionQueue.add(leasePendingDeletion)) {
+                LOG.warn("Unable to enqueue lease " + lease.getLeaseKey() + " for deletion.");
+            }
+        }
+    }
+
+    /**
+     * Check if lease was already enqueued for deletion.
+     * //TODO: Optimize verifying duplicate entries https://sim.amazon.com/issues/KinesisLTR-597.
+     * @param leasePendingDeletion
+     * @return true if enqueued for deletion; false otherwise.
+     */
+    public boolean isEnqueuedForDeletion(LeasePendingDeletion leasePendingDeletion) {
+        return deletionQueue.contains(leasePendingDeletion);
+    }
+
+    /**
+     * Returns how many leases are currently waiting in the queue pending deletion.
+     * @return number of leases pending deletion.
+     */
+    private int leasesPendingDeletion() {
+        return deletionQueue.size();
+    }
+
+    private boolean timeToCheckForCompletedShard() {
+        return completedLeaseStopwatch.elapsed(TimeUnit.MILLISECONDS) >= completedLeaseCleanupIntervalMillis;
+    }
+
+    private boolean timeToCheckForGarbageShard() {
+        return garbageLeaseStopwatch.elapsed(TimeUnit.MILLISECONDS) >= garbageLeaseCleanupIntervalMillis;
+    }
+
+    public LeaseCleanupResult cleanupLease(LeasePendingDeletion leasePendingDeletion,
+                                           boolean timeToCheckForCompletedShard, boolean timeToCheckForGarbageShard)
+            throws DependencyException, ProvisionedThroughputException, InvalidStateException {
+        final KinesisClientLease lease = leasePendingDeletion.lease();
+        final ShardInfo shardInfo = leasePendingDeletion.shardInfo();
+
+        boolean cleanedUpCompletedLease = false;
+        boolean cleanedUpGarbageLease = false;
+        boolean alreadyCheckedForGarbageCollection = false;
+        boolean wereChildShardsPresent = false;
+        boolean wasResourceNotFound = false;
+
+        try {
+            if (cleanupLeasesUponShardCompletion && timeToCheckForCompletedShard) {
+                final KinesisClientLease leaseFromDDB = leaseManager.getLease(shardInfo.getShardId());
+                if(leaseFromDDB != null) {
+                    Set<String> childShardKeys = leaseFromDDB.getChildShardIds();
+                    if (CollectionUtils.isNullOrEmpty(childShardKeys)) {
+                        try {
+                            childShardKeys = getChildShardsFromService(shardInfo);
+
+                            if (CollectionUtils.isNullOrEmpty(childShardKeys)) {
+                                LOG.error("No child shards returned from service for shard " + shardInfo.getShardId());
+                            } else {
+                                wereChildShardsPresent = true;
+                                updateLeaseWithChildShards(leasePendingDeletion, childShardKeys);
+                            }
+                        } catch (ResourceNotFoundException e) {
+                            throw e;
+                        } finally {
+                            alreadyCheckedForGarbageCollection = true;
+                        }
+                    } else {
+                        wereChildShardsPresent = true;
+                    }
+                    try {
+                        cleanedUpCompletedLease = cleanupLeaseForCompletedShard(lease, childShardKeys);
+                    } catch (Exception e) {
+                        // Suppressing the exception here, so that we can attempt for garbage cleanup.
+                        LOG.warn("Unable to cleanup lease for shard " + shardInfo.getShardId());
+                    }
+                } else {
+                    LOG.info("Lease not present in lease table while cleaning the shard " + shardInfo.getShardId());
+                    cleanedUpCompletedLease = true;
+                }
+            }
+
+            if (!alreadyCheckedForGarbageCollection && timeToCheckForGarbageShard) {
+                try {
+                    wereChildShardsPresent = !CollectionUtils
+                            .isNullOrEmpty(getChildShardsFromService(shardInfo));
+                } catch (ResourceNotFoundException e) {
+                    throw e;
+                }
+            }
+        } catch (ResourceNotFoundException e) {
+            wasResourceNotFound = true;
+            cleanedUpGarbageLease = cleanupLeaseForGarbageShard(lease);
+        }
+
+        return new LeaseCleanupResult(cleanedUpCompletedLease, cleanedUpGarbageLease, wereChildShardsPresent,
+                wasResourceNotFound);
+    }
+
+    private Set<String> getChildShardsFromService(ShardInfo shardInfo) {
+        final String iterator = kinesisProxy.getIterator(shardInfo.getShardId(), ShardIteratorType.LATEST.toString());
+        return kinesisProxy.get(iterator, maxRecords).getChildShards().stream().map(c -> c.getShardId()).collect(Collectors.toSet());
+    }
+
+
+    // A lease that ended with SHARD_END from ResourceNotFoundException is safe to delete if it no longer exists in the
+    // stream (known explicitly from ResourceNotFound being thrown when processing this shard),
+    private boolean cleanupLeaseForGarbageShard(KinesisClientLease lease) throws DependencyException, ProvisionedThroughputException, InvalidStateException {
+        LOG.info("Deleting lease " + lease.getLeaseKey() + " as it is not present in the stream.");
+        leaseManager.deleteLease(lease);
+        return true;
+    }
+
+    private boolean allParentShardLeasesDeleted(KinesisClientLease lease) throws DependencyException, ProvisionedThroughputException, InvalidStateException {
+        for (String parentShard : lease.getParentShardIds()) {
+            final KinesisClientLease parentLease = leaseManager.getLease(parentShard);
+
+            if (parentLease != null) {
+                LOG.warn("Lease " + lease.getLeaseKey() + " has a parent lease " + parentLease.getLeaseKey() +
+                        " which is still present in the lease table, skipping deletion for this lease.");
+                return false;
+            }
+        }
+        return true;
+    }
+
+    // We should only be deleting the current shard's lease if
+    // 1. All of its children are currently being processed, i.e their checkpoint is not TRIM_HORIZON or AT_TIMESTAMP.
+    // 2. Its parent shard lease(s) have already been deleted.
+    private boolean cleanupLeaseForCompletedShard(KinesisClientLease lease, Set<String> childShardLeaseKeys)
+            throws DependencyException, ProvisionedThroughputException, InvalidStateException, IllegalStateException {
+        final Set<String> processedChildShardLeaseKeys = new HashSet<>();
+
+        for (String childShardLeaseKey : childShardLeaseKeys) {
+            final KinesisClientLease childShardLease = Optional.ofNullable(
+                    leaseManager.getLease(childShardLeaseKey))
+                    .orElseThrow(() -> new IllegalStateException(
+                            "Child lease " + childShardLeaseKey + " for completed shard not found in "
+                                    + "lease table - not cleaning up lease " + lease));
+
+            if (!childShardLease.getCheckpoint().equals(ExtendedSequenceNumber.TRIM_HORIZON) && !childShardLease
+                    .getCheckpoint().equals(ExtendedSequenceNumber.AT_TIMESTAMP)) {
+                processedChildShardLeaseKeys.add(childShardLease.getLeaseKey());
+            }
+        }
+
+        if (!allParentShardLeasesDeleted(lease) || !Objects.equals(childShardLeaseKeys, processedChildShardLeaseKeys)) {
+            return false;
+        }
+
+        LOG.info("Deleting lease " + lease.getLeaseKey() + " as it has been completely processed and processing of child shard(s) has begun.");
+        leaseManager.deleteLease(lease);
+
+        return true;
+    }
+
+    private void updateLeaseWithChildShards(LeasePendingDeletion leasePendingDeletion, Set<String> childShardKeys)
+            throws DependencyException, ProvisionedThroughputException, InvalidStateException {
+        final KinesisClientLease updatedLease = leasePendingDeletion.lease();
+        updatedLease.setChildShardIds(childShardKeys);
+
+        leaseManager.updateLease(updatedLease);
+    }
+
+    @VisibleForTesting
+    void cleanupLeases() {
+        LOG.info("Number of pending leases to clean before the scan : " + leasesPendingDeletion());
+        if (deletionQueue.isEmpty()) {
+            LOG.debug("No leases pending deletion.");
+        } else if (timeToCheckForCompletedShard() | timeToCheckForGarbageShard()) {
+            final Queue<LeasePendingDeletion> failedDeletions = new ConcurrentLinkedQueue<>();
+            boolean completedLeaseCleanedUp = false;
+            boolean garbageLeaseCleanedUp = false;
+
+            LOG.debug("Attempting to clean up " + deletionQueue.size() + " lease(s).");
+
+            while (!deletionQueue.isEmpty()) {
+                final LeasePendingDeletion leasePendingDeletion = deletionQueue.poll();
+                final String leaseKey = leasePendingDeletion.lease().getLeaseKey();
+                boolean deletionSucceeded = false;
+                try {
+                    final LeaseCleanupResult leaseCleanupResult = cleanupLease(leasePendingDeletion,
+                            timeToCheckForCompletedShard(), timeToCheckForGarbageShard());
+                    completedLeaseCleanedUp |= leaseCleanupResult.cleanedUpCompletedLease();
+                    garbageLeaseCleanedUp |= leaseCleanupResult.cleanedUpGarbageLease();
+
+                    if (leaseCleanupResult.leaseCleanedUp()) {
+                        LOG.debug("Successfully cleaned up lease " + leaseKey);
+                        deletionSucceeded = true;
+                    } else {
+                        LOG.warn("Unable to clean up lease " + leaseKey + " due to " + leaseCleanupResult);
+                    }
+                } catch (Exception e) {
+                    LOG.error("Failed to cleanup lease " + leaseKey + ". Will re-enqueue for deletion and retry on next " +
+                            "scheduled execution.", e);
+                }
+                if (!deletionSucceeded) {
+                    LOG.debug("Did not cleanup lease " + leaseKey + ". Re-enqueueing for deletion.");
+                    failedDeletions.add(leasePendingDeletion);
+                }
+            }
+            if (completedLeaseCleanedUp) {
+                LOG.debug("At least one completed lease was cleaned up - restarting interval");
+                completedLeaseStopwatch.reset().start();
+            }
+            if (garbageLeaseCleanedUp) {
+                LOG.debug("At least one garbage lease was cleaned up - restarting interval");
+                garbageLeaseStopwatch.reset().start();
+            }
+            deletionQueue.addAll(failedDeletions);
+
+            LOG.info("Number of pending leases to clean after the scan : " +  leasesPendingDeletion());
+        }
+    }
+
+    private class LeaseCleanupThread implements Runnable {
+        @Override
+        public void run() {
+            cleanupLeases();
+        }
+    }
+
+    @Value
+    @Accessors(fluent=true)
+    public static class LeaseCleanupResult {
+        boolean cleanedUpCompletedLease;
+        boolean cleanedUpGarbageLease;
+        boolean wereChildShardsPresent;
+        boolean wasResourceNotFound;
+
+        public boolean leaseCleanedUp() {
+            return cleanedUpCompletedLease | cleanedUpGarbageLease;
+        }
+    }
+}

src/main/java/com/amazonaws/services/kinesis/leases/impl/LeaseManager.java

@@ -241,7 +241,7 @@ public class LeaseManager<T extends Lease> implements ILeaseManager<T> {
      */
     @Override
     public boolean isLeaseTableEmpty() throws DependencyException, InvalidStateException, ProvisionedThroughputException {
-        return list(1).isEmpty();
+        return list(1, 1).isEmpty();
     }
 
     /**
@@ -254,6 +254,20 @@ public class LeaseManager<T extends Lease> implements ILeaseManager<T> {
      * @throws ProvisionedThroughputException if DynamoDB scan fail due to exceeded capacity
      */
     List<T> list(Integer limit) throws DependencyException, InvalidStateException, ProvisionedThroughputException {
+        return list(limit, Integer.MAX_VALUE);
+    }
+
+    /**
+     * List with the given page size, up to a limit of paginated calls.
+     *
+     * @param limit number of items to consider at a time - used by integration tests to force paging.
+     * @param maxPages max number of paginated scan calls.
+     * @return list of leases
+     * @throws InvalidStateException if table does not exist
+     * @throws DependencyException if DynamoDB scan fail in an unexpected way
+     * @throws ProvisionedThroughputException if DynamoDB scan fail due to exceeded capacity
+     */
+    private List<T> list(Integer limit, Integer maxPages) throws InvalidStateException, ProvisionedThroughputException, DependencyException {
         if (LOG.isDebugEnabled()) {
             LOG.debug("Listing leases from table " + table);
         }
@@ -278,7 +292,7 @@ public class LeaseManager<T extends Lease> implements ILeaseManager<T> {
                 }
 
                 Map<String, AttributeValue> lastEvaluatedKey = scanResult.getLastEvaluatedKey();
-                if (lastEvaluatedKey == null) {
+                if (lastEvaluatedKey == null || --maxPages <= 0) {
                     // Signify that we're done.
                     scanResult = null;
                     if (LOG.isDebugEnabled()) {
@@ -591,6 +605,37 @@ public class LeaseManager<T extends Lease> implements ILeaseManager<T> {
         return true;
     }
 
+    /**
+     * {@inheritDoc}
+     */
+    @Override
+    public void updateLeaseWithMetaInfo(T lease, UpdateField updateField)
+        throws DependencyException, InvalidStateException, ProvisionedThroughputException {
+        verifyNotNull(lease, "lease cannot be null");
+        verifyNotNull(updateField, "updateField cannot be null");
+
+        if (LOG.isDebugEnabled()) {
+            LOG.debug("Updating lease " + lease + " for field " + updateField);
+        }
+
+        UpdateItemRequest request = new UpdateItemRequest();
+        request.setTableName(table);
+        request.setKey(serializer.getDynamoHashKey(lease));
+        request.setExpected(serializer.getDynamoExistentExpectation(lease.getLeaseKey()));
+
+        Map<String, AttributeValueUpdate> updates = serializer.getDynamoUpdateLeaseUpdate(lease, updateField);
+        updates.putAll(serializer.getDynamoUpdateLeaseUpdate(lease));
+        request.setAttributeUpdates(updates);
+
+        try {
+            dynamoDBClient.updateItem(request);
+        } catch (ConditionalCheckFailedException e) {
+            LOG.warn("Lease update failed for lease with key " + lease.getLeaseKey() + " because the lease did not exist at the time of the update", e);
+        } catch (AmazonClientException e) {
+            throw convertAndRethrowExceptions("update", lease.getLeaseKey(), e);
+        }
+    }
+
     /*
      * This method contains boilerplate exception handling - it throws or returns something to be thrown. The
      * inconsistency there exists to satisfy the compiler when this method is used at the end of non-void methods.

src/main/java/com/amazonaws/services/kinesis/leases/impl/LeaseSerializer.java

@@ -137,6 +137,18 @@ public class LeaseSerializer implements ILeaseSerializer<Lease> {
         return result;
     }
 
+    @Override
+    public Map<String, ExpectedAttributeValue> getDynamoExistentExpectation(final String leaseKey) {
+        Map<String, ExpectedAttributeValue> result = new HashMap<>();
+
+        ExpectedAttributeValue expectedAV = new ExpectedAttributeValue();
+        expectedAV.setValue(DynamoUtils.createAttributeValue(leaseKey));
+        expectedAV.setExists(true);
+        result.put(LEASE_KEY_KEY, expectedAV);
+
+        return result;
+    }
+
     @Override
     public Map<String, AttributeValueUpdate> getDynamoLeaseCounterUpdate(Lease lease) {
         return getDynamoLeaseCounterUpdate(lease.getLeaseCounter());
@@ -177,6 +189,12 @@ public class LeaseSerializer implements ILeaseSerializer<Lease> {
         return new HashMap<String, AttributeValueUpdate>();
     }
 
+    @Override
+    public Map<String, AttributeValueUpdate> getDynamoUpdateLeaseUpdate(Lease lease, UpdateField updateField) {
+        // There is no application-specific data in Lease - just return a map that increments the counter.
+        return new HashMap<String, AttributeValueUpdate>();
+    }
+
     @Override
     public Collection<KeySchemaElement> getKeySchema() {
         List<KeySchemaElement> keySchema = new ArrayList<KeySchemaElement>();

src/main/java/com/amazonaws/services/kinesis/leases/impl/UpdateField.java

@@ -0,0 +1,26 @@
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.amazonaws.services.kinesis.leases.impl;
+
+/**
+ * These are the special fields that will be updated only once during the lifetime of the lease.
+ * Since these are meta information that will not affect lease ownership or data durability, we allow
+ * any elected leader  or worker to set these fields directly without any conditional checks.
+ * Note that though HASH_KEY_RANGE will be available during lease initialization in newer versions, we keep this
+ * for backfilling while rolling forward to newer versions.
+ */
+public enum UpdateField {
+    CHILD_SHARDS, HASH_KEY_RANGE
+}

src/main/java/com/amazonaws/services/kinesis/leases/interfaces/ILeaseManager.java

@@ -20,6 +20,7 @@ import com.amazonaws.services.kinesis.leases.exceptions.DependencyException;
 import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
 import com.amazonaws.services.kinesis.leases.exceptions.ProvisionedThroughputException;
 import com.amazonaws.services.kinesis.leases.impl.Lease;
+import com.amazonaws.services.kinesis.leases.impl.UpdateField;
 
 /**
  * Supports basic CRUD operations for Leases.
@@ -180,6 +181,19 @@ public interface ILeaseManager<T extends Lease> {
     public boolean updateLease(T lease)
         throws DependencyException, InvalidStateException, ProvisionedThroughputException;
 
+    /**
+     * Update application-specific fields of the given lease in DynamoDB. Does not update fields managed by the leasing
+     * library such as leaseCounter, leaseOwner, or leaseKey.
+     **
+     * @throws InvalidStateException if lease table does not exist
+     * @throws ProvisionedThroughputException if DynamoDB update fails due to lack of capacity
+     * @throws DependencyException if DynamoDB update fails in an unexpected way
+     */
+    default void updateLeaseWithMetaInfo(T lease, UpdateField updateField)
+            throws DependencyException, InvalidStateException, ProvisionedThroughputException {
+        throw new UnsupportedOperationException("updateLeaseWithMetaInfo is not implemented.");
+    }
+
     /**
      * Check (synchronously) if there are any leases in the lease table.
      * 

src/main/java/com/amazonaws/services/kinesis/leases/interfaces/ILeaseSerializer.java

@@ -23,6 +23,7 @@ import com.amazonaws.services.dynamodbv2.model.AttributeValueUpdate;
 import com.amazonaws.services.dynamodbv2.model.ExpectedAttributeValue;
 import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
 import com.amazonaws.services.kinesis.leases.impl.Lease;
+import com.amazonaws.services.kinesis.leases.impl.UpdateField;
 
 /**
  * Utility class that manages the mapping of Lease objects/operations to records in DynamoDB.
@@ -78,6 +79,13 @@ public interface ILeaseSerializer<T extends Lease> {
      */
     public Map<String, ExpectedAttributeValue> getDynamoNonexistantExpectation();
 
+    /**
+     * @return the attribute value map asserting that a lease does exist.
+     */
+    default Map<String, ExpectedAttributeValue> getDynamoExistentExpectation(final String leaseKey) {
+        throw new UnsupportedOperationException("DynamoExistentExpectation is not implemented");
+    }
+
     /**
      * @param lease
      * @return the attribute value map that increments a lease counter
@@ -104,6 +112,15 @@ public interface ILeaseSerializer<T extends Lease> {
      */
     public Map<String, AttributeValueUpdate> getDynamoUpdateLeaseUpdate(T lease);
 
+    /**
+     * @param lease
+     * @param updateField
+     * @return the attribute value map that updates application-specific data for a lease
+     */
+    default Map<String, AttributeValueUpdate> getDynamoUpdateLeaseUpdate(T lease, UpdateField updateField) {
+        throw new UnsupportedOperationException();
+    }
+
     /**
      * @return the key schema for creating a DynamoDB table to store leases
      */

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ConsumerStatesTest.java

@@ -124,7 +124,7 @@ public class ConsumerStatesTest {
         assertThat(state.successTransition(), equalTo(ShardConsumerState.INITIALIZING.getConsumerState()));
         for (ShutdownReason shutdownReason : ShutdownReason.values()) {
             assertThat(state.shutdownTransition(shutdownReason),
-                    equalTo(ShardConsumerState.SHUTDOWN_COMPLETE.getConsumerState()));
+                    equalTo(ShardConsumerState.SHUTTING_DOWN.getConsumerState()));
         }
 
         assertThat(state.getState(), equalTo(ShardConsumerState.WAITING_ON_PARENT_SHARDS));

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ExceptionThrowingLeaseManager.java

@@ -17,6 +17,7 @@ package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 import java.util.Arrays;
 import java.util.List;
 
+import com.amazonaws.services.kinesis.leases.impl.UpdateField;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 
@@ -53,6 +54,7 @@ class ExceptionThrowingLeaseManager implements ILeaseManager<KinesisClientLease>
         DELETELEASE(9),
         DELETEALL(10),
         UPDATELEASE(11),
+        UPDATELEASEWITHMETAINFO(12),
         NONE(Integer.MIN_VALUE);
 
         private Integer index;
@@ -197,6 +199,14 @@ class ExceptionThrowingLeaseManager implements ILeaseManager<KinesisClientLease>
         return leaseManager.updateLease(lease);
     }
 
+    @Override
+    public void updateLeaseWithMetaInfo(KinesisClientLease lease, UpdateField updateField)
+            throws DependencyException, InvalidStateException, ProvisionedThroughputException {
+        throwExceptions("updateLeaseWithMetaInfo", ExceptionThrowingLeaseManagerMethods.UPDATELEASEWITHMETAINFO);
+
+        leaseManager.updateLeaseWithMetaInfo(lease, updateField);
+    }
+
     @Override
     public KinesisClientLease getLease(String shardId)
         throws DependencyException, InvalidStateException, ProvisionedThroughputException {
@@ -215,7 +225,7 @@ class ExceptionThrowingLeaseManager implements ILeaseManager<KinesisClientLease>
     @Override
     public boolean isLeaseTableEmpty() throws DependencyException,
         InvalidStateException, ProvisionedThroughputException {
-        return false;
+        return leaseManager.listLeases().isEmpty();
     }
 
 }

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisDataFetcherTest.java

@@ -36,6 +36,7 @@ import java.util.Collections;
 import java.util.Date;
 import java.util.List;
 
+import com.amazonaws.services.kinesis.model.ChildShard;
 import org.junit.Assert;
 import org.junit.BeforeClass;
 import org.junit.Test;
@@ -132,7 +133,7 @@ public class KinesisDataFetcherTest {
     }
 
     @Test
-    public void testadvanceIteratorTo() throws KinesisClientLibException {
+    public void testadvanceIteratorTo() throws Exception {
         IKinesisProxy kinesis = mock(IKinesisProxy.class);
         ICheckpoint checkpoint = mock(ICheckpoint.class);
 
@@ -146,9 +147,13 @@ public class KinesisDataFetcherTest {
         GetRecordsResult outputA = new GetRecordsResult();
         List<Record> recordsA = new ArrayList<Record>();
         outputA.setRecords(recordsA);
+        outputA.setNextShardIterator("nextShardIteratorA");
+        outputA.setChildShards(Collections.emptyList());
         GetRecordsResult outputB = new GetRecordsResult();
         List<Record> recordsB = new ArrayList<Record>();
         outputB.setRecords(recordsB);
+        outputB.setNextShardIterator("nextShardIteratorB");
+        outputB.setChildShards(Collections.emptyList());
 
         when(kinesis.getIterator(SHARD_ID, AT_SEQUENCE_NUMBER, seqA)).thenReturn(iteratorA);
         when(kinesis.getIterator(SHARD_ID, AT_SEQUENCE_NUMBER, seqB)).thenReturn(iteratorB);
@@ -166,7 +171,7 @@ public class KinesisDataFetcherTest {
     }
 
     @Test
-    public void testadvanceIteratorToTrimHorizonLatestAndAtTimestamp() {
+    public void testadvanceIteratorToTrimHorizonLatestAndAtTimestamp() throws Exception{
         IKinesisProxy kinesis = mock(IKinesisProxy.class);
 
         KinesisDataFetcher fetcher = new KinesisDataFetcher(kinesis, SHARD_INFO);
@@ -189,7 +194,7 @@ public class KinesisDataFetcherTest {
     }
 
     @Test
-    public void testGetRecordsWithResourceNotFoundException() {
+    public void testGetRecordsWithResourceNotFoundException() throws Exception {
         // Set up arguments used by proxy
         String nextIterator = "TestShardIterator";
         int maxRecords = 100;
@@ -211,11 +216,12 @@ public class KinesisDataFetcherTest {
     }
     
     @Test
-    public void testNonNullGetRecords() {
+    public void testNonNullGetRecords() throws Exception {
         String nextIterator = "TestIterator";
         int maxRecords = 100;
         
         KinesisProxy mockProxy = mock(KinesisProxy.class);
+        when(mockProxy.getIterator(anyString(), anyString())).thenReturn("targetIterator");
         doThrow(new ResourceNotFoundException("Test Exception")).when(mockProxy).get(nextIterator, maxRecords);
 
         KinesisDataFetcher dataFetcher = new KinesisDataFetcher(mockProxy, SHARD_INFO);
@@ -232,17 +238,25 @@ public class KinesisDataFetcherTest {
         final String NEXT_ITERATOR_ONE = "NextIteratorOne";
         final String NEXT_ITERATOR_TWO = "NextIteratorTwo";
         when(kinesisProxy.getIterator(anyString(), anyString())).thenReturn(INITIAL_ITERATOR);
-        GetRecordsResult iteratorOneResults = mock(GetRecordsResult.class);
+
-        when(iteratorOneResults.getNextShardIterator()).thenReturn(NEXT_ITERATOR_ONE);
+        GetRecordsResult iteratorOneResults = new GetRecordsResult();
+        iteratorOneResults.setNextShardIterator(NEXT_ITERATOR_ONE);
+        iteratorOneResults.setChildShards(Collections.emptyList());
         when(kinesisProxy.get(eq(INITIAL_ITERATOR), anyInt())).thenReturn(iteratorOneResults);
 
-        GetRecordsResult iteratorTwoResults = mock(GetRecordsResult.class);
+        GetRecordsResult iteratorTwoResults = new GetRecordsResult();
+        iteratorTwoResults.setNextShardIterator(NEXT_ITERATOR_TWO);
+        iteratorTwoResults.setChildShards(Collections.emptyList());
         when(kinesisProxy.get(eq(NEXT_ITERATOR_ONE), anyInt())).thenReturn(iteratorTwoResults);
-        when(iteratorTwoResults.getNextShardIterator()).thenReturn(NEXT_ITERATOR_TWO);
 
-        GetRecordsResult finalResult = mock(GetRecordsResult.class);
+        GetRecordsResult finalResult = new GetRecordsResult();
+        finalResult.setNextShardIterator(null);
+        List<ChildShard> childShards = new ArrayList<>();
+        ChildShard childShard = new ChildShard();
+        childShard.setParentShards(Collections.singletonList("parentShardId"));
+        childShards.add(childShard);
+        finalResult.setChildShards(childShards);
         when(kinesisProxy.get(eq(NEXT_ITERATOR_TWO), anyInt())).thenReturn(finalResult);
-        when(finalResult.getNextShardIterator()).thenReturn(null);
 
         KinesisDataFetcher dataFetcher = new KinesisDataFetcher(kinesisProxy, SHARD_INFO);
         dataFetcher.initialize("TRIM_HORIZON",
@@ -276,13 +290,14 @@ public class KinesisDataFetcherTest {
     }
     
     @Test
-    public void testRestartIterator() {
+    public void testRestartIterator() throws Exception{
         GetRecordsResult getRecordsResult = mock(GetRecordsResult.class);
-        GetRecordsResult restartGetRecordsResult = new GetRecordsResult();
+        GetRecordsResult restartGetRecordsResult = mock(GetRecordsResult.class);
         Record record = mock(Record.class);
         final String initialIterator = "InitialIterator";
         final String nextShardIterator = "NextShardIterator";
         final String restartShardIterator = "RestartIterator";
+        final String restartNextShardIterator = "RestartNextIterator";
         final String sequenceNumber = "SequenceNumber";
         final String iteratorType = "AT_SEQUENCE_NUMBER";
         KinesisProxy kinesisProxy = mock(KinesisProxy.class);
@@ -292,6 +307,7 @@ public class KinesisDataFetcherTest {
         when(kinesisProxy.get(eq(initialIterator), eq(10))).thenReturn(getRecordsResult);
         when(getRecordsResult.getRecords()).thenReturn(Collections.singletonList(record));
         when(getRecordsResult.getNextShardIterator()).thenReturn(nextShardIterator);
+        when(getRecordsResult.getChildShards()).thenReturn(Collections.emptyList());
         when(record.getSequenceNumber()).thenReturn(sequenceNumber);
 
         fetcher.initialize(InitialPositionInStream.LATEST.toString(), INITIAL_POSITION_LATEST);
@@ -300,6 +316,8 @@ public class KinesisDataFetcherTest {
         verify(kinesisProxy).get(eq(initialIterator), eq(10));
 
         when(kinesisProxy.getIterator(eq(SHARD_ID), eq(iteratorType), eq(sequenceNumber))).thenReturn(restartShardIterator);
+        when(restartGetRecordsResult.getNextShardIterator()).thenReturn(restartNextShardIterator);
+        when(restartGetRecordsResult.getChildShards()).thenReturn(Collections.emptyList());
         when(kinesisProxy.get(eq(restartShardIterator), eq(10))).thenReturn(restartGetRecordsResult);
 
         fetcher.restartIterator();
@@ -309,7 +327,7 @@ public class KinesisDataFetcherTest {
     }
     
     @Test (expected = IllegalStateException.class)
-    public void testRestartIteratorNotInitialized() {
+    public void testRestartIteratorNotInitialized() throws Exception {
         KinesisDataFetcher dataFetcher = new KinesisDataFetcher(kinesisProxy, SHARD_INFO);
         dataFetcher.restartIterator();
     }
@@ -354,6 +372,8 @@ public class KinesisDataFetcherTest {
         List<Record> expectedRecords = new ArrayList<Record>();
         GetRecordsResult response = new GetRecordsResult();
         response.setRecords(expectedRecords);
+        response.setNextShardIterator("testNextShardIterator");
+        response.setChildShards(Collections.emptyList());
 
         when(kinesis.getIterator(SHARD_ID, initialPositionInStream.getTimestamp())).thenReturn(iterator);
         when(kinesis.getIterator(SHARD_ID, AT_SEQUENCE_NUMBER, seqNo)).thenReturn(iterator);

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PeriodicShardSyncManagerTest.java

@@ -0,0 +1,616 @@
+/*
+ * Copyright 2019 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
+
+import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
+import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
+import com.amazonaws.services.kinesis.leases.impl.HashKeyRangeForLease;
+import com.amazonaws.services.kinesis.leases.impl.KinesisClientLease;
+import com.amazonaws.services.kinesis.leases.interfaces.ILeaseManager;
+import com.amazonaws.services.kinesis.metrics.impl.NullMetricsFactory;
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+import com.amazonaws.services.kinesis.model.HashKeyRange;
+import com.amazonaws.services.kinesis.model.Shard;
+import com.amazonaws.util.CollectionUtils;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static com.amazonaws.services.kinesis.clientlibrary.lib.worker.PeriodicShardSyncManager.MAX_HASH_KEY;
+import static com.amazonaws.services.kinesis.clientlibrary.lib.worker.PeriodicShardSyncManager.MIN_HASH_KEY;
+import static com.amazonaws.services.kinesis.leases.impl.HashKeyRangeForLease.deserialize;
+import static org.mockito.Mockito.when;
+
+@RunWith(MockitoJUnitRunner.class)
+public class PeriodicShardSyncManagerTest {
+
+    private static final String WORKER_ID = "workerId";
+    public static final long LEASES_RECOVERY_AUDITOR_EXECUTION_FREQUENCY_MILLIS = 2 * 60 * 1000L;
+    public static final int LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD = 3;
+
+    /** Manager for PERIODIC shard sync strategy */
+    private PeriodicShardSyncManager periodicShardSyncManager;
+
+    /** Manager for SHARD_END shard sync strategy */
+    private PeriodicShardSyncManager auditorPeriodicShardSyncManager;
+
+    @Mock
+    private LeaderDecider leaderDecider;
+    @Mock
+    private ShardSyncTask shardSyncTask;
+    @Mock
+    private ILeaseManager<KinesisClientLease> leaseManager;
+    @Mock
+    private IKinesisProxy kinesisProxy;
+
+    private IMetricsFactory metricsFactory = new NullMetricsFactory();
+
+    @Before
+    public void setup() {
+        periodicShardSyncManager = new PeriodicShardSyncManager(WORKER_ID, leaderDecider, shardSyncTask,
+                metricsFactory, leaseManager, kinesisProxy, false, LEASES_RECOVERY_AUDITOR_EXECUTION_FREQUENCY_MILLIS,
+                LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD);
+        auditorPeriodicShardSyncManager = new PeriodicShardSyncManager(WORKER_ID, leaderDecider, shardSyncTask,
+                metricsFactory, leaseManager, kinesisProxy, true, LEASES_RECOVERY_AUDITOR_EXECUTION_FREQUENCY_MILLIS, 
+                LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD);
+    }
+
+    @Test
+    public void testForFailureWhenHashRangesAreIncomplete() {
+        List<KinesisClientLease> hashRanges = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "23"));
+            add(deserialize("6", "23"));
+            add(deserialize("25", MAX_HASH_KEY.toString())); // Missing interval here
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+        Assert.assertTrue(PeriodicShardSyncManager
+                .checkForHoleInHashKeyRanges(hashRanges).isPresent());
+    }
+
+    @Test
+    public void testForSuccessWhenHashRangesAreComplete() {
+        List<KinesisClientLease> hashRanges = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "23"));
+            add(deserialize("6", "23"));
+            add(deserialize("24", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+        Assert.assertFalse(PeriodicShardSyncManager
+                .checkForHoleInHashKeyRanges(hashRanges).isPresent());
+    }
+
+    @Test
+    public void testForSuccessWhenUnsortedHashRangesAreComplete() {
+        List<KinesisClientLease> hashRanges = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize("4", "23"));
+            add(deserialize("2", "3"));
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("24", MAX_HASH_KEY.toString()));
+            add(deserialize("6", "23"));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+        Assert.assertFalse(PeriodicShardSyncManager
+                .checkForHoleInHashKeyRanges(hashRanges).isPresent());
+    }
+
+    @Test
+    public void testForSuccessWhenHashRangesAreCompleteForOverlappingLeasesAtEnd() {
+        List<KinesisClientLease> hashRanges = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "23"));
+            add(deserialize("6", "23"));
+            add(deserialize("24", MAX_HASH_KEY.toString()));
+            add(deserialize("24", "45"));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+        Assert.assertFalse(PeriodicShardSyncManager
+                .checkForHoleInHashKeyRanges(hashRanges).isPresent());
+    }
+
+    @Test
+    public void testIfShardSyncIsInitiatedWhenNoLeasesArePassed() throws Exception {
+        when(leaseManager.listLeases()).thenReturn(null);
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertTrue(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+
+    }
+
+    @Test
+    public void testIfShardSyncIsInitiatedWhenEmptyLeasesArePassed() throws Exception {
+        when(leaseManager.listLeases()).thenReturn(Collections.emptyList());
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertTrue(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+    }
+
+    @Test
+    public void testIfShardSyncIsInitiatedWhenConfidenceFactorIsNotReached() throws Exception {
+        List<KinesisClientLease> leases = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "23"));
+            add(deserialize("6", "23")); // Hole between 23 and 25
+            add(deserialize("25", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+
+        when(leaseManager.listLeases()).thenReturn(leases);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+    }
+
+    @Test
+    public void testIfShardSyncIsInitiatedWhenConfidenceFactorIsReached() throws Exception {
+        List<KinesisClientLease> leases = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "23"));
+            add(deserialize("6", "23")); // Hole between 23 and 25
+            add(deserialize("25", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+
+        when(leaseManager.listLeases()).thenReturn(leases);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertTrue(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+    }
+
+    @Test
+    public void testIfShardSyncIsInitiatedWhenHoleIsDueToShardEnd() throws Exception {
+        List<KinesisClientLease> leases = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("6", "23")); // Introducing hole here through SHARD_END checkpoint
+            add(deserialize("4", "23"));
+            add(deserialize("24", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            if (lease.getHashKeyRange().startingHashKey().toString().equals("4")) {
+                lease.setCheckpoint(ExtendedSequenceNumber.SHARD_END);
+            } else {
+                lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            }
+            return lease;
+        }).collect(Collectors.toList());
+
+        when(leaseManager.listLeases()).thenReturn(leases);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertTrue(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+    }
+
+    @Test
+    public void testIfShardSyncIsInitiatedWhenNoLeasesAreUsedDueToShardEnd() throws Exception {
+        List<KinesisClientLease> leases = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "23"));
+            add(deserialize("6", "23"));
+            add(deserialize("24", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.SHARD_END);
+            return lease;
+        }).collect(Collectors.toList());
+
+        when(leaseManager.listLeases()).thenReturn(leases);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertTrue(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+    }
+
+    @Test
+    public void testIfShardSyncIsInitiatedWhenHoleShifts() throws Exception {
+        List<KinesisClientLease> leases1 = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "23"));
+            add(deserialize("6", "23")); // Hole between 23 and 25
+            add(deserialize("25", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+
+        when(leaseManager.listLeases()).thenReturn(leases1);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+
+        List<KinesisClientLease> leases2 = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3")); // Hole between 3 and 5
+            add(deserialize("5", "23"));
+            add(deserialize("6", "23"));
+            add(deserialize("25", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+
+        // Resetting the holes
+        when(leaseManager.listLeases()).thenReturn(leases2);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertTrue(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+    }
+
+    @Test
+    public void testIfShardSyncIsInitiatedWhenHoleShiftsMoreThanOnce() throws Exception {
+        List<KinesisClientLease> leases1 = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "23"));
+            add(deserialize("6", "23")); // Hole between 23 and 25
+            add(deserialize("25", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+
+        when(leaseManager.listLeases()).thenReturn(leases1);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+
+        List<KinesisClientLease> leases2 = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3")); // Hole between 3 and 5
+            add(deserialize("5", "23"));
+            add(deserialize("6", "23"));
+            add(deserialize("25", MAX_HASH_KEY.toString()));
+        }}.stream().map(hashKeyRangeForLease -> {
+            KinesisClientLease lease = new KinesisClientLease();
+            lease.setHashKeyRange(hashKeyRangeForLease);
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            return lease;
+        }).collect(Collectors.toList());
+
+        // Resetting the holes
+        when(leaseManager.listLeases()).thenReturn(leases2);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+        // Resetting the holes again
+        when(leaseManager.listLeases()).thenReturn(leases1);
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertTrue(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+    }
+
+    @Test
+    public void testIfMissingHashRangeInformationIsFilledBeforeEvaluatingForShardSync() throws Exception {
+        final int[] shardCounter = { 0 };
+        List<HashKeyRangeForLease> hashKeyRangeForLeases = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("4", "20"));
+            add(deserialize("21", "23"));
+            add(deserialize("24", MAX_HASH_KEY.toString()));
+        }};
+
+        List<Shard> kinesisShards = hashKeyRangeForLeases.stream()
+                .map(hashKeyRange -> new Shard()
+                        .withShardId("shard-" + ++shardCounter[0])
+                        .withHashKeyRange(new HashKeyRange()
+                                .withStartingHashKey(hashKeyRange.serializedStartingHashKey())
+                                .withEndingHashKey(hashKeyRange.serializedEndingHashKey())))
+                .collect(Collectors.toList());
+
+        when(kinesisProxy.getShardList()).thenReturn(kinesisShards);
+
+        final int[] leaseCounter = { 0 };
+        List<KinesisClientLease> leases = hashKeyRangeForLeases.stream()
+                .map(hashKeyRange -> {
+                    KinesisClientLease lease = new KinesisClientLease();
+                    lease.setLeaseKey("shard-" + ++leaseCounter[0]);
+                    // Setting the hash range only for the last two leases
+                    if (leaseCounter[0] >= 3) {
+                        lease.setHashKeyRange(hashKeyRange);
+                    }
+                    lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+                    return lease;
+                }).collect(Collectors.toList());
+
+        when(leaseManager.listLeases()).thenReturn(leases);
+
+        // Assert that SHARD_END shard sync should never trigger, but PERIODIC shard sync should always trigger
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+
+        // Assert that all the leases now have hash ranges set
+        for (KinesisClientLease lease : leases) {
+            Assert.assertNotNull(lease.getHashKeyRange());
+        }
+    }
+
+    @Test
+    public void testIfMissingHashRangeInformationIsFilledBeforeEvaluatingForShardSyncInHoleScenario() throws Exception {
+        final int[] shardCounter = { 0 };
+        List<HashKeyRangeForLease> hashKeyRangeForLeases = new ArrayList<HashKeyRangeForLease>() {{
+            add(deserialize(MIN_HASH_KEY.toString(), "1"));
+            add(deserialize("2", "3"));
+            add(deserialize("5", "20")); // Hole between 3 and 5
+            add(deserialize("21", "23"));
+            add(deserialize("24", MAX_HASH_KEY.toString()));
+        }};
+
+        List<Shard> kinesisShards = hashKeyRangeForLeases.stream()
+                .map(hashKeyRange -> new Shard()
+                        .withShardId("shard-" + ++shardCounter[0])
+                        .withHashKeyRange(new HashKeyRange()
+                                .withStartingHashKey(hashKeyRange.serializedStartingHashKey())
+                                .withEndingHashKey(hashKeyRange.serializedEndingHashKey())))
+                .collect(Collectors.toList());
+
+        when(kinesisProxy.getShardList()).thenReturn(kinesisShards);
+
+        final int[] leaseCounter = { 0 };
+        List<KinesisClientLease> leases = hashKeyRangeForLeases.stream()
+                .map(hashKeyRange -> {
+                    KinesisClientLease lease = new KinesisClientLease();
+                    lease.setLeaseKey("shard-" + ++leaseCounter[0]);
+                    // Setting the hash range only for the last two leases
+                    if (leaseCounter[0] >= 3) {
+                        lease.setHashKeyRange(hashKeyRange);
+                    }
+                    lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+                    return lease;
+                }).collect(Collectors.toList());
+
+        when(leaseManager.listLeases()).thenReturn(leases);
+
+        // Assert that shard sync should trigger after breaching threshold
+        for (int i = 1; i < LEASES_RECOVERY_AUDITOR_INCONSISTENCY_CONFIDENCE_THRESHOLD; i++) {
+            Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        }
+        Assert.assertTrue(periodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+        Assert.assertTrue(auditorPeriodicShardSyncManager.checkForShardSync().shouldDoShardSync());
+
+        // Assert that all the leases now have hash ranges set
+        for (KinesisClientLease lease : leases) {
+            Assert.assertNotNull(lease.getHashKeyRange());
+        }
+    }
+
+    @Test
+    public void testFor1000DifferentValidSplitHierarchyTreeTheHashRangesAreAlwaysComplete() {
+        for (int i = 0; i < 1000; i++) {
+            int maxInitialLeaseCount = 100;
+            List<KinesisClientLease> leases = generateInitialLeases(maxInitialLeaseCount);
+            reshard(leases, 5, ReshardType.SPLIT, maxInitialLeaseCount, false);
+            Collections.shuffle(leases);
+            Assert.assertFalse(periodicShardSyncManager.hasHoleInLeases(leases).isPresent());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.hasHoleInLeases(leases).isPresent());
+        }
+    }
+
+    @Test
+    public void testFor1000DifferentValidMergeHierarchyTreeWithSomeInProgressParentsTheHashRangesAreAlwaysComplete() {
+        for (int i = 0; i < 1000; i++) {
+            int maxInitialLeaseCount = 100;
+            List<KinesisClientLease> leases = generateInitialLeases(maxInitialLeaseCount);
+            reshard(leases, 5, ReshardType.MERGE, maxInitialLeaseCount, true);
+            Collections.shuffle(leases);
+            Assert.assertFalse(periodicShardSyncManager.hasHoleInLeases(leases).isPresent());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.hasHoleInLeases(leases).isPresent());
+        }
+    }
+
+    @Test
+    public void testFor1000DifferentValidReshardHierarchyTreeWithSomeInProgressParentsTheHashRangesAreAlwaysComplete() {
+        for (int i = 0; i < 1000; i++) {
+            int maxInitialLeaseCount = 100;
+            List<KinesisClientLease> leases = generateInitialLeases(maxInitialLeaseCount);
+            reshard(leases, 5, ReshardType.ANY, maxInitialLeaseCount, true);
+            Collections.shuffle(leases);
+            Assert.assertFalse(periodicShardSyncManager.hasHoleInLeases(leases).isPresent());
+            Assert.assertFalse(auditorPeriodicShardSyncManager.hasHoleInLeases(leases).isPresent());
+        }
+    }
+
+    private List<KinesisClientLease> generateInitialLeases(int initialShardCount) {
+        long hashRangeInternalMax = 10000000;
+        List<KinesisClientLease> initialLeases = new ArrayList<>();
+        long leaseStartKey = 0;
+        for (int i = 1; i <= initialShardCount; i++) {
+            final KinesisClientLease lease = new KinesisClientLease();
+            long leaseEndKey;
+            if (i != initialShardCount) {
+                leaseEndKey = (hashRangeInternalMax / initialShardCount) * i;
+                lease.setHashKeyRange(deserialize(leaseStartKey + "", leaseEndKey + ""));
+            } else {
+                leaseEndKey = 0;
+                lease.setHashKeyRange(deserialize(leaseStartKey + "", MAX_HASH_KEY.toString()));
+            }
+            lease.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            lease.setLeaseKey("shard-" + i);
+            initialLeases.add(lease);
+            leaseStartKey = leaseEndKey + 1;
+        }
+
+        return initialLeases;
+    }
+
+    private void reshard(List<KinesisClientLease> initialLeases, int depth, ReshardType reshardType, int leaseCounter,
+                         boolean shouldKeepSomeParentsInProgress) {
+        for (int i = 0; i < depth; i++) {
+            if (reshardType == ReshardType.SPLIT) {
+                leaseCounter = split(initialLeases, leaseCounter);
+            } else if (reshardType == ReshardType.MERGE) {
+                leaseCounter = merge(initialLeases, leaseCounter, shouldKeepSomeParentsInProgress);
+            } else {
+                if (isHeads()) {
+                    leaseCounter = split(initialLeases, leaseCounter);
+                } else {
+                    leaseCounter = merge(initialLeases, leaseCounter, shouldKeepSomeParentsInProgress);
+                }
+            }
+        }
+    }
+
+    private int merge(List<KinesisClientLease> initialLeases, int leaseCounter, boolean shouldKeepSomeParentsInProgress) {
+        List<KinesisClientLease> leasesEligibleForMerge = initialLeases.stream()
+                .filter(l -> CollectionUtils.isNullOrEmpty(l.getChildShardIds())).collect(Collectors.toList());
+
+        int leasesToMerge = (int) ((leasesEligibleForMerge.size() - 1) / 2.0 * Math.random());
+        for (int i = 0; i < leasesToMerge; i += 2) {
+            KinesisClientLease parent1 = leasesEligibleForMerge.get(i);
+            KinesisClientLease parent2 = leasesEligibleForMerge.get(i + 1);
+            if (parent2.getHashKeyRange().startingHashKey()
+                    .subtract(parent1.getHashKeyRange().endingHashKey()).equals(BigInteger.ONE)) {
+                parent1.setCheckpoint(ExtendedSequenceNumber.SHARD_END);
+
+                if (!shouldKeepSomeParentsInProgress || (shouldKeepSomeParentsInProgress && isOneFromDiceRoll())) {
+                    parent2.setCheckpoint(ExtendedSequenceNumber.SHARD_END);
+                }
+
+                KinesisClientLease child = new KinesisClientLease();
+                child.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+                child.setLeaseKey("shard-" + ++leaseCounter);
+                child.setHashKeyRange(new HashKeyRangeForLease(parent1.getHashKeyRange().startingHashKey(),
+                        parent2.getHashKeyRange().endingHashKey()));
+                parent1.setChildShardIds(Collections.singletonList(child.getLeaseKey()));
+                parent2.setChildShardIds(Collections.singletonList(child.getLeaseKey()));
+                child.setParentShardIds(Sets.newHashSet(parent1.getLeaseKey(), parent2.getLeaseKey()));
+
+                initialLeases.add(child);
+            }
+        }
+
+        return leaseCounter;
+    }
+
+    private int split(List<KinesisClientLease> initialLeases, int leaseCounter) {
+        List<KinesisClientLease> leasesEligibleForSplit = initialLeases.stream()
+                .filter(l -> CollectionUtils.isNullOrEmpty(l.getChildShardIds())).collect(Collectors.toList());
+
+        int leasesToSplit = (int) (leasesEligibleForSplit.size() * Math.random());
+        for (int i = 0; i < leasesToSplit; i++) {
+            KinesisClientLease parent = leasesEligibleForSplit.get(i);
+            parent.setCheckpoint(ExtendedSequenceNumber.SHARD_END);
+
+            KinesisClientLease child1 = new KinesisClientLease();
+            child1.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            child1.setHashKeyRange(new HashKeyRangeForLease(parent.getHashKeyRange().startingHashKey(),
+                    parent.getHashKeyRange().startingHashKey().add(parent.getHashKeyRange().endingHashKey())
+                            .divide(new BigInteger("2"))));
+            child1.setLeaseKey("shard-" + ++leaseCounter);
+
+            KinesisClientLease child2 = new KinesisClientLease();
+            child2.setCheckpoint(ExtendedSequenceNumber.TRIM_HORIZON);
+            child2.setHashKeyRange(new HashKeyRangeForLease(parent.getHashKeyRange().startingHashKey()
+                    .add(parent.getHashKeyRange().endingHashKey()).divide(new BigInteger("2")).add(BigInteger.ONE),
+                    parent.getHashKeyRange().endingHashKey()));
+            child2.setLeaseKey("shard-" + ++leaseCounter);
+
+            child1.setParentShardIds(Sets.newHashSet(parent.getLeaseKey()));
+            child2.setParentShardIds(Sets.newHashSet(parent.getLeaseKey()));
+            parent.setChildShardIds(Lists.newArrayList(child1.getLeaseKey(), child2.getLeaseKey()));
+
+            initialLeases.add(child1);
+            initialLeases.add(child2);
+        }
+
+        return leaseCounter;
+    }
+
+    private boolean isHeads() {
+        return Math.random() <= 0.5;
+    }
+
+    private boolean isOneFromDiceRoll() {
+        return Math.random() <= 0.16;
+    }
+
+    private enum ReshardType {
+        SPLIT,
+        MERGE,
+        ANY
+    }
+}

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PrefetchGetRecordsCacheIntegrationTest.java

@@ -29,6 +29,7 @@ import static org.mockito.Mockito.when;
 
 import java.nio.ByteBuffer;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
@@ -74,6 +75,8 @@ public class PrefetchGetRecordsCacheIntegrationTest {
     private IKinesisProxy proxy;
     @Mock
     private ShardInfo shardInfo;
+    @Mock
+    private KinesisClientLibLeaseCoordinator leaseCoordinator;
     
     @Before
     public void setup() {
@@ -171,7 +174,7 @@ public class PrefetchGetRecordsCacheIntegrationTest {
     }
     
     @Test
-    public void testExpiredIteratorException() {
+    public void testExpiredIteratorException() throws Exception {
         when(dataFetcher.getRecords(eq(MAX_RECORDS_PER_CALL))).thenAnswer(new Answer<DataFetcherResult>() {
             @Override
             public DataFetcherResult answer(final InvocationOnMock invocationOnMock) throws Throwable {
@@ -215,6 +218,8 @@ public class PrefetchGetRecordsCacheIntegrationTest {
             GetRecordsResult getRecordsResult = new GetRecordsResult();
             getRecordsResult.setRecords(new ArrayList<>(records));
             getRecordsResult.setMillisBehindLatest(1000L);
+            getRecordsResult.setNextShardIterator("testNextShardIterator");
+            getRecordsResult.setChildShards(Collections.emptyList());
 
             return new AdvancingResult(getRecordsResult);
         }

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/PrefetchGetRecordsCacheTest.java

@@ -31,6 +31,7 @@ import static org.mockito.Mockito.when;
 import java.nio.ByteBuffer;
 import java.time.Duration;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
@@ -98,6 +99,8 @@ public class PrefetchGetRecordsCacheTest {
 
         when(getRecordsRetrievalStrategy.getRecords(eq(MAX_RECORDS_PER_CALL))).thenReturn(getRecordsResult);
         when(getRecordsResult.getRecords()).thenReturn(records);
+        when(getRecordsResult.getNextShardIterator()).thenReturn("testNextShardIterator");
+        when(getRecordsResult.getChildShards()).thenReturn(Collections.emptyList());
     }
 
     @Test
@@ -203,7 +206,7 @@ public class PrefetchGetRecordsCacheTest {
     }
     
     @Test
-    public void testExpiredIteratorException() {
+    public void testExpiredIteratorException() throws Exception{
         getRecordsCache.start();
         
         when(getRecordsRetrievalStrategy.getRecords(MAX_RECORDS_PER_CALL)).thenThrow(ExpiredIteratorException.class).thenReturn(getRecordsResult);

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardConsumerTest.java

@@ -39,6 +39,7 @@ import java.io.File;
 import java.math.BigInteger;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collection;
 import java.util.Date;
 import java.util.List;
 import java.util.ListIterator;
@@ -53,6 +54,8 @@ import java.util.concurrent.Future;
 import java.util.concurrent.RejectedExecutionException;
 import java.util.concurrent.TimeUnit;
 
+import com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager;
+import com.amazonaws.services.kinesis.leases.impl.LeaseManager;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.hamcrest.Description;
@@ -137,6 +140,7 @@ public class ShardConsumerTest {
         recordsFetcherFactory = spy(new SimpleRecordsFetcherFactory());
         when(config.getRecordsFetcherFactory()).thenReturn(recordsFetcherFactory);
         when(config.getLogWarningForTaskAfterMillis()).thenReturn(Optional.empty());
+        when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
     }
 
     /**
@@ -245,7 +249,7 @@ public class ShardConsumerTest {
     @SuppressWarnings("unchecked")
     @Test
     public final void testRecordProcessorThrowable() throws Exception {
-        ShardInfo shardInfo = new ShardInfo("s-0-0", "testToken", null, ExtendedSequenceNumber.TRIM_HORIZON);
+        ShardInfo shardInfo = new ShardInfo("s-0-0", UUID.randomUUID().toString(), null, ExtendedSequenceNumber.TRIM_HORIZON);
         StreamConfig streamConfig =
                 new StreamConfig(streamProxy,
                         1,
@@ -271,6 +275,7 @@ public class ShardConsumerTest {
 
         final ExtendedSequenceNumber checkpointSequenceNumber = new ExtendedSequenceNumber("123");
         final ExtendedSequenceNumber pendingCheckpointSequenceNumber = null;
+        when(streamProxy.getIterator(anyString(), anyString(), anyString())).thenReturn("startingIterator");
         when(leaseManager.getLease(anyString())).thenReturn(null);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         when(checkpoint.getCheckpointObject(anyString())).thenReturn(
@@ -473,6 +478,8 @@ public class ShardConsumerTest {
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         when(leaseManager.getLease(eq(parentShardId))).thenReturn(parentLease);
         when(parentLease.getCheckpoint()).thenReturn(ExtendedSequenceNumber.TRIM_HORIZON);
+        when(recordProcessorCheckpointer.getLastCheckpointValue()).thenReturn(ExtendedSequenceNumber.SHARD_END);
+        when(streamConfig.getStreamProxy()).thenReturn(streamProxy);
 
         final ShardConsumer consumer =
                 new ShardConsumer(shardInfo,
@@ -505,6 +512,9 @@ public class ShardConsumerTest {
         assertThat(consumer.getShutdownReason(), equalTo(ShutdownReason.REQUESTED));
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.WAITING_ON_PARENT_SHARDS)));
         consumer.consumeShard();
+        assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.SHUTTING_DOWN)));
+        Thread.sleep(50L);
+        consumer.beginShutdown();
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.SHUTDOWN_COMPLETE)));
         assertThat(consumer.isShutdown(), is(true));
         verify(shutdownNotification, times(1)).shutdownComplete();
@@ -538,7 +548,7 @@ public class ShardConsumerTest {
         int numRecs = 10;
         BigInteger startSeqNum = BigInteger.ONE;
         String streamShardId = "kinesis-0-0";
-        String testConcurrencyToken = "testToken";
+        String testConcurrencyToken = UUID.randomUUID().toString();
         List<Shard> shardList = KinesisLocalFileDataCreator.createShardList(1, "kinesis-0-", startSeqNum);
         // Close the shard so that shutdown is called with reason terminate
         shardList.get(0).getSequenceNumberRange().setEndingSequenceNumber(
@@ -573,7 +583,12 @@ public class ShardConsumerTest {
                 .thenReturn(getRecordsCache);
         when(leaseManager.getLease(anyString())).thenReturn(null);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
-        when(leaseCoordinator.getCurrentlyHeldLease(shardInfo.getShardId())).thenReturn(new KinesisClientLease());
+        List<String> parentShardIds = new ArrayList<>();
+        parentShardIds.add("parentShardId");
+        KinesisClientLease currentLease = createLease(streamShardId, "leaseOwner", parentShardIds);
+        currentLease.setCheckpoint(new ExtendedSequenceNumber("testSequenceNumbeer"));
+        when(leaseManager.getLease(streamShardId)).thenReturn(currentLease);
+        when(leaseCoordinator.getCurrentlyHeldLease(shardInfo.getShardId())).thenReturn(currentLease);
 
         RecordProcessorCheckpointer recordProcessorCheckpointer = new RecordProcessorCheckpointer(
                 shardInfo,
@@ -606,8 +621,7 @@ public class ShardConsumerTest {
                         shardSyncer,
                         shardSyncStrategy);
 
-        when(shardSyncStrategy.onShardConsumerShutDown(shardList)).thenReturn(new TaskResult(null));
+        when(leaseCoordinator.updateLease(any(KinesisClientLease.class), any(UUID.class))).thenReturn(true);
-
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.WAITING_ON_PARENT_SHARDS)));
         consumer.consumeShard(); // check on parent shards
         Thread.sleep(50L);
@@ -657,7 +671,7 @@ public class ShardConsumerTest {
         }
         assertThat(consumer.getCurrentState(), equalTo(ConsumerStates.ShardConsumerState.SHUTDOWN_COMPLETE));
 
-        assertThat(processor.getShutdownReason(), is(equalTo(ShutdownReason.ZOMBIE)));
+        assertThat(processor.getShutdownReason(), is(equalTo(ShutdownReason.TERMINATE)));
 
         verify(getRecordsCache).shutdown();
 
@@ -681,7 +695,7 @@ public class ShardConsumerTest {
         int numRecs = 10;
         BigInteger startSeqNum = BigInteger.ONE;
         String streamShardId = "kinesis-0-0";
-        String testConcurrencyToken = "testToken";
+        String testConcurrencyToken = UUID.randomUUID().toString();
         List<Shard> shardList = KinesisLocalFileDataCreator.createShardList(3, "kinesis-0-", startSeqNum);
         // Close the shard so that shutdown is called with reason terminate
         shardList.get(0).getSequenceNumberRange().setEndingSequenceNumber(
@@ -696,7 +710,11 @@ public class ShardConsumerTest {
         final int idleTimeMS = 0; // keep unit tests fast
         ICheckpoint checkpoint = new InMemoryCheckpointImpl(startSeqNum.toString());
         checkpoint.setCheckpoint(streamShardId, ExtendedSequenceNumber.TRIM_HORIZON, testConcurrencyToken);
-        when(leaseManager.getLease(anyString())).thenReturn(null);
+        List<String> parentShardIds = new ArrayList<>();
+        parentShardIds.add("parentShardId");
+        KinesisClientLease currentLease = createLease(streamShardId, "leaseOwner", parentShardIds);
+        currentLease.setCheckpoint(new ExtendedSequenceNumber("testSequenceNumbeer"));
+        when(leaseManager.getLease(streamShardId)).thenReturn(currentLease);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
 
         TransientShutdownErrorTestStreamlet processor = new TransientShutdownErrorTestStreamlet();
@@ -749,7 +767,8 @@ public class ShardConsumerTest {
                         shardSyncer,
                         shardSyncStrategy);
 
-        when(shardSyncStrategy.onShardConsumerShutDown(shardList)).thenReturn(new TaskResult(null));
+        when(leaseCoordinator.getCurrentlyHeldLease(shardInfo.getShardId())).thenReturn(currentLease);
+        when(leaseCoordinator.updateLease(any(KinesisClientLease.class), any(UUID.class))).thenReturn(true);
 
         assertThat(consumer.getCurrentState(), is(equalTo(ConsumerStates.ShardConsumerState.WAITING_ON_PARENT_SHARDS)));
         consumer.consumeShard(); // check on parent shards
@@ -939,7 +958,7 @@ public class ShardConsumerTest {
     @SuppressWarnings("unchecked")
     @Test
     public final void testConsumeShardInitializedWithPendingCheckpoint() throws Exception {
-        ShardInfo shardInfo = new ShardInfo("s-0-0", "testToken", null, ExtendedSequenceNumber.TRIM_HORIZON);
+        ShardInfo shardInfo = new ShardInfo("s-0-0", UUID.randomUUID().toString(), null, ExtendedSequenceNumber.TRIM_HORIZON);
         StreamConfig streamConfig =
                 new StreamConfig(streamProxy,
                         1,
@@ -967,6 +986,7 @@ public class ShardConsumerTest {
 
         final ExtendedSequenceNumber checkpointSequenceNumber = new ExtendedSequenceNumber("123");
         final ExtendedSequenceNumber pendingCheckpointSequenceNumber = new ExtendedSequenceNumber("999");
+        when(streamProxy.getIterator(anyString(), anyString(), anyString())).thenReturn("startingIterator");
         when(leaseManager.getLease(anyString())).thenReturn(null);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         when(config.getRecordsFetcherFactory()).thenReturn(new SimpleRecordsFetcherFactory());
@@ -1125,6 +1145,14 @@ public class ShardConsumerTest {
         return userRecords;
     }
 
+    private KinesisClientLease createLease(String leaseKey, String leaseOwner, Collection<String> parentShardIds) {
+        KinesisClientLease lease = new KinesisClientLease();
+        lease.setLeaseKey(leaseKey);
+        lease.setLeaseOwner(leaseOwner);
+        lease.setParentShardIds(parentShardIds);
+        return lease;
+    }
+
     Matcher<InitializationInput> initializationInputMatcher(final ExtendedSequenceNumber checkpoint,
                                                             final ExtendedSequenceNumber pendingCheckpoint) {
         return new TypeSafeMatcher<InitializationInput>() {

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardObjectHelper.java

@@ -25,24 +25,24 @@ import com.amazonaws.services.kinesis.model.Shard;
 /**
  * Helper class to create Shard, SequenceRange and related objects.
  */
-class ShardObjectHelper {
+public class ShardObjectHelper {
 
     private static final int EXPONENT = 128;
     
     /**
      * Max value of a sequence number (2^128 -1). Useful for defining sequence number range for a shard.
      */
-    static final String MAX_SEQUENCE_NUMBER = new BigInteger("2").pow(EXPONENT).subtract(BigInteger.ONE).toString();
+    public static final String MAX_SEQUENCE_NUMBER = new BigInteger("2").pow(EXPONENT).subtract(BigInteger.ONE).toString();
     
     /**
      * Min value of a sequence number (0). Useful for defining sequence number range for a shard.
      */
-    static final String MIN_SEQUENCE_NUMBER = BigInteger.ZERO.toString();
+    public static final String MIN_SEQUENCE_NUMBER = BigInteger.ZERO.toString();
 
     /**
      * Max value of a hash key (2^128 -1). Useful for defining hash key range for a shard.
      */
-    static final String MAX_HASH_KEY = new BigInteger("2").pow(EXPONENT).subtract(BigInteger.ONE).toString();
+    public static final String MAX_HASH_KEY = new BigInteger("2").pow(EXPONENT).subtract(BigInteger.ONE).toString();
     
     /**
      * Min value of a hash key (0). Useful for defining sequence number range for a shard.
@@ -63,7 +63,7 @@ class ShardObjectHelper {
      * @param sequenceNumberRange
      * @return
      */
-    static Shard newShard(String shardId,
+    public static Shard newShard(String shardId,
             String parentShardId,
             String adjacentParentShardId,
             SequenceNumberRange sequenceNumberRange) {
@@ -78,7 +78,7 @@ class ShardObjectHelper {
      * @param hashKeyRange
      * @return
      */
-    static Shard newShard(String shardId,
+    public static Shard newShard(String shardId,
             String parentShardId,
             String adjacentParentShardId,
             SequenceNumberRange sequenceNumberRange,
@@ -98,7 +98,7 @@ class ShardObjectHelper {
      * @param endingSequenceNumber
      * @return
      */
-    static SequenceNumberRange newSequenceNumberRange(String startingSequenceNumber, String endingSequenceNumber) {
+    public static SequenceNumberRange newSequenceNumberRange(String startingSequenceNumber, String endingSequenceNumber) {
         SequenceNumberRange range = new SequenceNumberRange();
         range.setStartingSequenceNumber(startingSequenceNumber);
         range.setEndingSequenceNumber(endingSequenceNumber);
@@ -110,14 +110,14 @@ class ShardObjectHelper {
      * @param endingHashKey
      * @return
      */
-    static HashKeyRange newHashKeyRange(String startingHashKey, String endingHashKey) {
+    public static HashKeyRange newHashKeyRange(String startingHashKey, String endingHashKey) {
         HashKeyRange range = new HashKeyRange();
         range.setStartingHashKey(startingHashKey);
         range.setEndingHashKey(endingHashKey);
         return range;
     }
     
-    static List<String> getParentShardIds(Shard shard) {
+    public static List<String> getParentShardIds(Shard shard) {
         List<String> parentShardIds = new ArrayList<>(2);
         if (shard.getAdjacentParentShardId() != null) {
             parentShardIds.add(shard.getAdjacentParentShardId());

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShutdownTaskTest.java

@@ -14,21 +14,37 @@
  */
 package com.amazonaws.services.kinesis.clientlibrary.lib.worker;
 
+import static junit.framework.TestCase.assertTrue;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
 import static org.mockito.Matchers.any;
 import static org.mockito.Matchers.anyString;
 import static org.mockito.Mockito.doNothing;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.spy;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
 
 import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
 import java.util.HashSet;
 import java.util.List;
 import java.util.Set;
+import java.util.UUID;
 
+import com.amazonaws.services.kinesis.clientlibrary.exceptions.internal.BlockedOnParentShardException;
 import com.amazonaws.services.kinesis.clientlibrary.proxies.ShardListWrappingShardClosureVerificationResponse;
+import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownInput;
+import com.amazonaws.services.kinesis.leases.exceptions.CustomerApplicationException;
+import com.amazonaws.services.kinesis.leases.exceptions.InvalidStateException;
+import com.amazonaws.services.kinesis.leases.impl.UpdateField;
+import com.amazonaws.services.kinesis.leases.impl.LeaseCleanupManager;
+import com.amazonaws.services.kinesis.metrics.impl.NullMetricsFactory;
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+import com.amazonaws.services.kinesis.model.ChildShard;
 import com.amazonaws.services.kinesis.model.HashKeyRange;
 import com.amazonaws.services.kinesis.model.SequenceNumberRange;
 import com.amazonaws.services.kinesis.model.Shard;
@@ -49,6 +65,7 @@ import com.amazonaws.services.kinesis.leases.interfaces.ILeaseManager;
 import org.junit.runner.RunWith;
 import org.mockito.Mock;
 import org.mockito.runners.MockitoJUnitRunner;
+import static com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask.RETRY_RANDOM_MAX_RANGE;
 
 /**
  *
@@ -60,20 +77,30 @@ public class ShutdownTaskTest {
             InitialPositionInStreamExtended.newInitialPosition(InitialPositionInStream.TRIM_HORIZON);
 
     Set<String> defaultParentShardIds = new HashSet<>();
-    String defaultConcurrencyToken = "testToken4398";
+    String defaultConcurrencyToken = UUID.randomUUID().toString();
     String defaultShardId = "shardId-0";
     ShardInfo defaultShardInfo = new ShardInfo(defaultShardId,
             defaultConcurrencyToken,
             defaultParentShardIds,
             ExtendedSequenceNumber.LATEST);
-    IRecordProcessor defaultRecordProcessor = new TestStreamlet();
     ShardSyncer shardSyncer = new KinesisShardSyncer(new KinesisLeaseCleanupValidator());
+    IMetricsFactory metricsFactory = new NullMetricsFactory();
 
 
+    @Mock
+    private IKinesisProxy kinesisProxy;
     @Mock
     private GetRecordsCache getRecordsCache;
     @Mock
     private ShardSyncStrategy shardSyncStrategy;
+    @Mock
+    private ILeaseManager<KinesisClientLease> leaseManager;
+    @Mock
+    private KinesisClientLibLeaseCoordinator leaseCoordinator;
+    @Mock
+    private IRecordProcessor defaultRecordProcessor;
+    @Mock
+    private LeaseCleanupManager leaseCleanupManager;
 
     /**
      * @throws java.lang.Exception
@@ -95,6 +122,12 @@ public class ShutdownTaskTest {
     @Before
     public void setUp() throws Exception {
         doNothing().when(getRecordsCache).shutdown();
+        final KinesisClientLease parentLease = createLease(defaultShardId, "leaseOwner", Collections.emptyList());
+        parentLease.setCheckpoint(new ExtendedSequenceNumber("3298"));
+        when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
+        when(leaseCoordinator.getCurrentlyHeldLease(defaultShardId)).thenReturn(parentLease);
+        when(leaseManager.getLease(defaultShardId)).thenReturn(parentLease);
+
     }
 
     /**
@@ -111,12 +144,6 @@ public class ShutdownTaskTest {
     public final void testCallWhenApplicationDoesNotCheckpoint() {
         RecordProcessorCheckpointer checkpointer = mock(RecordProcessorCheckpointer.class);
         when(checkpointer.getLastCheckpointValue()).thenReturn(new ExtendedSequenceNumber("3298"));
-        IKinesisProxy kinesisProxy = mock(IKinesisProxy.class);
-        List<Shard> shards = constructShardListForGraphA();
-        when(kinesisProxy.getShardList()).thenReturn(shards);
-        when(kinesisProxy.verifyShardClosure(anyString())).thenReturn(new ShardListWrappingShardClosureVerificationResponse(true, shards));
-        KinesisClientLibLeaseCoordinator leaseCoordinator = mock(KinesisClientLibLeaseCoordinator.class);
-        ILeaseManager<KinesisClientLease> leaseManager = mock(KinesisClientLeaseManager.class);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         boolean cleanupLeasesOfCompletedShards = false;
         boolean ignoreUnexpectedChildShards = false;
@@ -132,31 +159,29 @@ public class ShutdownTaskTest {
                 TASK_BACKOFF_TIME_MILLIS,
                 getRecordsCache,
                 shardSyncer,
-                shardSyncStrategy);
+                shardSyncStrategy,
+                constructSplitChildShards(),
+                leaseCleanupManager);
         TaskResult result = task.call();
-        Assert.assertNotNull(result.getException());
+        assertNotNull(result.getException());
-        Assert.assertTrue(result.getException() instanceof IllegalArgumentException);
+        Assert.assertTrue(result.getException() instanceof CustomerApplicationException);
+        final String expectedExceptionMessage = "Customer application throws exception for shard shardId-0";
+        Assert.assertEquals(expectedExceptionMessage, result.getException().getMessage());
     }
 
     /**
      * Test method for {@link ShutdownTask#call()}.
      */
     @Test
-    public final void testCallWhenSyncingShardsThrows() {
+    public final void testCallWhenCreatingLeaseThrows() throws Exception {
         RecordProcessorCheckpointer checkpointer = mock(RecordProcessorCheckpointer.class);
         when(checkpointer.getLastCheckpointValue()).thenReturn(ExtendedSequenceNumber.SHARD_END);
-        List<Shard> shards = constructShardListForGraphA();
-        IKinesisProxy kinesisProxy = mock(IKinesisProxy.class);
-        when(kinesisProxy.getShardList()).thenReturn(shards);
-        when(kinesisProxy.verifyShardClosure(anyString())).thenReturn(new ShardListWrappingShardClosureVerificationResponse(true, shards));
-        KinesisClientLibLeaseCoordinator leaseCoordinator = mock(KinesisClientLibLeaseCoordinator.class);
-        ILeaseManager<KinesisClientLease> leaseManager = mock(KinesisClientLeaseManager.class);
-        when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         boolean cleanupLeasesOfCompletedShards = false;
         boolean ignoreUnexpectedChildShards = false;
 
-        when(shardSyncStrategy.onShardConsumerShutDown(shards)).thenReturn(new TaskResult(new KinesisClientLibIOException("")));
+        final String exceptionMessage = "InvalidStateException is thrown.";
+        when(leaseManager.createLeaseIfNotExists(any(KinesisClientLease.class))).thenThrow(new InvalidStateException(exceptionMessage));
         ShutdownTask task = new ShutdownTask(defaultShardInfo,
                 defaultRecordProcessor,
                 checkpointer,
@@ -169,30 +194,152 @@ public class ShutdownTaskTest {
                 TASK_BACKOFF_TIME_MILLIS,
                 getRecordsCache,
                 shardSyncer,
-                shardSyncStrategy);
+                shardSyncStrategy,
+                constructSplitChildShards(),
+                leaseCleanupManager);
         TaskResult result = task.call();
-        verify(shardSyncStrategy).onShardConsumerShutDown(shards);
-        Assert.assertNotNull(result.getException());
-        Assert.assertTrue(result.getException() instanceof KinesisClientLibIOException);
         verify(getRecordsCache).shutdown();
+        verify(leaseCoordinator).dropLease(any(KinesisClientLease.class));
+        Assert.assertNull(result.getException());
     }
 
     @Test
-    public final void testCallWhenShardEnd() {
+    public final void testCallWhenParentInfoNotPresentInLease() throws Exception {
         RecordProcessorCheckpointer checkpointer = mock(RecordProcessorCheckpointer.class);
         when(checkpointer.getLastCheckpointValue()).thenReturn(ExtendedSequenceNumber.SHARD_END);
-        List<Shard> shards = constructShardListForGraphA();
-        IKinesisProxy kinesisProxy = mock(IKinesisProxy.class);
-        when(kinesisProxy.getShardList()).thenReturn(shards);
-        when(kinesisProxy.verifyShardClosure(anyString())).thenReturn(new ShardListWrappingShardClosureVerificationResponse(true, shards));
-        KinesisClientLibLeaseCoordinator leaseCoordinator = mock(KinesisClientLibLeaseCoordinator.class);
-        ILeaseManager<KinesisClientLease> leaseManager = mock(KinesisClientLeaseManager.class);
-        when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         boolean cleanupLeasesOfCompletedShards = false;
         boolean ignoreUnexpectedChildShards = false;
 
-        when(shardSyncStrategy.onShardConsumerShutDown(shards)).thenReturn(new TaskResult(null));
+        KinesisClientLease currentLease = createLease(defaultShardId, "leaseOwner", Collections.emptyList());
+        currentLease.setCheckpoint(new ExtendedSequenceNumber("3298"));
+        KinesisClientLease adjacentParentLease = createLease("ShardId-1", "leaseOwner", Collections.emptyList());
+        when(leaseCoordinator.getCurrentlyHeldLease(defaultShardId)).thenReturn( currentLease);
+        when(leaseManager.getLease(defaultShardId)).thenReturn(currentLease);
+        when(leaseManager.getLease("ShardId-1")).thenReturn(null, null, null, null, null, adjacentParentLease);
+
+        // Make first 5 attempts with partial parent info in lease table
+        for (int i = 0; i < 5; i++) {
+            ShutdownTask task = spy(new ShutdownTask(defaultShardInfo,
+                                                     defaultRecordProcessor,
+                                                     checkpointer,
+                                                     ShutdownReason.TERMINATE,
+                                                     kinesisProxy,
+                                                     INITIAL_POSITION_TRIM_HORIZON,
+                                                     cleanupLeasesOfCompletedShards,
+                                                     ignoreUnexpectedChildShards,
+                                                     leaseCoordinator,
+                                                     TASK_BACKOFF_TIME_MILLIS,
+                                                     getRecordsCache,
+                                                     shardSyncer,
+                                                     shardSyncStrategy,
+                                                     constructMergeChildShards(),
+                                                     leaseCleanupManager));
+            when(task.isOneInNProbability(RETRY_RANDOM_MAX_RANGE)).thenReturn(false);
+            TaskResult result = task.call();
+            assertNotNull(result.getException());
+            assertTrue(result.getException() instanceof BlockedOnParentShardException);
+            assertTrue(result.getException().getMessage().contains("has partial parent information in lease table"));
+            verify(task, times(1)).isOneInNProbability(RETRY_RANDOM_MAX_RANGE);
+            verify(getRecordsCache, never()).shutdown();
+            verify(defaultRecordProcessor, never()).shutdown(any(ShutdownInput.class));
+        }
+
+        // Make next attempt with complete parent info in lease table
+        ShutdownTask task = spy(new ShutdownTask(defaultShardInfo,
+                                                 defaultRecordProcessor,
+                                                 checkpointer,
+                                                 ShutdownReason.TERMINATE,
+                                                 kinesisProxy,
+                                                 INITIAL_POSITION_TRIM_HORIZON,
+                                                 cleanupLeasesOfCompletedShards,
+                                                 ignoreUnexpectedChildShards,
+                                                 leaseCoordinator,
+                                                 TASK_BACKOFF_TIME_MILLIS,
+                                                 getRecordsCache,
+                                                 shardSyncer,
+                                                 shardSyncStrategy,
+                                                 constructMergeChildShards(),
+                                                 leaseCleanupManager));
+        when(task.isOneInNProbability(RETRY_RANDOM_MAX_RANGE)).thenReturn(false);
+        TaskResult result = task.call();
+        assertNull(result.getException());
+        verify(task, never()).isOneInNProbability(RETRY_RANDOM_MAX_RANGE);
+        verify(getRecordsCache).shutdown();
+        verify(defaultRecordProcessor).shutdown(any(ShutdownInput.class));
+        verify(leaseCoordinator, never()).dropLease(currentLease);
+    }
+
+    @Test
+    public final void testCallTriggersLeaseLossWhenParentInfoNotPresentInLease() throws Exception {
+        RecordProcessorCheckpointer checkpointer = mock(RecordProcessorCheckpointer.class);
+        when(checkpointer.getLastCheckpointValue()).thenReturn(ExtendedSequenceNumber.SHARD_END);
+        when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
+        boolean cleanupLeasesOfCompletedShards = false;
+        boolean ignoreUnexpectedChildShards = false;
+
+        KinesisClientLease currentLease = createLease(defaultShardId, "leaseOwner", Collections.emptyList());
+        when(leaseCoordinator.getCurrentlyHeldLease(defaultShardId)).thenReturn( currentLease);
+        when(leaseManager.getLease(defaultShardId)).thenReturn(currentLease);
+        when(leaseManager.getLease("ShardId-1")).thenReturn(null, null, null, null, null, null, null, null, null, null, null);
+
+        for (int i = 0; i < 10; i++) {
+            ShutdownTask task = spy(new ShutdownTask(defaultShardInfo,
+                                                     defaultRecordProcessor,
+                                                     checkpointer,
+                                                     ShutdownReason.TERMINATE,
+                                                     kinesisProxy,
+                                                     INITIAL_POSITION_TRIM_HORIZON,
+                                                     cleanupLeasesOfCompletedShards,
+                                                     ignoreUnexpectedChildShards,
+                                                     leaseCoordinator,
+                                                     TASK_BACKOFF_TIME_MILLIS,
+                                                     getRecordsCache,
+                                                     shardSyncer,
+                                                     shardSyncStrategy,
+                                                     constructMergeChildShards(),
+                                                     leaseCleanupManager));
+            when(task.isOneInNProbability(RETRY_RANDOM_MAX_RANGE)).thenReturn(false);
+            TaskResult result = task.call();
+            assertNotNull(result.getException());
+            assertTrue(result.getException() instanceof BlockedOnParentShardException);
+            assertTrue(result.getException().getMessage().contains("has partial parent information in lease table"));
+            verify(task, times(1)).isOneInNProbability(RETRY_RANDOM_MAX_RANGE);
+            verify(getRecordsCache, never()).shutdown();
+            verify(defaultRecordProcessor, never()).shutdown(any(ShutdownInput.class));
+        }
+
+        ShutdownTask task = spy(new ShutdownTask(defaultShardInfo,
+                                                 defaultRecordProcessor,
+                                                 checkpointer,
+                                                 ShutdownReason.TERMINATE,
+                                                 kinesisProxy,
+                                                 INITIAL_POSITION_TRIM_HORIZON,
+                                                 cleanupLeasesOfCompletedShards,
+                                                 ignoreUnexpectedChildShards,
+                                                 leaseCoordinator,
+                                                 TASK_BACKOFF_TIME_MILLIS,
+                                                 getRecordsCache,
+                                                 shardSyncer,
+                                                 shardSyncStrategy,
+                                                 constructMergeChildShards(),
+                                                 leaseCleanupManager));
+        when(task.isOneInNProbability(RETRY_RANDOM_MAX_RANGE)).thenReturn(true);
+        TaskResult result = task.call();
+        assertNull(result.getException());
+        verify(task, times(1)).isOneInNProbability(RETRY_RANDOM_MAX_RANGE);
+        verify(getRecordsCache).shutdown();
+        verify(defaultRecordProcessor).shutdown(any(ShutdownInput.class));
+        verify(leaseCoordinator).dropLease(currentLease);
+    }
+
+    @Test
+    public final void testCallWhenShardEnd() throws Exception {
+        RecordProcessorCheckpointer checkpointer = mock(RecordProcessorCheckpointer.class);
+        when(checkpointer.getLastCheckpointValue()).thenReturn(ExtendedSequenceNumber.SHARD_END);
+        boolean cleanupLeasesOfCompletedShards = false;
+        boolean ignoreUnexpectedChildShards = false;
+
         ShutdownTask task = new ShutdownTask(defaultShardInfo,
                                              defaultRecordProcessor,
                                              checkpointer,
@@ -205,36 +352,28 @@ public class ShutdownTaskTest {
                                              TASK_BACKOFF_TIME_MILLIS,
                                              getRecordsCache,
                                              shardSyncer,
-                                             shardSyncStrategy);
+                                             shardSyncStrategy,
+                                             constructSplitChildShards(),
+                                             leaseCleanupManager);
         TaskResult result = task.call();
-        verify(shardSyncStrategy).onShardConsumerShutDown(shards);
+        verify(leaseManager, times(2)).createLeaseIfNotExists(any(KinesisClientLease.class));
-        verify(kinesisProxy, times(1)).verifyShardClosure(anyString());
+        verify(leaseManager).updateLeaseWithMetaInfo(any(KinesisClientLease.class), any(UpdateField.class));
         Assert.assertNull(result.getException());
         verify(getRecordsCache).shutdown();
-        verify(leaseCoordinator, never()).dropLease(any());
     }
 
     @Test
-    public final void testCallWhenFalseShardEnd() {
+    public final void testCallWhenShardNotFound() throws Exception {
         ShardInfo shardInfo = new ShardInfo("shardId-4",
                                                    defaultConcurrencyToken,
                                                    defaultParentShardIds,
                                                    ExtendedSequenceNumber.LATEST);
         RecordProcessorCheckpointer checkpointer = mock(RecordProcessorCheckpointer.class);
         when(checkpointer.getLastCheckpointValue()).thenReturn(ExtendedSequenceNumber.SHARD_END);
-        List<Shard> shards = constructShardListForGraphA();
-        IKinesisProxy kinesisProxy = mock(IKinesisProxy.class);
-        when(kinesisProxy.getShardList()).thenReturn(shards);
-        when(kinesisProxy.verifyShardClosure(anyString())).thenReturn(new ShardListWrappingShardClosureVerificationResponse(false, shards));
-        KinesisClientLibLeaseCoordinator leaseCoordinator = mock(KinesisClientLibLeaseCoordinator.class);
-        ILeaseManager<KinesisClientLease> leaseManager = mock(KinesisClientLeaseManager.class);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
-        when(leaseCoordinator.getCurrentlyHeldLease(shardInfo.getShardId())).thenReturn(new KinesisClientLease());
         boolean cleanupLeasesOfCompletedShards = false;
         boolean ignoreUnexpectedChildShards = false;
 
-        when(shardSyncStrategy.onShardConsumerShutDown(shards)).thenReturn(new TaskResult(null));
-
         ShutdownTask task = new ShutdownTask(shardInfo,
                                              defaultRecordProcessor,
                                              checkpointer,
@@ -247,31 +386,24 @@ public class ShutdownTaskTest {
                                              TASK_BACKOFF_TIME_MILLIS,
                                              getRecordsCache,
                                              shardSyncer,
-                                             shardSyncStrategy);
+                                             shardSyncStrategy,
+                                             Collections.emptyList(),
+                                             leaseCleanupManager);
         TaskResult result = task.call();
-        verify(shardSyncStrategy, never()).onShardConsumerShutDown(shards);
+        verify(leaseManager, never()).createLeaseIfNotExists(any(KinesisClientLease.class));
-        verify(kinesisProxy, times(1)).verifyShardClosure(anyString());
+        verify(leaseManager, never()).updateLeaseWithMetaInfo(any(KinesisClientLease.class), any(UpdateField.class));
         Assert.assertNull(result.getException());
         verify(getRecordsCache).shutdown();
-        verify(leaseCoordinator).dropLease(any());
     }
 
     @Test
-    public final void testCallWhenLeaseLost() {
+    public final void testCallWhenLeaseLost() throws Exception {
         RecordProcessorCheckpointer checkpointer = mock(RecordProcessorCheckpointer.class);
         when(checkpointer.getLastCheckpointValue()).thenReturn(new ExtendedSequenceNumber("3298"));
-        List<Shard> shards = constructShardListForGraphA();
-        IKinesisProxy kinesisProxy = mock(IKinesisProxy.class);
-        when(kinesisProxy.getShardList()).thenReturn(shards);
-        when(kinesisProxy.verifyShardClosure(anyString())).thenReturn(new ShardListWrappingShardClosureVerificationResponse(false, shards));
-        KinesisClientLibLeaseCoordinator leaseCoordinator = mock(KinesisClientLibLeaseCoordinator.class);
-        ILeaseManager<KinesisClientLease> leaseManager = mock(KinesisClientLeaseManager.class);
-        when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
         boolean cleanupLeasesOfCompletedShards = false;
         boolean ignoreUnexpectedChildShards = false;
 
-        when(shardSyncStrategy.onShardConsumerShutDown(shards)).thenReturn(new TaskResult(null));
         ShutdownTask task = new ShutdownTask(defaultShardInfo,
                                              defaultRecordProcessor,
                                              checkpointer,
@@ -284,13 +416,14 @@ public class ShutdownTaskTest {
                                              TASK_BACKOFF_TIME_MILLIS,
                                              getRecordsCache,
                                              shardSyncer,
-                                             shardSyncStrategy);
+                                             shardSyncStrategy,
+                                             Collections.emptyList(),
+                                             leaseCleanupManager);
         TaskResult result = task.call();
-        verify(shardSyncStrategy, never()).onShardConsumerShutDown(shards);
+        verify(leaseManager, never()).createLeaseIfNotExists(any(KinesisClientLease.class));
-        verify(kinesisProxy, never()).getShardList();
+        verify(leaseManager, never()).updateLeaseWithMetaInfo(any(KinesisClientLease.class), any(UpdateField.class));
         Assert.assertNull(result.getException());
         verify(getRecordsCache).shutdown();
-        verify(leaseCoordinator, never()).dropLease(any());
     }
 
     /**
@@ -299,10 +432,54 @@ public class ShutdownTaskTest {
     @Test
     public final void testGetTaskType() {
         KinesisClientLibLeaseCoordinator leaseCoordinator = mock(KinesisClientLibLeaseCoordinator.class);
-        ShutdownTask task = new ShutdownTask(null, null, null, null, null, null, false, false, leaseCoordinator, 0, getRecordsCache, shardSyncer, shardSyncStrategy);
+        ShutdownTask task = new ShutdownTask(null, null, null, null,
+                                             null, null, false,
+                                             false, leaseCoordinator, 0,
+                                             getRecordsCache, shardSyncer, shardSyncStrategy, Collections.emptyList(), leaseCleanupManager);
         Assert.assertEquals(TaskType.SHUTDOWN, task.getTaskType());
     }
 
+    private List<ChildShard> constructSplitChildShards() {
+        List<ChildShard> childShards = new ArrayList<>();
+        List<String> parentShards = new ArrayList<>();
+        parentShards.add(defaultShardId);
+
+        ChildShard leftChild = new ChildShard();
+        leftChild.setShardId("ShardId-1");
+        leftChild.setParentShards(parentShards);
+        leftChild.setHashKeyRange(ShardObjectHelper.newHashKeyRange("0", "49"));
+        childShards.add(leftChild);
+
+        ChildShard rightChild = new ChildShard();
+        rightChild.setShardId("ShardId-2");
+        rightChild.setParentShards(parentShards);
+        rightChild.setHashKeyRange(ShardObjectHelper.newHashKeyRange("50", "99"));
+        childShards.add(rightChild);
+        return childShards;
+    }
+
+    private List<ChildShard> constructMergeChildShards() {
+        List<ChildShard> childShards = new ArrayList<>();
+        List<String> parentShards = new ArrayList<>();
+        parentShards.add(defaultShardId);
+        parentShards.add("ShardId-1");
+
+        ChildShard childShard = new ChildShard();
+        childShard.setShardId("ShardId-2");
+        childShard.setParentShards(parentShards);
+        childShard.setHashKeyRange(ShardObjectHelper.newHashKeyRange("0", "99"));
+        childShards.add(childShard);
+
+        return childShards;
+    }
+
+    private KinesisClientLease createLease(String leaseKey, String leaseOwner, Collection<String> parentShardIds) {
+        KinesisClientLease lease = new KinesisClientLease();
+        lease.setLeaseKey(leaseKey);
+        lease.setLeaseOwner(leaseOwner);
+        lease.setParentShardIds(parentShardIds);
+        return lease;
+    }
 
     /*
      * Helper method to construct a shard list for graph A. Graph A is defined below.

src/test/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/WorkerTest.java

@@ -72,6 +72,7 @@ import com.amazonaws.services.kinesis.leases.impl.KinesisClientLeaseBuilder;
 import com.amazonaws.services.kinesis.leases.impl.KinesisClientLeaseManager;
 import com.amazonaws.services.kinesis.leases.impl.LeaseManager;
 import com.amazonaws.services.kinesis.leases.interfaces.LeaseSelector;
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsScope;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.hamcrest.Condition;
@@ -158,6 +159,7 @@ public class WorkerTest {
 
     private static final String KINESIS_SHARD_ID_FORMAT = "kinesis-0-0-%d";
     private static final String CONCURRENCY_TOKEN_FORMAT = "testToken-%d";
+    private static final String WORKER_ID = "workerId";
 
     private RecordsFetcherFactory recordsFetcherFactory;
     private KinesisClientLibConfiguration config;
@@ -172,6 +174,8 @@ public class WorkerTest {
     @Mock
     private IKinesisProxy proxy;
     @Mock
+    private StreamConfig streamConfig;
+    @Mock
     private WorkerThreadPoolExecutor executorService;
     @Mock
     private WorkerCWMetricsFactory cwMetricsFactory;
@@ -194,9 +198,12 @@ public class WorkerTest {
 
     @Before
     public void setup() {
-        config = spy(new KinesisClientLibConfiguration("app", null, null, null));
+        config = spy(new KinesisClientLibConfiguration("app", null, null, WORKER_ID));
+        config.withMaxInitializationAttempts(1);
         recordsFetcherFactory = spy(new SimpleRecordsFetcherFactory());
         when(config.getRecordsFetcherFactory()).thenReturn(recordsFetcherFactory);
+        when(leaseCoordinator.getLeaseManager()).thenReturn(mock(ILeaseManager.class));
+        when(streamConfig.getStreamProxy()).thenReturn(kinesisProxy);
     }
 
     // CHECKSTYLE:IGNORE AnonInnerLengthCheck FOR NEXT 50 LINES
@@ -244,7 +251,7 @@ public class WorkerTest {
     @Test
     public final void testGetStageName() {
         final String stageName = "testStageName";
-        config = new KinesisClientLibConfiguration(stageName, null, null, null);
+        config = new KinesisClientLibConfiguration(stageName, null, null, WORKER_ID);
         Worker worker = new Worker(v1RecordProcessorFactory, config);
         Assert.assertEquals(stageName, worker.getApplicationName());
     }
@@ -253,8 +260,7 @@ public class WorkerTest {
     public final void testCreateOrGetShardConsumer() {
         final String stageName = "testStageName";
         IRecordProcessorFactory streamletFactory = SAMPLE_RECORD_PROCESSOR_FACTORY_V2;
-        config = new KinesisClientLibConfiguration(stageName, null, null, null);
+        config = new KinesisClientLibConfiguration(stageName, null, null, WORKER_ID);
-        IKinesisProxy proxy = null;
         ICheckpoint checkpoint = null;
         int maxRecords = 1;
         int idleTimeInMilliseconds = 1000;
@@ -303,7 +309,6 @@ public class WorkerTest {
     public void testWorkerLoopWithCheckpoint() {
         final String stageName = "testStageName";
         IRecordProcessorFactory streamletFactory = SAMPLE_RECORD_PROCESSOR_FACTORY_V2;
-        IKinesisProxy proxy = null;
         ICheckpoint checkpoint = null;
         int maxRecords = 1;
         int idleTimeInMilliseconds = 1000;
@@ -372,8 +377,7 @@ public class WorkerTest {
     public final void testCleanupShardConsumers() {
         final String stageName = "testStageName";
         IRecordProcessorFactory streamletFactory = SAMPLE_RECORD_PROCESSOR_FACTORY_V2;
-        config = new KinesisClientLibConfiguration(stageName, null, null, null);
+        config = new KinesisClientLibConfiguration(stageName, null, null, WORKER_ID);
-        IKinesisProxy proxy = null;
         ICheckpoint checkpoint = null;
         int maxRecords = 1;
         int idleTimeInMilliseconds = 1000;
@@ -429,12 +433,14 @@ public class WorkerTest {
     }
 
     @Test
-    public final void testInitializationFailureWithRetries() {
+    public final void testInitializationFailureWithRetries() throws Exception {
         String stageName = "testInitializationWorker";
         IRecordProcessorFactory recordProcessorFactory = new TestStreamletFactory(null, null);
-        config = new KinesisClientLibConfiguration(stageName, null, null, null);
+        config = new KinesisClientLibConfiguration(stageName, null, null, WORKER_ID);
+        config.withMaxInitializationAttempts(2);
+
         int count = 0;
-        when(proxy.getShardList()).thenThrow(new RuntimeException(Integer.toString(count++)));
+        when(proxy.getShardListWithFilter(any())).thenThrow(new RuntimeException(Integer.toString(count++)));
         int maxRecords = 2;
         long idleTimeInMilliseconds = 1L;
         StreamConfig streamConfig =
@@ -443,6 +449,7 @@ public class WorkerTest {
                         idleTimeInMilliseconds,
                         callProcessRecordsForEmptyRecordList, skipCheckpointValidationValue, INITIAL_POSITION_LATEST);
         when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
+        when(leaseManager.isLeaseTableEmpty()).thenReturn(true);
         ExecutorService execService = Executors.newSingleThreadExecutor();
         long shardPollInterval = 0L;
         Worker worker =
@@ -465,6 +472,79 @@ public class WorkerTest {
         Assert.assertTrue(count > 0);
     }
 
+    @Test
+    public final void testInitializationWaitsWhenLeaseTableIsEmpty() throws Exception {
+        final String stageName = "testInitializationWorker";
+        when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
+        when(leaseManager.isLeaseTableEmpty()).thenReturn(true);
+
+        final int maxRecords = 2;
+        final long idleTimeInMilliseconds = 1L;
+        final StreamConfig streamConfig = new StreamConfig(proxy, maxRecords, idleTimeInMilliseconds,
+                callProcessRecordsForEmptyRecordList, skipCheckpointValidationValue, INITIAL_POSITION_LATEST);
+
+        final long shardPollInterval = 0L;
+        final Worker worker =
+                new Worker(stageName,
+                        v2RecordProcessorFactory,
+                        config,
+                        streamConfig, INITIAL_POSITION_TRIM_HORIZON,
+                        shardPollInterval,
+                        shardSyncIntervalMillis,
+                        cleanupLeasesUponShardCompletion,
+                        leaseCoordinator,
+                        leaseCoordinator,
+                        Executors.newSingleThreadExecutor(),
+                        nullMetricsFactory,
+                        taskBackoffTimeMillis,
+                        failoverTimeMillis,
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        shardPrioritization);
+
+        final long startTime = System.currentTimeMillis();
+        worker.shouldInitiateLeaseSync();
+        final long endTime = System.currentTimeMillis();
+
+        assertTrue(endTime - startTime > Worker.MIN_WAIT_TIME_FOR_LEASE_TABLE_CHECK_MILLIS);
+        assertTrue(endTime - startTime < Worker.MAX_WAIT_TIME_FOR_LEASE_TABLE_CHECK_MILLIS + Worker.LEASE_TABLE_CHECK_FREQUENCY_MILLIS);
+    }
+
+    @Test
+    public final void testInitializationDoesntWaitWhenLeaseTableIsNotEmpty() throws Exception {
+        final String stageName = "testInitializationWorker";
+        when(leaseCoordinator.getLeaseManager()).thenReturn(leaseManager);
+        when(leaseManager.isLeaseTableEmpty()).thenReturn(false);
+
+        final int maxRecords = 2;
+        final long idleTimeInMilliseconds = 1L;
+        final StreamConfig streamConfig = new StreamConfig(proxy, maxRecords, idleTimeInMilliseconds,
+                callProcessRecordsForEmptyRecordList, skipCheckpointValidationValue, INITIAL_POSITION_LATEST);
+
+        final long shardPollInterval = 0L;
+        final Worker worker =
+                new Worker(stageName,
+                        v2RecordProcessorFactory,
+                        config,
+                        streamConfig, INITIAL_POSITION_TRIM_HORIZON,
+                        shardPollInterval,
+                        shardSyncIntervalMillis,
+                        cleanupLeasesUponShardCompletion,
+                        leaseCoordinator,
+                        leaseCoordinator,
+                        Executors.newSingleThreadExecutor(),
+                        nullMetricsFactory,
+                        taskBackoffTimeMillis,
+                        failoverTimeMillis,
+                        KinesisClientLibConfiguration.DEFAULT_SKIP_SHARD_SYNC_AT_STARTUP_IF_LEASES_EXIST,
+                        shardPrioritization);
+
+        final long startTime = System.currentTimeMillis();
+        worker.shouldInitiateLeaseSync();
+        final long endTime = System.currentTimeMillis();
+
+        assertTrue(endTime - startTime < Worker.MIN_WAIT_TIME_FOR_LEASE_TABLE_CHECK_MILLIS);
+    }
+
     /**
      * Runs worker with threadPoolSize == numShards
      * Test method for {@link com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker#run()}.
@@ -536,7 +616,6 @@ public class WorkerTest {
 
     @Test
     public final void testWorkerShutsDownOwnedResources() throws Exception {
-
         final long failoverTimeMillis = 20L;
 
         // Make sure that worker thread is run before invoking shutdown.
@@ -576,6 +655,7 @@ public class WorkerTest {
 
         final ExecutorService executorService = mock(ThreadPoolExecutor.class);
         final CWMetricsFactory cwMetricsFactory = mock(CWMetricsFactory.class);
+        when(cwMetricsFactory.createMetrics()).thenReturn(mock(IMetricsScope.class));
         // Make sure that worker thread is run before invoking shutdown.
         final CountDownLatch workerStarted = new CountDownLatch(1);
         doAnswer(new Answer<Boolean>() {
@@ -787,7 +867,6 @@ public class WorkerTest {
 
 
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        StreamConfig streamConfig = mock(StreamConfig.class);
         IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
 
         ExtendedSequenceNumber checkpoint = new ExtendedSequenceNumber("123", 0L);
@@ -875,7 +954,6 @@ public class WorkerTest {
     public void testShutdownCallableNotAllowedTwice() throws Exception {
 
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        StreamConfig streamConfig = mock(StreamConfig.class);
         IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
 
         ExtendedSequenceNumber checkpoint = new ExtendedSequenceNumber("123", 0L);
@@ -940,7 +1018,6 @@ public class WorkerTest {
     public void testGracefulShutdownSingleFuture() throws Exception {
 
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        StreamConfig streamConfig = mock(StreamConfig.class);
         IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
 
         ExtendedSequenceNumber checkpoint = new ExtendedSequenceNumber("123", 0L);
@@ -1028,7 +1105,6 @@ public class WorkerTest {
 
 
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        StreamConfig streamConfig = mock(StreamConfig.class);
         IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
 
 
@@ -1111,7 +1187,6 @@ public class WorkerTest {
         when(completedLease.getParentShardIds()).thenReturn(Collections.singleton(parentShardId));
         when(completedLease.getCheckpoint()).thenReturn(ExtendedSequenceNumber.SHARD_END);
         when(completedLease.getConcurrencyToken()).thenReturn(UUID.randomUUID());
-        final StreamConfig streamConfig = mock(StreamConfig.class);
         final IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
         final List<KinesisClientLease> leases = Collections.singletonList(completedLease);
         final List<ShardInfo> currentAssignments = new ArrayList<>();
@@ -1159,7 +1234,6 @@ public class WorkerTest {
     public void testRequestShutdownWithLostLease() throws Exception {
 
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        StreamConfig streamConfig = mock(StreamConfig.class);
         IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
 
         ExtendedSequenceNumber checkpoint = new ExtendedSequenceNumber("123", 0L);
@@ -1272,7 +1346,6 @@ public class WorkerTest {
     public void testRequestShutdownWithAllLeasesLost() throws Exception {
 
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        StreamConfig streamConfig = mock(StreamConfig.class);
         IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
 
         ExtendedSequenceNumber checkpoint = new ExtendedSequenceNumber("123", 0L);
@@ -1390,7 +1463,6 @@ public class WorkerTest {
     public void testLeaseCancelledAfterShutdownRequest() throws Exception {
 
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        StreamConfig streamConfig = mock(StreamConfig.class);
         IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
 
         ExtendedSequenceNumber checkpoint = new ExtendedSequenceNumber("123", 0L);
@@ -1474,7 +1546,6 @@ public class WorkerTest {
     public void testEndOfShardAfterShutdownRequest() throws Exception {
 
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        StreamConfig streamConfig = mock(StreamConfig.class);
         IMetricsFactory metricsFactory = mock(IMetricsFactory.class);
 
         ExtendedSequenceNumber checkpoint = new ExtendedSequenceNumber("123", 0L);
@@ -1704,11 +1775,69 @@ public class WorkerTest {
         Assert.assertSame(leaseManager, worker.getLeaseCoordinator().getLeaseManager());
     }
 
+    @Test
+    public void testBuilderWithDefaultShardSyncStrategy() {
+        IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
+        Worker worker = new Worker.Builder()
+                .recordProcessorFactory(recordProcessorFactory)
+                .config(config)
+                .build();
+
+        Assert.assertNotNull(worker.getLeaderDecider());
+        Assert.assertNotNull(worker.getPeriodicShardSyncManager());
+    }
+
+    @Test
+    public void testBuilderWithPeriodicShardSyncStrategyAndDefaultLeaderDecider() {
+        IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
+        when(config.getShardSyncStrategyType()).thenReturn(ShardSyncStrategyType.PERIODIC);
+        Worker worker = new Worker.Builder()
+                .recordProcessorFactory(recordProcessorFactory)
+                .config(config)
+                .build();
+
+        Assert.assertNotNull(worker.getLeaderDecider());
+        Assert.assertNotNull(worker.getPeriodicShardSyncManager());
+    }
+
+    @Test
+    public void testBuilderWithPeriodicShardSyncStrategyAndCustomLeaderDecider() {
+        IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
+        when(config.getShardSyncStrategyType()).thenReturn(ShardSyncStrategyType.PERIODIC);
+
+        LeaderDecider leaderDecider = mock(LeaderDecider.class);
+        Worker worker = new Worker.Builder()
+                .recordProcessorFactory(recordProcessorFactory)
+                .config(config)
+                .leaderDecider(leaderDecider)
+                .build();
+
+        Assert.assertSame(leaderDecider, worker.getLeaderDecider());
+        Assert.assertNotNull(worker.getPeriodicShardSyncManager());
+    }
+
+    @Test
+    public void testCustomLeaderDeciderNotAllowedForShardEndShardSync() {
+        IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
+        when(config.getShardSyncStrategyType()).thenReturn(ShardSyncStrategyType.SHARD_END);
+
+        LeaderDecider leaderDecider = mock(LeaderDecider.class);
+        Worker worker = new Worker.Builder()
+                .recordProcessorFactory(recordProcessorFactory)
+                .config(config)
+                .leaderDecider(leaderDecider)
+                .build();
+
+        // Worker should override custom leaderDecider and use default instead
+        Assert.assertNotSame(leaderDecider, worker.getLeaderDecider());
+        Assert.assertNotNull(worker.getPeriodicShardSyncManager());
+    }
+
     @Test
     public void testBuilderSetRegionAndEndpointToClient() {
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
         final String endpoint = "TestEndpoint";
-        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, null)
+        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, WORKER_ID)
                 .withRegionName(Regions.US_WEST_2.getName())
                 .withKinesisEndpoint(endpoint)
                 .withDynamoDBEndpoint(endpoint);
@@ -1736,7 +1865,7 @@ public class WorkerTest {
     public void testBuilderSetRegionToClient() {
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
         String region = Regions.US_WEST_2.getName();
-        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, null)
+        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, WORKER_ID)
                 .withRegionName(region);
 
         Worker.Builder builder = new Worker.Builder();
@@ -1763,7 +1892,7 @@ public class WorkerTest {
     @Test
     public void testBuilderGenerateClients() {
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
-        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, null);
+        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, WORKER_ID);
         Worker.Builder builder = spy(new Worker.Builder().recordProcessorFactory(recordProcessorFactory).config(config));
         ArgumentCaptor<AwsClientBuilder> builderCaptor = ArgumentCaptor.forClass(AwsClientBuilder.class);
 
@@ -1789,7 +1918,7 @@ public class WorkerTest {
     public void testBuilderGenerateClientsWithRegion() {
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
         String region = Regions.US_WEST_2.getName();
-        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, null)
+        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, WORKER_ID)
                 .withRegionName(region);
         ArgumentCaptor<AwsClientBuilder> builderCaptor = ArgumentCaptor.forClass(AwsClientBuilder.class);
 
@@ -1809,7 +1938,7 @@ public class WorkerTest {
         IRecordProcessorFactory recordProcessorFactory = mock(IRecordProcessorFactory.class);
         String region = Regions.US_WEST_2.getName();
         String endpointUrl = "TestEndpoint";
-        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, null)
+        KinesisClientLibConfiguration config = new KinesisClientLibConfiguration("TestApp", null, null, WORKER_ID)
                 .withRegionName(region).withKinesisEndpoint(endpointUrl).withDynamoDBEndpoint(endpointUrl);
 
         Worker.Builder builder = spy(new Worker.Builder());
@@ -2055,7 +2184,8 @@ public class WorkerTest {
     private List<Shard> createShardListWithOneSplit() {
         List<Shard> shards = new ArrayList<Shard>();
         SequenceNumberRange range0 = ShardObjectHelper.newSequenceNumberRange("39428", "987324");
-        SequenceNumberRange range1 = ShardObjectHelper.newSequenceNumberRange("987325", null);
+        SequenceNumberRange range1 = ShardObjectHelper.newSequenceNumberRange("39428", "100000");
+        SequenceNumberRange range2 = ShardObjectHelper.newSequenceNumberRange("100001", "987324");
         HashKeyRange keyRange =
                 ShardObjectHelper.newHashKeyRange(ShardObjectHelper.MIN_HASH_KEY, ShardObjectHelper.MAX_HASH_KEY);
         Shard shard0 = ShardObjectHelper.newShard("shardId-0", null, null, range0, keyRange);
@@ -2064,6 +2194,9 @@ public class WorkerTest {
         Shard shard1 = ShardObjectHelper.newShard("shardId-1", "shardId-0", null, range1, keyRange);
         shards.add(shard1);
 
+        Shard shard2 = ShardObjectHelper.newShard("shardId-2", "shardId-0", null, range2, keyRange);
+        shards.add(shard2);
+
         return shards;
     }
 

src/test/java/com/amazonaws/services/kinesis/clientlibrary/proxies/KinesisLocalFileProxy.java

@@ -25,6 +25,7 @@ import java.nio.charset.Charset;
 import java.nio.charset.CharsetEncoder;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.Date;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -33,6 +34,8 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
+import com.amazonaws.services.kinesis.model.ChildShard;
+import com.amazonaws.services.kinesis.model.ShardFilter;
 import com.amazonaws.util.CollectionUtils;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.logging.Log;
@@ -387,14 +390,33 @@ public class KinesisLocalFileProxy implements IKinesisProxy {
              */
             response.setNextShardIterator(serializeIterator(iterator.shardId, lastRecordsSeqNo.add(BigInteger.ONE)
                     .toString()));
+            response.setChildShards(Collections.emptyList());
             LOG.debug("Returning a non null iterator for shard " + iterator.shardId);
         } else {
+            response.setChildShards(constructChildShards(iterator));
             LOG.info("Returning null iterator for shard " + iterator.shardId);
         }
 
         return response;
     }
 
+    private List<ChildShard> constructChildShards(IteratorInfo iterator) {
+        List<ChildShard> childShards = new ArrayList<>();
+        List<String> parentShards = new ArrayList<>();
+        parentShards.add(iterator.shardId);
+
+        ChildShard leftChild = new ChildShard();
+        leftChild.setShardId("shardId-1");
+        leftChild.setParentShards(parentShards);
+        childShards.add(leftChild);
+
+        ChildShard rightChild = new ChildShard();
+        rightChild.setShardId("shardId-2");
+        rightChild.setParentShards(parentShards);
+        childShards.add(rightChild);
+        return childShards;
+    }
+
     /**
      * {@inheritDoc}
      */
@@ -425,6 +447,16 @@ public class KinesisLocalFileProxy implements IKinesisProxy {
         return shards;
     }
 
+    /**
+     * {@inheritDoc}
+     */
+    @Override
+    public List<Shard> getShardListWithFilter(ShardFilter shardFilter) throws ResourceNotFoundException {
+        List<Shard> shards = new LinkedList<Shard>();
+        shards.addAll(shardList);
+        return shards;
+    }
+
     /**
      * {@inheritDoc}
      */

src/test/java/com/amazonaws/services/kinesis/leases/impl/KinesisClientLeaseBuilder.java

@@ -30,6 +30,8 @@ public class KinesisClientLeaseBuilder {
     private ExtendedSequenceNumber pendingCheckpoint;
     private Long ownerSwitchesSinceCheckpoint = 0L;
     private Set<String> parentShardIds  = new HashSet<>();
+    private Set<String> childShardIds = new HashSet<>();
+    private HashKeyRangeForLease hashKeyRangeForLease;
 
     public KinesisClientLeaseBuilder withLeaseKey(String leaseKey) {
         this.leaseKey = leaseKey;
@@ -76,8 +78,18 @@ public class KinesisClientLeaseBuilder {
         return this;
     }
 
+    public KinesisClientLeaseBuilder withChildShardIds(Set<String> childShardIds) {
+        this.childShardIds = childShardIds;
+        return this;
+    }
+
+    public KinesisClientLeaseBuilder withHashKeyRange(HashKeyRangeForLease hashKeyRangeForLease) {
+        this.hashKeyRangeForLease = hashKeyRangeForLease;
+        return this;
+    }
+
     public KinesisClientLease build() {
         return new KinesisClientLease(leaseKey, leaseOwner, leaseCounter, concurrencyToken, lastCounterIncrementNanos,
-                checkpoint, pendingCheckpoint, ownerSwitchesSinceCheckpoint, parentShardIds);
+                checkpoint, pendingCheckpoint, ownerSwitchesSinceCheckpoint, parentShardIds, childShardIds, hashKeyRangeForLease);
     }
 }

src/test/java/com/amazonaws/services/kinesis/leases/impl/LeaseCleanupManagerTest.java

@@ -0,0 +1,300 @@
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.amazonaws.services.kinesis.leases.impl;
+
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardInfo;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardObjectHelper;
+import com.amazonaws.services.kinesis.clientlibrary.proxies.IKinesisProxy;
+import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
+import com.amazonaws.services.kinesis.leases.LeasePendingDeletion;
+import com.amazonaws.services.kinesis.metrics.impl.NullMetricsFactory;
+import com.amazonaws.services.kinesis.metrics.interfaces.IMetricsFactory;
+import com.amazonaws.services.kinesis.model.ChildShard;
+import com.amazonaws.services.kinesis.model.GetRecordsResult;
+import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.stream.Collectors;
+
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@RunWith(MockitoJUnitRunner.class)
+public class LeaseCleanupManagerTest {
+
+    private ShardInfo shardInfo;
+    private String concurrencyToken = "1234";
+    private int maxRecords = 1;
+
+    private String getShardId = "getShardId";
+    private String splitParent = "splitParent";
+    private String mergeParent1 = "mergeParent-1";
+    private String mergeParent2 = "mergeParent-2";
+
+    private long leaseCleanupIntervalMillis = Duration.ofSeconds(1).toMillis();
+    private long completedLeaseCleanupIntervalMillis = Duration.ofSeconds(0).toMillis();
+    private long garbageLeaseCleanupIntervalMillis = Duration.ofSeconds(0).toMillis();
+    private boolean cleanupLeasesOfCompletedShards = true;
+    private LeaseCleanupManager leaseCleanupManager;
+    private static final IMetricsFactory NULL_METRICS_FACTORY = new NullMetricsFactory();
+
+    @Mock
+    private LeaseManager leaseManager;
+    @Mock
+    private LeaseCoordinator leaseCoordinator;
+    @Mock
+    private IKinesisProxy kinesis;
+    @Mock
+    private ScheduledExecutorService deletionThreadPool;
+
+    @Before
+    public void setUp() {
+        shardInfo = new ShardInfo(getShardId, concurrencyToken, Collections.emptySet(),
+                ExtendedSequenceNumber.LATEST);
+        leaseCleanupManager = new LeaseCleanupManager(kinesis, leaseManager, deletionThreadPool, NULL_METRICS_FACTORY,
+                cleanupLeasesOfCompletedShards, leaseCleanupIntervalMillis, completedLeaseCleanupIntervalMillis, garbageLeaseCleanupIntervalMillis, maxRecords);
+    }
+
+    /**
+     * Tests subsequent calls to start {@link LeaseCleanupManager}.
+     */
+    @Test
+    public final void testSubsequentStarts() {
+        leaseCleanupManager.start();
+        Assert.assertTrue(leaseCleanupManager.isRunning());
+        leaseCleanupManager.start();
+    }
+
+    /**
+     * Tests that when both child shard leases are present, we are able to delete the parent shard for the completed
+     * shard case.
+     */
+    @Test
+    public final void testParentShardLeaseDeletedSplitCase() throws Exception {
+        shardInfo = new ShardInfo("ShardId-0", concurrencyToken, Collections.emptySet(),
+                ExtendedSequenceNumber.LATEST);
+
+        verifyExpectedDeletedLeasesCompletedShardCase(shardInfo, childShardsForSplit(), ExtendedSequenceNumber.LATEST, 1);
+    }
+
+    /**
+     * Tests that when both child shard leases are present, we are able to delete the parent shard for the completed
+     * shard case.
+     */
+    @Test
+    public final void testParentShardLeaseDeletedMergeCase() throws Exception {
+        shardInfo = new ShardInfo("ShardId-0", concurrencyToken, Collections.emptySet(),
+                ExtendedSequenceNumber.LATEST);
+
+        verifyExpectedDeletedLeasesCompletedShardCase(shardInfo, childShardsForMerge(), ExtendedSequenceNumber.LATEST, 1);
+    }
+
+    /**
+     * Tests that if cleanupLeasesOfCompletedShards is not enabled by the customer, then no leases are cleaned up for
+     * the completed shard case.
+     */
+    @Test
+    public final void testNoLeasesDeletedWhenNotEnabled() throws Exception {
+        shardInfo = new ShardInfo("ShardId-0", concurrencyToken, Collections.emptySet(),
+                ExtendedSequenceNumber.LATEST);
+        cleanupLeasesOfCompletedShards = false;
+
+        leaseCleanupManager = new LeaseCleanupManager(kinesis, leaseManager, deletionThreadPool, NULL_METRICS_FACTORY,
+                cleanupLeasesOfCompletedShards, leaseCleanupIntervalMillis, completedLeaseCleanupIntervalMillis, garbageLeaseCleanupIntervalMillis, maxRecords);
+
+        verifyExpectedDeletedLeasesCompletedShardCase(shardInfo, childShardsForSplit(), ExtendedSequenceNumber.LATEST, 0);
+    }
+
+    /**
+     * Tests that if some of the child shard leases are missing, we fail fast and don't delete the parent shard lease
+     * for the completed shard case.
+     */
+    @Test
+    public final void testNoCleanupWhenSomeChildShardLeasesAreNotPresent() throws Exception {
+        List<ChildShard> childShards = childShardsForSplit();
+
+        shardInfo = new ShardInfo("ShardId-0", concurrencyToken, Collections.emptySet(),
+                ExtendedSequenceNumber.LATEST);
+
+        verifyExpectedDeletedLeasesCompletedShardCase(shardInfo, childShards, ExtendedSequenceNumber.LATEST, false, 0);
+    }
+
+    /**
+     * Tests that if some child shard leases haven't begun processing (at least one lease w/ checkpoint TRIM_HORIZON),
+     * we don't delete them for the completed shard case.
+     */
+    @Test
+    public final void testParentShardLeaseNotDeletedWhenChildIsAtTrim() throws Exception {
+        testParentShardLeaseNotDeletedWhenChildIsAtPosition(ExtendedSequenceNumber.TRIM_HORIZON);
+    }
+
+    /**
+     * Tests that if some child shard leases haven't begun processing (at least one lease w/ checkpoint AT_TIMESTAMP),
+     * we don't delete them for the completed shard case.
+     */
+    @Test
+    public final void testParentShardLeaseNotDeletedWhenChildIsAtTimestamp() throws Exception {
+        testParentShardLeaseNotDeletedWhenChildIsAtPosition(ExtendedSequenceNumber.AT_TIMESTAMP);
+    }
+
+    private  void testParentShardLeaseNotDeletedWhenChildIsAtPosition(ExtendedSequenceNumber extendedSequenceNumber)
+            throws Exception {
+        shardInfo = new ShardInfo("ShardId-0", concurrencyToken, Collections.emptySet(),
+                ExtendedSequenceNumber.LATEST);
+
+        verifyExpectedDeletedLeasesCompletedShardCase(shardInfo, childShardsForMerge(), extendedSequenceNumber, 0);
+    }
+
+    /**
+     * Tests that if a lease's parents are still present, we do not delete the lease.
+     */
+    @Test
+    public final void testLeaseNotDeletedWhenParentsStillPresent() throws Exception {
+        shardInfo = new ShardInfo("ShardId-0", concurrencyToken, Collections.singleton("parent"),
+                ExtendedSequenceNumber.LATEST);
+
+        verifyExpectedDeletedLeasesCompletedShardCase(shardInfo, childShardsForMerge(), ExtendedSequenceNumber.LATEST, 0);
+    }
+
+    /**
+     * Tests ResourceNotFound case for if a shard expires, that we delete the lease when shardExpired is found.
+     */
+    @Test
+    public final void testLeaseDeletedWhenShardDoesNotExist() throws Exception {
+        shardInfo = new ShardInfo("ShardId-0", concurrencyToken, Collections.emptySet(),
+                ExtendedSequenceNumber.LATEST);
+        final KinesisClientLease heldLease = LeaseHelper.createLease(shardInfo.getShardId(), "leaseOwner", Collections.singleton("parentShardId"));
+
+        testLeaseDeletedWhenShardDoesNotExist(heldLease);
+    }
+
+    /**
+     * Tests ResourceNotFound case when completed lease cleanup is disabled.
+     * @throws Exception
+     */
+    @Test
+    public final void testLeaseDeletedWhenShardDoesNotExistAndCleanupCompletedLeaseDisabled() throws Exception {
+        shardInfo = new ShardInfo("ShardId-0", concurrencyToken, Collections.emptySet(),
+                ExtendedSequenceNumber.LATEST);
+        final KinesisClientLease heldLease = LeaseHelper.createLease(shardInfo.getShardId(), "leaseOwner", Collections.singleton("parentShardId"));
+
+        cleanupLeasesOfCompletedShards = false;
+
+        leaseCleanupManager = new LeaseCleanupManager(kinesis, leaseManager, deletionThreadPool, NULL_METRICS_FACTORY,
+                cleanupLeasesOfCompletedShards, leaseCleanupIntervalMillis, completedLeaseCleanupIntervalMillis, garbageLeaseCleanupIntervalMillis, maxRecords);
+
+        testLeaseDeletedWhenShardDoesNotExist(heldLease);
+    }
+
+    public void testLeaseDeletedWhenShardDoesNotExist(KinesisClientLease heldLease) throws Exception {
+        when(leaseCoordinator.getCurrentlyHeldLease(shardInfo.getShardId())).thenReturn(heldLease);
+        when(kinesis.get(anyString(), anyInt())).thenThrow(ResourceNotFoundException.class);
+        when(kinesis.getIterator(anyString(), anyString())).thenThrow(ResourceNotFoundException.class);
+        when(leaseManager.getLease(heldLease.getLeaseKey())).thenReturn(heldLease);
+
+        leaseCleanupManager.enqueueForDeletion(new LeasePendingDeletion(heldLease, shardInfo));
+        leaseCleanupManager.cleanupLeases();
+
+        verify(leaseManager, times(1)).deleteLease(heldLease);
+    }
+
+    private void verifyExpectedDeletedLeasesCompletedShardCase(ShardInfo shardInfo, List<ChildShard> childShards,
+                                                                     ExtendedSequenceNumber extendedSequenceNumber,
+                                                                     int expectedDeletedLeases) throws Exception {
+        verifyExpectedDeletedLeasesCompletedShardCase(shardInfo, childShards, extendedSequenceNumber, true, expectedDeletedLeases);
+    }
+
+    private void verifyExpectedDeletedLeasesCompletedShardCase(ShardInfo shardInfo, List<ChildShard> childShards,
+                                                                     ExtendedSequenceNumber extendedSequenceNumber,
+                                                                     boolean childShardLeasesPresent,
+                                                                     int expectedDeletedLeases) throws Exception {
+
+        final KinesisClientLease lease = LeaseHelper.createLease(shardInfo.getShardId(), "leaseOwner", shardInfo.getParentShardIds(),
+                childShards.stream().map(c -> c.getShardId()).collect(Collectors.toSet()));
+        final List<KinesisClientLease> childShardLeases = childShards.stream().map(c -> LeaseHelper.createLease(
+                c.getShardId(), "leaseOwner",  Collections.singleton(shardInfo.getShardId()),
+                Collections.emptyList(), extendedSequenceNumber)).collect(Collectors.toList());
+
+        final List<KinesisClientLease> parentShardLeases = lease.getParentShardIds().stream().map(p ->
+                LeaseHelper.createLease(p, "leaseOwner",  Collections.emptyList(),
+                        Collections.singleton(shardInfo.getShardId()), extendedSequenceNumber)).collect(Collectors.toList());
+
+        when(leaseManager.getLease(lease.getLeaseKey())).thenReturn(lease);
+        for (Lease parentShardLease : parentShardLeases) {
+            when(leaseManager.getLease(parentShardLease.getLeaseKey())).thenReturn(parentShardLease);
+        }
+        if (childShardLeasesPresent) {
+            for (Lease childShardLease : childShardLeases) {
+                when(leaseManager.getLease(childShardLease.getLeaseKey())).thenReturn(childShardLease);
+            }
+        }
+
+        when(kinesis.getIterator(any(String.class), any(String.class))).thenReturn("123");
+
+        final GetRecordsResult getRecordsResult = new GetRecordsResult();
+                getRecordsResult.setRecords(Collections.emptyList());
+                getRecordsResult.setChildShards(childShards);
+
+        when(kinesis.get(any(String.class), any(Integer.class))).thenReturn(getRecordsResult);
+
+        leaseCleanupManager.enqueueForDeletion(new LeasePendingDeletion(lease, shardInfo));
+        leaseCleanupManager.cleanupLeases();
+
+        verify(leaseManager, times(expectedDeletedLeases)).deleteLease(any(Lease.class));
+    }
+
+    private List<ChildShard> childShardsForSplit() {
+        final List<String> parentShards = Arrays.asList(splitParent);
+
+        final ChildShard leftChild = new ChildShard();
+                leftChild.setShardId("leftChild");
+                leftChild.setParentShards(parentShards);
+                leftChild.setHashKeyRange(ShardObjectHelper.newHashKeyRange("0", "49"));
+
+        final ChildShard rightChild = new ChildShard();
+                rightChild.setShardId("rightChild");
+                rightChild.setParentShards(parentShards);
+                rightChild.setHashKeyRange(ShardObjectHelper.newHashKeyRange("50", "99"));
+
+        return  Arrays.asList(leftChild, rightChild);
+    }
+
+    private List<ChildShard> childShardsForMerge() {
+        final List<String> parentShards = Arrays.asList(mergeParent1, mergeParent2);
+
+        final ChildShard child = new ChildShard();
+                child.setShardId("onlyChild");
+                child.setParentShards(parentShards);
+                child.setHashKeyRange(ShardObjectHelper.newHashKeyRange("0", "99"));
+
+        return Collections.singletonList(child);
+    }
+}

src/test/java/com/amazonaws/services/kinesis/leases/impl/LeaseHelper.java

@@ -0,0 +1,44 @@
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates.
+ * Licensed under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.amazonaws.services.kinesis.leases.impl;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
+
+import java.util.Collection;
+import java.util.Collections;
+
+public class LeaseHelper {
+
+    public static KinesisClientLease createLease(String leaseKey, String leaseOwner, Collection<String> parentShardIds) {
+        return createLease(leaseKey, leaseOwner, parentShardIds, Collections.emptySet(), ExtendedSequenceNumber.LATEST);
+    }
+
+    public static KinesisClientLease  createLease(String leaseKey, String leaseOwner, Collection<String> parentShardIds, Collection<String> childShardIds) {
+        return createLease(leaseKey, leaseOwner, parentShardIds, childShardIds, ExtendedSequenceNumber.LATEST);
+    }
+
+    public static KinesisClientLease  createLease(String leaseKey, String leaseOwner, Collection<String> parentShardIds,
+                                    Collection<String> childShardIds, ExtendedSequenceNumber extendedSequenceNumber) {
+        KinesisClientLease  lease = new KinesisClientLease ();
+        lease.setLeaseKey(leaseKey);
+        lease.setLeaseOwner(leaseOwner);
+        lease.setParentShardIds(parentShardIds);
+        lease.setChildShardIds(childShardIds);
+        lease.setCheckpoint(extendedSequenceNumber);
+
+        return lease;
+    }
+}

src/test/java/com/amazonaws/services/kinesis/leases/impl/LeaseManagerIntegrationTest.java

@@ -27,6 +27,7 @@ import com.amazonaws.services.dynamodbv2.model.ListTablesResult;
 import com.amazonaws.services.dynamodbv2.model.TableDescription;
 import com.amazonaws.services.dynamodbv2.model.TableStatus;
 import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
+import com.amazonaws.services.kinesis.model.HashKeyRange;
 import junit.framework.Assert;
 
 import org.junit.Test;
@@ -124,6 +125,37 @@ public class LeaseManagerIntegrationTest extends LeaseIntegrationTest {
         Assert.assertFalse(leaseManager.renewLease(leaseCopy));
     }
 
+    /**
+     * Tests leaseManager.updateLeaseWithMetaInfo() when the lease is deleted before updating it with meta info
+     */
+    @Test
+    public void testDeleteLeaseThenUpdateLeaseWithMetaInfo() throws LeasingException {
+        TestHarnessBuilder builder = new TestHarnessBuilder(leaseManager);
+        KinesisClientLease lease = builder.withLease("1").build().get("1");
+        final String leaseKey = lease.getLeaseKey();
+        leaseManager.deleteLease(lease);
+        leaseManager.updateLeaseWithMetaInfo(lease, UpdateField.HASH_KEY_RANGE);
+        final KinesisClientLease deletedLease = leaseManager.getLease(leaseKey);
+        Assert.assertNull(deletedLease);
+    }
+
+    /**
+     * Tests leaseManager.updateLeaseWithMetaInfo() on hashKeyRange update
+     */
+    @Test
+    public void testUpdateLeaseWithMetaInfo() throws LeasingException {
+        TestHarnessBuilder builder = new TestHarnessBuilder(leaseManager);
+        KinesisClientLease lease = builder.withLease("1").build().get("1");
+        final String leaseKey = lease.getLeaseKey();
+        final HashKeyRangeForLease hashKeyRangeForLease = HashKeyRangeForLease.fromHashKeyRange(new HashKeyRange()
+                                                                                                   .withStartingHashKey("1")
+                                                                                                   .withEndingHashKey("2"));
+        lease.setHashKeyRange(hashKeyRangeForLease);
+        leaseManager.updateLeaseWithMetaInfo(lease, UpdateField.HASH_KEY_RANGE);
+        final KinesisClientLease updatedLease = leaseManager.getLease(leaseKey);
+        Assert.assertEquals(lease, updatedLease);
+    }
+
     /**
      * Tests takeLease when the lease is not already owned.
      */