From 2447513de386240623d9aa9aeef16a4334a71905 Mon Sep 17 00:00:00 2001 From: Roberto Tyley Date: Mon, 4 Oct 2021 13:34:27 +0100 Subject: [PATCH] Fix DynamoDBLeaseTaker logging of available leases (#846) When `DynamoDBLeaseTaker` is deciding what leases to take to satisfy the target lease-count of it's worker, it can either take *expired* leases that haven't been updated by any other worker for a while, or failing that, 'steal' a limited number (by default, 1) of leases from other workers, in order to ensure its worker is gradually getting closer to it's target. It turns out that the log message it prints out when taking expired leases is misleading. For instance, in this message - the leases weren't stolen, they were taken from the available 'expired' leases: ``` Worker xxxxxx saw 68 total leases, 0 available leases, 6 workers. Target is 12 leases, I have 2 leases, I will take 10 leases ``` ...yet the message says there were 0 available leases, so how is that possible? The truth is, there were 10 available expired leases, but the log message was printing out the value of `expiredLeases.size()` - and `expiredLeases` has been mutated over the course of the method, with entries removed and added to `leasesToTake`: https://github.com/awslabs/amazon-kinesis-client/blob/0c5042dadf794fe988438436252a5a8fe70b6b0b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java#L421 The fix to logging in this commit just records `numAvailableLeases` at the start of the method, so we can give an accurate log message: ``` Worker xxxxxx saw 68 total leases, 10 available leases, 6 workers. Target is 12 leases, I have 2 leases, I will take 10 leases ``` --- .../amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java index 7aff9513..06b2527b 100644 --- a/amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java +++ b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java @@ -347,6 +347,7 @@ public class DynamoDBLeaseTaker implements LeaseTaker { MetricsUtil.addWorkerIdentifier(scope, workerIdentifier); List veryOldLeases = new ArrayList<>(); + final int numAvailableLeases = expiredLeases.size(); int numLeases = 0; int numWorkers = 0; int numLeasesToReachTarget = 0; @@ -438,7 +439,7 @@ public class DynamoDBLeaseTaker implements LeaseTaker { log.info( "Worker {} saw {} total leases, {} available leases, {} " + "workers. Target is {} leases, I have {} leases, I will take {} leases", - workerIdentifier, numLeases, expiredLeases.size(), numWorkers, target, myCount, + workerIdentifier, numLeases, numAvailableLeases, numWorkers, target, myCount, leasesToTake.size()); }