* Fix to shutdown PrefetchRecordsPublisher in gracefull manner
Previously when the lease expires PrefetchRecordsPublisher shutdown
the process forecefully by interupting the threads,
which lead to leak in apache http client connection
Now changed to code to shutdown the PrefetchRecordsPublisher
process in more gracefull manager and handled interrupted exception
* Fixed failing unit test
* Add awaitTerminationTimeoutMillis as paramter for PrefetchRecordsPublisher
Since clients can configure there own awaitTerminationTimeoutMillis,
add it as sepearate parameter with default value
* Fix setting interrupot status after shutdown
Co-authored-by: Monishkumar Gajendran <monishku@amazon.com>
* Create DynamoDB tables on On-Demand billing mode by default.
This will enable KCL to create the DynamoDB tables - Lease and
ShardProgress with the On-Demand billing mode instead of provisioned
billing mode.
* Keep previous table creation function for backward compatibility.
* Added unit tests.
* Added more unit tests.
When `DynamoDBLeaseTaker` is deciding what leases to take to satisfy the
target lease-count of it's worker, it can either take *expired* leases
that haven't been updated by any other worker for a while, or failing
that, 'steal' a limited number (by default, 1) of leases from other
workers, in order to ensure its worker is gradually getting closer to
it's target.
It turns out that the log message it prints out when taking expired
leases is misleading. For instance, in this message - the leases
weren't stolen, they were taken from the available 'expired' leases:
```
Worker xxxxxx saw 68 total leases, 0 available leases, 6 workers. Target is 12 leases, I have 2 leases, I will take 10 leases
```
...yet the message says there were 0 available leases, so how is that
possible? The truth is, there were 10 available expired leases, but the
log message was printing out the value of `expiredLeases.size()` - and
`expiredLeases` has been mutated over the course of the method, with
entries removed and added to `leasesToTake`:
0c5042dadf/amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java (L421)
The fix to logging in this commit just records `numAvailableLeases` at
the start of the method, so we can give an accurate log message:
```
Worker xxxxxx saw 68 total leases, 10 available leases, 6 workers. Target is 12 leases, I have 2 leases, I will take 10 leases
```
* Add default `orphanedStreamInitialPositionInStream` in `MultiStreamTracker`, so a custom initial position can be passed to Scheduler to initialize the orphaned stream's config
* Renamed `getDefaultStreamConfig` to `getOrphanedStreamConfig`
* Refactored `SchedulerTest` setup and implement `TestMultiStreamTracker` to test `MultiStreamTracker` interface default methods. Note that this is a workaround for using mockito 1.x to test default interface methods. mockito >= 2.7.13 supports Spy on interface directly, which can be used to test default methods without implementing a concrete class. However, mockito 2.x has a number of breaking changes, so future work will be needed to refactor unit tests and upgrade to mockito >= 2.7.13
Signed-off-by: Rex Chen <shuningc@amazon.com>
* Fixing bug where ShardFilter parameter for ListShards was being passed in for paginated calls.
This resulted in a bug where all calls for ListShards when initializing
the lease table would fail, since Kinesis only requires the NextToken
parameter when making paginated calls.
* Adding some logging for listShards.
Co-authored-by: Joshua Kim <kimjos@amazon.com>
