* Remove a possible deadlock on polling queue fill
Adding new items to the receive queue for the PrefetchRecordsPublisher
when at capacity would deadlock retrievals as it was already holding
a lock on this.
The method addArrivedRecordsInput did not need to be synchronized on
this as it didn't change any of the protected
state (requestedResponses). There is a call to drainQueueForRequests
immediately after the addArrivedRecordsInput that will ensure newly
arrived data is dispatched.
This fixes#448
* Small fix on the reasoning comment
* Adjust the test to act more like the ShardConsumer
The ShardConsuemr, which is the principal user of the
PrefetchRecordsPublisher, uses RxJava to consume from publisher. This
test uses RxJava to consume, and notifies the test thread once
MAX_ITEMS * 3 have been received. This ensures that we cycle through
the queue at least 3 times.
* Removed the upper limit on the retrievals
The way RxJava's request management makes it possible that more
requests than we might expect can happen.
Added the SequenceNumberValidator that will be used in the checkpoint
process to ensure that the sequence number is valid for the shard
being checkpointed.
* Run multiple instance of scheduler on one JVM
* handling creation of shardSyncer in DynamoDBLeaseManagementFactory and LeaseManagementConfig
* remove multi-threading unit test and do some small refactorings
* refectoring
* deprecate ShardSyncer and use HierarchichalShardSyncer instead; change the order for metricsFactory and HierarchichalShardSyncer in ShardConsumerArgument
* fix typos and use mock object of shardSyncer
* delete improper comments
* fix comments
* remove duplicated comments
Reverted 3 commits:
Revert "Change version number to 2.0.3-experimental"
Revert: 54c171dc2a.
Revert "Experimental support for sequence number validation in the publisher (#401)"
Revert: 592499f7bc.
Revert "Support Validating Records are From to the Expected Shard (#400)"
Revert: 01f5db8049.
* This feature enables customers to perform actions on DynamoDB lease tables once created and in the active state
* Introducing TableCreatorCallback for DynamoDB lease management
* Introducing DoesNothingTableCreatorCallback
* Intoducing TableCreatorCallback config in LeaseManagementConfig, with DoesNothingTableCreatorCallback as the default
* Introducing TableCreatorCallbackInput object.
* Updating the javadoc
* Handling ReadTimeouts gracefully
* Emitting logging messages at DEBUG level for retryable exceptions
* Introducing SubscribeToShardRetryableException
* Addressing comments
* Making private ThrowableCategory class static
* Creating static instances for acquiretimeout and readtimeout categories
* Cleaned up imports
* Renamed and moved SubscribeToShardRetryableException to RetryableRetrievalException
* Renamed UNKNOWN exception type to Other
* Moved sequence number validation to an experimental feature
Moved the sequence number validation to become an experimental feature
that can be removed in the future.
Added an annotation for experimental features.
* Delete merge conflict again?
* Add some reminder that this stuff is experimental
* Added a reason field, and some reasons
Added a reason value to the annotation, and updated two of the unusual places.
* SequenceNumberValidator for verifying shardId's
Added a SequenceNumberValidator that will can extract, and verify
shardId's from a v2 sequence number.
* Added documentation and bit length test
Added documentation for the public methods.
Added a bit length test for the reader that will reject sequence
numbers that don't fit the expectations.
* Added more comments and further document public operations
Added comments in the only SequenceNumberReader explaining how things
are expected to work.
Further documented the class and operations with expectations and outcomes.
* Added configuration to allow failing on mismatched records
Allow configuration which will cause the FanOutRecordsPublisher to
throw an exception when it detects records that aren't for the shard
it's processing.
* Some debug logging to understand mismatched sequence numbers
Added some logging messages for sequence numbers when starting up.
* Added debug support and logging
Added @ToString to InitialPositionInStreamExtended for debugging
purposes.
Added a debug log about the initialization of the
FanOutRecordsPublisher to ensure that the publisher is being
initialized as expected.
Changed the CoordinatorFactory methods to be default methods. This
will not require an implementer to implement the deprecated methods.
Updated the documentation on the methods to indicate resolve order as
a list.
Cleaning up configuration
Deprecating createGracefulShutdownCoordinator from CoordinatorFactory
Deprecating createWorkerStateChangeListener from CoordinatorFactory
Introduing gracefulShutdownCoordinator and workerStateChangeListener configurations to CoordinatorConfig
Switching to use CoordinatorFactory only if the new configurations in code are set to null
* Deprecating createGracefulShutdownCoordinator from CoordinatorFactory
* Deprecating createWorkerStateChangeListener from CoordinatorFactory
* Introduing gracefulShutdownCoordinator and workerStateChangeListener configurations to CoordinatorConfig
* Switching to use CoordinatorFactory only if the new configurations in code are set to null
* Update release notes for release 2.0.1
* Updated version to 2.0.1
* Added notes for PR #371, and cleaned up formatting
* Reordered release notes, and added additional fixes
Ordered the release notes in ascending order of their issue #
Added release notes for #374 and #375
* Add release notes to the change log
outstandingRequests was actually representing the available space in
the RxJava queue. This renames it to better match reality.
Also changed to only make the request if there is available queue
space. We now decrement availableQueueSpace ahead of determine
whether to request another item.
* Added missing lock around the call to request.
Calls to Subscription#request weren't synchronized correctly. This
was only really an issue if there is a large number of errors
occurring.
* Reject operations where the subscriber doesn't match.
If the original subscriber doesn't match the current subscriber reject
operations completely.
If the flow is null, but the subscriber still matches error out the
subscription. The original subscriber will restart.
For canceling only accept the cancel request if the original
subscriber matches the current subscriber.
* Remove unneeded if statement
Don't really need to check if the subscriber is still current, as this
is synchronized.
Subscribe to shard ends periodically and the KCL needs to reconnect at
the last continuation sequence number. If the continuation sequence
number happens to be the last record returned using AT_SEQUENCE_NUMBER
will cause the record to be returned again.
