* Remove a possible deadlock on polling queue fill
Adding new items to the receive queue for the PrefetchRecordsPublisher
when at capacity would deadlock retrievals as it was already holding
a lock on this.
The method addArrivedRecordsInput did not need to be synchronized on
this as it didn't change any of the protected
state (requestedResponses). There is a call to drainQueueForRequests
immediately after the addArrivedRecordsInput that will ensure newly
arrived data is dispatched.
This fixes#448
* Small fix on the reasoning comment
* Adjust the test to act more like the ShardConsumer
The ShardConsuemr, which is the principal user of the
PrefetchRecordsPublisher, uses RxJava to consume from publisher. This
test uses RxJava to consume, and notifies the test thread once
MAX_ITEMS * 3 have been received. This ensures that we cycle through
the queue at least 3 times.
* Removed the upper limit on the retrievals
The way RxJava's request management makes it possible that more
requests than we might expect can happen.
Added the SequenceNumberValidator that will be used in the checkpoint
process to ensure that the sequence number is valid for the shard
being checkpointed.
* Run multiple instance of scheduler on one JVM
* handling creation of shardSyncer in DynamoDBLeaseManagementFactory and LeaseManagementConfig
* remove multi-threading unit test and do some small refactorings
* refectoring
* deprecate ShardSyncer and use HierarchichalShardSyncer instead; change the order for metricsFactory and HierarchichalShardSyncer in ShardConsumerArgument
* fix typos and use mock object of shardSyncer
* delete improper comments
* fix comments
* remove duplicated comments
Reverted 3 commits:
Revert "Change version number to 2.0.3-experimental"
Revert: 54c171dc2a.
Revert "Experimental support for sequence number validation in the publisher (#401)"
Revert: 592499f7bc.
Revert "Support Validating Records are From to the Expected Shard (#400)"
Revert: 01f5db8049.
* This feature enables customers to perform actions on DynamoDB lease tables once created and in the active state
* Introducing TableCreatorCallback for DynamoDB lease management
* Introducing DoesNothingTableCreatorCallback
* Intoducing TableCreatorCallback config in LeaseManagementConfig, with DoesNothingTableCreatorCallback as the default
* Introducing TableCreatorCallbackInput object.
* Updating the javadoc
* Handling ReadTimeouts gracefully
* Emitting logging messages at DEBUG level for retryable exceptions
* Introducing SubscribeToShardRetryableException
* Addressing comments
* Making private ThrowableCategory class static
* Creating static instances for acquiretimeout and readtimeout categories
* Cleaned up imports
* Renamed and moved SubscribeToShardRetryableException to RetryableRetrievalException
* Renamed UNKNOWN exception type to Other
* Moved sequence number validation to an experimental feature
Moved the sequence number validation to become an experimental feature
that can be removed in the future.
Added an annotation for experimental features.
* Delete merge conflict again?
* Add some reminder that this stuff is experimental
* Added a reason field, and some reasons
Added a reason value to the annotation, and updated two of the unusual places.
* SequenceNumberValidator for verifying shardId's
Added a SequenceNumberValidator that will can extract, and verify
shardId's from a v2 sequence number.
* Added documentation and bit length test
Added documentation for the public methods.
Added a bit length test for the reader that will reject sequence
numbers that don't fit the expectations.
* Added more comments and further document public operations
Added comments in the only SequenceNumberReader explaining how things
are expected to work.
Further documented the class and operations with expectations and outcomes.
* Added configuration to allow failing on mismatched records
Allow configuration which will cause the FanOutRecordsPublisher to
throw an exception when it detects records that aren't for the shard
it's processing.
* Some debug logging to understand mismatched sequence numbers
Added some logging messages for sequence numbers when starting up.
* Added debug support and logging
Added @ToString to InitialPositionInStreamExtended for debugging
purposes.
Added a debug log about the initialization of the
FanOutRecordsPublisher to ensure that the publisher is being
initialized as expected.
Changed the CoordinatorFactory methods to be default methods. This
will not require an implementer to implement the deprecated methods.
Updated the documentation on the methods to indicate resolve order as
a list.
