* Changed Prefetch to catch `SdkException` instead of `SdkClientException`
* With SDK 2.x service exceptions are of the type `SdkServiceException`
* Adding `*.iml` to .gitignore
* Started to play around with restarting from last processed
After a failure the KCL would instead restart from what the
ShardConsumer says it last processed.
* Extracted the InternalSubscriber to its own class
Extracted the InternalSubscriber to ShardConsumerSubscriber to make
testing easier. Added tests for the ShardConsumerSubscriber that
verifies error handling and other components of the class.
Added tests that verify the restart from behavior.
* Moved the ProcessRecordsInputMatcher to its own class
* Initial changes to for restarting of the PrefetchRecordsPublisher
* Remove code coverage configuration
* Switched to using explicit locks to deal with blocking queue
When the blocking queue is full it would normally enter into a fully
parked state, but would continue to hold the lock.
This changes the process to only block for a second when attempting to
enqueue a response, and if it doesn't succeed check to see if it's
been reset before attempting again.
* Changed locking around the restart, and how fetcher gets updated
Changed the locking around the restart to use a reader/writer lock
instead of single lock with a yield.
Changed how the fetcher is reset to not restart from an
advanceIteratorTo which would retrieve a new shard iterator. Instead
the resetIterator method takes both the iterator to start from, the
last accepted sequence number, and the initial position.
* Changed test to ensure that PositionResetException is thrown
Changed the test to wait for the queue to reach capacity before
restarting the PrefetchRecordsPublisher. This should mostly ensure
that calling restartFrom will trigger a throw of a
PositionResetException.
Added @VisibleFortest on the queue since it was already being used in testing.
* Move to snapshot
* Ensure that only one thread can be sending data at a time
In the test the TestPublisher is accessed from two threads: the test
thread, and the dispatch thread. Both have the possibility of calling
send() under certain conditions. This changes it so that only one of
the threads can actively be sending data at a time.
TestPublisher#requested was changed to volatile to ensure that calling
cancel can correctly set it to zero.
* Block the test until the blocking thread is in control
This test is somewhat of an odd case as it intends to test what
happens when nothing is dispatched to the ShardConsumerSubcriber for
some amount of time, but data is queued for dispatch. To do this we
block the single thread of the executor with a lock to ensure that
items pile up in the queue so that should the restart work incorrectly
we will see lost data.
The AWS SDK depends on being able to find which region it should
operate in. In many cases this is either explicitly set or retrieved
from a variety of sources. For these test cases we don't actually
care what the region. To ensure that the tests operate as expected the
region is set before the test runs, and reset upon completion.
* Introducing MultiLangDaemon support for Enhanced Fan-Out.
* MultiLangDaemon now supports the following command line options.
* `--properties-file`: Properties file that the KCL should use to set up the Scheduler.
* `--log-configuration`: logback.xml that the KCL should use for logging.
* Updated AWS SDK dependency to 2.2.0.
* MultiLangDaemon now uses logback for logging.
* Remove a possible deadlock on polling queue fill
Adding new items to the receive queue for the PrefetchRecordsPublisher
when at capacity would deadlock retrievals as it was already holding
a lock on this.
The method addArrivedRecordsInput did not need to be synchronized on
this as it didn't change any of the protected
state (requestedResponses). There is a call to drainQueueForRequests
immediately after the addArrivedRecordsInput that will ensure newly
arrived data is dispatched.
This fixes#448
* Small fix on the reasoning comment
* Adjust the test to act more like the ShardConsumer
The ShardConsuemr, which is the principal user of the
PrefetchRecordsPublisher, uses RxJava to consume from publisher. This
test uses RxJava to consume, and notifies the test thread once
MAX_ITEMS * 3 have been received. This ensures that we cycle through
the queue at least 3 times.
* Removed the upper limit on the retrievals
The way RxJava's request management makes it possible that more
requests than we might expect can happen.
Added the SequenceNumberValidator that will be used in the checkpoint
process to ensure that the sequence number is valid for the shard
being checkpointed.
* Run multiple instance of scheduler on one JVM
* handling creation of shardSyncer in DynamoDBLeaseManagementFactory and LeaseManagementConfig
* remove multi-threading unit test and do some small refactorings
* refectoring
* deprecate ShardSyncer and use HierarchichalShardSyncer instead; change the order for metricsFactory and HierarchichalShardSyncer in ShardConsumerArgument
* fix typos and use mock object of shardSyncer
* delete improper comments
* fix comments
* remove duplicated comments
Reverted 3 commits:
Revert "Change version number to 2.0.3-experimental"
Revert: 54c171dc2a.
Revert "Experimental support for sequence number validation in the publisher (#401)"
Revert: 592499f7bc.
Revert "Support Validating Records are From to the Expected Shard (#400)"
Revert: 01f5db8049.
* This feature enables customers to perform actions on DynamoDB lease tables once created and in the active state
* Introducing TableCreatorCallback for DynamoDB lease management
* Introducing DoesNothingTableCreatorCallback
* Intoducing TableCreatorCallback config in LeaseManagementConfig, with DoesNothingTableCreatorCallback as the default
* Introducing TableCreatorCallbackInput object.
* Updating the javadoc
* Handling ReadTimeouts gracefully
* Emitting logging messages at DEBUG level for retryable exceptions
* Introducing SubscribeToShardRetryableException
* Addressing comments
* Making private ThrowableCategory class static
* Creating static instances for acquiretimeout and readtimeout categories
* Cleaned up imports
* Renamed and moved SubscribeToShardRetryableException to RetryableRetrievalException
* Renamed UNKNOWN exception type to Other
* Moved sequence number validation to an experimental feature
Moved the sequence number validation to become an experimental feature
that can be removed in the future.
Added an annotation for experimental features.
* Delete merge conflict again?
* Add some reminder that this stuff is experimental
* Added a reason field, and some reasons
Added a reason value to the annotation, and updated two of the unusual places.
* SequenceNumberValidator for verifying shardId's
Added a SequenceNumberValidator that will can extract, and verify
shardId's from a v2 sequence number.
* Added documentation and bit length test
Added documentation for the public methods.
Added a bit length test for the reader that will reject sequence
numbers that don't fit the expectations.
* Added more comments and further document public operations
Added comments in the only SequenceNumberReader explaining how things
are expected to work.
Further documented the class and operations with expectations and outcomes.
* Added configuration to allow failing on mismatched records
Allow configuration which will cause the FanOutRecordsPublisher to
throw an exception when it detects records that aren't for the shard
it's processing.
* Some debug logging to understand mismatched sequence numbers
Added some logging messages for sequence numbers when starting up.
* Added debug support and logging
Added @ToString to InitialPositionInStreamExtended for debugging
purposes.
Added a debug log about the initialization of the
FanOutRecordsPublisher to ensure that the publisher is being
initialized as expected.
Changed the CoordinatorFactory methods to be default methods. This
will not require an implementer to implement the deprecated methods.
Updated the documentation on the methods to indicate resolve order as
a list.
Cleaning up configuration
Deprecating createGracefulShutdownCoordinator from CoordinatorFactory
Deprecating createWorkerStateChangeListener from CoordinatorFactory
Introduing gracefulShutdownCoordinator and workerStateChangeListener configurations to CoordinatorConfig
Switching to use CoordinatorFactory only if the new configurations in code are set to null
* Deprecating createGracefulShutdownCoordinator from CoordinatorFactory
* Deprecating createWorkerStateChangeListener from CoordinatorFactory
* Introduing gracefulShutdownCoordinator and workerStateChangeListener configurations to CoordinatorConfig
* Switching to use CoordinatorFactory only if the new configurations in code are set to null
