Commit graph

209 commits

Author SHA1 Message Date
Sahil Palvia
0326e217f6
Updating version to 2.0.4-SNAPSHOT (#438) 2018-10-09 11:55:50 -07:00
shask-amazon
31ab0af901 Added an API on LeaseCoordinator and LeaseTaker to get all leases for… (#428)
* Added an API on LeaseCoordinator and LeaseTaker to get all leases for the application
2018-10-09 07:56:13 -07:00
Justin Pfifer
e972617bfc
Release note for release 2.0.3 (#436)
Added release notes, and changelog for the 2.0.3 changes.

Made the warning clearer that it only affects the 2.0 version of the
Amazon Kinesis Client
2018-10-08 15:51:06 -07:00
Justin Pfifer
8e6a8c7da3 Advance version of the AWS SDK to 2.0.6 (#434)
Fixes a bug when making SubscribeToShard requests over HTTP 1.1.
Using HTTP 1.1 for SubscribeToShard isn't supported, and may be
break at any time.
2018-10-04 10:40:31 -07:00
Sahil Palvia
9e420d83e4 Fixing issue with KinesisClientUtil class (#433)
* Passing HttpClientBuilder to the client instead of HttpClient
2018-10-03 08:46:34 -07:00
Justin Pfifer
e86bf3d7f3
Revert experimental features from master (#431)
Reverted 3 commits:

Revert "Change version number to 2.0.3-experimental"
Revert: 54c171dc2a.

Revert "Experimental support for sequence number validation in the publisher (#401)"
Revert: 592499f7bc.

Revert "Support Validating Records are From to the Expected Shard (#400)"
Revert: 01f5db8049.
2018-10-02 14:50:25 -07:00
Sahil Palvia
a88d4ba602
Introducing callback for DynamoDB lease table (#413)
* This feature enables customers to perform actions on DynamoDB lease tables once created and in the active state
* Introducing TableCreatorCallback for DynamoDB lease management
* Introducing DoesNothingTableCreatorCallback
* Intoducing TableCreatorCallback config in LeaseManagementConfig, with DoesNothingTableCreatorCallback as the default
* Introducing TableCreatorCallbackInput object.
* Updating the javadoc
2018-09-25 10:06:24 -07:00
Sahil Palvia
0d6335d434
Upgrading Guava dependency to version 26.0-jre (Issue #416) (#420) 2018-09-25 10:00:08 -07:00
Sahil Palvia
d85d6f4187
Fixing spelling error for Acquire timeout (#410) 2018-09-20 21:52:31 +05:30
Sahil Palvia
131b1e4b0f Cleaning up tests (#408)
* Deleting all unused tests for KinesisProxy
2018-09-19 15:00:35 -07:00
Sahil Palvia
a893da6942 Graceful handling of ReadTimeoutExceptions (#403)
* Handling ReadTimeouts gracefully

* Emitting logging messages at DEBUG level for retryable exceptions
* Introducing SubscribeToShardRetryableException

* Addressing comments

* Making private ThrowableCategory class static
* Creating static instances for acquiretimeout and readtimeout categories
* Cleaned up imports
* Renamed and moved SubscribeToShardRetryableException to RetryableRetrievalException
* Renamed UNKNOWN exception type to Other
2018-09-19 10:15:40 -07:00
Justin Pfifer
54c171dc2a
Change version number to 2.0.3-experimental 2018-09-18 18:38:59 -07:00
Justin Pfifer
592499f7bc Experimental support for sequence number validation in the publisher (#401)
* Moved sequence number validation to an experimental feature

Moved the sequence number validation to become an experimental feature
that can be removed in the future.

Added an annotation for experimental features.

* Delete merge conflict again?

* Add some reminder that this stuff is experimental

* Added a reason field, and some reasons

Added a reason value to the annotation, and updated two of the unusual places.
2018-09-19 03:35:36 +05:30
Justin Pfifer
01f5db8049 Support Validating Records are From to the Expected Shard (#400)
* SequenceNumberValidator for verifying shardId's

Added a SequenceNumberValidator that will can extract, and verify
shardId's from a v2 sequence number.

* Added documentation and bit length test

Added documentation for the public methods.
Added a bit length test for the reader that will reject sequence
numbers that don't fit the expectations.

* Added more comments and further document public operations

Added comments in the only SequenceNumberReader explaining how things
are expected to work.

Further documented the class and operations with expectations and outcomes.

* Added configuration to allow failing on mismatched records

Allow configuration which will cause the FanOutRecordsPublisher to
throw an exception when it detects records that aren't for the shard
it's processing.
2018-09-18 03:03:46 +05:30
Justin Pfifer
e8735a4742
Debugging Logs for Initialization of FanOutRecordsPublisher (#398)
* Some debug logging to understand mismatched sequence numbers

Added some logging messages for sequence numbers when starting up.

* Added debug support and logging

Added @ToString to InitialPositionInStreamExtended for debugging
purposes.
Added a debug log about the initialization of the
FanOutRecordsPublisher to ensure that the publisher is being
initialized as expected.
2018-09-11 12:16:12 -07:00
Justin Pfifer
c8a3a031f4
Updated to 2.0.3-SNAPSHOT (#397) 2018-09-11 10:41:47 -07:00
Justin Pfifer
a8badc22d0
Release notes for 2.0.2 (#392)
Release notes for 2.0.2, and advance the version
2018-09-04 10:44:30 -07:00
jiaxul
ea49eef19e Introduce initialization exception handler in KCL V2 (#369)
Added a new method to the WorkerStateChangeListener that is called once all attempts to initialize the scheduler have failed.
2018-08-31 07:54:52 -07:00
Justin Pfifer
0634a3c836
Merge pull request #388 from pfifer/default-deprecated
Change the CoordinatorFactory deprecated methods to default
2018-08-30 08:17:33 -07:00
Justin Pfifer
70f937b68b
Merge pull request #387 from maghis/patch-1
Fixed typo in terminate log message
2018-08-30 08:06:47 -07:00
Pfifer, Justin
106c055744 Change the CoordinatorFactory deprecated methods to default
Changed the CoordinatorFactory methods to be default methods.  This
will not require an implementer to implement the deprecated methods.

Updated the documentation on the methods to indicate resolve order as
a list.
2018-08-30 08:03:18 -07:00
Massimo Andreasi Bassi
b6472df41a
Fixed typo in terminate log message 2018-08-30 10:57:57 -04:00
Justin Pfifer
03af17f6ee
Merge pull request #386 from sahilpalvia/commons-lang-fix
Upgrading KCL to use commons-lang 3.7
2018-08-29 14:14:58 -07:00
Sahil Palvia
0ec3004b0b Updating the import in the configuration test. 2018-08-27 13:12:28 -07:00
Sahil Palvia
878b1a4af9 Upgrading KCL to use only commons-lang3
* Removing commons-lang 2.6 dependency
* Upgrading imports to use commons-lang3
2018-08-27 13:07:39 -07:00
Justin Pfifer
7734561e18
Merge pull request #385 from sahilpalvia/coordinator-config-fix
Cleaning up configuration

Deprecating createGracefulShutdownCoordinator from CoordinatorFactory
Deprecating createWorkerStateChangeListener from CoordinatorFactory
Introduing gracefulShutdownCoordinator and workerStateChangeListener configurations to CoordinatorConfig
Switching to use CoordinatorFactory only if the new configurations in code are set to null
2018-08-27 11:51:06 -07:00
Sahil Palvia
7fa9e10991 Cleaning up configuration
* Deprecating createGracefulShutdownCoordinator from CoordinatorFactory
* Deprecating createWorkerStateChangeListener from CoordinatorFactory
* Introduing gracefulShutdownCoordinator and workerStateChangeListener configurations to CoordinatorConfig
* Switching to use CoordinatorFactory only if the new configurations in code are set to null
2018-08-27 11:26:04 -07:00
Justin Pfifer
99667e8f50 Respect logWarningForTaskAfterMillis for logging a retrieval warning (#383)
Only log a warning if it's been > logWarningForTaskAfterMillis time
since data was last received.
2018-08-23 13:16:53 -07:00
Justin Pfifer
e3eff0dc3c
Merge pull request #379 from sahilpalvia/readme_fix
Updating README and version information
2018-08-22 11:45:29 -07:00
Sahil Palvia
9459cc6a9e Updating README and version information
* Adding information on how to import the project using maven
* Updating project version to 2.0.2-SNAPSHOT
2018-08-21 17:02:42 -07:00
Justin Pfifer
2e598fe715
Merge pull request #378 from pfifer/release-notes-2.0.1-missed
Updates release notes for PR #368
2018-08-21 12:31:11 -07:00
Pfifer, Justin
a720c8670b Updates release notes for PR #368
Updated the configuration class/method, and added the PR to the
release notes.
2018-08-21 11:49:16 -07:00
Justin Pfifer
9d6eb6b1a8 Advance version and release notes for 2.0.1 (#377)
* Update release notes for release 2.0.1

* Updated version to 2.0.1

* Added notes for PR #371, and cleaned up formatting

* Reordered release notes, and added additional fixes

Ordered the release notes in ascending order of their issue #

Added release notes for #374 and #375

* Add release notes to the change log
2018-08-21 10:12:45 -07:00
Justin Pfifer
90acdc02bf Rename outstandingRquests to availableQueueSpace (#375)
outstandingRequests was actually representing the available space in
the RxJava queue.  This renames it to better match reality.

Also changed to only make the request if there is available queue
space.  We now decrement availableQueueSpace ahead of determine
whether to request another item.
2018-08-20 16:17:25 -07:00
Justin Pfifer
5533d370cd Don't allow activities if the subscriber is current, or the connection to Kinesis is broken (#374)
* Added missing lock around the call to request.

Calls to Subscription#request weren't synchronized correctly.  This
was only really an issue if there is a large number of errors
occurring.

* Reject operations where the subscriber doesn't match.

If the original subscriber doesn't match the current subscriber reject
operations completely.
If the flow is null, but the subscriber still matches error out the
subscription.  The original subscriber will restart.

For canceling only accept the cancel request if the original
subscriber matches the current subscriber.

* Remove unneeded if statement

Don't really need to check if the subscriber is still current, as this
is synchronized.
2018-08-20 13:45:48 -07:00
Justin Pfifer
c1e38f0126 Update the lastRequestTime to prevent overlapping restarts. (#373)
When a restart occurs due to no activity set the lastRequestTime to
now to prevent ti from overlapping itself.
2018-08-20 09:25:57 -07:00
Justin Pfifer
2e2c892b7e Upgrade to AWS SDK 2.0.1 (#372)
Upgraded to the new newest version of the SDK.
2018-08-17 09:42:04 -07:00
Justin Pfifer
e8d2190162 Use AFTER_SEQUENCE_NUMBER when reconnecting (#371)
Subscribe to shard ends periodically and the KCL needs to reconnect at
the last continuation sequence number.  If the continuation sequence
number happens to be the last record returned using AT_SEQUENCE_NUMBER
will cause the record to be returned again.
2018-08-16 13:37:52 -07:00
muktiranjan
e694ab7724 Moving the max number of Scheduler initialization attempts parameter … (#368)
* Moving the max number of Scheduler initialization attempts parameter to CoordinatorConfig

* Changing the max initialization attempts variable name
2018-08-15 13:25:07 -07:00
Sahil Palvia
205cf051f3
Cleaning up LeaseManagementConfig (#366)
* Cleaning up LeaseManagementConfig:

* Removing unused metrics factory variable from LeaseManagementConfig.

* Revert "Cleaning up LeaseManagementConfig:"

This reverts commit b16ba37966.

* Deprecating metrics factory in LeaseManagementConfig

* Marking metrics factory in LeaseManagementConfig for deprecation
2018-08-15 11:14:15 -07:00
Justin Pfifer
6973152f60
Merge pull request #360 from sahilpalvia/dynamodb-iops-fix
Ensure that lease tables are created with the specified IOPs values.
2018-08-13 07:50:21 -07:00
Justin Pfifer
9951062a5d
Merge pull request #359 from sahilpalvia/rnf-fix
Fixing issue with ResourceNotFound around SubscribeToShard
2018-08-13 07:02:09 -07:00
Justin Pfifer
69899cc394
Merge pull request #363 from muktiranjan/development
Making the maximum number of Scheduler initialization attempts configurable
2018-08-13 07:01:28 -07:00
Mukti Ranjan Sahoo
e8553ed5a9 Making the maximum number of Scheduler initialization attempts configurable 2018-08-10 18:11:07 -07:00
Sahil Palvia
f1cbf15075 Introducing changes to avoid breaking changes
* Introducing chained constructors in DynamoDBLeaseManagementFactory
* Introducing TableConstants to maintain Default IOPS in one place
2018-08-10 14:57:58 -07:00
Sahil Palvia
e780421036 Removing mock and introducing @Mock 2018-08-10 14:27:15 -07:00
Sahil Palvia
cd7dc1f9b1 Updating version to 2.0.1-SNAPSHOT for multilang 2018-08-10 10:40:28 -07:00
Sahil Palvia
b396626f7e Changes made according to comments:
* Creating handleFlowError method
* Using mocks instead of unnecessary spies
2018-08-10 10:18:15 -07:00
Sahil Palvia
ca88ee9bc6 Reverting some changes:
* Reverting the constructor, and adding chained constructor
* Reverting support for initial iops methods
* Adding deprecated tags and notes to javadoc
2018-08-10 10:08:59 -07:00
Sahil Palvia
51ec96bf9a Fixing IOPs issue with lease table
* Making DynamoDBLeaseCoordinator take IOPs configuration in the constructor
* InitialLeaseTableReadCapacity and InitialLeaseTableWriteCapacity for the DynamoDBLeaseCoordinator class throws UnsupportedException
2018-08-09 15:14:40 -07:00