luciano/amazon-kinesis-client
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
228 commits 48 branches 124 tags 11 MiB
Commit graph

228 commits

This branch
This branch
All branches
Author SHA1 Message Date
Chunxue Yang
0a5724c332 KCL 1.x ShardEnd Shard Sync and Lease table ChildShard persistence 2020-07-06 15:10:17 -07:00
Micah Jaffe
3a88a60a4e
Add periodic hash range auditor, hole detection, and recovery (#43) 
* Add periodic hash range auditor, hole detection, and recovery

* Add unit tests for hash range hole recovery

* Fix max hash range bug

* Address PR feedback

* Fix DDB hash key persistence bug
 2020-06-22 08:45:46 -07:00
Micah Jaffe
6b474b7390
Add configurable max initialization attempts (#39) 
* Add configurable max initialization attempts

* Add maxInitializationAttempts to unit test setup
 2020-05-15 14:02:44 -07:00
Micah Jaffe
fdad20aff4
Add leader-elected periodic shard sync manager as auditor to ShardEndShardSync strategy (#35) 
* Add leader-elected periodic shard sync manager as auditor to ShardEndShardSync strategy

* Address PR feedback and add Builder unit tests
 2020-05-14 09:04:44 -07:00
Micah Jaffe
45e76b0fd5
Add leader-elected shard sync on application bootstrap (#28) 
* Add leader-elected shard sync on application bootstrap

* Add additional testing and address PR feedback

* Remove runShardSync leader election from boostrap case

* Remove random UUID workerId and update unit tests
 2020-05-07 17:08:53 -07:00
Micah Jaffe
550d7af5b1
Add lease sync strategy for empty lease table (#24) 
* Add lease sync strategy for empty lease table

* Fix ShardSyncer unit tests to reflect new empty lease table shard sync logic
 2020-05-04 17:06:57 -07:00
Micah Jaffe
c7cd2f1e75 Migrate KinesisProxy to ListShardsWithFilter for empty lease table case 2020-04-30 13:28:27 -07:00
ychunxue
e1111d487e
Merge pull request #699 from ychunxue/v1.x 
Post Release 1.13.3
 2020-03-02 16:50:54 -08:00
Chunxue Yang
418807668d Fixing Snapshot version 2020-03-02 16:45:48 -08:00
Chunxue Yang
f93d01f183 Post Release 1.13.3 2020-03-02 16:43:12 -08:00
ychunxue
74fc856dab
Merge pull request #698 from ychunxue/v1.x 
Prepare for KCL 1.13.3 release
 2020-03-02 14:17:52 -08:00
Chunxue Yang
3cd1bdccf7 Prepare for KCL 1.13.3 release 2020-03-02 13:50:08 -08:00
ychunxue
e9e64f8511
Merge pull request #694 from ychunxue/v1.x 
Stop passing latest shards to the next ShardSyncTask in ShardSyncTask…
 2020-03-02 10:18:24 -08:00
Chunxue Yang
e95344b13e Adding unit test changes 2020-03-02 09:48:00 -08:00
Chunxue Yang
5760a27962 Stop passing latest shards to the next ShardSyncTask in ShardSyncTaskManager. 2020-02-25 16:18:56 -08:00
ychunxue
de52856b45
Merge pull request #684 from parijatsinha/v1.12_fix 
Refactored shard closure verification performed by ShutdownTask
 2020-02-24 09:58:52 -08:00
parijas
94a6324724 Addressed PR comments 2020-02-11 14:13:19 -08:00
parijas
1d50d766f4 Refactored shard closure verification performed by ShutdownTask 2020-02-03 17:00:32 -08:00
Cory Bradshaw
02c2036d5d Preparing for Release 1.13.2 2020-01-13 09:56:16 -08:00
Cory-Bradshaw
a7079a81e4
Adding backward compatible constructors that use the default DDB Billing Mode (#673) 
Co-authored-by: cobrads <44237866+cobrads@users.noreply.github.com>
 2020-01-09 10:10:06 -08:00
ychunxue
ca15bd4ef5
Merge pull request #671 from ychunxue/v1.x 
Prepare KCL release 1.13.1
 2019-12-30 15:20:15 -08:00
Chunxue Yang
53133fd385 Trim the old release notes in README file 2019-12-30 15:11:39 -08:00
Chunxue Yang
5f50f08dcc Prepare KCL release 1.13.1 2019-12-30 15:01:56 -08:00
ychunxue
e8180c9804
Merge pull request #670 from Cory-Bradshaw/v1.x 
Fixing LeaseManagementIntegrationTest
 2019-12-30 14:46:06 -08:00
Cory Bradshaw
40bc62e1f9 Fixing LeaseManagementIntegrationTest by using mocks to setup assumptions about table existance 2019-12-30 14:27:51 -08:00
ychunxue
bd59461c2c
Merge pull request #659 from parijatsinha/bug-fix 
Ensure ShardSyncTask invocation from ShardSyncTaskManager for pending ShardEnd
 2019-12-26 09:44:37 -08:00
parijas
5b3078f801 Addressed review comments 2019-12-17 11:40:20 -08:00
parijas
2d6b92e8ac Ensure ShardSyncTask invocation from ShardSyncTaskManager for pending shard sync requests 2019-12-02 16:05:58 -08:00
Cory-Bradshaw
f369f2114a
Adding DDB BillingMode Support (#656) 
* Adding DDB BillingMode Support
 2019-11-29 14:27:17 -08:00
ychunxue
635a101ab4
Merge pull request #648 from ychunxue/v1.x 
Post Rlease 1.13.0.
Updating KCL to SNAPSHOT version
 2019-11-06 10:42:57 -08:00
Chunxue Yang
a3e720e8e2 Post Rlease 1.13.0 2019-11-06 10:29:04 -08:00
ychunxue
7663673f3b
Merge pull request #646 from ychunxue/v1.x 
Prepare for KCL 1.13.0 release
 2019-11-05 15:25:03 -08:00
Chunxue Yang
bddf573c03 Release Prepare 2019-11-05 14:44:49 -08:00
Chunxue Yang
ebaa5ef788 Update the KCL version to 1.13.0 2019-11-05 14:31:50 -08:00
ychunxue
3947b55853
Merge pull request #640 from aggarwal/better-graceful-shutdown-2 
handling completed and blocked tasks better during graceful shutdown
 2019-11-01 09:53:34 -07:00
Shitanshu Aggarwal
5cb2a3500c handling completed and blocked tasks better during graceful shutdown 2019-10-30 03:11:29 +00:00
Cory-Bradshaw
d9a5789bb2
Merge pull request #632 from jushkem/v1.13.0-snap 
Update version to v1.13.0-SNAPSHOT
 2019-10-18 09:59:30 -07:00
Joshua Kim
cce5b5a620 Update version to v1.13.0-SNAPSHOT 2019-10-18 16:45:37 +00:00
Joshua Kim
1bdd1d1671
Preparing for v1.12.0 (#627) 2019-10-17 17:41:35 -07:00
Joshua Kim
34f5e33450
Updating AWS SDK to 1.11.655 (#626) 2019-10-17 16:55:36 -07:00
ychunxue
ffb34b6d96 Shard End fix for KCL v1.x (#623) 
* KCL shardend fix for V1

* Address Comments

* Address more comments

* Force lease lost before shutting down ShardConsumer with Zombie state

* Updating version

* Addressing comments

* Addressing comments

* Fixing unit test

* Addressing comments

* Adding default implementation for onShardConsumerShutDown in ShardSyncStrategy interface

* Method name changes

* Addressing comments

* Addressing comments

* Addressing comments

* Revert the access change for getShardList method
 2019-10-17 15:14:29 -07:00
Micah Jaffe
cc8aa5ef74
Update Sonatype to dedicated AWS endpoint (#618) 2019-10-04 13:53:21 -07:00
Micah Jaffe
3e36f0c7d0 Update version to 1.11.3-SNAPSHOT (#595) 2019-08-16 14:08:17 -07:00
Micah Jaffe
b537403d81 Preparation for v1.11.2 (#594) 2019-08-15 10:45:00 -07:00
Parijat Sinha
ba984fe279 Adding support for metrics in PeriodicShardSyncer (#592) 
* Changes to support injection of ShardSyncer, LeaseTaker, and LeaseRenewer into KCL Worker

* Additional checks around injection of LeaseRenewer and LeaseRenewerThreadPool

* Changed accessor on InitialPositionInStreamExtended to public to allow ShardSyncer injection

* Changed ShardSyncer to a public interface. Renamed implementation to KinesisShardSyncer.

* Removed wild card imports introduced in previous commit

* Minor refactoring in Worker Builder

* Added license info to ShardSyncer interface. Minor refactoring

* Changes to chain constructor in LeaseCoordinator

* Changed accessor on InitialPositionInStreamExtended factory methods. Minor changes in Worker builder.

* Changes to support periodic shard sync

* Patching changes left out in merge

* Overriding shard-sync idle time to 0 for periodic shard-sync

* Addressed PR feedback

* Addresed PR #579 review comments

* Modified constructor for DeterministicShuffleShardSyncLeaderDecider

* Addressed PR comments

* Fixed failing test

* Removed redundant member varible

* Re-enable metrics for shard-sync

* Removed unused method from ShardSyncTaskManager
 2019-08-13 16:37:57 -07:00
Micah Jaffe
8f58bafacc Update to 1.11.2-SNAPSHOT (#589) 2019-08-13 11:50:35 -07:00
Micah Jaffe
0e910a1dbc Updated logging message around ShardEnd checkpointing failures (#590) 2019-08-13 11:49:40 -07:00
Micah Jaffe
41f996b833
Add logging around shard end codepaths (#585) 
* Add logging around shard end codepaths

* Update logging messaging
 2019-08-09 14:05:08 -07:00
Micah Jaffe
2ca3cbd21c Release 1.11.1 for Amazon Kinesis Client Library for Java (#588) 
* Preparation for v1.11.1

* Preparation for v1.11.1
 2019-08-09 10:27:50 -07:00
Micah Jaffe
c00203f028 Update AWS SDK to version 1.11.603 (#587) 2019-08-09 09:53:25 -07:00