Commit graph

99 commits

Author SHA1 Message Date
furq-aws
48fab1fe71
Modify RecordProcessorCheckpointer#advancePosition Metrics usage to ensure proper closure (#1224) 2024-02-23 10:22:20 -08:00
Brendan Lynch
e731486b7f
Updated Guava version, added backoff for ShardSync Integ test, and reintroduced SNAPSHOT version (#1214)
* Updated guava version to 32.1.1 and added backoff logic for ShardSync Integration test

* Added back snapshot version classifier

---------

Co-authored-by: Brendan Lynch <brenplyn@amazon.com>
2023-10-11 14:54:49 -07:00
lucienlu-aws
db30ac956b
Add support for Stream ARNs (#1108)
Add support for referencing streams by streamARN
2023-06-07 13:55:51 -07:00
lucienlu-aws
fe1a34f6e5
Prevent HashRangesAreAlwaysComplete test from building invalid hierarchy trees (#1111) 2023-05-17 16:41:28 -07:00
gguptp
372f98b21a
Every other change for DynamoDBStreamsKinesis Adapter Compatibility (#995)
Co-authored-by: Nicholas Gutierrez <nichgu@amazon.com>
2022-10-06 10:08:50 -07:00
catalinclapon
cf71e4e4e4
Update the Worker shutdown logic to make sure that the LeaseCleanupManager also terminates all the threads that it has started (#816)
* Update the Worker shutdown logic to make sure that the LeaseCleanupManager also terminates all the threads that it has started

* Minor javadoc fixes
2021-05-13 12:53:52 -07:00
Joshua Kim
f2b9006a98
Removing cached shard progress, adding guardrails for duplicate shard responses. (#811)
Co-authored-by: Joshua Kim <kimjos@amazon.com>
2021-05-03 13:50:54 -07:00
Ashwin Giridharan
9ed1e10fe5 Fixing unit test 2021-01-26 23:21:39 -08:00
Ashwin Giridharan
5142ce9bc5 Fixing exception type 2021-01-26 21:40:22 -08:00
Ashwin Giridharan
b2a747f3ea 1. Update the checkpoint with non SHARD_END sequence number only if the DDB sequence number is not SHARD_END.
2. Verify the shard end checkpointing by directly looking up the ddb lease entry
2021-01-25 23:02:41 -08:00
Ashwin Giridharan
9cb5020022 1. Fix for updating HashRange directly while creating leases and including HashRange in seralizing and deserializing to/from DDB
2. Fix for making LeaseCleanupManager non-singleton to avoid cross-table interference in multiple apps running in same JVM
3. Fixing updateMetaInfo method to not update other lease table fields
4. Preventing shard deletion in LeaseCleanupManager if a valid shard does not have child shards in lease table and in Kinesis Service
5. Adding childshards update support in updateMetaInfo
6. Fixing LeaseCleanupManager to call updateMetaInfo instead of update for childshard update in lease

7. Fixing unit tests to accommodate HashRange changes
2021-01-23 17:58:42 -08:00
Joshua Kim
700daa7a87 Fixing cache miss bug in list shards 2020-08-17 15:45:41 -04:00
Cai41
41c08c3b04 Also consider TableStatus.UPDATING status 2020-08-17 15:45:41 -04:00
Chunxue Yang
32fc14f535 Rebasing and addressing comments 2020-07-31 13:51:30 -07:00
Chunxue Yang
22737c4a5b addressing comments 2020-07-28 17:51:50 -07:00
Chunxue Yang
f713017528 LeaseCleanupManager change 2020-07-28 13:11:57 -07:00
Chunxue Yang
c3b41c3b55 refactoring shutdownTask 2020-07-27 18:18:32 -07:00
Chunxue Yang
5f7d4b3bc6 merging and resolving conflicts 2020-07-27 15:19:35 -07:00
Joshua Kim
6f16b168a4
Lease cleanup v1.x (#73)
* Moving lease cleanup to ShutdownTask.
* Introducing LeaseCleanupManager with relevant configs.
2020-07-27 16:08:52 -04:00
Chunxue Yang
8a296a5aa3 change the retry logic 2020-07-21 11:08:56 -07:00
Chunxue Yang
5bf24bda43 fix for premature childShard lease creation 2020-07-15 12:33:53 -07:00
Chunxue Yang
995511afe5 Fixing updateParentShardupdateCurrentLeaseWithChildShards method 2020-07-09 16:03:49 -07:00
Chunxue Yang
849ffea028 Periodic shard sync metrics and config 2020-07-09 12:07:30 -07:00
Chunxue Yang
058403f943 addressing comments 2020-07-09 11:20:37 -07:00
Chunxue Yang
36ae2f2428 Bug fix: No longer need full shard sync for shardEnd 2020-07-08 17:45:02 -07:00
Chunxue Yang
5ba87a34ab Merge remote-tracking branch 'ashwin/ltr-v1.x' into periodicShardSyncMetric 2020-07-07 17:17:50 -07:00
ychunxue
53cc7fc347
Merge pull request #67 from ychunxue/ltr1x_latest
KCL 1.x ShardEnd Shard Sync and Lease table ChildShard persistence
2020-07-07 13:49:53 -07:00
Chunxue Yang
0a5724c332 KCL 1.x ShardEnd Shard Sync and Lease table ChildShard persistence 2020-07-06 15:10:17 -07:00
Joshua Kim
0760688375
KinesisShardSyncer only create leases for one level of leases (#49)
Co-authored-by: Joshua Kim <kimjos@amazon.com>
2020-07-06 14:50:21 -04:00
Chunxue Yang
57f8c120d0 Periodc Shard Sync Metrics and Config 2020-07-02 10:15:44 -07:00
Micah Jaffe
3a88a60a4e
Add periodic hash range auditor, hole detection, and recovery (#43)
* Add periodic hash range auditor, hole detection, and recovery

* Add unit tests for hash range hole recovery

* Fix max hash range bug

* Address PR feedback

* Fix DDB hash key persistence bug
2020-06-22 08:45:46 -07:00
Micah Jaffe
6b474b7390
Add configurable max initialization attempts (#39)
* Add configurable max initialization attempts

* Add maxInitializationAttempts to unit test setup
2020-05-15 14:02:44 -07:00
Micah Jaffe
fdad20aff4
Add leader-elected periodic shard sync manager as auditor to ShardEndShardSync strategy (#35)
* Add leader-elected periodic shard sync manager as auditor to ShardEndShardSync strategy

* Address PR feedback and add Builder unit tests
2020-05-14 09:04:44 -07:00
Micah Jaffe
45e76b0fd5
Add leader-elected shard sync on application bootstrap (#28)
* Add leader-elected shard sync on application bootstrap

* Add additional testing and address PR feedback

* Remove runShardSync leader election from boostrap case

* Remove random UUID workerId and update unit tests
2020-05-07 17:08:53 -07:00
Micah Jaffe
550d7af5b1
Add lease sync strategy for empty lease table (#24)
* Add lease sync strategy for empty lease table

* Fix ShardSyncer unit tests to reflect new empty lease table shard sync logic
2020-05-04 17:06:57 -07:00
Micah Jaffe
c7cd2f1e75 Migrate KinesisProxy to ListShardsWithFilter for empty lease table case 2020-04-30 13:28:27 -07:00
Chunxue Yang
e95344b13e Adding unit test changes 2020-03-02 09:48:00 -08:00
parijas
94a6324724 Addressed PR comments 2020-02-11 14:13:19 -08:00
parijas
1d50d766f4 Refactored shard closure verification performed by ShutdownTask 2020-02-03 17:00:32 -08:00
Cory Bradshaw
40bc62e1f9 Fixing LeaseManagementIntegrationTest by using mocks to setup assumptions about table existance 2019-12-30 14:27:51 -08:00
ychunxue
bd59461c2c
Merge pull request #659 from parijatsinha/bug-fix
Ensure ShardSyncTask invocation from ShardSyncTaskManager for pending ShardEnd
2019-12-26 09:44:37 -08:00
parijas
5b3078f801 Addressed review comments 2019-12-17 11:40:20 -08:00
parijas
2d6b92e8ac Ensure ShardSyncTask invocation from ShardSyncTaskManager for pending shard sync requests 2019-12-02 16:05:58 -08:00
Cory-Bradshaw
f369f2114a
Adding DDB BillingMode Support (#656)
* Adding DDB BillingMode Support
2019-11-29 14:27:17 -08:00
Shitanshu Aggarwal
5cb2a3500c handling completed and blocked tasks better during graceful shutdown 2019-10-30 03:11:29 +00:00
ychunxue
ffb34b6d96 Shard End fix for KCL v1.x (#623)
* KCL shardend fix for V1

* Address Comments

* Address more comments

* Force lease lost before shutting down ShardConsumer with Zombie state

* Updating version

* Addressing comments

* Addressing comments

* Fixing unit test

* Addressing comments

* Adding default implementation for onShardConsumerShutDown in ShardSyncStrategy interface

* Method name changes

* Addressing comments

* Addressing comments

* Addressing comments

* Revert the access change for getShardList method
2019-10-17 15:14:29 -07:00
Parijat Sinha
7a1d3031c5 Introducing support for periodic shard sync (#579)
* Changes to support injection of ShardSyncer, LeaseTaker, and LeaseRenewer into KCL Worker

* Additional checks around injection of LeaseRenewer and LeaseRenewerThreadPool

* Changed accessor on InitialPositionInStreamExtended to public to allow ShardSyncer injection

* Changed ShardSyncer to a public interface. Renamed implementation to KinesisShardSyncer.

* Removed wild card imports introduced in previous commit

* Minor refactoring in Worker Builder

* Added license info to ShardSyncer interface. Minor refactoring

* Changes to chain constructor in LeaseCoordinator

* Changed accessor on InitialPositionInStreamExtended factory methods. Minor changes in Worker builder.

* Changes to support periodic shard sync

* Patching changes left out in merge

* Overriding shard-sync idle time to 0 for periodic shard-sync

* Addressed PR feedback

* Addresed PR #579 review comments

* Modified constructor for DeterministicShuffleShardSyncLeaderDecider

* Addressed PR comments

* Fixed failing test

* Removed redundant member varible
2019-08-07 15:37:20 -07:00
Anoop Garlapati
3aefdfa28a Better handling of get checkpoint object in KinesisClientLibLeaseCoordinator (#558)
Improve the handling of get checkpoint object in KinesisClientLibLeaseCoordinator
by no longer triggering a null pointer exception when lease is null.
2019-07-09 11:31:30 -07:00
awslankakamal
bb81bfaa66 Updating license to Apache License 2.0 (#522) 2019-04-05 16:08:24 -07:00
achitojha
fbdd449759 Separated out some lease taker logic as interface implementations to … (#490)
Added interfaces to allow external users to control the lease selection, and cleanup.
2019-02-20 11:48:40 -08:00