luciano/amazon-kinesis-client
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
455 commits 48 branches 124 tags 11 MiB
Commit graph

455 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ashwin Giridharan
b636edd007 ShutdownTask bug fix 2020-06-24 13:07:42 -07:00
Ashwin Giridharan
bd6a7d8b90 Logging to track pending leases to cleanup 2020-06-24 12:43:55 -07:00
Ashwin Giridharan
a2b6019336 Changes to enqueue lease for deletion in potential garbage cleanup scenario. 2020-06-24 12:39:18 -07:00
ashwing
1b86184428
Merge pull request #64 from ashwing/ltr_1_leader_stream_sync 
Ltr 1 leader stream sync
 2020-06-23 15:45:04 -07:00
Ashwin Giridharan
67ce7a783d Fixing lease cleanup issues with multistreaming 2020-06-23 03:03:42 -07:00
Ashwin Giridharan
0246e1e852 Only making leader to do the stream sync 2020-06-22 14:30:28 -07:00
ychunxue
db2c22e046
Merge pull request #61 from ychunxue/ltr_1_createChildLeaseFix 
search child shards in DDB instead of in local cache
 2020-06-17 12:10:25 -07:00
ashwing
7bb60b0c69
Merge pull request #62 from ashwing/ltr_1_skip_shardsync_on_empty_stream_shards 
Skipping ddb scan and shard sync for streams that has no shards retur…
 2020-06-17 11:57:48 -07:00
Chunxue Yang
27613763ac search child shards in DDB instead of in local cache 2020-06-17 11:31:22 -07:00
Ashwin Giridharan
8d40e5bdd4 Skipping ddb scan and shard sync for streams that has no shards returned from service 2020-06-17 10:49:57 -07:00
ashwing
50a1fe8922
Merge pull request #60 from ashwing/ltr_1_old_streams_cleanup 
Removing the old streams from active streams list without cleaning up the leases
 2020-06-17 10:40:49 -07:00
Ashwin Giridharan
69a05b409b Removing the old streams from active streams list without cleaning up the leases 2020-06-17 10:25:11 -07:00
ashwing
6021261b55
Merge pull request #59 from Renjuju/cherry-pick-rnf-exception 
[Cherry-pick from multi_stream_2] Empty lease response for ResourceNotFound exceptions (#27)
 2020-06-16 15:44:36 -07:00
Renju Radhakrishnan
4a89c26f6f Empty lease response for ResourceNotFound exceptions (#27) 2020-06-16 13:42:20 -07:00
ashwing
562face833
Merge pull request #55 from ashwing/ltr_1_periodic_auditor_metrics_configs 
Ltr 1 periodic auditor metrics configs
 2020-06-16 10:21:23 -07:00
ashwing
a24e7dcf03
Merge pull request #57 from Renjuju/cherry-pick-leases-list-to-set 
[Cherry-pick] Convert expiredLeases to set (#56)
 2020-06-16 10:19:45 -07:00
Ashwin Giridharan
e7e8196f29 Fixed typo 2020-06-16 10:09:13 -07:00
ashwing
65388573f9
Merge pull request #58 from ashwing/ltr_1_lease_cleanup_fix 
Lease cleanup handling for garbage shard
 2020-06-16 09:10:46 -07:00
Ashwin Giridharan
374e47b208 updating wereChildShardsPresent hint in all cases 2020-06-15 23:13:47 -07:00
Ashwin Giridharan
4d65f56038 Added comments on the garbage cleanup logic - 2 2020-06-15 22:02:33 -07:00
Ashwin Giridharan
7b0dc0d3e5 Added comments on the garbage cleanup logic 2020-06-15 22:00:56 -07:00
Ashwin Giridharan
8ec14baca1 Handlign garbage shard case while delivering lease cleanup 2020-06-15 21:53:30 -07:00
Ashwin Giridharan
b60dd60f35 Code refactoring - 1 2020-06-15 21:20:33 -07:00
Ashwin Giridharan
37281e9493 Deleting lease immediately in RNF Exception 2020-06-15 20:57:55 -07:00
Renju Radhakrishnan
aaf37be0cd Convert expiredLeases to set (#56) 2020-06-15 11:01:44 -07:00
Ashwin Giridharan
c1cbb6cf6d Removing one of the lease clean up strategy as it is no longer required with distributed lease cleanup 2020-06-12 22:22:18 -07:00
Ashwin Giridharan
ce38178399 Caching consumerArn for StreamIdentifier in FanOutRetrievalFactory 2020-06-12 22:20:38 -07:00
Ashwin Giridharan
2530481dba Periodic Shard Sync Auditor - Metrics and Customer auditor configs 2020-06-12 18:12:02 -07:00
ashwing
1aeb8ed9a1
Merge pull request #53 from ashwing/ltr_ddb_scan_opt 
Limiting max page for ddb scan in isEmptyLeaseTable check to be 1
 2020-06-11 18:42:48 -07:00
Ashwin Giridharan
6fc654802c Limiting max page for ddb scan in isEmptyLeaseTable check to be 1 2020-06-11 18:36:42 -07:00
Joshua Kim
c8422745d9
Removing lease cleanup from HierarchicalShardSyncer (#30) 
* Removing lease cleanup from HierarchicalShardSyncer
 2020-06-11 19:56:04 -04:00
Joshua Kim
2e2211a9b7
Merge pull request #31 from ashwing/ShutdownTaskCleanup 
* Adding LeaseCleanupManager
* Added lease cleanup unit tests
* Adding lease cleanup path to shard consumer shutdown
 2020-06-11 19:48:58 -04:00
Joshua Kim
445dd106b2 Adding logging 2020-06-11 19:48:24 -04:00
Joshua Kim
dc078ead39 Reverting pom change 2020-06-10 19:49:06 -04:00
Joshua Kim
2f1838483c Adding Lease cleanup in shutdown task. 2020-06-10 19:45:11 -04:00
Joshua Kim
c7cdbd5d8b pom change 2020-06-10 18:57:53 -04:00
ashwing
61aeae3ea1
Merge pull request #48 from ashwing/ltr_1_hsyncperstreamconfig 
Hierachical stream syncer per stream config changes
 2020-06-10 10:35:24 -07:00
ashwing
291d1b4059
Merge pull request #44 from ashwing/ltr_1_ms_consumerarn 
Dedicated consumerArn for streams in multiStream mode
 2020-06-09 18:18:21 -07:00
Ashwin Giridharan
b323e7c487 Introducing dedicated shard syncer for each of the streamconfig 2020-06-09 17:14:02 -07:00
Ashwin Giridharan
700c7a563a Adding unit test case 2020-06-07 21:17:24 -07:00
Ashwin Giridharan
b5d0301b31 Adding stream information to Fanout Consumer Registration logs 2020-06-03 01:08:13 -07:00
Ashwin Giridharan
f69398a2b2 Adding dedicated consumerArn support for streams in multistreaming mode 2020-06-03 00:48:11 -07:00
Micah Jaffe
113029e33c
Update lease sync algorithm to create leases for no more than one level of the shard hierarchy (#41) 
* Update NonEmptyLeaseTableSynchronizer algorithm to create leases for no more than one level of the shard hierarchy

* Add and fix unit tests for LATEST, TRIM_HORIZON, AT_TIMESTAMP

* Update unit tests

* Fix edge case of reading from TRIM with partial lease for one child shard

* Revert child shards optimization for TRIM/TIMESTAMP case
 2020-05-31 10:28:05 -07:00
Ashwin Giridharan
f2ba3bcd2f Changes for partial lease table hole detection, missing hash range detection and recovery 2020-05-29 09:10:14 -07:00
Ashwin Giridharan
67d2b082fd Fixing retry logic 2020-05-26 11:10:10 -07:00
Ashwin Giridharan
08ca1b61bc Changing test case name 2020-05-25 13:37:10 -07:00
Ashwin Giridharan
72a6d5e084 Adding more unit test cases and fixing an edge case 2020-05-25 13:25:43 -07:00
Ashwin Giridharan
fc4781e347 Addressed review comments 2020-05-21 00:57:24 -07:00
ychunxue
52f1be5c67
Merge pull request #32 from ychunxue/ltr_1_updateLeaseFix 
Update Parent lease fix
 2020-05-07 14:58:24 -07:00
Ashwin Giridharan
02ea8cd70f Added TODO for childshards update fix 2020-05-07 01:36:51 -07:00