luciano/amazon-kinesis-client
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
510 commits 48 branches 124 tags 11 MiB
Commit graph

510 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ashwin Giridharan
710b784f73 KCL changes pertaining to distributed lease discovery, distributed lease cleanup, removing redundant shasrd sync and lease uptaste operastions. Refer release notes for more details. 2020-08-10 11:33:08 -07:00
Renju Radhakrishnan
27494ea043
Set expectation criteria to use leaseKey (#88) 
* Set expectation criteria to use leaseKey

* Set exists to true

* Add integration tests for leaseMetaInfo update
 2020-07-28 15:18:40 -07:00
ashwing
ca661bf30d
Merge pull request #85 from joshua-kim/ltr_1 
Fixing bug where isRunning is not set
 2020-07-27 16:42:51 -07:00
ashwing
11034dd5b8
Merge pull request #86 from ashwing/ltr_1_lease_update_cond_check 
Add conditional check while updating the lease table meta info
 2020-07-27 16:40:50 -07:00
Joshua Kim
a48141682c Moving debug to info logs to startup 2020-07-27 19:39:46 -04:00
Ashwin Giridharan
e2a1d71053 Update exception msg 2020-07-27 16:37:40 -07:00
Joshua Kim
b7a5d7968b Removing unnecessary code 2020-07-27 19:31:52 -04:00
Ashwin Giridharan
bdf019748f Add conditional check while updating the lease table meta info 2020-07-27 16:26:51 -07:00
Joshua Kim
15c45d2029 Adding a unit test 2020-07-27 19:17:27 -04:00
Joshua Kim
cadbc0503b reverting whitespace 2020-07-27 18:15:57 -04:00
Joshua Kim
9c8b89e0c6 Moving to after watches and executor initialized 2020-07-27 18:14:11 -04:00
Joshua Kim
5a0a6c600e Adding safety check around stopwatches 2020-07-27 18:08:35 -04:00
Joshua Kim
5470bc7488 Revert "Pom change to 2.10.65" 
This reverts commit 44f2cb7d5b.
 2020-07-27 17:59:29 -04:00
Joshua Kim
bc5dae270f Fixing bug where isRunning is not set 2020-07-27 17:58:42 -04:00
Joshua Kim
44f2cb7d5b Pom change to 2.10.65 2020-07-27 17:58:42 -04:00
ashwing
e235777c17
Merge pull request #83 from ashwing/ltr_1_shardsync_executor_overload_fix 
Avoiding ShardSync Task sleep when we skip the shard sync due to no s…
 2020-07-22 12:16:47 -07:00
Ashwin Giridharan
75fe6060f6 Adding logging around the shard sync executor queuesize 2020-07-22 11:37:25 -07:00
Ashwin Giridharan
8aec062e64 Avoiding ShardSync Task sleep when we skip the shard sync due to no shards found 2020-07-22 10:36:22 -07:00
Joshua Kim
ff703459e1
Exposing interface for getting child shards (#81) 
* Exposing interface for getting child shards

* javadocs

* more java docs

* Addressing comments

* Tuning random max range

Co-authored-by: Joshua Kim <kimjos@amazon.com>
 2020-07-17 18:32:50 -04:00
Renju Radhakrishnan
c68ab705bd
Fix NonGreedyTake integration test and add extra integration test (#80) 
* Fix NonGreedyTake integration test and add extra integration test for very old leases

* Fix spacing in comment
 2020-07-17 14:48:25 -07:00
ashwing
afd0f7c08a
Merge pull request #77 from ashwing/ltr_1_shutdown_probability_based_exception 
Breaking loop on ShutdownException using 10-faced dice
 2020-07-16 20:42:26 -07:00
ashwing
f46bd444d4
Merge pull request #78 from ashwing/ltr_1_rnf_log 
Log details of ResourceNotFoundException while cleaning up leases
 2020-07-16 20:41:39 -07:00
Ashwin Giridharan
4c899b2930 Log details of ResourceNotFoundException while cleaning up leases 2020-07-16 20:34:48 -07:00
Ashwin Giridharan
60b6066189 Breaking loop on ShutdownException using 10-faced dice 2020-07-16 12:14:53 -07:00
Ashwin Giridharan
7b704246e2 Fixing NPE in log 2020-07-15 14:52:12 -07:00
ashwing
4457ebc350
Merge pull request #74 from ashwing/ltr_1_fix_for_premature_childshard_creation 
Ltr 1 fix for premature childshard creation
 2020-07-15 11:35:07 -07:00
Ashwin Giridharan
264e5bd4aa Shutdown task to throw retryable exception intermittently to allow parent shard lease detection in merge cases. 2020-07-15 10:08:50 -07:00
Ashwin Giridharan
d9f5557ff5 Added unit test cases 2020-07-14 16:59:31 -07:00
Ashwin Giridharan
ff1bee5791 Fix for avoiding premature creation of child lease when other parent lease is not created yet. This can happen in the case of fast reshard operations. 2020-07-14 16:59:31 -07:00
ashwing
5fc55e0c8c
Merge pull request #68 from ychunxue/childShardValidation 
ChildShard validation in Data Fetcher level and error handling in shu…
 2020-07-14 16:33:53 -07:00
Chunxue Yang
85a5423657 Addressing more comments 2020-07-14 15:02:04 -07:00
Chunxue Yang
a0094b0df8 Addressing comments 2020-07-08 12:33:16 -07:00
Renju Radhakrishnan
347ae9eb68
Add very old leases + update lease taker to always emit metrics (#54) 
* Add very old leases + metrics update

* Add conditional emisison for metrics in case of very old leases

* move code blocks around to ensure all metrics are emitted
 2020-07-07 16:58:40 -07:00
Chunxue Yang
b49d8ea8cc ChildShard validation in Data Fetcher level and error handling in shutdownTask 2020-07-07 16:02:12 -07:00
Joshua Kim
74ffd4060c Revert "Fixing bug for multistream not using list shards with filter. (#66)" 
This reverts commit f693311ac8.
 2020-07-01 19:07:50 -04:00
Joshua Kim
f693311ac8
Fixing bug for multistream not using list shards with filter. (#66) 
Fixing bug for mulitstreaming not using list shards with filter.
 2020-07-01 19:01:13 -04:00
ashwing
842c755511
Merge pull request #65 from ashwing/ltr_1_lease_cleanup_logging 
Ltr 1 lease cleanup logging
 2020-06-24 17:15:07 -07:00
Ashwin Giridharan
950faf7475 Fixing condition check 2020-06-24 14:34:01 -07:00
Ashwin Giridharan
b636edd007 ShutdownTask bug fix 2020-06-24 13:07:42 -07:00
Cai41
87677255e1 DDB is considered healthy in UPDATING status 2020-06-24 13:07:24 -07:00
Ashwin Giridharan
bd6a7d8b90 Logging to track pending leases to cleanup 2020-06-24 12:43:55 -07:00
Ashwin Giridharan
a2b6019336 Changes to enqueue lease for deletion in potential garbage cleanup scenario. 2020-06-24 12:39:18 -07:00
ashwing
98d723b576
Merge pull request #63 from ashwing/ltr_1_duplicate_recseq_fix 
Fixing Prefetch publisher cache restart issue
 2020-06-23 17:36:16 -07:00
ashwing
1b86184428
Merge pull request #64 from ashwing/ltr_1_leader_stream_sync 
Ltr 1 leader stream sync
 2020-06-23 15:45:04 -07:00
Ashwin Giridharan
60af78f7cb Adding unit test case for validating internal state on initial prefetcher failures 2020-06-23 14:49:15 -07:00
Ashwin Giridharan
67ce7a783d Fixing lease cleanup issues with multistreaming 2020-06-23 03:03:42 -07:00
Ashwin Giridharan
ef39ecd0df Log changes 2020-06-22 23:37:38 -07:00
Ashwin Giridharan
d4f3c0b14a Fixing Prefetch publisher cache restart issue 2020-06-22 23:23:41 -07:00
Ashwin Giridharan
0246e1e852 Only making leader to do the stream sync 2020-06-22 14:30:28 -07:00
ychunxue
db2c22e046
Merge pull request #61 from ychunxue/ltr_1_createChildLeaseFix 
search child shards in DDB instead of in local cache
 2020-06-17 12:10:25 -07:00