Commit graph

  • c3b41c3b55 refactoring shutdownTask Chunxue Yang 2020-07-27 18:18:32 -0700
  • ca661bf30d
    Merge pull request #85 from joshua-kim/ltr_1 ashwing 2020-07-27 16:42:51 -0700
  • 11034dd5b8
    Merge pull request #86 from ashwing/ltr_1_lease_update_cond_check ashwing 2020-07-27 16:40:50 -0700
  • a48141682c Moving debug to info logs to startup Joshua Kim 2020-07-27 19:39:46 -0400
  • e2a1d71053 Update exception msg Ashwin Giridharan 2020-07-27 16:37:40 -0700
  • b7a5d7968b Removing unnecessary code Joshua Kim 2020-07-27 19:31:52 -0400
  • bdf019748f Add conditional check while updating the lease table meta info Ashwin Giridharan 2020-07-27 16:26:51 -0700
  • 15c45d2029 Adding a unit test Joshua Kim 2020-07-27 19:17:27 -0400
  • 5f7d4b3bc6 merging and resolving conflicts Chunxue Yang 2020-07-27 15:19:35 -0700
  • cadbc0503b reverting whitespace Joshua Kim 2020-07-27 18:15:57 -0400
  • 9c8b89e0c6 Moving to after watches and executor initialized Joshua Kim 2020-07-27 18:14:11 -0400
  • 5a0a6c600e Adding safety check around stopwatches Joshua Kim 2020-07-27 18:08:35 -0400
  • 5470bc7488 Revert "Pom change to 2.10.65" Joshua Kim 2020-07-27 17:59:29 -0400
  • bc5dae270f Fixing bug where isRunning is not set Joshua Kim 2020-07-27 17:58:39 -0400
  • 44f2cb7d5b Pom change to 2.10.65 Joshua Kim 2020-07-21 06:09:27 -0400
  • 6f16b168a4
    Lease cleanup v1.x (#73) Joshua Kim 2020-07-27 16:08:52 -0400
  • 4bd195320d
    Bump commons-beanutils in /amazon-kinesis-client-multilang dependabot[bot] 2020-07-27 14:25:27 +0000
  • e235777c17
    Merge pull request #83 from ashwing/ltr_1_shardsync_executor_overload_fix ashwing 2020-07-22 12:16:47 -0700
  • 75fe6060f6 Adding logging around the shard sync executor queuesize Ashwin Giridharan 2020-07-22 11:37:25 -0700
  • 8aec062e64 Avoiding ShardSync Task sleep when we skip the shard sync due to no shards found Ashwin Giridharan 2020-07-22 10:36:22 -0700
  • 8a296a5aa3 change the retry logic Chunxue Yang 2020-07-21 11:08:56 -0700
  • ff703459e1
    Exposing interface for getting child shards (#81) Joshua Kim 2020-07-17 18:32:50 -0400
  • c68ab705bd
    Fix NonGreedyTake integration test and add extra integration test (#80) Renju Radhakrishnan 2020-07-17 14:48:25 -0700
  • afd0f7c08a
    Merge pull request #77 from ashwing/ltr_1_shutdown_probability_based_exception ashwing 2020-07-16 20:42:26 -0700
  • f46bd444d4
    Merge pull request #78 from ashwing/ltr_1_rnf_log ashwing 2020-07-16 20:41:39 -0700
  • 4c899b2930 Log details of ResourceNotFoundException while cleaning up leases Ashwin Giridharan 2020-07-16 20:34:48 -0700
  • 089c6ab18e
    Making isLeaseTableEmpty use one paginated scan call. (#75) Joshua Kim 2020-07-16 16:27:25 -0400
  • 60b6066189 Breaking loop on ShutdownException using 10-faced dice Ashwin Giridharan 2020-07-16 12:14:53 -0700
  • 7b704246e2 Fixing NPE in log Ashwin Giridharan 2020-07-15 14:52:12 -0700
  • 5bf24bda43 fix for premature childShard lease creation Chunxue Yang 2020-07-15 12:33:53 -0700
  • 4457ebc350
    Merge pull request #74 from ashwing/ltr_1_fix_for_premature_childshard_creation ashwing 2020-07-15 11:35:07 -0700
  • 264e5bd4aa Shutdown task to throw retryable exception intermittently to allow parent shard lease detection in merge cases. Ashwin Giridharan 2020-07-15 10:08:50 -0700
  • d9f5557ff5 Added unit test cases Ashwin Giridharan 2020-07-14 01:58:01 -0700
  • ff1bee5791 Fix for avoiding premature creation of child lease when other parent lease is not created yet. This can happen in the case of fast reshard operations. Ashwin Giridharan 2020-07-14 01:17:05 -0700
  • 5fc55e0c8c
    Merge pull request #68 from ychunxue/childShardValidation ashwing 2020-07-14 16:33:53 -0700
  • 85a5423657 Addressing more comments Chunxue Yang 2020-07-14 15:02:04 -0700
  • a1eafbb69a Addressing comments Chunxue Yang 2020-07-10 10:43:29 -0700
  • 995511afe5 Fixing updateParentShardupdateCurrentLeaseWithChildShards method Chunxue Yang 2020-07-09 16:03:49 -0700
  • 849ffea028 Periodic shard sync metrics and config Chunxue Yang 2020-07-09 12:07:30 -0700
  • 4fd63989d3
    Merge pull request #70 from ychunxue/ltr-v1.x-bugfix ychunxue 2020-07-09 11:26:24 -0700
  • 058403f943 addressing comments Chunxue Yang 2020-07-09 11:20:37 -0700
  • 36ae2f2428 Bug fix: No longer need full shard sync for shardEnd Chunxue Yang 2020-07-08 17:45:02 -0700
  • a0094b0df8 Addressing comments Chunxue Yang 2020-07-08 12:33:16 -0700
  • 5ba87a34ab Merge remote-tracking branch 'ashwin/ltr-v1.x' into periodicShardSyncMetric Chunxue Yang 2020-07-07 17:17:50 -0700
  • 347ae9eb68
    Add very old leases + update lease taker to always emit metrics (#54) Renju Radhakrishnan 2020-07-07 16:58:40 -0700
  • b49d8ea8cc ChildShard validation in Data Fetcher level and error handling in shutdownTask Chunxue Yang 2020-07-07 16:02:12 -0700
  • 53cc7fc347
    Merge pull request #67 from ychunxue/ltr1x_latest ychunxue 2020-07-07 13:49:53 -0700
  • 0a5724c332 KCL 1.x ShardEnd Shard Sync and Lease table ChildShard persistence Chunxue Yang 2020-05-13 22:09:29 -0700
  • 0760688375
    KinesisShardSyncer only create leases for one level of leases (#49) Joshua Kim 2020-07-06 14:50:21 -0400
  • 57f8c120d0 Periodc Shard Sync Metrics and Config Chunxue Yang 2020-07-02 10:15:44 -0700
  • 74ffd4060c Revert "Fixing bug for multistream not using list shards with filter. (#66)" Joshua Kim 2020-07-01 19:07:50 -0400
  • f693311ac8
    Fixing bug for multistream not using list shards with filter. (#66) Joshua Kim 2020-07-01 19:01:13 -0400
  • b059fde302
    Merge f625aea63d into 87677255e1 jegoshj 2020-06-29 14:40:52 +0200
  • 264f1a7006
    Merge 14915fc01c into 87677255e1 Kirill Kozlov 2020-06-28 13:32:01 +0000
  • 842c755511
    Merge pull request #65 from ashwing/ltr_1_lease_cleanup_logging ashwing 2020-06-24 17:15:07 -0700
  • 950faf7475 Fixing condition check Ashwin Giridharan 2020-06-24 14:34:01 -0700
  • b636edd007 ShutdownTask bug fix Ashwin Giridharan 2020-06-24 13:07:42 -0700
  • 87677255e1 DDB is considered healthy in UPDATING status Cai41 2020-06-05 15:45:22 -0700
  • bd6a7d8b90 Logging to track pending leases to cleanup Ashwin Giridharan 2020-06-24 12:43:55 -0700
  • a2b6019336 Changes to enqueue lease for deletion in potential garbage cleanup scenario. Ashwin Giridharan 2020-06-24 12:39:18 -0700
  • 98d723b576
    Merge pull request #63 from ashwing/ltr_1_duplicate_recseq_fix ashwing 2020-06-23 17:36:16 -0700
  • 1b86184428
    Merge pull request #64 from ashwing/ltr_1_leader_stream_sync ashwing 2020-06-23 15:45:04 -0700
  • 60af78f7cb Adding unit test case for validating internal state on initial prefetcher failures Ashwin Giridharan 2020-06-23 14:49:15 -0700
  • 67ce7a783d Fixing lease cleanup issues with multistreaming Ashwin Giridharan 2020-06-23 03:03:42 -0700
  • ef39ecd0df Log changes Ashwin Giridharan 2020-06-22 23:37:38 -0700
  • d4f3c0b14a Fixing Prefetch publisher cache restart issue Ashwin Giridharan 2020-06-22 23:23:41 -0700
  • 0246e1e852 Only making leader to do the stream sync Ashwin Giridharan 2020-06-22 14:30:28 -0700
  • 3a88a60a4e
    Add periodic hash range auditor, hole detection, and recovery (#43) Micah Jaffe 2020-06-22 08:45:46 -0700
  • db2c22e046
    Merge pull request #61 from ychunxue/ltr_1_createChildLeaseFix ychunxue 2020-06-17 12:10:25 -0700
  • 7bb60b0c69
    Merge pull request #62 from ashwing/ltr_1_skip_shardsync_on_empty_stream_shards ashwing 2020-06-17 11:57:48 -0700
  • 27613763ac search child shards in DDB instead of in local cache Chunxue Yang 2020-06-17 11:31:22 -0700
  • 8d40e5bdd4 Skipping ddb scan and shard sync for streams that has no shards returned from service Ashwin Giridharan 2020-06-17 10:49:57 -0700
  • 50a1fe8922
    Merge pull request #60 from ashwing/ltr_1_old_streams_cleanup ashwing 2020-06-17 10:40:49 -0700
  • 69a05b409b Removing the old streams from active streams list without cleaning up the leases Ashwin Giridharan 2020-06-17 10:25:11 -0700
  • 6021261b55
    Merge pull request #59 from Renjuju/cherry-pick-rnf-exception ashwing 2020-06-16 15:44:36 -0700
  • 4a89c26f6f Empty lease response for ResourceNotFound exceptions (#27) Renju Radhakrishnan 2020-05-05 21:06:10 -0700
  • 562face833
    Merge pull request #55 from ashwing/ltr_1_periodic_auditor_metrics_configs ashwing 2020-06-16 10:21:23 -0700
  • a24e7dcf03
    Merge pull request #57 from Renjuju/cherry-pick-leases-list-to-set ashwing 2020-06-16 10:19:45 -0700
  • e7e8196f29 Fixed typo Ashwin Giridharan 2020-06-16 10:09:13 -0700
  • 65388573f9
    Merge pull request #58 from ashwing/ltr_1_lease_cleanup_fix ashwing 2020-06-16 09:10:46 -0700
  • 374e47b208 updating wereChildShardsPresent hint in all cases Ashwin Giridharan 2020-06-15 23:13:47 -0700
  • 4d65f56038 Added comments on the garbage cleanup logic - 2 Ashwin Giridharan 2020-06-15 22:02:33 -0700
  • 7b0dc0d3e5 Added comments on the garbage cleanup logic Ashwin Giridharan 2020-06-15 22:00:56 -0700
  • 8ec14baca1 Handlign garbage shard case while delivering lease cleanup Ashwin Giridharan 2020-06-15 21:53:30 -0700
  • b60dd60f35 Code refactoring - 1 Ashwin Giridharan 2020-06-15 21:20:33 -0700
  • 37281e9493 Deleting lease immediately in RNF Exception Ashwin Giridharan 2020-06-15 20:57:55 -0700
  • aaf37be0cd Convert expiredLeases to set (#56) Renju Radhakrishnan 2020-06-15 13:59:43 -0400
  • c1cbb6cf6d Removing one of the lease clean up strategy as it is no longer required with distributed lease cleanup Ashwin Giridharan 2020-06-12 09:28:30 -0700
  • ce38178399 Caching consumerArn for StreamIdentifier in FanOutRetrievalFactory Ashwin Giridharan 2020-06-12 22:20:38 -0700
  • 2530481dba Periodic Shard Sync Auditor - Metrics and Customer auditor configs Ashwin Giridharan 2020-06-12 18:12:02 -0700
  • 1aeb8ed9a1
    Merge pull request #53 from ashwing/ltr_ddb_scan_opt ashwing 2020-06-11 18:42:48 -0700
  • 6fc654802c Limiting max page for ddb scan in isEmptyLeaseTable check to be 1 Ashwin Giridharan 2020-06-11 18:01:46 -0700
  • c8422745d9
    Removing lease cleanup from HierarchicalShardSyncer (#30) Joshua Kim 2020-06-11 19:56:04 -0400
  • 2e2211a9b7
    Merge pull request #31 from ashwing/ShutdownTaskCleanup Joshua Kim 2020-06-11 19:48:58 -0400
  • 445dd106b2 Adding logging Joshua Kim 2020-06-11 19:48:24 -0400
  • dc078ead39 Reverting pom change Joshua Kim 2020-06-10 19:49:06 -0400
  • 2f1838483c Adding Lease cleanup in shutdown task. Joshua Kim 2020-05-06 08:51:06 -0400
  • c7cdbd5d8b pom change Joshua Kim 2020-06-04 03:49:07 -0400
  • 61aeae3ea1
    Merge pull request #48 from ashwing/ltr_1_hsyncperstreamconfig ashwing 2020-06-10 10:35:24 -0700
  • 291d1b4059
    Merge pull request #44 from ashwing/ltr_1_ms_consumerarn ashwing 2020-06-09 18:18:21 -0700