Commit graph

115 commits

Author SHA1 Message Date
Wei
94930bfacb Addressed comment 2017-10-09 15:50:09 -07:00
Wei
8055b6379a Addressed comments 2017-10-09 15:17:33 -07:00
Wei
eff10a8253 Addressed Comments 2017-10-06 14:39:18 -07:00
Wei
be34a50f59 Addressed comments 2017-10-06 13:34:57 -07:00
Wei
a2fd07c7b0 Added set metrics to InitializeTask 2017-10-06 13:08:04 -07:00
Wei
e0c4ad2680 Get rid of sysout 2017-10-06 11:36:16 -07:00
Wei
dddcd64bd7 Passed the operation by caller 2017-10-06 10:51:10 -07:00
Wei
81c13d2a35 Merge remote-tracking branch 'upstream/prefetch' into prefetch 2017-10-05 18:04:37 -07:00
Sahil Palvia
0cf34461a7 Updated the documentation. Changed the variable name from maxSize to maxPendingProcessRecordsInput. 2017-10-04 12:00:36 -07:00
Sahil Palvia
5717bab9b6 Addressing comments from PR. Catching expected exceptions, changing log message for unexcepted exception. Changing threadname. 2017-10-04 11:14:55 -07:00
Sahil Palvia
80216b9a59 Added shardId to thread name. Added documentation to the methods. 2017-10-03 22:57:33 -07:00
Sahil Palvia
ffdfe82b79 Fixing issue with prefetch thread, where it kept on dying and falling behind. Catching throwable instead of error. Assigning thread name to the prefetch thread. 2017-10-03 15:29:38 -07:00
Wei
192aef6619 Addressed comments 2017-10-03 13:39:29 -07:00
Wei
2fc4267b83 Merged upstream 2017-10-03 13:17:58 -07:00
Wei
6ab1503fe3 emit metric 2017-09-29 14:49:11 -07:00
Wei
abed367da0 Handle Custom Metric Scope 2017-09-28 14:47:31 -07:00
Sahil Palvia
cd1f944add Making sure to call shutdown on strategy once the cache is interrupted. 2017-09-27 21:01:29 -07:00
Sahil Palvia
7b76d1d56e Moving the idleMillisBetweenCalls to the cache. Fixed unit and integ tests. Calling StrategyShutdown from within the cache thread. 2017-09-27 15:50:23 -07:00
Justin Pfifer
060913d49c Merge Release 1.8.5 (#233)
* Only advance the shard iterator when we accept a result to return

This changes the retriever strategy to only accept the shard iterator
when we have accepted a result to return.  This is for the
asynchronous retriever where multiple threads may contend for the same
iterator slot.  This ensures only the one selected for the response will
advance the shard iterator.

* Release 1.8.5 of the Amazon Kinesis Client for Java (#232)

* Release 1.8.5 of the Amazon Kinesis Client for Java

Release 1.8.5 (September 26, 2017)
* Only advance the shard iterator for the accepted response.  
  This fixes a race condition in the `KinesisDataFetcher` when it's being used to make asynchronous requests.  The shard iterator is now only advanced when the retriever calls `DataFetcherResult#accept()`.
  * PR #230
  * Issue #231

* Change the TerminalResult to return an empty GetRecordsResult

Changes the TerminalResult to provide an empty GetRecordsResult, which
components downstream depend on.

* Fix unit test for behavior change of TerminalResult

Got distracted, and forgot to run the unit tests.
2017-09-27 10:31:04 -07:00
Sahil Palvia
e437fa8436 Adding Integ testing for prefetch. 2017-09-26 16:24:30 -07:00
Sahil Palvia
d3b81cb463 Changing default cache size to 8 megs 2017-09-26 14:44:35 -07:00
Sahil Palvia
f96062efe3 Modifying the sane defaults to have 3 advanced calls, 15 megs cache size and 30000 records. 2017-09-26 14:41:32 -07:00
Sahil Palvia
34dd6f416a Adding idleMillisBetweenCalls config, to slow down the aggressive get calls while prefetching. 2017-09-26 14:39:13 -07:00
Sahil Palvia
b5d5618a94 Updating the config to set the cache type. 2017-09-25 11:56:02 -07:00
Sahil Palvia
86c9f3d5b9 Merge remote-tracking branch 'upstream/master' into prefetch 2017-09-25 11:29:38 -07:00
Sahil Palvia
6321dcafd7 Merging changes into the branch and updating the tests. 2017-09-25 11:25:31 -07:00
Pfifer, Justin
b1c27bd386 Recreate the completion service instead of reusing them.
This prevents cancelled tasks from being polled in subsequent calls.
2017-09-22 14:45:54 -07:00
BtXin
49b761c5e2 Merging changes (#225)
* integrated prefetch with shardconsumer

* fixed tests

* added fatory methods

* added tests and fixed broken tests

* Resolved conflicts

* Addressed comments

* Integrated the changes
2017-09-22 14:20:08 -07:00
Sahil Palvia
9249f28092 Merging master into prefetch (#223)
* MultiLangDaemon: Make shutdown grace configurable (#204)

Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.

* Release 1.8.2 of the Amazon Kinesis Client for Java (#218)

* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204

* Calling shutdown on the RetrievalStrategy (#222)

Fixes a bug where the retriever wasn't being shutdown when a record processor was being shutdown.

* Release 1.8.3 of the Amazon Kinesis Client for Java (#224)

* Call shutdown on the retriever when the record processor is being shutdown
  This fixes a bug that could leak threads if using the
  AsynchronousGetRecordsRetrievalStrategy is being used.
  The asynchronous retriever is only used when
  KinesisClientLibConfiguration#retryGetRecordsInSeconds, and
  KinesisClientLibConfiguration#maxGetRecordsThreadPool are set.
  * PR #222
2017-09-22 13:32:49 -07:00
Sahil Palvia
8af4db39c6 Merge remote-tracking branch 'upstream/master' into prefetch 2017-09-22 13:29:20 -07:00
Justin Pfifer
5558449043 Release 1.8.3 of the Amazon Kinesis Client for Java (#224)
* Call shutdown on the retriever when the record processor is being shutdown
  This fixes a bug that could leak threads if using the
  AsynchronousGetRecordsRetrievalStrategy is being used.
  The asynchronous retriever is only used when
  KinesisClientLibConfiguration#retryGetRecordsInSeconds, and
  KinesisClientLibConfiguration#maxGetRecordsThreadPool are set.
  * PR #222
2017-09-22 12:25:52 -07:00
Sahil Palvia
38900c51ad Updating branch to be upto date with the master 2017-09-22 11:51:21 -07:00
Sahil Palvia
4dd9423170 Calling shutdown on the RetrievalStrategy (#222)
Fixes a bug where the retriever wasn't being shutdown when a record processor was being shutdown.
2017-09-22 11:43:34 -07:00
Sahil Palvia
7d24bc54f8 Addressing comments and fixing the failing test. 2017-09-21 13:23:22 -07:00
Sahil Palvia
48699f4812 Updating the test 2017-09-21 11:52:00 -07:00
Sahil Palvia
ce8dd88846 Added a test case for BlockingGetRecordsCache. Updated the comments on BlockingGetRecords cache. Synchronized the added and removed methods instead of using volatile variables. 2017-09-21 11:50:53 -07:00
Sahil Palvia
3404ddfcf4 Adding licenses to files, addressing comments. Throwing IllegalStateException if threadpool is not started. Exposing the start method in the GetRecordsCache interface. Adding new test case for the new behavior. 2017-09-21 11:27:49 -07:00
Sahil Palvia
ed1f9d4dca Merge remote-tracking branch 'upstream/prefetch' into prefetch 2017-09-21 11:10:39 -07:00
Justin Pfifer
f9701f9651 Merge Release 1.8.2 (#220)
* MultiLangDaemon: Make shutdown grace configurable (#204)

Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.

* Release 1.8.2 of the Amazon Kinesis Client for Java (#218)

* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204
2017-09-21 10:23:38 -07:00
Sahil Palvia
a3cbfaff31 Changed KinesisDataFetecher.getRecords not to return null back. Added test for KinesisDataFetcher.getRecords. Adding test classes for GetRecordsCache implemetations. Removing the DataFetchingStrategy from the PrefetchGetRecordsCache. 2017-09-20 17:28:05 -07:00
Justin Pfifer
9a82b6bd05 Release 1.8.2 of the Amazon Kinesis Client for Java (#218)
* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204
2017-09-20 08:52:36 -07:00
Sahil Palvia
a8edb70552 Addressing PR comments. 2017-09-19 14:48:43 -07:00
Sahil Palvia
5172f4f936 Adding null condition to getRecords in the KinesisDataFetcher class. Changing the abstract class back to an interface. 2017-09-19 14:25:25 -07:00
Sahil Palvia
14ebfb8f0f Adding Blocking cache and spilting into blocking and prefetching cache. Changing the GetRecordsCache interface to abstract class. 2017-09-19 13:57:32 -07:00
Sahil Palvia
bcee1ae395 Adding default caching class and enum for fetching strategy. 2017-09-19 12:06:50 -07:00
Alex Charlton
01d2688bc6 MultiLangDaemon: Make shutdown grace configurable (#204)
Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.
2017-09-19 08:52:31 -07:00
Sahil Palvia
66b809ef7b Adding default cache stub 2017-09-18 16:32:54 -07:00
Sahil Palvia
c70ab1fc72 Fixing minor error 2017-09-18 15:02:46 -07:00
Sahil Palvia
9def913453 Reverting back to the interface 2017-09-18 15:02:06 -07:00
Sahil Palvia
8ceb5f2492 Addressing comments and adding initial documentation and changing the retreiver from interface to class. 2017-09-18 14:59:48 -07:00