Commit graph

143 commits

Author SHA1 Message Date
Pfifer, Justin
cc2e92001a Change the TerminalResult to return an empty GetRecordsResult
Changes the TerminalResult to provide an empty GetRecordsResult, which
components downstream depend on.
2017-09-27 09:58:33 -07:00
Pfifer, Justin
f8425d9283 Merge branch 'master' into prefetch
Merged release 1.8.5
2017-09-27 08:37:08 -07:00
Justin Pfifer
c96c8afeac Merge pull request #229 from sahilpalvia/prefetch
Fixing tests and integrating to use the getRecordsCache.
2017-09-27 08:03:03 -07:00
Sahil Palvia
e437fa8436 Adding Integ testing for prefetch. 2017-09-26 16:24:30 -07:00
Sahil Palvia
d3b81cb463 Changing default cache size to 8 megs 2017-09-26 14:44:35 -07:00
Sahil Palvia
f96062efe3 Modifying the sane defaults to have 3 advanced calls, 15 megs cache size and 30000 records. 2017-09-26 14:41:32 -07:00
Sahil Palvia
34dd6f416a Adding idleMillisBetweenCalls config, to slow down the aggressive get calls while prefetching. 2017-09-26 14:39:13 -07:00
Justin Pfifer
9720b1b249 Release 1.8.5 of the Amazon Kinesis Client for Java (#232)
* Release 1.8.5 of the Amazon Kinesis Client for Java

Release 1.8.5 (September 26, 2017)
* Only advance the shard iterator for the accepted response.  
  This fixes a race condition in the `KinesisDataFetcher` when it's being used to make asynchronous requests.  The shard iterator is now only advanced when the retriever calls `DataFetcherResult#accept()`.
  * PR #230
  * Issue #231
2017-09-26 14:31:59 -07:00
Justin Pfifer
6b25de917b Merge pull request #230 from pfifer/data-fetcher-accept
Only advance the shard iterator when we accept a result to return
2017-09-26 13:47:57 -07:00
Pfifer, Justin
4b20556f37 Only advance the shard iterator when we accept a result to return
This changes the retriever strategy to only accept the shard iterator
when we have accepted a result to return.  This is for the
asynchronous retriever where multiple threads may contend for the same
iterator slot.  This ensures only the one selected for the response will
advance the shard iterator.
2017-09-26 12:39:53 -07:00
Sahil Palvia
b5d5618a94 Updating the config to set the cache type. 2017-09-25 11:56:02 -07:00
Sahil Palvia
1deae5cd07 Updating the artifact version to the next SNAPSHOT 2017-09-25 11:31:40 -07:00
Sahil Palvia
86c9f3d5b9 Merge remote-tracking branch 'upstream/master' into prefetch 2017-09-25 11:29:38 -07:00
Sahil Palvia
6321dcafd7 Merging changes into the branch and updating the tests. 2017-09-25 11:25:31 -07:00
Justin Pfifer
23c46267cf Merge pull request #228 from pfifer/release-1.8.4
Release 1.8.4 of the Amazon Kinesis Client for Java
2017-09-22 16:32:19 -07:00
Pfifer, Justin
c1df703d67 Release 1.8.4 of the Amazon Kinesis Client for Java
* Create a new completion service for each request.
  This ensures that canceled tasks are discarded.  This will prevent a cancellation exception causing issues processing records.
  * PR #227
  * Issue #226
2017-09-22 16:23:20 -07:00
Justin Pfifer
e8039b31ae Merge pull request #227 from pfifer/cancel-fix
Recreate the completion service instead of reusing them.
2017-09-22 16:18:52 -07:00
Pfifer, Justin
b1c27bd386 Recreate the completion service instead of reusing them.
This prevents cancelled tasks from being polled in subsequent calls.
2017-09-22 14:45:54 -07:00
BtXin
49b761c5e2 Merging changes (#225)
* integrated prefetch with shardconsumer

* fixed tests

* added fatory methods

* added tests and fixed broken tests

* Resolved conflicts

* Addressed comments

* Integrated the changes
2017-09-22 14:20:08 -07:00
Sahil Palvia
9249f28092 Merging master into prefetch (#223)
* MultiLangDaemon: Make shutdown grace configurable (#204)

Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.

* Release 1.8.2 of the Amazon Kinesis Client for Java (#218)

* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204

* Calling shutdown on the RetrievalStrategy (#222)

Fixes a bug where the retriever wasn't being shutdown when a record processor was being shutdown.

* Release 1.8.3 of the Amazon Kinesis Client for Java (#224)

* Call shutdown on the retriever when the record processor is being shutdown
  This fixes a bug that could leak threads if using the
  AsynchronousGetRecordsRetrievalStrategy is being used.
  The asynchronous retriever is only used when
  KinesisClientLibConfiguration#retryGetRecordsInSeconds, and
  KinesisClientLibConfiguration#maxGetRecordsThreadPool are set.
  * PR #222
2017-09-22 13:32:49 -07:00
Sahil Palvia
8af4db39c6 Merge remote-tracking branch 'upstream/master' into prefetch 2017-09-22 13:29:20 -07:00
Justin Pfifer
5558449043 Release 1.8.3 of the Amazon Kinesis Client for Java (#224)
* Call shutdown on the retriever when the record processor is being shutdown
  This fixes a bug that could leak threads if using the
  AsynchronousGetRecordsRetrievalStrategy is being used.
  The asynchronous retriever is only used when
  KinesisClientLibConfiguration#retryGetRecordsInSeconds, and
  KinesisClientLibConfiguration#maxGetRecordsThreadPool are set.
  * PR #222
2017-09-22 12:25:52 -07:00
Sahil Palvia
38900c51ad Updating branch to be upto date with the master 2017-09-22 11:51:21 -07:00
Sahil Palvia
4dd9423170 Calling shutdown on the RetrievalStrategy (#222)
Fixes a bug where the retriever wasn't being shutdown when a record processor was being shutdown.
2017-09-22 11:43:34 -07:00
Justin Pfifer
fac15f80f8 Merge pull request #221 from sahilpalvia/prefetch
Adding test classes for the Blocking and Prefetch getRecords cache.
2017-09-21 13:30:41 -07:00
Sahil Palvia
7d24bc54f8 Addressing comments and fixing the failing test. 2017-09-21 13:23:22 -07:00
Sahil Palvia
48699f4812 Updating the test 2017-09-21 11:52:00 -07:00
Sahil Palvia
ce8dd88846 Added a test case for BlockingGetRecordsCache. Updated the comments on BlockingGetRecords cache. Synchronized the added and removed methods instead of using volatile variables. 2017-09-21 11:50:53 -07:00
Sahil Palvia
3404ddfcf4 Adding licenses to files, addressing comments. Throwing IllegalStateException if threadpool is not started. Exposing the start method in the GetRecordsCache interface. Adding new test case for the new behavior. 2017-09-21 11:27:49 -07:00
Sahil Palvia
ed1f9d4dca Merge remote-tracking branch 'upstream/prefetch' into prefetch 2017-09-21 11:10:39 -07:00
Justin Pfifer
f9701f9651 Merge Release 1.8.2 (#220)
* MultiLangDaemon: Make shutdown grace configurable (#204)

Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.

* Release 1.8.2 of the Amazon Kinesis Client for Java (#218)

* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204
2017-09-21 10:23:38 -07:00
Sahil Palvia
92c0b43b2a Merge remote-tracking branch 'upstream/prefetch' into prefetch 2017-09-20 17:29:19 -07:00
Sahil Palvia
a3cbfaff31 Changed KinesisDataFetecher.getRecords not to return null back. Added test for KinesisDataFetcher.getRecords. Adding test classes for GetRecordsCache implemetations. Removing the DataFetchingStrategy from the PrefetchGetRecordsCache. 2017-09-20 17:28:05 -07:00
Justin Pfifer
9a82b6bd05 Release 1.8.2 of the Amazon Kinesis Client for Java (#218)
* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204
2017-09-20 08:52:36 -07:00
Justin Pfifer
40aaece7c3 Merge pull request #216 from sahilpalvia/prefetch
Adding default caching class and fetching stratergy
2017-09-19 16:09:13 -07:00
Sahil Palvia
a8edb70552 Addressing PR comments. 2017-09-19 14:48:43 -07:00
Sahil Palvia
5172f4f936 Adding null condition to getRecords in the KinesisDataFetcher class. Changing the abstract class back to an interface. 2017-09-19 14:25:25 -07:00
Sahil Palvia
14ebfb8f0f Adding Blocking cache and spilting into blocking and prefetching cache. Changing the GetRecordsCache interface to abstract class. 2017-09-19 13:57:32 -07:00
Sahil Palvia
b6f41d21f8 Merge remote-tracking branch 'upstream/prefetch' into prefetch 2017-09-19 12:09:33 -07:00
Sahil Palvia
bcee1ae395 Adding default caching class and enum for fetching strategy. 2017-09-19 12:06:50 -07:00
Alex Charlton
01d2688bc6 MultiLangDaemon: Make shutdown grace configurable (#204)
Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.
2017-09-19 08:52:31 -07:00
Sahil Palvia
c92a5b556c Adding the cache and the retriver stubs. (#215)
* Adding the cache and the retriver stubs.

* Addressing comments and adding initial documentation and changing the retreiver from interface to class.

* Reverting back to the interface

* Fixing minor error

* Adding default cache stub
2017-09-18 16:35:31 -07:00
Sahil Palvia
66b809ef7b Adding default cache stub 2017-09-18 16:32:54 -07:00
Sahil Palvia
c70ab1fc72 Fixing minor error 2017-09-18 15:02:46 -07:00
Sahil Palvia
9def913453 Reverting back to the interface 2017-09-18 15:02:06 -07:00
Sahil Palvia
8ceb5f2492 Addressing comments and adding initial documentation and changing the retreiver from interface to class. 2017-09-18 14:59:48 -07:00
Sahil Palvia
190d8fb5aa Adding the cache and the retriver stubs. 2017-09-18 13:26:09 -07:00
Sahil Palvia
244da44d29 Allow Configuring GetRecords Calls to Timeout. (#214)
It's now possible to configure GetRecords calls to timeout if they
take to long.  This can be used to terminate a long running request to
ensure that record processors continue to make progress

This feature was added with contributions from @pfifer, @sahilpalvia,
and @BtXin.
2017-09-18 10:58:16 -07:00
Sahil Palvia
656b17ceaa Adding logging for DynamoDB ProvisionedThroughputExceededExcpetion (#212)
* Adding logging to ProvisionedThroughputExceededException.

* Addressing CR comments and changing log level from error to warn

* Updated the comments as per cr comments.
2017-09-15 11:04:30 -07:00
Walid Baruni
1ec0b656c9 Add Support for Two Phase Commit (#188)
Adds support for making two phase commits.  When a checkpoint is started, but fails to complete the next instance of the record processor will receive the attempted checkpoint position at initialization time.
2017-08-15 12:56:32 -07:00