Commit graph

136 commits

Author SHA1 Message Date
Sahil Palvia
d3b81cb463 Changing default cache size to 8 megs 2017-09-26 14:44:35 -07:00
Sahil Palvia
f96062efe3 Modifying the sane defaults to have 3 advanced calls, 15 megs cache size and 30000 records. 2017-09-26 14:41:32 -07:00
Sahil Palvia
34dd6f416a Adding idleMillisBetweenCalls config, to slow down the aggressive get calls while prefetching. 2017-09-26 14:39:13 -07:00
Sahil Palvia
b5d5618a94 Updating the config to set the cache type. 2017-09-25 11:56:02 -07:00
Sahil Palvia
1deae5cd07 Updating the artifact version to the next SNAPSHOT 2017-09-25 11:31:40 -07:00
Sahil Palvia
86c9f3d5b9 Merge remote-tracking branch 'upstream/master' into prefetch 2017-09-25 11:29:38 -07:00
Sahil Palvia
6321dcafd7 Merging changes into the branch and updating the tests. 2017-09-25 11:25:31 -07:00
Justin Pfifer
23c46267cf Merge pull request #228 from pfifer/release-1.8.4
Release 1.8.4 of the Amazon Kinesis Client for Java
2017-09-22 16:32:19 -07:00
Pfifer, Justin
c1df703d67 Release 1.8.4 of the Amazon Kinesis Client for Java
* Create a new completion service for each request.
  This ensures that canceled tasks are discarded.  This will prevent a cancellation exception causing issues processing records.
  * PR #227
  * Issue #226
2017-09-22 16:23:20 -07:00
Justin Pfifer
e8039b31ae Merge pull request #227 from pfifer/cancel-fix
Recreate the completion service instead of reusing them.
2017-09-22 16:18:52 -07:00
Pfifer, Justin
b1c27bd386 Recreate the completion service instead of reusing them.
This prevents cancelled tasks from being polled in subsequent calls.
2017-09-22 14:45:54 -07:00
BtXin
49b761c5e2 Merging changes (#225)
* integrated prefetch with shardconsumer

* fixed tests

* added fatory methods

* added tests and fixed broken tests

* Resolved conflicts

* Addressed comments

* Integrated the changes
2017-09-22 14:20:08 -07:00
Sahil Palvia
9249f28092 Merging master into prefetch (#223)
* MultiLangDaemon: Make shutdown grace configurable (#204)

Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.

* Release 1.8.2 of the Amazon Kinesis Client for Java (#218)

* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204

* Calling shutdown on the RetrievalStrategy (#222)

Fixes a bug where the retriever wasn't being shutdown when a record processor was being shutdown.

* Release 1.8.3 of the Amazon Kinesis Client for Java (#224)

* Call shutdown on the retriever when the record processor is being shutdown
  This fixes a bug that could leak threads if using the
  AsynchronousGetRecordsRetrievalStrategy is being used.
  The asynchronous retriever is only used when
  KinesisClientLibConfiguration#retryGetRecordsInSeconds, and
  KinesisClientLibConfiguration#maxGetRecordsThreadPool are set.
  * PR #222
2017-09-22 13:32:49 -07:00
Sahil Palvia
8af4db39c6 Merge remote-tracking branch 'upstream/master' into prefetch 2017-09-22 13:29:20 -07:00
Justin Pfifer
5558449043 Release 1.8.3 of the Amazon Kinesis Client for Java (#224)
* Call shutdown on the retriever when the record processor is being shutdown
  This fixes a bug that could leak threads if using the
  AsynchronousGetRecordsRetrievalStrategy is being used.
  The asynchronous retriever is only used when
  KinesisClientLibConfiguration#retryGetRecordsInSeconds, and
  KinesisClientLibConfiguration#maxGetRecordsThreadPool are set.
  * PR #222
2017-09-22 12:25:52 -07:00
Sahil Palvia
38900c51ad Updating branch to be upto date with the master 2017-09-22 11:51:21 -07:00
Sahil Palvia
4dd9423170 Calling shutdown on the RetrievalStrategy (#222)
Fixes a bug where the retriever wasn't being shutdown when a record processor was being shutdown.
2017-09-22 11:43:34 -07:00
Justin Pfifer
fac15f80f8 Merge pull request #221 from sahilpalvia/prefetch
Adding test classes for the Blocking and Prefetch getRecords cache.
2017-09-21 13:30:41 -07:00
Sahil Palvia
7d24bc54f8 Addressing comments and fixing the failing test. 2017-09-21 13:23:22 -07:00
Sahil Palvia
48699f4812 Updating the test 2017-09-21 11:52:00 -07:00
Sahil Palvia
ce8dd88846 Added a test case for BlockingGetRecordsCache. Updated the comments on BlockingGetRecords cache. Synchronized the added and removed methods instead of using volatile variables. 2017-09-21 11:50:53 -07:00
Sahil Palvia
3404ddfcf4 Adding licenses to files, addressing comments. Throwing IllegalStateException if threadpool is not started. Exposing the start method in the GetRecordsCache interface. Adding new test case for the new behavior. 2017-09-21 11:27:49 -07:00
Sahil Palvia
ed1f9d4dca Merge remote-tracking branch 'upstream/prefetch' into prefetch 2017-09-21 11:10:39 -07:00
Justin Pfifer
f9701f9651 Merge Release 1.8.2 (#220)
* MultiLangDaemon: Make shutdown grace configurable (#204)

Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.

* Release 1.8.2 of the Amazon Kinesis Client for Java (#218)

* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204
2017-09-21 10:23:38 -07:00
Sahil Palvia
92c0b43b2a Merge remote-tracking branch 'upstream/prefetch' into prefetch 2017-09-20 17:29:19 -07:00
Sahil Palvia
a3cbfaff31 Changed KinesisDataFetecher.getRecords not to return null back. Added test for KinesisDataFetcher.getRecords. Adding test classes for GetRecordsCache implemetations. Removing the DataFetchingStrategy from the PrefetchGetRecordsCache. 2017-09-20 17:28:05 -07:00
Justin Pfifer
9a82b6bd05 Release 1.8.2 of the Amazon Kinesis Client for Java (#218)
* Add support for two phase checkpoints
  Applications can now set a pending checkpoint, before completing the checkpoint operation. Once the application has completed its checkpoint steps, the final checkpoint will clear the pending checkpoint.
  Should the checkpoint fail the attempted sequence number is provided in the InitializationInput#getPendingCheckpointSequenceNumber otherwise the value will be null.
  * PR #188
* Support timeouts, and retry for GetRecords calls.
  Applications can now set timeouts for GetRecord calls to Kinesis.  As part of setting the timeout, the application must also provide a thread pool size for concurrent requests.
  * PR #214
* Notification when the lease table is throttled
  When writes, or reads, to the lease table are throttled a warning will be emitted.  If you're seeing this warning you should increase the IOPs for your lease table to prevent processing delays.
  * PR #212
* Support configuring the graceful shutdown timeout for MultiLang Clients
  This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.  The timeout can be configured by adding shutdownGraceMillis to the properties file set to the number of milliseconds to wait.
  * PR #204
2017-09-20 08:52:36 -07:00
Justin Pfifer
40aaece7c3 Merge pull request #216 from sahilpalvia/prefetch
Adding default caching class and fetching stratergy
2017-09-19 16:09:13 -07:00
Sahil Palvia
a8edb70552 Addressing PR comments. 2017-09-19 14:48:43 -07:00
Sahil Palvia
5172f4f936 Adding null condition to getRecords in the KinesisDataFetcher class. Changing the abstract class back to an interface. 2017-09-19 14:25:25 -07:00
Sahil Palvia
14ebfb8f0f Adding Blocking cache and spilting into blocking and prefetching cache. Changing the GetRecordsCache interface to abstract class. 2017-09-19 13:57:32 -07:00
Sahil Palvia
b6f41d21f8 Merge remote-tracking branch 'upstream/prefetch' into prefetch 2017-09-19 12:09:33 -07:00
Sahil Palvia
bcee1ae395 Adding default caching class and enum for fetching strategy. 2017-09-19 12:06:50 -07:00
Alex Charlton
01d2688bc6 MultiLangDaemon: Make shutdown grace configurable (#204)
Allow configuring the amount of time that the graceful shutdown process will wait for the client to complete its shutdown.
2017-09-19 08:52:31 -07:00
Sahil Palvia
c92a5b556c Adding the cache and the retriver stubs. (#215)
* Adding the cache and the retriver stubs.

* Addressing comments and adding initial documentation and changing the retreiver from interface to class.

* Reverting back to the interface

* Fixing minor error

* Adding default cache stub
2017-09-18 16:35:31 -07:00
Sahil Palvia
66b809ef7b Adding default cache stub 2017-09-18 16:32:54 -07:00
Sahil Palvia
c70ab1fc72 Fixing minor error 2017-09-18 15:02:46 -07:00
Sahil Palvia
9def913453 Reverting back to the interface 2017-09-18 15:02:06 -07:00
Sahil Palvia
8ceb5f2492 Addressing comments and adding initial documentation and changing the retreiver from interface to class. 2017-09-18 14:59:48 -07:00
Sahil Palvia
190d8fb5aa Adding the cache and the retriver stubs. 2017-09-18 13:26:09 -07:00
Sahil Palvia
244da44d29 Allow Configuring GetRecords Calls to Timeout. (#214)
It's now possible to configure GetRecords calls to timeout if they
take to long.  This can be used to terminate a long running request to
ensure that record processors continue to make progress

This feature was added with contributions from @pfifer, @sahilpalvia,
and @BtXin.
2017-09-18 10:58:16 -07:00
Sahil Palvia
656b17ceaa Adding logging for DynamoDB ProvisionedThroughputExceededExcpetion (#212)
* Adding logging to ProvisionedThroughputExceededException.

* Addressing CR comments and changing log level from error to warn

* Updated the comments as per cr comments.
2017-09-15 11:04:30 -07:00
Walid Baruni
1ec0b656c9 Add Support for Two Phase Commit (#188)
Adds support for making two phase commits.  When a checkpoint is started, but fails to complete the next instance of the record processor will receive the attempted checkpoint position at initialization time.
2017-08-15 12:56:32 -07:00
Justin Pfifer
cdbbff31e8 Advance to 1.8.2-SNAPSHOT (#203)
Advanced the version to 1.8.2
2017-08-14 09:05:12 -07:00
Justin Pfifer
79296dad2e Release 1.8.1 of the Amazon Kinesis Client (#198)
Support timeouts for calls to the MultiLang Daemon
This adds support for setting a timeout when dispatching records to
the client record processor. If the record processor doesn't respond
within the timeout the parent Java process will be terminated. This
is a temporary fix to handle cases where the KCL becomes blocked
while waiting for a client record processor.

The timeout for the this can be set by adding
  `timeoutInSeconds = <timeout value>`.
The default for this is no timeout.

Setting this can cause the KCL to exit suddenly, before using this
ensure that you have an automated restart for your application

Related #195
Related #185
2017-08-02 10:41:14 -07:00
Pfifer, Justin
526a1fa0df Updated AWS SDK to 1.11.171 2017-08-02 10:33:23 -07:00
Sahil Palvia
e8f9ad3f0a Adding support for timeout for multilang protocol, related to issue 185 (#195)
* Adding timeout to waitForStatusMessage future call. Introducing new config properties timeoutEnabled and timeoutInSeconds. Halting the JVM if timeout is reached.

* Adding test cases for halt jvm code. Made the configuration objects for timeout optional.

* Addressing code review comments and making appropriate changes.
2017-08-01 11:37:39 -07:00
Justin Pfifer
7d56c4aef1 Advance Version to 1.8.1-SNAPSHOT (#194)
Advance version of the amazon-kinesis-client to 1.8.1-SNAPSHOT in
preparation of next release cycle.
2017-07-25 14:19:34 -07:00
Justin Pfifer
af51312307 Release 1.8.0 (#193)
* Execute graceful shutdown on its own thread
  * PR #191
  * Issue #167
* Added support for controlling the size of the lease renewer thread pool
  * PR #177
  * Issue #171
* Require Java 8 and later
  Java 8 is now required for versions 1.8.0 of the amazon-kinesis-client and later.
  * PR #176
2017-07-25 09:22:48 -07:00
Justin Pfifer
c067cefa1f Make Graceful Shutdown Run On Its Own Thread (#191)
* Initial start of fix for requested shutdown

* Execute the requested shutdown in a separate thread.

Fix for Issue #167

* Reworked some of the shutdown logic to make the relationships clearer.

* Added/Updated Copyright Statements

* Add Missing License Statements
2017-07-21 08:30:26 -07:00