luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
177 commits 6 branches 9 tags 7.5 MiB
Commit graph

177 commits

This branch
This branch
All branches
Author SHA1 Message Date
Vincent
049445e259 Add unit tests for the GetRecords function (#64) 
A closed shard returns the last sequence number and no error. The current implementation leads to an infinite loop if the shard is closed. NextShardIterator checking is enough. That's why I remove the getShardIterator call
 2018-07-28 10:36:22 -07:00
Vincent
a1239221d8 Ignore IDE files and fix code based Gometalinter (#63) 
- scanError.Error is removed since it is not used.
- session.New() is deprecated, The NewKinesisClient's function signature change so it can
  returns the error from the NewSession().
 2018-07-24 20:10:38 -07:00
Emanuel Ramos
cf5d22abff Adding Option func to set maxInterval (#60) 2018-07-13 07:26:21 -07:00
Umur Gedik
9ccee87b62 Add an option to start consuming from latest (#59) 
* Add an option to client to start consuming from latest
 2018-06-27 20:05:09 -07:00
Emanuel Ramos
b7be26418a Add postgres checkpoint implementation (#55) 2018-06-17 19:27:10 -07:00
Prometheus
739e9e39a5 Make it possible to let user use 3rd party logging library (#56) 2018-06-12 18:07:33 -07:00
Prometheus
e6a489c76b Scanerror signals the consumer if we should continue scanning for next record & whether to checkpoint. (#54) 
* remove ValidateCheckpoint

* update for checkpoint can not customize retryer

* implement the scan error as in PR 44

* at least log if record processor has error

* mistakenly removed this line

* propage error up. ignore invalid state
 2018-06-08 08:40:42 -07:00
Prometheus
b05f5b3ac6 Update readme for checkpoint <EOM> (#53) 
* remove ValidateCheckpoint
* update for checkpoint can not customize retryer
 2018-06-07 21:10:28 -07:00
Prometheus
d058203b6e Make what aws error to trigger retry decided by caller (#52) 
* remove ValidateCheckpoint
* make retrying on error decided by caller
 2018-06-04 20:07:58 -07:00
Prometheus
9a7e102a05 remove ValidateCheckpoint (#51) 2018-06-01 19:34:46 -07:00
Prometheus
992cc42419 DDB uses default AWS config settings to ping table; won't work with WithDyanmoClient. Misc update on example and README (#50) 2018-06-01 16:14:42 -07:00
Prometheus
9e0a97916d Use AWS resource iface, overwrite default dynamodb, more explicit in example about overwrite default AWS resrouce client (#49) 
* use custom kinesis client

* use aws sdk interface, add missing api for ddb

* add overwrite default dynamodbclien usage
 2018-05-31 17:41:14 -07:00
Anant Prakash
2a5856ec99 Correct AWS_REGION (#48) 2018-05-30 06:45:38 -07:00
Anant Prakash
2fb47e63bc Fix typo in README.md (#47) 2018-05-24 08:34:10 -07:00
Tim Bart
4bdbbefa34 Use functional options to configure KinesisClient (#46) 2018-04-27 22:12:11 -07:00
Harlow Ward
64cdf69249
Add interval flush for DDB checkpoint (#40) 
Add interval flush for DDB checkpoint

* Allow checkpointing on a specified interval
* Add shutdown method to checkpoint to force flush

Minor changes:

* Swap order of input params for checkpoint (app, table)

Addresses: https://github.com/harlow/kinesis-consumer/issues/39
 2017-12-30 20:21:10 -08:00
Harlow Ward
955f74d553 Have new func return type 2017-11-26 18:22:09 -08:00
Harlow Ward
e5e057d6aa Add additional context to new comment 2017-11-26 18:17:41 -08:00
Harlow Ward
a62e7514e4 Make the Kinesis client exportable 2017-11-26 18:16:32 -08:00
Harlow Ward
b875bb56e7 Introduce Client Interface 
Testing the components of the consumer where proving difficult because
the consumer code was so tightly coupled with the Kinesis client
library.

* Extract the concept of Client interface
* Create default client w/ kinesis connection
* Test with fake client to avoid round trip to kinesis
 2017-11-26 16:00:11 -08:00
Harlow Ward
058f383e30 Add cancellation of pipeline from signal interrupts 2017-11-26 16:00:03 -08:00
Harlow Ward
89570130f5 Leverage bigger batchsize when seeding example data 2017-11-26 15:59:17 -08:00
Harlow Ward
edf0467eb0 Format errors from caller 2017-11-23 11:29:58 -08:00
Harlow Ward
86f1df782e Return the shard scan errors to top-level caller 2017-11-23 08:49:37 -08:00
Harlow Ward
b0245d688b Add more test coverage for Redis Checkpoint 2017-11-22 21:28:39 -08:00
Harlow Ward
6401371727 Simplify checkpoint interface; reduce input vars 2017-11-22 20:01:31 -08:00
Harlow Ward
3f081bd05a Fix position of input params for Options 2017-11-22 17:54:47 -08:00
Harlow Ward
4ffe3ec55a Add logs for start scan and checkpoints 2017-11-22 17:52:41 -08:00
Harlow Ward
3770136f64 Allow user to override no-op checkpoint with Option 2017-11-22 17:44:42 -08:00
Harlow Ward
c64b40b4ad Increment counts in correct usage points 2017-11-22 14:21:19 -08:00
Harlow Ward
c91f6233ef Add counter for exposing scanner metrics 2017-11-22 14:10:11 -08:00
Harlow Ward
84c0820f4a
Serverless options 2017-11-22 10:57:29 -08:00
Harlow Ward
7db78c24f4
Update with alternative options 2017-11-22 10:55:22 -08:00
Harlow Ward
b783b8fb5f
Update formatting on notes section 2017-11-22 10:50:09 -08:00
Harlow Ward
90d2903fe6 Use stdlib logging, default to discard 2017-11-22 10:46:39 -08:00
Harlow Ward
9a35af8df6 Update the checkpoint diagram 2017-11-21 09:04:39 -08:00
Harlow Ward
4d6a85e901 Make the Checkpoint a required input for Consumer 
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.

* Add Checkpoint as required param for new consumer
 2017-11-21 08:58:16 -08:00
Harlow Ward
154595b9a3
Add change log to summarize repo activity (#38) 
Make it visible to users when breaking changes happen in the repo

* Add changelog w/ link to previous stable version of library
 2017-11-20 17:27:39 -08:00
Harlow Ward
8d2cc5bc20 Return error from scan instead of terminating the program 2017-11-20 11:45:41 -08:00
Harlow Ward
60ce796c07
Add new diagram for consumer checkpoint storge 2017-11-20 11:14:39 -08:00
Harlow Ward
9620261104
Add checkpoint diagram 2017-11-20 11:06:46 -08:00
Harlow Ward
28837eee9e
Add link to blog post about Kinesis and Lambda 2017-11-20 10:29:30 -08:00
Harlow Ward
836e4bd5c1 Update godeps 2017-11-20 10:13:47 -08:00
Harlow Ward
d6602175e3
Add screenshot for fields needed in DDB checkpoint 2017-11-20 09:55:43 -08:00
Harlow Ward
1038843ed8
Use italics for Kinesis to Firehose note 2017-11-20 09:45:57 -08:00
Harlow Ward
99d82c2c01
Note about Kinesis to Firehose functionality 2017-11-20 09:45:00 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option (#37) 
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence

Implements: https://github.com/harlow/kinesis-consumer/issues/26
 2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience (#35) 
Major changes:

* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config 

https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

Minor changes:

* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
 2017-11-20 08:21:40 -08:00
Harlow Ward
e4efcdb56e Remove duplicated function for getShardIterator 2017-10-15 17:45:38 -07:00
Pierre Massat
c04f3d8a94 Get a new shard iterator on error (#32) 
* Get a new shard iterator on error
* Check for nil instead of empty string
* Get a new shard iterator on error
 2017-10-15 17:40:30 -07:00