Commit graph

20 commits

Author SHA1 Message Date
Harlow Ward
3b3b252fa5 Add all examples to example directory 2021-12-04 10:42:17 -08:00
Emanuel Ramos
b7be26418a Add postgres checkpoint implementation (#55) 2018-06-17 19:27:10 -07:00
Prometheus
739e9e39a5 Make it possible to let user use 3rd party logging library (#56) 2018-06-12 18:07:33 -07:00
Prometheus
e6a489c76b Scanerror signals the consumer if we should continue scanning for next record & whether to checkpoint. (#54)
* remove ValidateCheckpoint

* update for checkpoint can not customize retryer

* implement the scan error as in PR 44

* at least log if record processor has error

* mistakenly removed this line

* propage error up. ignore invalid state
2018-06-08 08:40:42 -07:00
Prometheus
d058203b6e Make what aws error to trigger retry decided by caller (#52)
* remove ValidateCheckpoint
* make retrying on error decided by caller
2018-06-04 20:07:58 -07:00
Prometheus
9a7e102a05 remove ValidateCheckpoint (#51) 2018-06-01 19:34:46 -07:00
Prometheus
992cc42419 DDB uses default AWS config settings to ping table; won't work with WithDyanmoClient. Misc update on example and README (#50) 2018-06-01 16:14:42 -07:00
Prometheus
9e0a97916d Use AWS resource iface, overwrite default dynamodb, more explicit in example about overwrite default AWS resrouce client (#49)
* use custom kinesis client

* use aws sdk interface, add missing api for ddb

* add overwrite default dynamodbclien usage
2018-05-31 17:41:14 -07:00
Harlow Ward
64cdf69249
Add interval flush for DDB checkpoint (#40)
Add interval flush for DDB checkpoint

* Allow checkpointing on a specified interval
* Add shutdown method to checkpoint to force flush

Minor changes:

* Swap order of input params for checkpoint (app, table)

Addresses: https://github.com/harlow/kinesis-consumer/issues/39
2017-12-30 20:21:10 -08:00
Harlow Ward
058f383e30 Add cancellation of pipeline from signal interrupts 2017-11-26 16:00:03 -08:00
Harlow Ward
edf0467eb0 Format errors from caller 2017-11-23 11:29:58 -08:00
Harlow Ward
86f1df782e Return the shard scan errors to top-level caller 2017-11-23 08:49:37 -08:00
Harlow Ward
6401371727 Simplify checkpoint interface; reduce input vars 2017-11-22 20:01:31 -08:00
Harlow Ward
3770136f64 Allow user to override no-op checkpoint with Option 2017-11-22 17:44:42 -08:00
Harlow Ward
c91f6233ef Add counter for exposing scanner metrics 2017-11-22 14:10:11 -08:00
Harlow Ward
90d2903fe6 Use stdlib logging, default to discard 2017-11-22 10:46:39 -08:00
Harlow Ward
4d6a85e901 Make the Checkpoint a required input for Consumer
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.

* Add Checkpoint as required param for new consumer
2017-11-21 08:58:16 -08:00
Harlow Ward
8d2cc5bc20 Return error from scan instead of terminating the program 2017-11-20 11:45:41 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option (#37)
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence

Implements: https://github.com/harlow/kinesis-consumer/issues/26
2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience (#35)
Major changes:

* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config 

https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

Minor changes:

* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
2017-11-20 08:21:40 -08:00