Commit graph

13 commits

Author SHA1 Message Date
Pierre Massat
c04f3d8a94 Get a new shard iterator on error (#32)
* Get a new shard iterator on error
* Check for nil instead of empty string
* Get a new shard iterator on error
2017-10-15 17:40:30 -07:00
Pierre Massat
c56cefb667 Expose the shard ID in the buffer (#30) 2016-12-26 08:24:34 -07:00
Harlow Ward
fedb6812fb Add checkpoint interface for custom checkpoints (#29)
To allow other checkpoint backends we extract a checkpoint interface
which future checkpoints can implement.

* Add checkpoint interface for custom checkpoints
* Create RedisCheckpoint to implement checkpoint interface
* Swap out hosie redis library for go-redis

Minor changes

* Allow configuration of Redis endpoint with env var `REDIS_URL`
* Replace gvt with govendor
2016-12-04 00:08:06 -08:00
Harlow Ward
ceca88b96a Add required fields to Config
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
2016-05-07 18:10:31 -07:00
Harlow Ward
a12c15a191 Use no-op Info logging by default 2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36 Use config object for optional params
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.

https://peter.bourgon.org/go-best-practices-2016/#configuration
2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe Add logging when records are emitted w/ record count 2016-05-01 10:43:42 -07:00
Harlow Ward
49b5a94c7e Use Apex log for logging (#27)
* Use Apex log for logging
2016-04-30 22:23:35 -07:00
Harlow Ward
e843ae5928 Remove poll interval in favor of aws retry backoff 2016-04-30 18:05:04 -07:00
Harlow Ward
dded9d0a0e Leverage the default AWS retry logic 2016-04-30 17:04:44 -07:00
Harlow Ward
c29698550f Add config options to Consumer
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
2016-02-09 22:31:15 -08:00
Harlow Ward
caa2fc0b0a Add Kinesis Firehose example 2016-02-08 13:21:54 -08:00
Harlow Ward
f0e6461cb6 Refactor to use handler func
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.

* Add refactoring note and SHA to README
2016-02-06 17:50:17 -08:00