Commit graph

10 commits

Author SHA1 Message Date
Vincent
d3b76346f5 Break down the big function and Add tests for Scan (#65) 2018-09-03 09:59:39 -07:00
Harlow Ward
23811ec99a Add test for stopping scan 2018-07-29 10:27:01 -07:00
Harlow Ward
d6ded158bf Refactor the consumer tests
A previous PR from @vincent6767 had nicer mock Kinesis client that
simplified setting up data for the tests.

Mock client pulled from:
https://github.com/harlow/kinesis-consumer/pull/64
2018-07-29 10:13:03 -07:00
Harlow Ward
fb98fbe244
Remove the client wrapper (#58)
Having an additional Client has added some confusion (https://github.com/harlow/kinesis-consumer/issues/45) on how to provide a
custom kinesis client. Allowing `WithClient` to accept a Kinesis client
it cleans up the interface.

Major changes:

* Remove the Client wrapper; prefer using kinesis client directly
* Change `ScanError` to `ScanStatus` as the return value isn't necessarily an error

Note: these are breaking changes, if you need last stable release please see here: https://github.com/harlow/kinesis-consumer/releases/tag/v0.2.0
2018-07-28 22:53:33 -07:00
Vincent
a1239221d8 Ignore IDE files and fix code based Gometalinter (#63)
- scanError.Error is removed since it is not used.
- session.New() is deprecated, The NewKinesisClient's function signature change so it can
  returns the error from the NewSession().
2018-07-24 20:10:38 -07:00
Prometheus
e6a489c76b Scanerror signals the consumer if we should continue scanning for next record & whether to checkpoint. (#54)
* remove ValidateCheckpoint

* update for checkpoint can not customize retryer

* implement the scan error as in PR 44

* at least log if record processor has error

* mistakenly removed this line

* propage error up. ignore invalid state
2018-06-08 08:40:42 -07:00
Harlow Ward
b875bb56e7 Introduce Client Interface
Testing the components of the consumer where proving difficult because
the consumer code was so tightly coupled with the Kinesis client
library.

* Extract the concept of Client interface
* Create default client w/ kinesis connection
* Test with fake client to avoid round trip to kinesis
2017-11-26 16:00:11 -08:00
Harlow Ward
afae1bea36 Use config object for optional params
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.

https://peter.bourgon.org/go-best-practices-2016/#configuration
2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe Add logging when records are emitted w/ record count 2016-05-01 10:43:42 -07:00
Harlow Ward
c29698550f Add config options to Consumer
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
2016-02-09 22:31:15 -08:00