Commit graph

6 commits

Author SHA1 Message Date
Harlow Ward
7018c0c47e
Introduce Group interface and AllGroup (#91)
* Introduce Group interface and AllGroup

As we move towards consumer groups we'll need to support the current
"consume all shards" strategy, and setup the codebase for the
anticipated "consume balanced shards."
2019-06-09 13:42:25 -07:00
Harlow Ward
97fe4e66ff
Use shard broker to monitor and process new shards (#85)
* Use shard broker to start processing new shards

The addition of a shard broker will allow the consumer to be notified
when new shards are added to the stream so it can consume them.

Fixes: https://github.com/harlow/kinesis-consumer/issues/36
2019-04-09 22:03:12 -07:00
Harlow Ward
c4f363a517 Example data is in repo, no need to download 2019-04-07 16:33:56 -07:00
Harlow Ward
4ffe3ec55a Add logs for start scan and checkpoints 2017-11-22 17:52:41 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience (#35)
Major changes:

* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config 

https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

Minor changes:

* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
2017-11-20 08:21:40 -08:00
Harlow Ward
f4c8d8090d Use TJ's Kinesis producer library 2016-05-01 16:54:47 -07:00
Renamed from examples/seed/README.md (Browse further)