luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add new diagram for consumer checkpoint storge

Browse source
This commit is contained in:
Harlow Ward 2017-11-20 11:14:39 -08:00 committed by GitHub
parent 9620261104
commit 60ce796c07
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
README.md
View file

@ -1,6 +1,6 @@
# Golang Kinesis Consumer
Kinesis consumer applications written in Go. This library is intended to be a lightweight wrapper around the Kinesis API to scan records, set checkpoints, and gracefully recover from network errors.
Kinesis consumer applications written in Go. This library is intended to be a lightweight wrapper around the Kinesis API to read records, save checkpoints (with swappable backends), and gracefully recover from network errors.
_NOTE: With the release of [Kinesis to Firehose](http://docs.aws.amazon.com/firehose/latest/dev/writing-with-kinesis-streams.html) it's possible to archive data directly to S3, Redshift, or Elasticsearch without running a consumer application._
@ -54,8 +54,8 @@ The consumer requires the following config:
It also accepts the following optional overrides:
* Kinesis Client
* Checkpoint Storage
* Logger
* Checkpoint
```go
// new kinesis client
@ -69,11 +69,11 @@ c, err := consumer.New(
)
```
### Checkpoint
### Checkpoint Storage
To record the progress of the consumer in the stream we store the last sequence number the consumer has read from a particular shard. This will allow consumers to re-launch and pick up at the position in the stream where they left off.
<img width="694" alt="kinesis-checkpoints" src="https://user-images.githubusercontent.com/739782/33036220-95649ce8-cde2-11e7-99dc-725c38d3d902.png">
<img width="687" alt="kinesis-checkpoints" src="https://user-images.githubusercontent.com/739782/33036582-b6f3c4b4-cde3-11e7-9334-c4bfbe34d984.png">
The default checkpoint uses Redis on localhost; to set a custom Redis URL use ENV vars: