luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Replace Checkpoint with Store interface (#90)

Browse source 
As we work towards introducing consumer groups to the repository we need a more generic name for the persistence layer for storing checkpoints and leases for given shards. 

* Rename `checkpoint` to `store`
This commit is contained in:
Harlow Ward 2019-07-28 21:18:40 -07:00 committed by GitHub
parent d05d6c2d5e
commit c72f561abd
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
28 changed files with 134 additions and 129 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

53
README.md
View file

@ -2,7 +2,7 @@
[![Build Status](https://travis-ci.com/harlow/kinesis-consumer.svg?branch=master)](https://travis-ci.com/harlow/kinesis-consumer) [![GoDoc](https://godoc.org/github.com/harlow/kinesis-consumer?status.svg)](https://godoc.org/github.com/harlow/kinesis-consumer) [![Build Status](https://travis-ci.com/harlow/kinesis-consumer.svg?branch=master)](https://travis-ci.com/harlow/kinesis-consumer) [![GoDoc](https://godoc.org/github.com/harlow/kinesis-consumer?status.svg)](https://godoc.org/github.com/harlow/kinesis-consumer)
__Note:__ This repo is under active development adding [Consumer Groups #42](https://github.com/harlow/kinesis-consumer/issues/42). Master should always be deployable, but expect breaking changes in master over the next few months. __Note:__ This repo is under active development adding [Consumer Groups #42](https://github.com/harlow/kinesis-consumer/issues/42). Master should always be deployable, but expect breaking changes in master over the next few months.
Latest stable release https://github.com/harlow/kinesis-consumer/releases/tag/v0.3.2 Latest stable release https://github.com/harlow/kinesis-consumer/releases/tag/v0.3.2
@ -104,9 +104,13 @@ err := c.Scan(ctx, func(r *consumer.Record) error {
}) })
``` ```
## Checkpoint ## Options
To record the progress of the consumer in the stream we use a checkpoint to store the last sequence number the consumer has read from a particular shard. The boolean value SkipCheckpoint of consumer.ScanError determines if checkpoint will be activated. ScanError is returned by the record processing callback. The consumer allows the following optional overrides.
### Storage
To record the progress of the consumer in the stream (checkpoint) we use a storage layer to persist the last sequence number the consumer has read from a particular shard. The boolean value ErrSkipCheckpoint of consumer.ScanError determines if checkpoint will be activated. ScanError is returned by the record processing callback.
This will allow consumers to re-launch and pick up at the position in the stream where they left off. This will allow consumers to re-launch and pick up at the position in the stream where they left off.
@ -114,33 +118,42 @@ The uniq identifier for a consumer is `[appName, streamName, shardID]`
<img width="722" alt="kinesis-checkpoints" src="https://user-images.githubusercontent.com/739782/33085867-d8336122-ce9a-11e7-8c8a-a8afeb09dff1.png"> <img width="722" alt="kinesis-checkpoints" src="https://user-images.githubusercontent.com/739782/33085867-d8336122-ce9a-11e7-8c8a-a8afeb09dff1.png">
Note: The default checkpoint is no-op. Which means the scan will not persist any state and the consumer will start from the beginning of the stream each time it is re-started. Note: The default storage is in-memory (no-op). Which means the scan will not persist any state and the consumer will start from the beginning of the stream each time it is re-started.
To persist scan progress choose one of the following checkpoints: The consumer accpets a `WithStorage` option to set the storage layer:
### Redis Checkpoint ```go
c, err := consumer.New(*stream, consumer.WithStorage(db))
if err != nil {
log.Log("consumer error: %v", err)
}
```
To persist scan progress choose one of the following storage layers:
#### Redis
The Redis checkpoint requries App Name, and Stream Name: The Redis checkpoint requries App Name, and Stream Name:
```go ```go
import checkpoint "github.com/harlow/kinesis-consumer/checkpoint/redis" import storage "github.com/harlow/kinesis-consumer/store/redis"
// redis checkpoint // redis checkpoint
ck, err := checkpoint.New(appName) db, err := storage.New(appName)
if err != nil { if err != nil {
log.Fatalf("new checkpoint error: %v", err) log.Fatalf("new checkpoint error: %v", err)
} }
``` ```
### DynamoDB Checkpoint #### DynamoDB
The DynamoDB checkpoint requires Table Name, App Name, and Stream Name: The DynamoDB checkpoint requires Table Name, App Name, and Stream Name:
```go ```go
import checkpoint "github.com/harlow/kinesis-consumer/checkpoint/ddb" import storage "github.com/harlow/kinesis-consumer/store/ddb"
// ddb checkpoint // ddb checkpoint
ck, err := checkpoint.New(appName, tableName) db, err := storage.New(appName, tableName)
if err != nil { if err != nil {
log.Fatalf("new checkpoint error: %v", err) log.Fatalf("new checkpoint error: %v", err)
} }
@ -154,7 +167,7 @@ myDynamoDbClient := dynamodb.New(session.New(aws.NewConfig()))
// Region: aws.String("us-west-2"), // Region: aws.String("us-west-2"),
// }) // })
ck, err := checkpoint.New(*app, *table, checkpoint.WithDynamoClient(myDynamoDbClient)) db, err := storage.New(*app, *table, checkpoint.WithDynamoClient(myDynamoDbClient))
if err != nil { if err != nil {
log.Fatalf("new checkpoint error: %v", err) log.Fatalf("new checkpoint error: %v", err)
} }
@ -173,15 +186,15 @@ Sort key: shard_id
<img width="727" alt="screen shot 2017-11-22 at 7 59 36 pm" src="https://user-images.githubusercontent.com/739782/33158557-b90e4228-cfbf-11e7-9a99-73b56a446f5f.png"> <img width="727" alt="screen shot 2017-11-22 at 7 59 36 pm" src="https://user-images.githubusercontent.com/739782/33158557-b90e4228-cfbf-11e7-9a99-73b56a446f5f.png">
### Postgres Checkpoint #### Postgres
The Postgres checkpoint requires Table Name, App Name, Stream Name and ConnectionString: The Postgres checkpoint requires Table Name, App Name, Stream Name and ConnectionString:
```go ```go
import checkpoint "github.com/harlow/kinesis-consumer/checkpoint/postgres" import storage "github.com/harlow/kinesis-consumer/store/postgres"
// postgres checkpoint // postgres checkpoint
ck, err := checkpoint.New(app, table, connStr) db, err := storage.New(app, table, connStr)
if err != nil { if err != nil {
log.Fatalf("new checkpoint error: %v", err) log.Fatalf("new checkpoint error: %v", err)
} }
@ -201,15 +214,15 @@ CREATE TABLE kinesis_consumer (
The table name has to be the same that you specify when creating the checkpoint. The primary key composed by namespace and shard_id is mandatory in order to the checkpoint run without issues and also to ensure data integrity. The table name has to be the same that you specify when creating the checkpoint. The primary key composed by namespace and shard_id is mandatory in order to the checkpoint run without issues and also to ensure data integrity.
### Mysql Checkpoint #### Mysql
The Mysql checkpoint requires Table Name, App Name, Stream Name and ConnectionString (just like the Postgres checkpoint!): The Mysql checkpoint requires Table Name, App Name, Stream Name and ConnectionString (just like the Postgres checkpoint!):
```go ```go
import checkpoint "github.com/harlow/kinesis-consumer/checkpoint/mysql" import storage "github.com/harlow/kinesis-consumer/store/mysql"
// mysql checkpoint // mysql checkpoint
ck, err := checkpoint.New(app, table, connStr) db, err := storage.New(app, table, connStr)
if err != nil { if err != nil {
log.Fatalf("new checkpoint error: %v", err) log.Fatalf("new checkpoint error: %v", err)
} }
@ -229,10 +242,6 @@ CREATE TABLE kinesis_consumer (
The table name has to be the same that you specify when creating the checkpoint. The primary key composed by namespace and shard_id is mandatory in order to the checkpoint run without issues and also to ensure data integrity. The table name has to be the same that you specify when creating the checkpoint. The primary key composed by namespace and shard_id is mandatory in order to the checkpoint run without issues and also to ensure data integrity.
## Options
The consumer allows the following optional overrides.
### Kinesis Client ### Kinesis Client
Override the Kinesis client if there is any special config needed: Override the Kinesis client if there is any special config needed:

16
allgroup.go
View file

@ -11,13 +11,13 @@ import (
// NewAllGroup returns an intitialized AllGroup for consuming // NewAllGroup returns an intitialized AllGroup for consuming
// all shards on a stream // all shards on a stream
func NewAllGroup(ksis kinesisiface.KinesisAPI, ck Checkpoint, streamName string, logger Logger) *AllGroup { func NewAllGroup(ksis kinesisiface.KinesisAPI, store Store, streamName string, logger Logger) *AllGroup {
return &AllGroup{ return &AllGroup{
ksis: ksis, ksis: ksis,
shards: make(map[string]*kinesis.Shard), shards: make(map[string]*kinesis.Shard),
streamName: streamName, streamName: streamName,
logger: logger, logger: logger,
checkpoint: ck, Store: store,
} }
} }
@ -27,7 +27,7 @@ type AllGroup struct {
ksis kinesisiface.KinesisAPI ksis kinesisiface.KinesisAPI
streamName string streamName string
logger Logger logger Logger
checkpoint Checkpoint Store
shardMu sync.Mutex shardMu sync.Mutex
shards map[string]*kinesis.Shard shards map[string]*kinesis.Shard
@ -59,16 +59,6 @@ func (g *AllGroup) Start(ctx context.Context, shardc chan *kinesis.Shard) {
} }
} }
// GetCheckpoint returns the current checkpoint for provided stream
func (g *AllGroup) GetCheckpoint(streamName, shardID string) (string, error) {
return g.checkpoint.Get(streamName, shardID)
}
// SetCheckpoint sets the current checkpoint for provided stream
func (g *AllGroup) SetCheckpoint(streamName, shardID, sequenceNumber string) error {
return g.checkpoint.Set(streamName, shardID, sequenceNumber)
}
// findNewShards pulls the list of shards from the Kinesis API // findNewShards pulls the list of shards from the Kinesis API
// and uses a local cache to determine if we are already processing // and uses a local cache to determine if we are already processing
// a particular shard. // a particular shard.

13
checkpoint.go
View file

@ -1,13 +0,0 @@
package consumer
// Checkpoint interface used track consumer progress in the stream
type Checkpoint interface {
Get(streamName, shardID string) (string, error)
Set(streamName, shardID, sequenceNumber string) error
}
// noopCheckpoint implements the checkpoint interface with discard
type noopCheckpoint struct{}
func (n noopCheckpoint) Set(string, string, string) error { return nil }
func (n noopCheckpoint) Get(string, string) (string, error) { return "", nil }

16
consumer.go
View file

@ -23,12 +23,12 @@ func New(streamName string, opts ...Option) (*Consumer, error) {
return nil, fmt.Errorf("must provide stream name") return nil, fmt.Errorf("must provide stream name")
} }
// new consumer with no-op checkpoint, counter, and logger // new consumer with noop storage, counter, and logger
c := &Consumer{ c := &Consumer{
streamName: streamName, streamName: streamName,
initialShardIteratorType: kinesis.ShardIteratorTypeLatest, initialShardIteratorType: kinesis.ShardIteratorTypeLatest,
store: &noopStore{},
counter: &noopCounter{}, counter: &noopCounter{},
checkpoint: &noopCheckpoint{},
logger: &noopLogger{ logger: &noopLogger{
logger: log.New(ioutil.Discard, "", log.LstdFlags), logger: log.New(ioutil.Discard, "", log.LstdFlags),
}, },
@ -39,7 +39,7 @@ func New(streamName string, opts ...Option) (*Consumer, error) {
opt(c) opt(c)
} }
// default client if none provided // default client
if c.client == nil { if c.client == nil {
newSession, err := session.NewSession(aws.NewConfig()) newSession, err := session.NewSession(aws.NewConfig())
if err != nil { if err != nil {
@ -48,9 +48,9 @@ func New(streamName string, opts ...Option) (*Consumer, error) {
c.client = kinesis.New(newSession) c.client = kinesis.New(newSession)
} }
// default group if none provided // default group consumes all shards
if c.group == nil { if c.group == nil {
c.group = NewAllGroup(c.client, c.checkpoint, c.streamName, c.logger) c.group = NewAllGroup(c.client, c.store, streamName, c.logger)
} }
return c, nil return c, nil
@ -61,10 +61,10 @@ type Consumer struct {
streamName string streamName string
initialShardIteratorType string initialShardIteratorType string
client kinesisiface.KinesisAPI client kinesisiface.KinesisAPI
logger Logger
group Group
checkpoint Checkpoint
counter Counter counter Counter
group Group
logger Logger
store Store
} }
// ScanFunc is the type of the function called for each message read // ScanFunc is the type of the function called for each message read

18
consumer_test.go
View file

@ -57,7 +57,7 @@ func TestScan(t *testing.T) {
c, err := New("myStreamName", c, err := New("myStreamName",
WithClient(client), WithClient(client),
WithCounter(ctr), WithCounter(ctr),
WithCheckpoint(cp), WithStorage(cp),
) )
if err != nil { if err != nil {
t.Fatalf("new consumer error: %v", err) t.Fatalf("new consumer error: %v", err)
@ -90,7 +90,7 @@ func TestScan(t *testing.T) {
t.Errorf("counter error expected %d, got %d", 2, val) t.Errorf("counter error expected %d, got %d", 2, val)
} }
val, err := cp.Get("myStreamName", "myShard") val, err := cp.GetCheckpoint("myStreamName", "myShard")
if err != nil && val != "lastSeqNum" { if err != nil && val != "lastSeqNum" {
t.Errorf("checkout error expected %s, got %s", "lastSeqNum", val) t.Errorf("checkout error expected %s, got %s", "lastSeqNum", val)
} }
@ -119,7 +119,7 @@ func TestScanShard(t *testing.T) {
c, err := New("myStreamName", c, err := New("myStreamName",
WithClient(client), WithClient(client),
WithCounter(ctr), WithCounter(ctr),
WithCheckpoint(cp), WithStorage(cp),
) )
if err != nil { if err != nil {
t.Fatalf("new consumer error: %v", err) t.Fatalf("new consumer error: %v", err)
@ -156,7 +156,7 @@ func TestScanShard(t *testing.T) {
} }
// sets checkpoint // sets checkpoint
val, err := cp.Get("myStreamName", "myShard") val, err := cp.GetCheckpoint("myStreamName", "myShard")
if err != nil && val != "lastSeqNum" { if err != nil && val != "lastSeqNum" {
t.Fatalf("checkout error expected %s, got %s", "lastSeqNum", val) t.Fatalf("checkout error expected %s, got %s", "lastSeqNum", val)
} }
@ -219,7 +219,7 @@ func TestScanShard_SkipCheckpoint(t *testing.T) {
var cp = &fakeCheckpoint{cache: map[string]string{}} var cp = &fakeCheckpoint{cache: map[string]string{}}
c, err := New("myStreamName", WithClient(client), WithCheckpoint(cp)) c, err := New("myStreamName", WithClient(client), WithStorage(cp))
if err != nil { if err != nil {
t.Fatalf("new consumer error: %v", err) t.Fatalf("new consumer error: %v", err)
} }
@ -229,7 +229,7 @@ func TestScanShard_SkipCheckpoint(t *testing.T) {
var fn = func(r *Record) error { var fn = func(r *Record) error {
if aws.StringValue(r.SequenceNumber) == "lastSeqNum" { if aws.StringValue(r.SequenceNumber) == "lastSeqNum" {
cancel() cancel()
return SkipCheckpoint return ErrSkipCheckpoint
} }
return nil return nil
@ -240,7 +240,7 @@ func TestScanShard_SkipCheckpoint(t *testing.T) {
t.Fatalf("scan shard error: %v", err) t.Fatalf("scan shard error: %v", err)
} }
val, err := cp.Get("myStreamName", "myShard") val, err := cp.GetCheckpoint("myStreamName", "myShard")
if err != nil && val != "firstSeqNum" { if err != nil && val != "firstSeqNum" {
t.Fatalf("checkout error expected %s, got %s", "firstSeqNum", val) t.Fatalf("checkout error expected %s, got %s", "firstSeqNum", val)
} }
@ -301,7 +301,7 @@ type fakeCheckpoint struct {
mu sync.Mutex mu sync.Mutex
} }
func (fc *fakeCheckpoint) Set(streamName, shardID, sequenceNumber string) error { func (fc *fakeCheckpoint) SetCheckpoint(streamName, shardID, sequenceNumber string) error {
fc.mu.Lock() fc.mu.Lock()
defer fc.mu.Unlock() defer fc.mu.Unlock()
@ -310,7 +310,7 @@ func (fc *fakeCheckpoint) Set(streamName, shardID, sequenceNumber string) error
return nil return nil
} }
func (fc *fakeCheckpoint) Get(streamName, shardID string) (string, error) { func (fc *fakeCheckpoint) GetCheckpoint(streamName, shardID string) (string, error) {
fc.mu.Lock() fc.mu.Lock()
defer fc.mu.Unlock() defer fc.mu.Unlock()

3
examples/consumer/cp-dynamo/README.md
View file

@ -7,9 +7,8 @@ Read records from the Kinesis stream
Export the required environment vars for connecting to the Kinesis stream: Export the required environment vars for connecting to the Kinesis stream:
``` ```
export AWS_ACCESS_KEY= export AWS_PROFILE=
export AWS_REGION= export AWS_REGION=
export AWS_SECRET_KEY=
``` ```
### Run the consumer ### Run the consumer

18
examples/consumer/cp-dynamo/main.go
View file

@ -19,7 +19,7 @@ import (
"github.com/aws/aws-sdk-go/service/dynamodb" "github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/kinesis" "github.com/aws/aws-sdk-go/service/kinesis"
consumer "github.com/harlow/kinesis-consumer" consumer "github.com/harlow/kinesis-consumer"
checkpoint "github.com/harlow/kinesis-consumer/checkpoint/ddb" storage "github.com/harlow/kinesis-consumer/store/ddb"
) )
// kick off a server for exposing scan metrics // kick off a server for exposing scan metrics
@ -69,8 +69,8 @@ func main() {
myKsis := kinesis.New(sess) myKsis := kinesis.New(sess)
myDdbClient := dynamodb.New(sess) myDdbClient := dynamodb.New(sess)
// ddb checkpoint // ddb persitance
ck, err := checkpoint.New(*app, *table, checkpoint.WithDynamoClient(myDdbClient), checkpoint.WithRetryer(&MyRetryer{})) ddb, err := storage.New(*app, *table, storage.WithDynamoClient(myDdbClient), storage.WithRetryer(&MyRetryer{}))
if err != nil { if err != nil {
log.Log("checkpoint error: %v", err) log.Log("checkpoint error: %v", err)
} }
@ -81,7 +81,7 @@ func main() {
// consumer // consumer
c, err := consumer.New( c, err := consumer.New(
*stream, *stream,
consumer.WithCheckpoint(ck), consumer.WithStorage(ddb),
consumer.WithLogger(log), consumer.WithLogger(log),
consumer.WithCounter(counter), consumer.WithCounter(counter),
consumer.WithClient(myKsis), consumer.WithClient(myKsis),
@ -111,17 +111,17 @@ func main() {
log.Log("scan error: %v", err) log.Log("scan error: %v", err)
} }
if err := ck.Shutdown(); err != nil { if err := ddb.Shutdown(); err != nil {
log.Log("checkpoint shutdown error: %v", err) log.Log("storage shutdown error: %v", err)
} }
} }
// MyRetryer used for checkpointing // MyRetryer used for storage
type MyRetryer struct { type MyRetryer struct {
checkpoint.Retryer storage.Retryer
} }
// ShouldRetry implements custom logic for when a checkpont should retry // ShouldRetry implements custom logic for when errors should retry
func (r *MyRetryer) ShouldRetry(err error) bool { func (r *MyRetryer) ShouldRetry(err error) bool {
if awsErr, ok := err.(awserr.Error); ok { if awsErr, ok := err.(awserr.Error); ok {
switch awsErr.Code() { switch awsErr.Code() {

5
examples/consumer/cp-mysql/README.md
View file

@ -7,9 +7,8 @@ Read records from the Kinesis stream using mysql as checkpoint
Export the required environment vars for connecting to the Kinesis stream: Export the required environment vars for connecting to the Kinesis stream:
```shell ```shell
export AWS_ACCESS_KEY= export AWS_PROFILE=
export AWS_REGION= export AWS_REGION=
export AWS_SECRET_KEY=
``` ```
## Run the consumer ## Run the consumer
@ -18,4 +17,4 @@ export AWS_SECRET_KEY=
Connection string should look something like Connection string should look something like
user:password@/dbname user:password@/dbname

4
examples/consumer/cp-mysql/main.go
View file

@ -10,7 +10,7 @@ import (
"os/signal" "os/signal"
consumer "github.com/harlow/kinesis-consumer" consumer "github.com/harlow/kinesis-consumer"
checkpoint "github.com/harlow/kinesis-consumer/checkpoint/mysql" checkpoint "github.com/harlow/kinesis-consumer/store/mysql"
) )
func main() { func main() {
@ -33,7 +33,7 @@ func main() {
// consumer // consumer
c, err := consumer.New( c, err := consumer.New(
*stream, *stream,
consumer.WithCheckpoint(ck), consumer.WithStorage(ck),
consumer.WithCounter(counter), consumer.WithCounter(counter),
) )
if err != nil { if err != nil {

3
examples/consumer/cp-postgres/README.md
View file

@ -7,9 +7,8 @@ Read records from the Kinesis stream using postgres as checkpoint
Export the required environment vars for connecting to the Kinesis stream: Export the required environment vars for connecting to the Kinesis stream:
```shell ```shell
export AWS_ACCESS_KEY= export AWS_PROFILE=
export AWS_REGION= export AWS_REGION=
export AWS_SECRET_KEY=
``` ```
## Run the consumer ## Run the consumer

4
examples/consumer/cp-postgres/main.go
View file

@ -10,7 +10,7 @@ import (
"os/signal" "os/signal"
consumer "github.com/harlow/kinesis-consumer" consumer "github.com/harlow/kinesis-consumer"
checkpoint "github.com/harlow/kinesis-consumer/checkpoint/postgres" checkpoint "github.com/harlow/kinesis-consumer/store/postgres"
) )
func main() { func main() {
@ -33,7 +33,7 @@ func main() {
// consumer // consumer
c, err := consumer.New( c, err := consumer.New(
*stream, *stream,
consumer.WithCheckpoint(ck), consumer.WithStorage(ck),
consumer.WithCounter(counter), consumer.WithCounter(counter),
) )
if err != nil { if err != nil {

4
examples/consumer/cp-redis/main.go
View file

@ -9,7 +9,7 @@ import (
"os/signal" "os/signal"
consumer "github.com/harlow/kinesis-consumer" consumer "github.com/harlow/kinesis-consumer"
checkpoint "github.com/harlow/kinesis-consumer/checkpoint/redis" checkpoint "github.com/harlow/kinesis-consumer/store/redis"
) )
// A myLogger provides a minimalistic logger satisfying the Logger interface. // A myLogger provides a minimalistic logger satisfying the Logger interface.
@ -43,7 +43,7 @@ func main() {
// consumer // consumer
c, err := consumer.New( c, err := consumer.New(
*stream, *stream,
consumer.WithCheckpoint(ck), consumer.WithStorage(ck),
consumer.WithLogger(logger), consumer.WithLogger(logger),
) )
if err != nil { if err != nil {

1
go.mod
View file

@ -7,6 +7,7 @@ require (
github.com/go-sql-driver/mysql v1.4.1 github.com/go-sql-driver/mysql v1.4.1
github.com/lib/pq v0.0.0-20180523175426-90697d60dd84 github.com/lib/pq v0.0.0-20180523175426-90697d60dd84
github.com/pkg/errors v0.8.0 github.com/pkg/errors v0.8.0
github.com/stretchr/testify v1.3.0 // indirect
google.golang.org/appengine v1.6.1 // indirect google.golang.org/appengine v1.6.1 // indirect
gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0
gopkg.in/redis.v5 v5.2.9 gopkg.in/redis.v5 v5.2.9

7
go.sum
View file

@ -2,6 +2,8 @@ github.com/apex/log v1.0.0 h1:5UWeZC54mWVtOGSCjtuvDPgY/o0QxmjQgvYZ27pLVGQ=
github.com/apex/log v1.0.0/go.mod h1:yA770aXIDQrhVOIGurT/pVdfCpSq1GQV/auzMN5fzvY= github.com/apex/log v1.0.0/go.mod h1:yA770aXIDQrhVOIGurT/pVdfCpSq1GQV/auzMN5fzvY=
github.com/aws/aws-sdk-go v1.15.0 h1:uxi9gcf4jxEX7r8oWYMEkYB4kziKet+1cHPmq52LjC4= github.com/aws/aws-sdk-go v1.15.0 h1:uxi9gcf4jxEX7r8oWYMEkYB4kziKet+1cHPmq52LjC4=
github.com/aws/aws-sdk-go v1.15.0/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0= github.com/aws/aws-sdk-go v1.15.0/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-ini/ini v1.25.4/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8= github.com/go-ini/ini v1.25.4/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
github.com/go-ini/ini v1.38.1 h1:hbtfM8emWUVo9GnXSloXYyFbXxZ+tG6sbepSStoe1FY= github.com/go-ini/ini v1.38.1 h1:hbtfM8emWUVo9GnXSloXYyFbXxZ+tG6sbepSStoe1FY=
github.com/go-ini/ini v1.38.1/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8= github.com/go-ini/ini v1.38.1/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
@ -14,6 +16,11 @@ github.com/lib/pq v0.0.0-20180523175426-90697d60dd84 h1:it29sI2IM490luSc3RAhp5Wu
github.com/lib/pq v0.0.0-20180523175426-90697d60dd84/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= github.com/lib/pq v0.0.0-20180523175426-90697d60dd84/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/pkg/errors v0.8.0 h1:WdK/asTD0HN+q6hsWO3/vpuAkAr+tw6aNJNDFFf0+qw= github.com/pkg/errors v0.8.0 h1:WdK/asTD0HN+q6hsWO3/vpuAkAr+tw6aNJNDFFf0+qw=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=

6
options.go
View file

@ -5,10 +5,10 @@ import "github.com/aws/aws-sdk-go/service/kinesis/kinesisiface"
// Option is used to override defaults when creating a new Consumer // Option is used to override defaults when creating a new Consumer
type Option func(*Consumer) type Option func(*Consumer)
// WithCheckpoint overrides the default checkpoint // WithStorage overrides the default storage
func WithCheckpoint(checkpoint Checkpoint) Option { func WithStorage(store Store) Option {
return func(c *Consumer) { return func(c *Consumer) {
c.checkpoint = checkpoint c.store = store
} }
} }

13
store.go Normal file
View file

@ -0,0 +1,13 @@
package consumer
// Store interface used to persist scan progress
type Store interface {
GetCheckpoint(streamName, shardID string) (string, error)
SetCheckpoint(streamName, shardID, sequenceNumber string) error
}
// noopStore implements the storage interface with discard
type noopStore struct{}
func (n noopStore) GetCheckpoint(string, string) (string, error) { return "", nil }
func (n noopStore) SetCheckpoint(string, string, string) error { return nil }

8
checkpoint/ddb/ddb.go → store/ddb/ddb.go
View file

@ -87,7 +87,7 @@ type item struct {
// Get determines if a checkpoint for a particular Shard exists. // Get determines if a checkpoint for a particular Shard exists.
// Typically used to determine whether we should start processing the shard with // Typically used to determine whether we should start processing the shard with
// TRIM_HORIZON or AFTER_SEQUENCE_NUMBER (if checkpoint exists). // TRIM_HORIZON or AFTER_SEQUENCE_NUMBER (if checkpoint exists).
func (c *Checkpoint) Get(streamName, shardID string) (string, error) { func (c *Checkpoint) GetCheckpoint(streamName, shardID string) (string, error) {
namespace := fmt.Sprintf("%s-%s", c.appName, streamName) namespace := fmt.Sprintf("%s-%s", c.appName, streamName)
params := &dynamodb.GetItemInput{ params := &dynamodb.GetItemInput{
@ -106,7 +106,7 @@ func (c *Checkpoint) Get(streamName, shardID string) (string, error) {
resp, err := c.client.GetItem(params) resp, err := c.client.GetItem(params)
if err != nil { if err != nil {
if c.retryer.ShouldRetry(err) { if c.retryer.ShouldRetry(err) {
return c.Get(streamName, shardID) return c.GetCheckpoint(streamName, shardID)
} }
return "", err return "", err
} }
@ -116,9 +116,9 @@ func (c *Checkpoint) Get(streamName, shardID string) (string, error) {
return i.SequenceNumber, nil return i.SequenceNumber, nil
} }
// Set stores a checkpoint for a shard (e.g. sequence number of last record processed by application). // SetCheckpoint stores a checkpoint for a shard (e.g. sequence number of last record processed by application).
// Upon failover, record processing is resumed from this point. // Upon failover, record processing is resumed from this point.
func (c *Checkpoint) Set(streamName, shardID, sequenceNumber string) error { func (c *Checkpoint) SetCheckpoint(streamName, shardID, sequenceNumber string) error {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()

0
checkpoint/ddb/ddb_test.go → store/ddb/ddb_test.go
View file

0
checkpoint/ddb/retryer.go → store/ddb/retryer.go
View file

0
checkpoint/ddb/retryer_test.go → store/ddb/retryer_test.go
View file

8
checkpoint/mysql/mysql.go → store/mysql/mysql.go
View file

@ -77,10 +77,10 @@ func (c *Checkpoint) GetMaxInterval() time.Duration {
return c.maxInterval return c.maxInterval
} }
// Get determines if a checkpoint for a particular Shard exists. // GetCheckpoint determines if a checkpoint for a particular Shard exists.
// Typically used to determine whether we should start processing the shard with // Typically used to determine whether we should start processing the shard with
// TRIM_HORIZON or AFTER_SEQUENCE_NUMBER (if checkpoint exists). // TRIM_HORIZON or AFTER_SEQUENCE_NUMBER (if checkpoint exists).
func (c *Checkpoint) Get(streamName, shardID string) (string, error) { func (c *Checkpoint) GetCheckpoint(streamName, shardID string) (string, error) {
namespace := fmt.Sprintf("%s-%s", c.appName, streamName) namespace := fmt.Sprintf("%s-%s", c.appName, streamName)
var sequenceNumber string var sequenceNumber string
@ -97,9 +97,9 @@ func (c *Checkpoint) Get(streamName, shardID string) (string, error) {
return sequenceNumber, nil return sequenceNumber, nil
} }
// Set stores a checkpoint for a shard (e.g. sequence number of last record processed by application). // SetCheckpoint stores a checkpoint for a shard (e.g. sequence number of last record processed by application).
// Upon failover, record processing is resumed from this point. // Upon failover, record processing is resumed from this point.
func (c *Checkpoint) Set(streamName, shardID, sequenceNumber string) error { func (c *Checkpoint) SetCheckpoint(streamName, shardID, sequenceNumber string) error {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()

0
checkpoint/mysql/mysql_databaseutils_test.go → store/mysql/mysql_databaseutils_test.go
View file

18
checkpoint/mysql/mysql_test.go → store/mysql/mysql_test.go
View file

@ -72,7 +72,7 @@ func TestNew_WithMaxIntervalOption(t *testing.T) {
ck.Shutdown() ck.Shutdown()
} }
func TestCheckpoint_Get(t *testing.T) { func TestCheckpoint_GetCheckpoint(t *testing.T) {
appName := "streamConsumer" appName := "streamConsumer"
tableName := "checkpoint" tableName := "checkpoint"
connString := "user:password@/dbname" connString := "user:password@/dbname"
@ -98,7 +98,7 @@ func TestCheckpoint_Get(t *testing.T) {
tableName) tableName)
mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnRows(expectedRows) mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnRows(expectedRows)
gotSequenceNumber, err := ck.Get(streamName, shardID) gotSequenceNumber, err := ck.GetCheckpoint(streamName, shardID)
if gotSequenceNumber != expectedSequenceNumber { if gotSequenceNumber != expectedSequenceNumber {
t.Errorf("expected sequence number equals %v, but got %v", expectedSequenceNumber, gotSequenceNumber) t.Errorf("expected sequence number equals %v, but got %v", expectedSequenceNumber, gotSequenceNumber)
@ -134,7 +134,7 @@ func TestCheckpoint_Get_NoRows(t *testing.T) {
tableName) tableName)
mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnError(sql.ErrNoRows) mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnError(sql.ErrNoRows)
gotSequenceNumber, err := ck.Get(streamName, shardID) gotSequenceNumber, err := ck.GetCheckpoint(streamName, shardID)
if gotSequenceNumber != "" { if gotSequenceNumber != "" {
t.Errorf("expected sequence number equals empty, but got %v", gotSequenceNumber) t.Errorf("expected sequence number equals empty, but got %v", gotSequenceNumber)
@ -170,7 +170,7 @@ func TestCheckpoint_Get_QueryError(t *testing.T) {
tableName) tableName)
mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnError(errors.New("an error")) mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnError(errors.New("an error"))
gotSequenceNumber, err := ck.Get(streamName, shardID) gotSequenceNumber, err := ck.GetCheckpoint(streamName, shardID)
if gotSequenceNumber != "" { if gotSequenceNumber != "" {
t.Errorf("expected sequence number equals empty, but got %v", gotSequenceNumber) t.Errorf("expected sequence number equals empty, but got %v", gotSequenceNumber)
@ -184,7 +184,7 @@ func TestCheckpoint_Get_QueryError(t *testing.T) {
ck.Shutdown() ck.Shutdown()
} }
func TestCheckpoint_Set(t *testing.T) { func TestCheckpoint_SetCheckpoint(t *testing.T) {
appName := "streamConsumer" appName := "streamConsumer"
tableName := "checkpoint" tableName := "checkpoint"
connString := "user:password@/dbname" connString := "user:password@/dbname"
@ -197,7 +197,7 @@ func TestCheckpoint_Set(t *testing.T) {
t.Fatalf("error occurred during the checkpoint creation. cause: %v", err) t.Fatalf("error occurred during the checkpoint creation. cause: %v", err)
} }
err = ck.Set(streamName, shardID, expectedSequenceNumber) err = ck.SetCheckpoint(streamName, shardID, expectedSequenceNumber)
if err != nil { if err != nil {
t.Errorf("expected error equals nil, but got %v", err) t.Errorf("expected error equals nil, but got %v", err)
@ -218,7 +218,7 @@ func TestCheckpoint_Set_SequenceNumberEmpty(t *testing.T) {
t.Fatalf("error occurred during the checkpoint creation. cause: %v", err) t.Fatalf("error occurred during the checkpoint creation. cause: %v", err)
} }
err = ck.Set(streamName, shardID, expectedSequenceNumber) err = ck.SetCheckpoint(streamName, shardID, expectedSequenceNumber)
if err == nil { if err == nil {
t.Errorf("expected error equals not nil, but got %v", err) t.Errorf("expected error equals not nil, but got %v", err)
@ -249,7 +249,7 @@ func TestCheckpoint_Shutdown(t *testing.T) {
result := sqlmock.NewResult(0, 1) result := sqlmock.NewResult(0, 1)
mock.ExpectExec(expectedSQLRegexString).WithArgs(namespace, shardID, expectedSequenceNumber).WillReturnResult(result) mock.ExpectExec(expectedSQLRegexString).WithArgs(namespace, shardID, expectedSequenceNumber).WillReturnResult(result)
err = ck.Set(streamName, shardID, expectedSequenceNumber) err = ck.SetCheckpoint(streamName, shardID, expectedSequenceNumber)
if err != nil { if err != nil {
t.Fatalf("unable to set checkpoint for data initialization. cause: %v", err) t.Fatalf("unable to set checkpoint for data initialization. cause: %v", err)
@ -287,7 +287,7 @@ func TestCheckpoint_Shutdown_SaveError(t *testing.T) {
expectedSQLRegexString := fmt.Sprintf(`REPLACE INTO %s \(namespace, shard_id, sequence_number\) VALUES \(\?, \?, \?\)`, tableName) expectedSQLRegexString := fmt.Sprintf(`REPLACE INTO %s \(namespace, shard_id, sequence_number\) VALUES \(\?, \?, \?\)`, tableName)
mock.ExpectExec(expectedSQLRegexString).WithArgs(namespace, shardID, expectedSequenceNumber).WillReturnError(errors.New("an error")) mock.ExpectExec(expectedSQLRegexString).WithArgs(namespace, shardID, expectedSequenceNumber).WillReturnError(errors.New("an error"))
err = ck.Set(streamName, shardID, expectedSequenceNumber) err = ck.SetCheckpoint(streamName, shardID, expectedSequenceNumber)
if err != nil { if err != nil {
t.Fatalf("unable to set checkpoint for data initialization. cause: %v", err) t.Fatalf("unable to set checkpoint for data initialization. cause: %v", err)

13
checkpoint/postgres/postgres.go → store/postgres/postgres.go
View file

@ -6,6 +6,7 @@ import (
"fmt" "fmt"
"sync" "sync"
"time" "time"
// this is the postgres package so it makes sense to be here // this is the postgres package so it makes sense to be here
_ "github.com/lib/pq" _ "github.com/lib/pq"
) )
@ -77,10 +78,10 @@ func (c *Checkpoint) GetMaxInterval() time.Duration {
return c.maxInterval return c.maxInterval
} }
// Get determines if a checkpoint for a particular Shard exists. // GetCheckpoint determines if a checkpoint for a particular Shard exists.
// Typically used to determine whether we should start processing the shard with // Typically used to determine whether we should start processing the shard with
// TRIM_HORIZON or AFTER_SEQUENCE_NUMBER (if checkpoint exists). // TRIM_HORIZON or AFTER_SEQUENCE_NUMBER (if checkpoint exists).
func (c *Checkpoint) Get(streamName, shardID string) (string, error) { func (c *Checkpoint) GetCheckpoint(streamName, shardID string) (string, error) {
namespace := fmt.Sprintf("%s-%s", c.appName, streamName) namespace := fmt.Sprintf("%s-%s", c.appName, streamName)
var sequenceNumber string var sequenceNumber string
@ -97,9 +98,9 @@ func (c *Checkpoint) Get(streamName, shardID string) (string, error) {
return sequenceNumber, nil return sequenceNumber, nil
} }
// Set stores a checkpoint for a shard (e.g. sequence number of last record processed by application). // SetCheckpoint stores a checkpoint for a shard (e.g. sequence number of last record processed by application).
// Upon failover, record processing is resumed from this point. // Upon failover, record processing is resumed from this point.
func (c *Checkpoint) Set(streamName, shardID, sequenceNumber string) error { func (c *Checkpoint) SetCheckpoint(streamName, shardID, sequenceNumber string) error {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
@ -149,8 +150,8 @@ func (c *Checkpoint) save() error {
upsertCheckpoint := fmt.Sprintf(`INSERT INTO %s (namespace, shard_id, sequence_number) upsertCheckpoint := fmt.Sprintf(`INSERT INTO %s (namespace, shard_id, sequence_number)
VALUES($1, $2, $3) VALUES($1, $2, $3)
ON CONFLICT (namespace, shard_id) ON CONFLICT (namespace, shard_id)
DO DO
UPDATE UPDATE
SET sequence_number= $3;`, c.tableName) SET sequence_number= $3;`, c.tableName)
for key, sequenceNumber := range c.checkpoints { for key, sequenceNumber := range c.checkpoints {

0
checkpoint/postgres/postgres_databaseutils_test.go → store/postgres/postgres_databaseutils_test.go
View file

18
checkpoint/postgres/postgres_test.go → store/postgres/postgres_test.go
View file

@ -72,7 +72,7 @@ func TestNew_WithMaxIntervalOption(t *testing.T) {
ck.Shutdown() ck.Shutdown()
} }
func TestCheckpoint_Get(t *testing.T) { func TestCheckpoint_GetCheckpoint(t *testing.T) {
appName := "streamConsumer" appName := "streamConsumer"
tableName := "checkpoint" tableName := "checkpoint"
connString := "UserID=root;Password=myPassword;Host=localhost;Port=5432;Database=myDataBase;" connString := "UserID=root;Password=myPassword;Host=localhost;Port=5432;Database=myDataBase;"
@ -98,7 +98,7 @@ func TestCheckpoint_Get(t *testing.T) {
tableName) tableName)
mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnRows(expectedRows) mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnRows(expectedRows)
gotSequenceNumber, err := ck.Get(streamName, shardID) gotSequenceNumber, err := ck.GetCheckpoint(streamName, shardID)
if gotSequenceNumber != expectedSequenceNumber { if gotSequenceNumber != expectedSequenceNumber {
t.Errorf("expected sequence number equals %v, but got %v", expectedSequenceNumber, gotSequenceNumber) t.Errorf("expected sequence number equals %v, but got %v", expectedSequenceNumber, gotSequenceNumber)
@ -134,7 +134,7 @@ func TestCheckpoint_Get_NoRows(t *testing.T) {
tableName) tableName)
mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnError(sql.ErrNoRows) mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnError(sql.ErrNoRows)
gotSequenceNumber, err := ck.Get(streamName, shardID) gotSequenceNumber, err := ck.GetCheckpoint(streamName, shardID)
if gotSequenceNumber != "" { if gotSequenceNumber != "" {
t.Errorf("expected sequence number equals empty, but got %v", gotSequenceNumber) t.Errorf("expected sequence number equals empty, but got %v", gotSequenceNumber)
@ -170,7 +170,7 @@ func TestCheckpoint_Get_QueryError(t *testing.T) {
tableName) tableName)
mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnError(errors.New("an error")) mock.ExpectQuery(expectedSQLRegexString).WithArgs(namespace, shardID).WillReturnError(errors.New("an error"))
gotSequenceNumber, err := ck.Get(streamName, shardID) gotSequenceNumber, err := ck.GetCheckpoint(streamName, shardID)
if gotSequenceNumber != "" { if gotSequenceNumber != "" {
t.Errorf("expected sequence number equals empty, but got %v", gotSequenceNumber) t.Errorf("expected sequence number equals empty, but got %v", gotSequenceNumber)
@ -184,7 +184,7 @@ func TestCheckpoint_Get_QueryError(t *testing.T) {
ck.Shutdown() ck.Shutdown()
} }
func TestCheckpoint_Set(t *testing.T) { func TestCheckpoint_SetCheckpoint(t *testing.T) {
appName := "streamConsumer" appName := "streamConsumer"
tableName := "checkpoint" tableName := "checkpoint"
connString := "UserID=root;Password=myPassword;Host=localhost;Port=5432;Database=myDataBase;" connString := "UserID=root;Password=myPassword;Host=localhost;Port=5432;Database=myDataBase;"
@ -197,7 +197,7 @@ func TestCheckpoint_Set(t *testing.T) {
t.Fatalf("error occurred during the checkpoint creation. cause: %v", err) t.Fatalf("error occurred during the checkpoint creation. cause: %v", err)
} }
err = ck.Set(streamName, shardID, expectedSequenceNumber) err = ck.SetCheckpoint(streamName, shardID, expectedSequenceNumber)
if err != nil { if err != nil {
t.Errorf("expected error equals nil, but got %v", err) t.Errorf("expected error equals nil, but got %v", err)
@ -218,7 +218,7 @@ func TestCheckpoint_Set_SequenceNumberEmpty(t *testing.T) {
t.Fatalf("error occurred during the checkpoint creation. cause: %v", err) t.Fatalf("error occurred during the checkpoint creation. cause: %v", err)
} }
err = ck.Set(streamName, shardID, expectedSequenceNumber) err = ck.SetCheckpoint(streamName, shardID, expectedSequenceNumber)
if err == nil { if err == nil {
t.Errorf("expected error equals not nil, but got %v", err) t.Errorf("expected error equals not nil, but got %v", err)
@ -249,7 +249,7 @@ func TestCheckpoint_Shutdown(t *testing.T) {
result := sqlmock.NewResult(0, 1) result := sqlmock.NewResult(0, 1)
mock.ExpectExec(expectedSQLRegexString).WithArgs(namespace, shardID, expectedSequenceNumber).WillReturnResult(result) mock.ExpectExec(expectedSQLRegexString).WithArgs(namespace, shardID, expectedSequenceNumber).WillReturnResult(result)
err = ck.Set(streamName, shardID, expectedSequenceNumber) err = ck.SetCheckpoint(streamName, shardID, expectedSequenceNumber)
if err != nil { if err != nil {
t.Fatalf("unable to set checkpoint for data initialization. cause: %v", err) t.Fatalf("unable to set checkpoint for data initialization. cause: %v", err)
@ -287,7 +287,7 @@ func TestCheckpoint_Shutdown_SaveError(t *testing.T) {
expectedSQLRegexString := fmt.Sprintf(`INSERT INTO %s \(namespace, shard_id, sequence_number\) VALUES\(\$1, \$2, \$3\) ON CONFLICT \(namespace, shard_id\) DO UPDATE SET sequence_number= \$3;`, tableName) expectedSQLRegexString := fmt.Sprintf(`INSERT INTO %s \(namespace, shard_id, sequence_number\) VALUES\(\$1, \$2, \$3\) ON CONFLICT \(namespace, shard_id\) DO UPDATE SET sequence_number= \$3;`, tableName)
mock.ExpectExec(expectedSQLRegexString).WithArgs(namespace, shardID, expectedSequenceNumber).WillReturnError(errors.New("an error")) mock.ExpectExec(expectedSQLRegexString).WithArgs(namespace, shardID, expectedSequenceNumber).WillReturnError(errors.New("an error"))
err = ck.Set(streamName, shardID, expectedSequenceNumber) err = ck.SetCheckpoint(streamName, shardID, expectedSequenceNumber)
if err != nil { if err != nil {
t.Fatalf("unable to set checkpoint for data initialization. cause: %v", err) t.Fatalf("unable to set checkpoint for data initialization. cause: %v", err)

8
checkpoint/redis/redis.go → store/redis/redis.go
View file

@ -36,15 +36,15 @@ type Checkpoint struct {
client *redis.Client client *redis.Client
} }
// Get fetches the checkpoint for a particular Shard. // GetCheckpoint fetches the checkpoint for a particular Shard.
func (c *Checkpoint) Get(streamName, shardID string) (string, error) { func (c *Checkpoint) GetCheckpoint(streamName, shardID string) (string, error) {
val, _ := c.client.Get(c.key(streamName, shardID)).Result() val, _ := c.client.Get(c.key(streamName, shardID)).Result()
return val, nil return val, nil
} }
// Set stores a checkpoint for a shard (e.g. sequence number of last record processed by application). // SetCheckpoint stores a checkpoint for a shard (e.g. sequence number of last record processed by application).
// Upon failover, record processing is resumed from this point. // Upon failover, record processing is resumed from this point.
func (c *Checkpoint) Set(streamName, shardID, sequenceNumber string) error { func (c *Checkpoint) SetCheckpoint(streamName, shardID, sequenceNumber string) error {
if sequenceNumber == "" { if sequenceNumber == "" {
return fmt.Errorf("sequence number should not be empty") return fmt.Errorf("sequence number should not be empty")
} }

6
checkpoint/redis/redis_test.go → store/redis/redis_test.go
View file

@ -12,10 +12,10 @@ func Test_CheckpointLifecycle(t *testing.T) {
} }
// set // set
c.Set("streamName", "shardID", "testSeqNum") c.SetCheckpoint("streamName", "shardID", "testSeqNum")
// get // get
val, err := c.Get("streamName", "shardID") val, err := c.GetCheckpoint("streamName", "shardID")
if err != nil { if err != nil {
t.Fatalf("get checkpoint error: %v", err) t.Fatalf("get checkpoint error: %v", err)
} }
@ -30,7 +30,7 @@ func Test_SetEmptySeqNum(t *testing.T) {
t.Fatalf("new checkpoint error: %v", err) t.Fatalf("new checkpoint error: %v", err)
} }
err = c.Set("streamName", "shardID", "") err = c.SetCheckpoint("streamName", "shardID", "")
if err == nil { if err == nil {
t.Fatalf("should not allow empty sequence number") t.Fatalf("should not allow empty sequence number")
} }