553e2392fd
* fix nil pointer dereference on AWS errors
* return Start errors to Scan consumer
before the previous commit e465b09, client errors panicked the
reader, so consumers would pick up sharditerator errors by virtue of
their server crashing and burning.
Now that client errors are properly returned, the behaviour of
listShards is problematic because it absorbs any client errors it gets.
The result of these two things now is that if you hit an aws error, your server will go into an
endless scan loop you can't detect and can't easily recover from.
To avoid that, listShards will now stop if it hits a client error.
---------
Co-authored-by: Jarrad Whitaker <jwhitaker 📧 swift-nav.com>
21 lines
590 B
Go
21 lines
590 B
Go
package consumer
|
|
|
|
import (
|
|
"context"
|
|
|
|
"github.com/aws/aws-sdk-go-v2/service/kinesis/types"
|
|
)
|
|
|
|
// Group interface used to manage which shard to process
|
|
type Group interface {
|
|
Start(ctx context.Context, shardc chan types.Shard) error
|
|
GetCheckpoint(streamName, shardID string) (string, error)
|
|
SetCheckpoint(streamName, shardID, sequenceNumber string) error
|
|
}
|
|
|
|
type CloseableGroup interface {
|
|
Group
|
|
// Allows shard processors to tell the group when the shard has been
|
|
// fully processed. Should be called only once per shardID.
|
|
CloseShard(ctx context.Context, shardID string) error
|
|
}
|