Commit graph

10 commits

Author SHA1 Message Date
Jason Tackaberry
3372ea1d17 Add option for immediate rescan
When immediate rescan is enabled, a poll from Kinesis that returns records and
also indicates we are still behind the latest record will skip waiting for the
next scan interval and rescan immediately.

Related, the initial scan for a shard is performed immediately.  The
scan ticker only applies to subsequent scans.

These changes make higher scan intervals feasible, allowing for less
chatty clients while also improving overall effective throughput.
2020-08-01 19:14:03 -04:00
chumbert2
400ef07463
Allow to override Limit parameter in GetRecords (#113)
Add option maxRecords to set the maximum number of records that can
be returned by GetRecords. Default value: 10,000. Use WithMaxRecords
to change the default value.

See

https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/#GetRecordsInput
https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
2020-04-27 21:12:20 -07:00
0livd
bc5c012fd3 Add scanInterval option (#105)
By default a consumer will scan for records every 250ms.
This interval can be configured with WithScanInterval.
2020-01-17 10:22:10 -08:00
Dimas Yudha P
a252eb38c6 Readme and linter fix (#102)
* update readme.md adding goreport

* update readme and fix issue found by linter

* update readme.md add Go badges

* update readme.md, fix go badges
2019-08-31 18:43:26 -07:00
Andrew Shannon Brown
14db23eaf3 Support creating an iterator with an initial timestamp (#99)
* Allow setting initial timestamp

* Fix writing to closed channel

* Allow cancelling of request
2019-08-14 09:33:35 -07:00
Harlow Ward
5da0865ac1 Add WithGroup option 2019-07-28 21:34:54 -07:00
Harlow Ward
a9c97d3b93 Update examples to use Store interface 2019-07-28 21:33:19 -07:00
Harlow Ward
c72f561abd
Replace Checkpoint with Store interface (#90)
As we work towards introducing consumer groups to the repository we need a more generic name for the persistence layer for storing checkpoints and leases for given shards. 

* Rename `checkpoint` to `store`
2019-07-28 21:18:40 -07:00
Harlow Ward
d05d6c2d5e Update comments for exported functions 2019-07-28 10:54:01 -07:00
Harlow Ward
76158d24ab
Introduce ScanFunc signature and remove ScanStatus (#77)
Major changes:

```go
type ScanFunc func(r *Record) error
```

* Simplify the callback func signature by removing `ScanStatus` 
* Leverage context for cancellation 
* Add custom error `SkipCheckpoint` for special cases when we don't want to checkpoint

Minor changes:

* Use kinesis package constants for shard iterator types
* Move optional config to new file

See conversation on #75 for more details
2019-04-07 16:29:12 -07:00