luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
188 commits 6 branches 9 tags 7.5 MiB
Commit graph

188 commits

This branch
This branch
All branches
Author SHA1 Message Date
Harlow Ward
b3a77c3ce4 Remove note about consumer groups 2021-08-22 09:00:13 -07:00
jonandrewj
a334486111
Implement a WithShardClosedHandler option for the consumer (#135) 2021-07-30 14:16:15 -07:00
jonandrewj
c75a9237b6
Add InternalFailureException to the list of retriable errors (#132) 
* Add InternalFailureException to the list of retriable errors
 2021-02-25 14:54:48 -08:00
Frank Meyer
27055f2ace
Wrap underlying errors via %w verb (#130) 
As introduced in Go 1.13. This enables user of this library
to check for an underlying wrapped error type via errors.Is and
errors.As functions.
 2021-02-05 09:10:34 -08:00
James Regovich
799ccf2d40
Add support for aggregated records (#127) 
Add config option for aggregated records and deaggregation on records in ScanShard

This PR adds an option to consume aggregated records.
 2020-10-13 20:41:18 -07:00
Jason Tackaberry
e60d217333
Include MillisBehindLatest in Record for ScanFunc (#124) 2020-08-01 22:05:17 -07:00
Jason Tackaberry
3f2519e51c
Run initial scan immediately (#123) 
Rather than starting the shard scan loop and waiting for next scan tick before
fetching any data, do the first poll immediately, and then wait.
 2020-08-01 15:45:37 -07:00
Jason Tackaberry
97ffabeaa5
Include ShardID in Record passed to ScanFunc (#121) 
* Include ShardID in Record passed to ScanFunc
* Update mock to explicitly use kinesis.Record

Supports change wherein consumer.Record is changed from an alias of
kinesis.Record to a composition containing it.
 2020-07-30 14:18:38 -07:00
Harlow Ward
bae065cf53 Make sure gofmt run on all files 2020-07-21 20:31:38 -07:00
Harlow Ward
3b95644d77 Move license to file without prefix 2020-07-21 20:31:04 -07:00
Nicolas Augusto Sassi
ef5ce02f91
Update README.md (#119) 
fix typo
 2020-07-21 20:27:21 -07:00
Harlow Ward
89db667ce5
Bump AWS SDK to v1.33.3 (#118) 
* Bump AWS SDK to v1.33.3
* Bump redis version
 2020-07-21 20:27:03 -07:00
chumbert2
400ef07463
Allow to override Limit parameter in GetRecords (#113) 
Add option maxRecords to set the maximum number of records that can
be returned by GetRecords. Default value: 10,000. Use WithMaxRecords
to change the default value.

See

https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/#GetRecordsInput
https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
 2020-04-27 21:12:20 -07:00
Jason Pepper
dcd9d048fb
Don't send StreamName when calling ListShards with NextToken. (#110) 
Co-authored-by: Jason Pepper <jason.pepper@crypsisgroup.com>
 2020-03-19 21:45:53 -07:00
0livd
bc5c012fd3 Add scanInterval option (#105) 
By default a consumer will scan for records every 250ms.
This interval can be configured with WithScanInterval.
 2020-01-17 10:22:10 -08:00
Patrik Karlström
217999854b Only initialize ddb client if none is provided (#106) 2020-01-17 10:02:37 -08:00
Andrew Shannon Brown
f85f25c15e Add an in-memory checkpoint to the API (#103) 
* Add an in-memory checkpoint to the API

* Rename memory to store

* Rename test package to store
 2019-09-08 13:13:04 -07:00
Harlow Ward
b87510458e
Fix example command indentation 2019-09-02 08:13:08 -07:00
Harlow Ward
dfb0faaa87 Add missing code-block highlighting 2019-09-02 08:11:40 -07:00
Harlow Ward
b451fc4cc2 Use stdin for example data reader instead of file path 2019-09-02 08:08:21 -07:00
Harlow Ward
e3ee95b282 Fix typo in context cancellation docs 2019-09-02 07:47:10 -07:00
Harlow Ward
4c2aaf78a2 Add consumer example without checkpointing 2019-09-02 07:44:26 -07:00
Harlow Ward
3ae979bf82 Move example consumers under cmd directory 2019-09-02 07:36:31 -07:00
Harlow Ward
1a141cfbaa
Move notice above the build status 2019-09-02 07:30:33 -07:00
Harlow Ward
3a98baa012 Update godoc for allgroup description 2019-09-02 07:26:44 -07:00
Harlow Ward
ed40b5d9b4 Default to kinesalite when running example consumers 2019-09-02 07:25:07 -07:00
Dimas Yudha P
a252eb38c6 Readme and linter fix (#102) 
* update readme.md adding goreport

* update readme and fix issue found by linter

* update readme.md add Go badges

* update readme.md, fix go badges
 2019-08-31 18:43:26 -07:00
Matias Morán Losada
71bbc397e2 An attempt to work around a gopkg.in/module error in a dependency (#100) 
* An attempt to work around a gopkg.in/module error in a dependency

* replace go-sqlmock module to new source in tests
 2019-08-31 13:19:23 -07:00
Andrew Shannon Brown
14db23eaf3 Support creating an iterator with an initial timestamp (#99) 
* Allow setting initial timestamp

* Fix writing to closed channel

* Allow cancelling of request
 2019-08-14 09:33:35 -07:00
Patrick Moore
81a8ac4221 Allow use of existing Redis client for checkpoint store (#96) 2019-08-05 15:04:27 -07:00
Harlow Ward
35c48ef1c9
Only retry expired shard iterator errors (#95) 
Fixes https://github.com/harlow/kinesis-consumer/issues/92
 2019-07-30 19:48:20 -07:00
Harlow Ward
5da0865ac1 Add WithGroup option 2019-07-28 21:34:54 -07:00
Harlow Ward
a9c97d3b93 Update examples to use Store interface 2019-07-28 21:33:19 -07:00
Harlow Ward
d2cf65fa7a Update Redis library version 
The Redis library was pinned to an older vesion using gopkg.in.
This updates to latest version and pins w/ go mod.

https://github.com/harlow/kinesis-consumer/issues/93
 2019-07-28 21:27:28 -07:00
Harlow Ward
00b5f64fa7 Clean up storage w/ store for consistency 2019-07-28 21:20:29 -07:00
Harlow Ward
c72f561abd
Replace Checkpoint with Store interface (#90) 
As we work towards introducing consumer groups to the repository we need a more generic name for the persistence layer for storing checkpoints and leases for given shards. 

* Rename `checkpoint` to `store`
 2019-07-28 21:18:40 -07:00
Harlow Ward
d05d6c2d5e Update comments for exported functions 2019-07-28 10:54:01 -07:00
Harlow Ward
7018c0c47e
Introduce Group interface and AllGroup (#91) 
* Introduce Group interface and AllGroup

As we move towards consumer groups we'll need to support the current
"consume all shards" strategy, and setup the codebase for the
anticipated "consume balanced shards."
 2019-06-09 13:42:25 -07:00
Harlow Ward
9cd2e57ba4
Add note about development on Consumer Groups 2019-05-28 19:52:32 -07:00
keperry
8493100b85 Switch to go modules (#88) 2019-05-06 16:05:39 -07:00
James Greenhill
b48acfa5d4 Add Mysql support for checkpointing (#87) 2019-04-12 22:15:49 -07:00
Harlow Ward
f7f98a4bc6 Default the consumer library to read from latest 
Having consumers start at latest record seems like a reasonable default.
This can also be overridden with optional config, make sure that is
documented for users of the library.

Fixes: https://github.com/harlow/kinesis-consumer/issues/86
 2019-04-09 22:11:46 -07:00
Harlow Ward
97fe4e66ff
Use shard broker to monitor and process new shards (#85) 
* Use shard broker to start processing new shards

The addition of a shard broker will allow the consumer to be notified
when new shards are added to the stream so it can consume them.

Fixes: https://github.com/harlow/kinesis-consumer/issues/36
 2019-04-09 22:03:12 -07:00
Harlow Ward
c4f363a517 Example data is in repo, no need to download 2019-04-07 16:33:56 -07:00
Harlow Ward
76158d24ab
Introduce ScanFunc signature and remove ScanStatus (#77) 
Major changes:

```go
type ScanFunc func(r *Record) error
```

* Simplify the callback func signature by removing `ScanStatus` 
* Leverage context for cancellation 
* Add custom error `SkipCheckpoint` for special cases when we don't want to checkpoint

Minor changes:

* Use kinesis package constants for shard iterator types
* Move optional config to new file

See conversation on #75 for more details
 2019-04-07 16:29:12 -07:00
Harlow Ward
24de74fd14
Fix the CI and Doc links 2019-02-18 11:10:02 -08:00
Harlow Ward
8fd7675ea4
Add TravisCI setup (#83) 
We've had a few PRs hit master without running the test, this should help make sure we always know the PR status before merging.
 2019-02-18 11:05:01 -08:00
Emanuel Ramos
245d1bd6b5 change cancel place (#82) 2019-02-18 07:59:20 -08:00
lordfarhan40
2037463c62 Fix getShardID does not return more than 100 shards (#81) 2019-02-14 20:45:32 -08:00
Harlow Ward
2f58b136fe Add dummy users data for producing onto stream 
Pulled from: https://github.com/awslabs/amazon-kinesis-connectors
 2018-12-30 07:28:18 -08:00