luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
178 commits 6 branches 9 tags 7.5 MiB
Commit graph

178 commits

This branch
This branch
All branches
Author SHA1 Message Date
Nicolas Augusto Sassi
ef5ce02f91
Update README.md (#119) 
fix typo
 2020-07-21 20:27:21 -07:00
Harlow Ward
89db667ce5
Bump AWS SDK to v1.33.3 (#118) 
* Bump AWS SDK to v1.33.3
* Bump redis version
 2020-07-21 20:27:03 -07:00
chumbert2
400ef07463
Allow to override Limit parameter in GetRecords (#113) 
Add option maxRecords to set the maximum number of records that can
be returned by GetRecords. Default value: 10,000. Use WithMaxRecords
to change the default value.

See

https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/#GetRecordsInput
https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
 2020-04-27 21:12:20 -07:00
Jason Pepper
dcd9d048fb
Don't send StreamName when calling ListShards with NextToken. (#110) 
Co-authored-by: Jason Pepper <jason.pepper@crypsisgroup.com>
 2020-03-19 21:45:53 -07:00
0livd
bc5c012fd3 Add scanInterval option (#105) 
By default a consumer will scan for records every 250ms.
This interval can be configured with WithScanInterval.
 2020-01-17 10:22:10 -08:00
Patrik Karlström
217999854b Only initialize ddb client if none is provided (#106) 2020-01-17 10:02:37 -08:00
Andrew Shannon Brown
f85f25c15e Add an in-memory checkpoint to the API (#103) 
* Add an in-memory checkpoint to the API

* Rename memory to store

* Rename test package to store
 2019-09-08 13:13:04 -07:00
Harlow Ward
b87510458e
Fix example command indentation 2019-09-02 08:13:08 -07:00
Harlow Ward
dfb0faaa87 Add missing code-block highlighting 2019-09-02 08:11:40 -07:00
Harlow Ward
b451fc4cc2 Use stdin for example data reader instead of file path 2019-09-02 08:08:21 -07:00
Harlow Ward
e3ee95b282 Fix typo in context cancellation docs 2019-09-02 07:47:10 -07:00
Harlow Ward
4c2aaf78a2 Add consumer example without checkpointing 2019-09-02 07:44:26 -07:00
Harlow Ward
3ae979bf82 Move example consumers under cmd directory 2019-09-02 07:36:31 -07:00
Harlow Ward
1a141cfbaa
Move notice above the build status 2019-09-02 07:30:33 -07:00
Harlow Ward
3a98baa012 Update godoc for allgroup description 2019-09-02 07:26:44 -07:00
Harlow Ward
ed40b5d9b4 Default to kinesalite when running example consumers 2019-09-02 07:25:07 -07:00
Dimas Yudha P
a252eb38c6 Readme and linter fix (#102) 
* update readme.md adding goreport

* update readme and fix issue found by linter

* update readme.md add Go badges

* update readme.md, fix go badges
 2019-08-31 18:43:26 -07:00
Matias Morán Losada
71bbc397e2 An attempt to work around a gopkg.in/module error in a dependency (#100) 
* An attempt to work around a gopkg.in/module error in a dependency

* replace go-sqlmock module to new source in tests
 2019-08-31 13:19:23 -07:00
Andrew Shannon Brown
14db23eaf3 Support creating an iterator with an initial timestamp (#99) 
* Allow setting initial timestamp

* Fix writing to closed channel

* Allow cancelling of request
 2019-08-14 09:33:35 -07:00
Patrick Moore
81a8ac4221 Allow use of existing Redis client for checkpoint store (#96) 2019-08-05 15:04:27 -07:00
Harlow Ward
35c48ef1c9
Only retry expired shard iterator errors (#95) 
Fixes https://github.com/harlow/kinesis-consumer/issues/92
 2019-07-30 19:48:20 -07:00
Harlow Ward
5da0865ac1 Add WithGroup option 2019-07-28 21:34:54 -07:00
Harlow Ward
a9c97d3b93 Update examples to use Store interface 2019-07-28 21:33:19 -07:00
Harlow Ward
d2cf65fa7a Update Redis library version 
The Redis library was pinned to an older vesion using gopkg.in.
This updates to latest version and pins w/ go mod.

https://github.com/harlow/kinesis-consumer/issues/93
 2019-07-28 21:27:28 -07:00
Harlow Ward
00b5f64fa7 Clean up storage w/ store for consistency 2019-07-28 21:20:29 -07:00
Harlow Ward
c72f561abd
Replace Checkpoint with Store interface (#90) 
As we work towards introducing consumer groups to the repository we need a more generic name for the persistence layer for storing checkpoints and leases for given shards. 

* Rename `checkpoint` to `store`
 2019-07-28 21:18:40 -07:00
Harlow Ward
d05d6c2d5e Update comments for exported functions 2019-07-28 10:54:01 -07:00
Harlow Ward
7018c0c47e
Introduce Group interface and AllGroup (#91) 
* Introduce Group interface and AllGroup

As we move towards consumer groups we'll need to support the current
"consume all shards" strategy, and setup the codebase for the
anticipated "consume balanced shards."
 2019-06-09 13:42:25 -07:00
Harlow Ward
9cd2e57ba4
Add note about development on Consumer Groups 2019-05-28 19:52:32 -07:00
keperry
8493100b85 Switch to go modules (#88) 2019-05-06 16:05:39 -07:00
James Greenhill
b48acfa5d4 Add Mysql support for checkpointing (#87) 2019-04-12 22:15:49 -07:00
Harlow Ward
f7f98a4bc6 Default the consumer library to read from latest 
Having consumers start at latest record seems like a reasonable default.
This can also be overridden with optional config, make sure that is
documented for users of the library.

Fixes: https://github.com/harlow/kinesis-consumer/issues/86
 2019-04-09 22:11:46 -07:00
Harlow Ward
97fe4e66ff
Use shard broker to monitor and process new shards (#85) 
* Use shard broker to start processing new shards

The addition of a shard broker will allow the consumer to be notified
when new shards are added to the stream so it can consume them.

Fixes: https://github.com/harlow/kinesis-consumer/issues/36
 2019-04-09 22:03:12 -07:00
Harlow Ward
c4f363a517 Example data is in repo, no need to download 2019-04-07 16:33:56 -07:00
Harlow Ward
76158d24ab
Introduce ScanFunc signature and remove ScanStatus (#77) 
Major changes:

```go
type ScanFunc func(r *Record) error
```

* Simplify the callback func signature by removing `ScanStatus` 
* Leverage context for cancellation 
* Add custom error `SkipCheckpoint` for special cases when we don't want to checkpoint

Minor changes:

* Use kinesis package constants for shard iterator types
* Move optional config to new file

See conversation on #75 for more details
 2019-04-07 16:29:12 -07:00
Harlow Ward
24de74fd14
Fix the CI and Doc links 2019-02-18 11:10:02 -08:00
Harlow Ward
8fd7675ea4
Add TravisCI setup (#83) 
We've had a few PRs hit master without running the test, this should help make sure we always know the PR status before merging.
 2019-02-18 11:05:01 -08:00
Emanuel Ramos
245d1bd6b5 change cancel place (#82) 2019-02-18 07:59:20 -08:00
lordfarhan40
2037463c62 Fix getShardID does not return more than 100 shards (#81) 2019-02-14 20:45:32 -08:00
Harlow Ward
2f58b136fe Add dummy users data for producing onto stream 
Pulled from: https://github.com/awslabs/amazon-kinesis-connectors
 2018-12-30 07:28:18 -08:00
Harlow Ward
4f374e4425 Update example to use new AWS Session 2018-12-29 10:45:26 -08:00
Harlow Ward
94f0b2ae1e Fix import cycle error for postgres tests 2018-12-29 10:37:24 -08:00
Harlow Ward
3527b603d3 Add shard iterator type as config option 
Fixes: https://github.com/harlow/kinesis-consumer/issues/74
 2018-12-28 19:39:47 -08:00
Harlow Ward
5688ff2820 Specify stop scan when returning scan status 2018-12-28 19:34:16 -08:00
Prometheus
c061203d5b add basic test for dynamodb (#73) 2018-11-07 18:53:00 -08:00
Vincent
cb35697903 Write unit tests for Postgres package (#69) 
Changes:
* I add postgres_databaseutils_test.go for containing all utilities for mocking the database.
* I put appName as a required parameter passed to the New() since it is needed to form the namespace column name.
* I add GetMaxInterval() since it might be needed by the consumer and the test.
* I move the SQL string package variables directly into the function that need it so we could avoid maintenance nightmare in the future.
 2018-10-14 11:23:37 -06:00
Vincent
d3b76346f5 Break down the big function and Add tests for Scan (#65) 2018-09-03 09:59:39 -07:00
Harlow Ward
23811ec99a Add test for stopping scan 2018-07-29 10:27:01 -07:00
Harlow Ward
d6ded158bf Refactor the consumer tests 
A previous PR from @vincent6767 had nicer mock Kinesis client that
simplified setting up data for the tests.

Mock client pulled from:
https://github.com/harlow/kinesis-consumer/pull/64
 2018-07-29 10:13:03 -07:00
Harlow Ward
911282363e Update go dep and prune unused packages 2018-07-28 22:58:40 -07:00