luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
121 commits 6 branches 9 tags 7.5 MiB
Commit graph

121 commits

This branch
This branch
All branches
Author SHA1 Message Date
Edward Tsang
cba40aeeb8 Merge remote-tracking branch 'origin/master' 2018-06-05 11:49:51 -07:00
Prometheus
d058203b6e Make what aws error to trigger retry decided by caller (#52) 
* remove ValidateCheckpoint
* make retrying on error decided by caller
 2018-06-04 20:07:58 -07:00
Prometheus
9a7e102a05 remove ValidateCheckpoint (#51) 2018-06-01 19:34:46 -07:00
Edward Tsang
a800065101 remove ValidateCheckpoint 2018-06-01 18:25:08 -07:00
Prometheus
992cc42419 DDB uses default AWS config settings to ping table; won't work with WithDyanmoClient. Misc update on example and README (#50) 2018-06-01 16:14:42 -07:00
Prometheus
9e0a97916d Use AWS resource iface, overwrite default dynamodb, more explicit in example about overwrite default AWS resrouce client (#49) 
* use custom kinesis client

* use aws sdk interface, add missing api for ddb

* add overwrite default dynamodbclien usage
 2018-05-31 17:41:14 -07:00
Anant Prakash
2a5856ec99 Correct AWS_REGION (#48) 2018-05-30 06:45:38 -07:00
Anant Prakash
2fb47e63bc Fix typo in README.md (#47) 2018-05-24 08:34:10 -07:00
Tim Bart
4bdbbefa34 Use functional options to configure KinesisClient (#46) 2018-04-27 22:12:11 -07:00
Harlow Ward
64cdf69249
Add interval flush for DDB checkpoint (#40) 
Add interval flush for DDB checkpoint

* Allow checkpointing on a specified interval
* Add shutdown method to checkpoint to force flush

Minor changes:

* Swap order of input params for checkpoint (app, table)

Addresses: https://github.com/harlow/kinesis-consumer/issues/39
 2017-12-30 20:21:10 -08:00
Harlow Ward
955f74d553 Have new func return type 2017-11-26 18:22:09 -08:00
Harlow Ward
e5e057d6aa Add additional context to new comment 2017-11-26 18:17:41 -08:00
Harlow Ward
a62e7514e4 Make the Kinesis client exportable 2017-11-26 18:16:32 -08:00
Harlow Ward
b875bb56e7 Introduce Client Interface 
Testing the components of the consumer where proving difficult because
the consumer code was so tightly coupled with the Kinesis client
library.

* Extract the concept of Client interface
* Create default client w/ kinesis connection
* Test with fake client to avoid round trip to kinesis
 2017-11-26 16:00:11 -08:00
Harlow Ward
058f383e30 Add cancellation of pipeline from signal interrupts 2017-11-26 16:00:03 -08:00
Harlow Ward
89570130f5 Leverage bigger batchsize when seeding example data 2017-11-26 15:59:17 -08:00
Harlow Ward
edf0467eb0 Format errors from caller 2017-11-23 11:29:58 -08:00
Harlow Ward
86f1df782e Return the shard scan errors to top-level caller 2017-11-23 08:49:37 -08:00
Harlow Ward
b0245d688b Add more test coverage for Redis Checkpoint 2017-11-22 21:28:39 -08:00
Harlow Ward
6401371727 Simplify checkpoint interface; reduce input vars 2017-11-22 20:01:31 -08:00
Harlow Ward
3f081bd05a Fix position of input params for Options 2017-11-22 17:54:47 -08:00
Harlow Ward
4ffe3ec55a Add logs for start scan and checkpoints 2017-11-22 17:52:41 -08:00
Harlow Ward
3770136f64 Allow user to override no-op checkpoint with Option 2017-11-22 17:44:42 -08:00
Harlow Ward
c64b40b4ad Increment counts in correct usage points 2017-11-22 14:21:19 -08:00
Harlow Ward
c91f6233ef Add counter for exposing scanner metrics 2017-11-22 14:10:11 -08:00
Harlow Ward
84c0820f4a
Serverless options 2017-11-22 10:57:29 -08:00
Harlow Ward
7db78c24f4
Update with alternative options 2017-11-22 10:55:22 -08:00
Harlow Ward
b783b8fb5f
Update formatting on notes section 2017-11-22 10:50:09 -08:00
Harlow Ward
90d2903fe6 Use stdlib logging, default to discard 2017-11-22 10:46:39 -08:00
Harlow Ward
9a35af8df6 Update the checkpoint diagram 2017-11-21 09:04:39 -08:00
Harlow Ward
4d6a85e901 Make the Checkpoint a required input for Consumer 
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.

* Add Checkpoint as required param for new consumer
 2017-11-21 08:58:16 -08:00
Harlow Ward
154595b9a3
Add change log to summarize repo activity (#38) 
Make it visible to users when breaking changes happen in the repo

* Add changelog w/ link to previous stable version of library
 2017-11-20 17:27:39 -08:00
Harlow Ward
8d2cc5bc20 Return error from scan instead of terminating the program 2017-11-20 11:45:41 -08:00
Harlow Ward
60ce796c07
Add new diagram for consumer checkpoint storge 2017-11-20 11:14:39 -08:00
Harlow Ward
9620261104
Add checkpoint diagram 2017-11-20 11:06:46 -08:00
Harlow Ward
28837eee9e
Add link to blog post about Kinesis and Lambda 2017-11-20 10:29:30 -08:00
Harlow Ward
836e4bd5c1 Update godeps 2017-11-20 10:13:47 -08:00
Harlow Ward
d6602175e3
Add screenshot for fields needed in DDB checkpoint 2017-11-20 09:55:43 -08:00
Harlow Ward
1038843ed8
Use italics for Kinesis to Firehose note 2017-11-20 09:45:57 -08:00
Harlow Ward
99d82c2c01
Note about Kinesis to Firehose functionality 2017-11-20 09:45:00 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option (#37) 
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence

Implements: https://github.com/harlow/kinesis-consumer/issues/26
 2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience (#35) 
Major changes:

* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config 

https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

Minor changes:

* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
 2017-11-20 08:21:40 -08:00
Harlow Ward
e4efcdb56e Remove duplicated function for getShardIterator 2017-10-15 17:45:38 -07:00
Pierre Massat
c04f3d8a94 Get a new shard iterator on error (#32) 
* Get a new shard iterator on error
* Check for nil instead of empty string
* Get a new shard iterator on error
 2017-10-15 17:40:30 -07:00
Pierre Massat
c56cefb667 Expose the shard ID in the buffer (#30) 2016-12-26 08:24:34 -07:00
Harlow Ward
fedb6812fb Add checkpoint interface for custom checkpoints (#29) 
To allow other checkpoint backends we extract a checkpoint interface
which future checkpoints can implement.

* Add checkpoint interface for custom checkpoints
* Create RedisCheckpoint to implement checkpoint interface
* Swap out hosie redis library for go-redis

Minor changes

* Allow configuration of Redis endpoint with env var `REDIS_URL`
* Replace gvt with govendor
 2016-12-04 00:08:06 -08:00
Harlow Ward
cc936aed04 Update stream name label 2016-05-27 18:43:55 -06:00
Harlow Ward
a2e443925d Update config example 2016-05-07 18:15:55 -07:00
Harlow Ward
ceca88b96a Add required fields to Config 
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
 2016-05-07 18:10:31 -07:00
Harlow Ward
f4c8d8090d Use TJ's Kinesis producer library 2016-05-01 16:54:47 -07:00