Harlow Ward
64cdf69249
Add interval flush for DDB checkpoint ( #40 )
...
Add interval flush for DDB checkpoint
* Allow checkpointing on a specified interval
* Add shutdown method to checkpoint to force flush
Minor changes:
* Swap order of input params for checkpoint (app, table)
Addresses: https://github.com/harlow/kinesis-consumer/issues/39
2017-12-30 20:21:10 -08:00
Harlow Ward
955f74d553
Have new func return type
2017-11-26 18:22:09 -08:00
Harlow Ward
e5e057d6aa
Add additional context to new comment
2017-11-26 18:17:41 -08:00
Harlow Ward
a62e7514e4
Make the Kinesis client exportable
2017-11-26 18:16:32 -08:00
Harlow Ward
b875bb56e7
Introduce Client Interface
...
Testing the components of the consumer where proving difficult because
the consumer code was so tightly coupled with the Kinesis client
library.
* Extract the concept of Client interface
* Create default client w/ kinesis connection
* Test with fake client to avoid round trip to kinesis
2017-11-26 16:00:11 -08:00
Harlow Ward
058f383e30
Add cancellation of pipeline from signal interrupts
2017-11-26 16:00:03 -08:00
Harlow Ward
89570130f5
Leverage bigger batchsize when seeding example data
2017-11-26 15:59:17 -08:00
Harlow Ward
edf0467eb0
Format errors from caller
2017-11-23 11:29:58 -08:00
Harlow Ward
86f1df782e
Return the shard scan errors to top-level caller
2017-11-23 08:49:37 -08:00
Harlow Ward
b0245d688b
Add more test coverage for Redis Checkpoint
2017-11-22 21:28:39 -08:00
Harlow Ward
6401371727
Simplify checkpoint interface; reduce input vars
2017-11-22 20:01:31 -08:00
Harlow Ward
3f081bd05a
Fix position of input params for Options
2017-11-22 17:54:47 -08:00
Harlow Ward
4ffe3ec55a
Add logs for start scan and checkpoints
2017-11-22 17:52:41 -08:00
Harlow Ward
3770136f64
Allow user to override no-op checkpoint with Option
2017-11-22 17:44:42 -08:00
Harlow Ward
c64b40b4ad
Increment counts in correct usage points
2017-11-22 14:21:19 -08:00
Harlow Ward
c91f6233ef
Add counter for exposing scanner metrics
2017-11-22 14:10:11 -08:00
Harlow Ward
84c0820f4a
Serverless options
2017-11-22 10:57:29 -08:00
Harlow Ward
7db78c24f4
Update with alternative options
2017-11-22 10:55:22 -08:00
Harlow Ward
b783b8fb5f
Update formatting on notes section
2017-11-22 10:50:09 -08:00
Harlow Ward
90d2903fe6
Use stdlib logging, default to discard
2017-11-22 10:46:39 -08:00
Harlow Ward
9a35af8df6
Update the checkpoint diagram
2017-11-21 09:04:39 -08:00
Harlow Ward
4d6a85e901
Make the Checkpoint a required input for Consumer
...
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.
* Add Checkpoint as required param for new consumer
2017-11-21 08:58:16 -08:00
Harlow Ward
154595b9a3
Add change log to summarize repo activity ( #38 )
...
Make it visible to users when breaking changes happen in the repo
* Add changelog w/ link to previous stable version of library
2017-11-20 17:27:39 -08:00
Harlow Ward
8d2cc5bc20
Return error from scan instead of terminating the program
2017-11-20 11:45:41 -08:00
Harlow Ward
60ce796c07
Add new diagram for consumer checkpoint storge
2017-11-20 11:14:39 -08:00
Harlow Ward
9620261104
Add checkpoint diagram
2017-11-20 11:06:46 -08:00
Harlow Ward
28837eee9e
Add link to blog post about Kinesis and Lambda
2017-11-20 10:29:30 -08:00
Harlow Ward
836e4bd5c1
Update godeps
2017-11-20 10:13:47 -08:00
Harlow Ward
d6602175e3
Add screenshot for fields needed in DDB checkpoint
2017-11-20 09:55:43 -08:00
Harlow Ward
1038843ed8
Use italics for Kinesis to Firehose note
2017-11-20 09:45:57 -08:00
Harlow Ward
99d82c2c01
Note about Kinesis to Firehose functionality
2017-11-20 09:45:00 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option ( #37 )
...
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence
Implements: https://github.com/harlow/kinesis-consumer/issues/26
2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience ( #35 )
...
Major changes:
* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config
https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
Minor changes:
* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
2017-11-20 08:21:40 -08:00
Harlow Ward
e4efcdb56e
Remove duplicated function for getShardIterator
2017-10-15 17:45:38 -07:00
Pierre Massat
c04f3d8a94
Get a new shard iterator on error ( #32 )
...
* Get a new shard iterator on error
* Check for nil instead of empty string
* Get a new shard iterator on error
2017-10-15 17:40:30 -07:00
Pierre Massat
c56cefb667
Expose the shard ID in the buffer ( #30 )
2016-12-26 08:24:34 -07:00
Harlow Ward
fedb6812fb
Add checkpoint interface for custom checkpoints ( #29 )
...
To allow other checkpoint backends we extract a checkpoint interface
which future checkpoints can implement.
* Add checkpoint interface for custom checkpoints
* Create RedisCheckpoint to implement checkpoint interface
* Swap out hosie redis library for go-redis
Minor changes
* Allow configuration of Redis endpoint with env var `REDIS_URL`
* Replace gvt with govendor
2016-12-04 00:08:06 -08:00
Harlow Ward
cc936aed04
Update stream name label
2016-05-27 18:43:55 -06:00
Harlow Ward
a2e443925d
Update config example
2016-05-07 18:15:55 -07:00
Harlow Ward
ceca88b96a
Add required fields to Config
...
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
2016-05-07 18:10:31 -07:00
Harlow Ward
f4c8d8090d
Use TJ's Kinesis producer library
2016-05-01 16:54:47 -07:00
Harlow Ward
8d71bbd0ec
Update S3 emitter initialization
2016-05-01 15:07:58 -07:00
Harlow Ward
07efe9750c
Bump gvt vendored libraries
2016-05-01 14:56:50 -07:00
Harlow Ward
ff5ff00ad7
Remove hr line break from footer
2016-05-01 12:46:23 -07:00
Harlow Ward
9a17d4098e
Add footer links to social profiles
2016-05-01 12:45:27 -07:00
Harlow Ward
a12c15a191
Use no-op Info logging by default
2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36
Use config object for optional params
...
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.
https://peter.bourgon.org/go-best-practices-2016/#configuration
2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe
Add logging when records are emitted w/ record count
2016-05-01 10:43:42 -07:00
Harlow Ward
c7a69e2c92
Add benchmark test to buffer
2016-05-01 10:42:51 -07:00
Harlow Ward
d5bdd3f4bc
Leverage AWS S3 retries
2016-05-01 10:42:28 -07:00
