Harlow Ward
3770136f64
Allow user to override no-op checkpoint with Option
2017-11-22 17:44:42 -08:00
Harlow Ward
c64b40b4ad
Increment counts in correct usage points
2017-11-22 14:21:19 -08:00
Harlow Ward
c91f6233ef
Add counter for exposing scanner metrics
2017-11-22 14:10:11 -08:00
Harlow Ward
84c0820f4a
Serverless options
2017-11-22 10:57:29 -08:00
Harlow Ward
7db78c24f4
Update with alternative options
2017-11-22 10:55:22 -08:00
Harlow Ward
b783b8fb5f
Update formatting on notes section
2017-11-22 10:50:09 -08:00
Harlow Ward
90d2903fe6
Use stdlib logging, default to discard
2017-11-22 10:46:39 -08:00
Harlow Ward
9a35af8df6
Update the checkpoint diagram
2017-11-21 09:04:39 -08:00
Harlow Ward
4d6a85e901
Make the Checkpoint a required input for Consumer
...
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.
* Add Checkpoint as required param for new consumer
2017-11-21 08:58:16 -08:00
Harlow Ward
154595b9a3
Add change log to summarize repo activity ( #38 )
...
Make it visible to users when breaking changes happen in the repo
* Add changelog w/ link to previous stable version of library
2017-11-20 17:27:39 -08:00
Harlow Ward
8d2cc5bc20
Return error from scan instead of terminating the program
2017-11-20 11:45:41 -08:00
Harlow Ward
60ce796c07
Add new diagram for consumer checkpoint storge
2017-11-20 11:14:39 -08:00
Harlow Ward
9620261104
Add checkpoint diagram
2017-11-20 11:06:46 -08:00
Harlow Ward
28837eee9e
Add link to blog post about Kinesis and Lambda
2017-11-20 10:29:30 -08:00
Harlow Ward
836e4bd5c1
Update godeps
2017-11-20 10:13:47 -08:00
Harlow Ward
d6602175e3
Add screenshot for fields needed in DDB checkpoint
2017-11-20 09:55:43 -08:00
Harlow Ward
1038843ed8
Use italics for Kinesis to Firehose note
2017-11-20 09:45:57 -08:00
Harlow Ward
99d82c2c01
Note about Kinesis to Firehose functionality
2017-11-20 09:45:00 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option ( #37 )
...
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence
Implements: https://github.com/harlow/kinesis-consumer/issues/26
2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience ( #35 )
...
Major changes:
* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config
https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
Minor changes:
* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
2017-11-20 08:21:40 -08:00
Harlow Ward
e4efcdb56e
Remove duplicated function for getShardIterator
2017-10-15 17:45:38 -07:00
Pierre Massat
c04f3d8a94
Get a new shard iterator on error ( #32 )
...
* Get a new shard iterator on error
* Check for nil instead of empty string
* Get a new shard iterator on error
2017-10-15 17:40:30 -07:00
Pierre Massat
c56cefb667
Expose the shard ID in the buffer ( #30 )
2016-12-26 08:24:34 -07:00
Harlow Ward
fedb6812fb
Add checkpoint interface for custom checkpoints ( #29 )
...
To allow other checkpoint backends we extract a checkpoint interface
which future checkpoints can implement.
* Add checkpoint interface for custom checkpoints
* Create RedisCheckpoint to implement checkpoint interface
* Swap out hosie redis library for go-redis
Minor changes
* Allow configuration of Redis endpoint with env var `REDIS_URL`
* Replace gvt with govendor
2016-12-04 00:08:06 -08:00
Harlow Ward
cc936aed04
Update stream name label
2016-05-27 18:43:55 -06:00
Harlow Ward
a2e443925d
Update config example
2016-05-07 18:15:55 -07:00
Harlow Ward
ceca88b96a
Add required fields to Config
...
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
2016-05-07 18:10:31 -07:00
Harlow Ward
f4c8d8090d
Use TJ's Kinesis producer library
2016-05-01 16:54:47 -07:00
Harlow Ward
8d71bbd0ec
Update S3 emitter initialization
2016-05-01 15:07:58 -07:00
Harlow Ward
07efe9750c
Bump gvt vendored libraries
2016-05-01 14:56:50 -07:00
Harlow Ward
ff5ff00ad7
Remove hr line break from footer
2016-05-01 12:46:23 -07:00
Harlow Ward
9a17d4098e
Add footer links to social profiles
2016-05-01 12:45:27 -07:00
Harlow Ward
a12c15a191
Use no-op Info logging by default
2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36
Use config object for optional params
...
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.
https://peter.bourgon.org/go-best-practices-2016/#configuration
2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe
Add logging when records are emitted w/ record count
2016-05-01 10:43:42 -07:00
Harlow Ward
c7a69e2c92
Add benchmark test to buffer
2016-05-01 10:42:51 -07:00
Harlow Ward
d5bdd3f4bc
Leverage AWS S3 retries
2016-05-01 10:42:28 -07:00
Harlow Ward
49b5a94c7e
Use Apex log for logging ( #27 )
...
* Use Apex log for logging
2016-04-30 22:23:35 -07:00
Harlow Ward
e843ae5928
Remove poll interval in favor of aws retry backoff
2016-04-30 18:05:04 -07:00
Harlow Ward
dded9d0a0e
Leverage the default AWS retry logic
2016-04-30 17:04:44 -07:00
Harlow Ward
e150d4832b
Add line-breaks after blocks
2016-04-30 16:21:49 -07:00
Harlow Ward
8f1d408c52
Fix recoverable error tests
2016-04-30 16:20:01 -07:00
Harlow Ward
7fccc9da9f
Update function name to represent functionality
2016-04-30 09:14:25 -07:00
Harlow Ward
855af23684
Merge pull request #25 from robbles/master
...
Add gvt manifest
2016-02-16 10:50:01 -08:00
Rob O'Dwyer
ce3df40c90
add section about gvt to README.md
2016-02-09 23:39:57 -08:00
Harlow Ward
c29698550f
Add config options to Consumer
...
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
2016-02-09 22:31:15 -08:00
Rob O'Dwyer
179e9502b0
add gvt manifest
2016-02-09 13:30:54 -08:00
Harlow Ward
6119d5c7bf
Update language for syntax highlighting
2016-02-08 19:42:26 -08:00
Harlow Ward
066e5844fe
Update note about AWS Firehose service
2016-02-08 19:41:36 -08:00
Harlow Ward
aae268108e
Move emitters to separate packages
2016-02-08 19:39:09 -08:00
