Harlow Ward
3770136f64
Allow user to override no-op checkpoint with Option
2017-11-22 17:44:42 -08:00
Harlow Ward
c91f6233ef
Add counter for exposing scanner metrics
2017-11-22 14:10:11 -08:00
Harlow Ward
90d2903fe6
Use stdlib logging, default to discard
2017-11-22 10:46:39 -08:00
Harlow Ward
4d6a85e901
Make the Checkpoint a required input for Consumer
...
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.
* Add Checkpoint as required param for new consumer
2017-11-21 08:58:16 -08:00
Harlow Ward
8d2cc5bc20
Return error from scan instead of terminating the program
2017-11-20 11:45:41 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option ( #37 )
...
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence
Implements: https://github.com/harlow/kinesis-consumer/issues/26
2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience ( #35 )
...
Major changes:
* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config
https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
Minor changes:
* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
2017-11-20 08:21:40 -08:00
Harlow Ward
ceca88b96a
Add required fields to Config
...
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
2016-05-07 18:10:31 -07:00
Harlow Ward
f4c8d8090d
Use TJ's Kinesis producer library
2016-05-01 16:54:47 -07:00
Harlow Ward
a12c15a191
Use no-op Info logging by default
2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36
Use config object for optional params
...
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.
https://peter.bourgon.org/go-best-practices-2016/#configuration
2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe
Add logging when records are emitted w/ record count
2016-05-01 10:43:42 -07:00
Harlow Ward
d5bdd3f4bc
Leverage AWS S3 retries
2016-05-01 10:42:28 -07:00
Harlow Ward
7fccc9da9f
Update function name to represent functionality
2016-04-30 09:14:25 -07:00
Harlow Ward
c29698550f
Add config options to Consumer
...
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
2016-02-09 22:31:15 -08:00
Harlow Ward
aae268108e
Move emitters to separate packages
2016-02-08 19:39:09 -08:00
Harlow Ward
caa2fc0b0a
Add Kinesis Firehose example
2016-02-08 13:21:54 -08:00
Harlow Ward
f0e6461cb6
Refactor to use handler func
...
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.
* Add refactoring note and SHA to README
2016-02-06 17:50:17 -08:00
Harlow Ward
509f68de89
Add error checking for tmp file
2016-01-09 09:52:47 -08:00
Harlow Ward
d542fa996f
Use AWS SDK and concurrency
2016-01-09 09:46:28 -08:00
Harlow Ward
0d5e9b7b02
Use AWS SDK for S3 Emitter
...
* Handle retries for S3 Emitter
* Update Pipeline to use AWS backoff
* Leverage aws error kit in recoverable error helpers
2015-08-16 17:52:10 -07:00
Harlow Ward
18173842fb
Use AWS SDK
...
Limit the amount of library dependencies by using the official AWS SDK.
https://github.com/harlow/kinesis-connectors/issues/19
2015-08-15 23:17:49 -07:00
Harlow Ward
3d9e6e2485
Add Redshift Pipeline example
2015-05-25 23:18:14 -07:00
Harlow Ward
cd54569c61
Use go-kit Log interface
2015-05-25 22:40:40 -07:00
Harlow Ward
25e390e8b4
Add sample pipeline
...
* Reduce the size of main README in favor of adding functioning examples
* Add S3 Pipeline exmaple
* Add example of seeing the stream
2015-05-23 15:27:40 -07:00
