luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
111 commits 6 branches 9 tags 7.5 MiB
Commit graph

56 commits

Author SHA1 Message Date
Harlow Ward
6401371727 Simplify checkpoint interface; reduce input vars 2017-11-22 20:01:31 -08:00
Harlow Ward
3f081bd05a Fix position of input params for Options 2017-11-22 17:54:47 -08:00
Harlow Ward
3770136f64 Allow user to override no-op checkpoint with Option 2017-11-22 17:44:42 -08:00
Harlow Ward
c91f6233ef Add counter for exposing scanner metrics 2017-11-22 14:10:11 -08:00
Harlow Ward
84c0820f4a
Serverless options 2017-11-22 10:57:29 -08:00
Harlow Ward
7db78c24f4
Update with alternative options 2017-11-22 10:55:22 -08:00
Harlow Ward
b783b8fb5f
Update formatting on notes section 2017-11-22 10:50:09 -08:00
Harlow Ward
90d2903fe6 Use stdlib logging, default to discard 2017-11-22 10:46:39 -08:00
Harlow Ward
9a35af8df6 Update the checkpoint diagram 2017-11-21 09:04:39 -08:00
Harlow Ward
4d6a85e901 Make the Checkpoint a required input for Consumer 
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.

* Add Checkpoint as required param for new consumer
 2017-11-21 08:58:16 -08:00
Harlow Ward
8d2cc5bc20 Return error from scan instead of terminating the program 2017-11-20 11:45:41 -08:00
Harlow Ward
60ce796c07
Add new diagram for consumer checkpoint storge 2017-11-20 11:14:39 -08:00
Harlow Ward
9620261104
Add checkpoint diagram 2017-11-20 11:06:46 -08:00
Harlow Ward
28837eee9e
Add link to blog post about Kinesis and Lambda 2017-11-20 10:29:30 -08:00
Harlow Ward
d6602175e3
Add screenshot for fields needed in DDB checkpoint 2017-11-20 09:55:43 -08:00
Harlow Ward
1038843ed8
Use italics for Kinesis to Firehose note 2017-11-20 09:45:57 -08:00
Harlow Ward
99d82c2c01
Note about Kinesis to Firehose functionality 2017-11-20 09:45:00 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option (#37) 
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence

Implements: https://github.com/harlow/kinesis-consumer/issues/26
 2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience (#35) 
Major changes:

* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config 

https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

Minor changes:

* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
 2017-11-20 08:21:40 -08:00
Harlow Ward
fedb6812fb Add checkpoint interface for custom checkpoints (#29) 
To allow other checkpoint backends we extract a checkpoint interface
which future checkpoints can implement.

* Add checkpoint interface for custom checkpoints
* Create RedisCheckpoint to implement checkpoint interface
* Swap out hosie redis library for go-redis

Minor changes

* Allow configuration of Redis endpoint with env var `REDIS_URL`
* Replace gvt with govendor
 2016-12-04 00:08:06 -08:00
Harlow Ward
a2e443925d Update config example 2016-05-07 18:15:55 -07:00
Harlow Ward
ceca88b96a Add required fields to Config 
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
 2016-05-07 18:10:31 -07:00
Harlow Ward
ff5ff00ad7 Remove hr line break from footer 2016-05-01 12:46:23 -07:00
Harlow Ward
9a17d4098e Add footer links to social profiles 2016-05-01 12:45:27 -07:00
Harlow Ward
a12c15a191 Use no-op Info logging by default 2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36 Use config object for optional params 
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.

https://peter.bourgon.org/go-best-practices-2016/#configuration
 2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe Add logging when records are emitted w/ record count 2016-05-01 10:43:42 -07:00
Harlow Ward
49b5a94c7e Use Apex log for logging (#27) 
* Use Apex log for logging
 2016-04-30 22:23:35 -07:00
Harlow Ward
e843ae5928 Remove poll interval in favor of aws retry backoff 2016-04-30 18:05:04 -07:00
Harlow Ward
855af23684 Merge pull request #25 from robbles/master 
Add gvt manifest
 2016-02-16 10:50:01 -08:00
Rob O'Dwyer
ce3df40c90 add section about gvt to README.md 2016-02-09 23:39:57 -08:00
Harlow Ward
c29698550f Add config options to Consumer 
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
 2016-02-09 22:31:15 -08:00
Harlow Ward
6119d5c7bf Update language for syntax highlighting 2016-02-08 19:42:26 -08:00
Harlow Ward
066e5844fe Update note about AWS Firehose service 2016-02-08 19:41:36 -08:00
Harlow Ward
aae268108e Move emitters to separate packages 2016-02-08 19:39:09 -08:00
Harlow Ward
f0e6461cb6 Refactor to use handler func 
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.

* Add refactoring note and SHA to README
 2016-02-06 17:50:17 -08:00
Harlow Ward
3d9e6e2485 Add Redshift Pipeline example 2015-05-25 23:18:14 -07:00
Harlow Ward
e17988cfc9 Update Logging instructions 2015-05-25 23:09:59 -07:00
Harlow Ward
a27a13f405 Update README.md 2015-05-23 16:18:10 -07:00
Harlow Ward
9a30e97f10 Add link to exmaples directory 2015-05-23 16:10:08 -07:00
Harlow Ward
25e390e8b4 Add sample pipeline 
* Reduce the size of main README in favor of adding functioning examples
* Add S3 Pipeline exmaple
* Add example of seeing the stream
 2015-05-23 15:27:40 -07:00
Harlow Ward
fdd39482bb Add Installation sub-header to Usage 2015-05-23 13:57:52 -07:00
Harlow Ward
b9a489b940 Update header size. Remove note 2015-05-23 13:56:03 -07:00
Harlow Ward
6ca2abb7f8 User Log package for default logger 
* Modify the default logger to it also logs Line Numbers
* Update README w/ logging instructions
 2015-05-23 13:52:57 -07:00
Harlow Ward
9077d288d2 Rename License file and add Contributing sections 
* Make it clear from project root we are using MIT-LICENSE
* Add contributing section w/ overview of how to submit code to repo
 2015-05-23 10:24:53 -07:00
Harlow Ward
bde3e96ad9 Add Discard Logger 
For situations where we don't want any logs produced by the application.

* Remove references to Logger from README
* Add implementation of DiscardLogger
 2015-05-22 23:38:06 -07:00
Harlow Ward
82d10ab78d Simplify example (no Redshift cluster required) 2015-05-03 21:01:21 -07:00
Harlow Ward
cd71fd41bc Add Logger interface 
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.

* Add Logger interface
* Use logger interface in pipeline and emitters
 2015-05-03 21:01:14 -07:00
Harlow Ward
da7a1dd87a Update README.md 2014-12-20 19:46:38 -08:00
Harlow Ward
1bce60bae6 Update example with Redshift Manifest Pipeline 2014-12-20 19:45:34 -08:00