luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
88 commits 6 branches 9 tags 7.5 MiB
Commit graph

88 commits

This branch
This branch
All branches
Author SHA1 Message Date
Harlow Ward
60ce796c07
Add new diagram for consumer checkpoint storge 2017-11-20 11:14:39 -08:00
Harlow Ward
9620261104
Add checkpoint diagram 2017-11-20 11:06:46 -08:00
Harlow Ward
28837eee9e
Add link to blog post about Kinesis and Lambda 2017-11-20 10:29:30 -08:00
Harlow Ward
836e4bd5c1 Update godeps 2017-11-20 10:13:47 -08:00
Harlow Ward
d6602175e3
Add screenshot for fields needed in DDB checkpoint 2017-11-20 09:55:43 -08:00
Harlow Ward
1038843ed8
Use italics for Kinesis to Firehose note 2017-11-20 09:45:57 -08:00
Harlow Ward
99d82c2c01
Note about Kinesis to Firehose functionality 2017-11-20 09:45:00 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option (#37) 
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence

Implements: https://github.com/harlow/kinesis-consumer/issues/26
 2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience (#35) 
Major changes:

* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config 

https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

Minor changes:

* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
 2017-11-20 08:21:40 -08:00
Harlow Ward
e4efcdb56e Remove duplicated function for getShardIterator 2017-10-15 17:45:38 -07:00
Pierre Massat
c04f3d8a94 Get a new shard iterator on error (#32) 
* Get a new shard iterator on error
* Check for nil instead of empty string
* Get a new shard iterator on error
 2017-10-15 17:40:30 -07:00
Pierre Massat
c56cefb667 Expose the shard ID in the buffer (#30) 2016-12-26 08:24:34 -07:00
Harlow Ward
fedb6812fb Add checkpoint interface for custom checkpoints (#29) 
To allow other checkpoint backends we extract a checkpoint interface
which future checkpoints can implement.

* Add checkpoint interface for custom checkpoints
* Create RedisCheckpoint to implement checkpoint interface
* Swap out hosie redis library for go-redis

Minor changes

* Allow configuration of Redis endpoint with env var `REDIS_URL`
* Replace gvt with govendor
 2016-12-04 00:08:06 -08:00
Harlow Ward
cc936aed04 Update stream name label 2016-05-27 18:43:55 -06:00
Harlow Ward
a2e443925d Update config example 2016-05-07 18:15:55 -07:00
Harlow Ward
ceca88b96a Add required fields to Config 
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
 2016-05-07 18:10:31 -07:00
Harlow Ward
f4c8d8090d Use TJ's Kinesis producer library 2016-05-01 16:54:47 -07:00
Harlow Ward
8d71bbd0ec Update S3 emitter initialization 2016-05-01 15:07:58 -07:00
Harlow Ward
07efe9750c Bump gvt vendored libraries 2016-05-01 14:56:50 -07:00
Harlow Ward
ff5ff00ad7 Remove hr line break from footer 2016-05-01 12:46:23 -07:00
Harlow Ward
9a17d4098e Add footer links to social profiles 2016-05-01 12:45:27 -07:00
Harlow Ward
a12c15a191 Use no-op Info logging by default 2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36 Use config object for optional params 
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.

https://peter.bourgon.org/go-best-practices-2016/#configuration
 2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe Add logging when records are emitted w/ record count 2016-05-01 10:43:42 -07:00
Harlow Ward
c7a69e2c92 Add benchmark test to buffer 2016-05-01 10:42:51 -07:00
Harlow Ward
d5bdd3f4bc Leverage AWS S3 retries 2016-05-01 10:42:28 -07:00
Harlow Ward
49b5a94c7e Use Apex log for logging (#27) 
* Use Apex log for logging
 2016-04-30 22:23:35 -07:00
Harlow Ward
e843ae5928 Remove poll interval in favor of aws retry backoff 2016-04-30 18:05:04 -07:00
Harlow Ward
dded9d0a0e Leverage the default AWS retry logic 2016-04-30 17:04:44 -07:00
Harlow Ward
e150d4832b Add line-breaks after blocks 2016-04-30 16:21:49 -07:00
Harlow Ward
8f1d408c52 Fix recoverable error tests 2016-04-30 16:20:01 -07:00
Harlow Ward
7fccc9da9f Update function name to represent functionality 2016-04-30 09:14:25 -07:00
Harlow Ward
855af23684 Merge pull request #25 from robbles/master 
Add gvt manifest
 2016-02-16 10:50:01 -08:00
Rob O'Dwyer
ce3df40c90 add section about gvt to README.md 2016-02-09 23:39:57 -08:00
Harlow Ward
c29698550f Add config options to Consumer 
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
 2016-02-09 22:31:15 -08:00
Rob O'Dwyer
179e9502b0 add gvt manifest 2016-02-09 13:30:54 -08:00
Harlow Ward
6119d5c7bf Update language for syntax highlighting 2016-02-08 19:42:26 -08:00
Harlow Ward
066e5844fe Update note about AWS Firehose service 2016-02-08 19:41:36 -08:00
Harlow Ward
aae268108e Move emitters to separate packages 2016-02-08 19:39:09 -08:00
Harlow Ward
caa2fc0b0a Add Kinesis Firehose example 2016-02-08 13:21:54 -08:00
Harlow Ward
f0e6461cb6 Refactor to use handler func 
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.

* Add refactoring note and SHA to README
 2016-02-06 17:50:17 -08:00
Harlow Ward
509f68de89 Add error checking for tmp file 2016-01-09 09:52:47 -08:00
Harlow Ward
d542fa996f Use AWS SDK and concurrency 2016-01-09 09:46:28 -08:00
Harlow Ward
0d5e9b7b02 Use AWS SDK for S3 Emitter 
* Handle retries for S3 Emitter
* Update Pipeline to use AWS backoff
* Leverage aws error kit in recoverable error helpers
 2015-08-16 17:52:10 -07:00
Harlow Ward
18173842fb Use AWS SDK 
Limit the amount of library dependencies by using the official AWS SDK.

https://github.com/harlow/kinesis-connectors/issues/19
 2015-08-15 23:17:49 -07:00
Harlow Ward
8277ffc6be Use GoKit logger interface directly 2015-05-26 18:50:34 -07:00
Harlow Ward
3d9e6e2485 Add Redshift Pipeline example 2015-05-25 23:18:14 -07:00
Harlow Ward
e17988cfc9 Update Logging instructions 2015-05-25 23:09:59 -07:00
Harlow Ward
cd54569c61 Use go-kit Log interface 2015-05-25 22:40:40 -07:00
dan
9371fb938c Logging and DB connection reuse 
* Reuse Redshift DB connection
* Add more logging
 2015-05-25 22:23:40 -07:00