luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
73 commits 6 branches 9 tags 7.5 MiB
Commit graph

35 commits

Author SHA1 Message Date
Harlow Ward
ceca88b96a Add required fields to Config 
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
 2016-05-07 18:10:31 -07:00
Harlow Ward
ff5ff00ad7 Remove hr line break from footer 2016-05-01 12:46:23 -07:00
Harlow Ward
9a17d4098e Add footer links to social profiles 2016-05-01 12:45:27 -07:00
Harlow Ward
a12c15a191 Use no-op Info logging by default 2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36 Use config object for optional params 
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.

https://peter.bourgon.org/go-best-practices-2016/#configuration
 2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe Add logging when records are emitted w/ record count 2016-05-01 10:43:42 -07:00
Harlow Ward
49b5a94c7e Use Apex log for logging (#27) 
* Use Apex log for logging
 2016-04-30 22:23:35 -07:00
Harlow Ward
e843ae5928 Remove poll interval in favor of aws retry backoff 2016-04-30 18:05:04 -07:00
Harlow Ward
855af23684 Merge pull request #25 from robbles/master 
Add gvt manifest
 2016-02-16 10:50:01 -08:00
Rob O'Dwyer
ce3df40c90 add section about gvt to README.md 2016-02-09 23:39:57 -08:00
Harlow Ward
c29698550f Add config options to Consumer 
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
 2016-02-09 22:31:15 -08:00
Harlow Ward
6119d5c7bf Update language for syntax highlighting 2016-02-08 19:42:26 -08:00
Harlow Ward
066e5844fe Update note about AWS Firehose service 2016-02-08 19:41:36 -08:00
Harlow Ward
aae268108e Move emitters to separate packages 2016-02-08 19:39:09 -08:00
Harlow Ward
f0e6461cb6 Refactor to use handler func 
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.

* Add refactoring note and SHA to README
 2016-02-06 17:50:17 -08:00
Harlow Ward
3d9e6e2485 Add Redshift Pipeline example 2015-05-25 23:18:14 -07:00
Harlow Ward
e17988cfc9 Update Logging instructions 2015-05-25 23:09:59 -07:00
Harlow Ward
a27a13f405 Update README.md 2015-05-23 16:18:10 -07:00
Harlow Ward
9a30e97f10 Add link to exmaples directory 2015-05-23 16:10:08 -07:00
Harlow Ward
25e390e8b4 Add sample pipeline 
* Reduce the size of main README in favor of adding functioning examples
* Add S3 Pipeline exmaple
* Add example of seeing the stream
 2015-05-23 15:27:40 -07:00
Harlow Ward
fdd39482bb Add Installation sub-header to Usage 2015-05-23 13:57:52 -07:00
Harlow Ward
b9a489b940 Update header size. Remove note 2015-05-23 13:56:03 -07:00
Harlow Ward
6ca2abb7f8 User Log package for default logger 
* Modify the default logger to it also logs Line Numbers
* Update README w/ logging instructions
 2015-05-23 13:52:57 -07:00
Harlow Ward
9077d288d2 Rename License file and add Contributing sections 
* Make it clear from project root we are using MIT-LICENSE
* Add contributing section w/ overview of how to submit code to repo
 2015-05-23 10:24:53 -07:00
Harlow Ward
bde3e96ad9 Add Discard Logger 
For situations where we don't want any logs produced by the application.

* Remove references to Logger from README
* Add implementation of DiscardLogger
 2015-05-22 23:38:06 -07:00
Harlow Ward
82d10ab78d Simplify example (no Redshift cluster required) 2015-05-03 21:01:21 -07:00
Harlow Ward
cd71fd41bc Add Logger interface 
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.

* Add Logger interface
* Use logger interface in pipeline and emitters
 2015-05-03 21:01:14 -07:00
Harlow Ward
da7a1dd87a Update README.md 2014-12-20 19:46:38 -08:00
Harlow Ward
1bce60bae6 Update example with Redshift Manifest Pipeline 2014-12-20 19:45:34 -08:00
Harlow Ward
05920cc5d3 Update development notice 2014-12-18 18:26:59 -08:00
Harlow Ward
106f0d66bb Add Redshift Manifest functionality 
Use a manifest file for batch importing files uploaded to S3.
 2014-12-10 21:59:42 -08:00
Harlow Ward
02a7c648a3 Update link for AWS attribution 2014-11-15 18:10:19 -08:00
Harlow Ward
a9a625aa79 Update example usage in README 
* Showcase the new StringToStringTransformer as a simple way of getting
data directly from Kinesis to Redshift.
 2014-11-15 17:36:28 -08:00
Harlow Ward
06b40e6ed8 Base pipeline components 
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
 2014-11-14 20:45:34 -08:00
Harlow Ward
0fe80b708b Initial commit 
* Add installation instructions to README.
* Make note that this repo is under development.
 2014-07-25 00:22:06 -07:00