luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
145 commits 6 branches 9 tags 7.5 MiB
Commit graph

145 commits

This branch
This branch
All branches
Author SHA1 Message Date
Harlow Ward
0d5e9b7b02 Use AWS SDK for S3 Emitter 
* Handle retries for S3 Emitter
* Update Pipeline to use AWS backoff
* Leverage aws error kit in recoverable error helpers
 2015-08-16 17:52:10 -07:00
Harlow Ward
18173842fb Use AWS SDK 
Limit the amount of library dependencies by using the official AWS SDK.

https://github.com/harlow/kinesis-connectors/issues/19
 2015-08-15 23:17:49 -07:00
Harlow Ward
8277ffc6be Use GoKit logger interface directly 2015-05-26 18:50:34 -07:00
Harlow Ward
3d9e6e2485 Add Redshift Pipeline example 2015-05-25 23:18:14 -07:00
Harlow Ward
e17988cfc9 Update Logging instructions 2015-05-25 23:09:59 -07:00
Harlow Ward
cd54569c61 Use go-kit Log interface 2015-05-25 22:40:40 -07:00
dan
9371fb938c Logging and DB connection reuse 
* Reuse Redshift DB connection
* Add more logging
 2015-05-25 22:23:40 -07:00
Harlow Ward
a27a13f405 Update README.md 2015-05-23 16:18:10 -07:00
Harlow Ward
9a30e97f10 Add link to exmaples directory 2015-05-23 16:10:08 -07:00
Harlow Ward
25e390e8b4 Add sample pipeline 
* Reduce the size of main README in favor of adding functioning examples
* Add S3 Pipeline exmaple
* Add example of seeing the stream
 2015-05-23 15:27:40 -07:00
Harlow Ward
fdd39482bb Add Installation sub-header to Usage 2015-05-23 13:57:52 -07:00
Harlow Ward
b9a489b940 Update header size. Remove note 2015-05-23 13:56:03 -07:00
Harlow Ward
6ca2abb7f8 User Log package for default logger 
* Modify the default logger to it also logs Line Numbers
* Update README w/ logging instructions
 2015-05-23 13:52:57 -07:00
dan
e52fcb4f8c Checkpoint after filtered messages in the pipeline 
When records are filtered we still want to count them in the checkpoint.
This will allow the checkpoint to pick up at the appropriate spot if any
messages are filtered out.
 2015-05-23 12:56:38 -07:00
Harlow Ward
b38102eec7 Silence log warnings during test run 2015-05-23 12:48:31 -07:00
dan
95fe49e7e0 Add specs for AWS Backoff 2015-05-23 12:44:52 -07:00
dan
8c660f79fb Add Retries to Redshift Basic Emitter 
* Move AWS exponential backoff to its own file
 2015-05-23 12:36:59 -07:00
dan
f4de27dc13 Add recoverability in the Pipeline 
* Logic for handling Recoverable errors correctly
* Handle AWS exponential backoff
 2015-05-23 12:12:08 -07:00
Harlow Ward
9077d288d2 Rename License file and add Contributing sections 
* Make it clear from project root we are using MIT-LICENSE
* Add contributing section w/ overview of how to submit code to repo
 2015-05-23 10:24:53 -07:00
Harlow Ward
bde3e96ad9 Add Discard Logger 
For situations where we don't want any logs produced by the application.

* Remove references to Logger from README
* Add implementation of DiscardLogger
 2015-05-22 23:38:06 -07:00
dan
ff1cff0293 Add default logger 
Requiring the Logger be passed in to each of the structs was causing
Panics if the Logger was omitted.

* Add function for overriding the default logger
* Remove panics caused by initialization
 2015-05-22 23:19:58 -07:00
Harlow Ward
82d10ab78d Simplify example (no Redshift cluster required) 2015-05-03 21:01:21 -07:00
Harlow Ward
cd71fd41bc Add Logger interface 
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.

* Add Logger interface
* Use logger interface in pipeline and emitters
 2015-05-03 21:01:14 -07:00
Harlow Ward
5cf67a3c85 Add MIT License 2015-04-10 09:32:30 -07:00
Harlow Ward
da7a1dd87a Update README.md 2014-12-20 19:46:38 -08:00
Harlow Ward
1bce60bae6 Update example with Redshift Manifest Pipeline 2014-12-20 19:45:34 -08:00
Harlow Ward
b7f4cfc841 Remove custom config functions 2014-12-20 19:40:25 -08:00
Harlow Ward
3d290c4512 Shorten the CreateStream function name 2014-12-20 18:55:27 -08:00
Harlow Ward
05920cc5d3 Update development notice 2014-12-18 18:26:59 -08:00
Harlow Ward
236a6eefd0 Add tags to gitignore 2014-12-13 13:25:36 -08:00
Harlow Ward
4366c364d0 Remove unused DeleteStream function 
The function wasn't being used and should be removed from codebase.
 2014-12-10 22:04:28 -08:00
Harlow Ward
106f0d66bb Add Redshift Manifest functionality 
Use a manifest file for batch importing files uploaded to S3.
 2014-12-10 21:59:42 -08:00
Harlow Ward
8e8ee5af73 Use golint to update Golang styles 
* Update comments and return statements
* Adjust usage of Kinesis library (upgraded local source)
 2014-12-10 15:38:19 -08:00
Harlow Ward
02a7c648a3 Update link for AWS attribution 2014-11-15 18:10:19 -08:00
Harlow Ward
a9a625aa79 Update example usage in README 
* Showcase the new StringToStringTransformer as a simple way of getting
data directly from Kinesis to Redshift.
 2014-11-15 17:36:28 -08:00
Harlow Ward
4d00c430cf Sort exported fields alphabetically 2014-11-15 17:20:25 -08:00
Harlow Ward
7c631ba8c0 Add StringToString transformer 
In some cases we'll want to save the data from the stream directly with
no transformation needed. This will allow us to return the raw data
string from the stream

* Add new StringToStringTransformer
* Remove Record from codebase in favor of more generic interface
 2014-11-15 17:07:12 -08:00
Harlow Ward
59f488e6c7 Add line-break after exported fields 
Match the Golang std library style of adding line-breaks after exported
fields.

http://golang.org/src/pkg/sync/cond.go?s=609:772#L12
 2014-11-15 17:06:56 -08:00
Harlow Ward
5095681ad0 Group exported fields 
Godoc removes un-exported fields and puts a line-break in the
documentation when exported fields and un-exported fields aren't grouped
together. Will follow this standard going forward.
 2014-11-15 16:15:18 -08:00
Harlow Ward
b98adcf659 Rename Model to Record 
To match the DSL of the Kinesis library rename the Model interface to
Record.
 2014-11-15 15:54:54 -08:00
Harlow Ward
ad57271712 Documentation for Redis Checkpoint 
* Add Godoc style comments for each of the functions
 2014-11-15 14:04:52 -08:00
Harlow Ward
1b045f6107 Bump sleep time to 10s 
Noticed GetRecords error when a Shard's lease expired.

* Added a timeout so we don't hit the rate limit on this type of error.
* Increased the timeout when there are no Records on the stream.
 2014-11-15 13:51:55 -08:00
Harlow Ward
f921eca908 Add support for JSONPaths in Redshift Load 
By specifying a `jsonpaths` parameter to an S3 file containing ordinal
attribute position we can store free-form JSON in S3 and then reference
the col order from the COPY command.

http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html
 2014-11-15 13:49:37 -08:00
Harlow Ward
06b40e6ed8 Base pipeline components 
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
 2014-11-14 20:45:34 -08:00
Harlow Ward
0fe80b708b Initial commit 
* Add installation instructions to README.
* Make note that this repo is under development.
 2014-07-25 00:22:06 -07:00