Commit graph

12 commits

Author SHA1 Message Date
Harlow Ward
aae268108e Move emitters to separate packages 2016-02-08 19:39:09 -08:00
Harlow Ward
f0e6461cb6 Refactor to use handler func
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.

* Add refactoring note and SHA to README
2016-02-06 17:50:17 -08:00
Harlow Ward
0d5e9b7b02 Use AWS SDK for S3 Emitter
* Handle retries for S3 Emitter
* Update Pipeline to use AWS backoff
* Leverage aws error kit in recoverable error helpers
2015-08-16 17:52:10 -07:00
Harlow Ward
18173842fb Use AWS SDK
Limit the amount of library dependencies by using the official AWS SDK.

https://github.com/harlow/kinesis-connectors/issues/19
2015-08-15 23:17:49 -07:00
Harlow Ward
cd54569c61 Use go-kit Log interface 2015-05-25 22:40:40 -07:00
dan
9371fb938c Logging and DB connection reuse
* Reuse Redshift DB connection
* Add more logging
2015-05-25 22:23:40 -07:00
dan
ff1cff0293 Add default logger
Requiring the Logger be passed in to each of the structs was causing
Panics if the Logger was omitted.

* Add function for overriding the default logger
* Remove panics caused by initialization
2015-05-22 23:19:58 -07:00
Harlow Ward
cd71fd41bc Add Logger interface
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.

* Add Logger interface
* Use logger interface in pipeline and emitters
2015-05-03 21:01:14 -07:00
Harlow Ward
106f0d66bb Add Redshift Manifest functionality
Use a manifest file for batch importing files uploaded to S3.
2014-12-10 21:59:42 -08:00
Harlow Ward
8e8ee5af73 Use golint to update Golang styles
* Update comments and return statements
* Adjust usage of Kinesis library (upgraded local source)
2014-12-10 15:38:19 -08:00
Harlow Ward
b98adcf659 Rename Model to Record
To match the DSL of the Kinesis library rename the Model interface to
Record.
2014-11-15 15:54:54 -08:00
Harlow Ward
06b40e6ed8 Base pipeline components
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
2014-11-14 20:45:34 -08:00