Commit graph

10 commits

Author SHA1 Message Date
dan
e52fcb4f8c Checkpoint after filtered messages in the pipeline
When records are filtered we still want to count them in the checkpoint.
This will allow the checkpoint to pick up at the appropriate spot if any
messages are filtered out.
2015-05-23 12:56:38 -07:00
dan
8c660f79fb Add Retries to Redshift Basic Emitter
* Move AWS exponential backoff to its own file
2015-05-23 12:36:59 -07:00
dan
f4de27dc13 Add recoverability in the Pipeline
* Logic for handling Recoverable errors correctly
* Handle AWS exponential backoff
2015-05-23 12:12:08 -07:00
dan
ff1cff0293 Add default logger
Requiring the Logger be passed in to each of the structs was causing
Panics if the Logger was omitted.

* Add function for overriding the default logger
* Remove panics caused by initialization
2015-05-22 23:19:58 -07:00
Harlow Ward
cd71fd41bc Add Logger interface
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.

* Add Logger interface
* Use logger interface in pipeline and emitters
2015-05-03 21:01:14 -07:00
Harlow Ward
106f0d66bb Add Redshift Manifest functionality
Use a manifest file for batch importing files uploaded to S3.
2014-12-10 21:59:42 -08:00
Harlow Ward
8e8ee5af73 Use golint to update Golang styles
* Update comments and return statements
* Adjust usage of Kinesis library (upgraded local source)
2014-12-10 15:38:19 -08:00
Harlow Ward
b98adcf659 Rename Model to Record
To match the DSL of the Kinesis library rename the Model interface to
Record.
2014-11-15 15:54:54 -08:00
Harlow Ward
1b045f6107 Bump sleep time to 10s
Noticed GetRecords error when a Shard's lease expired.

* Added a timeout so we don't hit the rate limit on this type of error.
* Increased the timeout when there are no Records on the stream.
2014-11-15 13:51:55 -08:00
Harlow Ward
06b40e6ed8 Base pipeline components
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
2014-11-14 20:45:34 -08:00