dan
2ff748a2d4
checkpoint after filtered messages in the pipeline
...
checkpoint after filtered messages in the pipeline
checkpoint after filtered messages in the pipeline
2015-04-07 11:26:20 -07:00
dan
0dce2a6045
debug for errors that are killing the pipeline
2015-04-07 10:42:07 -07:00
dan
f1af849e81
handle recoverable error properly
2015-04-07 08:53:06 -07:00
dan
e3efa383c1
change dependency on log4go to different folder
2015-04-06 11:38:20 -07:00
dan
18163c1599
change the l4g.Critical, os.Exit combo to log.Fatal, because l4g defers output so the critical logs are not being seen
2015-04-04 10:24:23 -07:00
dan
02a9d5d173
use log4go output
2015-04-03 23:05:49 -07:00
dan
5301dfe963
fix the recoverable error logic to prevent crash
2015-04-03 22:48:38 -07:00
dan
6502115563
add recoverability in the pipeline, especially for throughput errors
2015-04-03 15:38:04 -07:00
dan
797b575ad1
add recoverability in the pipeline, especially for throughput errors
2015-04-03 15:33:34 -07:00
Harlow Ward
106f0d66bb
Add Redshift Manifest functionality
...
Use a manifest file for batch importing files uploaded to S3.
2014-12-10 21:59:42 -08:00
Harlow Ward
8e8ee5af73
Use golint to update Golang styles
...
* Update comments and return statements
* Adjust usage of Kinesis library (upgraded local source)
2014-12-10 15:38:19 -08:00
Harlow Ward
b98adcf659
Rename Model to Record
...
To match the DSL of the Kinesis library rename the Model interface to
Record.
2014-11-15 15:54:54 -08:00
Harlow Ward
1b045f6107
Bump sleep time to 10s
...
Noticed GetRecords error when a Shard's lease expired.
* Added a timeout so we don't hit the rate limit on this type of error.
* Increased the timeout when there are no Records on the stream.
2014-11-15 13:51:55 -08:00
Harlow Ward
06b40e6ed8
Base pipeline components
...
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
2014-11-14 20:45:34 -08:00
