dan
efa20e0ff6
Merge branch 'master' into hw-logger-interface
...
Conflicts:
pipeline.go
redshift_basic_emitter.go
redshift_manifest_emitter.go
s3_emitter.go
s3_manifest_emitter.go
2015-05-15 12:03:07 -07:00
dan
e42dd7f73d
move aws backoff to its own file, and use it for redshift basic emitter
2015-05-06 09:14:00 -07:00
Harlow Ward
cd71fd41bc
Add Logger interface
...
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.
* Add Logger interface
* Use logger interface in pipeline and emitters
2015-05-03 21:01:14 -07:00
dan
e5af96fb54
handle more errors
2015-04-21 15:14:24 -07:00
dan
46e5d62884
handle more kinesis recoverable errors
2015-04-08 09:54:20 -07:00
dan
a028ee862f
add more recoverable errors
2015-04-07 14:40:34 -07:00
dan
969ba18824
if there are no records available, keep going until the nextiterator returns nothing
2015-04-07 13:42:50 -07:00
dan
2ff748a2d4
checkpoint after filtered messages in the pipeline
...
checkpoint after filtered messages in the pipeline
checkpoint after filtered messages in the pipeline
2015-04-07 11:26:20 -07:00
dan
0dce2a6045
debug for errors that are killing the pipeline
2015-04-07 10:42:07 -07:00
dan
f1af849e81
handle recoverable error properly
2015-04-07 08:53:06 -07:00
dan
e3efa383c1
change dependency on log4go to different folder
2015-04-06 11:38:20 -07:00
dan
18163c1599
change the l4g.Critical, os.Exit combo to log.Fatal, because l4g defers output so the critical logs are not being seen
2015-04-04 10:24:23 -07:00
dan
02a9d5d173
use log4go output
2015-04-03 23:05:49 -07:00
dan
5301dfe963
fix the recoverable error logic to prevent crash
2015-04-03 22:48:38 -07:00
dan
6502115563
add recoverability in the pipeline, especially for throughput errors
2015-04-03 15:38:04 -07:00
dan
797b575ad1
add recoverability in the pipeline, especially for throughput errors
2015-04-03 15:33:34 -07:00
Harlow Ward
106f0d66bb
Add Redshift Manifest functionality
...
Use a manifest file for batch importing files uploaded to S3.
2014-12-10 21:59:42 -08:00
Harlow Ward
8e8ee5af73
Use golint to update Golang styles
...
* Update comments and return statements
* Adjust usage of Kinesis library (upgraded local source)
2014-12-10 15:38:19 -08:00
Harlow Ward
b98adcf659
Rename Model to Record
...
To match the DSL of the Kinesis library rename the Model interface to
Record.
2014-11-15 15:54:54 -08:00
Harlow Ward
1b045f6107
Bump sleep time to 10s
...
Noticed GetRecords error when a Shard's lease expired.
* Added a timeout so we don't hit the rate limit on this type of error.
* Increased the timeout when there are no Records on the stream.
2014-11-15 13:51:55 -08:00
Harlow Ward
06b40e6ed8
Base pipeline components
...
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
2014-11-14 20:45:34 -08:00
