Pierre Massat
c56cefb667
Expose the shard ID in the buffer ( #30 )
2016-12-26 08:24:34 -07:00
Harlow Ward
fedb6812fb
Add checkpoint interface for custom checkpoints ( #29 )
...
To allow other checkpoint backends we extract a checkpoint interface
which future checkpoints can implement.
* Add checkpoint interface for custom checkpoints
* Create RedisCheckpoint to implement checkpoint interface
* Swap out hosie redis library for go-redis
Minor changes
* Allow configuration of Redis endpoint with env var `REDIS_URL`
* Replace gvt with govendor
2016-12-04 00:08:06 -08:00
Harlow Ward
cc936aed04
Update stream name label
2016-05-27 18:43:55 -06:00
Harlow Ward
a2e443925d
Update config example
2016-05-07 18:15:55 -07:00
Harlow Ward
ceca88b96a
Add required fields to Config
...
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
2016-05-07 18:10:31 -07:00
Harlow Ward
f4c8d8090d
Use TJ's Kinesis producer library
2016-05-01 16:54:47 -07:00
Harlow Ward
8d71bbd0ec
Update S3 emitter initialization
2016-05-01 15:07:58 -07:00
Harlow Ward
07efe9750c
Bump gvt vendored libraries
2016-05-01 14:56:50 -07:00
Harlow Ward
ff5ff00ad7
Remove hr line break from footer
2016-05-01 12:46:23 -07:00
Harlow Ward
9a17d4098e
Add footer links to social profiles
2016-05-01 12:45:27 -07:00
Harlow Ward
a12c15a191
Use no-op Info logging by default
2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36
Use config object for optional params
...
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.
https://peter.bourgon.org/go-best-practices-2016/#configuration
2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe
Add logging when records are emitted w/ record count
2016-05-01 10:43:42 -07:00
Harlow Ward
c7a69e2c92
Add benchmark test to buffer
2016-05-01 10:42:51 -07:00
Harlow Ward
d5bdd3f4bc
Leverage AWS S3 retries
2016-05-01 10:42:28 -07:00
Harlow Ward
49b5a94c7e
Use Apex log for logging ( #27 )
...
* Use Apex log for logging
2016-04-30 22:23:35 -07:00
Harlow Ward
e843ae5928
Remove poll interval in favor of aws retry backoff
2016-04-30 18:05:04 -07:00
Harlow Ward
dded9d0a0e
Leverage the default AWS retry logic
2016-04-30 17:04:44 -07:00
Harlow Ward
e150d4832b
Add line-breaks after blocks
2016-04-30 16:21:49 -07:00
Harlow Ward
8f1d408c52
Fix recoverable error tests
2016-04-30 16:20:01 -07:00
Harlow Ward
7fccc9da9f
Update function name to represent functionality
2016-04-30 09:14:25 -07:00
Harlow Ward
855af23684
Merge pull request #25 from robbles/master
...
Add gvt manifest
2016-02-16 10:50:01 -08:00
Rob O'Dwyer
ce3df40c90
add section about gvt to README.md
2016-02-09 23:39:57 -08:00
Harlow Ward
c29698550f
Add config options to Consumer
...
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
2016-02-09 22:31:15 -08:00
Rob O'Dwyer
179e9502b0
add gvt manifest
2016-02-09 13:30:54 -08:00
Harlow Ward
6119d5c7bf
Update language for syntax highlighting
2016-02-08 19:42:26 -08:00
Harlow Ward
066e5844fe
Update note about AWS Firehose service
2016-02-08 19:41:36 -08:00
Harlow Ward
aae268108e
Move emitters to separate packages
2016-02-08 19:39:09 -08:00
Harlow Ward
caa2fc0b0a
Add Kinesis Firehose example
2016-02-08 13:21:54 -08:00
Harlow Ward
f0e6461cb6
Refactor to use handler func
...
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.
* Add refactoring note and SHA to README
2016-02-06 17:50:17 -08:00
Harlow Ward
509f68de89
Add error checking for tmp file
2016-01-09 09:52:47 -08:00
Harlow Ward
d542fa996f
Use AWS SDK and concurrency
2016-01-09 09:46:28 -08:00
Harlow Ward
0d5e9b7b02
Use AWS SDK for S3 Emitter
...
* Handle retries for S3 Emitter
* Update Pipeline to use AWS backoff
* Leverage aws error kit in recoverable error helpers
2015-08-16 17:52:10 -07:00
Harlow Ward
18173842fb
Use AWS SDK
...
Limit the amount of library dependencies by using the official AWS SDK.
https://github.com/harlow/kinesis-connectors/issues/19
2015-08-15 23:17:49 -07:00
Harlow Ward
8277ffc6be
Use GoKit logger interface directly
2015-05-26 18:50:34 -07:00
Harlow Ward
3d9e6e2485
Add Redshift Pipeline example
2015-05-25 23:18:14 -07:00
Harlow Ward
e17988cfc9
Update Logging instructions
2015-05-25 23:09:59 -07:00
Harlow Ward
cd54569c61
Use go-kit Log interface
2015-05-25 22:40:40 -07:00
dan
9371fb938c
Logging and DB connection reuse
...
* Reuse Redshift DB connection
* Add more logging
2015-05-25 22:23:40 -07:00
Harlow Ward
a27a13f405
Update README.md
2015-05-23 16:18:10 -07:00
Harlow Ward
9a30e97f10
Add link to exmaples directory
2015-05-23 16:10:08 -07:00
Harlow Ward
25e390e8b4
Add sample pipeline
...
* Reduce the size of main README in favor of adding functioning examples
* Add S3 Pipeline exmaple
* Add example of seeing the stream
2015-05-23 15:27:40 -07:00
Harlow Ward
fdd39482bb
Add Installation sub-header to Usage
2015-05-23 13:57:52 -07:00
Harlow Ward
b9a489b940
Update header size. Remove note
2015-05-23 13:56:03 -07:00
Harlow Ward
6ca2abb7f8
User Log package for default logger
...
* Modify the default logger to it also logs Line Numbers
* Update README w/ logging instructions
2015-05-23 13:52:57 -07:00
dan
e52fcb4f8c
Checkpoint after filtered messages in the pipeline
...
When records are filtered we still want to count them in the checkpoint.
This will allow the checkpoint to pick up at the appropriate spot if any
messages are filtered out.
2015-05-23 12:56:38 -07:00
Harlow Ward
b38102eec7
Silence log warnings during test run
2015-05-23 12:48:31 -07:00
dan
95fe49e7e0
Add specs for AWS Backoff
2015-05-23 12:44:52 -07:00
dan
8c660f79fb
Add Retries to Redshift Basic Emitter
...
* Move AWS exponential backoff to its own file
2015-05-23 12:36:59 -07:00
dan
f4de27dc13
Add recoverability in the Pipeline
...
* Logic for handling Recoverable errors correctly
* Handle AWS exponential backoff
2015-05-23 12:12:08 -07:00
