Harlow Ward
49b5a94c7e
Use Apex log for logging ( #27 )
...
* Use Apex log for logging
2016-04-30 22:23:35 -07:00
Harlow Ward
e843ae5928
Remove poll interval in favor of aws retry backoff
2016-04-30 18:05:04 -07:00
Harlow Ward
dded9d0a0e
Leverage the default AWS retry logic
2016-04-30 17:04:44 -07:00
Harlow Ward
e150d4832b
Add line-breaks after blocks
2016-04-30 16:21:49 -07:00
Harlow Ward
8f1d408c52
Fix recoverable error tests
2016-04-30 16:20:01 -07:00
Harlow Ward
7fccc9da9f
Update function name to represent functionality
2016-04-30 09:14:25 -07:00
Harlow Ward
855af23684
Merge pull request #25 from robbles/master
...
Add gvt manifest
2016-02-16 10:50:01 -08:00
Rob O'Dwyer
ce3df40c90
add section about gvt to README.md
2016-02-09 23:39:57 -08:00
Harlow Ward
c29698550f
Add config options to Consumer
...
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
2016-02-09 22:31:15 -08:00
Rob O'Dwyer
179e9502b0
add gvt manifest
2016-02-09 13:30:54 -08:00
Harlow Ward
6119d5c7bf
Update language for syntax highlighting
2016-02-08 19:42:26 -08:00
Harlow Ward
066e5844fe
Update note about AWS Firehose service
2016-02-08 19:41:36 -08:00
Harlow Ward
aae268108e
Move emitters to separate packages
2016-02-08 19:39:09 -08:00
Harlow Ward
caa2fc0b0a
Add Kinesis Firehose example
2016-02-08 13:21:54 -08:00
Harlow Ward
f0e6461cb6
Refactor to use handler func
...
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.
* Add refactoring note and SHA to README
2016-02-06 17:50:17 -08:00
Harlow Ward
509f68de89
Add error checking for tmp file
2016-01-09 09:52:47 -08:00
Harlow Ward
d542fa996f
Use AWS SDK and concurrency
2016-01-09 09:46:28 -08:00
Harlow Ward
0d5e9b7b02
Use AWS SDK for S3 Emitter
...
* Handle retries for S3 Emitter
* Update Pipeline to use AWS backoff
* Leverage aws error kit in recoverable error helpers
2015-08-16 17:52:10 -07:00
Harlow Ward
18173842fb
Use AWS SDK
...
Limit the amount of library dependencies by using the official AWS SDK.
https://github.com/harlow/kinesis-connectors/issues/19
2015-08-15 23:17:49 -07:00
Harlow Ward
8277ffc6be
Use GoKit logger interface directly
2015-05-26 18:50:34 -07:00
Harlow Ward
3d9e6e2485
Add Redshift Pipeline example
2015-05-25 23:18:14 -07:00
Harlow Ward
e17988cfc9
Update Logging instructions
2015-05-25 23:09:59 -07:00
Harlow Ward
cd54569c61
Use go-kit Log interface
2015-05-25 22:40:40 -07:00
dan
9371fb938c
Logging and DB connection reuse
...
* Reuse Redshift DB connection
* Add more logging
2015-05-25 22:23:40 -07:00
Harlow Ward
a27a13f405
Update README.md
2015-05-23 16:18:10 -07:00
Harlow Ward
9a30e97f10
Add link to exmaples directory
2015-05-23 16:10:08 -07:00
Harlow Ward
25e390e8b4
Add sample pipeline
...
* Reduce the size of main README in favor of adding functioning examples
* Add S3 Pipeline exmaple
* Add example of seeing the stream
2015-05-23 15:27:40 -07:00
Harlow Ward
fdd39482bb
Add Installation sub-header to Usage
2015-05-23 13:57:52 -07:00
Harlow Ward
b9a489b940
Update header size. Remove note
2015-05-23 13:56:03 -07:00
Harlow Ward
6ca2abb7f8
User Log package for default logger
...
* Modify the default logger to it also logs Line Numbers
* Update README w/ logging instructions
2015-05-23 13:52:57 -07:00
dan
e52fcb4f8c
Checkpoint after filtered messages in the pipeline
...
When records are filtered we still want to count them in the checkpoint.
This will allow the checkpoint to pick up at the appropriate spot if any
messages are filtered out.
2015-05-23 12:56:38 -07:00
Harlow Ward
b38102eec7
Silence log warnings during test run
2015-05-23 12:48:31 -07:00
dan
95fe49e7e0
Add specs for AWS Backoff
2015-05-23 12:44:52 -07:00
dan
8c660f79fb
Add Retries to Redshift Basic Emitter
...
* Move AWS exponential backoff to its own file
2015-05-23 12:36:59 -07:00
dan
f4de27dc13
Add recoverability in the Pipeline
...
* Logic for handling Recoverable errors correctly
* Handle AWS exponential backoff
2015-05-23 12:12:08 -07:00
Harlow Ward
9077d288d2
Rename License file and add Contributing sections
...
* Make it clear from project root we are using MIT-LICENSE
* Add contributing section w/ overview of how to submit code to repo
2015-05-23 10:24:53 -07:00
Harlow Ward
bde3e96ad9
Add Discard Logger
...
For situations where we don't want any logs produced by the application.
* Remove references to Logger from README
* Add implementation of DiscardLogger
2015-05-22 23:38:06 -07:00
dan
ff1cff0293
Add default logger
...
Requiring the Logger be passed in to each of the structs was causing
Panics if the Logger was omitted.
* Add function for overriding the default logger
* Remove panics caused by initialization
2015-05-22 23:19:58 -07:00
Harlow Ward
82d10ab78d
Simplify example (no Redshift cluster required)
2015-05-03 21:01:21 -07:00
Harlow Ward
cd71fd41bc
Add Logger interface
...
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.
* Add Logger interface
* Use logger interface in pipeline and emitters
2015-05-03 21:01:14 -07:00
Harlow Ward
5cf67a3c85
Add MIT License
2015-04-10 09:32:30 -07:00
Harlow Ward
da7a1dd87a
Update README.md
2014-12-20 19:46:38 -08:00
Harlow Ward
1bce60bae6
Update example with Redshift Manifest Pipeline
2014-12-20 19:45:34 -08:00
Harlow Ward
b7f4cfc841
Remove custom config functions
2014-12-20 19:40:25 -08:00
Harlow Ward
3d290c4512
Shorten the CreateStream function name
2014-12-20 18:55:27 -08:00
Harlow Ward
05920cc5d3
Update development notice
2014-12-18 18:26:59 -08:00
Harlow Ward
236a6eefd0
Add tags to gitignore
2014-12-13 13:25:36 -08:00
Harlow Ward
4366c364d0
Remove unused DeleteStream function
...
The function wasn't being used and should be removed from codebase.
2014-12-10 22:04:28 -08:00
Harlow Ward
106f0d66bb
Add Redshift Manifest functionality
...
Use a manifest file for batch importing files uploaded to S3.
2014-12-10 21:59:42 -08:00
Harlow Ward
8e8ee5af73
Use golint to update Golang styles
...
* Update comments and return statements
* Adjust usage of Kinesis library (upgraded local source)
2014-12-10 15:38:19 -08:00
