dan
f4de27dc13
Add recoverability in the Pipeline
...
* Logic for handling Recoverable errors correctly
* Handle AWS exponential backoff
2015-05-23 12:12:08 -07:00
Harlow Ward
9077d288d2
Rename License file and add Contributing sections
...
* Make it clear from project root we are using MIT-LICENSE
* Add contributing section w/ overview of how to submit code to repo
2015-05-23 10:24:53 -07:00
Harlow Ward
bde3e96ad9
Add Discard Logger
...
For situations where we don't want any logs produced by the application.
* Remove references to Logger from README
* Add implementation of DiscardLogger
2015-05-22 23:38:06 -07:00
dan
ff1cff0293
Add default logger
...
Requiring the Logger be passed in to each of the structs was causing
Panics if the Logger was omitted.
* Add function for overriding the default logger
* Remove panics caused by initialization
2015-05-22 23:19:58 -07:00
Harlow Ward
82d10ab78d
Simplify example (no Redshift cluster required)
2015-05-03 21:01:21 -07:00
Harlow Ward
cd71fd41bc
Add Logger interface
...
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.
* Add Logger interface
* Use logger interface in pipeline and emitters
2015-05-03 21:01:14 -07:00
Harlow Ward
5cf67a3c85
Add MIT License
2015-04-10 09:32:30 -07:00
Harlow Ward
da7a1dd87a
Update README.md
2014-12-20 19:46:38 -08:00
Harlow Ward
1bce60bae6
Update example with Redshift Manifest Pipeline
2014-12-20 19:45:34 -08:00
Harlow Ward
b7f4cfc841
Remove custom config functions
2014-12-20 19:40:25 -08:00
Harlow Ward
3d290c4512
Shorten the CreateStream function name
2014-12-20 18:55:27 -08:00
Harlow Ward
05920cc5d3
Update development notice
2014-12-18 18:26:59 -08:00
Harlow Ward
236a6eefd0
Add tags to gitignore
2014-12-13 13:25:36 -08:00
Harlow Ward
4366c364d0
Remove unused DeleteStream function
...
The function wasn't being used and should be removed from codebase.
2014-12-10 22:04:28 -08:00
Harlow Ward
106f0d66bb
Add Redshift Manifest functionality
...
Use a manifest file for batch importing files uploaded to S3.
2014-12-10 21:59:42 -08:00
Harlow Ward
8e8ee5af73
Use golint to update Golang styles
...
* Update comments and return statements
* Adjust usage of Kinesis library (upgraded local source)
2014-12-10 15:38:19 -08:00
Harlow Ward
02a7c648a3
Update link for AWS attribution
2014-11-15 18:10:19 -08:00
Harlow Ward
a9a625aa79
Update example usage in README
...
* Showcase the new StringToStringTransformer as a simple way of getting
data directly from Kinesis to Redshift.
2014-11-15 17:36:28 -08:00
Harlow Ward
4d00c430cf
Sort exported fields alphabetically
2014-11-15 17:20:25 -08:00
Harlow Ward
7c631ba8c0
Add StringToString transformer
...
In some cases we'll want to save the data from the stream directly with
no transformation needed. This will allow us to return the raw data
string from the stream
* Add new StringToStringTransformer
* Remove Record from codebase in favor of more generic interface
2014-11-15 17:07:12 -08:00
Harlow Ward
59f488e6c7
Add line-break after exported fields
...
Match the Golang std library style of adding line-breaks after exported
fields.
http://golang.org/src/pkg/sync/cond.go?s=609:772#L12
2014-11-15 17:06:56 -08:00
Harlow Ward
5095681ad0
Group exported fields
...
Godoc removes un-exported fields and puts a line-break in the
documentation when exported fields and un-exported fields aren't grouped
together. Will follow this standard going forward.
2014-11-15 16:15:18 -08:00
Harlow Ward
b98adcf659
Rename Model to Record
...
To match the DSL of the Kinesis library rename the Model interface to
Record.
2014-11-15 15:54:54 -08:00
Harlow Ward
ad57271712
Documentation for Redis Checkpoint
...
* Add Godoc style comments for each of the functions
2014-11-15 14:04:52 -08:00
Harlow Ward
1b045f6107
Bump sleep time to 10s
...
Noticed GetRecords error when a Shard's lease expired.
* Added a timeout so we don't hit the rate limit on this type of error.
* Increased the timeout when there are no Records on the stream.
2014-11-15 13:51:55 -08:00
Harlow Ward
f921eca908
Add support for JSONPaths in Redshift Load
...
By specifying a `jsonpaths` parameter to an S3 file containing ordinal
attribute position we can store free-form JSON in S3 and then reference
the col order from the COPY command.
http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html
2014-11-15 13:49:37 -08:00
Harlow Ward
06b40e6ed8
Base pipeline components
...
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
2014-11-14 20:45:34 -08:00
Harlow Ward
0fe80b708b
Initial commit
...
* Add installation instructions to README.
* Make note that this repo is under development.
2014-07-25 00:22:06 -07:00
