luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
49 commits 6 branches 9 tags 7.5 MiB
Commit graph

49 commits

This branch
This branch
All branches
Author SHA1 Message Date
dan
05330c6a9d Merge branch 'hw-updates' into ez-recover-errors 
Conflicts:
	pipeline.go
	redshift_basic_emitter.go
 2015-05-15 13:15:20 -07:00
dan
b045031df8 add default logger to remove panics caused by initialization 2015-05-15 12:32:45 -07:00
dan
efa20e0ff6 Merge branch 'master' into hw-logger-interface 
Conflicts:
	pipeline.go
	redshift_basic_emitter.go
	redshift_manifest_emitter.go
	s3_emitter.go
	s3_manifest_emitter.go
 2015-05-15 12:03:07 -07:00
dan
e6a0af7ffd move aws backoff to its own file, and use it for redshift basic emitter 2015-05-06 09:18:15 -07:00
dan
e42dd7f73d move aws backoff to its own file, and use it for redshift basic emitter 2015-05-06 09:14:00 -07:00
dan
2a285c52d5 add mysql checkpoint to kinesis-connectors 2015-05-06 08:44:14 -07:00
Harlow Ward
82d10ab78d Simplify example (no Redshift cluster required) 2015-05-03 21:01:21 -07:00
Harlow Ward
cd71fd41bc Add Logger interface 
To allow for different logging endpoints we'll introduce a Logger
interface that will be passed into the pipeline during initialization.

* Add Logger interface
* Use logger interface in pipeline and emitters
 2015-05-03 21:01:14 -07:00
dan
e5af96fb54 handle more errors 2015-04-21 15:14:24 -07:00
Harlow Ward
5cf67a3c85 Add MIT License 2015-04-10 09:32:30 -07:00
dan
46e5d62884 handle more kinesis recoverable errors 2015-04-08 09:54:20 -07:00
dan
a028ee862f add more recoverable errors 2015-04-07 14:40:34 -07:00
dan
969ba18824 if there are no records available, keep going until the nextiterator returns nothing 2015-04-07 13:42:50 -07:00
dan
2ff748a2d4 checkpoint after filtered messages in the pipeline 
checkpoint after filtered messages in the pipeline

checkpoint after filtered messages in the pipeline
 2015-04-07 11:26:20 -07:00
dan
0dce2a6045 debug for errors that are killing the pipeline 2015-04-07 10:42:07 -07:00
dan
f1af849e81 handle recoverable error properly 2015-04-07 08:53:06 -07:00
dan
6f8ff3f11d use a prefix for files generated on s3 2015-04-06 14:52:04 -07:00
dan
9ed761edc6 re-use the database connection to redshift 2015-04-06 14:51:46 -07:00
dan
e3efa383c1 change dependency on log4go to different folder 2015-04-06 11:38:20 -07:00
dan
18163c1599 change the l4g.Critical, os.Exit combo to log.Fatal, because l4g defers output so the critical logs are not being seen 2015-04-04 10:24:23 -07:00
dan
852cf2a7d2 more log4go additions 2015-04-03 23:17:19 -07:00
dan
d746c2529a more log4go additions 2015-04-03 23:14:08 -07:00
dan
b89c69d86b more log4go additions 2015-04-03 23:09:33 -07:00
dan
02a9d5d173 use log4go output 2015-04-03 23:05:49 -07:00
dan
5301dfe963 fix the recoverable error logic to prevent crash 2015-04-03 22:48:38 -07:00
dan
6502115563 add recoverability in the pipeline, especially for throughput errors 2015-04-03 15:38:04 -07:00
dan
797b575ad1 add recoverability in the pipeline, especially for throughput errors 2015-04-03 15:33:34 -07:00
dan
c5c7dd49c9 add / between bucket and filename in copy statement 2015-03-29 12:22:29 -07:00
Harlow Ward
da7a1dd87a Update README.md 2014-12-20 19:46:38 -08:00
Harlow Ward
1bce60bae6 Update example with Redshift Manifest Pipeline 2014-12-20 19:45:34 -08:00
Harlow Ward
b7f4cfc841 Remove custom config functions 2014-12-20 19:40:25 -08:00
Harlow Ward
3d290c4512 Shorten the CreateStream function name 2014-12-20 18:55:27 -08:00
Harlow Ward
05920cc5d3 Update development notice 2014-12-18 18:26:59 -08:00
Harlow Ward
236a6eefd0 Add tags to gitignore 2014-12-13 13:25:36 -08:00
Harlow Ward
4366c364d0 Remove unused DeleteStream function 
The function wasn't being used and should be removed from codebase.
 2014-12-10 22:04:28 -08:00
Harlow Ward
106f0d66bb Add Redshift Manifest functionality 
Use a manifest file for batch importing files uploaded to S3.
 2014-12-10 21:59:42 -08:00
Harlow Ward
8e8ee5af73 Use golint to update Golang styles 
* Update comments and return statements
* Adjust usage of Kinesis library (upgraded local source)
 2014-12-10 15:38:19 -08:00
Harlow Ward
02a7c648a3 Update link for AWS attribution 2014-11-15 18:10:19 -08:00
Harlow Ward
a9a625aa79 Update example usage in README 
* Showcase the new StringToStringTransformer as a simple way of getting
data directly from Kinesis to Redshift.
 2014-11-15 17:36:28 -08:00
Harlow Ward
4d00c430cf Sort exported fields alphabetically 2014-11-15 17:20:25 -08:00
Harlow Ward
7c631ba8c0 Add StringToString transformer 
In some cases we'll want to save the data from the stream directly with
no transformation needed. This will allow us to return the raw data
string from the stream

* Add new StringToStringTransformer
* Remove Record from codebase in favor of more generic interface
 2014-11-15 17:07:12 -08:00
Harlow Ward
59f488e6c7 Add line-break after exported fields 
Match the Golang std library style of adding line-breaks after exported
fields.

http://golang.org/src/pkg/sync/cond.go?s=609:772#L12
 2014-11-15 17:06:56 -08:00
Harlow Ward
5095681ad0 Group exported fields 
Godoc removes un-exported fields and puts a line-break in the
documentation when exported fields and un-exported fields aren't grouped
together. Will follow this standard going forward.
 2014-11-15 16:15:18 -08:00
Harlow Ward
b98adcf659 Rename Model to Record 
To match the DSL of the Kinesis library rename the Model interface to
Record.
 2014-11-15 15:54:54 -08:00
Harlow Ward
ad57271712 Documentation for Redis Checkpoint 
* Add Godoc style comments for each of the functions
 2014-11-15 14:04:52 -08:00
Harlow Ward
1b045f6107 Bump sleep time to 10s 
Noticed GetRecords error when a Shard's lease expired.

* Added a timeout so we don't hit the rate limit on this type of error.
* Increased the timeout when there are no Records on the stream.
 2014-11-15 13:51:55 -08:00
Harlow Ward
f921eca908 Add support for JSONPaths in Redshift Load 
By specifying a `jsonpaths` parameter to an S3 file containing ordinal
attribute position we can store free-form JSON in S3 and then reference
the col order from the COPY command.

http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html
 2014-11-15 13:49:37 -08:00
Harlow Ward
06b40e6ed8 Base pipeline components 
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
 2014-11-14 20:45:34 -08:00
Harlow Ward
0fe80b708b Initial commit 
* Add installation instructions to README.
* Make note that this repo is under development.
 2014-07-25 00:22:06 -07:00