Harlow Ward
afae1bea36
Use config object for optional params
...
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.
https://peter.bourgon.org/go-best-practices-2016/#configuration
2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe
Add logging when records are emitted w/ record count
2016-05-01 10:43:42 -07:00
Harlow Ward
c29698550f
Add config options to Consumer
...
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
2016-02-09 22:31:15 -08:00
Harlow Ward
f0e6461cb6
Refactor to use handler func
...
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.
* Add refactoring note and SHA to README
2016-02-06 17:50:17 -08:00
Harlow Ward
18173842fb
Use AWS SDK
...
Limit the amount of library dependencies by using the official AWS SDK.
https://github.com/harlow/kinesis-connectors/issues/19
2015-08-15 23:17:49 -07:00
Harlow Ward
7c631ba8c0
Add StringToString transformer
...
In some cases we'll want to save the data from the stream directly with
no transformation needed. This will allow us to return the raw data
string from the stream
* Add new StringToStringTransformer
* Remove Record from codebase in favor of more generic interface
2014-11-15 17:07:12 -08:00
Harlow Ward
b98adcf659
Rename Model to Record
...
To match the DSL of the Kinesis library rename the Model interface to
Record.
2014-11-15 15:54:54 -08:00
Harlow Ward
f921eca908
Add support for JSONPaths in Redshift Load
...
By specifying a `jsonpaths` parameter to an S3 file containing ordinal
attribute position we can store free-form JSON in S3 and then reference
the col order from the COPY command.
http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html
2014-11-15 13:49:37 -08:00
Harlow Ward
06b40e6ed8
Base pipeline components
...
* Create base Interfaces for Pipeline
* Add first base implementations for Pipeline
* Add initial test for core functionality
2014-11-14 20:45:34 -08:00
