kinesis-consumer/README.md

# Golang Kinesis Connectors

__Kinesis connector applications written in Go__

Inspired by the [Amazon Kinesis Connector Library][1]. These components are used for extracting streaming event data
into S3, Redshift, DynamoDB, and more. See the [API Docs][2] for package documentation.

[1]: https://github.com/awslabs/amazon-kinesis-connectors
[2]: http://godoc.org/github.com/harlow/kinesis-connectors

![golang_kinesis_connector](https://cloud.githubusercontent.com/assets/739782/4262283/2ee2550e-3b97-11e4-8cd1-21a5d7ee0964.png)

## Overview

Each Amazon Kinesis connector application is a pipeline that determines how records from an Amazon Kinesis stream will be handled. Records are retrieved from the stream, transformed according to a user-defined data model, buffered for batch processing, and then emitted to the appropriate AWS service.

A connector pipeline uses the following interfaces:

* __Pipeline:__ The pipeline implementation itself.
* __Transformer:__ Defines the transformation of records from the Amazon Kinesis stream in order to suit the user-defined data model. Includes methods for custom serializer/deserializers.
* __Filter:__ Defines a method for excluding irrelevant records from the processing.
* __Buffer:__ Defines a system for batching the set of records to be processed. The application can specify three thresholds: number of records, total byte count, and time. When one of these thresholds is crossed, the buffer is flushed and the data is emitted to the destination.
* __Emitter:__ Defines a method that makes client calls to other AWS services and persists the records stored in the buffer. The records can also be sent to another Amazon Kinesis stream.

### Installation

Get the package source:

    $ go get github.com/harlow/kinesis-connectors

### Example Pipelines

Examples pipelines:

* [S3 Pipeline](https://github.com/harlow/kinesis-connectors/tree/master/examples/s3-pipeline)
* [Redshift Basic Pipeline](https://github.com/harlow/kinesis-connectors/tree/master/examples/redshift-pipeline)

### Logging

Default logging is handled by [go-kit package log](https://github.com/go-kit/kit/tree/master/log). Applications can override the default loging behaviour by implementing the [Logger interface][log_interface].

```go
connector.SetLogger(NewCustomLogger())
```

[log_interface]: https://github.com/harlow/kinesis-connectors/blob/master/logger.go

## Contributing

Please see [CONTRIBUTING.md] for more information. Thank you, [contributors]!

[LICENSE]: /MIT-LICENSE
[CONTRIBUTING.md]: /CONTRIBUTING.md

## License

Copyright (c) 2015 Harlow Ward. It is free software, and may
be redistributed under the terms specified in the [LICENSE] file.

[contributors]: https://github.com/harlow/kinesis-connectors/graphs/contributors
Base pipeline components * Create base Interfaces for Pipeline * Add first base implementations for Pipeline * Add initial test for core functionality 2014-07-25 06:03:41 +00:00			`# Golang Kinesis Connectors`
Initial commit * Add installation instructions to README. * Make note that this repo is under development. 2014-07-25 06:03:41 +00:00
Update README.md 2015-05-23 23:18:10 +00:00			`__Kinesis connector applications written in Go__`
Initial commit * Add installation instructions to README. * Make note that this repo is under development. 2014-07-25 06:03:41 +00:00
Update development notice 2014-12-19 02:26:59 +00:00			`Inspired by the [Amazon Kinesis Connector Library][1]. These components are used for extracting streaming event data`
Update link for AWS attribution 2014-11-16 02:10:19 +00:00			`into S3, Redshift, DynamoDB, and more. See the [API Docs][2] for package documentation.`
Initial commit * Add installation instructions to README. * Make note that this repo is under development. 2014-07-25 06:03:41 +00:00
Rename License file and add Contributing sections * Make it clear from project root we are using MIT-LICENSE * Add contributing section w/ overview of how to submit code to repo 2015-05-23 15:52:08 +00:00			`[1]: https://github.com/awslabs/amazon-kinesis-connectors`
			`[2]: http://godoc.org/github.com/harlow/kinesis-connectors`

Update README.md 2015-05-23 23:18:10 +00:00			`![golang_kinesis_connector](https://cloud.githubusercontent.com/assets/739782/4262283/2ee2550e-3b97-11e4-8cd1-21a5d7ee0964.png)`

Base pipeline components * Create base Interfaces for Pipeline * Add first base implementations for Pipeline * Add initial test for core functionality 2014-07-25 06:03:41 +00:00			`## Overview`
Initial commit * Add installation instructions to README. * Make note that this repo is under development. 2014-07-25 06:03:41 +00:00
Base pipeline components * Create base Interfaces for Pipeline * Add first base implementations for Pipeline * Add initial test for core functionality 2014-07-25 06:03:41 +00:00			`Each Amazon Kinesis connector application is a pipeline that determines how records from an Amazon Kinesis stream will be handled. Records are retrieved from the stream, transformed according to a user-defined data model, buffered for batch processing, and then emitted to the appropriate AWS service.`
Initial commit * Add installation instructions to README. * Make note that this repo is under development. 2014-07-25 06:03:41 +00:00
Base pipeline components * Create base Interfaces for Pipeline * Add first base implementations for Pipeline * Add initial test for core functionality 2014-07-25 06:03:41 +00:00			`A connector pipeline uses the following interfaces:`

			`* __Pipeline:__ The pipeline implementation itself.`
			`* __Transformer:__ Defines the transformation of records from the Amazon Kinesis stream in order to suit the user-defined data model. Includes methods for custom serializer/deserializers.`
			`* __Filter:__ Defines a method for excluding irrelevant records from the processing.`
			`* __Buffer:__ Defines a system for batching the set of records to be processed. The application can specify three thresholds: number of records, total byte count, and time. When one of these thresholds is crossed, the buffer is flushed and the data is emitted to the destination.`
			`* __Emitter:__ Defines a method that makes client calls to other AWS services and persists the records stored in the buffer. The records can also be sent to another Amazon Kinesis stream.`

Add Installation sub-header to Usage 2015-05-23 20:57:52 +00:00			`### Installation`

			`Get the package source:`
Base pipeline components * Create base Interfaces for Pipeline * Add first base implementations for Pipeline * Add initial test for core functionality 2014-07-25 06:03:41 +00:00
			`$ go get github.com/harlow/kinesis-connectors`

Update README.md 2015-05-23 23:18:10 +00:00			`### Example Pipelines`
Add link to exmaples directory 2015-05-23 23:10:08 +00:00
Add Redshift Pipeline example 2015-05-26 06:18:14 +00:00			`Examples pipelines:`
Add link to exmaples directory 2015-05-23 23:10:08 +00:00
Add Redshift Pipeline example 2015-05-26 06:18:14 +00:00			`* [S3 Pipeline](https://github.com/harlow/kinesis-connectors/tree/master/examples/s3-pipeline)`
			`* [Redshift Basic Pipeline](https://github.com/harlow/kinesis-connectors/tree/master/examples/redshift-pipeline)`
Add link to exmaples directory 2015-05-23 23:10:08 +00:00
Update Logging instructions 2015-05-26 06:09:59 +00:00			`### Logging`

Add Redshift Pipeline example 2015-05-26 06:18:14 +00:00			`Default logging is handled by [go-kit package log](https://github.com/go-kit/kit/tree/master/log). Applications can override the default loging behaviour by implementing the [Logger interface][log_interface].`
Base pipeline components * Create base Interfaces for Pipeline * Add first base implementations for Pipeline * Add initial test for core functionality 2014-07-25 06:03:41 +00:00
User Log package for default logger * Modify the default logger to it also logs Line Numbers * Update README w/ logging instructions 2015-05-23 18:40:04 +00:00			```go
			`connector.SetLogger(NewCustomLogger())`
			```

			`[log_interface]: https://github.com/harlow/kinesis-connectors/blob/master/logger.go`

Update README.md 2015-05-23 23:18:10 +00:00			`## Contributing`
Rename License file and add Contributing sections * Make it clear from project root we are using MIT-LICENSE * Add contributing section w/ overview of how to submit code to repo 2015-05-23 15:52:08 +00:00
Add sample pipeline * Reduce the size of main README in favor of adding functioning examples * Add S3 Pipeline exmaple * Add example of seeing the stream 2015-05-23 22:22:58 +00:00			`Please see [CONTRIBUTING.md] for more information. Thank you, [contributors]!`
Rename License file and add Contributing sections * Make it clear from project root we are using MIT-LICENSE * Add contributing section w/ overview of how to submit code to repo 2015-05-23 15:52:08 +00:00
			`[LICENSE]: /MIT-LICENSE`
			`[CONTRIBUTING.md]: /CONTRIBUTING.md`

Update README.md 2015-05-23 23:18:10 +00:00			`## License`
Rename License file and add Contributing sections * Make it clear from project root we are using MIT-LICENSE * Add contributing section w/ overview of how to submit code to repo 2015-05-23 15:52:08 +00:00
			`Copyright (c) 2015 Harlow Ward. It is free software, and may`
			`be redistributed under the terms specified in the [LICENSE] file.`

			`[contributors]: https://github.com/harlow/kinesis-connectors/graphs/contributors`