60 lines
2.7 KiB
Markdown
60 lines
2.7 KiB
Markdown
# Golang Kinesis Connectors
|
|
|
|
__Kinesis connector applications written in Go__
|
|
|
|
Inspired by the [Amazon Kinesis Connector Library][1]. These components are used for extracting streaming event data
|
|
into S3, Redshift, DynamoDB, and more. See the [API Docs][2] for package documentation.
|
|
|
|
[1]: https://github.com/awslabs/amazon-kinesis-connectors
|
|
[2]: http://godoc.org/github.com/harlow/kinesis-connectors
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
Each Amazon Kinesis connector application is a pipeline that determines how records from an Amazon Kinesis stream will be handled. Records are retrieved from the stream, transformed according to a user-defined data model, buffered for batch processing, and then emitted to the appropriate AWS service.
|
|
|
|
A connector pipeline uses the following interfaces:
|
|
|
|
* __Pipeline:__ The pipeline implementation itself.
|
|
* __Transformer:__ Defines the transformation of records from the Amazon Kinesis stream in order to suit the user-defined data model. Includes methods for custom serializer/deserializers.
|
|
* __Filter:__ Defines a method for excluding irrelevant records from the processing.
|
|
* __Buffer:__ Defines a system for batching the set of records to be processed. The application can specify three thresholds: number of records, total byte count, and time. When one of these thresholds is crossed, the buffer is flushed and the data is emitted to the destination.
|
|
* __Emitter:__ Defines a method that makes client calls to other AWS services and persists the records stored in the buffer. The records can also be sent to another Amazon Kinesis stream.
|
|
|
|
### Installation
|
|
|
|
Get the package source:
|
|
|
|
$ go get github.com/harlow/kinesis-connectors
|
|
|
|
### Example Pipelines
|
|
|
|
Examples pipelines:
|
|
|
|
* [S3 Pipeline](https://github.com/harlow/kinesis-connectors/tree/master/examples/s3-pipeline)
|
|
* [Redshift Basic Pipeline](https://github.com/harlow/kinesis-connectors/tree/master/examples/redshift-pipeline)
|
|
|
|
### Logging
|
|
|
|
Default logging is handled by [go-kit package log](https://github.com/go-kit/kit/tree/master/log). Applications can override the default loging behaviour by implementing the [Logger interface][log_interface].
|
|
|
|
```go
|
|
connector.SetLogger(NewCustomLogger())
|
|
```
|
|
|
|
[log_interface]: https://github.com/harlow/kinesis-connectors/blob/master/logger.go
|
|
|
|
## Contributing
|
|
|
|
Please see [CONTRIBUTING.md] for more information. Thank you, [contributors]!
|
|
|
|
[LICENSE]: /MIT-LICENSE
|
|
[CONTRIBUTING.md]: /CONTRIBUTING.md
|
|
|
|
## License
|
|
|
|
Copyright (c) 2015 Harlow Ward. It is free software, and may
|
|
be redistributed under the terms specified in the [LICENSE] file.
|
|
|
|
[contributors]: https://github.com/harlow/kinesis-connectors/graphs/contributors
|