Golang Kinesis Connectors

Kinesis connector applications written in Go

With the release of Kinesis Firehose I'd recommend using Lambda Streams to Firehose for loading data to S3 and Redshift.

Inspired by the Amazon Kinesis Connector Library. This library is intended to be a lightweight wrapper around the Kinesis API to handle batching records, respecing ratelimits, setting checkpoints, and recovering gracefully from network errors.

Overview

The consumer expects a handler func that will process a buffer of incoming records.

func main() {
  var(
    app = flag.String("app", "", "The app name")
    stream = flag.String("stream", "", "The stream name")
  )
  flag.Parse()

  c := connector.NewConsumer(*app, *stream)
  c.Start(connector.HandlerFunc(func(b connector.Buffer) {
    fmt.Println(b.GetRecords())
    // process the records
  }))

  select {}
}

Installation

Get the package source:

$ go get github.com/harlow/kinesis-connectors

Examples

Use the seed stream code to put sample data onto the stream.

Firehose
S3

Logging

Default logging is handled by go-kit package log. Applications can override the default loging behaviour by implementing the Logger interface.

connector.SetLogger(NewCustomLogger())

Contributing

Please see CONTRIBUTING.md for more information. Thank you, contributors!

2.2 KiB Raw Blame History