Golang Kinesis Connectors

Kinesis connector applications written in Go

Note: Repo is going under refactoring to use a handler func to process batch data. The previous stable version of connectors exist at SHA 509f68de89efb74aa8d79a733749208edaf56b4d

Inspired by the Amazon Kinesis Connector Library. This library is used for extracting streaming event data from Kinesis into S3, Redshift, DynamoDB, and more. See the API Docs for package documentation.

Overview

The consumer expects a handler func that will process a buffer of incoming records.

func main() {
  var(
    app = flag.String("app", "", "The app name")
    stream = flag.String("stream", "", "The stream name")
  )
  flag.Parse()

  c := connector.NewConsumer(*app, *stream)
  c.Start(connector.HandlerFunc(func(b connector.Buffer) {
    fmt.Println(b.GetRecords())
    // process the records
  }))

  select {}
}

Installation

Get the package source:

$ go get github.com/harlow/kinesis-connectors

Example Pipelines

Examples pipelines:

• S3
• Redshift

Logging

Default logging is handled by go-kit package log. Applications can override the default loging behaviour by implementing the Logger interface.

connector.SetLogger(NewCustomLogger())

Contributing

Please see CONTRIBUTING.md for more information. Thank you, contributors!

License

Copyright (c) 2015 Harlow Ward. It is free software, and may be redistributed under the terms specified in the LICENSE file.