Harlow Ward afae1bea36 Use config object for optional params

After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.

https://peter.bourgon.org/go-best-practices-2016/#configuration

2016-05-01 12:20:44 -07:00

2.7 KiB

Raw Blame History

Golang Kinesis Connectors

Kinesis connector applications written in Go

With the new release of Kinesis Firehose I'd recommend using the Lambda Streams to Firehose project for loading data directly into S3 and Redshift.

Inspired by the Amazon Kinesis Connector Library. This library is intended to be a lightweight wrapper around the Kinesis API to handle batching records, setting checkpoints, respecting ratelimits, and recovering from network errors.

Overview

The consumer expects a handler func that will process a buffer of incoming records.

func main() {
  var(
    app = flag.String("app", "", "The app name")
    stream = flag.String("stream", "", "The stream name")
  )
  flag.Parse()

  // override library defaults
  cfg := connector.Config{
    MaxBatchCount: 400,
  }

  // create new consumer
  c := connector.NewConsumer(*app, *stream, cfg)

  // process records from the stream
  c.Start(connector.HandlerFunc(func(b connector.Buffer) {
    fmt.Println(b.GetRecords())
  }))

  select {}
}

Logging

Apex Log is used to log Info and Errors from within the libarary. The default handler is "text" and can be overrideen with other LogHandlers from the the Config struct:

import(
  "github.com/apex/log"
  "github.com/apex/log/handlers/json"
)

func main() {
  // ...

  cfg := connector.Config{
    LogHandler: json.New(os.Stderr),
  }
}

Installation

Get the package source:

$ go get github.com/harlow/kinesis-connectors

Fetching Dependencies

Install gvt:

$ export GO15VENDOREXPERIMENT=1
$ go get github.com/FiloSottile/gvt

Install dependencies into ./vendor/:

$ gvt restore

Examples

Use the seed stream code to put sample data onto the stream.

Firehose
S3

Contributing

Please see CONTRIBUTING.md for more information. Thank you, contributors!

2.7 KiB Raw Blame History