Golang Kinesis Consumer

Kinesis consumer applications written in Go

With the new release of Kinesis Firehose I'd recommend using the kinesis to firehose functionality for writing data directly to S3, Redshift, or Elasticsearch.

Installation

Get the package source:

$ go get github.com/harlow/kinesis-consumer

Overview

The consumer leverages a handler func that accepts a Kinesis record. The Scan method will consume all shards concurrently and call the callback func as it receives records from the stream.

import consumer "github.com/harlow/kinesis-consumer"

func main() {
	log.SetHandler(text.New(os.Stderr))
	log.SetLevel(log.DebugLevel)

	var (
		app    = flag.String("app", "", "App name") // name of consumer group
		stream = flag.String("stream", "", "Stream name")
	)
	flag.Parse()

	c, err := consumer.New(*app, *stream)
	if err != nil {
		log.Fatalf("new consumer error: %v", err)
	}

	c.Scan(context.TODO(), func(r *kinesis.Record) bool {
		fmt.Println(string(r.Data))

		return true // continue scanning
	})
}

Note: If you need to aggregate based on a specific shard the ScanShard method should be leverged instead.

Configuration

The consumer requires the following config:

• App Name (used for checkpoints)
• Stream Name (kinesis stream name)

It also accepts the following optional overrides:

• Kinesis Client
• Logger
• Checkpoint

svc := kinesis.New(session.New(aws.NewConfig()))

c, err := consumer.New(
	appName, 
	streamName,
	consumer.WithClient(svc),
)

Checkpoint

The default checkpoint uses Redis on localhost; to set a custom Redis URL use ENV vars:

REDIS_URL=redis.example.com:6379

• Add DDB as a checkpoint option

Logging

Apex Log is used for logging Info. Override the logs format with other Log Handlers. For example using the "json" log handler:

import(
  "github.com/apex/log"
  "github.com/apex/log/handlers/json"
)

func main() {
  // ...

  log.SetHandler(json.New(os.Stderr))
  log.SetLevel(log.DebugLevel)
}

Which will producde the following logs:

  INFO[0000] processing                app=test shard=shardId-000000000000 stream=test
  INFO[0008] checkpoint                app=test shard=shardId-000000000000 stream=test
  INFO[0012] checkpoint                app=test shard=shardId-000000000000 stream=test

Contributing

Please see CONTRIBUTING.md for more information. Thank you, contributors!

License

Copyright (c) 2015 Harlow Ward. It is free software, and may be redistributed under the terms specified in the LICENSE file.

www.hward.com  ·  GitHub @harlow  ·  Twitter @harlow_ward