Harlow Ward f0e6461cb6 Refactor to use handler func

The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.

* Add refactoring note and SHA to README

2016-02-06 17:50:17 -08:00

2.2 KiB

Raw Blame History

Golang Kinesis Connectors

Kinesis connector applications written in Go

Note: Repo is going under refactoring to use a handler func to process batch data. The previous stable version of connectors exist at SHA 509f68de89efb74aa8d79a733749208edaf56b4d

Inspired by the Amazon Kinesis Connector Library. This library is used for extracting streaming event data from Kinesis into S3, Redshift, DynamoDB, and more. See the API Docs for package documentation.

Overview

The consumer expects a handler func that will process a buffer of incoming records.

func main() {
  var(
    app = flag.String("app", "", "The app name")
    stream = flag.String("stream", "", "The stream name")
  )
  flag.Parse()

  c := connector.NewConsumer(*app, *stream)
  c.Start(connector.HandlerFunc(func(b connector.Buffer) {
    fmt.Println(b.GetRecords())
    // process the records
  }))

  select {}
}

Installation

Get the package source:

$ go get github.com/harlow/kinesis-connectors

Example Pipelines

Examples pipelines:

S3
Redshift

Logging

Default logging is handled by go-kit package log. Applications can override the default loging behaviour by implementing the Logger interface.

connector.SetLogger(NewCustomLogger())

Contributing

Please see CONTRIBUTING.md for more information. Thank you, contributors!

2.2 KiB Raw Blame History