f0e6461cb6
The previous pipeline model required a lot of setup and abstracted away the processing of records. By passing a HandlerFunc to the consumer we keep the business logic of processing of records closer to the use of the consumer. * Add refactoring note and SHA to README
596 B
596 B
S3 Pipeline
The S3 Connector Pipeline performs the following steps:
- Pull records from Kinesis and buffer them untill the desired threshold is met.
- Upload the batch of records to an S3 bucket.
- Set the current Shard checkpoint in Redis.
The pipleline config vars are loaded done with gcfg.
Environment Variables
Export the required environment vars for connecting to the Kinesis stream:
export AWS_ACCESS_KEY=
export AWS_REGION_NAME=
export AWS_SECRET_KEY=
Running the code
$ go run main.go -a appName -s streamName