25e390e8b4
* Reduce the size of main README in favor of adding functioning examples * Add S3 Pipeline exmaple * Add example of seeing the stream |
||
|---|---|---|
| .. | ||
| main.go | ||
| pipeline.cfg | ||
| README.md | ||
S3 Pipeline
The S3 Connector Pipeline performs the following steps:
- Pull records from Kinesis and buffer them untill the desired threshold is met.
- Upload the batch of records to an S3 bucket.
- Set the current Shard checkpoint in Redis.
The pipleline config vars are loaded done with gcfg.
Environment Variables
Export the required environment vars for connecting to the Kinesis stream:
export AWS_ACCESS_KEY=
export AWS_REGION_NAME=
export AWS_SECRET_KEY=
Running the code
$ go run main.go