25e390e8b4
* Reduce the size of main README in favor of adding functioning examples * Add S3 Pipeline exmaple * Add example of seeing the stream
25 lines
571 B
Markdown
25 lines
571 B
Markdown
# S3 Pipeline
|
|
|
|
The S3 Connector Pipeline performs the following steps:
|
|
|
|
1. Pull records from Kinesis and buffer them untill the desired threshold is met.
|
|
2. Upload the batch of records to an S3 bucket.
|
|
3. Set the current Shard checkpoint in Redis.
|
|
|
|
The pipleline config vars are loaded done with [gcfg].
|
|
|
|
[gcfg]: https://code.google.com/p/gcfg/
|
|
|
|
### Environment Variables
|
|
|
|
Export the required environment vars for connecting to the Kinesis stream:
|
|
|
|
```
|
|
export AWS_ACCESS_KEY=
|
|
export AWS_REGION_NAME=
|
|
export AWS_SECRET_KEY=
|
|
```
|
|
|
|
### Running the code
|
|
|
|
$ go run main.go
|