f921eca908
By specifying a `jsonpaths` parameter to an S3 file containing ordinal attribute position we can store free-form JSON in S3 and then reference the col order from the COPY command. http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html
16 lines
626 B
Go
16 lines
626 B
Go
package connector
|
|
|
|
// Buffer defines a buffer used to store records streamed through Kinesis. It is a part of the
|
|
// Pipeline utilized by the Pipeline.ProcessShard function. Records are stored in the buffer by calling
|
|
// the Add method. The buffer has two size limits defined: total total number of records and a
|
|
// time limit in seconds. The ShouldFlush() method may indicate that the buffer is full based on
|
|
// these limits.
|
|
type Buffer interface {
|
|
Add(data Model, sequenceNumber string)
|
|
FirstSequenceNumber() string
|
|
Flush()
|
|
LastSequenceNumber() string
|
|
NumRecordsInBuffer() int
|
|
Records() []Model
|
|
ShouldFlush() bool
|
|
}
|