luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
145 commits 6 branches 9 tags 7.5 MiB
Commit graph

47 commits

Author SHA1 Message Date
Edward Tsang
63829e7e07 remove double tagging f aws requestID 2018-11-14 10:35:07 -08:00
Edward Tsang
91787fac0e add aws req retry count. move tagging requestId to complete handler. some aws services fill that in at that stage. tag response content length. req throttled. 2018-11-14 10:22:59 -08:00
Edward Tsang
5ff0624a2d add back commented out code 2018-11-12 09:01:16 -08:00
Edward Tsang
a6eaeb2bfa need to add context as first parameter to pass through, fixed tests and readme 2018-11-07 15:45:13 -08:00
Edward Tsang
0528067bbe need to use AWS call ***WithContext to flow the context object from upper layer 2018-11-07 10:29:17 -08:00
Edward Tsang
43cf094b27 * instrumented consumer amd producer example with opentracing 
* add doc for jaeger and steps for running opentracing examples
* add context appropriate
* add utility for AWS tracing .. Createing tracer/logger etc
 2018-11-06 15:03:34 -08:00
Harlow Ward
fb98fbe244
Remove the client wrapper (#58) 
Having an additional Client has added some confusion (https://github.com/harlow/kinesis-consumer/issues/45) on how to provide a
custom kinesis client. Allowing `WithClient` to accept a Kinesis client
it cleans up the interface.

Major changes:

* Remove the Client wrapper; prefer using kinesis client directly
* Change `ScanError` to `ScanStatus` as the return value isn't necessarily an error

Note: these are breaking changes, if you need last stable release please see here: https://github.com/harlow/kinesis-consumer/releases/tag/v0.2.0
 2018-07-28 22:53:33 -07:00
Emanuel Ramos
b7be26418a Add postgres checkpoint implementation (#55) 2018-06-17 19:27:10 -07:00
Prometheus
739e9e39a5 Make it possible to let user use 3rd party logging library (#56) 2018-06-12 18:07:33 -07:00
Prometheus
e6a489c76b Scanerror signals the consumer if we should continue scanning for next record & whether to checkpoint. (#54) 
* remove ValidateCheckpoint

* update for checkpoint can not customize retryer

* implement the scan error as in PR 44

* at least log if record processor has error

* mistakenly removed this line

* propage error up. ignore invalid state
 2018-06-08 08:40:42 -07:00
Prometheus
d058203b6e Make what aws error to trigger retry decided by caller (#52) 
* remove ValidateCheckpoint
* make retrying on error decided by caller
 2018-06-04 20:07:58 -07:00
Prometheus
9a7e102a05 remove ValidateCheckpoint (#51) 2018-06-01 19:34:46 -07:00
Prometheus
992cc42419 DDB uses default AWS config settings to ping table; won't work with WithDyanmoClient. Misc update on example and README (#50) 2018-06-01 16:14:42 -07:00
Prometheus
9e0a97916d Use AWS resource iface, overwrite default dynamodb, more explicit in example about overwrite default AWS resrouce client (#49) 
* use custom kinesis client

* use aws sdk interface, add missing api for ddb

* add overwrite default dynamodbclien usage
 2018-05-31 17:41:14 -07:00
Anant Prakash
2a5856ec99 Correct AWS_REGION (#48) 2018-05-30 06:45:38 -07:00
Harlow Ward
64cdf69249
Add interval flush for DDB checkpoint (#40) 
Add interval flush for DDB checkpoint

* Allow checkpointing on a specified interval
* Add shutdown method to checkpoint to force flush

Minor changes:

* Swap order of input params for checkpoint (app, table)

Addresses: https://github.com/harlow/kinesis-consumer/issues/39
 2017-12-30 20:21:10 -08:00
Harlow Ward
058f383e30 Add cancellation of pipeline from signal interrupts 2017-11-26 16:00:03 -08:00
Harlow Ward
89570130f5 Leverage bigger batchsize when seeding example data 2017-11-26 15:59:17 -08:00
Harlow Ward
edf0467eb0 Format errors from caller 2017-11-23 11:29:58 -08:00
Harlow Ward
86f1df782e Return the shard scan errors to top-level caller 2017-11-23 08:49:37 -08:00
Harlow Ward
6401371727 Simplify checkpoint interface; reduce input vars 2017-11-22 20:01:31 -08:00
Harlow Ward
4ffe3ec55a Add logs for start scan and checkpoints 2017-11-22 17:52:41 -08:00
Harlow Ward
3770136f64 Allow user to override no-op checkpoint with Option 2017-11-22 17:44:42 -08:00
Harlow Ward
c91f6233ef Add counter for exposing scanner metrics 2017-11-22 14:10:11 -08:00
Harlow Ward
90d2903fe6 Use stdlib logging, default to discard 2017-11-22 10:46:39 -08:00
Harlow Ward
4d6a85e901 Make the Checkpoint a required input for Consumer 
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.

* Add Checkpoint as required param for new consumer
 2017-11-21 08:58:16 -08:00
Harlow Ward
8d2cc5bc20 Return error from scan instead of terminating the program 2017-11-20 11:45:41 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option (#37) 
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence

Implements: https://github.com/harlow/kinesis-consumer/issues/26
 2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience (#35) 
Major changes:

* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config 

https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

Minor changes:

* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
 2017-11-20 08:21:40 -08:00
Harlow Ward
ceca88b96a Add required fields to Config 
Taking some inspiration from:
https://github.com/tj/go-kinesis/blob/master/kinesis.go#L50-L75
 2016-05-07 18:10:31 -07:00
Harlow Ward
f4c8d8090d Use TJ's Kinesis producer library 2016-05-01 16:54:47 -07:00
Harlow Ward
a12c15a191 Use no-op Info logging by default 2016-05-01 12:40:30 -07:00
Harlow Ward
afae1bea36 Use config object for optional params 
After reading notes from Peter's talk I like the idea of using a config
object where consumers of the library can override the defaults.

https://peter.bourgon.org/go-best-practices-2016/#configuration
 2016-05-01 12:20:44 -07:00
Harlow Ward
3aa0f72efe Add logging when records are emitted w/ record count 2016-05-01 10:43:42 -07:00
Harlow Ward
d5bdd3f4bc Leverage AWS S3 retries 2016-05-01 10:42:28 -07:00
Harlow Ward
7fccc9da9f Update function name to represent functionality 2016-04-30 09:14:25 -07:00
Harlow Ward
c29698550f Add config options to Consumer 
The Firehose service can take a max batch size of 500. While created the
example the need for finer grained configuration was necessary.
 2016-02-09 22:31:15 -08:00
Harlow Ward
aae268108e Move emitters to separate packages 2016-02-08 19:39:09 -08:00
Harlow Ward
caa2fc0b0a Add Kinesis Firehose example 2016-02-08 13:21:54 -08:00
Harlow Ward
f0e6461cb6 Refactor to use handler func 
The previous pipeline model required a lot of setup and abstracted away
the processing of records. By passing a HandlerFunc to the consumer we
keep the business logic of processing of records closer to the use of
the consumer.

* Add refactoring note and SHA to README
 2016-02-06 17:50:17 -08:00
Harlow Ward
509f68de89 Add error checking for tmp file 2016-01-09 09:52:47 -08:00
Harlow Ward
d542fa996f Use AWS SDK and concurrency 2016-01-09 09:46:28 -08:00
Harlow Ward
0d5e9b7b02 Use AWS SDK for S3 Emitter 
* Handle retries for S3 Emitter
* Update Pipeline to use AWS backoff
* Leverage aws error kit in recoverable error helpers
 2015-08-16 17:52:10 -07:00
Harlow Ward
18173842fb Use AWS SDK 
Limit the amount of library dependencies by using the official AWS SDK.

https://github.com/harlow/kinesis-connectors/issues/19
 2015-08-15 23:17:49 -07:00
Harlow Ward
3d9e6e2485 Add Redshift Pipeline example 2015-05-25 23:18:14 -07:00
Harlow Ward
cd54569c61 Use go-kit Log interface 2015-05-25 22:40:40 -07:00
Harlow Ward
25e390e8b4 Add sample pipeline 
* Reduce the size of main README in favor of adding functioning examples
* Add S3 Pipeline exmaple
* Add example of seeing the stream
 2015-05-23 15:27:40 -07:00