Commit graph

15 commits

Author SHA1 Message Date
Edward Tsang
a6eaeb2bfa need to add context as first parameter to pass through, fixed tests and readme 2018-11-07 15:45:13 -08:00
Vincent
cb35697903 Write unit tests for Postgres package (#69)
Changes:
* I add postgres_databaseutils_test.go for containing all utilities for mocking the database.
* I put appName as a required parameter passed to the New() since it is needed to form the namespace column name.
* I add GetMaxInterval() since it might be needed by the consumer and the test.
* I move the SQL string package variables directly into the function that need it so we could avoid maintenance nightmare in the future.
2018-10-14 11:23:37 -06:00
Emanuel Ramos
cf5d22abff Adding Option func to set maxInterval (#60) 2018-07-13 07:26:21 -07:00
Emanuel Ramos
b7be26418a Add postgres checkpoint implementation (#55) 2018-06-17 19:27:10 -07:00
Prometheus
d058203b6e Make what aws error to trigger retry decided by caller (#52)
* remove ValidateCheckpoint
* make retrying on error decided by caller
2018-06-04 20:07:58 -07:00
Prometheus
9a7e102a05 remove ValidateCheckpoint (#51) 2018-06-01 19:34:46 -07:00
Prometheus
992cc42419 DDB uses default AWS config settings to ping table; won't work with WithDyanmoClient. Misc update on example and README (#50) 2018-06-01 16:14:42 -07:00
Prometheus
9e0a97916d Use AWS resource iface, overwrite default dynamodb, more explicit in example about overwrite default AWS resrouce client (#49)
* use custom kinesis client

* use aws sdk interface, add missing api for ddb

* add overwrite default dynamodbclien usage
2018-05-31 17:41:14 -07:00
Harlow Ward
64cdf69249
Add interval flush for DDB checkpoint (#40)
Add interval flush for DDB checkpoint

* Allow checkpointing on a specified interval
* Add shutdown method to checkpoint to force flush

Minor changes:

* Swap order of input params for checkpoint (app, table)

Addresses: https://github.com/harlow/kinesis-consumer/issues/39
2017-12-30 20:21:10 -08:00
Harlow Ward
b0245d688b Add more test coverage for Redis Checkpoint 2017-11-22 21:28:39 -08:00
Harlow Ward
6401371727 Simplify checkpoint interface; reduce input vars 2017-11-22 20:01:31 -08:00
Harlow Ward
3770136f64 Allow user to override no-op checkpoint with Option 2017-11-22 17:44:42 -08:00
Harlow Ward
4d6a85e901 Make the Checkpoint a required input for Consumer
The Checkpoint functionality is an important part of the library and
previously it wasn't obvious that the Consumer was defaulting to Redis
for this functionality.

* Add Checkpoint as required param for new consumer
2017-11-21 08:58:16 -08:00
Harlow Ward
6ee965ec0a
Add DDB as consumer checkpoint option (#37)
* Simplify the checkpoint interface
* Add DDB backend for checkpoint persistence

Implements: https://github.com/harlow/kinesis-consumer/issues/26
2017-11-20 09:37:30 -08:00
Harlow Ward
130c78456c
Simplify the consumer experience (#35)
Major changes:

* Remove intermediate batching of kinesis records
* Call the callback func with each record
* Use functional options for config 

https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

Minor changes:

* update README messaging about Kinesis -> Firehose functionality
* remove unused buffer and emitter code
2017-11-20 08:21:40 -08:00