luciano/amazon-kinesis-client-go
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
204 commits 11 branches 1 tag 8.2 MiB
Commit graph

204 commits

This branch
This branch
All branches
Author SHA1 Message Date
Xavi Ramirez
1be812a887 Added less hacky way of downloading jars 2017-08-07 19:22:10 +00:00
Xavi Ramirez
de04a27799 Return NonKayveeError if log contians invalid or empty json 2017-08-07 19:20:56 +00:00
Xavi Ramirez
eb230b94f7 Added stats to base kinesis client 2017-08-07 19:20:56 +00:00
Xavi Ramirez
5c373fa7d8 Fixed unit tests 2017-08-07 19:20:56 +00:00
Xavi Ramirez
4809cdb4e6 Another refactor to simply and likely speed up code. New code allows checkpoint and readline messages to be interleaved. 2017-08-07 19:20:56 +00:00
Xavi Ramirez
1bff01ff4f Removed unused params and properties 2017-08-07 19:20:56 +00:00
Xavi Ramirez
734b46274f Better coordinated checkpoints and readlines. Checkpoint function now returns all errors. 2017-08-07 19:20:56 +00:00
Xavi Ramirez
b19c9297d8 Adjusted some locks 2017-08-07 19:20:56 +00:00
Xavi Ramirez
c814742afa Another refactor 2017-08-07 19:20:56 +00:00
Xavi Ramirez
c5f75d6554 Fixed another race-condition. Ensure line is completely processed before allowing a checkpoint. 2017-08-07 19:20:56 +00:00
Xavi Ramirez
6a30e0eb8a Huge refactor. Batchers no longer run in their own go-routines, which allowed us to a lot of concurrency and simplify model. 2017-08-07 19:20:56 +00:00
Xavi Ramirez
ba951ff0da Refactor to fix dead locks and race conditions. 2017-08-07 19:20:56 +00:00
Xavi Ramirez
29f68f77eb Removed redundant retry logic 2017-08-07 19:20:56 +00:00
Xavi Ramirez
c6fe4cef37 Renamed variable from seq to pair 2017-08-07 19:20:56 +00:00
Xavi Ramirez
6e9457cbcf Better propogated shutdown signal 2017-08-07 19:20:56 +00:00
Xavi Ramirez
873544ae78 Run benchmarks when building 2017-08-07 19:20:56 +00:00
Xavi Ramirez
266d7d620d Fixed bug in decoder package 2017-08-07 19:20:56 +00:00
Xavi Ramirez
7a7716e824 Increased default read rate limit 2017-08-07 19:20:56 +00:00
Xavi Ramirez
04042290f5 Moved SequencePair to kcl package 2017-08-07 19:20:56 +00:00
Xavi
4c6d8453ef Merge pull request #5 from Clever/INFRA-2405-new-consumer-interface 
Implemented new consumer interface as well as example consumer
 2017-07-24 13:12:43 -07:00
Xavi Ramirez
212ec45d88 Added writter unit tests and fixed a slew of concurrency bugs 2017-07-21 01:35:54 +00:00
Xavi Ramirez
939fc5406f Batcher now returns error if bad config is provided 2017-07-21 01:33:59 +00:00
Xavi Ramirez
0cb14948a3 KCL more aggressively logs errors 2017-07-21 01:27:17 +00:00
Xavi Ramirez
ae37b57e61 Added IsEmpty to SequencePair 2017-07-21 01:25:02 +00:00
Xavi Ramirez
b0f769bfa7 Breaking change to KCL. Created Checkpointer interface to make testing easier. 2017-07-21 01:25:02 +00:00
Xavi Ramirez
164b9923be Reworded docs to remove references to firehose. Renamed config variables to be clearer. 2017-07-21 01:25:02 +00:00
Xavi Ramirez
e8f40e607a Added unit test for SequencePair IsLessThan function 2017-07-19 19:00:28 +00:00
Xavi Ramirez
1b2afcafc0 Renamed methods to be more generic and to remove reference to logs 2017-07-19 00:21:31 +00:00
Xavi Ramirez
f116c752f5 Added unit test to ensure correct error is returned 2017-07-19 00:20:20 +00:00
Xavi Ramirez
906f6b94fc Made ReadBurstLimit relative to read rate limit 2017-07-18 19:53:25 +00:00
Xavi Ramirez
3f42cb5e4a Created IsLessThan method to SequencePair to make code more readable 2017-07-18 19:52:26 +00:00
Xavi Ramirez
6d4df426a9 Added install deps step to circle yaml 2017-07-18 19:31:06 +00:00
Xavi Ramirez
8d273d6a1e Fixed and added unit tests 2017-07-18 19:19:40 +00:00
Xavi Ramirez
a329c40645 Fixed race condition 2017-07-18 19:13:39 +00:00
Xavi Ramirez
8bcef6cd91 Log file closed when kclProcessor is done running 2017-07-18 19:12:46 +00:00
Xavi Ramirez
4c67f39c99 Implemented new consumer interface as well as example consumer: 'batchconsumer' 2017-07-18 02:03:15 +00:00
Nathan Leiby
aef51b0801 Merge pull request #4 from Clever/use-golang-1.8 
Go 1.7 -> 1.8
 2017-05-25 15:29:33 -07:00
Nathan Leiby
bb9ebbca1d Use Go 1.8 in Circle build and golang-version-check 2017-05-24 12:40:55 -07:00
Xavi
92eaacfbf1 Merge pull request #3 from Clever/made-checkpointer-cacheable 
Consumers of this library can now cache the checkpointer object
 2017-05-22 16:49:23 -07:00
Xavi Ramirez
458e66e321 Added shutdown method to *Checkerpointer struct 2017-05-22 23:06:56 +00:00
Xavi Ramirez
5f507ab116 Moved retry and error handling logic to Checkpointer class 2017-05-22 23:06:16 +00:00
Xavi Ramirez
02b053c0ee Added vendor to .gitignore 2017-05-22 22:57:34 +00:00
Xavi Ramirez
3c56b57e6b Added mutex to Checkpoint function to avoid racey conditions 2017-05-19 22:30:11 +00:00
Xavi Ramirez
17cfe98efa Users of this library can now cache the checkpointer object 2017-05-19 22:01:57 +00:00
Rafael
8da04c944f Merge pull request #2 from Clever/shutdown-checkpoint 
allow for empty checkpoints on shutdown
 2017-05-17 17:52:42 -07:00
Rafael Garcia
bea2abc691 allow for empty checkpoints on shutdown 2017-05-18 00:46:29 +00:00
Nathan Leiby
695f85d99d Merge pull request #1 from Clever/checkpointing-fix 
checkpointing fix
 2017-04-05 15:05:39 -07:00
Nathan Leiby
e6229ce8bb checkpointing fix 
- always try checkpoint at least once (previously, a value of 0 retries
  meant it would never checkpoint)
- setup record processor using New(), so we get default checkpointing
  configuration
 2017-04-05 13:51:45 -07:00
Rafael Garcia
47eec92a1a more circle tweaks 2017-02-08 20:38:14 +00:00
Rafael Garcia
cd0fa0ab62 circle tweaks 2017-02-08 20:35:07 +00:00