Taylor Sutton
f8e9c34641
Refactor zlib decompression into the splitter package
...
Being in the batchconsumer package means it will work for anything
using KCL, but lambdas that subscribe to these log streams do not use
batchconsumer at all; instead they invoke the splitter package
directly. As such, if we want this functionality to be available to
lambda log consumers, it can't be in batchconsumer.
There are no functionality changes here, just moving code from an
unexported method in one place to an exported function in another
place. The tests also get moved along with it.
2020-11-12 11:37:13 -05:00
Taylor Sutton
4fe27d0d39
Clarifying comment - Logs from CW Logs are gzipped.
2020-11-11 12:39:51 -05:00
Taylor Sutton
6b2d1f8a56
Add zlib decompression to batch consumer.
...
This has become relevant as zlib is the compression method supported
by the Kinesis plugin for Fluent Bit.
2020-11-11 09:12:05 -05:00
Xavi Ramirez
e5850f1464
Added Initialize method to sender
2018-08-09 16:43:30 -07:00
Rafael Garcia
c255bf87f3
add test for default case
2018-01-25 15:13:11 +00:00
Rafael Garcia
945ed317c2
remove logger as parameter
2017-11-03 17:48:50 +00:00
Rafael Garcia
221743b3e2
write errors to stderr
2017-11-02 21:50:46 +00:00
Nathan Leiby
6039f69d86
(lint) go vet fixes
2017-08-10 14:28:13 -07:00
Nathan Leiby
5b8a86212d
remove DeployEnv (no longer needed)
2017-08-10 14:26:27 -07:00
Xavi Ramirez
55aeecddd7
Renamed IsEmpty to IsNil on SequencePair
2017-08-10 20:16:41 +00:00
Xavi Ramirez
45fad863d0
Made config object that's specific to BatcherManager
2017-08-10 20:11:24 +00:00
Xavi Ramirez
3a04dad4bf
Send single value instead of full config object to CheckpointManager
2017-08-10 20:01:07 +00:00
Xavi Ramirez
27cce44140
Made constructors of private structs private
2017-08-10 19:21:46 +00:00
Xavi Ramirez
b7743c9ea7
Added a more coherent shutdown pathway
2017-08-09 01:22:11 +00:00
Xavi Ramirez
eb230b94f7
Added stats to base kinesis client
2017-08-07 19:20:56 +00:00
Xavi Ramirez
6a30e0eb8a
Huge refactor. Batchers no longer run in their own go-routines, which allowed us to a lot of concurrency and simplify model.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
ba951ff0da
Refactor to fix dead locks and race conditions.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
29f68f77eb
Removed redundant retry logic
2017-08-07 19:20:56 +00:00
Xavi Ramirez
c6fe4cef37
Renamed variable from seq to pair
2017-08-07 19:20:56 +00:00
Xavi Ramirez
6e9457cbcf
Better propogated shutdown signal
2017-08-07 19:20:56 +00:00
Xavi Ramirez
04042290f5
Moved SequencePair to kcl package
2017-08-07 19:20:56 +00:00
Xavi Ramirez
212ec45d88
Added writter unit tests and fixed a slew of concurrency bugs
2017-07-21 01:35:54 +00:00
Xavi Ramirez
939fc5406f
Batcher now returns error if bad config is provided
2017-07-21 01:33:59 +00:00
Xavi Ramirez
164b9923be
Reworded docs to remove references to firehose. Renamed config variables to be clearer.
2017-07-21 01:25:02 +00:00
Xavi Ramirez
1b2afcafc0
Renamed methods to be more generic and to remove reference to logs
2017-07-19 00:21:31 +00:00
Xavi Ramirez
3f42cb5e4a
Created IsLessThan method to SequencePair to make code more readable
2017-07-18 19:52:26 +00:00
Xavi Ramirez
8d273d6a1e
Fixed and added unit tests
2017-07-18 19:19:40 +00:00
Xavi Ramirez
4c67f39c99
Implemented new consumer interface as well as example consumer: 'batchconsumer'
2017-07-18 02:03:15 +00:00
