Taylor Sutton
f8e9c34641
Refactor zlib decompression into the splitter package
...
Being in the batchconsumer package means it will work for anything
using KCL, but lambdas that subscribe to these log streams do not use
batchconsumer at all; instead they invoke the splitter package
directly. As such, if we want this functionality to be available to
lambda log consumers, it can't be in batchconsumer.
There are no functionality changes here, just moving code from an
unexported method in one place to an exported function in another
place. The tests also get moved along with it.
2020-11-12 11:37:13 -05:00
Taylor Sutton
4fe27d0d39
Clarifying comment - Logs from CW Logs are gzipped.
2020-11-11 12:39:51 -05:00
Taylor Sutton
6b2d1f8a56
Add zlib decompression to batch consumer.
...
This has become relevant as zlib is the compression method supported
by the Kinesis plugin for Fluent Bit.
2020-11-11 09:12:05 -05:00
Gavi Hirsch
1112894639
clear count stats after logging them
2020-09-11 14:04:42 -07:00
Xavi Ramirez
e80ff2239b
Added error message for failed log written to file
2018-12-11 16:32:33 -08:00
Xavi Ramirez
e5850f1464
Added Initialize method to sender
2018-08-09 16:43:30 -07:00
Rafael Garcia
c255bf87f3
add test for default case
2018-01-25 15:13:11 +00:00
Rafael Garcia
7b18d1bab4
write failed log message to file
2017-11-03 18:03:39 +00:00
Rafael Garcia
945ed317c2
remove logger as parameter
2017-11-03 17:48:50 +00:00
Rafael Garcia
221743b3e2
write errors to stderr
2017-11-02 21:50:46 +00:00
mohit
22495acfe4
derpfix: s/black/blank
2017-09-13 19:22:58 +00:00
mohit
5b91cd80da
DefaultCounters to log 0 for counter stats
...
this makes sure that we can get metrics when things are not flowing
through one of the consumers
2017-09-12 04:33:05 +00:00
mohit
8b44537934
fix typo in error log
2017-09-11 20:22:34 +00:00
Xavi Ramirez
f581bb4f6e
Increased default read rate.
2017-08-22 18:37:47 +00:00
Xavi Ramirez
8f7cbf9952
Fixed case checkpoints would only occur on shutdown. The archive consumer fell victim of this bug.
2017-08-20 23:41:28 +00:00
Nathan Leiby
6039f69d86
(lint) go vet fixes
2017-08-10 14:28:13 -07:00
Nathan Leiby
5b8a86212d
remove DeployEnv (no longer needed)
2017-08-10 14:26:27 -07:00
Xavi Ramirez
55aeecddd7
Renamed IsEmpty to IsNil on SequencePair
2017-08-10 20:16:41 +00:00
Xavi Ramirez
45fad863d0
Made config object that's specific to BatcherManager
2017-08-10 20:11:24 +00:00
Xavi Ramirez
3a04dad4bf
Send single value instead of full config object to CheckpointManager
2017-08-10 20:01:07 +00:00
Xavi Ramirez
6102f9c752
Added comment to why a channel is taking a channel.
2017-08-10 19:56:17 +00:00
Xavi Ramirez
68a16cfe6a
Added an external go routine to ensure stale batches get flushed
2017-08-10 19:47:08 +00:00
Xavi Ramirez
27cce44140
Made constructors of private structs private
2017-08-10 19:21:46 +00:00
Xavi Ramirez
1632c81fc7
Use errors.New instead of fmt.Errorf
2017-08-10 19:20:55 +00:00
Xavi Ramirez
b7743c9ea7
Added a more coherent shutdown pathway
2017-08-09 01:22:11 +00:00
Xavi Ramirez
eb230b94f7
Added stats to base kinesis client
2017-08-07 19:20:56 +00:00
Xavi Ramirez
5c373fa7d8
Fixed unit tests
2017-08-07 19:20:56 +00:00
Xavi Ramirez
734b46274f
Better coordinated checkpoints and readlines. Checkpoint function now returns all errors.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
6a30e0eb8a
Huge refactor. Batchers no longer run in their own go-routines, which allowed us to a lot of concurrency and simplify model.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
ba951ff0da
Refactor to fix dead locks and race conditions.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
29f68f77eb
Removed redundant retry logic
2017-08-07 19:20:56 +00:00
Xavi Ramirez
c6fe4cef37
Renamed variable from seq to pair
2017-08-07 19:20:56 +00:00
Xavi Ramirez
6e9457cbcf
Better propogated shutdown signal
2017-08-07 19:20:56 +00:00
Xavi Ramirez
7a7716e824
Increased default read rate limit
2017-08-07 19:20:56 +00:00
Xavi Ramirez
04042290f5
Moved SequencePair to kcl package
2017-08-07 19:20:56 +00:00
Xavi Ramirez
212ec45d88
Added writter unit tests and fixed a slew of concurrency bugs
2017-07-21 01:35:54 +00:00
Xavi Ramirez
939fc5406f
Batcher now returns error if bad config is provided
2017-07-21 01:33:59 +00:00
Xavi Ramirez
ae37b57e61
Added IsEmpty to SequencePair
2017-07-21 01:25:02 +00:00
Xavi Ramirez
164b9923be
Reworded docs to remove references to firehose. Renamed config variables to be clearer.
2017-07-21 01:25:02 +00:00
Xavi Ramirez
e8f40e607a
Added unit test for SequencePair IsLessThan function
2017-07-19 19:00:28 +00:00
Xavi Ramirez
1b2afcafc0
Renamed methods to be more generic and to remove reference to logs
2017-07-19 00:21:31 +00:00
Xavi Ramirez
f116c752f5
Added unit test to ensure correct error is returned
2017-07-19 00:20:20 +00:00
Xavi Ramirez
906f6b94fc
Made ReadBurstLimit relative to read rate limit
2017-07-18 19:53:25 +00:00
Xavi Ramirez
3f42cb5e4a
Created IsLessThan method to SequencePair to make code more readable
2017-07-18 19:52:26 +00:00
Xavi Ramirez
8d273d6a1e
Fixed and added unit tests
2017-07-18 19:19:40 +00:00
Xavi Ramirez
a329c40645
Fixed race condition
2017-07-18 19:13:39 +00:00
Xavi Ramirez
8bcef6cd91
Log file closed when kclProcessor is done running
2017-07-18 19:12:46 +00:00
Xavi Ramirez
4c67f39c99
Implemented new consumer interface as well as example consumer: 'batchconsumer'
2017-07-18 02:03:15 +00:00
