History

Taylor Sutton f8e9c34641 Refactor zlib decompression into the splitter package Being in the batchconsumer package means it will work for anything using KCL, but lambdas that subscribe to these log streams do not use batchconsumer at all; instead they invoke the splitter package directly. As such, if we want this functionality to be available to lambda log consumers, it can't be in batchconsumer. There are no functionality changes here, just moving code from an unexported method in one place to an exported function in another place. The tests also get moved along with it.		2020-11-12 11:37:13 -05:00
..
README.md	Implemented new consumer interface as well as example consumer: 'batchconsumer'	2017-07-18 02:03:15 +00:00
splitter.go	Refactor zlib decompression into the splitter package	2020-11-12 11:37:13 -05:00
splitter_test.go	Refactor zlib decompression into the splitter package	2020-11-12 11:37:13 -05:00

README.md

splitter

This splitter allows ingesting logs from a CWLogs subscription.

Splitter's expected input is batched logs from a CloudWatchLogs subscription to a Kinesis Stream. The CWLogs subscription has a special format which bundles several logs into a single record. The splitter takes this record and splits it into multiple logs. These logs are also modified to mimic the RSyslog format we expect from our other logs. This allows them to be decoded normally by the rest of the pipeline.