luciano/amazon-kinesis-client-go
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
amazon-kinesis-client-go/splitter
History
Taylor Sutton 6e8a99c50d Add functionality for handling KPL aggregates and zlib 
Adds a lowest-common-denominator function, KPLDeaggregate, for
handling records that might be KPL aggregated. Also adds a function,
DeaggregateAndSplitIfNecessary, to wrap the existing functionality of
SplitMessageIfNecessary with KPL deaggreation.

These functions are handy for non-KCL consumers, like Lambda
functions. KCL automatically applies deaggreation for you.

This change is backwards compatible - the previously exposed function
SplitMessageIfNecessary still does the same things.
2020-11-16 11:01:28 -05:00
..
README.md Implemented new consumer interface as well as example consumer: 'batchconsumer' 2017-07-18 02:03:15 +00:00
splitter.go Add functionality for handling KPL aggregates and zlib 2020-11-16 11:01:28 -05:00
splitter_test.go Add functionality for handling KPL aggregates and zlib 2020-11-16 11:01:28 -05:00

README.md

splitter

This splitter allows ingesting logs from a CWLogs subscription.

Splitter's expected input is batched logs from a CloudWatchLogs subscription to a Kinesis Stream. The CWLogs subscription has a special format which bundles several logs into a single record. The splitter takes this record and splits it into multiple logs. These logs are also modified to mimic the RSyslog format we expect from our other logs. This allows them to be decoded normally by the rest of the pipeline.