History

Nathan Leiby 0c02e93821 splitter: don't filter by env Filtering by env was an artifact of the previous way we originally had setup our CWLogs subscriptions, assuming that we could have multiple subscriptions per stream. :( You can't. Let's instead allow any CWLogs subscription's logs to be valid, which is needed for: https://github.com/Clever/kinesis-cwlogs-splitter/pull/1		2017-08-10 14:23:03 -07:00
..
README.md	Implemented new consumer interface as well as example consumer: 'batchconsumer'	2017-07-18 02:03:15 +00:00
splitter.go	splitter: don't filter by env	2017-08-10 14:23:03 -07:00
splitter_test.go	splitter: don't filter by env	2017-08-10 14:23:03 -07:00

README.md

splitter

This splitter allows ingesting logs from a CWLogs subscription.

Splitter's expected input is batched logs from a CloudWatchLogs subscription to a Kinesis Stream. The CWLogs subscription has a special format which bundles several logs into a single record. The splitter takes this record and splits it into multiple logs. These logs are also modified to mimic the RSyslog format we expect from our other logs. This allows them to be decoded normally by the rest of the pipeline.