History

Nathan Leiby efa90d4481 splitter: update log stream parsing AWS changed the format of the log stream for AWS Batch logs. This reflects that change. --- http://docs.aws.amazon.com/batch/latest/userguide/job_states.html Logs for RUNNING jobs are available in CloudWatch Logs; the log group is /aws/batch/job, and the log stream name format is jobDefinitionName/default/ecs_task_id (this format may change in the future). After a job reaches the RUNNING status, you can programmatically retrieve its log stream name with the DescribeJobs API operation. For more information, see View Log Data Sent to CloudWatch Logs in the Amazon CloudWatch Logs User Guide. By default, these logs are set to never expire, but you can modify the retention period. For more information, see Change Log Data Retention in CloudWatch Logs in the Amazon CloudWatch Logs User Guide.		2017-09-20 14:18:35 -07:00
..
README.md	Implemented new consumer interface as well as example consumer: 'batchconsumer'	2017-07-18 02:03:15 +00:00
splitter.go	splitter: update log stream parsing	2017-09-20 14:18:35 -07:00
splitter_test.go	splitter: update log stream parsing	2017-09-20 14:18:35 -07:00

README.md

splitter

This splitter allows ingesting logs from a CWLogs subscription.

Splitter's expected input is batched logs from a CloudWatchLogs subscription to a Kinesis Stream. The CWLogs subscription has a special format which bundles several logs into a single record. The splitter takes this record and splits it into multiple logs. These logs are also modified to mimic the RSyslog format we expect from our other logs. This allows them to be decoded normally by the rest of the pipeline.