Adds a lowest-common-denominator function, KPLDeaggregate, for
handling records that might be KPL aggregated. Also adds a function,
DeaggregateAndSplitIfNecessary, to wrap the existing functionality of
SplitMessageIfNecessary with KPL deaggreation.
These functions are handy for non-KCL consumers, like Lambda
functions. KCL automatically applies deaggreation for you.
This change is backwards compatible - the previously exposed function
SplitMessageIfNecessary still does the same things.
Being in the batchconsumer package means it will work for anything
using KCL, but lambdas that subscribe to these log streams do not use
batchconsumer at all; instead they invoke the splitter package
directly. As such, if we want this functionality to be available to
lambda log consumers, it can't be in batchconsumer.
There are no functionality changes here, just moving code from an
unexported method in one place to an exported function in another
place. The tests also get moved along with it.
AWS changed the format of the log stream for AWS Batch logs. This
reflects that change.
---
http://docs.aws.amazon.com/batch/latest/userguide/job_states.html
Logs for RUNNING jobs are available in CloudWatch Logs; the log group is /aws/batch/job, and the log stream name format is jobDefinitionName/default/ecs_task_id (this format may change in the future).
After a job reaches the RUNNING status, you can programmatically retrieve its log stream name with the DescribeJobs API operation. For more information, see View Log Data Sent to CloudWatch Logs in the Amazon CloudWatch Logs User Guide. By default, these logs are set to never expire, but you can modify the retention period. For more information, see Change Log Data Retention in CloudWatch Logs in the Amazon CloudWatch Logs User Guide.
Filtering by env was an artifact of the previous way we originally had
setup our CWLogs subscriptions, assuming that we could have multiple
subscriptions per stream. :( You can't.
Let's instead allow any CWLogs subscription's logs to be valid, which is
needed for:
https://github.com/Clever/kinesis-cwlogs-splitter/pull/1
