luciano/amazon-kinesis-client-go
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

splitter: update log stream parsing

Browse source 
AWS changed the format of the log stream for AWS Batch logs. This
reflects that change.

---

http://docs.aws.amazon.com/batch/latest/userguide/job_states.html

Logs for RUNNING jobs are available in CloudWatch Logs; the log group is /aws/batch/job, and the log stream name format is jobDefinitionName/default/ecs_task_id (this format may change in the future).
After a job reaches the RUNNING status, you can programmatically retrieve its log stream name with the DescribeJobs API operation. For more information, see View Log Data Sent to CloudWatch Logs in the Amazon CloudWatch Logs User Guide. By default, these logs are set to never expire, but you can modify the retention period. For more information, see Change Log Data Retention in CloudWatch Logs in the Amazon CloudWatch Logs User Guide.
This commit is contained in:
Nathan Leiby 2017-09-20 14:18:35 -07:00
parent 32d4d33f00
commit efa90d4481
2 changed files with 6 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
splitter/splitter.go
View file

@ -65,9 +65,11 @@ func unpack(input []byte) (LogEventBatch, error) {
// RFC3339Micro is the RFC3339 format in microseconds
const RFC3339Micro = "2006-01-02T15:04:05.999999-07:00"
// http://docs.aws.amazon.com/batch/latest/userguide/job_states.html
// "log stream name format is jobDefinitionName/default/ecs_task_id (this format may change in the future)."
const taskMeta = `([a-z0-9-]+)--([a-z0-9-]+)\/` + // env--app
`([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\/` + // task-id
`([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})` // container-id
`default\/` +
`([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})` // task-id
var taskRegex = regexp.MustCompile(taskMeta)

4
splitter/splitter_test.go
View file

@ -64,7 +64,7 @@ func TestFullLoop(t *testing.T) {
"messageType": "DATA_MESSAGE",
"owner": "123456789012",
"logGroup": "/aws/batch/job",
"logStream": "environment--app/11111111-2222-3333-4444-555566667777/88889999-0000-aaa-bbbb-ccccddddeeee",
"logStream": "environment--app/default/11111111-2222-3333-4444-555566667777",
"subscriptionFilters": [
"MySubscriptionFilter"
],
@ -103,7 +103,7 @@ func TestSplit(t *testing.T) {
MessageType: "DATA_MESSAGE",
Owner: "123456789012",
LogGroup: "/aws/batch/job",
LogStream: "env--app/12345678-1234-1234-1234-555566667777/88889999-0000-aaaa-bbbb-ccccddddeeee",
LogStream: "env--app/default/12345678-1234-1234-1234-555566667777",
SubscriptionFilters: []string{"MySubscriptionFilter"},
LogEvents: []LogEvent{
{