Commit graph

14 commits

Author SHA1 Message Date
Gavi Hirsch
30d3925119 glue log group format 2020-08-11 16:42:59 -07:00
Rafael Garcia
337d2063f5 support parsing AWS Glue job logs 2020-07-20 17:20:39 +00:00
Aaron Stein
598d57359f remove PID from slowquery log 2019-08-26 12:09:06 -07:00
Aaron Stein
45c48035c7 split slowquery logs 2019-08-26 12:08:01 -07:00
Aaron Stein
f747ccfdd0 rm constant PID 2019-08-23 10:27:43 -07:00
Rafael Garcia
18e6920dea expand task id regex for fargate 2019-02-27 23:15:19 +00:00
Rafael Garcia
aeb8af04e8 parse fargate logs 2019-02-25 20:52:52 +00:00
Rafael Garcia
ad2ef687bb fix lambda -> rsyslog message conversion to always have a task ID 2018-03-13 16:48:41 +00:00
Rafael Garcia
c255bf87f3 add test for default case 2018-01-25 15:13:11 +00:00
Rafael Garcia
10750e802b handle logs coming in from lambda functions 2018-01-25 01:23:03 +00:00
Nathan Leiby
efa90d4481 splitter: update log stream parsing
AWS changed the format of the log stream for AWS Batch logs. This
reflects that change.

---

http://docs.aws.amazon.com/batch/latest/userguide/job_states.html

Logs for RUNNING jobs are available in CloudWatch Logs; the log group is /aws/batch/job, and the log stream name format is jobDefinitionName/default/ecs_task_id (this format may change in the future).
After a job reaches the RUNNING status, you can programmatically retrieve its log stream name with the DescribeJobs API operation. For more information, see View Log Data Sent to CloudWatch Logs in the Amazon CloudWatch Logs User Guide. By default, these logs are set to never expire, but you can modify the retention period. For more information, see Change Log Data Retention in CloudWatch Logs in the Amazon CloudWatch Logs User Guide.
2017-09-20 14:18:35 -07:00
Nathan Leiby
0c02e93821 splitter: don't filter by env
Filtering by env was an artifact of the previous way we originally had
setup our CWLogs subscriptions, assuming that we could have multiple
subscriptions per stream. :( You can't.

Let's instead allow any CWLogs subscription's logs to be valid, which is
needed for:
https://github.com/Clever/kinesis-cwlogs-splitter/pull/1
2017-08-10 14:23:03 -07:00
Xavi Ramirez
8d273d6a1e Fixed and added unit tests 2017-07-18 19:19:40 +00:00
Xavi Ramirez
4c67f39c99 Implemented new consumer interface as well as example consumer: 'batchconsumer' 2017-07-18 02:03:15 +00:00