Commit graph

17 commits

Author SHA1 Message Date
Taylor Sutton
f5ce6fe4e7 Add decoding of JSON logs in Fluent format.
ParseAndEnhance used to be:
- Try to parse line as a syslog, extracting the log itself and other
  fields from syslog format
  - If that succeeds, try to parse the log as either a Kayvee log or
    an RDS slow query log.
  - Combine all these fields, and add on some "derived"
    fields (container_task|env|app).
- Not a syslog => error

Now it will be:
- Try to parse line as a syslog, same as before, including the
  Kayvee/RDS part
- If syslog parsing failed, try to parse as a Fluent log and extract
  some fields from the Fluent format (the log, timestamp, etc)
  - If that succeeds, try to parse the log itself as a Kayvee log.
  - Combine Kayvee fields (if found) and derived fields)
- If BOTH formats fields, it is an error.

The decoding makes a lot of assumptions:
- The names of the log field and timestamp field (even though,
theoretically, they are customizable in the fluentbit config.
- The timestamp format (again)
- The format of the Task Definition name (or at least part of it)
- All fluentbit logs should have hostname set to `aws-fargate`.

Perhaps these can be relaxed if necessary. They could probably be
replaced by some kind of config. As there is currently no config I
wanted to keep things simple as possible. If we need to re-evaluate
(for example if we start getting JSON logs that don't want to use the
same handling for container_task|env|app) we can reevaluate.
2020-08-17 15:02:59 -07:00
Daniel Xu
69d31088f3 Add tests for parsing slow query user 2019-10-14 17:19:12 -07:00
Aaron Stein
a66ded0e8f parse rds slowquery user 2019-10-01 14:39:11 -07:00
Aaron Stein
1261000e33 re-refactor log classification logic 2019-08-26 12:08:01 -07:00
Aaron Stein
df8b175e86 add decode test 2019-08-26 12:08:01 -07:00
Aaron Stein
c1d013f301 add PID-less test 2019-08-23 13:17:02 -07:00
Xavi
c4712bd220
Fixed type-o: guage -> gauge 2018-10-05 15:31:48 -07:00
Xavi Ramirez
8f2ad09efa Added timestamp, hostname, and rawlog to list of reserved fields 2018-05-10 23:56:32 +00:00
Xavi Ramirez
74634a6bf0 Added decoder_msg_type field to logs parsed by syslog 2018-05-10 23:56:14 +00:00
Xavi Ramirez
2ecbc081cd Removed stringifyNested, renameESReservedFields, and minimumTimestamp params from decoder. minimumTimestamp isn't needed anymore. stringifyNested and renameESReservedFields are very specific to the elasticsearch consumer and will be moved to the elasticsearch consumer. 2017-09-21 19:51:48 +00:00
Nathan Leiby
d1473f894a decode: s/type/decoder_msg_type
type is a really generally name. sometimes, it is used in other
libraries or various output as a valid field name. let's not make it a
reserved keyword that gets overriden by type == Kayvee
2017-08-17 14:36:54 -07:00
Xavi
4b4af77ba2 Merge pull request #9 from Clever/INFRA-2405-new-consumer-interface
Updated syslogparser library and added unit tests
2017-08-16 14:54:27 -07:00
Xavi Ramirez
e2f55b6d98 Updated syslogparser library and added unit tests 2017-08-16 20:58:30 +00:00
Nathan Leiby
1bebeb5aa5 decode: json decodes dimensions as array of interface
Previously, this would result in a log line failing to extracting any
dimensions because

```golang
dimensions.([]string)
```

would fail because

```golang
reflect.TypeOf(dimensions)
```

was `[]interface{}`.
2017-08-15 11:47:01 -07:00
Xavi Ramirez
de04a27799 Return NonKayveeError if log contians invalid or empty json 2017-08-07 19:20:56 +00:00
Xavi Ramirez
266d7d620d Fixed bug in decoder package 2017-08-07 19:20:56 +00:00
Xavi Ramirez
4c67f39c99 Implemented new consumer interface as well as example consumer: 'batchconsumer' 2017-07-18 02:03:15 +00:00