Amazon Kinesis Client for Go
f5ce6fe4e7
ParseAndEnhance used to be:
- Try to parse line as a syslog, extracting the log itself and other
fields from syslog format
- If that succeeds, try to parse the log as either a Kayvee log or
an RDS slow query log.
- Combine all these fields, and add on some "derived"
fields (container_task|env|app).
- Not a syslog => error
Now it will be:
- Try to parse line as a syslog, same as before, including the
Kayvee/RDS part
- If syslog parsing failed, try to parse as a Fluent log and extract
some fields from the Fluent format (the log, timestamp, etc)
- If that succeeds, try to parse the log itself as a Kayvee log.
- Combine Kayvee fields (if found) and derived fields)
- If BOTH formats fields, it is an error.
The decoding makes a lot of assumptions:
- The names of the log field and timestamp field (even though,
theoretically, they are customizable in the fluentbit config.
- The timestamp format (again)
- The format of the Task Definition name (or at least part of it)
- All fluentbit logs should have hostname set to `aws-fargate`.
Perhaps these can be relaxed if necessary. They could probably be
replaced by some kind of config. As there is currently no config I
wanted to keep things simple as possible. If we need to re-evaluate
(for example if we start getting JSON logs that don't want to use the
same handling for container_task|env|app) we can reevaluate.
|
||
|---|---|---|
| .circleci | ||
| .github | ||
| batchconsumer | ||
| cmd | ||
| decode | ||
| kcl | ||
| splitter | ||
| .gitignore | ||
| batchconsumer.properties | ||
| consumer.properties | ||
| golang.mk | ||
| Gopkg.lock | ||
| Gopkg.toml | ||
| Makefile | ||
| README.md | ||
amazon-kinesis-client-go
A port of amazon-kinesis-client-python to Go.
Running the Consumer
Edit the file consumer.properties to point at a Kinesis stream that has some data.
Build the consumer binary:
make build
Then run:
make run
This will download the jar files necessary to run the KCL, and then launch the KCL communicating with the consumer binary.