luciano/amazon-kinesis-client-go
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
180 commits 11 branches 1 tag 8.2 MiB
Commit graph

180 commits

This branch
This branch
All branches
Author SHA1 Message Date
Taylor Sutton
2a7e96157e
Merge pull request #57 from Clever/add-zlib-decompression 
Add zlib decompression to batch consumer.
 2020-11-11 12:42:11 -05:00
Taylor Sutton
4fe27d0d39 Clarifying comment - Logs from CW Logs are gzipped. 2020-11-11 12:39:51 -05:00
Taylor Sutton
6b2d1f8a56 Add zlib decompression to batch consumer. 
This has become relevant as zlib is the compression method supported
by the Kinesis plugin for Fluent Bit.
 2020-11-11 09:12:05 -05:00
Gavi Hirsch
611f0c0a60
Merge pull request #56 from Clever/INFRANG-3918 
clear count stats after logging them
 2020-09-11 15:02:06 -07:00
Gavi Hirsch
1112894639
clear count stats after logging them 2020-09-11 14:04:42 -07:00
Taylor Sutton
484b54bfe0
Merge pull request #55 from Clever/INFRANG-35-fluent-decode 
Add decoding of JSON logs in Fluent format.
 2020-08-18 08:44:17 -07:00
Taylor Sutton
f5ce6fe4e7 Add decoding of JSON logs in Fluent format. 
ParseAndEnhance used to be:
- Try to parse line as a syslog, extracting the log itself and other
  fields from syslog format
  - If that succeeds, try to parse the log as either a Kayvee log or
    an RDS slow query log.
  - Combine all these fields, and add on some "derived"
    fields (container_task|env|app).
- Not a syslog => error

Now it will be:
- Try to parse line as a syslog, same as before, including the
  Kayvee/RDS part
- If syslog parsing failed, try to parse as a Fluent log and extract
  some fields from the Fluent format (the log, timestamp, etc)
  - If that succeeds, try to parse the log itself as a Kayvee log.
  - Combine Kayvee fields (if found) and derived fields)
- If BOTH formats fields, it is an error.

The decoding makes a lot of assumptions:
- The names of the log field and timestamp field (even though,
theoretically, they are customizable in the fluentbit config.
- The timestamp format (again)
- The format of the Task Definition name (or at least part of it)
- All fluentbit logs should have hostname set to `aws-fargate`.

Perhaps these can be relaxed if necessary. They could probably be
replaced by some kind of config. As there is currently no config I
wanted to keep things simple as possible. If we need to re-evaluate
(for example if we start getting JSON logs that don't want to use the
same handling for container_task|env|app) we can reevaluate.
 2020-08-17 15:02:59 -07:00
Gavi Hirsch
9b32d93d1c
Merge pull request #54 from Clever/upgrade-kayvee-go-6.24.0 
upgrade kayvee-go to 6.24.0
 2020-08-14 19:19:26 -07:00
Gavi Hirsch
3bedc65483 upgrade kayvee-go to 6.24.0 2020-08-14 16:30:08 -07:00
Gavi Hirsch
b16af062fb
Merge pull request #53 from Clever/glue-2.0 
glue log group format
 2020-08-11 16:52:03 -07:00
Gavi Hirsch
30d3925119 glue log group format 2020-08-11 16:42:59 -07:00
Gavi Hirsch
ebd1e0e39c
Merge pull request #52 from Clever/dependabot/dep/github.com/stretchr/testify-1.6.1 
Bump github.com/stretchr/testify from 1.1.4 to 1.6.1
 2020-07-24 12:21:48 -07:00
Gavi Hirsch
bfe1a6ffb9 fix Gopkg.lock 2020-07-24 12:19:30 -07:00
dependabot-preview[bot]
35b07d7aed
Bump github.com/stretchr/testify from 1.1.4 to 1.6.1 
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.1.4 to 1.6.1.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.1.4...v1.6.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-07-24 17:38:31 +00:00
Gavi Hirsch
8a8b44112d
Merge pull request #51 from Clever/dependabot/dep/gopkg.in/Clever/kayvee-go.v6-6.23.0 
Bump gopkg.in/Clever/kayvee-go.v6 from 6.6.0 to 6.23.0
 2020-07-23 10:27:58 -07:00
dependabot-preview[bot]
8be774bc09
Bump gopkg.in/Clever/kayvee-go.v6 from 6.6.0 to 6.23.0 
Bumps [gopkg.in/Clever/kayvee-go.v6](https://github.com/Clever/kayvee-go) from 6.6.0 to 6.23.0.
- [Release notes](https://github.com/Clever/kayvee-go/releases)
- [Commits](https://github.com/Clever/kayvee-go/compare/v6.6.0...v6.23.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-07-23 17:24:47 +00:00
Rafael
504ebfad60
Merge pull request #49 from Clever/INFRANG-3968 
support parsing AWS Glue job logs
 2020-07-20 14:15:14 -04:00
Rafael Garcia
420ad243a4 add codeowners 2020-07-20 18:11:27 +00:00
Rafael Garcia
337d2063f5 support parsing AWS Glue job logs 2020-07-20 17:20:39 +00:00
Sayan
eb11747434
Merge pull request #48 from Clever/update-jar-link 
Fix maven repo link
 2020-04-06 13:50:16 -07:00
Sayan Samanta
d489b5039b update to repo1 link for jars 2020-04-06 13:31:10 -07:00
Taylor Sutton
e7db80a35a
Merge pull request #46 from Clever/go1.13 
Upgrade go to v1.13
 2019-10-18 11:36:05 -07:00
Taylor Sutton
5132214945 Upgrade go to v1.13 2019-10-18 09:45:22 -07:00
Daniel Xu
e265494afb
Merge pull request #45 from Clever/INFRA-3664 
Fix regex that parses the RDS slowquery user
 2019-10-14 17:21:53 -07:00
Daniel Xu
69d31088f3 Add tests for parsing slow query user 2019-10-14 17:19:12 -07:00
Daniel Xu
7446627fae Fix regex that parses the RDS slowquery user 2019-10-14 17:02:01 -07:00
Aaron Stein
9cbcf1096a
Merge pull request #44 from Clever/parse-rds 
Parse rds slowquery user
 2019-10-02 10:21:31 -07:00
Aaron Stein
a66ded0e8f parse rds slowquery user 2019-10-01 14:39:11 -07:00
Aaron Stein
07828aef7b
INFRA-2885: Split slowquery logs - Merge pull request #42 from Clever/INFRA-2885 
INFRA-2885: Split slowquery logs
 2019-08-26 12:16:17 -07:00
Aaron Stein
598d57359f remove PID from slowquery log 2019-08-26 12:09:06 -07:00
Aaron Stein
1261000e33 re-refactor log classification logic 2019-08-26 12:08:01 -07:00
Aaron Stein
ea59ce899b add match check 2019-08-26 12:08:01 -07:00
Aaron Stein
df8b175e86 add decode test 2019-08-26 12:08:01 -07:00
Aaron Stein
45c48035c7 split slowquery logs 2019-08-26 12:08:01 -07:00
Aaron Stein
2af69f28a2 update deps 2019-08-26 12:08:01 -07:00
Aaron Stein
c38be59053
Remove PID - Merge pull request #43 from Clever/rm-pid 
Remove PID
 2019-08-26 12:01:21 -07:00
Aaron Stein
4c9e2c582f rm debug file 2019-08-23 13:20:21 -07:00
Aaron Stein
c1d013f301 add PID-less test 2019-08-23 13:17:02 -07:00
Aaron Stein
f747ccfdd0 rm constant PID 2019-08-23 10:27:43 -07:00
johnhuangclever
4f8bee913e
Merge pull request #40 from Clever/go1.12-again 2019-06-01 01:05:48 -07:00
John Huang
4a28b40953 Golang 1.9 upgrade 2019-06-01 00:39:17 -07:00
Rafael
ebbd9e7a37
Merge pull request #39 from Clever/INFRA-3363-take-two 
expand ECS task id regex for fargate
 2019-03-01 10:37:30 -08:00
Rafael Garcia
18e6920dea expand task id regex for fargate 2019-02-27 23:15:19 +00:00
Rafael Garcia
aeb8af04e8 parse fargate logs 2019-02-25 20:52:52 +00:00
Xavi
2b62c6cbb1
Merge pull request #37 from Clever/INFRA-3343-rm-awscli-install-amazon-kinesis-client-go 
Removed unneeded "Install awscli for ECR publish" command
 2019-01-09 14:55:35 -08:00
Xavi Ramirez
df71c27e6f Removed unneeded "Install awscli for ECR publish" command 2019-01-08 18:40:29 -08:00
Xavi
a975ca3c3b
Merge pull request #36 from Clever/add-err-msg 
Added error message for failed log written to file
 2018-12-11 16:43:26 -08:00
Xavi Ramirez
e80ff2239b Added error message for failed log written to file 2018-12-11 16:32:33 -08:00
Xavi
23c68b42ff
Merge pull request #33 from wwwjfy/fix-shutdown-requested 
Fix shutdown requested
 2018-12-11 16:28:37 -08:00
Xavi
7351346041
Merge pull request #35 from wwwjfy/fix-checkpoint 
fix periodic checkpoint not triggered
 2018-12-11 16:23:51 -08:00