luciano/amazon-kinesis-client-go
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
182 commits 11 branches 1 tag 8.2 MiB
Commit graph

182 commits

This branch
This branch
All branches
Author SHA1 Message Date
Taylor Sutton
4dd769ffca
Merge pull request #58 from Clever/refactor-decompression 
Refactor zlib decompression into the splitter package
 2020-11-12 11:57:53 -05:00
Taylor Sutton
f8e9c34641 Refactor zlib decompression into the splitter package 
Being in the batchconsumer package means it will work for anything
using KCL, but lambdas that subscribe to these log streams do not use
batchconsumer at all; instead they invoke the splitter package
directly. As such, if we want this functionality to be available to
lambda log consumers, it can't be in batchconsumer.

There are no functionality changes here, just moving code from an
unexported method in one place to an exported function in another
place. The tests also get moved along with it.
 2020-11-12 11:37:13 -05:00
Taylor Sutton
2a7e96157e
Merge pull request #57 from Clever/add-zlib-decompression 
Add zlib decompression to batch consumer.
 2020-11-11 12:42:11 -05:00
Taylor Sutton
4fe27d0d39 Clarifying comment - Logs from CW Logs are gzipped. 2020-11-11 12:39:51 -05:00
Taylor Sutton
6b2d1f8a56 Add zlib decompression to batch consumer. 
This has become relevant as zlib is the compression method supported
by the Kinesis plugin for Fluent Bit.
 2020-11-11 09:12:05 -05:00
Gavi Hirsch
611f0c0a60
Merge pull request #56 from Clever/INFRANG-3918 
clear count stats after logging them
 2020-09-11 15:02:06 -07:00
Gavi Hirsch
1112894639
clear count stats after logging them 2020-09-11 14:04:42 -07:00
Taylor Sutton
484b54bfe0
Merge pull request #55 from Clever/INFRANG-35-fluent-decode 
Add decoding of JSON logs in Fluent format.
 2020-08-18 08:44:17 -07:00
Taylor Sutton
f5ce6fe4e7 Add decoding of JSON logs in Fluent format. 
ParseAndEnhance used to be:
- Try to parse line as a syslog, extracting the log itself and other
  fields from syslog format
  - If that succeeds, try to parse the log as either a Kayvee log or
    an RDS slow query log.
  - Combine all these fields, and add on some "derived"
    fields (container_task|env|app).
- Not a syslog => error

Now it will be:
- Try to parse line as a syslog, same as before, including the
  Kayvee/RDS part
- If syslog parsing failed, try to parse as a Fluent log and extract
  some fields from the Fluent format (the log, timestamp, etc)
  - If that succeeds, try to parse the log itself as a Kayvee log.
  - Combine Kayvee fields (if found) and derived fields)
- If BOTH formats fields, it is an error.

The decoding makes a lot of assumptions:
- The names of the log field and timestamp field (even though,
theoretically, they are customizable in the fluentbit config.
- The timestamp format (again)
- The format of the Task Definition name (or at least part of it)
- All fluentbit logs should have hostname set to `aws-fargate`.

Perhaps these can be relaxed if necessary. They could probably be
replaced by some kind of config. As there is currently no config I
wanted to keep things simple as possible. If we need to re-evaluate
(for example if we start getting JSON logs that don't want to use the
same handling for container_task|env|app) we can reevaluate.
 2020-08-17 15:02:59 -07:00
Gavi Hirsch
9b32d93d1c
Merge pull request #54 from Clever/upgrade-kayvee-go-6.24.0 
upgrade kayvee-go to 6.24.0
 2020-08-14 19:19:26 -07:00
Gavi Hirsch
3bedc65483 upgrade kayvee-go to 6.24.0 2020-08-14 16:30:08 -07:00
Gavi Hirsch
b16af062fb
Merge pull request #53 from Clever/glue-2.0 
glue log group format
 2020-08-11 16:52:03 -07:00
Gavi Hirsch
30d3925119 glue log group format 2020-08-11 16:42:59 -07:00
Gavi Hirsch
ebd1e0e39c
Merge pull request #52 from Clever/dependabot/dep/github.com/stretchr/testify-1.6.1 
Bump github.com/stretchr/testify from 1.1.4 to 1.6.1
 2020-07-24 12:21:48 -07:00
Gavi Hirsch
bfe1a6ffb9 fix Gopkg.lock 2020-07-24 12:19:30 -07:00
dependabot-preview[bot]
35b07d7aed
Bump github.com/stretchr/testify from 1.1.4 to 1.6.1 
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.1.4 to 1.6.1.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.1.4...v1.6.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-07-24 17:38:31 +00:00
Gavi Hirsch
8a8b44112d
Merge pull request #51 from Clever/dependabot/dep/gopkg.in/Clever/kayvee-go.v6-6.23.0 
Bump gopkg.in/Clever/kayvee-go.v6 from 6.6.0 to 6.23.0
 2020-07-23 10:27:58 -07:00
dependabot-preview[bot]
8be774bc09
Bump gopkg.in/Clever/kayvee-go.v6 from 6.6.0 to 6.23.0 
Bumps [gopkg.in/Clever/kayvee-go.v6](https://github.com/Clever/kayvee-go) from 6.6.0 to 6.23.0.
- [Release notes](https://github.com/Clever/kayvee-go/releases)
- [Commits](https://github.com/Clever/kayvee-go/compare/v6.6.0...v6.23.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-07-23 17:24:47 +00:00
Rafael
504ebfad60
Merge pull request #49 from Clever/INFRANG-3968 
support parsing AWS Glue job logs
 2020-07-20 14:15:14 -04:00
Rafael Garcia
420ad243a4 add codeowners 2020-07-20 18:11:27 +00:00
Rafael Garcia
337d2063f5 support parsing AWS Glue job logs 2020-07-20 17:20:39 +00:00
Sayan
eb11747434
Merge pull request #48 from Clever/update-jar-link 
Fix maven repo link
 2020-04-06 13:50:16 -07:00
Sayan Samanta
d489b5039b update to repo1 link for jars 2020-04-06 13:31:10 -07:00
Taylor Sutton
e7db80a35a
Merge pull request #46 from Clever/go1.13 
Upgrade go to v1.13
 2019-10-18 11:36:05 -07:00
Taylor Sutton
5132214945 Upgrade go to v1.13 2019-10-18 09:45:22 -07:00
Daniel Xu
e265494afb
Merge pull request #45 from Clever/INFRA-3664 
Fix regex that parses the RDS slowquery user
 2019-10-14 17:21:53 -07:00
Daniel Xu
69d31088f3 Add tests for parsing slow query user 2019-10-14 17:19:12 -07:00
Daniel Xu
7446627fae Fix regex that parses the RDS slowquery user 2019-10-14 17:02:01 -07:00
Aaron Stein
9cbcf1096a
Merge pull request #44 from Clever/parse-rds 
Parse rds slowquery user
 2019-10-02 10:21:31 -07:00
Aaron Stein
a66ded0e8f parse rds slowquery user 2019-10-01 14:39:11 -07:00
Aaron Stein
07828aef7b
INFRA-2885: Split slowquery logs - Merge pull request #42 from Clever/INFRA-2885 
INFRA-2885: Split slowquery logs
 2019-08-26 12:16:17 -07:00
Aaron Stein
598d57359f remove PID from slowquery log 2019-08-26 12:09:06 -07:00
Aaron Stein
1261000e33 re-refactor log classification logic 2019-08-26 12:08:01 -07:00
Aaron Stein
ea59ce899b add match check 2019-08-26 12:08:01 -07:00
Aaron Stein
df8b175e86 add decode test 2019-08-26 12:08:01 -07:00
Aaron Stein
45c48035c7 split slowquery logs 2019-08-26 12:08:01 -07:00
Aaron Stein
2af69f28a2 update deps 2019-08-26 12:08:01 -07:00
Aaron Stein
c38be59053
Remove PID - Merge pull request #43 from Clever/rm-pid 
Remove PID
 2019-08-26 12:01:21 -07:00
Aaron Stein
4c9e2c582f rm debug file 2019-08-23 13:20:21 -07:00
Aaron Stein
c1d013f301 add PID-less test 2019-08-23 13:17:02 -07:00
Aaron Stein
f747ccfdd0 rm constant PID 2019-08-23 10:27:43 -07:00
johnhuangclever
4f8bee913e
Merge pull request #40 from Clever/go1.12-again 2019-06-01 01:05:48 -07:00
John Huang
4a28b40953 Golang 1.9 upgrade 2019-06-01 00:39:17 -07:00
Rafael
ebbd9e7a37
Merge pull request #39 from Clever/INFRA-3363-take-two 
expand ECS task id regex for fargate
 2019-03-01 10:37:30 -08:00
Rafael Garcia
18e6920dea expand task id regex for fargate 2019-02-27 23:15:19 +00:00
Rafael Garcia
aeb8af04e8 parse fargate logs 2019-02-25 20:52:52 +00:00
Xavi
2b62c6cbb1
Merge pull request #37 from Clever/INFRA-3343-rm-awscli-install-amazon-kinesis-client-go 
Removed unneeded "Install awscli for ECR publish" command
 2019-01-09 14:55:35 -08:00
Xavi Ramirez
df71c27e6f Removed unneeded "Install awscli for ECR publish" command 2019-01-08 18:40:29 -08:00
Xavi
a975ca3c3b
Merge pull request #36 from Clever/add-err-msg 
Added error message for failed log written to file
 2018-12-11 16:43:26 -08:00
Xavi Ramirez
e80ff2239b Added error message for failed log written to file 2018-12-11 16:32:33 -08:00