luciano/amazon-kinesis-client-go
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
204 commits 11 branches 1 tag 8.2 MiB
Commit graph

204 commits

This branch
This branch
All branches
Author SHA1 Message Date
Taylor Sutton
f78d1c6b9b
Merge pull request #71 from Clever/INFRANG-6435 
notify CI failure
 2024-09-19 07:23:19 -07:00
Tanmay Sardesai
a8a4e207fe notify CI failure 2024-09-18 13:04:48 -04:00
Taylor Sutton
e66f1922b7
Merge pull request #68 from Clever/INFRANG-5293 
Update github known hosts key
 2023-05-04 09:43:18 -07:00
Chris Martin
dc3e5e097c Update github known hosts key 2023-04-25 14:56:16 -04:00
Nikhil Bhatia
9a1371d965
Merge pull request #67 from Clever/ci-status-notify 
Microplane: add Github Action workflow for ci-notify
 2022-03-29 20:30:42 -07:00
nbhatia823
cf19a05ced Microplane: add Github Action workflow for ci-notify 2022-03-29 15:12:09 -07:00
Rafael
3a42d32e53
Merge pull request #66 from Clever/go-modules 
dep -> go modules
 2021-06-03 19:12:19 -04:00
Gavi Hirsch
d461ebebb6 dep -> go modules 2021-06-03 15:41:20 -07:00
Taylor Sutton
c11acd24ec
Merge pull request #64 from Clever/dependabot/dep/github.com/golang/protobuf-1.5.2 
Bump github.com/golang/protobuf from 1.5.1 to 1.5.2
 2021-03-30 12:57:35 -07:00
dependabot-preview[bot]
ad3e33c7d6
Bump github.com/golang/protobuf from 1.5.1 to 1.5.2 
Bumps [github.com/golang/protobuf](https://github.com/golang/protobuf) from 1.5.1 to 1.5.2.
- [Release notes](https://github.com/golang/protobuf/releases)
- [Commits](https://github.com/golang/protobuf/compare/v1.5.1...v1.5.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2021-03-30 17:52:57 +00:00
Taylor Sutton
fa54b6ee39
Merge pull request #63 from Clever/dependabot/dep/github.com/golang/protobuf-1.5.1 
Bump github.com/golang/protobuf from 1.4.3 to 1.5.1
 2021-03-18 11:22:53 -07:00
dependabot-preview[bot]
c164f1d895
Bump github.com/golang/protobuf from 1.4.3 to 1.5.1 
Bumps [github.com/golang/protobuf](https://github.com/golang/protobuf) from 1.4.3 to 1.5.1.
- [Release notes](https://github.com/golang/protobuf/releases)
- [Commits](https://github.com/golang/protobuf/compare/v1.4.3...v1.5.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2021-03-18 17:21:17 +00:00
Gavi Hirsch
6ac018712d
Merge pull request #62 from Clever/INFRANG-4003 
add versioning
 2021-03-05 09:27:29 -08:00
Gavi Hirsch
7351a3d3f7 add versioning 2021-03-04 10:11:51 -08:00
Rafael
358de6589c
Merge pull request #61 from Clever/dependabot/dep/gopkg.in/Clever/kayvee-go.v6-6.24.1 
Bump gopkg.in/Clever/kayvee-go.v6 from 6.24.0 to 6.24.1
 2021-02-02 13:26:14 -05:00
dependabot-preview[bot]
fe41fdaac3
Bump gopkg.in/Clever/kayvee-go.v6 from 6.24.0 to 6.24.1 
Bumps [gopkg.in/Clever/kayvee-go.v6](https://github.com/Clever/kayvee-go) from 6.24.0 to 6.24.1.
- [Release notes](https://github.com/Clever/kayvee-go/releases)
- [Commits](https://github.com/Clever/kayvee-go/compare/v6.24.0...v6.24.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2021-02-02 18:20:42 +00:00
Tanmay Sardesai
102de58470
Merge pull request #60 from Clever/dependabot/dep/github.com/stretchr/testify-1.7.0 
Bump github.com/stretchr/testify from 1.6.1 to 1.7.0
 2021-01-19 10:01:36 -08:00
dependabot-preview[bot]
54f91f4073
Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.6.1 to 1.7.0.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.6.1...v1.7.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2021-01-13 18:18:37 +00:00
Taylor Sutton
29f86e9986
Merge pull request #59 from Clever/refactor-decompression 
Refactor decompression
 2020-11-16 20:18:07 -05:00
Taylor Sutton
42c3480288 Simplify main function's name. 2020-11-16 17:37:53 -05:00
Taylor Sutton
cc4f7716b3 More documentation around handling of CW log bundles; deflake test 2020-11-16 12:44:50 -05:00
Taylor Sutton
6e8a99c50d Add functionality for handling KPL aggregates and zlib 
Adds a lowest-common-denominator function, KPLDeaggregate, for
handling records that might be KPL aggregated. Also adds a function,
DeaggregateAndSplitIfNecessary, to wrap the existing functionality of
SplitMessageIfNecessary with KPL deaggreation.

These functions are handy for non-KCL consumers, like Lambda
functions. KCL automatically applies deaggreation for you.

This change is backwards compatible - the previously exposed function
SplitMessageIfNecessary still does the same things.
 2020-11-16 11:01:28 -05:00
Taylor Sutton
4dd769ffca
Merge pull request #58 from Clever/refactor-decompression 
Refactor zlib decompression into the splitter package
 2020-11-12 11:57:53 -05:00
Taylor Sutton
f8e9c34641 Refactor zlib decompression into the splitter package 
Being in the batchconsumer package means it will work for anything
using KCL, but lambdas that subscribe to these log streams do not use
batchconsumer at all; instead they invoke the splitter package
directly. As such, if we want this functionality to be available to
lambda log consumers, it can't be in batchconsumer.

There are no functionality changes here, just moving code from an
unexported method in one place to an exported function in another
place. The tests also get moved along with it.
 2020-11-12 11:37:13 -05:00
Taylor Sutton
2a7e96157e
Merge pull request #57 from Clever/add-zlib-decompression 
Add zlib decompression to batch consumer.
 2020-11-11 12:42:11 -05:00
Taylor Sutton
4fe27d0d39 Clarifying comment - Logs from CW Logs are gzipped. 2020-11-11 12:39:51 -05:00
Taylor Sutton
6b2d1f8a56 Add zlib decompression to batch consumer. 
This has become relevant as zlib is the compression method supported
by the Kinesis plugin for Fluent Bit.
 2020-11-11 09:12:05 -05:00
Gavi Hirsch
611f0c0a60
Merge pull request #56 from Clever/INFRANG-3918 
clear count stats after logging them
 2020-09-11 15:02:06 -07:00
Gavi Hirsch
1112894639
clear count stats after logging them 2020-09-11 14:04:42 -07:00
Taylor Sutton
484b54bfe0
Merge pull request #55 from Clever/INFRANG-35-fluent-decode 
Add decoding of JSON logs in Fluent format.
 2020-08-18 08:44:17 -07:00
Taylor Sutton
f5ce6fe4e7 Add decoding of JSON logs in Fluent format. 
ParseAndEnhance used to be:
- Try to parse line as a syslog, extracting the log itself and other
  fields from syslog format
  - If that succeeds, try to parse the log as either a Kayvee log or
    an RDS slow query log.
  - Combine all these fields, and add on some "derived"
    fields (container_task|env|app).
- Not a syslog => error

Now it will be:
- Try to parse line as a syslog, same as before, including the
  Kayvee/RDS part
- If syslog parsing failed, try to parse as a Fluent log and extract
  some fields from the Fluent format (the log, timestamp, etc)
  - If that succeeds, try to parse the log itself as a Kayvee log.
  - Combine Kayvee fields (if found) and derived fields)
- If BOTH formats fields, it is an error.

The decoding makes a lot of assumptions:
- The names of the log field and timestamp field (even though,
theoretically, they are customizable in the fluentbit config.
- The timestamp format (again)
- The format of the Task Definition name (or at least part of it)
- All fluentbit logs should have hostname set to `aws-fargate`.

Perhaps these can be relaxed if necessary. They could probably be
replaced by some kind of config. As there is currently no config I
wanted to keep things simple as possible. If we need to re-evaluate
(for example if we start getting JSON logs that don't want to use the
same handling for container_task|env|app) we can reevaluate.
 2020-08-17 15:02:59 -07:00
Gavi Hirsch
9b32d93d1c
Merge pull request #54 from Clever/upgrade-kayvee-go-6.24.0 
upgrade kayvee-go to 6.24.0
 2020-08-14 19:19:26 -07:00
Gavi Hirsch
3bedc65483 upgrade kayvee-go to 6.24.0 2020-08-14 16:30:08 -07:00
Gavi Hirsch
b16af062fb
Merge pull request #53 from Clever/glue-2.0 
glue log group format
 2020-08-11 16:52:03 -07:00
Gavi Hirsch
30d3925119 glue log group format 2020-08-11 16:42:59 -07:00
Gavi Hirsch
ebd1e0e39c
Merge pull request #52 from Clever/dependabot/dep/github.com/stretchr/testify-1.6.1 
Bump github.com/stretchr/testify from 1.1.4 to 1.6.1
 2020-07-24 12:21:48 -07:00
Gavi Hirsch
bfe1a6ffb9 fix Gopkg.lock 2020-07-24 12:19:30 -07:00
dependabot-preview[bot]
35b07d7aed
Bump github.com/stretchr/testify from 1.1.4 to 1.6.1 
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.1.4 to 1.6.1.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.1.4...v1.6.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-07-24 17:38:31 +00:00
Gavi Hirsch
8a8b44112d
Merge pull request #51 from Clever/dependabot/dep/gopkg.in/Clever/kayvee-go.v6-6.23.0 
Bump gopkg.in/Clever/kayvee-go.v6 from 6.6.0 to 6.23.0
 2020-07-23 10:27:58 -07:00
dependabot-preview[bot]
8be774bc09
Bump gopkg.in/Clever/kayvee-go.v6 from 6.6.0 to 6.23.0 
Bumps [gopkg.in/Clever/kayvee-go.v6](https://github.com/Clever/kayvee-go) from 6.6.0 to 6.23.0.
- [Release notes](https://github.com/Clever/kayvee-go/releases)
- [Commits](https://github.com/Clever/kayvee-go/compare/v6.6.0...v6.23.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2020-07-23 17:24:47 +00:00
Rafael
504ebfad60
Merge pull request #49 from Clever/INFRANG-3968 
support parsing AWS Glue job logs
 2020-07-20 14:15:14 -04:00
Rafael Garcia
420ad243a4 add codeowners 2020-07-20 18:11:27 +00:00
Rafael Garcia
337d2063f5 support parsing AWS Glue job logs 2020-07-20 17:20:39 +00:00
Sayan
eb11747434
Merge pull request #48 from Clever/update-jar-link 
Fix maven repo link
 2020-04-06 13:50:16 -07:00
Sayan Samanta
d489b5039b update to repo1 link for jars 2020-04-06 13:31:10 -07:00
Taylor Sutton
e7db80a35a
Merge pull request #46 from Clever/go1.13 
Upgrade go to v1.13
 2019-10-18 11:36:05 -07:00
Taylor Sutton
5132214945 Upgrade go to v1.13 2019-10-18 09:45:22 -07:00
Daniel Xu
e265494afb
Merge pull request #45 from Clever/INFRA-3664 
Fix regex that parses the RDS slowquery user
 2019-10-14 17:21:53 -07:00
Daniel Xu
69d31088f3 Add tests for parsing slow query user 2019-10-14 17:19:12 -07:00
Daniel Xu
7446627fae Fix regex that parses the RDS slowquery user 2019-10-14 17:02:01 -07:00